Why do We Need Large Batchsizes in Contrastive Learning? A Gradient-Bias Perspective
Authors: Changyou Chen
Jianyi Zhang
Yi Xu
Liqun Chen
Jiali Duan
Yiran Chen
Son Dinh Tran
Belinda Zeng
Trishul Chilimbi
Venue: NeurIPS 2022 Accept (2022)
Keywords: Bayesian data augmentation
contrastive learning
representation learning

Abstract: Contrastive learning (CL) has been the de facto technique for self-supervised representation learning (SSL), with impressive empirical success such as multi-modal representation learning. However, traditional CL loss only considers negative samples from a minibatch, which could cause biased gradients due to the non-decomposibility of the loss. For the first time, we consider optimizing a more generalized contrastive loss, where each data sample is associated with an infinite number of negative samples. We show that directly using minibatch stochastic optimization could lead to gradient bias. To remedy this, we propose an efficient Bayesian data augmentation technique to augment the contrastive loss into a decomposable one, where standard stochastic optimization can be directly applied without gradient bias. Specifically, our augmented loss defines a joint distribution over the model parameters and the augmented parameters, which can be conveniently optimized by a proposed stochastic expectation-maximization algorithm. Our framework is more general and is related to several popular SSL algorithms. We verify our framework on both small scale models and several large foundation models, including SSL of ImageNet and SSL for vision-language representation learning. Experiment results indicate the existence of gradient bias in all cases, and demonstrate the effectiveness of the proposed method on improving previous state of the arts. Remarkably, our method can outperform the strong MoCo-v3 under the same hyper-parameter setting with only around half of the minibatch size; and also obtains strong results in the recent public benchmark ELEVATER for few-shot image classification. 

Why do We Need Large Batchsizes in Contrastive Learning? A Gradient-Bias Perspective
Changyou Chen
Jianyi Zhang?
Yi Xu
Liqun Chen
Jiali Duan
Yiran Chen
Son Dinh Tran Duke University
Belinda Zeng
Trishul Chilimbi Amazon
University Buffalo
Abstract
Contrastive learning (CL) has been the de facto technique for self-supervised repre- sentation learning (SSL), with impressive empirical success such as multi-modal representation learning. However, traditional CL loss only considers negative samples from a minibatch, which could cause biased gradients due to the non- decomposibility of the loss. For the first time, we consider optimizing more generalized contrastive loss, where each data sample is associated with an infinite number of negative samples. We show that directly using minibatch stochastic optimization could lead to gradient bias. To remedy this, we propose an efficient Bayesian data augmentation technique to augment the contrastive loss into de- composable one, where standard stochastic optimization can be directly applied without gradient bias. Specifically, our augmented loss defines a joint distribution over the model parameters and the augmented parameters, which can be conve- niently optimized by proposed stochastic expectation- maximization algorithm. Our framework is more general and is related several popular SSL algorithms. We verify our framework on both small scale models and several large foundation models, including SSL of ImageNet and SSL for vision-language representation learning. Experiment results indicate the existence of gradient bias in all cases, and demonstrate the effectiveness of the proposed method improving previous state the arts. Remarkably, our method can outperform the strong MoCo-v3 under the same also obtains strong results in the recent public benchmark ELEVATER for few-shot image classification.
expectation-maxiiizaaioo
setting with only around half of the minibatch size; and
Introduction
Recent years have seen significant developments and applications of contrastive learning (CL), promising self-supervised learning (SSL. technique, to a variety of striking research breakthroughs such as learning of multi-modal foundation models [5] in text and image domains [46]
[3 67 for
natural-language and laiiunla representation learning. In addition, CL has also be applied
recently to other scenarios such as supervised learning [ and reinforcement learning [ K6 The
main idea of CL is define a contrastive loss where each data sample is entangled with both its positive and negative data samples, and optimize it to learn transformationninvariann features to better discriminate positive samples from negative ones. From a technical perspective, the contrastive loss is non non-decomposable, in the sense that each data sample is coupled with all the other samples in the loss function, i.e., to process one data sample, one has to get access to all its positive and negative samples from the other data samples. This is intractable when considering more general CL setting where the number negative samples are unbounded (potentially infinite). this case, if stochastic optimization is applied, only negative samples in the current minibatch are available. Thus, this
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

could lead to biased gradients and consequently sub-optimal solutions (see Section 2.1 for a formal
statement), which is also evidenced in several existing works 22 24
straightforward approach to overcoming the issue is to define some decomposable loss functions for learning. For example, based spectral theory, 2 proposes spectral contrastive learning, which defines the loss as the difference between positive and negative similarity scores. Furthermore, and SimSiam [T deal with this problem by completely dropping the negative samples. These methods, however, are developed from quite the standard contrastive learning, e.g., SimCLR that adopts the info-NCE loss [T]. Since standard contrastive learning has been shown to be relatively robust and easier to scale up to large foundation models [5] such as CLIP and its variants 4. 600 63 67]. it is important to study its equivalent form in terms of a decomposable loss, and understand what benefits can be brought by this new form to further boost model performance.
negative-sample-free SSL methods such as BYOL A
different perspective and are not equivalent
To this end, we propose an equivalent conditional decomposable form of the standard contrastive learning, by leveraging the well developed Bayesian data augmentation technique from the Bayesian learning community A.. The basic idea of Bayesian data augmentation is to augment a difficult- to-handle distribution to an easier one by augmenting the parameter space with auxiliary random variables, also known as auxiliary data (39] as each of them is associated with one data point. When applying it to contrastive learning. each positive data pair will be associated with an auxiliary variable/data allowing one to define a joint distribution over the model parameters and the auxiliary data. Conditioned on the auxiliary random variables, the potentially infinite negative samples will be separated from each other without introducing bias, thus making decomposable loss (details in Section 2.2.2). We call our model Decomsable Contrastive Learning (DeCL.. To learn from the proposed decomposable contrastive loss, we then propose a new stochastic (sEM) framework for maximal likelihood estimation (MLE) of the joint distribution. Specifically, with the proposed augmentation technique, the auxiliary random variables enjoy simple Gamma posterior distributions, conditioned on which the model parameter can be easily optimized by stochastic gradient descent. To summarize, our contributions include:
We propose an equivalent augmented form of the standard contrastive loss from a probability perspective, which enjoys conditional decomposibility for unbiased stochastic optimization.
We propose an efficient sEM algorithm for MLE of the joint distribution over model parameters and augmented data, which only needs minimal modifications based on various contrastive- optimization procedures such as SimCLR B and MoCo-v3 [14].
We verify our framework with extensive experiments in large-scale foundation models, including the uni-modal ImageNet and multi-modal vision-language representation learning, well as public benchmark ELEVATER for few-shot image classification [37] Experi- mental results suggests that, with the ability of gradient-bia bias mitigation, our framework can improve the current CL-based state of the arts.
2 Decomposing Contrastive Learning
2.1 Setup& & Gradient Bias in Contrastive Learning
In CL, a backbone network, parameterized by 6, is used for feature extraction/encodingg Given an input x. the feature extractor outputs its representation, written as z = enc (x; 0). The basic idea of CL is to maximize the representation similarities between positive data pairs while minimizing
those of negative pairs. Formally, we are given an unlabeled dataset D & {x, ). Each x; E D
will be applied a set of transformations to form an implicit augmented dataset D A {(x;, {x]) T is a sampled transformation function from the transformation family T. A positive pair is represented as (x,, xx)) for Va; and a negative pair is represented as (x, or xf) for # j. For both positive and negative pairs, the data points will be fed to the encoder and similarity scores will be calculated based on their representations. For notation simplicity, we will index positive similarity scores by 'j+t and negative similarity scores by k= e-g., we use stzt e (en 0), enc(x;; 0)) to denote the similarity score between x; and its j-th positive data point from {x²) }. where sim(...) denotes a similarity metric (positive value) such as
where xf ~ t(x,) and
WWe will use auxiliary (random) variable and auxiliary data interchangeably without distinction, as they are equivalently used in the Bayesian data augmentation community.

---

the exponential cosine similarity used in most contrastive learning methods (which we use unless explicitly stated); similarly, s used to denote the similarity score between x; and one of its negative data point. Note the similarity scores depend on 0, but is omitted in the representation for notation simplicity.
theory, each data point x, should be associated with an infinite number of negative pairs due to the availability of arbitrary transformation functions. Thus, we consider one of the most general form of contrastive loss with N; positive pairs and infinite negative pairs for each X;, formulated as
Lonn (D:0)
1
Σ
log (x;), with Cx.
1
Σ Sizt Ek= SiA
Sij
[D]
x;ED
practical setting is to consider one single positive sample in each Lx., i.e,, N; = giving us a Fix-" where the positive similarity score written as 8,t for notation simplicity. We will consider this simpler case in our following development, although generalizing it to the more general case of N; positive pairs is straightforward.
slightly simpler sub-loss Lx; -
Gradient Bias Note the general CL loss ① is in- tractable for direct optimization due to the infinite number of negative pairs for each data point. For practical feasibility, a common trick is to replace the infinite negative pairs with only those within a minibatch. This gives us an approximation of Lx; as Ex, A *,t + P/n- where B denotes the mini-
1.0
0.5
Loss
0.0
0.5
Ours
batch size. We argue that this approximation bi- ased due to the non-decomposibilliy nature of the standard CL loss Loon. in the sense that the individual loss Leon is associated with the negative samples that distribute across the whole data D. In other words, the overall loss Leom cannot be decomposed into a sum over independent sub-losses. Importantly, non- Lx and
Direct- Opt
1.0
10
30
50
Iteration
Figure 1: Trajectories of 10 random runs. Direct optimization (Direct-Opt, blue) fails to converge with heavy noise; whereas our method (red) can handle the noise and con- verges to the same optimal solution.
decomposibility indicates that Lx; * E
consequently, VoLx, * VOE [x.] where both the expectations are taken over the randomness of minibaching. Thus, using negative samples from a minibatch can cause gradient bias with minibatch stochastic optimization, leading to sub-optimal solutions. We demonstrate the problem with simple synthetic experiment, where we try to optimize the simplest CL form of e"1 /(e" + e*a) over 8j and
82. We inject noise into the denominator to mimic stochastic optimization, resulting in a stochastic objective of e e+ /(e" + 2² + &e) in each SGD iteration, where á is a random zero-mean Gaussian to 0.2. We compare direct optimization with SGD and our method (introduced
noise with variance below) on 10 random runs, starting from different initializations. The learning curves are plotted in Figure I It is observed that direct optimization diverges on different runs due the high injected noise, whereas our method can always converge to the same optimum with loss = 1. To further demonstrate the consequence of gradient bias, we inject noise of variance 0.1 and use a learning rate of 0.2 (other settings lead to similar conclusions) to run optimization until converged (around 500 steps). To guarantee& a unique optimal solution, we introduce regularizer of 0.1 (s + s2). At convergence, our method gives
2.97, 0.34, whereas direct optimization gives $1 = 0.15, 82 = 0.04, indicating a sub-optimal solution from direct optimization. We include more experimental results in
Appendix L
In the literature, common practice mitigate the gradient bias problem use large batch sizes, which has been demonstrated to be necessary by existing works 13. Although there are also some preliminary works trying mitigate the negative impact of minibatches 6 4 64]], whether they can mitigate gradient bias is not totally clear. Our method deals with this problem from a principled Bayesian perspective with new efficient Bayesian data augmentation technique.
2.2 Decomposable Contrastive Learning (DeCL)
2.2.1
The Overall Algorithm
'One can also consider infinite positive pairs, which also be naturally fit into framework.
3

---

We first present our over- all algorithm, then detail the derivations in the fol- lowing sections. Our algo- rithm is illustrated in Al- gorithm !I which is very simple and intuitive. The algorithm mainly consists of the following steps: 1) Load a minibatch, and cal- culate the similarity scores
Algorithm 1 Decomposable Contrastive Learning (DeCL)
1: # enc(:): encoder backbone to extract features
2: # (h,,""",,,,"")) A decreasing sequence in [0, 1]
3: t=
4: for x in loader do
load minibatch with B samples 5: X1, X2 aug(x),au(xx)
> augmentation
Calculate positive/negative similarity scores
(s+); A 85t (vector) and (ST).. _ 8ik- (matrix)
7: Estimate auxiliary variable/data U (vector) as
described in Section 2.2.3
(same as standard CL); 2) Estimate the so-called aux- iliary data U, which will be used to adaptively weight the negative data in the minibatch (see loss_ 1) 11:
8: loss_1 = (U mean(dim = -1) - log(s*)) )..ttanmmmme)) .eaa())
> decomposable contrastive loss
9:
loss_2 = (log(S .sum dim = - 1)) - log (sst)) )).t)a)mmm)n))) >
a smoothing loss induced from the stochastic EM principle
10:
loss = X x loss__ + (1 X₂) x loss_2
loss. (
update the encoder backbone
achieve gradient bias elimi- nation; 3) Construct the to- tal loss, which consists of a decomposable loss loss and smoothing loss loss 2 (coming from the stochastic EM algorithm described below), which we reformulate so that it is equivalent to the standard minibatch info-NCE loss 13: : 4) Optimize the loss to update the encoder backbone.
12:
1=1+1
13:
end for
2.2.2 Decomposing Contrastive Loss with Bayesian Data Augmentation
The basic idea is to leverage Bayesian data augmentation techniques 41 to disentangle the infinite negative similarity scores in the denominator of ( Originally, Bayesian data augmentation augments a complicated target distribution @ (xx) (e.g., g., some posterior distribution of interest) to a more well- behaved augmented distribution p(x, u) with an auxiliary random variable u, one can then directly deal with the augmented distribution p(x, u) instead of the original p(x). For example, instead of maximizing p(x). one can maximize p(x, u) w.r.t. x instead. This will yield equivalent solutions while obtaining the convenience of dealing with an easier distribution p(x. u). One important requirement to guarantee the augmented problem equivalent to the original one. In other words, the augmented distribution must satisfy: 1) the marginal distribution should recover the original distribution by marginalization, i.e., Ju p(x, u u)duu = p(x); and 2) the augmented distribution much easier to handle, e.g., for MLE.
We adopt this general idea to augment the contrastive loss and make it conditional decomposable. In other words, our goal is to construct an augmented objective, denoted as Lecon (D, U; 0) with auxiliary data U, such that EulCacom U): f Leon 0). and Lacon (D, U: 0) is expected to decouple negative samples given the auxiliary data U. The main challenge then comes to dealing with the entangling denominator of Lx.- This can be achieved by applying the Gamma identity, i.e., Specifically, we augment the denominator in the original contrastive loss into an exponential form, and defines a joint distribution p(0, U (D) over the model parameter and the augmented data U & {u.}i. where each u, is associated with a data sample x; E D:
ga f e
P(B.U & II
8 U.
II
(2)
EX.D
k
Theorem 1. The joint distribution is valid augmented distribution, i.e., marginalizing the up to some constant.
augmented variable U recovers the original contrastive loss (
Constructing a Decomposable Loss With the probability interpretation of the contrastive loss in P we can apply maximal likelihood estimation to optimize the model parameter O. Note the augmented random variables have simple Gamma conditional distributions, which can be easily sampled from. To this end, we propose an effective approximate method via stochastic expectation- maximization to optimize the joint distribution in Section 2.2.3 In the following, we first derive
'The x and the following p(x x) and p(x, u) are a bit different in meaning from those in our contrastive-eerning setting. These notation only used in this paragraph to explain the idea of Bayesian data augmentation.

---

the decomposable loss from 2 Suppose there are K negative samples for each data sample x,, where oo represents the population- setting. Note by scaling u, to K u, 2 is equivalent to p(0, U (D) x EEE 8,+E u. i *;+ II e e- au_ Conditioning on U and taking negative logarithm on both sides of p(0, U (D) give us new augmented loss:
K
)
Ldoon
Σ EX.E
u,
K
8/1
K
Σ Sik
log(s,))
Σ 1X
u, E
1iik
] log(s; ) (3)
where the expectation is taken over the distribution of negative samples of x,.
On Decomposibility Following the literature such as 31 we define decomposable loss as a loss LDD such that L(D) = -xx (X,) for some dataset D, Le., the sub-loss L; only depends on the single data point x, To prove our loss 3 is decomposable for a given U, we first construct an augmented dataset D' _ {xi, x x from the original dataset D - {x,} and the data augmentation family T as follows: 1) Randomly sample data sample x; D; 2) Randomly sample a transfor- mation function t ~ T and apply it to x; to generate a positive sample x = t(x,): :: 3) Randomly sample another data sample x' ~ D excluding x;; 4) Randomly sample transformation function
y'~77 T and apply it to x' to generate a negative sample x; = ='(x'.. ). From the construction, it clear
that each data points x! A (x:, x,,,x,+ x; ) in the new augmented dataset D' are identical independently distributed, and thus 5 is equivalent to L' EixçED (u, 8;a- log(s;+) )) - EixeEe C under the K + oo and infinite data limit. Since it easily verified that the sub-losses CJ's are independent from each other, our new loss ( is thus valid decomposable loss. Consequently, conditioned on U, one can safely apply stochastic optimization, where, at each iteration, a random miníbatch of data sampled from D' (which can be constructed from a minibatch of the original data D) and used to evaluate the stochastic gradients from B for parameter updates. To conclude, we have:
Corollary 2. Given the auxiliary variable U, the new contrastive loss ( decomposable. And thus, using minibatches for stochastic optimization maintains unbiased stochastic gradients.
2.2.3 Solving by Stochastic Expectation-Maximizaate
We adopt the MLE principle on the augmented joint distribution for model learning. Because the augmented variables U are local random variables that are associated to the original data, it is unwise to directly optimize them in the same way as the global model parameter O. We advocate that stochastic weweetaaioonmaxiizzzatiii distribution of U follows a simple Gamma distribution, which allows one to easily get samples from. There are different variants of sEM algorithms (2) 17 We adapt the method introduced in (2), and propose an efficient sEM variant to solve our problem. Specifically, based on our decomposable form of the contrastive loss, we define a joint distribution p(0, U (D') over (0, U) based on the augmented dataset D'. Since D' is of infinite size, we will apply stochastic optimization to optimize log U (D)) in each iteration 6, where Dj is a minibatch randomly sampled in this iteration. Standard stochastic optimization theory implies such a stochastic approximation step would converge under some standard conditions [ The problem now reduces to optimizing log p(0, U (D)) each iteration. To this end, we adopt the sEM algorithm proposed in [2] which is used to maximize a joint likelihood of a latent variable model relying only on samples of the latent variables. Specifically, with our notation and indexing iteration, the algorithm consists of the following steps in each iteration: 1) Simulation: sample the auxiliary data U, from their approximate posterior distribution; 2) Stochastic approxiation: update stochastic objective as (+) (.) + (log U, (D') Q()));; The sequence { is user-specified, and we recommend X; 1/1; 3) Maximization: maximize Q++1 (0). Based on the general algorithm, we make a few adjustments of this basic framework to adopt to our setting, and elaborate on the three steps in the following:
(sEM) is natural fit to this problem, the conditional
This corresponds to simulating the auxiliary data U, for the current minibatch. After scaling u, by K and taking the K + oo limit on @ it is known that the posterior distribution of u, follows: p(u; |0, D) Gamma(1, [B(= Unfortunately, the exact value the rate parameter E[sa-] is intractable, hindering exact dimulation from its Gamma posterior. Thanks to the theory in
Simulation
2 Theorem 3], if one can construct sequence of approximate distributions (ğ) (u;). .... . (u.)..... such that the sequence converges in mean to p(u, 00, D), then sampling from these (ğe (u.) can
5

---

also gaurantee convergence of the algorithm. To this end, we propose two methods to construct sequence of rate parameters to approximate E[s of the Gamma posterior p(u, |0. D): 1) Maintain global buffer to store the average similarities score K Σx 8ak- for each u, which is then updated by moving averages over the minibatches; 2) Directly use the momentum network to calculate the average similarity scores from the current minibatch to approximate the grund-true expected similarity scores. Note this strategy can be directly applied to models with a momentum network such as MoCo [4] without any architecture changes. In both ways, the rate parameters of the Gamma posteriors are expected to converge to the expectation of the corresponding similarity scores.
Stochastic Approximation According to the description above, the stochastic objective contains two terms: (0) and p(0 U, (D:). The latter endows convenient form, which has the same expression as B but with expectation evaluated on the current minibatch. The term (:(0) has the same form as log p(O, Ug (D)), but with statistics taken from previous iterations (Loss | in Algorithm D For optimization convenience and algorithm stability, marginalize out the auxiliary data U from previous iterations, giving us the same info-NCE loss evaluated on a minibatch (Loss_ in Algorithm D. as in standard contrastive learning. Although did not investigate different versions of x sequence, in practice, we find performing alternative optimization between Loss_ I and Loss_ is beneficial, e.g., we alternate between one step optimization of Loss_ I and one step optimization of Loss_2 2 in the Open-CLIP experiment; and simply directly optimize over Loss_I without the smoothing loss Loss_2 2 in the MoCo-v3 with ImageNet experiments.
Maximization Once (+) is constructed, which is in the form of a weighted average between the info-NCE loss and logp(0, (0, U, (Di) as explained above, optimization can be done in the same way as the standard CL, e.g., by adopting the same optimization algorithms and hyperparameters.
More detailed explanation and derivations of the proposed sEM algorithm can be found in the Appendix C Compared to standard contrastive learning, our method has an extra step of estimating the auxiliary variables U. This additional computational cost for U is negligible relative to the computation of other parts, e-g., the haok-propag step. Empirically, we also observe almost the same computational speed of our method compared to the standard contrastive learning.
2.2.4 Generalization Bound
By extending the techniques in 2 we study the finite sample generalization bound of the proposed decomposable contrastive loss. Let F. be a hypothesis class containing feature extractors from the data space X' to feature space R Similar to (27]. we extend Rademacher complexity (50 to function classes with high-dimensional outputs, and define the Rademacher complexity of F. on n data as
(F) = max.r.... Ex Ex suPpeFFAe[k] + (Σj o,fi (. where o is a uniform random vector in 13" and f=(z) is the i-th dimension of f(2). Note in CL, the function f is typically normalized, meaning (F) is controllable, e.g., it is typically less than or equal to one.
For conciseness, we slightly overload some of the previously defined notation. Let fpup E F be minimizer of the loss Loon(f (f) on infinite data, called population loss; and let Lacon denote the loss of G on finite dataset D, called empirical loss. Since Lacon an unbiased estimator of the population loss Lcons we can derive generalization bounds via off-the-shelff concentration inequalities.
Assumption 1. (Realizabilitt) At least one of the global minima of Ecom()) belongs to F.
The following theorem bounds the population loss of a feature extractor trained on finite data. Theorem 3. Assume [z) S for some K 0 and for all f F, X. Let fpop E F be a minimizer of the population loss Lean(f). Given a random dataset of size n, let femm E F be a minimizer of the empirical loss Ldcon. Denote a S b as a < Cb for some constant C. Under Assumption / with probability at least 1 - 8 over the randomness of data, the generalization loss is bounded as
(V
Leon (Jamp) S Leon (Spop) + +c R1/2(F) + C2
2/6 +
where constants C1 & kx) S kx2/3 exp(3) k is the featue dimension and B is the commonly used temperature parameter in the similarity score M1 14 12).

---

3 Related Work
We discuss the most related research in this section. more complete list of related works provided in Section A in the Appendix.
Spectral Contrastive Learning and Related The recently proposed spectral contrastive learning loss 277. with our notation, can be written as: - Lexte (#4k- 28,+) ). The attractionreepulsion
CL also shares similar objective [66]. The main difference of ours and theirs lie the scaling of similarity scores: our method scales positive similarity scores with logarithm, and has the mechanism of adaptively weighting the negative data. Moreover, our framework is theoretically equivalent the standard contrastive loss, whereas the above two are not.
SimCLR 13
SimCLR is an instance of CL based on the standard info-NCE contrastive loss. Since our method is independent of such specific implementation of CL, it can be directly applied to SimCLR by replacing the loss. We will demonstrate our method within SimCLR in the experiments.
MoCo The MoCo family, especially the recent MoCo-v3 [4], represents a strong baseline for contrastive learning. Similar to the SimCLR case, we can also directly replace the original MoCo contrastive loss with our loss, which will also be comprehensively studied in the experiments.
Negative-Sample-Fre SSL Another line of research toward SSL is via negative-sampi samp-s--mmaeeee methods, where only positive samples are considered, e.g., BYOL [4 and SimSiam [T Our framework can be considered as the contribution of nagative samples are adaptively weighted by the auxiliary random variables U. Especially, when the expected negative similarly scores become smaller (negative samples are far away). the adaptive weights will asymptotically approach zero and thus become negative-sample free.
bridge between contrastive and non-contrstive learning methods,
Improved Contrastive Learning There have been many efforts trying to improve contrastive learning from various perspectives, e.g., [6 modifies the info-NCE loss to mitigate the robustness of the loss to the minibatch size; [54] shows the necessity of large batch size in contrastive training and proposed an improved method; 12 identifies the log-K curve of contrastive learning and proposes an improved FlatNCE loss for fix; Other efforts include the SimSiam [66, relative predictive coding 1561. Wasserstein predictive coding (88. Barlow Twins [641. VICReg (4, gradient catching (20), and etc. All these methods do not explicitly address the gradient bias problem. There are also some previous work considering better ways sampling negative samples [22 &I 48 but they still face the gradient-bias problem. Our method provides a principled way to decouple the negative samples with Bayesian data augmentation, enabling an effective solution to mitigate gradient bias.
Experiments
We first demonstrate the robustness of our method against gradient bias on small-scale problems, and then test it on large models for single-modal image and multi-modal vision-language representation learning. All the experiments are conducted on a NVIDIA A100 GPU Server. All hyperparameters are adopted from the public codebases specified below without further tuning, although we believe more improvement can be achieved by further tuning the hyper- sperrpprameters, which is left as interesting future work. Our goal of experiments is to demonstrate the effectiveness of our method to mitigate gradient bias in contrastive learning, but not to achieve state-of-the-ars results. Thus, we do not compare our method with other con-contrastive learning methods, although we include some results in the Appendix. Code will be publicly available after internal reviewing and approval.
4.1 Robustness against Similarity-Score Noise
To illustrate the impact of similarity- score noise, we explore two ways to add score noise: one manually perturb similarity scores and the other adopts the intrinsic noise induced from using minibatches.
Manual Similarity-Score Noise We gradually add Gaussian noise to the similarity scores and monitor the performance changes. We compare our method with the popular SimCLR baseline ([3). where our model uses exactly the same network architecture as SimCLR. We conduct experiments on the popular CIFAR-10 10 dataset, and implement our method based on the publicly available codebase 47 where we strictly follow the default hyper-parameter setting without any tuning. We fix the minibatch size to 128 while varying the variance of the added Gaussian noise. The results are
plotted in Figure
It is clear that with the noise variance increasing, the validation accuracy of
7

---

Table 1: Validation top-1 accuracies (%) on the CIFAR- 10 and STL datasets.
CIFAR-1B R-10
Normalized Feature
Unnormalized Feature
Batch
64
128
256
512
64
128
256
512
Epoch
| 300
200
300
200
300
|
300
200
|
200
300
200
300
200
300
SimCLR
79.6
| 81,88
81.8
83.4
83.8
85.8
84.9
86.8
73.9
74.5
75.8
76.4
76.1
76.7
77.2
77.5
DcCL (Ours)
83.8
85.9
85.7
87.2
86.4
87.6
86.4
87.7
79.9
81.1
82.2
84.1
83.4
85.4
83.5
84.4
Normalized Feature
Unnormalized Feature
STL
Batch
64
128
256
512
64
128
256
512
Epoch 200
|
300
200
300
200
300
200
|
300
200
300
200
300
200
300
200
300
SimCLR
69.8
69.7
72.1
72,9
75.6
73.7
77.4
78.1
70.6
70.2
71.8
72.1
73.8
72.8
74.1
74.2
DeCL (Ours)
71.7
73.0
75.0
74.4
77.6
78.3
79.5
81.2
75.8
75.0
75.0
77.7
78.8
79.5
79.8
79.4
SimCLR drops much more significantly than ours, demonstrating the robustness of our method to similarity-score noise.
90
Intrinsic Similarity-Scone Noise Next, inves-
tigate the impact of similarity-score noise inherited
Top-1 Accuracy
80
from minibatches. In addition to the commonly used architecture for contrastive learning, where a normal- ization step is applied at the output of the backbone encoder network to project the feature representation onto hyper-sphere, we additionally test variant by removing the normalization step. The reason is that we find the normalization can make the impact of similarity-scor noise less prominent because the rep- much more compact
70
Ours (300eps)
60
Ours (400eps)
SimCLR (300eps)
50
SimCLR (400eps)
40
0.05
0.1
0.2
0.3
0.4
0.5
Noise Variance
resentations are constrained sub-space. When removing the feature normalization. we adopt exponential of negative Euclidean distance as the similarity metric. We compare our method with SimCLR on the CIFAR- 10 and STL datasets,
Figure 2: Top-1 accuracy (%) with increasing similarity score noise CIFAR-10.
and follow the default paree--rrrmtterrr settings as detailed in the codebase 47 We vary the mini- batch sizes along with different pretraining epoches. The results are shown in Table It observed that our method significantly improves over SimCLR in all cases. Particularly, the performance gap without feature normalization is more prominent, probably because the architecture is more vulnerable to similarity-scon noise, as explained above. Note the Github codebase 47 gives top-1 accuracy of 89. 1 % with minibatch of 512 and epochs of 500. For directly comparison, run our method with the same setting, which gives 89.9%, further demonstrating gradient bias in the large-batch setting.
We further compare our method with existing SSL methods including popular contrastive and non- contrastive learning methods. The results are presented in Table in the Appendix. In general, our method outperforms existing methods most case, especially when the batch sizes are small. Please refer to Section F in the Appendix for more detailed experiments
4.2 Single-Modal Image Representation Learning
We then evaluate our framework on foundation models for large-scale image representation learning. We adopt Mo [14], one of the recent state-of-the-art method, as our baseline. We strictly follow the default settings as the publicly available codebase [15] without further tuning, except replacing the original contrastive learning loss with our method. We run the models on the public ImageNet-1 dataset, and test it with different minibatch sizes, ranging among 128, 256 and 512. Note the reported results in the original paper are based on very large batchsizes such as 1024 and 4096, which were not able to test them due to the time and computation constraint. But we do fine our method can match those results with smaller batch sizes. Similar to the original paper, we test our method with different encoder backbones including ResNet50, ViT-Small and VaT-Base. For evaluation, we finetune the pretrained models for downstream image classification the ImageNet- 1K validation data. We
$ We did not perform experiments on the large ImageNet dataset due to the huge computational cost and the challenges in making fair experiment setting. However, some public results (e.g.. https://paperswiihcodee.c com/so/asll/////llieii.eo suppriseeee supervised-image mamage-lassifiiationooo indicate the superiority of contrastive learning methods over contrastive ones, and our method can improve over standard contrastive learning.
8

---

Table 2: Results of Pretraining on ImageNet-IK. 'a/b/c" in the KNN column are accuracies (%) with k 10, 20, 100, respectively. Results indicated by "MoCo-v3 are from [15].
Batch Size: 256
Batch Size: 512
Average Linear
Methods
kNN
Linear
kNN
Linear
MoCo-v3
49.7/50.0/48.7
59.5
56.6/57.0/55.8
68.2
63.85
ResNet-50
DeCL (Ours)
64.2
56.1/56.5/55.5
67.7
65.95
MoCo-v3
50.0/50.6/49.1
62.3
54.6/54.9/53.5
66.0
64.15
ViT-Small
DeCL (Ours)
52.0/52.5/50.7
63.4
57.6/57.8/55
66.4
64.90
MoCo-v3
59.1/59.5/57.9
68.1
61.5/61.5/6000
70.1
69.10
ViT-Base
DeCL (Ours)
60.7/61.1/59.4
69.1
61.4/61.8/60.4
70.3
69.70
DeCL (Ours): Batch_ Size-Siiee 2304
MoCo-v3 [4]: Batch Baich_Sizee 4096
ViT-Base
kNN: 69.7/70.0/68.9
Linear: 76.8
kNN: 69.3/69.2/67.8
Linear: 76.7
also test the generalization ability of the pretrained models by conducting transfer learning on the CIFAR-10, CIFAR-100, Flowers and Pets datasets, funetuning from the pretrained checkpoints.
Validation on ImageNet We evaluate the performance on ImageNet in terms of kNN and linear probing classification accuracy different checkpoints. For kNN evaluation, we follow previous work [8] by simply performing k-nearest neighbor classification in the embedding feature space, where k is set to 10, 20 and 100, respectively. For linear probing, we following [4]] by fixing the pretrained weights of the backbone network and supervised finetuning an added linear layer. We adopt the standard top-1 and top-5 accuracies on the validation data. The final results summarized in Table 2 It is observed that our method outperforms MoCo-v3 all cases. Remarkably, on the ViT--Bse/16 16 atchitecture, it slightly outperforms the CL-based art, achieving top-1 accuracy of 68. 5% and 76.8%, respectively, in terms of KNN and linear probing, but with around half of the minibatch size compared to previous (67.8% and 76. of MoCo-v3 [4 I5)). Note only the last row in the table is directly comparable to the numbers in the official Github. All the other results use smaller batch sizes, thus are expected with lower accuracy.
Transfer Learning We transfer the pretrained checkpoints and prepare the datasets by strictly following the instructions from the MoCo-v3 website [15. Similarly, we use the default settings for the experiments. The results are summarized in Table Consistent with previous findings [14]. due to the sizes of these downstream tasks, the performance gaps between different methods are not as significant as other comparisons. Nevertheless, with only around half of the batch size, our method perform similarly with previously reported state of the arts
Few-Shot Image Classification To further demonstrate the quality of our learned representations, we adopt the recently developed ELEVATER benchmark (37], which performs 5-shot transfer learning 20 public image classifications datasets. Each data set contains 5 randomly selected training samples in each class, and the model is trained for 50 epochs before the test score is reported, Three random seeds are considered for each data set. We deploy the automatic hyper-parameter tuning pipeline implemented in ELEVATER to make fair linear probing comparison of pre-trained models based on ViT-Base. The original metrics of each dataset are used, with more details provided in (37. The results are shown in Table Consistently, our method achieves an average accuracy of 54.7%, outperforming MoCo-v3 by 4.5% on average. More significantly, our results represents a new state of the art among the models pretrained on ImageNet- 1K (to date Oct 7, 2022), outperforming the CACR model [66] with an average accuracy of 54.51%.
4.3 Multi-Modal Vision-Language Representation Learning
Finally, we test our framework for vision-language representation learning by adopting the publicly available Open-Cla CLIP codebase 29]. Similarly to CLIP (4). Open-Clip uses contrastive learning to jointly embed images and the corresponding text. We follow the default setting used in the codebase to train the 3M images of the Conceptual Captions (CC) dataset (57), consisting of 2.89M training images and 13K validation images. We replace their contrastive learning algorithms (variant of SimCLR) with our method. The backbone architecture is the ResNet-50x4. To evaluate the pretrained models, we test them the CC validation dataset, well as applying them for zero-shot evaluate on

---

Table 3: Transfer learning accuracy with different checkpoints, The number appended to the end of the methods represents minibatch size; "a/b'' in our and MoCo-v3 method means top-1 and top-5 accuracies (%). 3M [4]" are results from 14] trained with much larger minibatch sizes.
CIFAR- 10
35
CIFAR 100 35
Flowers 102 44
Pets 44
Average
Random init.
77.8
48.5
54,4
40.1
55.2
Supervised 34
98.1
87.1
89.5
93.8
92.1
MoCo-v3-512
98.3/99.9
88.2/97.6
97.5/99.6
90.9/98.8
93.7/ 99.0
MoCo-v3-4096 [1]
98.9
90.5
97.7
93.2
95.1
DeCL (Ours) 512 -12
98,4/100
89.0/98.0
97.1/99.4
90.8/99.8 8
93 33.8/99.3
DeCL (Ours) 2304
98.9/100.0
90.5/98.4
97.7/99.5
92.7/99.1
95.0/99.3
Table 4: Performance of our method on the ELEVATER benchmark. Data-1 to Data-20 correspond to datasets Caltech101, CIFARIO, CIFARIOO, Country211, DescriTextures, EuroSAT, FER2013, FGVC Aircraft. Food101, GTSRB, HatefulMemes, KITTI, MNIST, Oxford Flowers, Oxford Pets, PutchCamelyon, Rendered SST2, RESISC45, Stanford Cars, VOC2007, respectively.
Dataset
Data-
Data-2
Dara-3
Data-4
Data-5
Data-6
Data-7
Data-8
Data-9
Data-10
Mean Acc.
v3
80.8
78.5
60.5
$7.1
77.1
20.5
11.8
36.6
31.4
50.2
DeCL (Ours)
86.0
87.8
65.4
4.5
55.0
78.8
19.3
22.9
41.3
38.5
54.7
Gains
45.2
+9.3
+4.9
-0.3
+1.7
-1.2
+11.1
+4.7
+7.1
+4.5
Data-11
Duta-12 12
Data-13 13
Data- 14
Data-15 15
Data-16
Data-17
Data-18
Data-19
Data-20
# Wins
MoCo-v3
(Ours)
51.5
44,9
72.9
85.4
78.7
63.7
50.1
63.3
21.0
63.5
16
Gains
+0.8
Table 5: Results on Multi-Modal Vision-Language Representation Learning on Conceptual Captions.
Text Retrieval
Image Retrieval
Val Loss
ImageNet Zero-Shot
Methods
R@1 If
R@5t
R@ 101
R@11
R@
R@ 10
Top-1
Top-5
OpenCLIP
30.0
52.0
60.9
29.7
51.8
60.7
2.02
19.2
37.3
DeCL (Ours)
30.2
53.8
62.6
30.2
53.3
62.3
1.52
19.7
38.3
the ImageNet-IK IK validation dataset. The evaluation tasks on the CC dataset are text-to-image and image-to-text retrievals. Following previous works, we adopt R@ 1, R@5, R@ 10 and validation loss as the evaluation metrics. For zero-shot ImageNet classification, we adopt top-1 and top-5 accuracies. For computational convenience, following [29,, we pretrain the model for 30 epochs. The results are summarized in Table 5 Our method gives consistently better results in all metrics, especially in the @@ 10 score and top-5 accuracy, validating the effectiveness of our method in gradient bias reduction.
5 Conclusion
We investigate the problem of gradient bias in contrastive learning with respect to infinite negative- sample size, by formulating
probability framework with Bayesian data augmentation. An efficient and simple stochastic EM algorithm is proposed, which constitutes a decomposable CL framework. Generalization bound on finite training samples is also developed. Experimental results on various settings, ranging from small to large models and from single to multiple modality settings, demonstrate the effectiveness of our framework. Interesting future works include investigating more accurate methods to estimate the augmented variables and improving on the multi-modal learning setting. Furthermore, one potential advantage of our framework potential for efficient distributed training, especially in the asynchronized setting, one only needs to maintain the auxiliary data without communicating with local data.
Limitation & Potential Negative Societal Impacts This paper identify the gradient bias problem in contrastive learning and proposes efficient and effective solutions to mitigate the problem. However, better approximations are still needed to tackle the problem, egg., better methods to estimate the auxiliary data. Thus, there is still room to improve the proposed method. Finally, as this work is technical in nature and does not include sensitive data, we do not foresee potential negative societal impacts.
10

---

References
[1] LIGHTLY benchmark.
html
Accessed: 2022-08-03.
[2] Stéphanie Allassonnière and Juliette Chevallier, A new class of stochastic em algorithms. escap- ing local maxima and handling intractable sampling. Comput. Stat. Data Anal., 159:107159, 2021.
[3] Andrea Banino, Adria Puigdomenech Badia, Jacob C Walker, Tim Scholtes, Jovana Mitrovic, and Charles Blundell. CoBERL: Contrastive BERT for reinforcement learning. In International Conference on Learning Representations, 2022.
[4] Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Mariane-invarinnc--covaanan regular ization for self-supervised learning. ArXiv, 0ttp22105049006 2021.
[5] Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, S. Buch, D. Card, Rodrigo Castellon, Niladri S. Chatterji, Annie Chen, Kathleen Creel, Jared Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren E. Gillespie, Karan Goel, Noah D. Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong. Kyle Hsu, Jing Huang. Thomas F. Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, O. Khattab, Pang Wei Koh, Mark S. Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir P. Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Benjamin Newman, Allen Nie. Juan Carlos Niebles, Hamed Nilforoshan, J. F. Nyarko, Giray Ogut, Laurel Orr, Isabel Papadimitriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Robert Reich, Hongyu Ren, Frieda Rong. Yusuf H. Roohani, Camilo Ruiz, Jack Ryan, Christopher R'e, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishna Parasuram Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang, William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei A. Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. On the opportunities and risks of foundation models. ArXiv, abs/2108 08.5558. 2021.
[6] Léon Bottou, Frank E. Curtis, and Jorge Nocedal. Optimization methods for large-scale machine learning. SIAM Review, 60(2):223-11 2018.
[7] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. ArXiv, abs/2006.09882, 2020.
[8] Mathilde Caron, Hugo Touvron, Ishan Misra, Herv' e l'egou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in supe-supe vision transformers. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 9630-9640, 2021.
[9] C. Chen. Dependent Normalized Random Measures. PhD thesis, The Australian National University, 2014.
[10] Changyou Chen, Vinayak A. Rao, Wray L. Buntine, and Yee whye Teh. Dependent normalized random measures. In ICML. 2013.
[11] Jianfei Chen, Jun Zhu, Yee Whye Teh, and Tong Zhang. Stochastic expectation maximization with variance reduction. In NewrIPS, 2018.
[12] Junya Chen, Zhe Gan, Xuan Li, Qing Guo, Liqun Chen, Shuyang Gao, Tagyoung Chung, Yi Xu, Belinda Zeng, Wenlian Lu, et al. Simpler, faster, stronger: Breaking the log-k curse on contrastive learners with flatnce. arXiv preprint arXiv:2107.01155 152, 2021.
[13] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. ArXiv, 2tt//2002.5509,,22200
11

---

[14] X. Chen, S. Xie, and K. He. An empirical study of training self-supervised vision transformers. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 9620-9629. Los Alamitos, CA, USA, oct 2021. IEFE Computer Society.
[15] Xinlei Chen. Moco v3 for self- superviee resnet and vit. faceboobressarrch/ooc--- 2022.
https://github. com/
[16] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. 202/ IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15745- 15753, 2021.
[17] Bernard Delyon, Mare Lavielle, and Éric Moulines. Convergence of stochastic approximation version of the em algorithm. Annals of Statistics, 27:94-128, 1999.
[18) Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. With a little help from my friends: Nearest-neighhor contrastive learning of visual representa- tions. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 9568-9577, 2021.
[19] Nicholas Frosst, Nicolas Papernot, and Geoffrey E. Hinton. Analyzing and improving represen- tations with the soft nearest neighbor loss. In ICML, 2019.
[20] Luyu Gao, Yunyi Zhang, Jiawei Han, and Jamie Callan. Scaling deep contrastive learning batch size under memory limited setup. In REPLANLP, 2021.
[21] Yuting Gao, Jia-Xin Zhuang, Ke Li, Hao Cheng, Xiaowei Guo, Feiyue Huang, Rongrong Ji, and Xing Sun. Disco: Remedy self-supervised learning on lightweight models with distilled contrastive learning. ArXiv, abs/2 104.91444 2021.
[22] Songwei Ge, Shlok Kumar Mishra, Chun-Liang Li, Haohan Wang, and David Jacobs. Robust contrastive learning using negative samples with diminished semantics. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
[23] Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. ArXiv, abs/1803.00723 2018.
[24] Jean-Bastien Grill, Florian Strub, Florent Altch'e, e, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Ávila Pires, Zhaohan Daniel Guo, Mohammad Ghesh- laghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent: A new approach to sulevsuue learning. ArXiv, abs/2006.07733. 2020.
[25] Michael Gutmann and Aapo Hyvärinen. Noise- conts-contre estimation: A new estimation principle for unnormalized statistical models. In Yee Whye Teh and Mike Titterington, editors, Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, volume of Proceedings of Machine Learning Research, pages 297-304, Chia Laguna Resort, Sardinia, Italy, 13-15 May 2010. PMLR.
[26] Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning invariant mapping. 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2:1 1-135-1142, 2006.
[27] Jeff HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable guarantees for self- supervised deep learning with spectral contrastive loss. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
(28) Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum contrast for unsupervised visual representation learning. 2020 IEEF/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 9726-9735, 2020.
[29] Gabriel Ilharco, Mitchell Wortsman, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, July 2021. If you use this software, please cite it as below.
12

---

[30] Yannis Kalantidis, Mert Bulent Sariyildiz, Noe Pion, Philippe Weinzaepfel, and Diane Larlus. Hard negative mixing for contrastive learning. 2021.
[31]
Purushottam Kar, Harikrishna Narasimhan, and Prateek Jain. Online and stochastic gradient methods for non-decomposable loss functions. In NIPS, 2014.
[32] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balean, and H. Lin, editors, Advances Neural Information Processing Systems, volume 33, pages 18661-18673. Curran Associates, Inc., 2020.
[33] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. ArXiv, abs/2004. 1362, 2020.
[34] Alexander Kolesnikov, Alexey Dosovitskiy, Dirk Weissenborn, Georg Heigold, Jakob Uszkoreit, Lucas Beyer, Matthias Minderer, Mostafa Dehghani, Neil Houlsby, Sylvain Gelly. Thomas Unterthiner, and Xiaohua Zhai. An image is worth 16x16 words: Transformers for image recognition at scale. 2021.
[35] Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.
[36] Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Curl: Contrastive unsupervised repre- sentations for reinforcement learning. Proceedings of the 37th International Conference on Machine Learning, Vienna, Austria, PMLR 119, 2020. arXiv:2004.04136.
[37] Chunyuan Li, Haotian Liu, Liunian Harold Li, Pengchuan Zhang. Jyoti Aneja, Jianwei Yang, Ping Jin, Yong Jae Lee, Houdong Hu, Zicheng Liu, and Jianfeng Gao. ELEVATER: A bench- mark and toolkit for evaluating language-augmente visual models. In Thirly-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
[38] Chunyuan Li, Jianwei Yang, Pengchuan Zhang, Mei Gao, Bin Xiao, Xiyang Dai, Lu Yuan, and Jianfeng Gao. Efficient self-supervised vision transformers for representation learning. In International Conference on Learning Representations, 2022.
[39] Xiao-Li Meng and David A. Van Dyk. Seeking efficient data augmentation schemes via conditional and marginal augmentation. Biometrika, 86:301-320, 1999.
[40] T.K. Moon. The expectation-maxiine maximization algorithm. IEEE Signal Processing Magazine, 13(6):47-60, 1996.
[41] Peter Neal and Theodore Kypraios. Exact bayesian inference via data augmentation. Statistics and Computing, 25:333-347, 2015.
[42] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. 2008 Sixth Indian Conference Computer Vision, Graphics & Image Processing, pages 722-729, 2008
[43] Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. ECCV, 2016.
[44] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In 2012 IEEE Conference Computer Vision and Pattern Recognition, pages 3498-3505, 2012.
[45] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. arXiv preprint 10Xiv22103.00020, 2021.
[46] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021.
[47] Hao Ren. Simelr. httpss//gthhuo.cc///ttttooossss
2022.
13

---

[48] Joshua David Robinson, Ching Yao Chuang, Suvrit Sra, and Stefanie Jegelka. Contrastive learning with hard negative samples. In International Conference on Learning Representations. 2021.
[49] Ruslan Salakhutdinov and Geoff Hinton. Learning nonlinear embedding by preserving class neighbourhood structure. In Marina Meila and Xiaotong Shen, editors, Proceedings of the Eleventh International Conference on Artificial Intelligence and Statistics, volume 2 of Proceedings of Machine Learning Research, pages 412-419, San Juan, Puerto Rico, 21-24 Mar 2007. PMLR.
[50] Shai Shalev-Shwarte and Shai Ben-David. Understanding Machine Learning From Theory to Algorithms. Cambridge University Press, 2014.
[51] Piyush Sharma, Nan Ding. Sebastian Goodman, and Radu Soricut. Conceptual captions: cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguisties (Volume I: Long Papers), pages 2556-2565, Melbourne, Australia, July 2018. Association for Computational Linguistics.
[52] Kihyuk Sohn. Improved deep metric learning with multi-class n-pair loss objective. In NIPS, 2016.
[53] Hyun Oh Song, Yu Xiang, Stefanie Jegelka, and Silvio Savarese. Deep metric learning via lifted structured feature embedding. 2016 IEEE Conference on Computer Vision and Pattern 2016.
Recognition (CVPR), pages
[54] Jiaming Song and Stefano Ermon. Understanding the limitations of variational mutual informa- tion estimators. In International Conference on Learning Representations, 2020.
[55] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In ECCV, 2020.
[56] Yao-Hung Hubert Tsai, Martin Q. Ma, Muqiao Yang, Han Zhao, Louis-Philiggs Morency, and Ruslan Salakhutdinov. Self-supervised representation learning with relative predictive coding. International Conference on Learning Representutions, 2021.
[57] Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. ArXiv, abs/1807.03777 2018.
[58] Kilian Q Weinberger and Lawrence K. Saul. Distance metric learning for large margin nearest neighbor classification. In NIPS, 2005.
[59] Zhirong Wu, Yuanjun Xiong, Stella X. Yu, and Dahua Lin. Unsupervised feature learning via con-parametric instance discrimination. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7733-3742, 2018.
[60] Jianwei Yang, Chengkun Li, Pengchuan Zhang, Bin Xiao, Ce Liu, Lu Yuan, and Jianfeng Gao. Unified contrastive learning in image-text-label space. ArXiv, abs/2204.03610, 2022.
[61] Chun-Hsiao Ych, Cheng- Yao Hong, Yen-Chi Hsu, Tyng-Luh Liu, Yubci Chen, and Yann LeCun. Decoupled contrastive learning. ArXiv, abs/2110.068453 2021.
[62] Mingzhang Yin and Mingyuan Zhou. Arm: Augment-reinforee merge gradient for stochastic binary networks. In ICLR, 2019.
[63] Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel C. F. Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, Ce Liu, Mengchen Liu, Zicheng Liu, Yumao Lu, Yu Shi, Lijuan Wang, Jianfeng Wang, Bin Xiao, Zhen Xiao, Jianwei Yang. Michael Zeng, Luowei Zhou, and Pengchuan Zhang. Florence: A new foundation model for computer vision. ArXiv, abs/21 11. 11.11432, 2021.
[64] Jure Zbontar, Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self- supervised learning via redundancy reduction. In ICML, 2021.
14

---

[65] Richard Zhang, Phillip Isola, and Alexei A. Efros. Colorful image colorization. In ECCV, 2016.
[66] Huangjie Zheng, Xu Chen, Jiangchao Yao, Hongxia Yang, Chunyuan Li, Ya Zhang, Hao Zhang, Ivor Wai-Hung Tsang, Jingren Zhou, and Mingyuan Zhou. Contrastive attraction and contrastive repulsion for representation learning. 2021.
[67] Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Noel C. F. Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, and Jianfeng Gao. Regionclip: Region-basee singuage-imag pretraining. 2021.
[68] Mingyuan Zhou and Lawrence Carin. Augment-and-conquee negative binomial processes. In NIPS, 2012.
[69] Mingyuan Zhou, Yulai Cong, and Bo Chen. Augmentable gamma belief networks. J. Mach. Learn. Res., 13:6:444 2016.
15

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [No]
(b) Did you specify all the training details (e.g., data splits. hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [N/A]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
16


Reviews:
[{"review_id": "vfOaXw3--Hm", "reviewer": "Reviewer_4KzQ", "summary": "In order to address the gradient bias and sub-optimal solution problems in contrastive learning, this paper proposes a novel Bayesian data augmentation method to decompose contrastive loss with an efficient sEM algorithm. The authors develop a generalization theory and verify their approach with extensive experiments comparing with several contrastive learning baseline methods. The experimental results demonstrate the effectiveness of this approach on single-modal image representation learning and multi-modal vision-language representation learning tasks. ", "questions": "1. In lines 235-236, you mentioned \"For optimization convenience and algorithm stability, we marginalize out the auxiliary data $U$ from previous iterations, is it a good solution, or what is the problem behind this? \n2. in lines 328-330, since the normalization can mitigate the gradient noise problem, do you still need another approach to deal with the intrinsic gradient noise? \n3. What is the training time for your proposed methods? Is it much slower than SimCLR in your experiments?", "limitations": "N/A", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The problem formulation is clear. The authors propose a novel approach to remedy two problems, i.e., gradient bias and sub-optimal solution, with a new contrastive loss applying the Bayesian data augmentation method. This paper points out a new research direction to improve the performance of contrastive learning. \n2. The proposed method only needs minimal modifications on most contrastive learning frameworks, e.g., SimCLR and MoCo-v3.  The authors provide a generalization theory and explain clearly how the proposed loss function can be used in most contrastive learning approaches. The authors show clear proofs for the proposed theorems. \n3. The experimental results (i.e., tables and figures) clearly demonstrate the effectiveness of this approach. Some improvements are clearly observed between the proposed method and the compared baseline methods on most evaluation metrics across all tasks. \n4. The paper is well written and easy to follow. The authors present a very good literature review. The notations, figures, and algorithms are easy to read.\n\nWeaknesses:\n1. The authors propose two methods to construct a sequence of rate parameters to approximate $E[s_{ik}-]$ of the Gamma posterior in lines 223-224. They should have provided sufficient analysis here. \n2. The authors provide some analysis and experimental results on the robustness of their method against gradient noise, it is not clear whether the gradient noise is equal to or the same as gradient bias. \n3. There are some minor issues. There is another meaningless red curve shown in Figure 1. To my understanding, $u_i$ in the negative sample part in Eq. (2) should also be boldface,  ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ihXSLKbUFU-", "reviewer": "Reviewer_SAtH", "summary": "The authors aimed to address the biased gradient, which is due to the non-decomposability of the loss, in the existing contrastive learning methods. A more generalized and decomposable loss function has been proposed by using Bayesian data augmentation technique. In the proposed loss, a joint distribution over the model parameters and the augmented parameters is defined, and can be optimized by stochastic EM algorithm. Extensive experiments were conducted on several datasets to confirm the efficiency of the proposed method.", "questions": ". Lines 94-118: The authors identified the gradient bias in contrastive learning as the current loss is non-decomposable. As mentioned later, this problem could be mitigated by using a large batch size. In Fig. 1, a motivation example is presented to show the consequence of the gradient bias. However, this example is not convincing as only one negative sample is used. In practice commonly hundreds or thousands negative samples are used along with some easy-to-implement training tricks, eg, normalization to mitigate the gradient bias. Overall, the gradient bias seems not to be a big issue in practical contrastive learning methods.\n\n. What's the connection between equations (1) and (3), ie., the existing and proposed loss function? Are they equivalent in terms of optimality?\n\n. Lines 180-194: From the analysis, the decomposibility is derived under the assumption of infinite negative samples. In practice, as there are only finite negative samples does the decomposibility still hold?\n\n. In Algorithm 1, how to properly design the sequence of {\\lambda_t} for convergence?\n\n. Theorem 3: The generalization gap is shown to be upper bounded by two terms. Any comments on these two terms?\n\n. From the provided experimental results, generally, the proposed method has more advantage when the batch size is small. When the batch size is large and the normalization is used the advantage is limited (eg, in Table 1: CIFAR-10/batch size 512, Table 2: batch size 512, Table 3: MoCo-v3-512 vs Ours-512). This again raises the question that, whether the gradient bias is a big issue for practical well-designed contrastive learning methods? The authors can provide comments on this for a better summary of comparison. ", "limitations": "Please see the Questions above for the detailed comments. \n\nNo potential negative societal impact related.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n. A new loss function is proposed to address the gradient bias with theoretical analysis.\n\n. Extensive experiments have been shown for the comparison between the proposed algorithm and state-of-the-art, eg., SimCLR and MOCO-V3.\n\n#############\n\nWeaknesses\n\nThe authors tried to address the gradient bias in contrastive learning. However, it's not convincing that the gradient bias is a big issue in existing contrastive learning methods. Especially usually a large number of negative samples are used along with some training tricks such as normalization. Such concern can also be confirmed from the experimental results. That is, generally, the proposed method has more advantage when the batch size is small. See Questions below for more detailed comments. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "SjRqz4Jrhb", "reviewer": "Reviewer_ZEaA", "summary": "In regards to the contrastive learning framework, the paper claims that the use of mini-batches of negative samples instead of the computationally infeasible option of using the entire dataset leads to bias in the parameter gradients w.r.t. the loss function. The primarily arises due to the fact that the loss terms corresponding to the positive and negative pairs are coupled, which prevents the expected loss of a mini-batch to be equal to the actual expected loss. This in turn results in sub-optimal final parameters and performance. To address this issue, the paper proposes a Bayesian data augmentation approach that effectively decouples the loss terms arising from the positive and negative pairs, which makes the expected mini-batch gradient to be equal to the actual expected loss terms. The paper shows experiments on large scale vision and vision-language tasks.", "questions": "1. Major technical flaw: The similarity scores in the contrastive loss function (like SimCLR) are typically in the exponent, but that is not the case in Eq. 1. This is problematic because similarity scores are typically the output of a cosine operator, which takes values between -1 and +1. Therefore, both the numerator and denominator can now take values between -1 and +1, and thus the combined term itself takes values between -inf and +inf. Applying the log function over such a term will cause numerical issues, and more importantly this is not a proper loss function. This problem creates issues in other places in the paper, which are discussed below.\n\n\n2. Fig. 1: The toy experiment in Fig. 1 does not demonstrate the gradient bias problem that is introduced in the abstract/introduction, which questions the primary motivation of the paper. There are two problems:\n    - Simulating the noise in mini-batches and showing that it causes divergence, does not necessarily show gradient bias, which is typically expected to lead to a sub-optimal solution. Divergence due to noise can happen even in unbiased gradient scenario, for instance, when the learning rate is too high. These details, which are not provided in the paper, are important. A more direct evidence would be when loss for both cases converge, but the biased gradient loss is sub-optimal.\n    - There is a technical issue with this experiment. In the contrastive loss, the similarity scores are present in the exponent of e. But it seems the similarity scores described in Eq. 1, and what’s used for this toy experiment do not use the exponent. This is bound to lead to numerical issues.\n\n3. Line 328-330 states: \"The reason is that we find the normalization can make the impact of gradient noise less prominent because the representations are constrained to a much more compact sub-space”. This statement undermines the motivation of the paper, which is to mitigate the gradient bias problem in contrastive methods to improve its performance. As stated, if normalizing the latent representation to lie on the unit L2 ball (which is the de facto procedure for all contrastive methods) makes the gradient noise less prominent, then the gradient bias problem does not seem very concerning. In addition and importantly, removing the feature normalization step (i.e. using dot product instead of cosine as similarity) is not a technically sound thing to do, because the contrastive loss function is no longer meaningful.\n\n4. In the opening paragraph of introduction, the authors state the gradient bias problem, which is the main subject of the paper. Following this, they rightfully acknowledge that non-contrastive methods like BYOL, do not need negative samples, which is the source of the gradient bias problem in contrastive methods. However, they dismiss such non-contrastive methods by stating that they are developed from a different perspective. But this does not rule out the possibility that non-contrastive methods cannot perform similarly to contrastive methods (in fact they sometimes report better performance), and may be addressing the gradient bias problem implicitly. Therefore to me, the claims here seems hand-wavy, and a deeper investigation is required to put non-contrastive methods in this context. This is also important to concretely establish that gradient bias is a real problem, meaning, no existing methods (not just SimCLR) solve it.\n\n5. Experiments reporting accuracy: While the accuracy numbers after removing the normalization step are better for the proposed method, the numbers for SimCLR are much worse than what is reported in literature as well as on the GitHub repo page from where the code is used for the experiments in Table 1 in this paper. Even for the normalized feature setting, SimCLR numbers are worse by ~5% for batch size 512 compared to that reported in the GitHub repo (citation 42 in the paper). My guess is that this is because of the use of Eq. 1, which is different from the original SimCLR objective.\n\n6. Experiments reporting accuracy: Once again, the performance of MoCo-v3 is far below the performance reported in their original paper. My guess is that the authors have use the loss function in Eq. (1), which does not put the similarity score in the exponent, which makes the loss function technically problematic.\n\nMinor comments:\n\n- This is more about convention, rather than a technical issue. Loss in Fig 1 converges to 1 for the unbiased case and is typically higher than the biased gradient case. However, typically, a lower value of loss is considered better.\n\n\n- Line 78: The positive pair is stated as the pair of the original sample and its augmented version. However, a positive pair typically consists of 2 augmentations, and not one original and one augmentation.\n\n- Line 129: Dimension of the matrix U is not specified on line 129.\n\n#### POST REBUTTAL###\nI thank the reviewers for their detailed response. Most of my concerns have been addressed. I have changed my score to weak accept. The reason for not providing a stronger score is because I am not fully convinced if the problem that the paper addresses has a very broad impact, since the proposed method is only significantly better compared to the original contrastive methods for certain batch sizes (typically small). Nonetheless, I think this paper proposes an interesting idea that does address an inherent problem in mini-batch based contrastive learning algorithms.", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- Contrastive learning has emerged as a successful mode of self-supervised representation learning. Thus any improvement to this framework will benefit the broader research community.\n- The proposed algorithm is novel, and a generalization bound is provided.\n- Experiments are conducted on large scale datasets.\n- The experiments show improvement of the proposed method over the baseline, under the contrastive objective (Eq 1) considered in the paper.\n\nWeakness:\n- There seems to be a major technical flaw in the contrastive objective (Eq. 1), which is used as the baseline (SimCLR) and as a starting point on which the proposed algorithm is built on. See comment 1 below.\n- The motivation behind the gradient bias problem, which is the primary subject of the paper, seems weak. See comments 2,3,4 below.\n- The experimental results, while they show that the proposed algorithm outperforms SimCLR, may not be reliable because the SimCLR performance in the paper are far below the numbers reported in existing literature for the same experimental setting. The reason for this is related to the previous point regarding Eq. 1. See comments 5,6 below.\n\nIf there was something I misunderstood regarding the major problems mentioned above, I'll be happy to reconsider them upon proper clarifications.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept