AUC-CL: A Batchsize-Robust Framework for Self-Supervised Contrastive Representation Learning
Authors: Rohan Sharma
Kaiyi Ji
zhiqiang xu
Changyou Chen
Venue: ICLR 2024 poster (2024)
Keywords: Contrastive Learning
AUC maximization
Representation Learning
Self Supervised Learning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We propose a batch-size robust contrastive loss, prove convergence guarantees and obtain state of the art performance against several established methods with low batch sizes.

Abstract: Self-supervised learning through contrastive representations is an emergent and promising avenue, aiming at alleviating the availability of labeled data. Recent research in the field also demonstrates its viability for several downstream tasks, henceforth leading to works that implement the contrastive principle through innovative loss functions and methods. However, despite achieving impressive progress, most methods depend on prohibitively large batch sizes and compute requirements for good performance. 
In this work, we propose the $\textbf{AUC}$-$\textbf{C}$ontrastive $\textbf{L}$earning, a new approach to contrastive learning that demonstrates robust and competitive performance in compute-limited regimes. 
We propose to incorporate the contrastive objective within the AUC-maximization framework, by noting that the AUC metric is maximized upon enhancing the probability of the network's binary prediction difference between positive and negative samples which inspires adequate embedding space arrangements in representation learning. Unlike standard contrastive methods, when performing stochastic optimization, our method maintains unbiased stochastic gradients and thus is more robust to batchsizes as opposed to standard stochastic optimization problems.
Remarkably, our method with a batch size of 256, outperforms several state-of-the-art methods that may need much larger batch sizes (e.g., 4096), on ImageNet and other standard datasets. Experiments on transfer learning, few-shot learning, and other downstream tasks also demonstrate the viability of our method.

Published as a conference paper at ICLR 2024
AUC-CL: A BATCHSIZE-ROBUS FRAMEWORK FOR SELF-SUPERVISH CONTRASTIVE REPRESENTATION LEARNING
Rohan Sharma', Kaiyi Ji', Zhiqiang Xu², Changyou Chen
University at Buffalo, 2MBZUAI (rohanjag, kaiyiji, changyou y @buf oalo..ooooo edu, (zhiqiane ang. xu) @mbzuai i .ac. ae
ABSTRACT
Self-supervised learning through contrastive representations is an emergent and promising avenue, aiming alleviating the availability of labeled data. Recent research in the field also demonstrates its viability for several downstream tasks, henceforth leading to works that implement the contrastive principle through inno- vative loss functions and methods. However, despite achieving impressive progress, most methods depend on prohibitively large batch sizes and compute requirements for good performance. In this work, we propose the JUC-Contrastive Learning, a new approach contrastive learning that demonstrates robust and competitive performance in compute- limuted regimes. We propose to incorporate the contrastive objective within the AUC-maximization framework, by noting that the AUC metric is maximized upon enhancing the probability of the network's s binary prediction difference between positive and negative samples which inspires adequate embed- ding space arrangements in representation learning. Unlike standard contrastive methods, when performing stochastic optimization, our method maintains unbiased stochastic gradients and thus is more robust to batchsizes opposed to standard stochastic optimization problems. Remarkably, our method with a batch size of 256, outperforms several state-of-the-ar methods that may need much larger batch sizes (e.g., 4096), on ImageNet and other standard datasets. Experiments on transfer learning and few-shot learning tasks also demonstrate the downstream viability of our method. Code is available at AUC-CL
INTRODUCTION
As deep learning continues to pervade across several facets of our livelihood, we come to an increasing realization that the developments in the field have outpaced the availability of data, especially labelled data. An increasing apprehension of these limits over time has led to research towards data efficient and viable alternative of self- superviedd learning (SSL). Recent SSL pretraining based methods (Grill al. 2020 Chen et al.) 2020 Caron et al.) 2020 Chen al., 2021 Lee et al., 2021 Li et al. 2021 Tomasev et al. 2022) have successfully recovered the performance of and often outperformed supervised methods for the downstream tasks on classification, object detection, recognition, and segmentation, to name a few, in the low data regime. Large scale language models such as BERT (Devlin et 2018), RoBERTa (Liu et al. 2019). XLM-R (Conneau al., 2019), and GPT (Brown et al. 2020). which have their roots in foundational methods such as Word2 Vec (Mikolov et al. 2013) and Glo Ve (Pennington al., 2014), serve as the exemplars of surprising cognizance language(s) using the self-supervised pretraining routine.
However, a significant drawback of these models the requirement of steep computing capacities and training times, largely due to the nature of the contrastive loss function. Several of the aforementioned methods leverage the ubiquitous and ergonomic cross-entropy function as the main objective for their training process. The objective function, albeit robust, often leads to an optimization bias in the computation of the gradient. For example, [Cen et al. (2022) illustrates this phenomenon using a simple synthetic experiment, showing that mere usage of SGD based optimization can lead to several sub-optimal solutions. More specifically, as shown by Yuan et al. (2022), the NT-Xent objective (a variant of the cross-entropy loss) used in SimCLR (Chen et al. 2020) and several other works can

---

Published as a conference paper at ICLR 2024
lead to considerable optimization error: "SimCLR suffers an optimization error least in the order of B) for the objective's 's gradient norm. Even with T oC, its optimization error is always dominated by O(1/VB)." This bias in the updates largely explains the degradation of performance in SimCLR and related methods with smaller batch sizes, which notably arises from the denominator of the contrastive objective (Chen al. 2022
In addition to the aforementioned works, Yeh et al. (2022) Chuang et al. (2020) also unambiguously
indicate an anomaly specific to the denominator of the contrastive loss. This leads the conclusion that, in expectation, the standard contrastive loss function is not an unbiased estimate of the true objective, which actually encompasses infinitely many negative samples for each positive one. This motivates us to study alternatives to overcome the intrinsic limitation of contrastive learning. To this end, we propose a new contrastive loss by incorporating the "contrastive" idea into the AUC- maximization framework. Specifically, since AUC only specifies two classes in the definition, we propose to treat the two classes as positive and negative classes for each data point. These positive and negative representations are then optimized through the AUC criterion, i.e., to maximize the area under the ROC curve for positive and negative classes. By optimizing the new loss, one can not only seamlessly model the -esitive-negativee sample interactions done in standard contrastive learning, but also enjoy merits inherited from the AUC loss, e.g., Yuan et al. (220)) demonstrates the AUC loss more robust noise compared to the cross-entropy loss. In addition. analysis that our loss function provides batch size independent gradient updates and assures us of provable convergence under mild assumptions. In summary, our major contributions are as follows:
show through an extensive
We re-examine some limitations of the standard contrastive learning framework, and propose a new contrastive loss that incorporates contrastive representation within the AUC maxi- mization framework. Our framework comes with nice theoretical properties and inspires superior performance under low batch size regimes.
We thoroughly analyze the function and its gradient updates and show that our function leads to unbiased gradient updates. Through extensive experiments, we demonstrate the robustness of our framework to small batch sizes.
We empirically demonstrate that our method outperforms several SOTA methods in SSL, as well as in some popular few-shot learning and transfer learning benchmarks.
2 PRELIMINARIES AND RELATED WORK
2.1 SELF-SUPERVISED LEARNING
Self-supervision in machine learning has become increasingly promising due to the alleviation the requirement for labelled data. Successful applications in downstream tasks further reflect well on its merits. The training mechanisms take varying formats that leverage several forms of co-occurrences. Temporally relevant information is utilized in works such Wang & Gupta (2015) Logeswaran & Lee (0118); Singh et al. (202I): Pan et al.) (2021); Gao et al. (2022). For vision based tasks, the highly popular method of using multiple image augmentations towards encouraging feature space separation
is leveraged by several works (Dosovitskiy et al.) 2014 Bachman et al. 2019 Tian et al.] 2020 Gao al. (02T). Methods that implement alternative contrastive objectives along feature dimensions include SimSiam (Chen & 2021). BYOL (Grill et al. 2020). DINO (Caron et al., 2021). VICReg
(Bardes et al., 2021). Barlow Twins (Zbontar et al. 202 and HaoChen et al. (202) which uses a
spectral contrastive objective.
Recent innovations falling under image sample contrastive methods that we consider relevant this work include the SimCLR (Chen et al. (2020), MoCo v3 (Chen et al., 2021). CLIP (Radford et al.) 202T), DCL (Yeh et al. 2022) and others, which utilize variants of the cross-entropy criterion. A prominent characteristic of these works is the requirement for large batch sizes in order to enhance performance, which often mandates a steep compute requirement and training time.
2.2 CONTRASTIVE LEARNING AND ITS LIMITATIONS
Contrastive Learning can date back to work in the early 90's (Becker & Hinton 1992 Bromley et al. 1993), based the motive of utilization of internally derived teaching signals neural
2

---

Published as a conference paper at ICLR 2024
network towards memory and data efficient systems. The core implementation has prevailed to the recent times, which is based on minimization of the distance between features generated from images/modallites belonging to unique entity. In recent times, (Chen et al. 2020 2021 Caron et al. 2021, Chen & He 202T), the field has successfully leveraged data augmentation strategies towards training large networks that often rival the supervised paradigm. In its most elementary form, the augmented versions of similar samples are drawn closer in the feature space and the dissimilar ones pushed apart. However, owing the lack of labels, these methods suffer from drawbacks that concern efficient sampling of the negatives.
Standard contrastive learning such as NTXent (Chen et al. 2020p and InfoNCE (Oord et al. 2018) use variant of the cross entropy as the loss function. For batch B (containing the indices of the data samples) and the corresponding features vectors {zi), the NTXent loss is formulated as
1 Σ log 2n JJEB
exp(sim(21, 2/)T)
C NTXent
2k=1 exp(sim(2i, (x)m(2
where sim(u, v) denotes the cosine similarity of features u and :; T is the temperature pa- rameter and
[B]. Of particular interest to this work is the denominator g(w,i) E2n Lk=1: (xp(sim(2) (2. 2x)/),, where w refers to the parameters of the network. This takes the form of a sum-exponent of similarities that ideally correspond to the negative class. However, the gradient estimate of this function with minibatches, given by
1 n
V. sim (z, 2j) + (g(
TV NTXent
n (ww,i)
is a biased estimator of the true gradient (Chen et al. 20222. In practice, large batch sizes are needed trade off the accuracy and computational efficiency.
There have been some efforts made to mitigate this dependence from several perspectives. For instance, Yeh al. (2022) construct decoupled objective that discards the presence of positive samples in the InfoNCE criterion and demonstrate gains. Chuang et al. (2220) also remedy the presence of positive samples in the denominator by constructing an estimator for bias correction. Ge
et al. 2021), Kalantidis et al. (2020) and Robiison et al. (2020) attempt negative mining to improve performance without explicitly addressing the bias issue.
To our knowledge, very little work has explicitly addressed the bias phenomenon and suggested solutions. Chen et al. (2022) demonstrate the bias phenomenon using a simple synthetic experiment and propose an expectation maximization based algorithm which adaptively weighs the negative samples in the minibatch. Yuan et al. (022)) quantify the bias in the order of O(I/VB) and suggest a solution that maintains track of the denominator terms and uses its moving average to compute the gradient at every step which comes at a cost to the memory and computation. Although these works provide remedies, the principle working mechanism is based on an asymptotically diminishing bias. We emphasise that our method not only is simplistic principled approach, it is also provably free of the bias and does not rely on momentum encoders, stop- -raddggadeett methods, negative mining or any additional tricks for better performance.
2.3 AUC MAXIMIZATION
Area under the ROC curve (AUC) is one practical metric for evaluation of model performance, which accounts for imbalances in dataset classes and overcomes the shortcomings of metrics such as accuracy. In principle. maximization of AUC leads towards enbancement of the prediction score of positive class, relative to the other negative classes. Some works that have studied pairwise maximization methods towards this objective are Gao & Zhou Toachims (2005), Herschhtl & Raskutti (2004). However, a natural challenge to the optimization process for AUC that arises as a consequence is the high computational complexity (O(N2)) which results from directly maximizing the positive class scores against the paired negative samples which inhibits learning from large-scale data. Ying al. (2016) proposed an equivalence that allows for min-max optimization objective that alleviates the O complexity of the objective and allows for an efficient optimization objective. Thereafter, improvements over this work were researched by Natole et al. (2018), Liu et al. (0188)
3

---

Published as a conference paper at ICLR 2024
and Yuan al. (2020). Notably, Yuan al. (2020) address laxities of the aforementioned methods, namely the adverse effects of easily classifialee data and the sensitivity to noisily labelled data, through margin based component in the loss function. The works above inspired to construct objective function that proves to be a prominent option for SSL. based tasks that is very reliable in limited compute related circumstances due its robustness to batch size.
3 THE PROPOSED METHOD
We first discuss some work of standard AUC metric maximisation frameworks, which was originally used for performance evaluation of classifiers. We then describe the motivation behind our formu- lation from the perspective of these approaches and subsequently, we propose our new framework for batchsize-robust contrastive representation learning by incorporating the contrastive idea into the AUC maximization formulation; and conduct theoretical analysis to show that the gradient updates resulting from our framework are unbiased and therefore robust to batch size variance. The convergence theory for our method is also developed.
3.1 NOTATIONS
Let D = (xi } denote the set of training images and let P denote the complete set of data augmentations with a randomly picked data augmentation denoted by A(.) P. For randomly selected sample E DD, consider the set M, defined as M; = {j A(xx) for VA E P, Vx, D)(x;) }) which denotes the set of indices of all samples and their augmentations excluding those of x, Analogous to this, for a batch B, B; U A(x,) for AA E P, vx, {xx,) Let E() define the encoder network comprising of the backbone network fw and the projector network gw. used for the generation of feature vectors for every image. Upon completion of training, discard the projector gw- We denote the cosine similarity between two encoded samples u = E(x.) and Exx,) by sim(u, v) defined by sim v) u'v
3.2 AUC MAXIMIZATION
We start by brief review of the AUC maximization framework, which, as problem for deep learning, is best formulated with a differentiable form. This can facilitate learning by application of backpropagation. Specifically, the AUC metric can be formulated using the Wilcoxon-Mal Maanonn---an---- Mann-Whitny statistics (Mann& & Whitney 1947 Wilcoxon 1992) as:
Definition 3.1. Given a classifier lw parameterized by w and labeled samples {x, x'ly = 1, 1},, the area under the receiver operator characteristic curve is defined by,
AUC(w) = P(hw (x) 2 ftw (x') y = 1, 1.y' = -1).
This formulation leads to an O(N2) complexity due to the explicit requirement of positive- negattve pairs for optimization. A widely used reformulation by introducing the square surrogate loss lends to an ease of optimization and alleviates the high complexity (Ying et al. 2016 Natole et al. 2018 Liu
ct al. 2018 Yuan et al., 2020). given by
L(w) = EB(( - Jow (x) + hw (x'))² y = 1, = --]]
To make the problem in more tractable, we use the following equivalent reformulation theorem ( (Yuan et al., (020).
Theorem 3.2. Minimising is equivalent to the following min-max problem:
min T(w) = min max B[Lr(W,, a3), where wERd
weRt OER
L((w ai) A (1w((x) - -(w))²² = 1 (1ew (x') - b(w))² [-1]] =-1]
+ max (2x(1= - a(w) + b(w)) -a?}
(5)
The optimal values of a, b, given w are a = a(w) = E(lo((x) y= 11, b b(() = E(/u((x) y -1] and a 1 1 a, respectively.
4

---

Published as a conference paper at ICLR 2024
X
Negerves
Negative pel dini oriis
Footive par s lorifie
X
>Ax
A,
Sample
Xx
Augmemanors
Enteedings
Fouture space
Enooder
Network
Simvlorlty Sc3le
Figure 1: The proposed AUC-CL framework: of positive pairs whereas A₂ enhances the distance between positive-negative pairs. Additionally, As. governed by the non-zere zero a parameter augments the gap between the expected values of similarities of positive and negative pairs, thus preventing collapse.
The component A serves to enhance the proximity
3.3 THE PROPOSED AUC-CL FRAMEWORK
We describe the incorporation the contrastive idea into the AUC maximization framework. In setting, adopt the cosine similarity metric to evaluate the feature space proximity of the data samples. Given a sample x, we generate feature vectors from two random augmentations of the sample, F(x.)) and F(x,). Thus, for batch of size [B] we derive two sets of augmented features, each of size [B]. We then evaluate the cosine similarity matrix using the two sets, which forms the matrix of scores that we eventually use our objective function.
Note both the contrastive loss and the AUC formula contain positive and negative data, yet their definitions are different. Therefore, one needs make transformations between them in order to achieve incorporation of contrastive loss into the AUC formula. To this end, we propose that each positive and negative classes in the AUC formula should correspond to pair of data samples in our setting. Specifically, we consider a score jim(i,j) between samples and Xj, each subjected to random augmentation operators and A' respectively, where (i [1, [B]], E [1, [B]]) belong the positive class in the AUC formula if it is derived from augmentations of the same source image. i.e. i = j. and denoted as yä+ = Similarly, it is called negative class (denoted as (y:g = -1)) if the pair is derived from separate images # j. Based on this, the original AUC maximization objective in is generalized to what we coin as the AUC contrastive learning:
min max Ex,.D....A' weRd aER
a: x; A, A, B.)]
(6)
where the loss 1/, (, 0r; x;, A. A', B,) at data x, and two data augmentations A and A' are given by (W,,,; 0; A, B3) [11 ii) +
ji) yest, Uis]
SEB
[(sim(i, - a(w))?
Ma+ +
(i)m( b(w))²
we] + max (2a(1 a(w) b(w)) a²}
JEB
(w)
ess (w)
a2(w)
Here, evaluate the component a1 (w) for the positive class and (www) for the negative class the notation suggests. Inspired by the conventional contrastive learning objectives, we take a sum over the negative terms Vig as a measure enhance the impact of the negative sample pairs, which in turn can improve the similarity scores for positive sample pairs. This leads to significant improvement in the performance.
A practical reformulation We propose reformulation the original problem in equation 6 to improve practical performance. 1) First, according to Theorem 3.2 the optimal values for a(w) and b( b(w) are: a(w): := E sim(i, i+) ). b(w) E jeB, (im((,)) Viyl- We substitute these optimal values into equation 6 and directly optimize over the encoder parameter This could lead to better convergence as one does not need to alternate between optimizing a and w. However, to avoid approximation errors, the expectations need to be exactly evaluated. In practice, only stochastic
5

---

Published as a conference paper at ICLR 2024
are available. Thus, replacing a(w) and b(w) with their stochastic estimations would induce bias if the evaluation function is non non-linear, e.g., a1 (w) and the (2(w) terms in equation 6 As a result, only do the substitution for a(w) and b(w) for the linear terms, e.g., the 0g (w) term. 2) Second, once we substitute a(w) and b(w) with their stochastic approximations in term 0z( w), the hyper-parameter has to be independent w.r.t. w in order to avoid bias introduced in stochastic approximation. Thus, we simply fix a in the experiments. We conduct ablations with various values of to illustrate the robustness of our function to the values. Additionally, one may opt to fix a and witness marginal improvements. We conjecture that this may allow ease in optimization for the uniformity task of representation learning, which non-trivial as compared to the alignment task Wang & Isola (020). As result, we propose the following new loss in practice as our objective function:
approximations calculated from
I' (w, b; x3, A, B.)
(sim(i, i+) a)²
Vii+ +
[sim(())) - -b)² Bes-1
(8)
JEB;
A
A2
+ 2a[1 sim(i, i+)
sim(i. | -a²
a²
Mii+
jEB,
Aa
Based the loss, we deseribe the algorithm for pretraining using contrastive loss in Algorithm An intuitive illustration of the method and the functions of its components is given in the Figure
3.4 CONVERGENCE ANALYSIS
We conduct the analysis of the gradient estima- tors for various components of our function in the under section A.I in the Appendix using the standard assumptions of smoothness and conti- nuity. In a high level, we evaluate the gradients of the various components of the Global Con- trastive Objective given by equation g and the stochastic gradient estimators of these compo-
Algorithm 1: AUC-CL
Input: Dataset D = {xI, Xn Parameters and networks: w. fws Iw for n=1 to N do
2: Sample a batch B + {I, ~ D)
3:
for all E [1....] do
4:
Fi, zit = A(z;), A'
nents which are used in batch wise optimization.
5:
2;: 244 (x.()( Iw (f (1(4))
We then establish the smoothness properties of these components and that the stochastic gradi- estimator is unbiased estimate of the true gradient, using standard assumptions. Subse- quently, using these properties, we establish the following convergence result.
6: end for
7:
for all
i {I...B} and j E (1... BJ do x 2;
8:
sim(i. = ||,م|||||
9:
end for
10: Evaluate I', according to
11: w 4- optimizer (w, L,)
Theorem 3.3. Assume the cosine similarity function sim(i, (ww) is smooth and Lipschitz continuous w.r.t. and max(a,b) b) S
end for Output: Encoder f
Choose the stepsize B 1 tie gradients in our algorithm are unbiased. Furthermore, for the optimization parameters (w.D) we have the following convergence results
VT. Then the stochas-
E VL(V))
where is drawn from 1 uniformly at random.
Remark 3.4. Theorem 3.3 indicates that our algorithm converges to a local optimum at a rate of
( (v)) w.r.t. the number updates T, which is independent batch sizes. Whereas for the standard contrastive loss with the cross-entropy related variants, the convergence rate is O(1//7L
/n 1/ 1/VB) for the learning rate 7 and batch size B Yuan et al. 2022). Consequently, this suffers from a convergence error that depends on the batch size B, which is at least O(1v/B) at the limit of
T - and 7) 0.

---

Published as a conference paper at ICLR 2024
4 EXPERIMENTS
Owing to the adaptable nature of our loss function, we simply adapt the widely existing framework of SimCLR (Chen al., 2020) and MoCo-v3 (Chen et al. 202T) codebases to AUC-CL., We retain the architectural choices and other configurations and explore different batch sizes. In all the experiments unless otherwise specified, similar to the work in Chen et al. (202T), we set the learning rate to be Ir BatchSize/ 2266 where Ir refers to the base learning rate parameter, hyperparameter that experiment with and realize that a stable value is set at 1r=1,0e-3 with higher values leading to slower convergence and accuracies lower by 2%. Moreover, we adopt learning rate warmup described by Goyal et al. (2117) for 40 epochs followed by cosine decay schedule (Loshchilov & Hutter (016). Our default optimizer is the AdamW (Ilya al. 2019
We follow the standard linear evaluation proto- col for our models and report the Top-1 Ima- geNet validation accuracy of a linear classifier trained over pretrained frozen networks. For Im- ageNet, we train using random resized crop strategy by cropping the images to 224x224 and resizing to 256x256 during training and using a center crop and resizing for validation data and train for 90 epochs using SGD optimizer and a batch size of 1024 with a learning rate of 0.1 with cosine annealing decay schedule and no weight decay. We train our models on four NVIDIA RTX A5000 GPUs with memory of 24.2 GB per GPU.
Table I: ImageNet Pretraining (ResNet-50). Top-1 accuracy linear evaluation results are listed. Results marked "*" are reproduced using LightlyAI
Method
Batch Size
100 ep
400 ep
800 sp
SimCLR
4096
66.5
69.8
70.1
BYOI.
4096
66.5
73.2
McCo-v3
4096
68.9
73.1
73.8
VICReg
2048
68.6
70.6
73.2
DINO
1024
68.3
72.9
BarlowTwins
1024
67.7
73.1
73.2
Zem CL.
1024
68.9
72.6
ARB
NNCLR+DCL
512
71.2
74.9
McCo-v2
256
67.4
71.0
72.2
SimCI DC
256
65.1
69.5
SimCLR+DCLW*
256
64.5
67.88
ImageNet We pretrain and finetune AUC- CL on ImageNet using the ResNet-50 and ViT-Small architectures using a batch size of 256. Additionally, we incorporate a multi- crop strategy of augmentation introduced in SwAV (Caron et al. 2020) for ImageNet with 2 x 22 + 96- global and local crops re- spectively. The results are given in Tables I and 2 We reproduce/omit the results for some of the listed methods due to unavailability, Remark- ably, our method outperforms several prominent methods that use much larger batch sizes in or- der to ensure strong performances. We also com- pare against some of the more recent methods of W-MSE (Ermolov et al. (222)). Zero-CI. CL (Zhang et al. 2022b) and ARB (Zhang al.) 2022a) that are batch size robust. We present supplementary findings utilizing a batch size of 1024 for other methods, while maintaining our batch size at 256 for training over 800 epochs in the Appendix under Table
SimSkam
256
68.1
70.8
71.3
W-MSE
256
69.4
72.5
AUC-CL
256
69.5
73.5
75.5
Table 2: ImageNet Pretraining (VIT-S), Top-1 accuracy for linear evaluation 1O-NN evalua- tion for k-NN are listed after pretraining for 300 epochs.
Method
Batch Size
k-NN
Lincar
SimCLR
1024
69.0
BYOL
1024
66.6
71.4
MoCo-v2
1024
62.0
71.6
MoCo-v3
1024
72.5
MoCo-v3
4096
73.2
SwaV
1024
64.7
71.8
AUC-CL
256
70.7
73.7
ImageNet-1000 Cifar-10 and Cifar-100 We conduct further experiments on standard benchmark image datasets, namely Cifar- 10 (Krizhevsky al., a) and Cifar-100 (Krizhevsky et al. al.] b). Moreover, we also evaluate our function on smaller scale of the ImageNet (Deng et al. (009) denoted by ImageNet- 100 which consists of a subset of randomly selected 100 classes from the full dataset, in keeping with the protocol followed by Wu et al. (2019) and Yuan et al. (2022). Following general practice, we pretrain using the backbone ResNet-18 and the set standard augmentations as given in Chen et al. (2020) and finetune using the same procedure. Table 3 lists our results. We use the popular benchmark given by da Costa et al. 2022) for the other methods. For all the other listed methods, the ImageNet- 100 data was trained using batch size of 128 for 400 epochs and list our results for the same. The Cifar datasets were trained using a batch size of 256 for 1000 epochs whereas we list our results after pretraining merely for 500 epochs using batch size of 128. As
7

---

Published as a conference paper at ICLR 2024
listed, AUC-CL outperforms all the listed methods. For the Cifar datasets, significant margin is gained upon training using merely half the batch size and number of epochs.
Table 3: Cifar and ImageNet- 100 pretraining- Linear evaluation (Acc@) 1) results. All methods trained on ImageNet-Ile 100 for 400 epochs using batch size of 256. The Cifar datasets are trained on for 1000 epochs using the same batch size. We train method on the Cifar datasets for 500 epochs and use a batch size of 128.
Method
Cifar-
Cifar-100
ImageNet-
Method
Cifar-10
Cifar- 100
ImageNet-
W-MSR
88.7
61.3
67.6
ARB
92.2
69.6
79.5
SwaV
89.2
64.9
74.0
NNCLR
91.9
79.8
SimSiam
90.5
66.0
74.5
McCo-v3
93.1
68.8
80.4
DINO
89.5
66.8
74.8
BYOL
92.6
70.5
80.2
Zero-FCL
90.5
70.2
75.7
Barlow Twins
92.1
70.9
80.4
Zero-ICL
90.5
69.3
78.0
AUC-CL
93.6
69.7
81.9
Few-Shot Linear Probing We further evaluate our ViT-S model through downstream linear probing for classification. We use the ELEVATER benchmark (Li 2022) which conducts 5-shot classification based tasks on 20 public image classification datasets. We run the benchmark with random initializations and use their automated hyperparameter tuning pipeline. Here we compare against other methods that are pretrained on ImageNet but use larger backbones, e.g., ViT-base. We demonstrably perform better than the mentioned methods on several datasets. Notably, our method outperforms greater majority of the datasets.
Table 4: ELEVATER benchmark. We compare against MoCo-v3 (Chen al. 2021), DeiT (Tovvron et al. 202T) and MAE (He et al. 2022). We list the backbone networks for each. Data-1 to Data-20 correspond to Caltech101, CIFARTO, 10, CIFARI 100, Country211, DescriTextures, FuroS FER2013, FGVC Aircraft, Food101, GTSRB, HatefulMemes, KITTI, MNIST, Oxford Flowers, Oxford Pets, PatchCamelyon, Rendered SST2, RESISC45, Stanford Cars and VOC2007 datasets respectively.
Method
Data-1
6
Datu-8
MAE
VTBB
59
34
21.2
2,8
35
64.4
21.3
7.0
7.7
17.5
33.4
MoCo-a3
VITBB
80.8
78.5
60.5
4.8
57.1
77.1
20.5
11.8
36.6
31.4
50.2
DeTT
WTB
86.2
70.1
61.5
4.4
52.9
62.5
14.5
24.1
41.9
46.7
54.1
AUC-CL
VIT-S
87.2
76.7
61.8
4.6
57.7
77.8
19.5
28.8
38.7
49.7
54,9
Method
Arch.
Data-11
Data-12
Data-13
Data-14
Data-15
Date-16
Data-17
Data-18
Data-19
Data-20
Wins
MAE
46.1
63.4
50.9
$4.9
38.9
18.3
MoCo-v3
VTTBB
50.7
46.7
64.1
79.5
76.3
54.7
50.1
61.1
13.4
47.9
DeTT
VitB
51.1
47.6
83.8
82.7
87.8
51,5
50.1
63.4
27.7
70.9
AUC-CL
50.
51.5
77.6
86.5
76.5
62.4
65.5
49.9
Other Comparisons We compare against another batch size robust method SogCLR (Yuan et al. 2022) using the ResNet-50 backbone and training on ImageNet and ImageNet-100 using identical set of parameters and augmentations in order to retain fairness. The results are given in Table 5 We again notice significant margin of improvement over the listed methods and across the batch sizes and epochs. Moreover, we witness greater robustness to batch size wherein the performance shifts are negligible with varying batch sizes. We further detail additional comparisons against MoCo-v3 A.3.2 and SimCLR A.3.3 the Appendix.
5
DISCUSSION
Cifar-10
Cifar-100
ImageNet-100
STL-10
68
85
Acclit
Acd88n
Acce1
92
67
78
84
90
66
76
83
64
128
256 Batch Size
512
64
128
256 Size
512
64
128
256
512
64
128
256
512
Batch
Batch Size
Batch
Size
Figure 2: Batch size robustness. Linear evaluation results (top-1 accuracy) using various batch sizes on the mentioned datasets compared against MoCo-v3, Our method retains stability in performance with varying batch sizes with relatively smaller variance.
8

---

Published as a conference paper at ICLR 2024
Table 5: Comparison by pretraining on ImageNet and ImageNet-100 against SimCLR (Chen et al. 2020) and SogCLR (Yuan et al. 2022) using the ResNet-50 backbone. Linear evaluation (acc@) results are listed. We retain the parameter and augmentation settings.
Batch size
Methods
ImageNet
ImageNet-
Epoch
100
200
400
100
200
400
SimCLR
62.6
64.0
64.1
68.5
72.7
75.7
128
SogCLR
64.9
66.2
67.4
72.2
76.7
79.3
AUC-CL
66.1
67.6
68.8
79.0
81.4
82.7
SimCLR
62.8
64.3
65.7
69.7
73.6
76.1
256
SogCLR
65.2
67.1
68.7
71.8
76.3
78.7
AUC-CL
66.2
67.7
69.1
78.9
81.2
82.8
SimCLR
63.8
65.6
66.7
70.9
74.1
75.9
512
SogCLR
65.0
67.2
68.8
71.8
75.8
78.2
AUC-CL
67.9
69.2
78.8
82.8
Intuition We illustrate a simplified intuition behind our formulation in the Figure [ The squared terms A1 and A2 enforce the similarity of the positive and negative pairs towards the values of and b. However, we observe that merely minimizing these two components and ignor- ing A3 leads to representational collapse. The component A3 therefore serves to counteract this collapse of the objective leading to an increase in the similarities of the positive pairs and decrease in those of the negatives, thus balancing the effects of each component towards desirable optimum. A combination of the three terms, leads to representations learned by our framework that would satisfy both positive-data compactness and negative-date repulsion, which is not considered explicitly in standard contrastive learning. Rearranging, the positive terms [(sim(i, (+) a)² 20(1 sim(t, (+)) yi+)] encourage the augmented views to be consistent in the feature space whereas EÿEs, [(sim(i,.) b)2 2asim(i, yーiー forces separation across negatives, encouraging representations match high entropy prior distribution.
dappiie margin robustness The function exhibits robustness to variations in batch size, attributed unbiased stochastic gradient estimators, derived theoretically (A.I). Notably, this characteristic isn't exclusive to our formulation. Our function's 's superior performance is evident, and we attribute this to its inherent nature, The retention in performance (Figure 2 with smaller batch sizes linked to robustness to increased noise in the batch, where smaller batch tends to have noisier set of similarity scores. Notably, our function introduces an adaptive margin term A3, a unique component in our formulation. This term proves particularly relevant for handling noise in stochastic samples. For instance, cases where the sim(i. (j) term is artificially high due to noise, sim(i, i) strategically adjusted to compensate for the noise, improving class clustering in the feature space and mitigating the effects of inherent noise. Additionally, we conduct experiments to establish that the component Ag whose influence is monitored by is critical to avert mode collapse in Table 11 in the Appendix.
Convergence We empirically analyze the convergence of our function by comparing against known arts in Figure (Appendix). We train using ResNet- 18 Cifar-10 10 using the configuration mentioned in A.3.3 and plot the k-NN Top-1 accuracy curves for the methods and notice that our method converges noticeably sooner. Specifically, our method outperforms SimSiam (second best) by over 8% at the epoch 100. We conjecture that the component A may be instrumental in such rapid early convergence. Aq monitors the uniformity objective Wang & Isola (020). An unbiased estimate of the component is therefore more asymptotically optimal and thus may be significant contributing factor in helping the model learn optimal feature spaces earlier than standard methods.
CONCLUSION
In this work, we investigate the popular SSL via contrastive learning and its variants. We delineate the drawbacks of the corresponding objectives and propose batchsize-robust objective inspired from AUC maximisation of data. We further analyze our objective for convergence and prove the function's 's robustness in convergence to batch size. Furthermore, under varying settings of representation learning, transfer learning and few shot classification, and compare against the state of the arts demonstrate the merits of our method agnostic to batch sizes.
conduct several experiments

---

Published as a conference paper at ICLR 2024
7 ACKNOWLEDGEMENT
This work is partially supported by NSF AI Pestitute-22298777 NSF RI-2223292, an Amazon research award, and an Adobe gift fund. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation, the Institute of Education Sciences, or the U.S. Department of Education.
BROADER IMPACT
The publication on self-supervised contrastive learning introduces a method that exhibits robust- ness to variations in batch size, advancing the field of unsupervised representation learning. This innovation enhances scalability and efficiency, making deep learning techniques more accessible across diverse computational infrastructures. By democratizing the utilization of self-supervised learning methodologies, the research fosters broader accessibility to cutting-edge technologies and holds potential for significant impacts across various applications, from natural language processing to computer vision.
10

---

Published as a conference paper at ICLR 2024
REFERENCES
Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-But E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Cur- ran Associates, Inc., 2019. URL https: httpppps:/ppcccccinggggooeoo neur http://s..s.rrrp.p.ppppppee/e//p Paperr pttppp/cccpppp///11///////
fi
Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: aariang-iivaiaanc-ovvviineeernge regproooo for self-supervised learning. CoRR, abs/2105.04906 2021. URL
2105. 04906
Suzanna Becker and Geoffrey E Hinton. Self-organizing neural network that discovers surfaces in random-dot stereograms. Nature, 355(6356):161-163, 1992.
Jane Bromley, Isabelle Guyon, Yann LeCun, Eduard Säckinger, and Roopak Shah. Signature verifi- cation using a" siamese" time delay neural network. Advances in neural information processing systems, 6, 1993.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901, 2020.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsuper- vised learning of visual features. CoRR, 0tx/1807.055522 2018. URL /tKLitttpp///rxxx.oorrg
/oas/1007. 05520
Mathilde Caron, Ishan Mísra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments, 2020. URL https
//arxiv.rrg/abs...
09882
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Fmerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9650-9660, October 2021.
Changyou Chen, Jianyi Zhang, Yi Xu, Liqun Chen, Jiali Duan, Yiran Chen, Son Tran, Belinda Zeng. and Trishul Chilimbi. Why do we need large batch sizes in contrastive learning? a gradient-bias perspective. In NeurIPS 2022, 2022. URL
htttps::/proceeddnnge wtwpeeeeiiinn.....inneeeeee nwwr httppppppr..llleeappp//////////a dbl
html]
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations, 2020. URL
05709
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15750-15758, June 2021.
Xinlei Chen, Saining Xie, and Kaiming He. An formers. CoRR, abs/2104.02057. 2021. oK//h/ppp.//arrr..rrrrr/////2..0
study of wtw.ddyyoorraiiinngeesssssrrrrro vision vision trans-
Ching-Yac Yao Chuang, Joshua Robinson, Yen-Chen Lin, Antonio Torralba, and Stefanie Jegelka. De- biased contrastive learning. Advances in neural information processing systems, 33:8765-8775, 2020.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Fran- cisco Guzmán. Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised cross-lingual representation learning at scale. arXiv preprint arXiv: 19111991119 1.02/ 16, 2019.
Victor Guilherme Turrisi da Costa, Enrico Fini, Moin Nabi, Nicu Sebe, and Elisa Ricci. solo-learn: A library of self-supervised methods for visual representation learning. Journal of Machine Learning Research, 23(56):1-4 2022. URL wtt.:htpp.. http://mm...orrrppppa////////1/1
html
11

---

Published as a conference paper at ICLR 2024
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A acage-scale hier- archical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255, 2009. doi: 10. 10L.1//VV..200090506660
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 04000, 2018,
Alexey Dosovitskiy, Jost Tobias Springenberg, Martin Riedmiller, and Thomas Brox. Discriminative unsupervised feature learning with convolutional neural networks. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger (eds.), Advances in Neural Information Processing httttppss//pooceeinng.eeeell
Systems, volume 27. Curran Associates, Inc., 2014.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint MxXiv:2010.11929 2020.
Aleksandr Ermolov, Aliaksandr Siarohin, Enver Sangineto, and Nicu Sebe. Whitening for self- supervised representation learning. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 3015-3024. PMLR, 18-24 Jul 2021. URL https:
hRt http::://ppcceeiingg
mlr. wwl..pres//ee http///1///eroloo.//// html
Junyu Gao, Mengyuan Chen, and Changsheng Xu. Fine-grained temporal contrastive learning for superv-superviied temporal action localization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 20099, June 2022.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simese: Simple contrastive learning of sentence embeddings. CoRR, 0bbs//244.08811, 2021. URL//ttcees//arrai..or////////// 08821
Wei Gao and Zui-Hua Zhou. On the consistency of auc pairwise optimization. In 7wenty-Fourth International Joint Conference on Artificial Intelligence, 2015.
songwei Ge, Shlok Mishra, Chun-Liang Li, Haohan Wang, and David Jacobs. Robust contrastive learning using negative samples with diminished semantics. Advances in Neural Information Processing Systems, 14:27356-27366 2021.
Spyros Gidaris, Andrei Bursuc, Gilles Puy, Nikos Komodakis, Matthieu Cord, and Patrick Pérez. Online bag-o--visuallwords generation for unsupervised representation learning. CoRR, abs abs/2012.11552,, 2020. URL https: http://wwww...rr//////22........
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint 0rXiv:1706.02671 2017.
-aantBatteen Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent: A new approach to self-supervised learning, 2020. URL httpthtss///riiiii wttpxiiwwrg//bb//000000 07733
Jeff Z. HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable guarantees for self-supervised deep learning with spectral contrastive loss. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 5000-5011. Curran Associates, Inc., 2021. URL pttt:p:://proceddingg..
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 16000-16009, June 2022.
Alan Herschtal and Bhavani Raskutti. Optimising area under the roc curve using gradient descent. In Proceedings of the twenty-first international conference Machine learning, pp. 49, 2004.
12

---

Published as a conference paper at ICLR 2024
Loshchilov Ilya, Hutter Frank, et al. Decoupled weight decay regularization. Proceedings of ICLR, 2019.
Thorsten Joachims. A support vector method for multivariate performance measures. In Proceedings of the 22nd international conference on Machine learning, pp. 377-384, 2005.
Yannis Kalantidis, Mert Bulent Sariyildiz, Noe Pion, Philippe Weinzaepfel, and Diane Larlus. Hard negative mixing for contrastive learning. Advances in Neural Information Processing Systems, 33: 21798-21809 21809, 2020.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-10 (canadian institute for advanced research). http:pp/www.......rrtttttee/e//a
http:/iiiiiiaaaaieeeieeieeaeeeee html
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-100 (canadian institute for advanced research). b. httt:htt:///wtt htt.:tttt://w..c...orrrttttt
far. htmll
Kuang-Huei Lee, Anurag Arnab, Sergio Guadarrama, John Camny, and Ian Fischer. Compressive visual representations, 2021. URL
12909
Chunyuan Li, Jianwei Yang, Pengchuan Zhang, Mei Gao, Bin Xiao, Xiyang Dai, Lu Yuan, and Jianfeng Gao. Efficient sepe-supe vision transformers for representation learning, 2021. URL https: //wpxvaavvvv///////////aa//aiai. wwtp/////////....//////
Chunyuan Li, Haotian Liu, Liunian Harold Li, Pengchuan Zhang. Jyoti Aneja, Jianwei Yang, Ping Jin, Yong Jae Lee, Houdong Hu, Zicheng Liu, and Jianfeng Gao. Elevater: A benchmark and toolkit for evaluating geguage-augmented visual models. Neural Information Processing Systems, 2022.
Mingrui Liu, Xiaoxuan Zhang, Zaiyi Chen, Xiaoyu Wang, and Tianbao Yang. Fast stochastic auc maximization with (LLLnnn-coovveeenn rate. In International Conference on Machine Learning, pp. 3189-3197. PMLR, 2018.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint 19Xiv:190..116600 2019.
Lajanugen Logeswaran and Honglak Lee. An efficient framework for learning sentence representa- tions, 2018. URL https:
abs/ 02893
Iyy Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:16066 03988, 2016.
H. B. Mann and D. R. Whitney, On Test of Whether one of Two Random Variables is Stochastically Larger than the Other. The Annals of Mathematicaal Staristics, 18(1):50 60. 1947. doi: 10.1214/ aoms/1 177730049.. (tLhttts::////
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representa- tions in vector space, 2013. /ttt/tttt//:atxttii
Michael Natole, Jr., Yiming Ying. and Siwei Lyu. Stochastic proximal algorithms for AUC maximiza- tion. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 3710-3719 PMLR, 10-15 Jul 2018. URL https: httts::/prroceeiingg http://............ee/////aaella
html
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In Indian Conference on Computer Vision, Graphics and Image Processing, Dec 2008
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03777 2018.
Tian Pan, Yibing Song, Tianyu Yang. Wenhao Jiang, and Wei Liu. Videomoco: Contrastive video representation learning with temporally adversarial examples. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 11205-11214 June 2021.
13

---

Published as a conference paper at ICLR 2024
Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In IEEE Conference on Computer Vision and Pattern Recognition, 2012.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 1532-1543, 2014.
Ben Poole, Chen Sun, Cordelia Schmid, Dilip Krishnan, Phillip Isola, and Yonglong Tian. What makes for good views for contrastive representation learning? In NeurIPS 2020, 2020.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. CoRR, abs/2103.00022. 2021. URLhhtppa /tttps::/arzsae ozg/ abs / 200..11131... 00020
Joshua Robinson, Ching- Ching-Yao Chuang, Suvrit Sra, and Stefanie Jegelka. Contrastive learning with hard negative samples. arXiv preprint arXiv:2010.44592 2020.
Ankit Singh, Omprakash Chakraborty, Ashutosh Varshney, Rameswar Panda, Rogerio Feris, Kate Saenko, and Abir Das. Semi-supervised action recognition with temporal contrastive learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 10389-10399, June 2021.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (eds.), Computer Vision ECCV 2020, pp. 776-794, Cham, 2020. Springer International Publishing. ISBN 978-3-030-58621-8.
Nenad Tomasev, Ioana Bica, Brian McWilliams, Lars Buesing, Razvan Pascanu, Charles Blundell, and Jovana Mitrovic. Pushing the limits of self-supervised resnets: Can we outperform supervised learming without labels imagenet?, 2022. http:::: http:pps:/aaxxi..rrog. abs
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Jegou. Training data-efficient image transformers amp: distillation through attention. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 10347-10357. PMLR. 18-24 Jul 2021. URL https: httpps/s//pcccc.......... http.//p....../../////.......... html
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning. volume 119 of Pro- ceedings of Machine Learning Research, pp. 9929-9939, PMLR, 13-18 Jul 2020. URL
htttps://proceed.ngg wwt../peeee///////aaaaaaaae httmml1mkk.mmmll
Xiaolong Wang and Abhinav Gupta. Unsupervised learning of visual representations using videos. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), December 2015.
Frank Wilcoxon. Individual comparisons by ranking methods. In Breakthroughs statistics, pp. 196-202. Springer, 1992.
Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu. Large scale incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Chun-Hsiao Yeh, Cheng- Yao Hong, Yen-Chi Hsu, Tyng-Luh Liu, Yubei Chen, and Yann LeCun. De- coupled contrastive learning. In Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner (eds.). Computer Vision ECCV 2022, pp. 668-684, Cham, 2022.
Springer Nature Switzerland. ISBN 978-3-031-19980
Yiming Ying, Longyin Wen, and Siwei Lyu. Stochastic online aue maximiza- tion. In D. Lee, M. Sugiyama, U. Luxburg. I. Guyon, and R. Garnett (eds.), Ad- vances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016. URL httppp://proeeccnngg... neuriss..uur......m http://ws.....ao...c.c/////////a c52 0eltl6ccc99d05522228888822222 Paperaa.ddddpapppddo. pddf
14

---

Published as a conference paper at ICLR 2024
Zhuoning Yuan, Yan Yan, Milan Sonka, and Tianbao Yang. Large-seal scale robust deep maximization: A new surrogate loss and empirical studies on medical image classification. 2020. doi: 10.48550/
ARXIV.2012.03173. URLhttps: arxiv. org/ abs 2012. 03173
Zhuoning Yuan, Yuexin Wu, Zi-Hao Qiu, Xianzhi Du, Lijun Zhang, Denny Zhou, and Tianbao Yang. Provable stochastic optimization for global contrastive learning: Small batch does not harm performance. In Kamalika Chaudhuri. Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning. volume 162 of Proceedings of Machine Learning Research, pp. 25760-25755 PMLR, 17-23 Jul 2022. URL https: Stctps://processtt
ngs. ml pre.s.ppeessseee v1 62 http:/ann2bb.hh...m html
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stephane Deny. Barlow twins: Sele-supe learning via redundancy reduction. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 12310-12320. PMLR, 18-24 Jul 2021. URL httpps:s::: httttps://prroceednng mlr. wll.ppress/vll v139 zbontar2la .tt22al...mmmmmmmmmmmmmm........
Shaofeng Zhang. Lyn Qiu, Feng Zhu, Junchi Yan, Hengrui Zhang, Rui Zhao, Hongyang Li, and Xiaokang Yang. Align representations with base: A new approach to self-supervised learning. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 16600-16609, June 2022a.
Shaofeng Zhang. Feng Zhu, Junchi Yan, Rui Zhao, and Xiaokang Yang. Zero-CL: Instance and feature decorrelation for negative-free symmetric contrastive learning. In International Confer- on Learning Representations, 2022b. httpstts:: /ttp:///opeeevveeeeeee net./ tatt/forruti forum?id
RAW9tCdVXL
15

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.1
PROOF OF
Proof. To optimize the objective in 8 with the standard widely available optimizers, adopt the following efficient mini-batch stochastic gradient estimator.
V (w, b; X A', B,) [(sim(i, (isiimiiii))))))) - a) V simmi i+) yoc+]
[((sim(i,)) b) V sim(i,) Mis-] + 2cx V Si 2+) yū-
jEB,
2a
V (wsim((.)) []s-],
(9)
jEB;
VoL(w, A, Bi) = Σ (b sim(i,) 9:5-1,
jEB;
where Bi is random mini-batch images drawn from M,, which still excludes the image x; itself. For the analysis, we consider the following simple update
B
W++1 W.
B
w
A.,
iEB
8
b =h
Σ
(10)
B
Our analysis adopts the following assumption. Note that the cosine similarity metric sim(i, j) for any two augmentations di, 2y and two operations A, A' is a function of w. Then, for the purpose of analysis, instead use the notation (w) capture the dependence
Assumption A.1. We assume for any i, A, A', the cosine similarity metric sim(i,,)(w) satisfies
siw( is Lo-Lipschitz continuous and Lı-smooth.
* max b) S
for some positive constant 7 > 0.
Note that the boundedness condition in the second item can be replaced by adding a projection of and onto bounded like ball for the updates in 10 However, for the simplicity, directly assume the boundedness, which is also observed during the optmization process in the experiments. The global contrastive objective in our case that based the entire dataset D is is given by minweRd, Ex, A.A.A'' ['' w, b; A', .(MM)) where
L(w, A. A', M,) [sim(i, i+) a)² yi++] Σ
[sim(i, 6)2 yi
JEM;
+ 2n sim(i, i+) yis+ +
Σ jim -i5- a²
JEM.
We first prove the smoothness of this objective function. Note that the cosine similarity sim(i, (w) bounded by 1, and hence we have max ()m a, $ + Then, the gradient of the this objective takes the form of
V L(W, by x, A. A'. Ma) ((sim i+) a)
V si i+) yo+]
Σ
[2simm())) V yo-1
JEM,
2a
Visim(i,
|is+] + 2x
Σ
Vwsim(i, |o-])
JEM.
= 2 Σ
[bb sim(i..) 134
JEM.
16

---

Published as a conference paper at ICLR 2024
Based on the gradient form here, obtain, for any two parameters (w, b), (w',
V w (,,, A,
V W.w'w' b,; A, A', Ma)l
< || [2(sim(i, (+) (w) - a) V w sim(i, iw) yu++]
[2(sim(1, (+) - a) (..mmi, (ww)) yi++]l
Σ ([(sim().)) (i yo--] JEM,
- (w.im 11--11l
+ 2a - Wwssmm( (+) Ma++] - V (im( [a+]]
+ 2a
Σ
[ wwwm
8y-]] (w.simii.)) /3g-]]l
JEM,
which, in conjunction with A.1 and using the fact that ab a't' = cal- + (a d')b', yields
<4(T+ 1)Linl/w w/ll + / w/ll 2(n L) 61 2am || w'll
s (4(T + 1) + 2Lzn - 22m w/l + 2(nn 1 1)Lobb 14
V2(((T + 1)Lmn + (Lo + 1)m + 20mL1) vlIw w/12 + 1/2
(11)
Similarly, for the gradient w.r.t. b, we have
|V b, M.)- ( 6';; A, ,MMll I 2(n 1)16 8' + 2(n -oool w w/l
(12)
2n(Lo + 1)V2 |w w'll |b 612
First
note
that
our stochastic gradient
estimator
L by b; A', A', B,) To see this, based the forms in
and
L. d; A, A', B,) are unbiased estimators. have
bVwL(w,bbx,AAA'
=E L(w, b, A, A', A, =EeEV w L(w, X,, A,) Kis A,
-VwL(W,bl
where the first equality follows because B; is sampled from M, A similar result obtained for VoL(w, b; x;, A, A', B.), i.c., LV,, 6; A, B.) = VoL(w, 6).
Based the smoothness results in IT and 12 and the unbiased estimation, we now ready to prove the main theorem. Let v = (w, b) denote all optimization parameters. From the smoothness results establish the smoothness of the overall objective L(v) L(w, b) as below. For
in and 12 any v and v
and
VT2
Th + 12 ||v v'll
(13)
Then, based on have
1 + L 2
I(ve1) <L(v;) + (Ve+1 -., vL(v.))
|
v=ll
17

---

Published as a conference paper at ICLR 2024
which, by taking the expectation E, [[||ey] on the both sides and using our unbiased gradient estimators, yields
VLg. +
E, L.L(v <L(V,) (lV))(v)))
12
" E2 V+41 - v.l12
2
/
L2
- -
B R,
(WL(W, B.)
2
iEB
+
Lw B:) 1²)
(14)
SEB
Note from the forms in and A.1
wc have
Σ
iEB
28(T + 1ymLo + 2anLo
1 Σ V.
b. 4n(T +1) (15)
B
iEB
Incorporating
into yields
L3 12
EvL(v sL(.)) VVV,)v 82
(4n(r + +1) 88T + 1)nLo + 2anLo).
2
Unconditioning vt. rearranging the above inequality and doing the telescoping over t from 0 to
1, we have
[V())) L(vo) miny L(v).
+ +BVE
L + L2 (An(T + 1) 8(T + 1)nLo + 2am
3T
2
<0( + ()
(16)
3T
which, in conjunction with VT
and the definition of t', finishes the proof.
A.2 AUC OPTIMIZATION AND CONTRASTIVE LEARNING
We further elucidate our motivations behind adaptation of the AUC optimization framework towards contrastive learning. AUC metric formulated for binary classification wherein, the objective of the network is to enhance the prediction scores for "positive" samples in comparison to the "negatives" (Equation 3.II. Thus by virtue of its construction, it aligns seamlessly for an application in contrastive learning wherein due to the lack of labels, one is compelled to enforce separation amongst classes through binary objective with "positives" being the augmentations of the same sample and "negatives", the augmentations of other samples within the batch. Additionally. AUC was originally devised to address the imbalance of classes whereby accuracy as a metric may lead to misleading evaluation of the network. A classic example of this phenomenon is often cited with dataset containing 100 samples, 99 of which are of the "positive" class and a network that predicts every sample "positive" will therefore have attained 99% accuracy. This aspect of the function resonates well with the context of contrastive learning in our application, as for one image in our batch of samples, the remaining images considered to be "negative".
A.3 ADDITIONAL RESULTS
A.3.1 LONGER PRETRAINING
In order to attain stronger convergence, we conduct the pretraining procedure using our method for 800 epochs on ImageNet with the Re backbone. Table we compare against the prominent methods for SSL. We retain the batch size of 256 for method, wherein the remaining methods have been trained using a larger batch size 1024. The parameters and setup of the method follows the description in 4 Yet again, our loss function avails superior result using far smaller batch size.
18

---

800 ep Published as a conference paper at ICLR 2024
Table ImageNet Longer Pretraining (ResNet-50). Top-1 accuracy for linear evaluation results are listed. We conduct pretraining for 800 epochs and compare against known arts.
Method
Batch Size
SimCLR Chen et al.
2020
1024
69.1
InfoMin Ponte
1024
73.0
Barlow [wins Zbontar ct 81
Bwins
2021
1024
73.2
OBOW Gidaris el al
2020
1024
73.8
BYOL Grrll
20200
1024
74.4
DCv2
aron ct al
2018
1024
75.2
SwAV
aron et al.
2020
1024
75.3
DINO
aron
2021
1024
75.3
AUC-CL
256
75.5
A.3.2 COMPARISON AGAINST MoCo-v3
Pretraining We train the MoCo-v3 architecture by replacing the objective function with ours whilst retaining the architecture and parameter settings. Our results are based on batch size of 128 on the datasets Cifar- 10, ImageNet-S and ImageNet using the ViT-small backbone, which are illustrated in Table 7 Our models were trained for 100 epochs with the default parameter and augmentation settings. Our model consistently outperforms the results when comparing to MoCo-v3 by values higher than 2% for Cifar-10 10 and Cifar-100 datasets and by over 5% for ImageNet.
Table Pre-training and linear evaluation vs MoCo-v3. "a/b/c'
kNN column are acc. (% with k 10, 20. 100, respectively. The Cifar results use the ResNet- 18 backbone and the ImageNet results use the Vit-Small backbone.
MoCo-v3
AUC-CL
Datasct
KNN
Linear
KNN
Linear
Cifar-10
84.6/84.5/84.
91.4
87.6/87.2/861
93.6
Cifar-100 100
51.3/53.5/52.7
66.6
52.9/54.7/53.4
69.7
ImageNet- 100
74.1/74.7/73.8
77.6
77.0//7.6/766
82.5
ImageNet
50.0/50.6/49.1
62.3
14.1/54.0/51.4
67.9
Transfer Learning We subsequently evaluate the model for transfer learning the Cifar- 10, Cifar-100 datasets, Flowers- 102 Nilsback & Zisserman (2008) and the Pets Parkhi et al. (2012) datasets after pretraining ImageNet. The protocol followed is identical mentioned in Chen et al. (2021) and the results are listed under Table 8
Table 8: Transfer Learning comparison with MoCov3 after pretraining on ImageNet. Numbers
to the method indicate the batch size used. "a/b' represent the Top-1 and Top-5 accuracies (%). The numbers listed under the 'Supervised' category are borrowed from Dosovitskiy et al. (2020)
Ci
C00
Flowers- 102
Pets
Average
Random Init.
77.8
48.5
54.4
40.1
55.2
Supervised
98.1
87.1
89.5
93.8
92.
Moco-v3-256
97.1/100.0
84.6/97.7
88.6/96.7
85.0/98.6
88.8/98.2
AUC-CL-122
98.2/100.0
85.4/97.9
88.9/97.5
87.1/99.3
89.9/98.7
A.3.3 COMPARISON AGAINST SIMCLR
We illustrate the performance our method on Cifar- 10, Cifar-100 and STL-10 in Table comparing against SimCLR for varying batch sizes and epochs. In this experiment, 50. As is standard practice. we replace the first 7 7 Conv layer with stride with 3 x with stride and remove the first max-pooling layer. The learning rate is fixed to le 3. The training is conducted
backbone of ResNet-
19

---

Published as a conference paper at ICLR 2024
Table SimCLR comparison. Top-1 I KNN evaluation results on Cifar-10, Cifar- 100 and STL-10 datasets.
BS
64
128
256
Dataset
Epoch
200
300
500
200
300
500
200
300
500
SimCLR
79.6
81.8
84.15
82.8
83.4
86.2
83.8
85.8
86.9
Cifar-10
AUC-CL
85.4
87.4
89.1
86.1
87.9
89.4
85.7
87.3
88.7
SimCLR
45.2
47.8
49.9
47.8
53.5
55.6
48.5
54.1
56.0
Cifar-100
AUC-CL
52.9
56.0
57.2
53.7
56.1
57.3
53.7
55.9
57.1
SimCLR
69.8
69.7
73.6
72.1
72.9
74.1
75.6
75.7
76.2
STL-10
AUC-CL
76.2
78.6
77.5
78.9
80.9
76.8
78.4
80.4
Table 10: Comparison against DCL for the listed batch sizes and datasets. The linear evaluation scores following the protocol mentioned in DCL are listed. All models are trained for 200 epochs. The backbone used for ImageNet was ResNet-50 whereas for the other datasets, the backbone was set to R8.
Method
BS/Dataset
256
512
DCL
ImageNet
64.3
65.9
65.8
AUC-CL
ImageNet
67.6
67.7
67.9
DCL
Cifar-10
85.7
85.3
84.7
AUC-CL
Cifar-10
88.4
87.9
88.3
DCL
Cifar-100
58.9
58.5
58.4
AUC-CI
Cifar-100
59.3
59.5
59.5
DCL
STI.100
86.1
85.7
85.6
AUC-CL
STL-10
86.5
86.3
86.5
for 500 epochs. We notice that our method significantly outperforms SimCLR for smaller batch sizes by margins of +7% for both datasets, and retains a consistent performance across the batch sizes. Upon convergence, our method outperforms SimCLR in Top-1 accuracy by an average of 3.8%. Subsequent to the epoch 20, our method overcomes the performance of SimCLR, and towards the end of training outperforms by over 3.2%. This trend is reflected across the datasets and batch-size, epoch settings, with the margins of oer-performance particularly stark for smaller batch sizes.
A.3.4 COMPARISON AGAINST DCL
We compare against the popular objective of DCL Yeh (2022) directly for the datasets of ImageNet, Cifar and STI. Here, identical to their work, we pretrain ResNet-50 and ResNet-18 architectures for ImageNet and the rest respectively, for 200 epochs. We follow the augmentation parameters as per their work in order to retain faimess and train our models for varying batch sizes. The results are listed in Table 10 across the comparisons.
We again witness a substantial margin improvement over DCL
A.3.5 ROBUSTNESS TO
We conduct extensive ablations to experiment with the significance of the component Ag in our main formulation g We train the model using the parameters and architecture described in section A.3.3 with a batch size 128 on Cifar-10, 10, while varying the value of the parameter a which modulates the influence of the component. The results are illustrated in Table [1 Here show the KNN evaluation results for at several epochs during training retaining identical settings. We establish that Ag is crucial to our formulation and prevents the mode collapse phenomenon often observed in SSL frameworks, where the features are mapped to a unique point in the hypersphere regardless of the class distinction. Moreover, for all other values of a, formulation retains its performance across
20

---

Published as a conference paper at ICLR 2024
Table 11: Robustness a. We evaluate our loss function on the Cifar-10 dataset for varying values of a using the KNN protocol for k=200. The values are the Top-1 1 KNN evaluation accuracies at various epochs and a parameter settings.
Epoch/a
0.0
0.1
0.5
0.7
1.0
25.1
77.7
78.3
77.6
76.8
100
10.0
82.9
83.4
82.4
81.5
150
10.00
85.18
85.70
84.76
84.41
200
10.00
86.80
86.61
85.89
85.98
250
10.00
87.40
87.64
87.21
87.06
90
80
K-NN (Acc@1)
60
50
40
30
BYOL
SimSiam
MoCo
SwaV
DINO
Ours
20
SimCLR
100
200
300
400
500
Epoch
Figure 3: I--N curves: Plot of the k-NN accuracy curves for various methods trained on Cifar-10 10 for 500 epochs using ResNet-18. 18. The values for the other methods were borrowed from AightlyAl
the epochs, with nominal differences, thus illustrating that the component crucial to the formulation well robust to variations in o, which therefore requires additional tuning.
21


Reviews:
[{"review_id": "OhKQMnIh80", "reviewer": "Reviewer_vuAt", "summary": "This paper introduces AUC-CL, a new approach to self-supervised contrastive learning that seeks to mitigate the dependencies on large batch sizes required by many existing contrastive methods. By integrating the contrastive objective within the AUC-maximization framework, the method prioritizes improving the binary prediction differences between positive and negative samples. This unique approach offers unbiased stochastic gradients during optimization, suggesting a higher resilience to batch size variations. Experimental results indicate that the AUC-Contrastive Learning, even with smaller batch sizes like 256, can rival or surpass performances of other methods requiring considerably larger batch sizes.", "questions": "1. The contrastive learning methods (e.g., SimCLR and MoCo) are sensitive to batch size and prefer larger batch size. However, there are some self-supervised leanring methods such as SimSiam, which do not have the concept of \"positive\" and \"negative\" pairs and are more robust to smaller batch sizes. Considering those methods, the contribution of this paper is diminished.\n\n2. In Table 3, the method proposed in this paper outperforms SimSiam in Cifar-10 dataset, with a 3.1% higher accuracy. However, if we take a look at the Figure 3, the accuracy gap between them is very small, especially in late training stage (after 400 epochs). Could you explain why there appears to be a conflict between these two results?\n\n3. As shown in Figure 3, the accuracy advantage of the proposed method gradually disappear as training proceeds, so it actually just converges faster at early training stage. Moreover, according to the convergence trend in Figure 3, if we train the model for 1,000 epochs for complete convergence as stated in Table 3, the proposed method will not even have an advantage convergence speed.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This paper addresses an important issue in contrast learning: model performance is particularly sensitive to batch size, preferring large batch size.\n\n2. The theoretical proof of the design in this paper is very detailed.", "weaknesses": "1. There are non-contrastive learning SSL methods (e.g., SimSiam) that are also less sensitive to batch size. Considering those methods, the novelty of this paper might be a concern.\n\n2. There appears to be a conflict between the experimental results in the table and the figure.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "U9ABYlB50Q", "reviewer": "Reviewer_nw59", "summary": "The paper introduces AUC-Contrastive Learning as a method to enable self-supervised learning even with a small batch size. This proposed method optimizes the model by making binary predictions on positive and negative samples, aiming to maximize the AUC score. Despite using a significantly smaller batch size, the method shows improved performance over existing contrastive approaches.", "questions": "There needs to be a clearer explanation regarding how a(w) and b(w) are replaced in Equation 7 and what 'a' and 'b' represent in Equation 8.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- The paper presents a contrastive learning technique based on the AUC score, effectively approximating a deterministic score. Its simplicity makes it easy to implement.\n\n- Unlike conventional contrastive learning-based self-supervised approaches, the proposed method demonstrates strong robustness to batch size variations while also showing performance enhancements.\n\n- Through extensive experiments, such as those involving various architectures, smaller datasets, and few-shot transfer, the paper proves the effectiveness of the method across different settings.", "weaknesses": "- The paper is generally not reader-friendly and lacks clarity. In several sections, terms are not defined or are vaguely explained. Specific issues include:\n    - On the second page, the terms B and T are not defined, making comprehension difficult.\n    - The introduction should provide a full explanation of AUC.\n    - On the third page, the term N is not explained.\n    - Figure 1 is distant from its related equation, making it hard to understand terms like A_1,2,3.\n    - In Eq. 8, there's a lack of clarity on how 'a' and 'b' are replaced.\n    - The algorithm section doesn’t exactly mention which loss function is being optimized.\n    - The paper introduces each method name without specifying which previous work it refers to.\n\n- The experimental results appear to be unfair, and several state-of-the-art methods are not reported.\n    - In Table 1, many methods didn't use the multi-crop strategy, but the proposed method did. This gives the proposed method an unjust \n benefit as it would've seen many more images per epoch, leading to potentially improved results.\n    - In Table 2, recent methods like DINO are not mentioned at all. DINO reported performance metrics (KNN: 72.8, Linear: 76.1 in 300 epochs) that are higher than the proposed method. The paper's claim that it outperforms several state-of-the-art methods is misleading, and a comprehensive comparison with all state-of-the-art methods is required.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cYe58tltJK", "reviewer": "Reviewer_pifE", "summary": "This paper proposes AUC-Contrastive Learning, which incorporates the contrastive objective within the AUC-maximization framework. Since it maintains unbiased stochastic gradients, it is more robust to batch sizes compared to the standard contrastive loss. It empirically shows that the method with a batch size of 256 outperforms or is on par with several state-of-the-art methods using larger batch sizes.", "questions": "See Weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The method is novel, and theoretically sound. It is great that there is clear link between the AUC-maximization and robustness to batch size.\n\n- The empirical results are promising. The proposed method with a small batch size can outperform or is on par with state-of-the-art methods with large batch size, which is beneficial for the efficiency of computation power.", "weaknesses": "- Some baselines are missing in the experiments. There are other methods for self-supervised representation learning, such as MAE[1] and MAGE[2]. Some of them have better performance than the contrastive learning methods. Although this paper focuses on improving contrastive representation learning, it is still necessary to compare with non-contrastive representation learning methods. \n\n- Some contrastive learning methods which do not need negative samples, e.g. BYOL and DINO, also shows some robustness to batch size. I think it is necessary to list their performance with smaller batch size (e.g., 256, the same as the proposed method) in Table 1.\n\n- Why does the paper use ViT-S on ImageNet in the experiment? As far as I aware, most of the representation learning methods use ViT-B or ViT-L for ImageNet evaluation. Also, do all methods converge at 400 epochs for ResNet-50 and 300 epochs for ViT-S? If the baseline methods can reach the same performance as the proposed method with longer epochs and larger architecture, then I don't think the paper can claim the outperformance over them.\n\n[1] He et al. Masked Autoencoders Are Scalable Vision Learners. CVPR 2022. \n\n[2] Li et al. MAGE: MAsked Generative Encoder to Unify Representation Learning and Image Synthesis. CVPR 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zRpLyeGhgM", "reviewer": "Reviewer_QVtp", "summary": "This paper proposes AUC-CL, a new batch-size robust framework for self-supervised contrastive representation learning. At first, the authors point out the limitations of existing NT-Xent objectives (i.e., performance is heavily influenced by batch size). Then, the authors theoretically analyze why SimCLR loss is sensitive to batch size from the gradient perspective. Finally, the author provides the theoretical analysis to demonstrate the convergence guarantee of AUC-CL. Although the relative performance is significant, the absolute accuracy is not competitive. Besides, the reasons why AUC-CL is batch-size robust are not clear, which seems to be an important property.", "questions": "As I have reviewed this submission in the other venue, and most of the concerns have been addressed, I have no further questions.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The paper is well-organized and easy to follow\n\nThe experimental results demonstrate the effectiveness of the proposed AUC-CL.", "weaknesses": "1. As stated in the summary, the motivations behind using the AUC framework in combination with contrastive learning, and why this is helpful for batch size robustness, are not clear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)