Decoupled Contrastive Learning
Authors: Chun-Hsiao Yeh
Cheng-Yao Hong
Yen-Chi Hsu
Tyng-Luh Liu
Yubei Chen
Yann LeCun
Venue: NeurIPS 2021 Submitted (2021)
Keywords: Contrastive Learning
Unsupervised Learning
Self-Supervised Learning

Abstract: Contrastive learning (CL) is one of the most successful paradigms for self-supervised learning (SSL). Specifically, contrastive learning treats two augmented ``views'' of the same sample as positive, pulling them close and treating all other samples as negative to push them far apart. Despite the evident success of CL SSL methods, there are several challenges in the existing methods as they may require special structures, large batches, or huge training epochs, etc. Our motivation in this work is to provide a simple, efficient, and yet competitive contrastive learning baseline. Through both theoretical and empirical studies, we identified a strong negative-positive-coupling (NPC) effect in the widely used cross-entropy loss in CL SSL methods. We hypothesize that the NPC effect may be a major cause of the inefficiency in many contrastive learning methods. By removing the NPC effect, we reach a decoupled contrastive learning (DCL) objective function, which significantly improves the training efficiency. DCL can achieve competitive performance, requiring neither large batches in SimCLR, momentum encoding in Moco, or large epochs. We demonstrate the benefit of DCL in various benchmarks. Further, DCL is also much less sensitive to suboptimal hyperparameters. Notably, our approach achieves $66.9\%$ ImageNet top-1 accuracy with 256 batch size within 200 epochs pre-training, which outperforms its baseline SimCLR by $5.1\%$. We believe DCL may provide a strong baseline for future contrastive learning-based SSL studies.

Decoupled Contrastive Learning
Anonymous Author(s) Affiliation Address email
Abstract
Contrastive learning (CL) is one of the most successful paradigms for self- supervised learning (SSL). Specifically, contrastive learning treats two augmented "views" of the same sample as positive, pulling them close and treating all other samples negative to push them far apart. Despite the evident success of CL SSL methods, there are several challenges in the existing methods as they may require special structures, large batches, huge training epochs, etc. Our motivation in this work is to provide a simple, efficient, and yet competitive contrastive learning baseline. Through both theoretical and empirical studies, we identified strong negative-osstive-coppigg (NPC) effect in the widely used cross-entropy loss in CL SSL methods. We hypothesize that the NPC effect may be a major cause of the inefficiency in many contrastive learning methods. By removing the NPC effect, we reach decoupled contrastive learning (DCL) objective function, which signifi- cantly improves the training efficiency. DCL can achieve competitive performance, requiring neither large batches SimCLR, momentum encoding in Moco, large epochs. We demonstrate the benefit of DCL in various benchmarks. Further, DCL is also much less sensitive to suboptimal hyperparameters. Notably, our approach achieves 66.9% ImageNet top-1 accuracy with 256 batch size within 200 epochs pre-training, which outperforms its baseline SimCLR by 5.1%. We believe DCL. may provide strong baseline for future contrastive learning-based SSL studies.
3
5
10
11
12
13
14
15
16
17
18
Introduction
20
21 As a fundamental task in machine learning, representation learning aims to extract features to reconstruct the raw data fully. It has been regarded as long-acting goal over the past decades. Recent progress on representation learning has achieved a significant milestone over self-supervised learning (SSL), facilitating feature learning with its competence in exploiting massive data without any annotated supervision. In the early stage of SSL, representation learning has focused on exploiting pretext tasks, which are addressed by generating pseudo-labels the unlaheled data through different transformations, such as solving jigsaw puzzles [1], colorization [2] and rotation prediction [3]. Though these approaches achieve some success in computer vision, there is large gap between these methods and supervised learning. Recently, there has been significant advancement in using contrastive learning [4, 5, 6, for self-supervised pre-training, which significantly closes the gap between the SSL method and supervised learning. Contrastive SSL methods, e.g., SimCLR [8], in general, try to pull different views of the same instance close and push different instances far apart in the representation space.
23
25
27
29
30
31
32
33
Despite the evident progress of the oftte-of-the-arr contrastive SSL methods, there have been several 35 challenges in future developing this direction: 1) The SOTA models [7] may require unique structures like the momentum encoder and large memory queues, which may complicate the understanding. 2)
37 The contrastive SSL. models [8] may depend on large batch size and huge epoch numbers to achieve competitive performance, posing computational challenge for academia explore this direction.
Submitted to 35th Conference on Neural Information Processing Systems (NeurIPS 2021). Do not distribute.

---

Batch 1024
Butcit sizx: 512
Batch wise: 226
136
Coefitcient Variation
Probsbility
0.0
6g
0.00
0.8
Conficciint Varation
0.04
0.4
0.02
0.2
02
Batch Size
(a)
(b)
Figure 1: An overview the batch size issue in the general contrastive approaches: (a) shows the NPC multiplier in different batch sizes. As the large batch size increasing the @a will approach 1 with small coefficient of variation. (b) illustrates the distribution qB.
39 3) They may be sensitive to hyperparameters and optimizers, introducing additional difficulty to reproduce the results various benchmarks.
Our motivation in this work is provide a simple, efficient, and yet competitive contrastive learning baseline. We choose SimCLR as our starting point, given its conceptual simplicity. By analyzing the objective function, we identified a Negative-Positee Possiiie--ooulliiii (NPC) multiplier qe in the gradient as shown in Proposition 1. The NPC multipliers modulate the gradient of each sample, and it mistakenly increases the impact of both negative samples and positive samples, given either of them is more informative. Such a coupling exacerbates when smaller batch sizes are used. By removing the coupling term, we reach new formulation, the decoupled contrastive learning (DCL). The new objective function significantly improves the training efficiency, requires neither large batches, momentum encoding, or large epochs to achieve competitive performance on various different benchmarks. Specifically, DCL reaches 66.9% ImageNet top-1 (linear probing) accuracy with batch
42
43
44
45
46
47
48
49
51 size 256, SGD optimizer within 200 epochs. Even DCL trained for 100 epochs, it still reaches 64.6% ImageNet top-1 accuracy with batch size 256.
53 In short, this work makes the following contributions:
54
1) We provide both theoretical analysis and empirical evidence to show the regative-positive coupling in the gradient of contrastive learning:
56
2) We introduce a new, decoupled contrastive learning (DCL) objective, which casts off the coupling phenomenon between positive and negative samples in contrastive learning, and significantly improves the training efficiency; Additionally, the proposed DCL objective is less sensitive the several important hyperparameters:
57
58
59
3) We demonstrate our approach via extensive experiments and analysis on both large and small-scale vision benchmarks, with an optimal configuration for the standard SimCLR baseline to have competitive performance within contrastive approaches.
61
2
Related work
64
2.1
Self-supervised representation learning
85
Self supervised representation learning (SSL) aims to learn a robust embedding space from data without human annotation. Previous can be roughly categorized into generative and discriminative. Generative approaches, such as autoencoders and adversarial learning, focus on reconstructing images from latent representations [9, 10]. Conversely, recent discriminative approaches, especially contrastive learning-based approaches, have gained the most ground and achieved state-of- state-of-theeart -he-thrrrt standard large-scale image classification benchmarks with increasingly more compute and data augmentations.
66
67
88
70

---

72
2.2
Contrastive learning
Contrastive learning (CL) constructs positive and negative sample pairs extract information from 74 the data itself. In CL, each anchor image in a batch has only one positive sample to construct positive
73
75 sample pair [11, 8. 7]. CPC [5] predicts the future output of sequential data by using current output
76 as prior knowledge, which can improve the feature representing the ability of the model. Instance discrimination [4] proposes a non-parametric cross-entropy loss optimize the model the instance 78 level. Inv. spread [12] makes use of data augmentation invariants and the spread-out property of
77
79 instance to learn features. MoCo [7] proposes a dictionary to maintain a negative sample set, thus increasing the number of negative sample pairs. Different from the aforementioned self-supervised
81 CL approaches, [13] proposes a supervised CL that considers all the same categories as positive pairs to increase the utility of images.
83
2.3 Collapsing issue via batch size and negative sample
84 In CL, the objective is to maximize the mutual information between the positive pairs. However, to avoid the "collapsing output", vast quantities of negative samples are needed so that the learning objectives obtain the maximum similarity and have the minimum similarity with negative samples. For instance, in SimCLR [8], training requires many negative samples, leading to a large batch size (i.e., 4096). Furthermore, to optimize such a huge batch, a specially designed optimizer LARS [14] is used. Similarly, MoCo [7] needs a vast queue (i.e., 65536) to achieve competitive performance. BYOL [15] does not collapse output without using any negative samples by considering all the images are positive and to maximize the similarity of "projection" and "prediction features. On the other hand, Simsam [16] leverages the Siamese network to introduce inductive biases for modeling invariance. With the small batch size (i.e.. 256), Simsam is rival to BYOL (4096). Unlike both approaches that achieved their success through empirical studies, this paper tackles from a theoretical perspective, proving that an intertwined multiplier qB of positive and negative is the main issue to contrastive learning.
85
86
87
86
89
90
91
92
93
94
96
Decouple negative and positive samples in contrastive learning
97
(a)
(b)
! 2
(1)
(2)
Z
Z
9(.)
9(-)
(-122
h
(1)
(2)
f(-)
f(-)
Zi
Xi
x
10)
Xi (22)
T
L
(1)
2
(2)
Σ
(0
T
Figure 2: Contrastive learning and negative-positive coupling (NPC). (a) In SimCLR, each sample X; has two augmented views (x(") x (2)). They are encoded by the same encoder f and further projected to (z(") 2 [ by a normalized MLP. (b) According to Equation 3. For the view x, the cross-entropy loss _ leads to a positive force z(2), which comes from the other view (2) of and
a negative force, which a weighted average of all the negative samples, i.e. {z, (5) E [1, 2), # i). However, the gradient - -V, (2) ( is proportional to the NPC multiplier.
We choose to start from SimCLR because of its conceptual simplicity. Given batch of N samples 99 (e.g. images), (x1....,xy) xx). let (1)), (x(2) be two augmented views of the sample v, and B be the set of all of the augmented views in the batch, i.c B {x {1,2), i E [L,N]]. As
3

---

shown by Figure 2(a), each of the views x; (k) is sent into the same encoder network f and the output
102 h (k) f(x!') ) is then projected by a normalized MLP projector that 2 ( = g(h, / (h, Il--
(k) For each augmented view x SimCLR solves classification problem by using the rest of the views in B as targets, and assigns the only positive label to x; where u * k. So SimCLR
104
creates cross-entropy loss function L!*) for each view x(*), and the overall loss function is
105
L = [xe (k)
106
exp((2)2,2, (//))//))
4
log exp( (z) 2,
(I)
(2)) /T) Eie(1,))e((( exp((z;)) 11,,,,,,))))))))))))/ /T)
(k)
107
Proposition 1. There exists
(1) coupling (NPC) multiplier 4B, in the gradient of
L D
108
V (1) 11
9g
(1)
(1)
2;
Eu exp(1e 1/7) j
V (
Vgn (1)
(2)
u 1,
(1)
exp eep 2 2/1//
(1)
2;
S.c xx )/T)
())
where the NPC multiplier 93.4 is:
exp(12, (1) (2)
22 (2)
BB.i
(1)
Eq((L22JeeD.MM.. exp( , //7)
(3)
(k) Due to the symmetry, similar NPC multiplier B exists in the gradient of L (k) E f1,2),
110
111 N..
112 As we can see, all the partial gradients in Equation 2 are modified by the common NPC multiplier
i in Equation 3. Equation makes intuitive sense: 1) When a positive sample pair {z, (1) (2)
113
, are
114
farther, the corresponding NPC multiplier BB is larger. This will makes the overall gradient larger.
Otherwise, the gradient is smaller. 2) When negative sample is closer to x (1) it makes TB larger.
(1)
115
116
Overall, the intuition here is that a positive sample farther from the target or a negative sample closer to the target is more informative. However, the positive samples and negative samples are strongly coupled. An outlier positive sample also makes the gradient from the negative samples significantly larger and vice versa.
117
118
119
120
Figure 1(b) shows the NPC multiplier Qa distribution shift w.r.t. different batch sizes for a pre-trained SimCLR baseline model. While all of the shown distributions have prominent fluctuation, the smaller
122
batch size makes (B cluster towards 0, while the larger batch size pushes the distribution towards 6(1. Figure ((a) shows the averaged NPC multiplier (qR) changes w.r.t. the batch size and the relative fluctuation. The small batch sizes introduce significant NPC fluctuation. Based on this observation, we propose remove the NPC multipliers from the gradients, which corresponds the
123
124
125
126
case O This leads to the decoupled contrastive learning formulation.
Proposition 2. Removing the positive pair from the denominator of Equation 2 leads to decoupled (k)
128
contrastive learning loss. we remove the NPC multiplier In. from Equation 2, we reach a
decoupled contrastive learning loss LDc = Exe(112).EED..N D.C where (A) is:
129
exp( (1) ,2 //7)
DCC - -log
(k)
(4)
exp( :
(2) [T] + Eie(1.2) exp((z)") (K) 1///T)
=x 2/ / /T/= log
(2)
Σ
exp((x!)) (xp((2()25))))1
(5)
IE (E(L

---

60.8
60.92
65.9
65.8
66.1
60.8
60.6
60.5
60.6
Top-1 accuracy (%)
64.3
65
Top-1 accuracy (%)
59
64
59.0
64.0
61.5
58
62
61.8
60-
60.6
MoCo vi DCL
SimCLR
58.9
54.7
56.8
32
128
256
512
1024
256
1024
4096
16384
65536
Batch size
Qeuu size
Figure 3: Comparisons ImageNet- with/without DCL under different numbers of (a): batch sizes for SimCLR [8] and (b): queues for MoCo [7]. Without DCL, the top-1 accuracy significantly drops when batch size (SimCLR) queues (MoCo) becomes very small.
130 The proofs of Proposition and 2 are given in Appendix. Further, we can generalize the loss function LDc to Dow by introducing weighting function for the positive pairs i.e. LDCW Exe(L2)) 4 DDC.s
132
(K) L
DCW, (z, (1) 2 (())) 22 )/T log
E
(K) exp((z) -2
(E(L.)).EE(I,N
133
where we can intuitively choose w to be a negative von Mises-Fisher weighting function that expl (xx //)) and [w] = 1. LDC is special case of Lpcw and we
134 (z) (2)
E. exp((((( x 1/o)
can see that lim,- to Lnow = Lpc. The intuition behind w=(x x. xx!2)) is that there is more learning signal when positive pair of samples are far from each other.
136
Experiments
137
138
This section evaluates our proposed decoupled contrastive learning (DCL) empirically and compares the general contrastive learning methods. We summarize our experiments and analysis the following: (1) our proposed work significantly outperforms the general contrastive learning on large and small-scale vision benchmarks: (2) we show the better version of DCL: LDCW could further improve the representation quality. (3) we further analyze our DCL with few learning epochs, which shows fast convergence of the proposed DCL.. Detailed experimental settings can be found in the Appendix.
139
140
142
143
144
145
4.1
Implementation details
To understand the effect of the sample decoupling, we consider our proposed DCL, which is based on the general contrastive learning, where model optimization irrelevant to the size of batches (i.c., negative samples). Extensive experiments and analysis are demonstrated large-scale benchmarks: ImageNet-IKe IK [19], ImageNet-100 [6], and small-scale benchmark: CIFAR [20], and STL10 [21]. Note that all of our experiments conducted with 8 Nvidía V100 GPUs single machine.
147
149
ImageNet For fair comparison on ImageNet data, we implement our proposed decoupled structure, DCL by following SimCLR [8] with ResNet-50 [22] as the encoder backbone and use cosine annealing schedule. We set the temperature r to 0.1 and the latent vector dimension 128. Following [23],
152
153
we evaluate the pre-trained models by training a linear classifier with frozen learned embedding on
155
ImageNet data. We further consider evaluating our approach on ImageNet- 100, a selected subset of
100 classes of ImageNet-IK
5

---

Table 1: Comparisons with/without DCL under different numbers of batch sizes from 32 to 512.
Results show the effectness of DCL on four widely used benchmarks. The performance of DCL keeps steadier than the SimCLR baseline while the batch size varied.
Dataset
ImageNet- 100 (linear)
CIFARIO (kNN)
Batch Size
32
64
128
256
512
32
64
128
256
512
SimCLR [8]
74.2
77.6
79.3
80.7
81.3
78.9
80.4
81.1
81.4
81.3
SimCLR w/ DCL
80.8
82.0
81.9
83.1
82.8
83.7
84.4
84.4
84.2
83.5
Dataset
CIFARION 100 (kNN)
STLI0 (kNN)
Batch Size
32
64
128
256
512
32
64
128
256
512
SimCLR [8]
49.4
50.3
51.8
52
52.4
74.1
76.2
76.9
77.3
77.6
SimCLR w/ DCL
51.1
54.3
54.6
54.9
55
82.0
82.8
81.8
81.2
81.0
Table 2: kNN top-1 accuracy (%) comparison of SSL approaches on small-scale benchmarks: CIFARIO, CIFARIOO, and STL110. Results show that DCL. consistently improves its SimCLR baseline. With multi-cropping [17], DCLW reaches competitive performance within other
contrastive learning approaches [8, 7, 4, 12, 18].
KNN (top-1) SimCLR
MoCo
MoCo CLD
NPID
NPID CLD
Inv. Spread
Exemplar
DCL
DCLW wo merop
CIFAR10
81.4
82.1
87.5
80.8
86.7
76.5
84.1
87.8
CIFAR100
52.0
53.1
58.1
$1.6
57.5
N/A 81.6
N/A
54.9
58.8
STLIO
77.3
80.8
79.1
83.6
79.3
81.2
84.1
157
CIFAR and STL10 For CIFARIO, CIFAR 100, STL10, ResNet-18 [22] is used the encoder architecture. We set the temperature to 0.07. All models are trained for 200 epochs with SGD optimizer and base Ir = 0.03 batchsize 256.. /256. We follow NPID [4] on using = 200 nearest
159
neighbor (kNN) classifier. Note that on STL10, we follow [24] to use both train set and unlabeled set for model pre-training
161
162
4.2
Experiments and analysis
163
DCL on ImageNet This section illustrates the effect of our DCL under different batch sizes and queues. The initial setup is have 1024 batch size (SimCLR [8]) and 65536 queues (MoCo [7) and gradually reduce the batch size (SimCLR) and queue (MoCo) to show the corresponding top-1 accuracy by linear evaluation. Figure 3 indicates that without DCL, the top-1 accuracy drastically drops when batch size (SimCLR) or queue (MoCo) becomes very small. While with DCL, the performance keeps steadier than baselines (SimCLR: -4.1% vs. 8.3%, MoCo: -0.4% vs. 5.9%).
185
186
187
188
169
Specifically, Figure 3 further shows that in SimCLR, the performance with DCL improves from 61.8% to 65.9% under 256 batch size; MoCo with DCL improves from 54.7% to 60.8% under 256 queues. The comparison fully demonstrates the necessity of DCL, especially when the number of negatives is small. Although batch size is increased to 1024, we also note that DCL (66.1%) still improves over the SimCLR baseline (65.1%)
171
172
173
We further observe the same phenomenon ImageNet-100 data. Table I shows that, while with DCL, the performance only drops 2.3% compare to the SimCLR baseline of 7.1%.
175
176 In summary, it is worth noting that, while the batch size is small, the strength of qB,i which is used to push the negative samples away from the positive sample, is also relatively weak. This phenomenon tends to reduce the efficiency of learning representation. While taking advantage of DCL alleviates the performance gap between small and large batch sizes. Hence, through the analysis, we find out
177
178
179
DCL can simply tackle the batch size issue in contrastive learning. With this considerable advantage given by DCL, general SSI. approaches can be implemented with fewer computational resources or lower standard platforms.
181

---

Table 3: Comparisons between SimCLR baseline, DCL, and DCLW. Results indicate that DCL improves the performance of baseline. and DCLW further provides an extra boost. Note that results are under the batch size 256 and epoch 200. All of models are both trained and evaluated with same experimental settings.
Baseline
DCL
DCLW
CIFARIO
81.8
84.2 (+3.1)
84.8 (+3.7)
CIFAR 100
51.8
54.6 (+2.8)
54.9 (+3.1)
ImageNet- 100
79.3
81.9 (+2.6)
82.8 (+3.5)
ImageNet-
61.8
65.9 (+4.1)
66.9 (+5.1)
Table 4: ImageNet- IK top-1 accuracy (%) on SimCLR and MoCo v2 with/without DCL under few training epochs. We further list results under 200 epochs for clear comparison. With DCL., the performance of SimCLR trained under 100 epochs nearly reaches its performance under 200 epochs. The MoCo v2 with DCL also reaches higher accuracy than the baseline under 100 epochs.
SimCLR[8)
SimCLR w/ DCL
MoCo v2[25]
MoCo DCL
100 epoch
57.5
64.6
63.6
64.4
200 epoch
61.8
65.9
67.5
67.7
183
DCL on CIFAR and STL10 In Table and Table 3, it is observed that DCL also demonstrates summary, DCL outperforms its baseline by 3.1% (CIFAR10) 10) and 2.8% (CIFAR 100) and keeps the performance relatively steady under batch size 256. also improve the kNN accuracy of the SimCLR baseline on STL.10 by 3.9%.
184
its effectiveness on small-scale benchmarks.
186
187
Decoupled objective with re-weighting DCLW We only replace DC with Dow with no pos-
sible advantage from additional tricks. That is, both our approach and the baselines apply the same training instruction the OpenSelfSup benchmark [23] for fairness. Note that we empirically choose 0.5 in the experiments.
189
191
Results in Table indicates that, DCLW achieves extra 5.1% (ImageNet- IK), 3.5% (ImageNet- 100) gains compared to the baseline. For CIFAR data, extra 3.7% (CIFAR10). 3.1% are gained from the addition of DCLW. worth to note that, trained with 200 epochs, our DCLW reaches 66.9% with batch size 256, surpassing the SimCLR [8] baseline: 66.2% with batch size 8192.
192
193
194
195
4.3
Small-scale benchmark results: STL10, CIFARIO, and CIFAR100
196
For STL10, CIFAR10, and CIFARI00, we implement our DCL with ResNet-18 [22] as encoder backbone by following small-scale benchmark of CLDD [24]. All the models are trained for 200 epochs with 256 batch size and evaluate by using kNN accuracies (k 200).
197
198
189
Results in Table 2 indicates that, our DCLW with multi-cropping [17] consistently outperforms the state-of-the-art baselines CIFARIO, STLI0, and CIFARI00, Our DCL also demonstrates its capability while comparing against other baselines. More analysis of large-scale benchmarks can be found in Appendix.
200
202
203
4.4
Ablations
204
We perform extensive ablations on the hyperparameters of our DCL and DCLW on both ImageNet data and other small-scale data, i.e., CIFARIO, CIFAR 100, and STL10. By seeking better configurations empirically, we see that our approach gives consistent gains over the standard SimCLR baseline. In other ablations, we see that our DCL, achieves more gains over both SimCI. and MoCo v2, i.e., contrastive learning baselines, also when training for 100 epochs only.
205
207
7

---

Losa (wi DCL)
Loss (wi DGLI Loss (baselinc) Accuracy1d (Wl DCL) Accurscydd (baseline)
Loss jee DCL) Loss (baselina]
Less (baselins)
Accuracy@11 (et DCL) Accuracy81 (baseline)
Accurecy@11 (wi DGLI Acourosy01 (baselite)
Coodts
Figure 4: During the SSL pre-training, DCL speeds up the model convergence and provides better performance than the baseline on CIFAR and STL10 data.
209
Few learning epochs Our DCL is inspired by the traditional contrastive learning framework, which needs large batch size, long learning epochs to achieve higher performance. The previous state- of-the-art, SimCLR [8], heavily rely on large quantities accuracy. (e.g., 69.3% with up to 1000 epochs). The purpose of our DCL is to achieve higher learning efficiency with few learning epochs. We demonstrate the effectiveness of DCL contrastive learning frameworks SimCLR and MoCo v2. We choose the batch size of 256 (queue of 65536) the baseline and train the model with only 100 epochs instead of the normal number of 200. We make sure other parameter settings the same for fair comparison. Table 4 shows the result ImageNet- IK using linear evaluation. With DCL, SimCLR can achieve 64.6% top-1 accuracy with only 100 epochs compared SimCLR baseline: 57.5%; MoCo v2 with DCL reaches 64.4% compared to MoCo v2 baseline: 63.6% with 100 epochs pre-training.
211
learning epochs to obtain high top-1
213
215
217
219
We further demonstrate that, with DCL, learning representation becomes faster during the early stage of training. The reason is that DCL successfully solves the decoupled issue between positive and negative pairs. Figure 4 (a), (b), and (c), show that our DCL improves the speed of convergence and reaches higher performance than the baseline on CIFAR and STL10 data.
221
223
Conclusion
224
225
In this paper, we identify the negative-positive positive- (NPC) effect in SimCLR. By removing the NPC effect. reach new objective function, decoupled contrastive learning (DCL). The proposed DCL. loss function requires minimal modification to the SimCLR baseline and provides efficient, reliable, and nontrivial performance improvement on various benchmarks. Given the conceptual simplicity of DCL and that it requires neither momentum encoding, large batch sizes, or
226
228
230
long epochs to reach competitive performance, we wish that DCL can serve as a strong baseline for the contrastive-based SSL methods.
References
[1]
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. European Conference on Computer Vision, pages 69-84. Springer, 2016.
234
235
[2] Richard Zhang, Phillip Isola, and Alexei Efros. Colorful image colorization. In European conference on computer vision, pages 649-666. Springer, 2016.
236
237
[3] Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. arXiv preprint 1803, 1003:01803.00728. 2018.
239
[4] Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3733-3742, 2018.
241
242
[5]
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv preprint arXiv:1807.0377 037408 2018.
243
244
[6] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. arXiv preprint 190.00996,, 2019.

---

248
[7] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie. and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9729-9738, 2020.
247
249
[8]
Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002.05709, 2020.
250
99 Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural information processing systems, pages 2672-2680, 2014.
252
253
[10]
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv: 1,06444, 2015.
255
256
Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning an invariant mapping. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2006), 17-22 June 2006, New York, NY, USA, pages 1735-1742.
258
IEEE Computer Society, 2006.
280
[12] Mang Ye, Xu Zhang, Pong C Yuen, and Shih-Fu Chang. Unsupervised embedding learning via invariant and spreading instance feature. In Proceedings of the IEEE Conference on computer vision and pattern recognition, pages 6210-6219, 2019.
261
263
[13] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. CoRR, abs/2004.1 11362, 2020.
264
285
266
[14] Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. arXiv preprint arXiv: 707.708.888.. 2017.
267
[15] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Ávila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent- A new approach to self-supervised learning. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeuriPS 2020, December 6-12, 2020, virtual, 2020.
269
270
271
272
273
274
275
[16]
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. CoRR, abs/2011.10566, 2020.
276
277
[17]
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. arXiv preprint arXiv:2006.0990 09882, 2020.
278
279
280
[18]
Alexey Dosovitskiy, Philipp Fischer, Jost Tobias Springenberg. Martin Riedmiller, and Thomas Brox. Discriminative unsupervised feature learning with exemplar convolutional neural net- works. IEEE transactions on pattern analysis and machine intelligence, 3749)1134-1177,
282
2015.
284
[19] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. leee, 2009.
285
287
[20]
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
288
289
21
Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsuper- vised feature learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 215-223. JMLR Workshop and Conference Proceedings. 2011.
290
291
282
[2 Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and paltern recognition,
283
294
pages 770-778, 2016.
[23] Xiaohang Zhan, Jiahao Xie, Ziwei Liu, Dahua Lin, and Chen Change Loy. OpenSelfSup: Open mmlab supfrvupe learning toolbox and benchmark. 2020.
296
297
[24] Xudong Wang, Ziwei Liu, and X Yu Stella. Unsupervised feature learning by cross-level instance-group discrimination.
9

---

299
[25] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning, arXiv preprint arXiv:2003.0429) 2020.
300
Checklist
301
302
1. For all authors...
303
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
304
contributions and scope? [Yes]
305
(b) Did you describe the limitations of your work? [Yes] See supplementary.
306
(c) Did you discuss any potential negative societal impacts of your work? [No]
307
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
308
2. If you including theoretical results...
310
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Section 3.
(b) Did you include complete proofs of all theoretical results? [Yes] See Section 3.
312
3. If you ran experiments..
313
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material straightforward to reproduce. In addition, we will release our code public after the review process.
URL)? [N/A] It can be
315
316
317
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section4.
(c) Did you report error bars (e.g., with respect to the random seed after running exper- iments multiple times)? [No]. We only report the mean value of 5 runs but without standard deviation.
320
322
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Section4.
323
324
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
325
(a) If your work uses existing assets, did you cite the creators? [Yes]
326
(b) Did you mention the license of the assets? [Yes]
327
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you'l 're using/curating? [Yes]
329
330
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No]
5. If you used crowdsourcing conducted research with human subjects...
333
(a) Did you include the full text of instructions given to participants and scu're us- ing/cuapplicable? [Yes]
334
335
(b) Did you describe any potential participant risks, with links to Inu're using/cuview Board (IRB) approvals, if applicable? [Yes]
(c) Did you include the estimated hourly wage paid to participants and the tou't 'ree using/cunt on participant compensation? [Yes]
338
10


Reviews:
[{"review_id": "wRvDM_VE-Rw", "reviewer": "Reviewer_K21n", "summary": "This paper observes that in contrastive leanring the gradient contribution from positive pairs and negative pairs is scaled by a common multiplier that tends to be small in the small negative batch size N regime, and close to 1 for large N. The authors propose a minor modification to the InfoNCE loss function - removing the term on the denominator pertaining to the positive pair - that suffices to remove the multiplier (i.e. set it equal to 1 always). Although the change is minor, the authors observe an improvement in robustness to batch size in SimCLR (and queue size in MoCo) and find that fewer epochs of training are required to attain a certain level of performance (though in the large epoch regime the performance saturates at a similar optimal performance level as the vanilla InfoNCE loss). ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nThis paper has some merits. The motivation of reducing the coupling of gradient information from negative and positive samples is reasonable, the method sensible and very simple to implement, and the empirical results are encouraging. \n\nHowever, despite those merits, I also have concerns that make me unwilling to recommend acceptance.  The objective derived is similar in form to the alignment and uniformity loss [1]. In [1] the order of the expectation and exponential is swapped, but the *key feature* of removing the positive term in the denominator is there. If you computed the gradients of the alignment and uniformity loss you would also find that it avoids the cancellation effect that this work posits as its main goal to remove.  It is *critically important* in my mind that [1] is discussed in the related work, and added as a baseline in experiments (especially for Fig. 3 on sensitivity to batch size and Fig. 4 on convergence speed). I will not feel confident enough to recommend acceptance without that comparison. \n\nI strongly suspect that the objective of [1], if tuned correctly, would yield similar results to those in this paper (if I have missed some fundamental difference between the objective given here and that in [1] then please point it out - I will listen carefully and update my review on this point if needed). As a result of this, I would argue that the main contribution of this paper is not the proposed decoupled loss itself, but the observation that the loss can be motivated from the point of view of removing this gradient multiplier. While I think this is an interesting observation, on its own I do not feel it is sufficient contribution by the standards of NeurIPS. \n\n---\n\n## Minor criticisms:\n\n- Somewhat slapdash reporting of results - e.g. Figure 4 doesn’t even tell the reader which dataset corresponds to which figure. Figure 1(a) talks about something called the “coefficient of variation” - I still don’t know what this is since I could not find an explanation anywhere in the text. Even if the explanation is hidden away somewhere, it shouldn't bee so hard for a reader to find it (do you just mean variance/standard dev.?). Please do a review of the pedagogical basics and make sure all figures are properly labeled and explained. \n- In Figure 3 why use MoCo-v1 when you can just as easily run MoCo-v2 experiments and obtain more competitive baseline comparisons?\n- Figure 3(b) has little practical consequence - in MoCo a larger memory bank is very cheap and generally no one feels a great need to shrink the size of the memory bank. Fig 3(a) on SimCLR is more interesting since batch size is a genuine constraint/consideration when using SimCLR.\n- Table 2: baseline results are somewhat weak. SimCLR/MoCo-v2 results for CIFAR10 & STL10 results can, with a little care, be in the 90s (here they are generally in the low 80s) and CIFAR100 results in the high 60s or 70s (here they are in the 50s). This is not a major issue, and isn’t a deciding factor by any means, but stronger baselines are always preferable. \n- The reweighed loss version L_DCW is not much of an innovation. Previous work [2,3] apply almost identical re-weighting schemes (to positive and negative pairs). There is no mention of this prior work in the text. \n\n---\n\n## Clarity: \n\nThis section does not form a component of my final assessment of this work as a research contribution, but is intended to help the authors with improving clarity.\n- Clarity is fairly poor in general. Ideas that should be easy and intuitive are written in a convoluted way and end up harder than necessary to understand. I would especially suggest revising section 3, which contains the main idea of the paper.\n- l.92 “Simsam” —> SimSiam\n- Fig1 is on page 2, but relates to formulae & discussions on Page 4. I would suggest moving the figure to a more logical position. \n- prop 2 is stated in a slightly odd (inverted) way. I would suggest a more linear sequence of logic: 1) define the new loss, 2) state that the new loss has gradients that are the same as the original loss, but with the “NPC multipliers” removed. \n\n---\n\n**Possible directions for improvement:**\n\n(This component does not factor into the assessment, but attempts to give constructive suggestions on possible directions)\n\nThe main goal of this work - making contrastive learning more efficient in terms of both training time and memory requirements - is an interesting one that many researchers have thought about. While perhaps the multiplier may play a role in this, there still remains the fundamental issue that the “alignment” loss and the “uniformity” are trying to enforce *very different* goals. The large epoch requirement for contrastive learning is (in my opinion) likely due to the conflict between these two learning objectives. Its would be very interesting to see an extension of this work to not just discuss scalar multipliers in front of gradient terms, but also discuss conflicting gradient *directions* (the alignment terms gradient likely points in a conflicting direction from the uniformity terms gradient). \n\n----\n\n**Update 23rd Aug:**\n\nScore moved from 5 to 6 based on information provided in the rebuttal.\n\n---\n\n**References:**\n\n[1] Understanding contrastive representation learning through alignment and uniformity on the hypersphere, Wang and Isola \n\n[2] Contrastive learning with hard negative samples, Robinson et al.\n\n[3] Contrastive Attraction and Contrastive Repulsion for Representation Learning, Zheng et al.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors do not discuss limitations or possible societal impact. Since the work is on fundamental methodology I am not hugely worried about the societal impact part. I would, however, appreciate a discussion of the limitations (especially since in the checklist they claim that they do include such a discussion in the appendix, for me only to find that there wasn't one). \n\nThe main limitation I would put forward is one discussed above - that even with the proposed cancellation in the loss, there still remains a fundamental tension between the alignment and uniformity terms in all contrastive losses, and that this may well be the main factor driving the large training time requirements of contrastive learning. Note: I am not trying to say that the authors have to resolve this limitation, but the purpose of the limitations discussion is to give the reader a more honest idea of where the future directions for further improvement may lie. ", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3hrs"}, "extra_text": {}}, {"review_id": "cLEfv7MZ1-", "reviewer": "Reviewer_taNp", "summary": "The authors propose an improved loss based on the infoNCE loss and show that SimCLR and MoCo with the proposed loss converge faster and require smaller batch sizes / memory banks.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "My main concern about the paper is that given there are so many papers discussing the losses for contrastive learning, the authors did not cite any of them [1-3]. Some of these papers also focus on batch size sensitivity. I would recommend authors do a more comprehensive literature search and compare the proposed loss with these previous losses.\n\nThe thermotical analysis provides interesting insights but the authors seem not to claim it in the right way. Based on Figure 1 and my understanding, small batch sizes lead to a smaller chance to utilize hard negatives in each iteration, and consequently a simpler problem to optimize, i.e. $q_B \\rightarrow 0$. Removing $q_B$ simply increases the signals provided by the simple negatives and positives. Instead, the authors claim the main issue to be a negative-positive coupling problem, which is not clear to me why that impacts the training.\n\nThe authors do not provide the code. Although it is simple to reproduce and the authors mention that this loss is insensitive to the hyperparameters, they do not provide any evidence to support this other than the claims in 4.4.\n\nThe authors state in the checklist that \"they describe the limitations of the work in supplementary material\", but they don't.\n\nThe empirical results of this paper are strong and the theoretical analysis is interesting. However, the paper lacks several relevant references. And there is evident space for the authors to improve in both experiments and writing. Therefore, I'm inclined to reject the paper at the current stage.\n\nMinor issues:\n\n1. line 14: Moco --> MoCo\n\n2. Figure 1: $q_b$ --> $q_B$\n\n3. line 211: rely --> relies\n\n4. line 271-272: please delete the official template.\n\n\n[1] Tsai, Yao-Hung Hubert, et al. \"Self-supervised Representation Learning with Relative Predictive Coding.\" ICLR 2021.\n[2] Hjelm, R. Devon, et al. \"Learning deep representations by mutual information estimation and maximization.\" ICLR 2019.\n[3] Ozair, Sherjil, et al. \"Wasserstein Dependency Measure for Representation Learning.\" NeurIPS 2019.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors claim the limitations to be addressed in supplementary material, but I didn't find any. The potential negative societal impact is not discussed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}, {"review_id": "ZSWrd1Z3VOj", "reviewer": "Reviewer_mww5", "summary": "This paper gives a theoretical analysis of the contrastive loss to identify a \"negative-positive coupling multiplier\" term. By removing this term from the loss function, the authors achieved better performance over SimCLR and MOCO, particularly when using small batch sizes. ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Pros:**\n\n1. The authors identified a common multiplier term in the gradients of the well known contrastive loss, and observed that this multiplier (NPC) behaved differently for small vs large batch sizes. Since NPC fluctuates wildly for small batches, but converges to 1 for large batches, they propose to set NPC to 1 during training. The technique is very simple and does not require further tricks.\n\n2. There is also an improved version that uses a weighting between positive and negative terms in the loss function. By choosing a weighting function cleverly they avoid having more hyperparameters to tune. (The temperature parameter needed to be re-tuned though.)\n\n3. For small batch sizes, the performance and training efficiency on various datasets exceed SimCLR baselines. This is nice because these are two known disadvantages of contrastive learning (bad at small batches, slow training).\n\n4. Writing is quite clear.\n\n**Cons:**\n\nI have some concerns with regards to the significance of the results. Contrastive learning is known to not do well at small batch sizes. Follow-up methods such as BYOL have achieved much better results at small batch sizes. While it is very nice that this paper has improved over SimCLR and MOCO at small batches, it still does not match the later SSL methods, and practical applications could be limited. The benefits may also disappear when models and training are scaled up.\n\n**Nit:**\n\nLine 127 - “Removing the positive pair from the denominator of Equation 2” - I think this should be Equation 1?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This is not applicable to this paper.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "YEtn2nEOp9C", "reviewer": "Reviewer_iJNi", "summary": "The authors study the problem of contrastive learning, and in particular consider the NT-Xent loss function used in popular contrastive learning methods, such as the SimCLR algorithm. Such algorithms typically require long training times and very large batch sizes to be effective, and the authors explore this phenomenon in more detail.\n\nConcretely, they identify a term in the gradient of this loss that has a potentially undesirable effect (the negative-positive coupling term), supported through empirical analysis. They adjust the contrastive loss function to prevent this term appearing in the resulting gradient, with the goal of improving learning speed and robustness to small batches. \n\nIn experiments on common SSL datasets including CIFAR10/100, STL10, and ImageNet, they find that by removing this coupling term, their method can improve significantly on competitive SimCLR and MoCo baselines, at much smaller batch sizes. Furthermore, they find that their method trains much faster with this adjustment to the loss.\n\nDrawbacks: The authors do not appear to include a discussion of limitations in the paper or in the supplementary material. Furthermore, some of their baselines (in particular SimCLR) seems to be  weaker than other reported numbers in the literature, so this is worth examining in more detail. ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Originality \nThe paper contributes a nice mathematical argument as to what in the gradient of the  NT-Xent loss function might cause noisier learning (worse performance at small batch sizes and long training times). To the best of my knowledge, this is an original contribution, as is the specific adjustment to the loss function. \n\n# Quality \nThe paper on the whole does a good job of identifying an important problem, hypothesising the cause (coupling term), providing some mathematical analysis and suggesting a solution, and then running empirical studies to verify their claims. \n\nThe experiments are fairly thorough, and consider important benchmark datasets used in the literature (both small and large scale).  Applying the method to both SimCLR and MoCo is a good idea, and it is  interesting to observe benefits in both settings. The improvement in learning speed is a very nice benefit of the method, and this was good to see analysed.\n\nMy main concern is that the baselines are weaker than numbers that I have seen reported in the literature. For example, consider this open source, popular implementation of SimCLR on CIFAR10: https://github.com/leftthomas/SimCLR. The reported result with kNN eval, batch size 512, and 500 epochs of training is 89.1%, which is significantly higher than numbers reported in the paper (best comparison: 81.3% in Table 1. This is 200 epochs, not 500, but I am not sure if we should expect an ~8% improvement with extra training alone).\n\nAlthough the results are impressive, I would like to see the proposed method improve on a stronger SimCLR baseline, rather than one that seems to be fairly weak, to be more confident that the method truly is effective.\n\nRelatedly, I am interested to understand why the performance of the proposed method appears to worsen with increased batch sizes in some situations. Is this just variation across seeds, or something more fundamental?  \n\nIt would also help if the authors provided confidence intervals/measure of spread for the numbers reported. They mention running 5 repeats, and a measure of spread would be helpful (especially in understanding things like the above).\n\nIf computationally feasible, I am interested in the authors running an experiment with DCL with a large batch size (\\~1024) and large number of training epochs (\\~500) to understand if the performance degrades at these scales. It is good to understand if that might be a limitation of the method. I understand it is not a central focus of the paper, but it would be good to address this.\n\nFurthermore, the authors mention they ran ablations in section 4.4. Can they provide more information about these? Was it just the reduction in learning epochs?\n\n# Clarity\nThe paper is well written and laid out, and was very easy to read.\n\n# Significance\nPotentially high, but some outstanding questions regarding weak baselines, scalability to large batches and training times. \n\n# Summary\nOverall, I think this is a good contribution, but the authors do not discuss limitations of the work and the quality of the baselines is somewhat uncertain. If the authors can address these points, I will consider acceptance. For now, I think this is slightly below acceptance.\n\n# Update post author response\nThank you for responding to my questions and providing the additional results. I am more encouraged by the method's performance now, and appreciate the discussion on limitations. It would be good for the authors to add all the relevant extra discussion into the final version of the paper (better comparisons to related work and additional experimental results). Given this and also the responses to the other reviewer comments, I am increasing my score accordingly.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "See above, but the authors do not mention limitations in their paper. This should be addressed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents an analysis of contrastive learning with the InfoNCE objective that identifies a strong dependence of the gradient on batch size, and proposes a new objective to remove this batch-size dependence by eliminating the positive pair from the denominator of the InfoNCE loss. This decouples the influence of the positive and negative pairs on the loss function (L_DC) in an additive manner, which is not true for the original InfoNCE loss. Through extensive experiments (in the original paper as well as rebuttal), the effectiveness over InfoNCE at small batch size is verified. 

Reveiewers raised several questions around the experimental setting and comparison to additional approaches (Barlow Twins, SimSiam, BYOL) that were largely addressed in the rebuttal. However, there remained concerns around the impact of yet another contrastive loss function given the success of non-contrastive losses like BYOL at small batch sizes. While the theoretical analysis and results in Fig 1 highlight the batch-size dependence of the gradient scale, there is not discussion of why removing this batch-size dependent impact should yield improved representation learning abilities, or how the new objective may be related to other proposed losses (but some of these concerns have been nicely addressed in the rebuttal, e.g. relation to alignment + uniformity losses). 

[[Addressing these concerns and incorporating more results from the rebuttal into the main paper would greatly improve the paper, but I cannot recommend it for acceptance in the current state.]]
Decision: Reject