Decoupled Contrastive Learning
Year: 2021

Abstract:
Contrastive learning (CL) is one of the most successful paradigms for self1 supervised learning (SSL). Specifically, contrastive learning treats two augmented 2 “views” of the same sample as positive, pulling them close and treating all other 3 samples as negative to push them far apart. Despite the evident success of CL SSL 4 methods, there are several challenges in the existing methods as they may require 5 special structures, large batches, or huge training epochs, etc. Our motivation in 6 this work is to provide a simple, efficient, and yet competitive contrastive learning 7 baseline. Through both theoretical and empirical studies, we identified a strong 8 negative-positive-coupling (NPC) effect in the widely used cross-entropy loss in 9 CL SSL methods. We hypothesize that the NPC effect may be a major cause of the 10 inefficiency in many contrastive learning methods. By removing the NPC effect, 11 we reach a decoupled contrastive learning (DCL) objective function, which signifi12 cantly improves the training efficiency. DCL can achieve competitive performance, 13 requiring neither large batches in SimCLR, momentum encoding in Moco, or large 14 epochs. We demonstrate the benefit of DCL in various benchmarks. Further, DCL 15 is also much less sensitive to suboptimal hyperparameters. Notably, our approach 16 achieves 66.9% ImageNet top-1 accuracy with 256 batch size within 200 epochs 17 pre-training, which outperforms its baseline SimCLR by 5.1%. We believe DCL 18 may provide a strong baseline for future contrastive learning-based SSL studies. 19

Contrastive learning (CL) is one of the most successful paradigms for self-1 supervised learning (SSL). Specifically, contrastive learning treats two augmented2 “views” of the same sample as positive, pulling them close and treating all other3 samples as negative to push them far apart. Despite the evident success of CL SSL4 methods, there are several challenges in the existing methods as they may require5 special structures, large batches, or huge training epochs, etc. Our motivation in6 this work is to provide a simple, efficient, and yet competitive contrastive learning7 baseline. Through both theoretical and empirical studies, we identified a strong8 negative-positive-coupling (NPC) effect in the widely used cross-entropy loss in9 CL SSL methods. We hypothesize that the NPC effect may be a major cause of the10 inefficiency in many contrastive learning methods. By removing the NPC effect,11 we reach a decoupled contrastive learning (DCL) objective function, which signifi-12 cantly improves the training efficiency. DCL can achieve competitive performance,13 requiring neither large batches in SimCLR, momentum encoding in Moco, or large14 epochs. We demonstrate the benefit of DCL in various benchmarks. Further, DCL15 is also much less sensitive to suboptimal hyperparameters. Notably, our approach16 achieves 66.9% ImageNet top-1 accuracy with 256 batch size within 200 epochs17 pre-training, which outperforms its baseline SimCLR by 5.1%. We believe DCL18 may provide a strong baseline for future contrastive learning-based SSL studies.19
1 Introduction20
As a fundamental task in machine learning, representation learning aims to extract features to21 reconstruct the raw data fully. It has been regarded as a long-acting goal over the past decades. Recent22 progress on representation learning has achieved a significant milestone over self-supervised learning23 (SSL), facilitating feature learning with its competence in exploiting massive raw data without any24 annotated supervision. In the early stage of SSL, representation learning has focused on exploiting25 pretext tasks, which are addressed by generating pseudo-labels to the unlabeled data through different26 transformations, such as solving jigsaw puzzles [1], colorization [2] and rotation prediction [3].27 Though these approaches achieve some success in computer vision, there is a large gap between28 these methods and supervised learning. Recently, there has been a significant advancement in using29 contrastive learning [4, 5, 6, 7, 8] for self-supervised pre-training, which significantly closes the gap30 between the SSL method and supervised learning. Contrastive SSL methods, e.g., SimCLR [8], in31 general, try to pull different views of the same instance close and push different instances far apart in32 the representation space.33
Despite the evident progress of the state-of-the-art contrastive SSL methods, there have been several34 challenges in future developing this direction: 1) The SOTA models [7] may require unique structures35 like the momentum encoder and large memory queues, which may complicate the understanding. 2)36 The contrastive SSL models [8] may depend on large batch size and huge epoch numbers to achieve37 competitive performance, posing a computational challenge for academia to explore this direction.38
Submitted to 35th Conference on Neural Information Processing Systems (NeurIPS 2021). Do not distribute.
3) They may be sensitive to hyperparameters and optimizers, introducing additional difficulty to39 reproduce the results on various benchmarks.40
Our motivation in this work is to provide a simple, efficient, and yet competitive contrastive learning41 baseline. We choose SimCLR as our starting point, given its conceptual simplicity. By analyzing the42 objective function, we identified a Negative-Positive-Coupling (NPC) multiplier qB in the gradient43 as shown in Proposition 1. The NPC multipliers modulate the gradient of each sample, and it44 mistakenly increases the impact of both negative samples and positive samples, given either of them45 is more informative. Such a coupling exacerbates when smaller batch sizes are used. By removing46 the coupling term, we reach a new formulation, the decoupled contrastive learning (DCL). The47 new objective function significantly improves the training efficiency, requires neither large batches,48 momentum encoding, or large epochs to achieve competitive performance on various different49 benchmarks. Specifically, DCL reaches 66.9% ImageNet top-1 (linear probing) accuracy with batch50 size 256, SGD optimizer within 200 epochs. Even if DCL is trained for 100 epochs, it still reaches51 64.6% ImageNet top-1 accuracy with batch size 256.52
In short, this work makes the following contributions:53
1) We provide both theoretical analysis and empirical evidence to show the negative-positive54 coupling in the gradient of contrastive learning;55
2) We introduce a new, decoupled contrastive learning (DCL) objective, which casts off the56 coupling phenomenon between positive and negative samples in contrastive learning, and57 significantly improves the training efficiency; Additionally, the proposed DCL objective is58 less sensitive the several important hyperparameters;59
3) We demonstrate our approach via extensive experiments and analysis on both large and60 small-scale vision benchmarks, with an optimal configuration for the standard SimCLR61 baseline to have a competitive performance within contrastive approaches.62
2 Related work63
2.1 Self-supervised representation learning64
Self-supervised representation learning (SSL) aims to learn a robust embedding space from data65 without human annotation. Previous arts can be roughly categorized into generative and discriminative.66 Generative approaches, such as autoencoders and adversarial learning, focus on reconstructing67 images from latent representations [9, 10]. Conversely, recent discriminative approaches, especially68 contrastive learning-based approaches, have gained the most ground and achieved state-of-the-art69 standard large-scale image classification benchmarks with increasingly more compute and data70 augmentations.71
2.2 Contrastive learning72
Contrastive learning (CL) constructs positive and negative sample pairs to extract information from73 the data itself. In CL, each anchor image in a batch has only one positive sample to construct a positive74 sample pair [11, 8, 7]. CPC [5] predicts the future output of sequential data by using current output75 as prior knowledge, which can improve the feature representing the ability of the model. Instance76 discrimination [4] proposes a non-parametric cross-entropy loss to optimize the model at the instance77 level. Inv. spread [12] makes use of data augmentation invariants and the spread-out property of78 instance to learn features. MoCo [7] proposes a dictionary to maintain a negative sample set, thus79 increasing the number of negative sample pairs. Different from the aforementioned self-supervised80 CL approaches, [13] proposes a supervised CL that considers all the same categories as positive pairs81 to increase the utility of images.82
2.3 Collapsing issue via batch size and negative sample83
In CL, the objective is to maximize the mutual information between the positive pairs. However, to84 avoid the “collapsing output”, vast quantities of negative samples are needed so that the learning85 objectives obtain the maximum similarity and have the minimum similarity with negative samples.86 For instance, in SimCLR [8], training requires many negative samples, leading to a large batch size87 (i.e., 4096). Furthermore, to optimize such a huge batch, a specially designed optimizer LARS [14]88 is used. Similarly, MoCo [7] needs a vast queue (i.e., 65536) to achieve competitive performance.89 BYOL [15] does not collapse output without using any negative samples by considering all the90 images are positive and to maximize the similarity of “projection” and “prediction ” features. On the91 other hand, Simsam [16] leverages the Siamese network to introduce inductive biases for modeling92 invariance. With the small batch size (i.e., 256), Simsam is a rival to BYOL (4096). Unlike both93 approaches that achieved their success through empirical studies, this paper tackles from a theoretical94 perspective, proving that an intertwined multiplier qB of positive and negative is the main issue to95 contrastive learning.96
3 Decouple negative and positive samples in contrastive learning97
(a) (b)
We choose to start from SimCLR because of its conceptual simplicity. Given a batch of N samples98 (e.g. images), {x1, . . . ,xN}, let x(1)i ,x (2) i be two augmented views of the sample xi and B be99 the set of all of the augmented views in the batch, i.e. B = {x(k)i |k ∈ {1, 2}, i ∈ [[1, N ]]}. As100
shown by Figure 2(a), each of the views x(k)i is sent into the same encoder network f and the output101 h (k) i = f(x (k) i ) is then projected by a normalized MLP projector that z (k) i = g(h (k) i )/‖g(h (k) i )‖.102 For each augmented view x(k)i , SimCLR solves a classification problem by using the rest of the103 views in B as targets, and assigns the only positive label to x(u)i , where u 6= k. So SimCLR104 creates a cross-entropy loss function L(k)i for each view x (k) i , and the overall loss function is105
L = ∑ k∈{1,2},i∈[[1,N ]] L (k) i .106
L (k) i = − log
exp(〈z(1)i , z (2) i 〉/τ)
exp(〈z(1)i , z (2) i 〉/τ) + ∑ l∈{1,2},j∈[[1,N ]],j 6=i exp(〈z (k) i , z (l) j 〉/τ)
(1)
Proposition 1. There exists a negative-positive coupling (NPC) multiplier q(1)B,i in the gradient of107
L (1) i :108  −∇ z (1) i L (1) i = q (1) B,i τ [ z (2) i − ∑ l∈{1,2},j∈[[1,N ]],j 6=i exp 〈z(1)i ,z (l) j 〉/τ∑ q∈{1,2},j∈[[1,N]],j 6=i exp(〈z (1) i ,z (q) j 〉/τ) · z(l)j ] −∇ z (2) i L (1) i = q (1) B,i τ · z (1) i
−∇ z (l) j L (1) i = −
q (1) B,i
τ
exp 〈z(1)i ,z (l) j 〉/τ∑
q∈{1,2},j∈[[1,N]],j 6=i exp(〈z (1) i ,z (q) j 〉/τ)
· z(1)i
(2)
where the NPC multiplier q(1)B,i is:109
q (1) B,i = 1− exp(〈z(1)i , z (2) i 〉/τ)∑
q∈{1,2},j∈[[1,N ]],j 6=i exp(〈z (1) i , z (q) j 〉/τ)
(3)
Due to the symmetry, a similar NPC multiplier q(k)B,i exists in the gradient of L (k) i , k ∈ {1, 2}, i ∈110 [[1, N ]].111
As we can see, all of the partial gradients in Equation 2 are modified by the common NPC multiplier112 q (k) B,i in Equation 3. Equation 3 makes intuitive sense: 1) When a positive sample pair {z (1) i , z (2) i } are113 farther, the corresponding NPC multiplier q(1)B,i is larger. This will makes the overall gradient larger.114
Otherwise, the gradient is smaller. 2) When a negative sample is closer to z(1)i , it makes q (1) B,i larger.115 Overall, the intuition here is that a positive sample farther from the target or a negative sample closer116 to the target is more informative. However, the positive samples and negative samples are strongly117 coupled. An outlier positive sample also makes the gradient from the negative samples significantly118 larger and vice versa.119
Figure 1(b) shows the NPC multiplier qB distribution shift w.r.t. different batch sizes for a pre-trained120 SimCLR baseline model. While all of the shown distributions have prominent fluctuation, the smaller121 batch size makes qB cluster towards 0, while the larger batch size pushes the distribution towards122 δ(1). Figure 1(a) shows the averaged NPC multiplier 〈qB〉 changes w.r.t. the batch size and the123 relative fluctuation. The small batch sizes introduce significant NPC fluctuation. Based on this124 observation, we propose to remove the NPC multipliers from the gradients, which corresponds to the125 case qB,N→∞. This leads to the decoupled contrastive learning formulation.126
Proposition 2. Removing the positive pair from the denominator of Equation 2 leads to a decoupled127 contrastive learning loss. If we remove the NPC multiplier q(k)B,i from Equation 2, we reach a128
decoupled contrastive learning loss LDC = ∑ k∈{1,2},i∈[[1,N ]] L (k) DC,i, where L (k) DC,i is:129
L (k) DC,i = − log exp(〈z(1)i , z (2) i 〉/τ)hhhhhhhhexp(〈z (1) i , z (2) i 〉/τ) + ∑ l∈{1,2},j∈[[1,N ]],j 6=i exp(〈z (k) i , z (l) j 〉/τ)
(4)
= −〈z(1)i , z (2) i 〉/τ + log ∑ l∈{1,2},j∈[[1,N ]],j 6=i exp(〈z(k)i , z (l) j 〉/τ) (5)
The proofs of Proposition 1 and 2 are given in Appendix. Further, we can generalize the loss130 function LDC to LDCW by introducing a weighting function for the positive pairs i.e. LDCW =131 ∑ k∈{1,2},i∈[[1,N ]] L (k) DCW,i.132
L (k) DCW,i = −w(z (1) i , z (2) i )(〈z (1) i , z (2) i 〉/τ) + log ∑ l∈{1,2},j∈[[1,N ]],j 6=i exp(〈z(k)i , z (l) j 〉/τ) (6)
where we can intuitively choose w to be a negative von Mises-Fisher weighting function that133
w(z (1) i , z (2) i ) = 2 −
exp(〈z(1)i ,z (2) i 〉/σ) E i [ exp(〈z(1)i ,z (2) i 〉/σ) ] and E [w] = 1. LDC is a special case of LDCW and we134 can see that limσ→∞ LDCW = LDC . The intuition behind w(z (1) i , z (2) i ) is that there is more learning135 signal when a positive pair of samples are far from each other.136
4 Experiments137
This section evaluates our proposed decoupled contrastive learning (DCL) empirically and compares138 it to the general contrastive learning methods. We summarize our experiments and analysis as the139 following: (1) our proposed work significantly outperforms the general contrastive learning on large140 and small-scale vision benchmarks; (2) we show the better version of DCL: LDCW could further141 improve the representation quality. (3) we further analyze our DCL with few learning epochs, which142 shows fast convergence of the proposed DCL. Detailed experimental settings can be found in the143 Appendix.144
4.1 Implementation details145
To understand the effect of the sample decoupling, we consider our proposed DCL, which is based on146 the general contrastive learning, where model optimization is irrelevant to the size of batches (i.e.,147 negative samples). Extensive experiments and analysis are demonstrated on large-scale benchmarks:148 ImageNet-1K [19], ImageNet-100 [6], and small-scale benchmark: CIFAR [20], and STL10 [21].149 Note that all of our experiments are conducted with 8 Nvidia V100 GPUs on a single machine.150
ImageNet For a fair comparison on ImageNet data, we implement our proposed decoupled structure,151 DCL by following SimCLR [8] with ResNet-50 [22] as the encoder backbone and use cosine annealing152 schedule. We set the temperature τ to 0.1 and the latent vector dimension to 128. Following [23],153 we evaluate the pre-trained models by training a linear classifier with frozen learned embedding on154 ImageNet data. We further consider evaluating our approach on ImageNet-100, a selected subset of155 100 classes of ImageNet-1K.156
CIFAR and STL10 For CIFAR10, CIFAR100, and STL10, ResNet-18 [22] is used as the encoder157 architecture. We set the temperature τ to 0.07. All models are trained for 200 epochs with SGD158 optimizer and a base lr = 0.03 ∗ batchsize/256. We follow NPID [4] on using k = 200 nearest159 neighbor (kNN) classifier. Note that on STL10, we follow [24] to use both train set and unlabeled160 set for model pre-training.161
4.2 Experiments and analysis162
DCL on ImageNet This section illustrates the effect of our DCL under different batch sizes and163 queues. The initial setup is to have 1024 batch size (SimCLR [8]) and 65536 queues (MoCo [7])164 and gradually reduce the batch size (SimCLR) and queue (MoCo) to show the corresponding top-1165 accuracy by linear evaluation. Figure 3 indicates that without DCL, the top-1 accuracy drastically166 drops when batch size (SimCLR) or queue (MoCo) becomes very small. While with DCL, the167 performance keeps steadier than baselines (SimCLR:−4.1% vs. −8.3%, MoCo: −0.4% vs. −5.9%).168 Specifically, Figure 3 further shows that in SimCLR, the performance with DCL improves from169 61.8% to 65.9% under 256 batch size; MoCo with DCL improves from 54.7% to 60.8% under 256170 queues. The comparison fully demonstrates the necessity of DCL, especially when the number of171 negatives is small. Although batch size is increased to 1024, we also note that our DCL (66.1%) still172 improves over the SimCLR baseline (65.1%).173
We further observe the same phenomenon on ImageNet-100 data. Table 1 shows that, while with174 DCL, the performance only drops 2.3% compare to the SimCLR baseline of 7.1%.175
In summary, it is worth noting that, while the batch size is small, the strength of qB,i, which is used to176 push the negative samples away from the positive sample, is also relatively weak. This phenomenon177 tends to reduce the efficiency of learning representation. While taking advantage of DCL alleviates178 the performance gap between small and large batch sizes. Hence, through the analysis, we find out179 DCL can simply tackle the batch size issue in contrastive learning. With this considerable advantage180 given by DCL, general SSL approaches can be implemented with fewer computational resources or181 lower standard platforms.182
DCL on CIFAR and STL10 In Table 1 and Table 3, it is observed that DCL also demonstrates183 its effectiveness on small-scale benchmarks. In summary, DCL outperforms its baseline by 3.1%184 (CIFAR10) and 2.8% (CIFAR100) and keeps the performance relatively steady under batch size 256.185 We also improve the kNN accuracy of the SimCLR baseline on STL10 by 3.9%.186
Decoupled objective with re-weighting DCLW We only replace LDC with LDCW with no pos-187 sible advantage from additional tricks. That is, both our approach and the baselines apply the same188 training instruction of the OpenSelfSup benchmark [23] for fairness. Note that we empirically choose189 σ = 0.5 in the experiments.190
Results in Table 3 indicates that, DCLW achieves extra 5.1% (ImageNet-1K), 3.5% (ImageNet-100)191 gains compared to the baseline. For CIFAR data, extra 3.7% (CIFAR10), 3.1% are gained from the192 addition of DCLW. It is worth to note that, trained with 200 epochs, our DCLW reaches 66.9% with193 batch size 256, surpassing the SimCLR [8] baseline: 66.2% with batch size 8192.194
4.3 Small-scale benchmark results: STL10, CIFAR10, and CIFAR100195
For STL10, CIFAR10, and CIFAR100, we implement our DCL with ResNet-18 [22] as encoder196 backbone by following small-scale benchmark of CLD [24]. All the models are trained for 200197 epochs with 256 batch size and evaluate by using kNN accuracies (k = 200).198
Results in Table 2 indicates that, our DCLW with multi-cropping [17] consistently outperforms the199 state-of-the-art baselines on CIFAR10, STL10, and CIFAR100. Our DCL also demonstrates its200 capability while comparing against other baselines. More analysis of large-scale benchmarks can be201 found in Appendix.202
4.4 Ablations203
We perform extensive ablations on the hyperparameters of our DCL and DCLW on both ImageNet data204 and other small-scale data, i.e., CIFAR10, CIFAR100, and STL10. By seeking better configurations205 empirically, we see that our approach gives consistent gains over the standard SimCLR baseline. In206 other ablations, we see that our DCL achieves more gains over both SimCLR and MoCo v2, i.e.,207 contrastive learning baselines, also when training for 100 epochs only.208
Few learning epochs Our DCL is inspired by the traditional contrastive learning framework, which209 needs a large batch size, long learning epochs to achieve higher performance. The previous state-210 of-the-art, SimCLR [8], heavily rely on large quantities of learning epochs to obtain high top-1211 accuracy. (e.g., 69.3% with up to 1000 epochs). The purpose of our DCL is to achieve higher learning212 efficiency with few learning epochs. We demonstrate the effectiveness of DCL in contrastive learning213 frameworks SimCLR and MoCo v2. We choose the batch size of 256 (queue of 65536) as the baseline214 and train the model with only 100 epochs instead of the normal number of 200. We make sure other215 parameter settings are the same for a fair comparison. Table 4 shows the result on ImageNet-1K216 using linear evaluation. With DCL, SimCLR can achieve 64.6% top-1 accuracy with only 100 epochs217 compared to SimCLR baseline: 57.5%; MoCo v2 with DCL reaches 64.4% compared to MoCo v2218 baseline: 63.6% with 100 epochs pre-training.219
We further demonstrate that, with DCL, learning representation becomes faster during the early stage220 of training. The reason is that DCL successfully solves the decoupled issue between positive and221 negative pairs. Figure 4 (a), (b), and (c), show that our DCL improves the speed of convergence and222 reaches higher performance than the baseline on CIFAR and STL10 data.223
5 Conclusion224
In this paper, we identify the negative-positive-coupling (NPC) effect in SimCLR. By removing225 the NPC effect, we reach a new objective function, decoupled contrastive learning (DCL). The226 proposed DCL loss function requires minimal modification to the SimCLR baseline and provides227 efficient, reliable, and nontrivial performance improvement on various benchmarks. Given the228 conceptual simplicity of DCL and that it requires neither momentum encoding, large batch sizes, or229 long epochs to reach competitive performance, we wish that DCL can serve as a strong baseline for230 the contrastive-based SSL methods.231
References232 [1] Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving233 jigsaw puzzles. In European Conference on Computer Vision, pages 69–84. Springer, 2016.234 [2] Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In European235 conference on computer vision, pages 649–666. Springer, 2016.236 [3] Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by237 predicting image rotations. arXiv preprint arXiv:1803.07728, 2018.238 [4] Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via239 non-parametric instance discrimination. In Proceedings of the IEEE Conference on Computer240 Vision and Pattern Recognition, pages 3733–3742, 2018.241
[5] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive242 predictive coding. arXiv preprint arXiv:1807.03748, 2018.243
[6] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. arXiv preprint244 arXiv:1906.05849, 2019.245
[7] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for246 unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on247 Computer Vision and Pattern Recognition, pages 9729–9738, 2020.248
[8] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework249 for contrastive learning of visual representations. arXiv preprint arXiv:2002.05709, 2020.250
[9] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil251 Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural252 information processing systems, pages 2672–2680, 2014.253
[10] Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with254 deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434, 2015.255
[11] Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning an256 invariant mapping. In 2006 IEEE Computer Society Conference on Computer Vision and257 Pattern Recognition (CVPR 2006), 17-22 June 2006, New York, NY, USA, pages 1735–1742.258 IEEE Computer Society, 2006.259
[12] Mang Ye, Xu Zhang, Pong C Yuen, and Shih-Fu Chang. Unsupervised embedding learning via260 invariant and spreading instance feature. In Proceedings of the IEEE Conference on computer261 vision and pattern recognition, pages 6210–6219, 2019.262
[13] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron263 Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. CoRR, abs/2004.11362,264 2020.265
[14] Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks.266 arXiv preprint arXiv:1708.03888, 2017.267
[15] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena268 Buchatskaya, Carl Doersch, Bernardo Ávila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar,269 Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent - A270 new approach to self-supervised learning. In Hugo Larochelle, Marc’Aurelio Ranzato, Raia271 Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information272 Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020,273 NeurIPS 2020, December 6-12, 2020, virtual, 2020.274
[16] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. CoRR,275 abs/2011.10566, 2020.276
[17] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin.277 Unsupervised learning of visual features by contrasting cluster assignments. arXiv preprint278 arXiv:2006.09882, 2020.279
[18] Alexey Dosovitskiy, Philipp Fischer, Jost Tobias Springenberg, Martin Riedmiller, and Thomas280 Brox. Discriminative unsupervised feature learning with exemplar convolutional neural net-281 works. IEEE transactions on pattern analysis and machine intelligence, 38(9):1734–1747,282 2015.283
[19] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-284 scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern285 recognition, pages 248–255. Ieee, 2009.286
[20] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images.287 2009.288
[21] Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsuper-289 vised feature learning. In Proceedings of the fourteenth international conference on artificial290 intelligence and statistics, pages 215–223. JMLR Workshop and Conference Proceedings, 2011.291
[22] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image292 recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition,293 pages 770–778, 2016.294
[23] Xiaohang Zhan, Jiahao Xie, Ziwei Liu, Dahua Lin, and Chen Change Loy. OpenSelfSup: Open295 mmlab self-supervised learning toolbox and benchmark. 2020.296
[24] Xudong Wang, Ziwei Liu, and X Yu Stella. Unsupervised feature learning by cross-level297 instance-group discrimination.298
[25] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum299 contrastive learning. arXiv preprint arXiv:2003.04297, 2020.300
Checklist301
1. For all authors...302 (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s303 contributions and scope? [Yes]304 (b) Did you describe the limitations of your work? [Yes] See supplementary.305 (c) Did you discuss any potential negative societal impacts of your work? [No]306 (d) Have you read the ethics review guidelines and ensured that your paper conforms to307 them? [Yes]308 2. If you are including theoretical results...309
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Section 3.310 (b) Did you include complete proofs of all theoretical results? [Yes] See Section 3.311
3. If you ran experiments...312 (a) Did you include the code, data, and instructions needed to reproduce the main exper-313 imental results (either in the supplemental material or as a URL)? [N/A] It can be314 straightforward to reproduce. In addition, we will release our code public after the315 review process.316
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they317 were chosen)? [Yes] See Section4.318
(c) Did you report error bars (e.g., with respect to the random seed after running exper-319 iments multiple times)? [No] . We only report the mean value of 5 runs but without320 standard deviation.321
(d) Did you include the total amount of compute and the type of resources used (e.g., type322 of GPUs, internal cluster, or cloud provider)? [Yes] See Section4.323
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...324 (a) If your work uses existing assets, did you cite the creators? [Yes]325 (b) Did you mention the license of the assets? [Yes]326 (c) Did you include any new assets either in the supplemental material or as a URL? [Yes]327 (d) Did you discuss whether and how consent was obtained from people whose data you’re328 using/curating? [Yes]329 (e) Did you discuss whether the data you are using/curating contains personally identifiable330 information or offensive content? [No]331 5. If you used crowdsourcing or conducted research with human subjects...332
(a) Did you include the full text of instructions given to participants and scu’re us-333 ing/cuapplicable? [Yes]334
(b) Did you describe any potential participant risks, with links to Inu’re using/cuview335 Board (IRB) approvals, if applicable? [Yes]336
(c) Did you include the estimated hourly wage paid to participants and the tou’re using/cunt337 on participant compensation? [Yes]338

Review summaries:
This paper observes that in contrastive leanring the gradient contribution from positive pairs and negative pairs is scaled by a common multiplier that tends to be small in the small negative batch size N regime, and close to 1 for large N. The authors propose a minor modification to the InfoNCE loss function - removing the term on the denominator pertaining to the positive pair - that suffices to remove the multiplier (i.e. set it equal to 1 always). Although the change is minor, the authors observe an improvement in robustness to batch size in SimCLR (and queue size in MoCo) and find that fewer epochs of training are required to attain a certain level of performance (though in the large epoch regime the performance saturates at a similar optimal performance level as the vanilla InfoNCE loss). 

The authors study the problem of contrastive learning, and in particular consider the NT-Xent loss function used in popular contrastive learning methods, such as the SimCLR algorithm. Such algorithms typically require long training times and very large batch sizes to be effective, and the authors explore this phenomenon in more detail.  Concretely, they identify a term in the gradient of this loss that has a potentially undesirable effect (the negative-positive coupling term), supported through empirical analysis. They adjust the contrastive loss function to prevent this term appearing in the resulting gradient, with the goal of improving learning speed and robustness to small batches.   In experiments on common SSL datasets including CIFAR10/100, STL10, and ImageNet, they find that by removing this coupling term, their method can improve significantly on competitive SimCLR and MoCo baselines, at much smaller batch sizes. Furthermore, they find that their method trains much faster with this adjustment to the loss.  Drawbacks: The authors do not appear to include a discussion of limitations in the paper or in the supplementary material. Furthermore, some of their baselines (in particular SimCLR) seems to be  weaker than other reported numbers in the literature, so this is worth examining in more detail.