Supervision Accelerates Pre-training in Contrastive Semi-Supervised Learning of Visual Representations
Authors: Mido Assran
Nicolas Ballas
Lluis Castrejon
Michael Rabbat
Venue:  (2021)
Keywords: semi-supervised learning
contrastive learning
self-supervised learning
deep learning
representation learning
metric learning
visual representations

Abstract: We investigate a strategy for improving the efficiency of contrastive learning of visual representations by leveraging a small amount of supervised information during pre-training. We propose a semi-supervised loss, SuNCEt, based on noise-contrastive estimation and neighbourhood component analysis, that aims to distinguish examples of different classes in addition to the self-supervised instance-wise pretext tasks. On ImageNet, we find that SuNCEt can be used to match the semi-supervised learning accuracy of previous contrastive approaches while using less than half the amount of pre-training and compute. Our main insight is that leveraging even a small amount of labeled data during pre-training, and not only during fine-tuning, provides an important signal that can significantly accelerate contrastive learning of visual representations.

Under review as a conference paper at ICLR 2021
SUPERVISION ACCELERATES PRE-TRAINING IN CONTRASTIVE OF VISUAL REPRESENTATIONS
LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
We investigate a strategy for improving the efficiency of contrastive learning of visual representations by leveraging small amount of supervised information during pre-training. We propose semi- cuni--upervised loss, SuNCEt, based on noise-contrastin estimation and neighbourhood component analysis, that aims to distinguish examples of different classes in addition the self-supervised instance- wise pretext tasks. On ImageNet, we find that SuNCEt can be used to match the semi-supervised learning accuracy of previous contrastive approaches while using less than half the amount of pre-training and compute. Our main insight is that leveraging even a small amount of labeled data during pre-training, and not only during fine-tuning, provides an important signal that significantly accelerate contrastive learning visual representations.
INTRODUCTION
Learning visual representations that are semantically meaningful with limited semantic annotations a longstanding challenge with the potential drastically improve the data-efficienccy of learning agents. Semi-supervised learning algorithms based on contrastive instance-wise pretext tasks learn representations with limited label information and have shown great promise (Hadsell al., 2006; Wu et al., 2018b; Bachman et al., 2019; Misra & van der Maaten, 2020; Chen et al., 2020a). Unfortunately, despite achieving state-of-te offte-of-thh-arr art performance, these cumi-supervised contrastive approaches typically require at least an order of magnitude more compute than standard supervised training with a cross-entropy loss (albeit without requiring access to the same amount of labeled data). Burdensome computational requirements not only make training laborious and particularly time- and energy consuming, they also exacerbate other issues, making more difficult to scale to more complex models and problems, and potentially inducing significant carbon footprints depending on the infrastructure used for training (Henderson et al., 2020)
In this work, investigate strategy for improving the computational efficiency of contrastive learning of visual representations by leveraging a small amount of supervised information during pre-training. We propose a semi-supervised loss, SuNCEt, based on noise-contrastive estimation (Gut- mann & Hyvärinen, 2010) and neighbourhood component analysis (Goldberger et al., 2005), that aims distinguishing examples of different classes in addition the self-supervised iustance-wise pretext tasks. We conduct a case-study with respect to the approach of Chen et al. (2020a) on the ImageNet (Russakovsky et al., 2015) and CIFARIO (Krizhevsky & Hinton, 2009) benchmarks. We find that using any available labels during pre-training (either in the form of a cross-entropy loss or SuNCEt ) can be used to reduce the amount pre-training required. Our most notable results on ImageNet are obtained with SuNCEt, where we can match the semi-supervised learning accuracy of previous contrastive approaches while using less than half the amount of pre-traing pre-training and compute, and require syper-paramete tuning. By combining SuNCEt with the contrastive SwAV method -
of Caron et al. (2020), we also achieve state-of of the top-5 accuracy on ImageNet with 10% labels. while cutting the pre-training epochs in half.

---

Under review as a conference paper at ICLR 2021
2 BACKGROUND
The goal of contrastive learning is to learn representations by comparison. Recently, this class of approaches has fueled rapid progress in unsupervised representation learning of images through self- supervision (Chopra et al., 2005; Hadsell et al., 2006; Bachman et al., 2019; Oord et al., 2018; Hénaff al.,, 2019; Tian al., 2019; Misra & van der Maaten, 2020; He et al., 2019; Arora al., 2019; Chen et al. 2020a; Caron et al., 2020; Grill et al., 2020; Chen et al., 2020b). In that context, contrastive approaches usually learn by maximizing the agreement between representations of different views of the same image, either directly, via instance discrimination, or indirectly through, cluster prototypes. Instance- wise approaches perform pairwise comparison of input data to push representations of similar inputs close to one another while pushing apart representations of dissimilar inputs, akin to form of distance-metric learning.
Sule-super contrastive approaches typically rely data-augmentation module, encoder network, and a contrastive loss. The data augmentation module stochastically maps an image 2; E R to different view. Denote by 2,,1, two possible views of an image 2;, and denote by fe the parameterized encoder, which maps an input image 1;,1 to a representation vector 2 = fo 1 E IRd. The encoder fe is usually parameterized as deep neural network with learnable parameters A. Given representation 2;,1, referred as an anchor embedding, and the representation of an alternative view of the same input 21,2, referred to as a positive sample, the goal is to optimize the encoder fu to output representations that enable one to easily discriminate between the positive sample and noise using multinomial logistic regression. This learning by picking out the positive sample from pool of negatives is in the spirit of noise-contrastive estimation (Gutmann & Hyvärinen, 2010). The noise samples in this context are often taken to be the representations of other images. For example, suppose we have a set of images (z:)se(m and apply the stochastic data-augmeentation to construct new set with two views of each image, (æi, I, seen). Denote by Z 1 (igg) the set of representations corresponding to these augmented images. Then the noise samples with respect to the anchor embedding ž;,1 E 2 are given by 2 (z,1, 212)) In this work, we minimize the normalized temperature-seat scaled cross entropy loss (Chen et al., 2020a) for instance-wise discrimination
(,p(sim(2.1,2,,22/
(ina(Z (2,1) = log Σ ExEZ((a.1) exp(sim(z,,x)) 2)/T)*
(1)
where sim(a, b) a T
denotes the cosine similarity and 0 is temperature parameter.
In typical semi-supervised contrastive learning setups, the encoder fe learned in a fully unsupervised pre-training phase. The goal of this pre-training is to learn a representation invariant to common data augmentations (cf. Hadsell et al. (2006); Misra & van der Maaten (2020)) such as random crop/flip. resizing, color distortions, and Gaussian blur. After pre-training on unlabeled data, labeled training instances are leveraged to fine-tune fo, e-g., using the canonical cross-entropy loss.
METHODOLOGY
Our goal is investigate strategy for improving the computational efficiency of contrastive learning of visual representations by leveraging the available supervised information during pre-training Here we explore a contrastive approach for utilizing available labels, but we also include additional numerical evaluations with a cross-entropy loss and parametric classifier in Section 4.
Contrastive approach. Consider set of labeled samples operated upon by the stochastic data- augmentation module. The associated set of parameterized embeddings are given by Zs(0) = (fa())) Let E denote an anchor image view with representation z = fe(@) and class label y. By slight overload of notation, denote by (0) the set of embeddings for images in S with class label y (same class as the anchor z). We define the Supervised Noise Contrastive Estimation (SuNCEt) loss as
[(z) log Σx,e2, exp(sim(z,z,))
E2 exp(sim(2, (x, 2x)/T)²
which is then averaged over all anchors
1 [S] Exezs(0) (()).
2

---

Under review as a conference paper at ICLR 2021
In each iteration of training we sample a few unlabeled images compute the self-supervised fettanee-discriminttee loss equation 1, and sample a few labeled images to construct the set S and compute the SuNCEt loss equation 2. We sum these two losses together and backpropagate through the encoder network. By convention, when "sampling unlabeled images," " we actually sample images from the entire training set (labeled and unlabeled). This simple procedure bears some similarity to unsupervised data augmentation (Xie et al., 2019), where supervised cross-entropy loss and parametric consistency loss are calculated each iteration.
Motivation. We motivate the form of the SuNCEr loss by leveraging the relationship between contrastive representation learning and distance- metric learning. Specifically, the SuNCEt loss can be seen as a form of neighborhood component analysis (Goldberger et al., 2005) with alternative similarity metric. Consider a classifier that predicts an image's class based on the similarity of the image's embedding to those of other labeled images 2j using temperature-scalee cosine similarity metric d(z,, x2) x" ls/([|= [Ir). Specifically, let the classifier randomly choose one point as its neighbour, with distribution as described below, and adopt the neighbour's class. Given the query embedding 2, denote the probability that the classifier selects point 2j E Zs) (z) as its neighhour by
(2)
exp(d(z,z,)
p(z)lz) = Ez0e M+} exp(d(z,x;)
Under mutual exclusivity (since the classifier only chooses one neighbour) and a uniform prior, the probability that the classifier predicts the class label ğ equal to some class given a query image æ with embedding z,
ex
p00 c2z
Σ p(zz(z)= = (2,) exp(((,,.)) (0)
(3)
where (0) C (f) is the set of embeddings of labeled images from class c. Minimizing the KL divergence between p( =) and the true class distribution (one-hot vector on the true class y), one arrives at the SuNCEt loss in equation 2. Assuming independence between labeled samples, the aggregate loss with respect to all labeled samples decomposes into the simple sum ExEZs(0) {(z). Numerical experiments in Appendix show that using SuNCEt during pre-training optimizes this aforementioned ao-parametric stochastic nearest-neight neighbours classifier and significantly out- performs inference with the more common K-Nearest Neighbours strategy.
Practical considerations. Rather than directly using the outputs of encoder fe to contrast samples, we feed the representations into a small multi-layer perceptron (MLP), hewl to project the repre- sentations into a lower dimensional subspace before evaluating the contrastive loss, following Chen et al. (2020a). That is, instead of using z ()) directly in equation 1 and equation 2, we use hemng (z) heg... (.)). The projection network hep is only used for optimizing the contrastive loss, and is discarded at the fine-tuning phase. In general, adding SuNCEt to pre-training script only takes a few lines of code. See Listing 2 in Appendix A for the pseudo-code used to compute SuNCEt loss on mini-bated batch of labeled images.
EXPERIMENTS
In this section, investigate the computational effects of SuNCEt when combined with the Sim- CLR self-supervised instance-wise pretext task defined in Section 2. We report results on the ImageNet (Russakovsky et al., 2015) and CIFAR10 (Krizhevsky & Hinton, 2009) benchmarks for comparison with related work. We also examine the combination of SuNCEt with the contrastive SwAV method of Caron et al. (2020) in Section and achieve state-of-ther -httofftth-atttt top-5ttoo--...o accuracy on ImageNet with 10% labels. while cutting the pre-training epochs in half. All methods are trained using the LARS optimizer (You al., 2017) along with cosine- -anne-annn learning-rate sched- ule (Loshchilov & Hutter, 2016). The standard procedure when evaluating semi-supervised learning methods on these data sets to assume that some percentage of the data is labeled, and treat the rest of the data as unlabeled. On ImageNet we directly use the same 1% and 10% data splits used
'The SuNCEt loss can certainly be combined with other instance-wise pretext tasks as well.
3

---

Under review as a conference paper at ICLR 2021
Pataflops 30331
10110
20220
40441
50551
60662
88.0
400
87
300sepochs
87
87.0
87.5
1060 0900-0
B7.1
150 co0cTo
SLR
450 <pochs
86
SimCUR 300-epochs
86.5
SUNCE:
SimGLi 200-spochs
85
100 opocna
500
1000
50000
100000
150000 Updates per GPU
200000
250000
300000
Pre-training epochs
Figure I: Top-5 validation accuracy of a ResNet50 pre-trained on ImageNet with access to 10% of the labels. Orange markers depict SimCIR self-supervised pre-training followed by fine-tuning. Blue markers depict the combination of SimCLR + SuNCEt. Using SuNCEt to leverage available labels during pre-training (not only fine-tuning). (i) accelerates convergence and produces better models (left sub-figure); and (if) can match the semi-supervised learning accuracy of SimCLR whith much less pre-training (right sub-figure). Orange shading in the right sub- sub-figure depicts compute saved. We train all methods using 64 V100 GPUs. One SimCLR epoch corresponds to 312 updates per GPU.
Table 1: Validation accuracy of ResNet50 pre-trained on ImageNet with 1% labels (right table). SimCLR implementations are pre-trained for a certain number epochs and then fine-tuned on the available labels. Using SuNCEt to leverage available labels during pre-training (not only line-tuning) accelerates training, and produces better models with much less compute.
10% of the labels (left table)
10% Labeled
Accuracy (%)
I Labeled
Accuracy (%)
Method
Epochs
Top
Top 5
Method
Epochs
Top 1
Top 5
Chen el al. (2020a) implementation
Chen al. (2020a) implementation
SimCLR
1000
65.6
87.8
SimCLR
1000
48.3
75.5
Our re- re-implemenntation
Our re- ie-ipllemennation
SimCLR
1000
66.1
87.8
SimCLR
1000
50.8
77.7
SimCLR
500
65.4
87.3
SimCLR
500
49.4
76.9
SuNNEE (ours)
500
66.7
88.2
SuNCEt (ours)
500
49.8
77.5
by Chen et al. (2020a). On CIFARIO. 10, we create the labeled data sets by independently selecting each point to be in the set labeled training points with some probability pt we run experiments for each p in (0.01, 0.05, 0.1, 0.2, 0.5, 1.0].
Architecture & data. The encoder network in our experiments is ResNet-50. On CIFAR1O modify the trunk the encoder following Chen et al. (2020a). While this network may not be optimal for CIFARIE 10 images, it enables fair comparison with previous work. For the projection network hop.s we use an MI.P with a single hidden-layer; the hidden layer has 2048 units and the output of the projection network is 1 128-dimensional real vector. The stochastic data augmentation module employs random cropping, random horizontal flips, and color jitter. On ImageNet, we also make use of Gaussian blur.
Fine-tuning. Upon completion of pre-training, all methods are fine-tuned on the available set of labeled data using SGD with Nesterov momentum (Sutskever et al., 2013). We adopt the same fine- tuning procedure as Chen et al. (2020a). Notably, when fine-tuning, do not employ weight-decay and only make use of basic data augmentations (random cropping and random horizontal flipping). Additional details on the fine-tuning procedure are provided in Appendix B.
4.1 IMAGENET
Experimental setup. Our default setup on ImageNet makes use of distributed training; we train each run 64 V100 GPUs and 640 CPU cores. We aggregate gradients using the standard all-reduce primitive and contrast representations across workers using an efficient all-gather primitive. We also synchronize batch-norm statisties across all workers each iteration to prevent the models
4

---

Under review as a conference paper at ICLR 2021
from leaking local information to improve the loss without improving representations (cf. Chen et al. (2020a)). We linearly warm-up the learning-rate from 0.6 to 4.8 during the first 10 epochs of training and use cosine-annealing schedule thereafter. We use momentum value of 0.9, weight decay 10-6, and temperature 0.1. These sypeer-paramtter are tuned for SimCLR (Chen et al., 2020a), but we also apply them the SimCLR SuNCEt combination.
We use a batch-size of 4096 (8192 contrastive samples) for SimCLR; each worker processes 128 contrastive samples per iteration. When implementing the SimCLR + SuNCEt combi- nation, we aim to keep the cost per-iteration roughly the same as the baseline, so we use a smaller unsupervised batch-size. Specifically, each worker processes 88 unlabeled samples per iteration, and 40 labeled samples (sub-sampling 20 classes in each iteration and sampling 2 images from each of the sub- sampled classes). With 10% of the images labeled, we turn off the SuNCEt loss after epoch 250; with 1% of the images labeled, we turn off the SuNCEt loss after epoch 30. We explore the effect of the switch-off epoch and the supervised batch-size (the fraction of labeled data in the sampled mini-batch) in Ap- pendix D, and find the ImageNet results to be relatively robust to these parameters.
66
65.6
25.1
65
05.5
64
63.8
63-
62
SimCin SUNCE: SimCLL
500
1000
Pre-training epochs
Figure 2: Top-1 accuracy of
a a ImageNet with access to 10%
Pre-trrinneetee on
of the labels. Orange markers depict
SimCLR self-supervised pre-training followed by fine- tuning Blue markers depict the combination of SimCLR + SuNCEt, Using SuNCE to Levalage_available labels during pre pre-training (not f accelerates convergence and produces better models.
SuNCEt. Figure shows the top-5 accuracy as a function of the amount pre-training when 10% of the data is labeled. Orange markers denote SimCLR self-supervised pre-training followed by fine-tuning. Blue markers denote the SimCLR + SuNCEt combination followed by fine-tuning. Using SuNCEt to leverage available labels during pre-training accelerates convergence and produces better models with much less compute. The orange shaded region in the right sub-figure explicitly shows the amount of compute saved by using SuNCEr during pre-training. To put these results in the context of our 64 GPU setup, one epoch of SimCLR corresponds to 312 updates per GPU. SimCLR+SuNCEt matches the best SimCLR top-5 accuracy while using only 44% of the compute, and matches the best SimCLR top-1 accuracy while using only 45% of the compute. It may be possible to push these savings further by optimizing the hyper-parameten, for SuNCEt.
Similarly, Figure shows the top: www.llaccuryyycco as a function of the amount pre pre-training wwenn 10% the data is fine-tuning, Blue markers denote the SimCLR SuNCEt combination followed by fine-tuning, It is interesting to note that the improvements in accuracy are significant under the same training epochs; e.g. in the 10% label setting, with 100 epochs of training we observe a +1.6% improvement (from -
Orakge.markers
markers denote SimCLR self-surreeee prettrannngg.., wtwiiiinngolooweeeeeeeooooo followed by.
61.8% to 63.4%) in top-I ImageNet accuracy (run-to-rum variation is on the order of 0.1 1-0.0% similarly with 500 epochs of training we observe a 1..3% improvement (from 65.5% to 66.7% Iop=1 ImageNet accuracy.
With 1% labeled data find that SuNCEt matches the best SimCLR 500-epoch top-5 accuracy while using only 81% of the compute, and matches the best SimCLR top-1 accuracy while using only 83% of compute.? While these savings significant when considering the overall cost of performing 500-epochs of pre-training on 64 V100 GPUs, we note that the improvements are slightly more modest compared to the 10% labeled data setting. This observation supports the hypothesis that improvements in convergence can be related to the availability of labeled data during pre-training. Table 1 shows the top-1 1 and top-5 model accuracies with 10% labeled data (left sub-table) and with 1% labeled data (right sub-table).
Cross-entropy. Next we experiment with leveraging labeled samples during pre-training using cross-entropy loss and a parametric linear classifier (as opposed the non-parametric SuNCEt loss). Similarly the SuNCEt experiments, we use the same hyper-parameters as in Chen al. (2020a) for pre-training. Figure 3 reports savings with respect to our SimCLR 1000-epoch baseline; the
Note that with 1% labeled data, our 500-epoch re-implementation of SimCLR outperforms the original 1000-epoch results of Chen et al. (2020a).
5

---

Under review as a conference paper at ICLR 2021
Top (66.1%)
Top 5 (87.8%)
SimCLR Cross.Ent. SimCLR + SUNCEL
SimCLR Cross.Ent.
SimCLR SUNCEL
30%
40%
60%
70%
Percentage of SimCLR- 10000 compute
Percentage of SimCLR-1000 compute
Figure 3: Percentage compute used by various methods to match the best SimCLR (ResNet50, ImageNet) top-1 validation accuracy (top plot) and top-5 validation accuracy (bottom plot) with 1000 epochs of pre-training. followed by fine-tuning with 10% of labels. Both SimCLR SuNCEt and SimCL. cross-entropy use the default SimCLR syper-parameters. The CinCI-R-cmss-entropy approach matches the best SimCI.R RR validation accuracy while using only 63% of the compute. These savings are lower than those provided by SuNCEr (which only requires 4444% of pre-training to match the best SimCLR accuracy), but are significant nonetheless.
cross-entropy approach matches the best SimCLR 1000-epoch top-1 and top-5 validation accuracy while using only 63% of the compute. These savings are lower than those provided by SuNCEt (which only requires 44% of pre-training to match the best SimCLR top-5 aceuracy and 45% of pre-training to match the best SimCLR top-1 accuracy), but are significant nonetheless.
With 1% labeled data, despite low training loss, SimCLR + cross-entropy does not obtain significantly greater than random validation accuracy with the SimCLR hyper-parameten (even if only leave the cross-entropy term on for 30 epochs to avoid overfitting). With only 12 samples per class in the 1% data setting, is quite easy overfit with cross-entropy loss, suggesting that more fine-grained tuning may be required. In contrast, recall that we observe 19% compute savings out of the box with SimCLR SuNCEt in this scenario with the default SimCLR syper-parameters.
Transfer. Our previous results show that leveraging labeled data during pre-training can result in computational savings. Next we investigate the effect of this procedure on downstream transfer tasks. We evaluate transfer learning performance of the 500-epoch pre-trate pre-trained ImageNet models on Pascal VOCO7 (Everingham et al., 2010) (1 (11-mAP), and CIFAR 10 and CIFAR (Krizhevsky & Hinton, 2009) (top-I), using the fine-tuning procedure described in Chen et al. (2020a) (cf. Appendix B for details). Transfer results are reported in Table 2. Using the SuNCEt loss to leverage available labels during pre-training always improves transfer over pure self-supervised pre-training for the same number of epochs. Moreover, on Pascal VOC07, the SimCLR + SuNCEt combination with only 500 epochs of pre-training significantly outperforms 1000 epochs of SimCLR pre-training.
Table 2:
Evaluating transfer learning performance a ResNet50 pre pre-trained on ImageNet. Using SuNCEt to leverage available labels during pre-training (not just fine-tuning) always improves transfer relative to self superviedd pre-training with same number of pre- pre-training epochs.
CIFARIO
CIFAR100
Pascal VOCO7
Method
Epochs
(Top-1)
(Top-1)
(11-mAP)
SimCLR SimCLR
1000
97.7
85.9
84.1
500
97.3
84.6
83.9
+ SuNCEt (ours)
500
97.6
85.5
85.1
10% labeled
4.2 CIFAR10
Experimental setup. Our training setup on CIFARIO uses a single V100 GPU and 10 CPU cores. We use a learning rate of 1.0, momentum 0.9, weight decay 10 and temperature 0.5. These hyper-parameters are tuned for SimCLR (Chen et al., 2020a), and we also apply them the SimCLR + SuNCEt combination. All methods are trained for default of 500 epochs. Results reported at intermediate epochs correspond to checkpoints from the 500 epochs training runs, and are intended to illustrate training dynamics. We use batch-size of 256 (512 contrastive samples) for SimCLR. When implementing the SimCLR + SuNCEt combination, we set out to keep the cost per-iteration roughly the same as the baseline, samples) and use supervised batch-size of 280 (sampling 28 images from each of the 10 classes in the labeled data set). Note that we only sample images from the available set of labeled data to
use smaller unsupervised batch-size of 128 (256 contrastive

---

Under review as a conference paper at ICLR 2021
90-
05
075
50
100
130
200
250
350
25.0
400
400
5.00
1000
202.0
300.0
400.0
500.5
600.0
700.0
800.0
100
Training Epochs
Potallops
(a) SimCLR test-set convergence fine-tuning on various percentages labeled data.
90
65
85
65
65
50
100
130
200
2500
350
25.0
400
40.0
5.00
100.0
200.
200.0
400.0
390.0
600.0
700.0
800.0
900.0
Training Epochs
Petaflpps
(b) SimCI. SuNCEr test-set convergence with fine-tuning on various percentages of labeled data.
Potaflago Stvod
impovemenn
+.0%
0.0%
55%
1006
22.0%
50
100
150
200
2010
300
330
400
430
500
-15066
15000
20:000
43000
20300
75000
Training Epochs
Moda Updatees
(c) SuNCEt improvement test-set convergence with (d) Computation saved by SuNCEr in reaching hest SimCLR test accuracy with fine-tuning on various percentages of labeled data.
fine-tuning on various percentages of labeled data.
Figure 4: Training ResNet50 with adjusted stem on CIFARIO given various percentages of labeled data. Evaluations reported at intermediate epochs correspond to checkpoints from the same run, with a 500 epoch learning-ral ratting-rate cosine-decay schedule. SuNCEt epochs are counted with respect to number of passes through the unsupervised data loader. Both the sample efficiency and computational efficiency of SimCL. improve with the availability of labeled data, if labeled data only used for fine-tuning. Using SuNCEt leverage available labels during pre-training further improves the sample efficiency (sub-figure (c)) and computational efficiency (sub-figure (d)) of SimCLR.
compute SuNCEt loss each iteration. We turn off SuNCEt loss after the first 100 epochs and revert back to completely self- sepervude learning for the remaining 400 epochs of pre-training to avoid overfitting to the small fraction available labeled data; we explore this point in Appendix E,
Results. Figure 4a shows the convergence of SimCLR with various amounts labeled data, both in terms of epochs (left sub-figure) and in terms of computation (right sub-figure). Both the sample efficiency (left sub-figure) and computational efficiency (right sub-figure) of SimCLR improve with the availability labeled data, even if labeled data is only used for fine-tuning. Figure 4b shows the convergence of the SimCLR + SuNCEt combination with various amounts of labeled data, both in terms of epochs (left sub-figure) and in terms of computation (right sub-figure). Epochs are counted with respect to the number of passes through the unsupervised data-loader. We observe similar trend in the SimCLR + SuNCEt combination, where both the sample efficiency and computational efficiency improve with the availability of labeled data. Figure 4c shows the improvement in Top-1
Thhe only exception to this rule is the set of experiments where 100% of the training data is labeled, in which case keep SuNCEr for the entire 500 epochs. We only observed overfitting on CIFAR10, ImageNet.
7

---

Under review as a conference paper at ICLR 2021
test accuracy throughout training (relative to SimCLR) when using SuNCEt during pre- training Not only does SuNCEt accelerate training from a sample efficiency point of view, but it also leads to better models at the end of training. Figure 4d teases apart the computational advantages by showing the amount of computation saved by the SimCLR + SuNCEt combination in reaching the best SimCLR accuracy. SuNCEt saves computation for any given amount of supervised samples. With only 1% of the training data labeled, SuNCEt can reach the best SimCLR test accuracy while conserving roughly 50 petaflops of computation and over 10000 model updates." 4 In the best case, SuNCEt, with the same exact hlper-paramees of SimCLR pre- pre-training to match the best SimCLR test accuracy. It may be possible push these savings further by optimizing hyper-parameter for SuNCEt.
as the self supervised baseline, only requires 22%
5 RELATED WORK
Table 3: Validation accuracy of ResNet50 pre-trained on ImageNet with access, to 10% of labels, Contrastive methods like SimCLR (Chen et al. 2020a) and SwAV (Caron al, 2020) can leverage SuNCEr during pre-training to surpass. their baseline semi superviie accuracy in half the number of pre-training epochs SuNCEt+SWAN is also competitive other semi supervisd approachhs and outperforms FixMatch-RandAugee in terms of top-5 accuracy.
Method
Epochs
Top-1
Top-5
Supervised (Zhai et al., 2019)
200
56.4
80.4
NPID+ (Wu.et atw.ll.22 201220000 8b; Misra & yan der (0atenn20200)
800
81,5.
PIRL (Misra & van der Maaten, 2020)
800
83.8
UDA+ + Rand
(Xie et 2019)
68,8
88.5.
FixMatch Rand (Sohn et al. 2020)
300
71.5
80.1
(y2 et al. 2020b)
1200_
68.4
89.2.
SimCLR (Chen et al. 2020a).
1000
65.6
87,8
SimCLR (ours)
500
66.7.
88.2.
SwAV (Caron et all 2020).
800
70,2.
89.9
SwAV+SuNCEt (ours)
400
70.8.
89.9
Self-supervised learning, There are a number of other self-supervised learning approaches in the literature, besides the discrmmiiiiiinniionnnn pretext task in SimCLR (Chen et al., 2020a;b). Some non-contrastive approaches learn feature representations by relative patch prediction (Doersch et al., 2015), by solving jigsaws (Noroozi & Favaro, 2016), by applying and predicting image rotations (Gidaris et al., 2018 8), by inpainting or colorization (Denton et al., 2016; Pathak et al., 2016; Zhang et al., 2016; 2017), by parametric vistance-iiscrmminatio (Dosovitskiy et al., 2014), and sometimes by combinations thereof (Doersch & Zisserman, 2017: Kolesnikov et al., 2019). Of the contrastive approaches, Contrastive Predictive Coding (CPC) (Oord et al., 2018; Hénaff et al., 2019) compares representations from neighbouring patches of the same image produce representations with a local regularity that are discriminative of particular samples. Non-Parametric Instance Diserimination (NPID) (Wu al., 2018h 20) aims to learn representations that enable each input image to be uniquely distinguished from the others, and makes use of a memory bank to train with many contrastive samples. The NPID training objective offers a non-parametric adaptation of Exemplar CNN (Dosovitskiy et al., 2014). Misra & van der Maaten (2020) generalizes the NPID method as Pretext-Invarian Representation Learning (PIRL) to contrast images both with and without data augmentations, and combine the method with other instance-wise pretext tasks. He al. (2019) proposes Momentum Contrast (MoCo) to build even larger memory banks by using an additional slowly progressing key encoder, thus benefiting from more contrastive samples while avoiding computational issues with larce-batch training. Grill et al. (2020) also contrasts representations with those of slowly progressing target encoder, but eliminates negative samples all together. There is also recent work (Li et al., 2020), which makes use of EM (McLachlan, 2004) and clustering algorithms for estimating cluster prototypes (Snell et al., 2017), and the recent SwAV method of Caron et al. (2020), which contrasts image representations with random cluster
'ne model update refers the process of completing a forward-hackwand pass, computing the loss, and performing optimization step.
8

---

Under review as a conference paper at ICLR 2021
prototypes. We, report results for SWAV+ in Table 3, trained with the same exact batch-size and rate as for SuNCEt+ SimCLR, The results are consistent with the SimCLR experiments in Section 4; we can match the baseline semi supervisd contrastive accuracy with less than half the wow coneer pre-training epochs...
Semi-supervised learning. Self-supervised learning methods are typically extended to the semi- supervised setting by fine-tuning the model on the available labeled data after completion of self- supervised pre-training. S4L (Zhai et al., 2019) is a recent exception to this general procedure, using cross-entropy loss during self-supervised pre-training. While Zhai et al. (2019) does not study contrastive approaches, nor the computational efficiency of S4L, it shows that S4L can be combined in a stage-wise approach with other semi-supervised methods such Virtual Adversarial Training (Miyato et al., 2018), Entropy Regularization (Grandvalet & Bengio, 2006), and Pseudo- Label (Lee, 2013) to improve the final accuracy of their model (see follow-up work (Tian al., 2020; Hendrycks et al., 2019)). Chen et al. (2020a) reports that the SimCLR approach with self- supervised pre-training and supervised fine-tuning outperforms the strong baseline combination of S4L with other semi-supervised tasks. Other semi-supervised learning methods not based on self-supervised learning include Unsupervised Data Augmentation (UDA) (Xie et al., 2019) and the MixMatch trilogy of work (Berthelot et al., 2019a;b; Sohn et al., 2020). FixMatch (Sohn et al., 2020) makes predictions on weakly augmented images and (when predictions are confident enough) uses those predictions as labels for strongly augmented views of those same images. An additional key feature of FixMatch is the use of learned data augmentations (Berthelot et al., 2019a; Cubuk et al., 2019). Of the con-contrastive methods, FixMatch sets the current state-of- state-of-the-arr on established semi-supervised learning benchmarks. Note that SwAV+SuNCEL is competitive with the other sepe-super approaches and outperforms FixMMath-RRnnAuugmen in terms of top-5 accuracy (cf. Table 3):
Supervised contrastive loss functions. Supervised contrastive losses have a rich history in the mistance-metric learning literature. Classically, these methods utilized triplet losses (Chechik et al., 2010; Hoffer & Ailon, 2015; Schroff et al., 2015) or max-margin losses (Weinberger & Saul, 2009; Taigman et al., 2014), and required computationally expensive serd-negative mining (Shrivastava et al., 2016) or peerssariilly-eneratte negatives (Duan et al., 2018) in order to obtain informa- tive contrastive samples that reveal information about the structure of the data. One of the first works to overcome expensive hard-negative mining is that of Sohn (2016), which suggests using several negative samples per anchor. Most similar to the SuNCEt Joss tha of Wu atall.C......... (2018a) which investigates neighborhood component analysis (NCA) (Goldberger et al., 2005) in the fully, supervised setting. However, their method approximates the NCA loss ly_ storing an embedding tensor for every single image in the dataset, adding non- coiiirivial memory overhead. The SuNCEt loss instead relics on noise contrastive estimation and does not have this limitation. Another more recent supervised contrastive loss is that proposed in Khosla et al. (2020) for the fully supervised setting; while their proposed method is more computationally draining than training with a standard cross- entropy loss, it is shown to improve model robustness. As mentioned, the loss can be seen as form of neighorteed component analysis (Geldberger ef al., 2005) with an alternative siterrative--iiiilrttty metrie.
The SuNCEt loss. is different from the Joss of Khosla et al. (2020. v.)., Howeyer, after the initial preprint of our work appeared on OpenReview, Khosla et al. (2020, v2, Section 15. 15-Change Log) additional contrastive loss of similar format s SuNCEt We provide brief comparison of the loss in Khosla et al. (2020, vl) and SuNCHt using the full set of labeled data in Appendix H. In short, when using the losses in conjunction with SimCLR and a small supervised batch-size, both methods perform similarly, However, when used independently with larger batches and more positive samples per anchor, their performance differs.
was updated with
CONCLUSION
This work demonstrates that small amount of supervised information leveraged during contrastive pre- ure-training (not just fine-tuning) can accelerate convergence. We posit that new methods and theory rethinking the role of supervision to not only improve model accuracy, but also learning efficiency is an exciting direction towards addressing the computational limitations of existing methods while utilizing limited semantic annotations.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Sanjeev Arora, Hrishikesh Khandeparkar, Mikhail Khodak, Orestis Plevrakis, and Nikunj Saun- shi. A theoretical analysis of contrastive unsupervised representation learning. arXiv preprint arXiv: 2019.
Philip Bachman, Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. In Advances in Neural Information Processing Systems, pp. 15509-15519, 2019.
David Berthelot, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. Remixmatch: Semi-supervised learning with distribution alignment and augmentation anchoring. arXiv preprint 17Xiv:1911.09783 2019a.
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. Mixmatch: A holistic approach to semi-supervised learning. Advances in Neural Information Processing Systems, pp. 5050-5060, 2019b.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. arXiv preprint arXiv:2006.0988) 09882, 2020.
Gal Chechik, Varun Sharma, Uri Shalit, and Samy Bengio. Large scale online learning of image similarity through ranking. Journal of Machine Learning Research, 11 (1Ma)):1109-111: 2010.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. preprint arXiv:2002.0570% 2020a.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self-supervised models are strong semi-supervised learners. arXiv preprint arXiv:2006. 10029, 2020b.
Sumit Chopra. Raia Hadsell, and Yann LeCun. Learning a similarity metric discriminatively, with application to face verification. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'OS) 005), volume pp. 539-546. IEEE, 2005.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoe V Le. Randaugment: Practical data augmentation with no separate search. arXiv preprint arXiv: /909. /37/9, 2019.
Emily Denton, Sam Gross, and Rob Fergus. Semi-supervised learning with context-conditional
generative adversarial networks. arXiv preprint drXiv:1611.064300 2016.
Carl Doersch and Andrew Zisserman. Multi-task self-supervised visual learning. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2051-2060, 2017.
Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1422-1430, 2015.
Alexey Dosovitskiy, Jost Tobias Springenberg, Martin Riedmiller, and Thomas Brox. Discrimina- tive unsupervised feature learning with convolutional neural networks. In Advances in neural information processing systems, pp. 766-774, 2014.
Yueqi Duan, Wenzhao Zheng, Xudong Lin, Jiwen Lu, and Jie Zhou. Deep adversarial metric learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2780-2789, 2018
Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journaall of computer vision, 88(2): 303-338, 2010.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. arXiv preprint arXiv: 1803.07728, 2018.
10

---

Under review as a conference paper at ICLR 2021
Jacob Goldberger, Geoffrey E Hinton, Sam T Roweis, and Russ R Salakhutdinov. Neighbourhood components analysis. In Advances in neural information processing systems, pp. 513-520, 2005.
Yves Grandvalet and Yoshua Bengio. Entropy regularization. Semi-supervised learning. pp. 151-168, 2006.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent: A new approach to self-super learning. arXiv preprint arXiv:2006.07733 07733, 2020.
Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pp. 297-304, 2010.
Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning invariant mapping. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR (V6), volume 2, pp. 1735-1742. IEEE, 2006.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. arXiv preprint arXiv: 191.05121 2019.
Olivier Hénaff, Aravind Srinivas, Jeffrey De Fauw, Ali Razavi, Carl Doersch, SM Eslami, and Aaron van den Oord. Data-efficient image recognition with contrastive predictive coding. arXiv preprint arXiv:1905.094 0wwp/999..0992,,... 2019.
Peter Henderson, Jieru Hu, Joshua Romoff, Emma Brunskill, Dan Jurafsky, and Joelle Pineau. Towards the systematic reporting of the energy and carbon footprints of machine learning. arXiv preprint arXiv:2002.00565) 2020.
Dan Hendrycks, Mantas Mazeika, Saurav Kadavath, and Dawn Song, Using self-supervised learning can improve model robustness and uncertainty. In Advances in Neural Information Processing Systems, pp. 15637-15648, 2019.
Elad Hoffer and Nir Ailon. Deep metric learning using triplet network. International Workshop on Cemilarity-Basele Pattern Recognition, pp. 84-92. Springer, 2015.
Prannay Khosla, Piotr Teterwak, Chen Wang. Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. arXiv preprint arXiv:2004. 113000 2020.
Alexander Kolesnikov, Xiaohua Zhai, and Lucas Beyer. Revisiting self-supervised visual representa- tion learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 1920-1929, 2019.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
Dong-Hyun Lee. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges representation learning, ICML, volume pp. 2013.
Junnan Li, Pan Zhou, Caiming Xiong, Richard Socher, and Steven CH Hoi. Prototypical contrastive learning of unsupervised representations. arXiv preprint XXiv:2005.04966. 2020.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv
preprint arXiv:1608. 1608.0393, 2016.
Geoffrey J McLachlan. Discriminant analysis and statistical pattern recognition, volume 544, John Wiley & Sons, 2004.
Ishan Misra and Laurens van der Maaten. Self-supervised learning of reppesatiiatrrrreeeettttttnnn In CVPR, 2020.
11

---

Under review as a conference paper at ICLR 2021
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii Virtual adversarial training: regularization method for supervised and semi-supervised learning. IEEE transactions on pattern analysis and machine intelligence, 1198):1979-1993, 2018.
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In European Conference on Computer Vision, pp. 69-84. Springer, 2016.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint /807.0340 2018.
Deepak Pathak, Philipp Krahenbuhi, Jeff Donahue, Trevor Darrell, and Alexei Efros. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2536-2544, 2016.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, (15(3):211225 2015.
Florian Schroff. Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEFE conference on computer vision and pattern recognition, pp. $15-823. 2015.
Abhinav Shrivastava, Abhinav Gupta, and Ross Girshick. Training region-based object detectors with online hard example mining. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 761-769, 2016.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in neural information processing systems, pp. 4077-4087, 2017.
Kihyuk Sohn. Improved deep metric learning with multi-class n-pair loss objective. Advances in neural information processing systems, pp. 1857-1865, 2016.
Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin D) Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel. Fixmatch: Simplifying cunissuperviee learning with consistency and confidence. arXiv preprint arXiv:2001.07683. 2020.
Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initialization and momentum in deep learning. In International conference on machine learning, pp. 1139-1147, 2013.
Yaniv Taigman, Ming Yang, Mare' Aurelio Ranzato, and Lior Wolf. Deepface: Closing the gap to human- performance in face verification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1701-1708. 2014.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. arXiv preprint arXiv: 2019.
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning. arXiv preprint arXiv:2005. 2020.
Kilian Q Weinberger and Lawrence K Saul. Distance metric learning for large margin nearest neighbor classification. Journal of Machine Learning Research, 10(Feb):207-244, 2009.
Zhirong Wu, Alexei A Efros, and Stella Yu. Improving generalization via scalable neighborhood component analysis. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 685-701, 2018a.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3733-3742, 2018b.
Qizhe Xie, Zihang Dai, Eduard Hovy. Shah-Thang Luong, and Quoc Le. Unsupervised data augmentation. arXiv preprint arXiv: /904. /2848, 2019.
12

---

Under review as a conference paper at ICLR 2021
Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. arXiv preprint 1708.
2017.
Xiaohua Zhai, Avital Oliver, Alexander Kolesnikov, and Lucas Beyer. S41: Self-supervised semi- supervised learning. In Proceedings of the IEEE international conference on computer vision, pp. 1476-1485, 2019.
Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In European conference on computer vision, pp. 649-666. Springer, 2016.
Richard Zhang, Phillip Isola, and Alexci A Efros. Split-brain autoencoders: Unsupervised learning by cross-channel prediction. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 1058-1067, 2017.
13

---

Under review as a conference paper at ICLR 2021
Appendix
A PSEUDO-CODE
Listing Pseudo-code for main training script computing SinCLR+SuNCEL when small fraction of labeled data is available during pre-training.
init image sampler for instance-dissrriinaation unsupervised_data lodde loader
-- init (labeled) image sampler for SuNCEt
supervised_datte _
loader
for epoch in range (aneeeuum_ppoch)
for itr, imgs in enumerate
:
compute insttaceediserrimnnttion loss
z (lp(encoder (annodee(imggs) (imgs))
ss ssl_loss = simclr(z)
compute supervised-contrstiive loss labeled data imgs, labels next (swpervised_data__
z (encodre (ancodeerimgss)) (imgs))
supervised_ loss = suncet(z, labels)
compute aggregate loss and update encoder & mlp loss supervised_ loss.bace backward
ssl_loss
hwwiiiier.ssepp.... Ir tr_scheduler. step()
Listing 2: Pseudo-code for computing SuNCEt on given tensor of image embeddings.
def suncet(z, labels):
normalize embeddings: [n d] d.i .unpuueze)uuu
compute similarity between embeddings: [n x n] exp_cs exrch.e 2.t()) / fitlatttreeed.ll_iile
compute loss for each sampled class and accumulate loss 0.
num classes for in set seabbssss)
batch-size of embeddings with class-label ( bs_cls (labels 1) sum()
num num_classes += 1
pos pos_cls cls torch. sum (exp_ [abells == [][; labels == 1], dim=1)
den_ cls torch. sum (exp_cs [labels 1], dim=1)
loss += torch. sum (torch. htsphl/ggg/ssssss.ddddddde.s))) / bs_cls
loss /= num_ classes
return loss
14

---

Under review as a conference paper at ICLR 2021
B ADDITIONAL DETAILS ABOUT FINE-TUNING
We follow the fine-tuning procedure of Chen et al. (2020a). Upon completion of pre-training, all methods fine-tuned the available labeled data using SGD with Nesterov momentum. We do not employ weight-decay during fine-tuning, and only make use of basic data augmentations (random cropping and random horizontal flipping). The weights of the linear classifier used to fine-tune the encoder network are initialized to zero. On CIFARIO. models are fine-tumed for 90 epochs. All results are reported on the standard CIFARIO test set. We use a batch-size of 256, along with momentum value of 0.9 and an initial learming-rate of 0.05 coupled with a cosine-annealle learning-rate schedule. On ImageNet, the 10% labeled data setting, models are fine-tuned for 30 epochs; in the 1% labeled data setting, models are fine-tuned for 60 epochs. We use batch-size of 4096, along with momentum value of 0.9 and an initial learning-rate of 0.8 coupled with a cosine-annealing learning-rate schedule. All results are reported on the standard ImageNet validation set using a single center-crop.
C ADDITIONAL DETAILS ABOUT TRANSFER
We follow the fine-tuning transfer procedure outlined in Chen et al. (2020a). Specifically, we fine-tune the pre-trained model for 20,000 steps using Nesterov momentum. We use a batch-size of 256 and set the momentum value to 0.9. We perform random resized crops and horizontal flipping. and select the learning rate and weight decay by performing grid search with grid of 7 logarithmically spaced learning rates between 0.0001 and 0.1 and 7 logarithmically spaced values of weight decay between 10-5 and 10 10-3, well
weight decay. We divide the weight decay values by the learning rate.
D EFFECT OF SUPERVISED BATCH-SIZE ON IMAGENET
What fraction of our sampled mini-batches should corre-
Petaflops
66.5
spond to labeled images for computing the SuNCEt loss? We fix the total num- bers of passes through the la- beled data and vary the frac- tion of labeled data sampled per mini-batch. Therefore, runs that sample less labeled data per mini-batch keep the SuNCEt loss on for more updates, whereas runs that sample more labeled data per mini-batch keep the SuNCEt loss for less updates.
66.0
30000
65.5
20000
88.0 Top
30000
87.5
20000
0.25
0.50
0.75
0.25
0.50
0.75
Fraction of labels mini-batch
Fraction of labels in mini-batch
(a) 10% labels
30000 Petaflops
49.75
TOp
49.50
20000
77.3
@ 77.2
30000
We train a ResNet50 on Im- ageNet for 500 epochs 64 V100 GPUs using the Sim- CLR + SuNCEt combina- tion with the default Sim- CLR optimization parame- ters described in Section 4. In one setting, 10% of the im- ages are labeled, and, in the other, 1% the images are labeled. The left sub-plots in Figure show how the best 1 and top-5 validation curacy vary as we change the fraction of labeled data per
20000
0.1
0.2
0.3
0.1
0.2 0.3
Fraction of labels in mini-batch
Fraction of labels in mini-batch
(b) 1% labels
Figure 5: Training a ResNet50 on ImageNet using the SimCLR + SuNCEr combination when given access to 10% of the labels (top sub-plot) and 1% of the labels (bottom sub-plot). We examine the best top-1 and top-5 validation accuracy, we vary fraction labeled samples per mini-batch. we only use a small fraction of labeled data in each mini-batch, then the best model accuracy drops. However, in general, the best final model accuracies, and the corresponding computational requirements, labeled data per mini-bate batch the corresponding swtich-off epoch.
the corresponding computational requirements, as
significantly affected by the fraction of
15

---

Under review as a conference paper at ICLR 2021
mini-batch. The right sub-plots show the compute (petaflops) used to obtain the corresponding models in the left subplots vary as we change the fraction of labeled data per mini-batch. we only small fraction of labeled data in each mini-batch, then the best model accuracy drops. However, in general, the best final model accuracies, and the corresponding computational requirements to obtain said models, not significantly affected by the fraction of labeled data per mini-batch and the corresponding switch-off epoch.
E LIMITATIONS ON CIFAR1O
The amount of time that CIFARIO is positively correlated with the amount labeled data. To shed light on this limitation, we conduct experiments where switch-off the SuNCEt loss at certain epoch, and revert to fully self-supervised learning for the remainder training. All models are trained for a total of 500 epochs; epochs are counted with respect the number of passes through the unsupervised data loader.
leave the SuNCEt loss without degrading performance
The left subplots in Figure 6 report the final model test-accuracy on CIFARIO as function of the switch-off epoch, for various percentages of available labeled data. The right subplots in Figure report the amount of petaflops needed to train the corresponding models in the left subplots.
To study the potential accuracy degradation our focus to the left subplots in Figure 6. When 20% more of the data is labeled (bottom three subplots), the final model accuracy relatively invariant to the switch-off epoch (lines are roughly horizontal). However, when less labeled data is available, the final model accuracy can degrade if we leave the SuNCEt loss on for too log (top three subplots). The magnitude of the degradation is positively correlated with the amount of available labeled data (lines become progressively more horizontal from top subplot to bottom subplot).
function of the switch-off epoch, first restrict
Petaflops
750
Top
90
500
80
250
750
90
500
80
250
750
90
500
80
10%
250
10%
750
90
500
80
20%
250
20%
750
90
500
80
-
50%
250
50%
750
90
500
80
100%
250
100%
100
200 300
400
500
100
200
300
400
500
SuNCEt Switch-Off Epoch
SuNCEt Switch-Off Epoch
Figure 6: Training ResNet50 with adjusted-stem CIFAR10 given various percentages of labeled data. Left subplots report the final model lest-accuracy function of the epoch which the SuNCEt loss switched off. Right subplots report amount petaflops needed train the corresponding models the lefts subplots. The number of epochs that one can utilize the SuNCEt loss for without degrading performance is positively correlated with the amount of available labeled data. To balance improvements in model accuracy with computational costs, it may also be beneficial switch off the SuNCEt loss carly in training, even if leaving does degrade performance.
16

---

Under review as a conference paper at ICLR 2021
SuNCEt
2.5
1%
InfoNCE
6
1%
4
0.0
2.5
4
2.5
10%
6
10%
0.0
2.5
20%
6
20%
4
0.0
50%
4-
2.5
100%
6
100%
0.0
100
200
300
400
500
0 100
200
300
500
Training Epochs
Training Epochs
Figure 7: Training ResNet50 with adjusted-stem CIFARIO given various percentages of labeled data. The SuNCEt loss is only used for the first 100 epochs of training, and then switched off for the reaminder of training. Left subplots report the supervised SuNCEt loss during training. Right subplots report the self-supervised InfoNCE loss during training. In practice, we turn off the SuNCEt loss when it has platcaued, and revert to fully self-supervised training thereafter. In these experiments, the SuNCEt loss has roughly plateaued at around 100 epochs of training.
From a computational perspective, it may also beneficial to turn off the SuNCEt loss at some point, even if leaving it on does not degrade performance. We hypothesize that once we have squeezed out from the labeled data, it is best revert all computational resources to
all the information that optimizing the (more slowly-convergent self-supervised instance-discriinnttio task. We see that leaving the SuNCEt loss for more epochs does not provide any significant improvement in model accuracy (left subplots in Figure 6), but the corresponding computational requirements still increase (right subplots in Figure 6).
Switching-off the SuNCEt loss when it has roughly plateaued provides a good strategy for balancing gains model accuracy with computational costs. We switch off the SuNCEt loss at epoch 100 in all of our CIFAR10 experiments in the main paper (except the experiment with 100% labeled data, where SuNCEt is left on for all 500 epochs of training). Figure 7 depicts the supervised SuNCEt loss during training for various percentages of available labeled data (left subplots), and the self-supervised InfoNCE loss during training for various percentages of available labeled data (right subplots). The SuNCEt loss has roughly plateaued after 100 training epochs (left subplots). Figure also suggests that the rate at which the SuNCEt loss platcaus is negatively correlated with the available amount of labeled data. This observation supports the intuition that should turn off the SuNCEt loss earlier in training if less labeled data is available (cf. Figure 6). In general, the strategy we adopt simply keep the number of passes through the labeled data fixed; meaning that less data will require less updates.
F ADDITIONAL EXPERIMENTS FOR SIMCLR + CROSS-ENTROPY (CIFARIO)
Experimental setup. Our training setup for SimCLR cross-entropy CIFARIO is identical to that used in Section 4 for SimCLR + SuNCEt. Specifically, we use a single V100 GPU and 10 CPU cores. We use learning-rai rate of 1.0. momentum 0.9, weight decay 1 and temperature 0.5. These hyper-parameters are tuned for SimCLR (Chen et al., 2020a), and we also apply them to the SimCLR cross-entropy combination.
17

---

Under review as a conference paper at ICLR 2021
90-
05
175
50
100
130
200
250
350
25.0
400
400
5.00
100.
202.0
300.0
400.0
500.5
600.0
700.0
800.0
100
Training Epochs
Potallops
(a) SimCLR test-set convergence fine-tuning on various percentages labeled data.
90
85
85
65
65
50
200
150
200
250
300
230
400
430
500
100.0
200.0
200.0
400.0
390.0
600.0
700.0
800.0
900.0
Training Spochs
Petaflpp
(b) SimCI.R R. cross-entropy test-set convergence with fine-tuning on various percentages of labeled data.
Potaflops Saved
16.0%
+4.0%
+2.0%
50%
100%
0.
50
100
150
200
250
300
300
400
430
300
-15000
15000
33200
43000
51200
75000
Training Epocht
Sarved
(c) SimCLR cross-entropy improvement test-sct (d) Computation saved hy SimCLR + cross-entropy in convergence (relative to plain SimCLR) with iine-tunigreacchnn the best SimCLR test accuracy with fine-tuning on various percentages of labeled data.
various percentages of labeled data.
Figure 8: Training ResNet50 with adjusted stem CIFARIO given various percentages of labeled data. Evaluations reported at intermediate epochs correspond to checkpoints from the same run, with a 500 epoch learning-rate rate cosine-decay decay schedule. Cross-entropy epochs are counted with respect to number of passes through the unsupervised data loader.
We batch-size of 256 (512 contrastive samples) for SimCLR. When implementing the SimCLR + cross-entropy combination, we set out to keep the cost per-iteration roughly the same as the baseline, so we use smaller unsupervised batch-size of 128 (256 contrastive samples) and a supervised batch-size of 280 (sampling 28 images from each of the 10 classes in the labeled data set). Note that we only sample images from the available set of labeled data compute the cross-entropy loss in each iteration.
We turn off the cross-entropy loss after the first 100 epochs and revert back to completely self- supervised learning for the remaining 400 epochs of pre-training to avoid overfitting to the small fraction of available labeled data.
Results. Figure 8a shows the convergence of SimCLR with various amounts of labeled data, both terms of epochs (left sub-figure) and in terms of computation (right sub-figure). Both the sample efficiency and computational efficiency of SimCLR improve with the availability of labeled data, even if labeled data only used for fine-tuning.
'The only exception to this rule is the set of experiments where 100% of the training data is labeled, in which case keep SuNCEt for the entire 500 epochs.
18

---

Under review as a conference paper at ICLR 2021
Figure 8b shows the convergence of the SimCLR cross-entropy combination with various amounts of labeled data, both in terms of epochs (left sub-figure) and in terms of computation (right sub- aub-figure). Epochs are counted with respect to the number of passes through the unsupervised data-loader. We observe a similar trend in the SimCLR + cross-entropy combination, where both the sample efficiency and computational efficiency improve with the availability of labeled data.
Figure 8c shows the improvement in Top test accuracy throughout training (relative to SimCLR) when using cross-entropy. Similarly SuNCEt, , we see that cross-entropy accelerates training from a sample efficiency point of view, and also leads to better models at the end of training.
Figure 8d shows the amount of computation saved by the SimCLR cross-entropy combination in reaching the best SimCLR accuracy. There are two x-axes in this figure. The top shows the petaflops saved and the bottom shows the number of model updates saved to reach the best SimCLR test accuracy. Similarly to SuNCEt, cross-entropy saves computation for any given amount of supervised samples. These results provide further evidence for our hypothesis, namely, that leveraging labeled data during self-supervised pre-training can accelerate convergence.
G NON-PARAMETRIC INFERENCE
In Section 3 we showed that, from a theoretical perspective. the SuNCEr loss optimizes a non- parametric classifier based on a type of stochastic nearest neighbours. Here we empirically evaluate this connection on ImageNet by classifying validation images using the inference procedure described in Section 3, and comparing to a K-Nearest Neighbours (KNN) classifier with the same similarity metric.
We consider the 10% labeled data setting and use the 400-epoch pre-trained+fine-uun Sim- CLR+SuNCEt models to compute image embeddings. Specifically, classify each point in the validation set by computing the SuNCEt class probabilities in equation 3 with respect to the small set of available labeled training images, and choosing the class with the highest probability. We refer to this non-parameteric inference procedure as SuNCEt -NPI. We employ basic data augmentations (random cropping and random horizontal flipping) to the labeled training images before computing their corresponding embeddings, and apply a single center-crop to the validation images. When performing inference using SuNCEt -NPI, we find best to use the temperature parameter used during training. 0.1 in this case, and, surprisingly, to also use the image embeddings obtained before the MLP projection head. We also find it best to use the image embeddings obtained before the MLP projection head when using the KNN classifier. We experiment with various values of K for the KNN classifier, and find K- = 10 to work best (surprisingly, better than larger values of K).
Table 4 shows the validation accuracy of these non-parametric classifiers. We consider (i) K- Nearest Neighbours (K=10); (ii) SuNCEt -NPI View), where we compute the SuNCEt class probabilities in equation 3 and use one embedding for each available labeled training image; (iii)
Table 4: Non- non-parametric inference. Validation accuracy of a ResNet50 pre-trained on ImageNet for 400-cpochs with access to 10% of labels using SimCLR+SuNCe with the default SimCLR hyper-parameters Inference is conducted non-parametrically by computing the similarity of validation images to the 10% labeled train images. We consider (i) K-Nearest Neighbours (K=10); (ii) SuNCEt -NPI Single-View)) where we compute the SuNCE class probabilities in equation and use one embedding for each labeled training image; (iii) SuNCEI -NPI (Multi- View), where we compute the SuNCEt class probabilitics in equation 3 and use multiple embedding for cach labeled training image. Validation accuracies obtained using SuNCEt -NPI are significantly greater those obtained using K- Neaarest Neighbours, suggesting that using SuNCEt during pre- pre-training optimizes the mon-parametric stochastic nearest classifier described in Section 3. Moreover, using multiple views for inference has no significant effect on classification accuracy (likely due to the invariance induced by self-supervised castanc-diicriminatte
10% Labeled
Accuracy (%)
Classification Method
Top 1
Top 5
KNN (K=10)
52.3
SuNCEt -NPI (Single-Vien
61.7
85.8
SuNCEt-No -NPI (Multi-View
61.6
85.7
19

---

Under review as a conference paper at ICLR 2021
SuNCEt -PII (iee), where we compute the SuNCEt -NPI class probabilities in equation 3 and use multiple embedding for each available labeled training image. The validation accuracies obtained by using SuNCEt -NPI are significantly greater than those obtained using K-Nearest Neighbours; suggesting that using SuNCHr during pre-training optimizes for the non-parametric stochastic nearest classifier descrihed in Section 3.
As a final observation, we find that using multiple views of training images for inference has no significant effect on the classification accuracy; this is likely due to the invariance induced by self- supervised
It should also be noted that the accuracies in Table 4 are obtained by comparing the validation images to only the 10% of labeled images used during pre-training. is almost certainly possible to iccrasse the accuracies for all methods in this table by conducting inference with respect to the entire training set. Moreoever/orroverr increase the SuNCEt -NPI accuracies by fine-tuning the pre-trained models using the SuNCEt loss,
may be possible to further
H CONTRASTIVE LOSSES AANANA
Here we briefly compare our SuNCEt loss loss www.ata.ooaaaae et 2020), which was proposed for the fully supervised setting. The loss Came with respect to an anchor = with class label y given by.
cxp(sim(z,z,) 2,)/7)
Σ
log
(0)
E (x(e2((0)1s exp(sim(z,zx))
(0)
whereas the SuNCEr loss is given by
(0 exp(sim(z, 2z)/T)
- logg
(x exp(sim(22 =x)/T)
When using the losses in conjunction with SimCLR and a small supervised batch-size, both methods perform similarly. Howeyer, when used independently with many positive samples per anchor, their performance differs. We train a ResNet50 for 100 epochs on ImageNet using the full set of labeled data, followed by 15 epochs of fine-tuning the entire network weights with cross-entropy loss. We use the default SimCLR tata-augmentations and
(learning rate 4.8).
Table 5: Validation accuracy of ResNet50 pre-trained on ImageNet with access to 100% of labels, using the table). training with SimCLR, using an unsupervised batch-size of 4,096 samples and a supervised batch-size of 1,280 samples. be
default R. soita-agmmnnattionn and hyper parametrs (seaming rate =.8.)
(Right table); training using only the supervised losses and batch-size of 16k (125 classes, 128 instances per, class). When using, the losses in conjunction with SimCLR and a small supervised batch-size. both methods perform similarly. However, when using the losses independently with many samples per anchor, the SuNCEt losstop
validation accuracy is more than +1000% higher.
Pre-train loss
Top-1
Top-5
Pre-train loss
Top-1
Top-5
rand, init
51.8
7
rand. init Lool
51.8
76_1
SimCLB Coup
75.4.
92.9
64.4
86.5.
SUNCLR+ (ours).
75,4
93.0
LANCEE_(our)
75.6
92,8
In the left table in Table See trained both wt methods jointly with SimCLR, using an unsupervised batch-size of 4,096 samples, and a supervised batch-size of 1,280, Both SuNCEt and Caup perform similarly, In the right sub-table in Table 5.we train the methods independently with a batch size of 16K; 125 classes and 128 instances per class. This is similar to the experimental setup in b
Section 4, where we each mini-batch contains 128 samples per class (2 sampee by each GPU),
We had difficulty getting the Khosla al. (2020) loss loss to converge with this many positive samples. even when we made the learning-rate maaeenng-rateeseee small, so we added a batch normaalizaoio (BN) layer before the final layer of the projection head, and this fixed the issue, We evaluated SuNCEt loss both - Son
with and without the BN layer, and it did not affect performance, so we left it in for the purpose of comparison. The SuNCEt accuracy is more than +00% higher.
20


Reviews:
[{"review_id": "kTY0UYywGsc", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary**:\nThis paper designs a new loss, called SuNCTt, to speed up the convergence of semi-supervised training. Specifically, the loss involves the computation of similarity between anchor and other images with the same class, and the similarity between anchor and other labeled images. It is claimed to be considered as the form of neighborhood component analysis. Together with the standard contrastive learning loss, it only uses less than half the amount of pre-training and computes to match the accuracy of the previous approaches.\n\n**Pros**:\n+ The whole idea makes sense. The comprehensive experiments, also shown in appendix, support claims in the paper that the + introduced loss is helpful for semi-supervised training  \n+ Overall, the paper is well written and the results part is well structured.\n\n\n**Concerns**:\n- It mentions some semi-supervised work in the related work. Is it possible to compare the results with theirs?\n- It would be good to show the results of using cross-entropy pre-trained on ImageNet. In addition to the saving compute, I am also curious about the final performance on ImageNet after training for 1000(500) epochs. \n- On page 5, it mentioned that, with 1% labeled data, it is not significantly greater than the random accuracy. However, in appendix F (figure 7b), the result of 1% data(CIFAR-10) looks good.  I wonder why it is, if I understand the contexts correctly.\n- I also wonder, for the labeled data, what will happen if we use explored metric learning methods, like triplet loss. For the triplet loss, we can leverage labels to define the positive and negative samples. Both triplet loss and SuNCTt may do the similar thing, but in different forms.\n\nOverall, I prefer the rating as above the threshold at the current stage. Hope the authors could address my concerns or questions in the rebuttal period.\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "_7q3-QZelKJ", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper combines the self-supervised contrastive loss with the supervised contrastive loss for semi-supervised learning. By leveraging a small amount of labeled data, this paper shows that the semi-supervised contrastive loss can achieve similar performance as self-supervised contrastive loss (SimCLR) with less than half of the compute.\n\nStrength:\n1. This paper is well-motivated. It is a good direction to explore contrastive learning in the semi-supervised learning setting.\n2. It is interesting to see that using a small amount of labeled data can reduce computation.\n3. The paper is well-written and easy to understand.\n\nWeakness:\n1. The proposed semi-supervised contrastive loss seems to be a straight-forward combination of the self-supervised contrastive loss and the supervised contrastive loss [Khosla et al. 2020]. Therefore, the technical novelty is limited. \n2. This paper does not compare with any of the existing semi-supervised learning methods (e.g. FixMatch), even though they are discussed in the related work. Self-supervised learning has been known to be computation expensive, but this paper also needs to justify that the proposed method is more efficient than existing semi-supervised learning methods.\n3. Figure 2 shows that combining CE with SimCLR already achieves good performance. Therefore, a simple baseline would be combining FixMatch with SimCLR. I would expect it to be comparable or even better than the proposed contrastive learning method, because previous papers (e.g. S4L) have shown that adding a self-supervised objective helps with semi-supervised learning.\n4. The improvement over SimCLR does not seem to be significant under the same training epochs. Given that labeled data is used, this improvement is mostly expected.  Furthermore, the supervised contrastive loss is turned off after some epochs, which means it plays a less important role. If SuNCEt is also trained for 1000 epochs, would it converge to a similar result as SimCLR?\n5. In Table 2, it would be interesting to also see the performance of fully-supervised learning.", "extra_scores": {}, "extra_text": {}}, {"review_id": "UackXCZunNf", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Pros:\n\n1. This submission is well written with lots of experiments. The claim is that supervised contrastive learning can speed up representation learning, which is well supported. \n\nCons:\n\n1. Instead of proposing a new idea, this paper adopts supervised contrastive learning and discovers that supervision can help accelerate the pretraining stage. Despite it maybe practical to speed up the experimental cycle, the technique contribution is rather limited. I may understands this paper wrong, so could authors clarify the propsoed SuNCEt loss with supervised contrastive loss? \n\n2. Since this paper is investigating semi-supervised learning, then some comparisons to existing literature is necessary. For example, comparisons to FixMatch. Right now, all the comparisons are only done on SimCLR, which are basically ablation studies.\n\n3. Performance improvement on CIFAR is marginal. I read the limitation section in Appendix, but not clear how it is related to the final results. \n\n4. I notice that the proposed SuNCEt loss is turned off after some training epochs. Especially when using 1% of the labeled data, this loss is turned off at epoch 30, which is quite early in the learn stage. Does SuNCEt loss hurt the training and why? And in practice, how do you determine the optimal time to turn off SuNCEt loss if you don't have access to all the labeled validation set? \n\nIn conclusion, despite the paper has some interesting results, it lacks of experiments to show its real contributions.", "extra_scores": {}, "extra_text": {}}, {"review_id": "4ua1vt1w_Ok", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new method applicable to a specific case in unsupervised learning: having access to a small amount of labels during the training phase where label signal is not used.\n\nFirst, I would like to say this is a well motivated task. Many recent works on self-supervised learning (image classification using instance discrimination signals) eventually need to use all the labels in the linear evaluation phase to obtain the final performance numbers. The authors also point out a similar view in the last paragraph in Section 3. \n\nIn the related works section, the work by Khosla et al. 2020 is mentioned. I wonder if it makes sense for the authors to add their numbers in the experiment comparison given that Khosla et al also use the original SimCLR as benchmark? In addition, I think a \"soft-nearest neighbor loss\" paper could be cited and compared (for example: Zhirong Wu, Alexei A Efros, and Stella Yu. Improving generalization via scalable neighbor- hood component analysis. In European Conference on Computer Vision (ECCV) 2018, 2018.)\nI understand that most supervised contrastive learning frameworks assume full label during the entire process, which is different from the setting in the paper. But it would be interesting (even critical) to compare the proposed method vs the other supervised contrastive learning methods when SuNCEt is given the full data.\n\nThe main contribution of the paper is the proposed SuNCEt loss, which is modified on top of the regular NCE loss in instance discrimination training. However, this “supervised constrastive learning“ only accelerates the SIMCLR learning process (by ~2x in terms of epochs from Table 1 and Table 2) and it does not significantly improve the accuracy over regular SIMCLR.\n\nOverall, I feel that the proposed loss is not a very novel idea (i.e., supervised contrastive learning) and the experiment results are not significantly better than prior arts.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes to speed-up self-supervised learning for semi-supervised learning by combining self-supervised pretraining and supervised fine-tuning into a single objective. The proposed supervised loss builds on Neighbourhood Components Analysis and soft nearest neighbor losses. Most reviewers are concerned about the novelty of the approach and the significance of empirical results. I agree with both concerns. I appreciate the comparison between $\log \sum \exp$ and $\sum \log \exp$, but it seems a simpler cross entropy loss also achieves a similar goal (potentially somewhat slower). I believe adding more experiments comparing different supervised loss functions across different architectures can help improve the paper.

Decision: Reject