Fast Training of Contrastive Learning with Intermediate Contrastive Loss
Authors: Chengyue Gong
Xingchao Liu
qiang liu
Venue:  (2021)

Abstract: Recently, representations learned by self-supervised approaches have significantly reduced the gap with their supervised counterparts in many different computer vision tasks. However, these self-supervised methods are computationally challenging. In this work, we focus on accelerating contrastive learning algorithms with little or even no loss of accuracy. Our insight is that, contrastive learning concentrates on optimizing similarity (dissimilarity) between pairs of inputs, and the similarity on the intermediate layers is a good surrogate of the final similarity. We exploit our observation by introducing additional intermediate contrastive losses. In this way, we can truncate the back-propagation and updates only a part of the parameters for each gradient descent update. Additionally, we do selection based on the intermediate losses to filter easy regions for each image, which further reduces the computational cost. We apply our method to recently-proposed MOCO, SimCLR, SwAV and notice that we can reduce the computational cost with little loss on the performance of ImageNet linear classification and other downstream tasks.

Under review as a conference paper at ICLR 2021
FAST TRAINING OF CONTRASTIVE LEARNING WITH INTERMEDIATE CONTRASTIVE Loss
Anonymous authors
Paper under double-blind review
ABSTRACT
Recently, representations learned by self-supervised approaches have significantly reduced the gap with their supervised counterparts in many different computer vi- sion tasks. However, these self-supervised methods are computationally challeng- ing. In this work, we focus on accelerating contrastive learning algorithms with little or even no loss of accuracy. Our insight is that, contrastive learning con- centrates on optimizing similarity (dissimilarity) between pairs of inputs, and the similarity on the intermediate layers is a good surrogate the final similarity. We exploit our observation by introducing additional intermediate contrastive losses. In this way, we can truncate the back-propee pro-paarpagttion and updates only a part of the parameters for each gradient descent update. Additionally, on the intermediate losses to filter easy regions for each image, which further re- duces the computational cost. We apply our method to secently-propssee MOCO (He et al. 2020). SimCLR (Chen et al. 2020a),, SWAV (Caron et al. (2020) and no- tiee that we can reduce the computational cost with little loss on the performance of ImageNet linear classification and other downstream tasks.
do selection based
INTRODUCTION
Recently, self- superviidd learning has been shown a promising approach for unsupervised and semi- supervised learning computer vision (Oord et al. 2018 Kolesnikov et al.) 2019 Zhai al. 2019] He et al. 2020 Chen et al., 2020b Grill et al. 2020). These methods learn unsupervised rep- resentation that can perform well on both ImageNet (Deng et al., 2009) linear classification and other down-stream tasks, e.g. pose estimation, detection, and semantic segmentation (He et al. 2020 Caron et al. 2020). A major type of self-supervised learning is contrastive learning, which constructs similar and dissimilar pairs the dataset and minimize constrative loss to learn mapping that yields similar (resp. dissimilar) pairs with similar (resp. dissimilar) representations.
Despite the recent successes, contrastive learning found to practically incur longer training time and higher computational cost compared with supervised learning (He et al. 2020 Grill et al. (020). For example, Hee al. (220)) and Chen et al. (2020a) requires 5 time than standard supervised learning on ImageNet. The enormous time and computational cost makes large-scale contrastive learning out of reach for many of the researchers and applications.
Grill
This work focuses on speed up contrastive learning. Our key observation is, because contrastive learning focuses on optimizing similarity (dissimilarity) between pairs of inputs, the similarity on the intermediate layers provide good surrogate of the final similarity, and computing the intermediate representation requires less computational cost. This in contrast with supervised learning, which requires to match the output of the final layer with a label and hence it is essential to calculate the final outputs.
We exploit the observation by introducing additional contrastive losses in the middle layers of the neural network. Instead of measuring only the contrastive loss of the representation in the last layer, compute the contrastive loss in the intermediate blocks. The intermediate losses enable the following two strategies that accelerates contrastive learning. (1) Partial Back- nack-propagation We start back-propagation randomly from of all the contrastive losses. Compared with doing full back-propei pook-prrpagttion in every optimization step, an intermediate starting point only requires computing the gradients for a part of the parameters. (2) Block-wise Hard Pair Selection: The intermediate

---

Under review as a conference paper at ICLR 2021
contrastive losses serve indicators of the similarity between the representations in early layers. These indicators can be used to filter out the simple pairs, thus reducing unnecessary computation.
We test the proposed method upon several recent sulf-supe learning algorithms, MOCO (He ct al. 2020), SwAV (Caron et al., 2020), SimCLR et al.) 2020a) and MOCO V2 (Chen et al. 2020b). We empirically show that method can save the training time with almost loss the final performance
the downstream tasks, e.g. ImageNet linear classification, PASCAL VOC object detection and segmentation. Our method largely reduces the training cost for contrastive learning methods by over 30%, and can serve as an alternative to standard self-supervised learning training pipeline the computation resources is limited.
METHOD
We first give brief introduction about contrastive learning in Sec. 2.1 and then introduce our method in Sec. [22 and Sec. [.. Our method is composed two major components: (1) using randomly early stopping for different mini-batch (2) using random crop and selecting hard regions on hidden states.
2.1 CONTRASTIVE LEARNING
Given an unlabeled set of images D = { we want to learn a representation map ʃ that extracts useful low-dimensional representations from the high-dimensional images x. contrastive learning, for each x ~ D, we use data augmentation or other techniques to construct a positive example 2 that is similar and a set of negative examples [== that are less similar to x than #+- Then we train the map f to maximize the similarity between (z) and fx++), ), while minimizing the similarity between f(x) and ((x= A popular choice contrastive loss is InfoNCE (Oord al. 2018).
exp ((x())
Linto
= log
(1)
Ek exp ((x()) + exp (+1(x)
The encoder aims to minimize the InfoNCE for all the images in the dataset,
L(J) EgED [[ [(x) ((x))))
where is a temperature hyper-parameterr Intuitively, Eq. 2 the log loss of a (K+1)-way softmax- based classifier that tries to classify fz) into the same class as f(x+).
There are many different methods to construct the positive and negative examples. In practice, people use context (Oord et al. 2018), data augmentation (Chen et al. 2020a He et al. 2020 Hénaff et al. 2019), colorization (Oord et al. 2018), clustering (Caron et al. the positive pair {#,20 x More generally, each image z can have multiple positive examples (Tan et al. 2019). Previous works have used in- in-batch data (Chen al. 2020a), memory bank (He al. 2020
2020 2018), etc., construct
or different regions one given image (Oord et al. 2018 to generate k=1 for each x.
{
2.2
ACCELERATED TRAINING BY PARTIAL BACK-PROPAGATION
Our approach stems from fundamental property of contrastive representation learning: contrastive loss measures the similarity between representations. Due to the hierarchical nature of deep neural networks, similar representation pairs from early layers are still similar even after being processed by the later layers. Therefore, we argue that full paok-propngttion is not necessary for contrastive representation learning. Rather, partial Back-propagation is enough for learning useful representa- tions in the final layer. Intermediate loss has been widely used in deep learning, from Inception Network (Szegedy al. 2015) DARTs (Liu et al. 2019).
We consider deep neural network f = fn o fn-1 ... f1. where f. refers to the i-th building block of the network. For feed-forward neural networks, the building block can be one hidden layer. For more complex network, e.g. ResNet (He et al.] (0166). the building block can be a sequential of convolutional layer, batch normalization layers (loffe & Szegedy 2015), and activation functions.
2

---

Under review as a conference paper at ICLR 2021
Selection Projection Head
Contrastive
Starting Point of BP
Gradient
Conv
Conv
Conv
Conv
Figure 1: Demonstration of partial back-propagation framework during training. The network in the figure has 4 blocks. For each block, we introduce additional projection head and contrastive loss. In this optimization step, we randomly choose start paok-propagation from the contrastive loss of the second block.
Algorithm 1 Accelerated Training by Partial Back- Phok-Propagation Input: network f = fn fn- ........................... f1, dataset D.
repeat
Randomly sample a mini-batch {= from D.
Randomly sample L; from Stoss-
Update f E {I,..., by mini-batch gradient descent on the loss 1 m Σx=1 Ci. until Convergence
Instead of using single contrastive loss C at the end of network, propose add auxiliary contrastive losses C, after each building block fi. obtaining a set of losses Sloss = (C,,..., C.). Expanding C; yields.
C, C(fit) EZND [Cinto (f (f1: fir(x+))
(3)
Here, (1x(()) is the intermediate representation after block fi. (I.( o... fi Note that we ignore the auxiliary projection head here for simple notation. In every optimization step, we randomly choose a loss L, from Stoss, and start proppaaaaann from L.,
Because propppropngntion from L, does not involve blocks fi+1 to fn, our method can reduce the computational burden from two aspects. (1) Reduction the number Multipyy-Accumulation operations (MACs). For example, consider the backward pass of standard ResNer-50 for one iteration, in the average case, our method can reducing the 8.2G MACs (He et al.) 2016) to 3.6G MACs. (2) Reduction in the storage and other additional costs. When training convolutional network, great proportion of the cost is the layer-wise storage and callback of the intermediate outputs and gradients (Chen al.] 2016 Yu al.) 2014 Kusumoto al.) (019). Our method decreases the depth of the back-propagation chain, and consequently decreases the additional storage and callback cost, Overall, both aspects ensure our method speed up the backward process in training.
2.3 BLOCK- WISE HARD PAIR SELECTION
Besides reducing the burden in back pack-propagation, the auxiliary losses enable us to judge whether the representations of æ and x4 are similar early stages of the forward pass. We can filter the simple pairs to further reduce the size of the computational graph, and in the meanwhile render the network focus on the difficult pairs. Our strategy not only significantly decreases the training time, but also helps recover the performance of standard training process, as demonstrated in Sec.
For x. and x+, let their intermediate representations after the i-th block be fis(2), fis: (2:4) Hx where C, H, W denote the number of channels, height, and width. We crop fix(*) and f1=(14) to obtain smaller patches and select the difficult pairs from them. We apply two- dimensional random crop (I)( and (1.) for M times, yielding two sets of smaller patches, B (sta MM (x:) and B+ (1L,(44))..., (M(x))) Here, PL,(2), (x

---

Under review as a conference paper at ICLR 2021
CxAxW
Cropped Patch Projection Head Contrastive
CxHxW
f200
fxa(x)
10
Choose the pair with highest loss
6r
Following Blocks
Following Blocks
(i.)
(x((4)
fao.)
Demonstration of hard pair selection when M From the four pairs, the pair with the highest contrastive loss is selected and passed to the following blocks. The procedure introduces little computational overhead except for a forward pass through the projection head. In comparison, smaller feature map significantly decreases the computational cost of the whole network.
Figure 2:
RCCBxx are cropped patches, and Á < H. W < W. We select the hard pair follows,
arg fis(t)) (R
Cito (fP, ((., p.q {1,2, M..
(4)
The above criterion picks the pair of representations with the largest contrastive loss. We then feed only (r) and (r+) to the later blocks in the network. Here, we only process the positive pairs, and do nothing for the negative pairs. Taking the properties of convolutional neural networks into account, our method can be viewed as progressively reducing the resolution of the original input images. We adopt standard image resolution for the first block, and apply our strategy in the others.
et al. 2020a: Misra & Maaten 2020), applying random crops on an image plays an essential role self-supervised learning. Caron et (2020) uses multiple crops of given images to do clustering. outpeforming standard SimCLR (Chen et al. 2020a) and other baselines (e.g. He et al., 20200, Chen et al., [020b] Hénaff et al., 2019). People also use multiple random crops and select the hard examples (Gong et al.) (020) to boost the performance. However, selecting the hard augmentation examples or using multi-crop will increase the computational cost, because they require multiple forward passes of the whole network for multiple input images. Our method, in contrast, acts on the feature map and filters the simple pairs in every intermediate block, resulting in both performance improvement and reduction of training time.
As noted in prior works
3 RELATED WORKS
Training a deep neural network usually requires high computation costs. Therefore, a long line of works have been devoted accelerating the training process of deep neural networks. Unlike efficient inference, methods for efficient training have been developed case by case (Han et al., 2016b Ye et al. 2020) Typically, researchers use domain knowledge design different strategies for different tasks to accelerate the training (Wu et al. 2020 Singh et al. 2018). In this work, we also leverage the special property of contrastive learning to develop the method.
Runtime Pruning Runtime Pruning removes redundant channels during training and uses the whole network with all removed channels for inference Lin al. (2017) uses RNN controller to remove useless filters during training. the controller decides if channel should be removed based on the feature map as inputs. Gao et al. (2018) replaces the RNN controller with another neural network The runtime pruning method can be applied to many of the supervised learning tasks. However, using the controller will introduce additional computation cost (Wang et al., 2019), and therefore usually could not accelerate the training by large margin in practice. Moreover, the tuning of the controller is still an ongoing problem.
Early skipping Early skipping techniques reduce computation by skipping part of the neural net- work whenever is unnecessary For example, in [Figurnov et af. (2017). an adaptive number of ResNet layers skipped within residual block for unimportant regions in object elassification.
4

---

Under review as a conference paper at ICLR 2021
The skipping mechanism is controlled by a halting scores predicted at the output branch of each residual unit. [i et al. (0117) handles easy regions in the early stages and harder cases are progres- sively fed forward to the next sub-model for further processing. These approaches are usually useful for specific tasks, e.g. segmentation, detection, which can take henefits from low-level features. For more general tasks, e-g. classification, it fails achieve good performance (Li et al. 2017).
Multi-scale training Multi-scale training is general approach in computer vision community, e.g- segmentation (He et al. 2017), detection (Singh et al. 2018), video (Wu al. 2020). However, the inevitable challenge is that the strategy is task-dependent or dataset-dependent. However, finding a general dynamic schedule for different cases adjust resolution extremely difficult (Wu al., 2020 Simonyan & Zisserman (2144). Usually, the strategy is a bit complicated. For example, Wu et al.
2020) proposes to schedule the batch size with both long and short cycles.
Summary and Discussion All these works are mainly based on two techniques: reducing resolution and skipping layers. Most of these methods are developed by either hand design or learning- -based method, e.g. reinforcement learning (Lin et al.] (017)). Compared to hand-designed method, the (which is not easy to tune) while the de- sign of search space still highly dependents on human knowledge. Therefore, in practice, it usually cannot find solution as good as hand-crafeee method or grid search. In this work, we design the acceleration method without the help of learning-bass based approach.
learning-based method introduces more hyyer-parametee
4 EXPERIMENTS
In experiments, we verify whether our proposed method can accelerate recent self-supervised learn- ing approaches with no loss on the performance on different tasks. We choose MOCO (He et al., 2020), simCLR (Chen et al. (200a)), SwAV (Caron et al. (020)) and MOCO V2 (Chen et al. 2020b) as our benehmark, and test the performance on object detection, segmentation and classification. Our results are averaged over 5 trials for all the experiments.
Training Settings We train the self- supervised methods on ImageNet
72.
(Deng et al. 2009) training set that
Test Accuracy
Test Accuracy
66
has 1.28 million images in 1000
classes. This dataset well- balancee
68.
62.
in its class distribution, and is the
most commonly-used benchmark to train sele-super representations.
MOCO V2
simCLR Accal sinCLR
64
Accal MOCO V2
4
12
16
12
16
We implement all the baseline al- gorithms and our methods all on the most widely-used ResNet-50 (He) al. 2016). For the hyper- parameters used in the training pro- cess, we keep almost all the hyper- parameters the same as their reporting in the paper, except batch size and learning rate. As our method can save GPU memory by selecting hard pair, we can use larger batch size. In practice, set the batch size as large as possible and linearly scale the learning rate (He et al. 2011)). We train all the methods and models on GPUs and compare with all the baselines trained in the same settings and machines.
(a) MOCO V2
(b) SimCLR
Figure 3: Linear classification ImageNet. The x-axis shows the training time (days), and the y-axis shows the top- accuracy for the linear classifier.
For the lase loss on ResNet-50, we set three auxiliary head (loss) after the three blocks. The projection head is several linear transformations followed by softmax layer and the contrastive loss (Chen et al. 2020a). For the hard region pair selection, we select hard pairs after the first and second block. The size of the selected regions is 3/4 of the original size.
Downstream Settings After training the self-supervised representations, we verify the performance on several downstream tasks. The main goal of unsupervised learning learn transferrable fea- tures, therefore we test the performance on different tasks except for the simple linear classification. We first verify the performance on ImageNet linear classification task with frozen features, which
We use NVIDIA TITAN V VOLTA 12GB, and record the training time for this setting.
5

---

Under review as a conference paper at ICLR 2021
is the most used benchmark. Then, we verify the performance segmentation and detection tasks, with both frozen features and unfrozen features.
4.1 IMAGENET LINEAR CLASSIFICATION
We evaluate the representations trained with various methods on ImageNet. We train all the models with 200, 400 or 800 epochs. For SwAV, we do not apply the multi-crop strategy described in their paper. We run the linear classification three times and report the final results.
We show the primary result in Table T It shows that applying to different self-supervised learning algorithms and different epochs, our method can always accelerate the training with little loss of accuracy. For simCLR and SwAV, we can even boost the top-1 accuracy (e.g. from 61.7% to 62.9% for simCLR and from 70.1% % to 70.7% for SwAV) with less training time. We further show the relationship between training time and top-1 accuracy in Figure 3 With the same time budget, the proposed algorithm can improve the performance of the linear classifier. Especially when the time GPUs), the proposed accelerating training method can boost
budget is limited (e.g. 3 days the performance with a large margin. For example, as shown in Figure 3 we can improve the MOCO V2 performance from 64% 66.5% when the training time is limited to 3 days. When more computation resources are given, our proposed method will converge to the same performance as the standard version.
Method simCLR
Epoch
Top-1
Time (Days)
200
61.7+0.2
4.3
Accel-simCLR
200
62.910.2
2.8
MOCO
200
60.6
4.3
Accel-MOCO
200
60.5+0.11
2.8
MOCO V2
200
67.5
4.3
Accel-MOCO V2
200
67.3+10.1
2.8
MOCO V2
800
71.1=0.1
17.2
Accel-MOCO V2
800
71,0+0.2
10.9
SwAV
400
70.1
8.1
Accel-SwAV
400
70.7
5.3
Table 1: Linear classification on ImageNet. Top-1 accuracy for linear models trained on frozen features from different self-suuprvised methods.
4.2 TRANSFERRING FEATURES
Since the main propose for self-supervised learning is to get universal representation, it necessary check whether the learned representations can be transferred more downstream tasks. We compare our accelerating method and the standard versions, transferred to various tasks including PASCAL VOC 2007 and PASCAL VOC 2012. Followed MOCO (He al., (020), we use an additional BN layer after the backbone, and synchronize the BN statistics across GPUs. We test the following models: simCLR trained with 200 epochs, MOCO trained with 200 epochs, MOCO V2 trained with 800 epochs and SWAV trained with 400 epochs.
VOC 2012 Segmentation We first evaluate all the models on PASCAL VOC 2012 segmentation tasks, and show the results in Table 2 We test two different settings: freezing the backbone or not, and report two metrics: mean IoU (Intersection-Oveel Union) and pixel-wise accuracy. We use PSP- Net (Zhao al. 2017) as the decoder. Table Table@shows that for all these settings and different models, our proposed accelerating method would not drop the performance on VOC 2012 segmentation task. Because of the small standard deviation (e.g. < 0.1), we do not list them in the table. We notice that although our accelerated MOCO V2 performs worse than standard MOCO V2 on ImageNet linear classification, they achieve the same performance on this task. On the other hand, the accelerated SWAV is no longer better than standard SwAV this task. It indicates that we should test the algorithm on various tasks to verify whether it would cause a drop in the performance.
VOC 2007 Object Detection We further test all these models the VOC 2007 object detection task. The detector Faster R-CNN with R50-C4 (He et al., (017)), using Detect ron2) (Wu et al.) 2019). The image scale is [480, 800] pixels during training and [800, 800) at inference. We evaluate

---

Under review as a conference paper at ICLR 2021
Method
Freeze
mean IoU
Accuracy
Freeze
mean IoU
Accuracy
Supervised
76.9+0.1
85.0=0.0
X
79.1
87.1=0.2
simCLR
72.411..1
82.0.10.0
X
74.8:00.1
83,410.1
Accel-simCLR
72.910.0
82.3
X
75.2
83.7=0.1
MOCO
7..1
83.2+0..1
X
0.1
Accel-MOCO
75.2+:0.2
83.310.1
76.9=0.1 X
84.6=0.1
MOCO V2
78.6+:0.2
86.6:1022
X
79.4
87.3=0.2
Accel-MOCO V2
78.6+:0.2
86.5+1022
X
79,3+0.2
87.3=0.2
SwAV
72.710.1
80.7H0.0
X
74.4
83.5=0.1
Accel-SwAV
72.7+10.1
80.8+10.0
X
74.5
83.5
Table 2: Image segmentation on PASCAL VOC 2012. The data is per-processed with (Hariharm al. 2011). For 'freeze', v denotes we freeze the backbone parameters, while x denotes parameters not frozen. We report mean loU (Intersection-O0 -eee--OOv---nnnn and pixel-wise accuracy to measure the performance.
the default VOC metric of AP50 (i.e., IoU threshold is 50%) and the more widely-used metrics, COCO-style AP. We do training on VoCtrainva12000 and do valuation on the VOCCeest200 We display the experiment results in Table On VOC 2007 object detection tasks, our accelerated versions do not have worse performance compared to the standard version for all the tested methods.
Method
Freeze
AP50
AP
Freeze
AP50
AP
Supervised
72.8+0.0
39.5+0.11
X
74.4+10.0
42.4+00.1
simCLR
71.2+0.0
37.4+0.2
X
73.340.1
40.1
Accel-simCLR
71.4+0.0
37.810.1
X
73.6=10.0
40.6+t.1
MOCO
73.640.1
40.510.1
X
77.9100.0
46.6110.1
Accel-MOCO
73.610.0
40.510.1
X
78.010.11
46.7.10.1
MOCO V2
77.740.1
46.510022
X
80.2.1..0
50.510.1
Accel-MOCO V2
77.540.1
46.2+0.1
X
80.110.0
50.240.1
SWAV
74.24:011
42.11:.2
X
78.2.00.0
46.8
Accel-SwAV
74.340.1
42.3+.022
X
78.4+10.1
46.9.10.1
Table 3: Object detection on PASCAL VOC 2007. For 'freeze', / denotes freezing the backbone parameters, while x denotes the parameters not frozen. We report AP (average precision) and AP50 to measure the performance.
4.3 FURTHER STUDY
We further do some study to empirically understand how each part of the proposed accelerating method contributes to the final result.
Layers Cluse to the Input are Important We first
study which part of given neural network is impor-
tant or even necessary for the representation learning during training.
Updated Layer
Un-Updated Layer
Shown in Table 4 we test which layer could be skipped during training. We test three different tings: 1) randomly selecting some not adjacent lay- ers to do backward, and pass the information through the not selected layers with skip connection 2) stop-
Figure 4:
The three update scheme de- scribed in context.
ping forward at one certain layer, and then doing backward the input 3) selecting several nearby blocks to do backward and not do always backward to input. We display these three approaches in Figure 4 We test these methods on MOCO V2 and simCLR, and report the results in Table Table 4 displays that doing backward to input achieves best performance, whle the other two approaches achieve worse results. It indicates that for these self-supervviee algorithms,, the blocks close to the input need more updates and gradient information. Recent work (Zhang et al. 2019) also shows that for supervised image classification, the layers close to the input are important for many different neural architectures.

---

Under review as a conference paper at ICLR 2021
Method
Accuracy
Time (GPU Days)
Method
Accuracy
Time (GPU Days)
MOCO V2
67.5+0.1
4.3
simCLR
61.7+0.2
4.3
Skipping
65,3:10.2
2.8
Skipping
61.5:10.3
2.8
BP-to-Input 67.3+0.1
2.8
BP-to-Input
62.9
2.8
Nearby-Blood 65.5++0.2
2.8
Nearby-Blowd Block
61.3.10.2
2.8
Table 4: Tested on ImageNet linear classification task, we empirically study which layer is important for celf-superviee learning.
Hard Pairs v.s. Easy Pairs v.s. Hard Examples In our framework, we filter easy regions (i.e. regions with lower contrastive loss) in the hidden states. To fully understand the role of the operation, we compare to two baselines under the same computation budget: 1) select easy region, 2) select hard examples in each mini-batch.
We show the performance of all these three approaches in Table It demonstrattss that selecting easy pairs drops the performance (e.g. for MOCO V2, compared to selecting hard pairs, selecting easy pairs will break top-1 accuracy from 67. 97.3% to 65.3%). We also notice that selecting hard examples would be a bit worse than selecting hard region pairs. This is reasonable for contrastive learning since many researchers have found that using multi-crop is helpful for contrastive learning, and can even boost the performance downstream tasks in some cases (Grill et al. 2020 Chen et al., [2020a] Misra & Maaten 2020). Moreover, selecting hard region pairs seize out the meaningful hard context for each given image. Predicting meaningful context itself can be useful for self-supervised learning (Oord et al. 2018).
Method MOCO V2
Accuracy
Method
Accuracy
67.5
simCLR
61.7+#0.2
Hard Pair
6.1
Hard Pair
62.910.2
Easy Pair
6..
Easy Pair
60.3
Hard Example
66.1
Hard Example
61.6
Table 5: We demonstrate an study of the role of selecting hard region pairs. Compared to selecting hard examples or easy regions, selecting hard region pairs can achieve better performance.
Ablation Study Finally, we conduct an ablation study simCLR and MOCO V2 to see the ad- vantages of each part for our proposed algorithm. As displayed in Table for MOCO V2, both the partial pook-propagttion and the block-wis. wise hard pair selection can achieve similar accuracy the standard version while using fewer computation resources. For simCLR, hard pair selection can even boost the performance of the original version with less training time, while partial back- proc-prrppgationn achieves similar performance as the standard simCLR. Combining these two approaches together can further save training time while having no loss of accuracy for both MOCO V2 and simCLR.
Method
Accuracy
Time (GPU Days)
Method
Accuracy
Time (GPU Days)
MOCO V2
67.5+F0.1
4.3
simCLR
61.7+0.2
4.3
+ PB
67.3++0.2
3.7
+ PB
61.8+0.11
3.7
HP
67.110.1
3.4
HP
63.1-0..2
3.4
+ PB + HP
67.3+10.1
2.8
+ PB + HP
62.9+0.2
2.8
Table 6: 'PB' denotes partial back propaaropg,,,on, 'HP' denotes hard pair selection for each image pair.
CONCLUSION AND DISCUSSION
Recent self-supervised learning has achieved great success in computer vision, e.g., matching or even outperforming the supervised learning model some vision tasks. However, the huge com- putation cost prevents many researchers from using on their research topics. In this paper, we focus accelerating the training of self-superviser learning based contrastive loss. Empirically, we show that our proposed method can achieve the same performance as several recent proposed self seff-supervised learning methods, with far less training time. In the future, plan further study
8

---

Under review as a conference paper at ICLR 2021
the theoretical property of these self-supervised approaches, whether it has some inherent property to achieve better performance or faster convergence. Furthermore, we will test our method on more tasks to whether it is more general approach.
REFERENCES
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsu- pervised learning of visual features. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 132-149, 2018.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. arXiv preprint arXiv:2006.0968 09882, 2020.
Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost. arXiv preprimt arXiv: 160.00664 2016.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. ICMI,, 2020a.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint Stt1::20030042997, 2020b.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In IEEE conference on computer vision and pattern recognition, pp- 248 255. leee, 2009.
Michael Figurnov, Maxwell Collins, Yukun Zhu, Li Zhang, Jonathan Huang, Dmitry Vetrov, and Ruslan Salakhutdinov. Spatially adaptive computation time for residual networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1049-1048, 2017.
Xitong Gao, Yiren Zhao, Lukasz Dudziak, Robert Mullins, and Cheng-zhong Xu. Dynamic channel pruning: Feature boosting and suppression. arXiv preprint arXiv:/810.053:1 2018.
Chengyue Gong, Tongzheng Ren, Mao Ye, and Qiang Liu. Maxup: A simple way to improve generalization of neural network training. arXiv preprint arXiv:2002.09024 2020.
Lean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi
Azar, al. Bootstrap your own latent: A new approach to self-supervised learning. arXiv preprint arXiv:2006.07733, 2020.
Song Han, Xingyu Liu, Huizi Mao, Jing Pu, Ardavan Pedram, Mark Horowitz, and William Dally. Eie: efficient inference engine on compressed deep neural network. ACM SIGARCH Computer Architecture News, 44(3):243-254, 2016a.
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. International Conference on Learning Representations, 2016b.
Bharath Hariharan, Pablo Arbeléez, Lubomir Bourdev, Subhransu Maji, and Jitendra Malik. Se- mantic contours from inverse detectors. In 2 International Conference on Computer Vision, pp. 991-998. IEEE, 2011.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference computer vision, pp. 2961-2969, 2017.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9729-9738, 2020.

---

Under review as a conference paper at ICLR 2021
Tong He, Zhi Zhang, Hang Zhang, Zhongyue Zhang, Junyuan Xie, and Mu Li. Bag of tricks for image classification with convolutional neural networks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 558-567 2019.
Olivier Hénaff, Aravind Srinivas, Jeffrey De Fauw, Ali Razavi, Carl Doersch, SM Eslami, and Aaron van den Oord. Data-efficient image recognition with contrastive predictive coding. arXiv preprint arXiv:I905. 00200,99222222 2019.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. International Conference on Machine Learning. 2015.
Alexander Kolesnikov, Xiaohua Zhai, and Lucas Beyer. Revisiting self-supervised visual represen- tation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recog- nition, pp. 1920-1929, 2019.
Mitsuru Kusumoto, Takuya Inoue, Gentaro Watanabe, Takuya Akiba, and Masanori Koyama. A graph theoretic framework of recomputation algorithms for memory-efficien backpropagation. In Advances in Neural Information Processing Systems, pp. 1163-1172, 2019.
Xiaoxiao Li, Ziwei Liu, Ping Luo, Chen Change Loy, and Xiaoou Tang. Not all pixels are equal: Difficulty-awant semantic segmentation via deep layer cascade. In Proceedings of the IEEE con- ference on computer vision and pultern recognition, pp. 3193-3202, 2017.
Ji Lin, Yongming Rao, Jiwen Lu, and Jie Zhou. Runtime neural pruning. In Advances in neural information processing systems, pp. 2181-2191, 2017. 2017.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. ICLR, 2019.
Ishan Misra and Laurens van der Maaten, Self-supervised learning of pretext-invariant representa- tions. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 6707-6717, 2020.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint 187.v:1807.03745 2018
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1556, 2014.
Bharat Singh, Mahyar Najibi, and Larry S Davis. Sniper: Efficient multi-scale training. In Advances in neural information processing systems, pp. 9310-9320, 2018.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1-9, 2015.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. arXiv preprint arXiv: (90:::996600000 1906.05899 05849, 2019.
Yue Wang, Ziyu Jiang, Xiaohan Chen, Pengfei Xu, Yang Zhao, Yingyan Lin, and Zhangyang Wang. E2-train: Training tftte-of-the-art enns with over 80% energy savings. In Advances in Neural Information Processing Systems, pp. 5138-51501 2019.
Chao-Yune Yuan Wu, Ross Girshick, Kaiming He, Christoph Feichtenhofer, and Philipp Krahenbuhl. A multigrid method for efficiently training video models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 153-162, 2020.
Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick. Detectron2, 2019.
Mao Ye, Chengyue Gong, Lizhen Nie, Denny Zhou, Adam Klivans, and Qiang Liu. Good sub- networks provably exist: Pruning via greedy forward selection. International Conference on Machine Learning, 2020.
10

---

Under review as a conference paper at ICLR 2021
Dong Yu, Adam Eversole, Mike Seltzer, Kaisheng Yao, Zhiheng Huang, Brian Guenter, Oleksii Kuchaiev, Yu Zhang, Frank Seide, Huaming Wang, et al. An introduction to computational net- works and the computational network toolkit. Technical report, Microsoft Technical Report MSR- TR-2014-1122 12, 2014.
Xiaohua Zhai, Avital Oliver, Alexander Kolesnikov, and Lucas Beyer. S41: Self-supervised semi- supervised learning. In Proceedings of the IEEE international conference on computer vision, pp- 1476-1485, 2019.
Chiyuan Zhang, Samy Bengio, and Yoram Singer. Are layers created equal? arXiv preprint arXiv:1902.01196, 2019.
Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 2881-2890, 2017.
A IMPLEMENT DETAILS
For the training of self-supervised learning algorithm, we use the open-source platforms to im- plement and keep almost all the hyper-parameters the same, except learning rate and batch size. For simCLR, we use 256 batch size, which achieves sightly worse result than [Chnn et al. (2020a) reported. For SwAV which use cluster centers as positive examples, we keep additional block-wise cluster centers for the loss. For MOCO and MOCO V2 which use memory bank, we keep additional layer-wise queen (memory bank), which costs additional memory but does almost no hurt training time.
For detection and segmentation, we use open-source code base and keep the syyper-prramete the same. We add an additional BN layer after the backbone for this setting.
B ABLATION STUDY
In the section, we do more ablation study to show the impact of different parts of our approach. We mainly show the result simCLR and SWAV.
B.1 DIFFERENT NUMBERS OF SELECTED REGION
In Eq we select hard pair over M candidates. In practice we notice that for simCLR trained with 200 epochs, the number of candidates M does not impact the final performance a lot. We show the results Table
Accuracy 0.2
62.8+t0.1
0.2
62.
Table 7: Top-1 accuracy for linear models trained on frozen features ImageNet. We change the number candidates for selecting hard pairs in this table.
B.2 INTERMEDIATE REPRESENTATIONS
We use the intermediate representations of simCLR and Accel-simCLR to do the linear classification task for ImageNet. As shown Table g the intermediate representations are improved when inter- mediate loss is used. Compared to the final layer representations, the intermediate representations of simCLR and Accel-simCLR have a larger gap on ImageNet top-1 accuracy.
receboohkeeeerrcc//a noco http:///eeeeeeaaaaaoeooooooccetn https://githhut
com
fSup
and http://aanttttttttaaaaaaa

---

Under review as a conference paper at ICLR 2021
Featl
Feat2
Feat3
Feat4
Final
simCLR Accel-simCLR
17.6=0.2
31,6::.22
41.9:10.2
54.1=0.2 55.6+0.2
61.7=00.2 62.9#0.2
19.4+0.2
33.5++0.3
43.5++.2
Table Top-1 accuracy for linear models trained on frozen features on ImageNet. 'Feat n' repre- sents the hidden after n-th block in ResNet-50, and 'Final' denotes the final-layer representations.
Method
Epoch
Top-1
Time (Days)
2 x 224
400
70.1
8.1
Accel-2 x 224
400
70.7:10.1
5.3
160 + 4 96
400
74.2+:.1
8.8
Accel-2 x 160 + 4 x 96
400
74.1+-0.1
5.7
Table 9: We conduct different multi-crop settings for SwAV.
B.3 MULTI-CROP FOR SWAV
For SwAV, we test two different settings, using 224 and 160 + x 96 during training for given image.
224 denotes for one image, generate 224 x 224 augmented images. 2 x 160 + 4 x 96 denotes for one image, we generate two 160 x 160 and four 96 x 96 augmented images. Table denotes that, for both settings, can accelerate the original version without loss on accuracy.
resolution
224
196
128
Accuracy
67.310.1
60.6-00.2
53.5
Table 10: Top-1 accuracy for linear models trained on frozen features on ImageNet. We change the resolution for selecting hard pairs in this table. We do the experiments on MOCO V2, using the official implementation?
B.4 REDUCING RESOLUTION HURTS PERFORMANCE
straightforward way accelerate the training process is smaller image resolution. Shown in Table T0, for MOCO V2, reducing the image resolution hurts the performance of the ImageNet linear classification by large margin. For example, using 128 x 128 resolution only achieves around 53.5% top-1 accuracy on ImageNet.
12


Reviews:
[{"review_id": "xNp0euf_-s", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this submission the authors suggest modifications to reduce the computational cost of contrastive learning. The authors propose to add constrastive loss heads at intermediate stages of the network. Using these intermediate losses as a proxy for the final contrastive loss they make two proposals for  reducing computational effort: 1) randomly choosing which contrastive loss to start from such that on average less of the network is actually applied. 2) randomly cropping the image at each intermediate loss and continuing only with hard patches for the positive samples. \n\nOverall, I am on the fence for this one. I think adding the intermediate contrastive losses is an interesting idea and the proposed changes already improve training times. However, the reductions in training times are not overwhelming and I believe there would be many more aspects to explore for earlier contrastive losses. Thus, I am not overly enthusiastic.\n\nPros:\n1) Earlier contrastive losses are an interesting idea!\n2) There is a reduction in training times at negligible cost in performance.\n3) The method is quite general over contrastive learning methods.\n\nCons:\n1) The reduction in training times is not overwhelming given the amount of changes.\n2) While the authors present a few analyses which parts of their method and network are important I don’t think these are particularly informative.\n3) Having losses for the individual parts of the network, the immediate idea would be to train the parts separately which would remove many more backpropagation operations and would allow parallelization of the network parts onto separate machines. This possibility is not explored here.\n4) Given that the stated aim is to reduce training times it seems that there is very little analysis on what actually takes time in this approach. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "edEYQunYam9", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes techniques to speedup contrastive self-supervised pre-training. The two key ideas are to back-prop only through a subset of layers in the network (from a random layer, back to the input), and to drop training instances which are not \"hard\" (in a way defined more precisely in the paper). By combining these techniques the paper shows that pre-training time can be reduced by ~30% without significant loss in performance on downstream tasks.\n\nThe proposed acceleration techniques are evaluated on multiple contrastive learning methods (SimCLR v1, MoCo v2, SwAV) and for multiple downstream tasks (image classification, segmentation, and object detection). The experiments convincingly demonstrate that the accelerated approach works well when pre-training for a relatively small number of epochs.\n\nMy main criticism is that the performance values reported in the paper do not always match what is reported in the literature. Presumably this is because the amount of pre-training is less than what was done in previous work. For example, SwAV reports achieving 75.3 top-1 on ImageNet with linear classification, while Table 1 in the paper only reports 70.1. Although the acceleration approach works well early in training, it is not clear whether the same benefits play out when one aims to achieve performance on par with that reported in the literature. More (longer) experiments are required to support this claim. \n\nOne would expect that, eventually, the approximation techniques introduced to speed up training, may have a negative effect on the ultimate performance. It would be useful to know at what point in training (or at what scale) this becomes apparent. And if it is not the case that the approximation techniques cause any loss in performance, it would be nice to have some deeper understanding of why that's the case.\n\nThe claim that the proposed approach reduces the (average) memory usage during training is accurate, although it should be clarified that this is the _average_ memory usage. The peak memory usage still occurs when all layers are used to produce the representation, and this is no different than the peak memory usage of standard methods. Anyways, I guess the main message is about reduction in training time, and memory savings could be down-played.\n\nFine-tuning the full architecture generally performs much better than just linear evaluation for image classification. Did you also investigate the performance of methods using fine-tuning? Also, the test accuracy reported in Fig 3(b) is much lower than that reported in the SimCLRv1 paper (Table 6). Is it clear how Accel-simCLR behaves when run for more pre-training epochs?\n\nMinor nit: Looking at the values in Table 1, I wonder why runtime isn't proportional to epochs for MoCo v2 (for both the standard and accelerated versions).\n\nHow many replicas are standard deviations computed over?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "BCyF5JJW5Wk", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThe paper proposes a clever trick to make instance contrastive learning faster by using the intermediate feature layers to perform the contrastive loss rather than just using the final 2048-d mean-pooled features as is typically done in MoCo or SimCLR. That way, the backprop costs are cheaper. The authors also use this intermediate representation based similarities to guide a better set of negatives to the layers on top. The authors demonstrate good results that show better time to accuracy (on the linear classifier) with both MoCo and SimCLR. A useful consquence of this paper is making contrastive unsupervised learning on ImageNet more accessible to people with less computation resources, ex PhD students in academic labs who may not have a DGX-1 or v3-TPU pods.\n\nPros:\n\nTackles an important problem - how can we make self-supervised learning even faster with clever engineering\nDelivers on the problem by proposing two sensible solutions - saving the cost of backprop by using intermediate layers for contrastive losses, and also better negative mining while doing the fprop in a computationally efficient manner.\nGood results in terms of time to accuracy on linear classifier - both with MoCo and SimCLR - two leading instance contrastive learning approaches.\nCons:\n\nNo code release as of yet. I believe the utility value of engineering driven papers is high but heavily relies of clean open source code that's usable by the community.\nNo results with fewer GPUs. If you can get results faster but still use 8 GPUs (though I do notice you have used 12 GB memory Titan V as opposed to 16 Gb memory GPUs used by MoCo, I don't think that's a deal breaker since MoCo ResNet-50s can be trained with the same HBM too). I think what's more important is to show you need fewer GPUs - can someone with a single GPU get results with the same amount of compute-time taken by Facebook to get MoCo's results.. That's what really increases accessibility to folks with less compute IMHO.\nReducing time to accuracy could be done in a host of other ways - eg - make everything run in fp16 or bfloat16 on the right kind of hardware - you would see a speedup of ~1.3x already. Use smaller image sizes while training, eg - instead of 224x224, use 192x192, use different temperatures (tune hyperparams) for training within as few epochs as possible. Use less unlabeled data to get the same accuracy. All these are orthogonal to your proposed method ofc, so not trying to compare them with yours. My point is that - why not push the broader goal to the extreme - can one combine all these engineering tricks in one piece, and design a way faster contrastive learning pipeline that can train with way less resources and still perform as well in much less wall clock time..\nThe Related work section can be improved - example, Contrastive Learning prior work should mention the CPCv2 work (Henaff et al 2019); and the idea of using intermediate layers and backprop. the losses through them should mention the Inception architecture (Szegedy et al. 2015) as well as Deep Classification work that did this few years ago for supervised learning.\n\n\nRating: Weak Reject - I would be open to reconsidering this if the authors commit to releasing their code, and also performing some benchmarking with respect to number of GPUs, and revising related work. I also encourage the authors to push further on making their pipeline even more efficient.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "8l_F4N5eqVe", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary**: \nThis paper proposes a new pipeline to speed up the training of contrastive learning. Specifically, besides the contrastive loss placed at the very end of the network, it introduces several additional intermediate losses. During the training, only one of them is used to compute gradients. With these intermediate losses, it can be also used to filter out the easy pairs. These two strategies can significantly accelerate contrastive learning while matching the performance of the recent methods. Authors conduct experiments and ablation studies to show the effectiveness of the proposed method on ImageNet and downstream tasks.\n\n**Pros**:\n+ The whole idea makes sense.  It introduces the intermediate losses to reduce the computation costs, and filter out easy samples to  recover the performance. \n+ Authors conduct experiments on linear classification and downstream task transfer learning to show good performance.\n+ Overall, the paper is well written and well organized.\n\n**Concerns**: \n- As we add multiple intermediate contrastive losses to the network, can we also evaluate the linear classification and downstream transfer task using the intermediate representations (after the project head)? I am curious about whether they can also obtain good performance.\n- I wonder what if we update the network using all loss L, but not randomly sampling one of them? Will it improve the baseline performance?\n- Does the hyper-parameter M affect the performance a lot when doing the hard pair selection? Can we do the ablation study on it?\n- As mentioned in [1], the low- and mid-level representations, not high-level presentations, make the instance discrimination good for the detection task. This may also support the effectiveness of adding intermediate losses. \n\n[1] What makes instance discrimination good for transfer learning?\n\n**Minor Comments**:\n* Page 2 (middle), below eq(2), same lass -> same class\n\nOverall, I prefer the rating as above the threshold at the current stage. Hope the authors could address my concerns or questions in the rebuttal period.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces modifications that allow to make the training of contrastive-learning-based models practical. The goal of the paper is very interesting, and the motivation clear. This paper tackles a very important issue with recent unsupervised feature learning methods.
However, while the goal is great, the present submission does not provide time improvements on par with the ambitions of this work. As noted by R2, many other hacks could be used in conjunction with the current work to scale this goal to the extreme, yielding time improvements which would be of a more impressive magnitude. In its current form, this paper unfortunately doesn’t meet the bar of acceptance.
Given the interesting scope of this work, I strongly encourage the authors to take the feedback from reviews and discussions into account and submit to another venue.
Decision: Reject