Contrastive Learning with Stronger Augmentations
Authors: Xiao Wang
Guo-Jun Qi
Venue:  (2021)
Keywords: Contrastive learning
Self-supervised learning
Unsupervised learning
Stronger augmentations

Abstract: Representation learning has been greatly improved with the advance of contrastive learning methods with the performance being closer to their supervised learning counterparts. Those methods have greatly benefited from various data augmentations that are carefully designated to maintain their identities so that the images transformed from the same instance can still be retrieved. Although stronger augmentations could expose novel patterns of representations to improve their generalizability, directly using stronger augmentations in instance discrimination-based contrastive learning may even deteriorate the performance, because the distortions induced from the stronger augmentations could ridiculously change the image structures and thus the transformed images cannot be viewed as the same as the original ones any more. Additional efforts are needed for us to explore the role of the stronger augmentations in further pushing the performance of unsupervised learning to the fully supervised upper bound. Instead of applying the stronger augmentations directly to minimize the contrastive loss, we propose to minimize the distribution divergence between the weakly and strongly augmented images over the representation bank to supervise the retrieval of strongly augmented queries from a pool of candidates. This avoids an overoptimistic assumption that could overfit the strongly augmented queries containing distorted visual structures into the positive targets in the representation bank, while still being able to distinguish them from the negative samples by leveraging the distributions of weakly augmented counterparts. The proposed method achieves top-1 accuracy of 76.2% on ImageNet with a standard ResNet-50 architecture with a single-layer classifier fine-tuned. This is almost the same as 76.5% of top-1 accuracy with a fully supervised ResNet-50. Moreover, it outperforms the previous self-supervised and supervised methods on both the transfer learning and object detection tasks.


Under review as a conference paper at ICLR 2021
CONTRASTIVE LEARNING WITH STRONGER AUGMEN- TATIONS
Anonymous authors Paper under double-blind review
ABSTRACT
Representation learning has been greatly improved with the advance of contrastive learning methods with the performance being closer to their supervised learning counterparts. Those methods have greatly benefited from various data augmenta- tions that are carefully designated to maintain their identities so that the images transformed from the same instance can still be retrieved. Although stronger aug- mentations could expose novel patterns of representations to improve their gener- alizability, directly using stronger augmentations instance discrimination biaaad contrastive learning may even deteriorate the performance, because the distor- tions induced from the stronger augmentations could ridiculously change the im- age structures and thus the transformed images can be viewed the same the original ones any more. Additional efforts are needed for us to explore the role of the stronger augmentations in further pushing the performance of unsupervised learning to the fully supervised upper bound. Instead of applying the stronger aug- mentations directly to minimize the contrastive loss, we propose minimize the distribution divergence between the weakly and strongly augmented images over the representation bank to supervise the retrieval of strongly augmented queries from pool of candidates. This avoids an overoptimistic assumption that could overfit the strongly augmented queries containing distorted visual structures into the positive targets in the representation bank, while still being able to distinguish them from the negative samples by leveraging the distributions of weakly aug- mented counterparts. The proposed method achieves top-1 accuracy of 76.2% on ImageNet with a standard ResNer-50 architecture with a single-layer classifier fine-tuned. This is almost the same as 76.5% of top-1 accuracy with a fully su- pervised ResNet-500. Moreover, it outperforms the previous self-supervised and supervised methods on both the transfer learning and object detection tasks.
INTRODUCTION
Deep neural network has shown its sweeping successes in learning from large-scale labeled datasets like ImageNet (Deng et al. (2009).. However, such successes hinge on the availability of a large amount of labeled examples that are expensive collect. To address this challenge, unsupervised visual representation learning and self-supervised learning, have been studied to learn feature rep- resentations without labels. Among them the contrastive learning (Hadsell
(2006); Misra & Maaten 20200; Chen et al. (2020b): He et al. (2020): Caron et al. (222)), showing great potentials to close the performance gap with supervised methods.
In contrastive learning Hadsell et al. (2006). each image is considered as an instance, and we wish train the network so that the representations of different augmentations of the same instance are as close as possible to each other (He et al. (2020); Chen et al.] (2020a); Wu et al. (2018) Hjelm) al. (2018); Oord et al. (2018); [Bachman al. (2019); /hhuang al. (2019); Tian al. (2019); Hénaff et al. (2019)). Meanwhile, the representations of different instances be also distinguished between each other.
It is worth noting that these methods usually rely image augmentations that are carefully desig- nated to maintain their instance identities so that the augmentation an instance can be accurately retrieved from a dictionary of instances. On the other hand, we believe stronger augmentations could expose novel patterns which can further improve the generalizability of learned representations and eventually close the gap with the fully supervised models. However, directly using stronger augmen-

---

Under review as a conference paper at ICLR 2021
tations in the contrastive learning could deteriorate the performance, because the induced distortions could ridiculously change the image structures and thus the transformed images cannot keep the identity of the original instances. Thus, additional efforts are needed for us to explore the role of the stronger augmentations to further boost the performance of self-supervised learning.
Thus we propose the CLSA (Contrastive Learning with Stronger Augmentations) framework to ad- dress this challenge. Instead of applying strongly augmented views to the contrastive loss, we pro- pose minimize the distribution divergence between the weakly and strongly augmented images over a representation bank to supervise the retrieval of stronger queries. This avoids an overop- timistic assumption that could overfit the strongly augmented queries containing distorted visual structures into the positive targets, while still being able to distinguish them from the negative sam- ples by leveraging the distributions of weakly augmented counterparts. The learned representation will not only explore the novel patterns exposed by the stronger augmentations, but also inherits the knowledge about the relative similarities to the negative samples.
The experiments on various datasets demonstrate that the proposed framework can greatly boost the performance by learning from stronger augmentations. On the ImageNet linear evaluation protocol, we reach a record 76.2% top-1 accuracy with the standard ResNet-50 backbone, which is almost as high as 76.5% top-1 accuracy of the fully supervised model. Mcanwhile, it also achieves the competitive performances on VOCO7 with the linear classifier on the pretrained ResNet-50 compared to the previous record of 88.9% top-1 accuracy. For the COCO object detection, the APs for small object detection has been improved to 24.4% from the previous best AP's of 20.8%. These results show that the CLSA can more effectively leverage stronger augmentations than the previous eeff-superiiee methods on downstream tasks. We also conduct ablation study to show a naive application of stronger augmen- tations in the contrastive learning would degrade the performances.
several downstream tasks. Among them is a top-1 accuracy of 93.6%
RELATED WORK
Unsupervised and self-supervised learning methods have been widely studied to close the gap with supervised learning. These methods can be categorized into four different major aspects.
Instance Discrimination and Contrastive Learning Each image is considered as an individual class in an instance discrimination setting (Bojanowski & Joulin (2017): Dosovitskiy et al. (2013); Wu et al. (2018): : Chen et al.] (2020a); He et al. (2020). It can be further formulated as contrastive learning (adselll et al. (2006)). In particular, Wu et al. (2018) built a memory bank that stores pre-computed representations from which positive examples
retrieved given some queries. Fol- lowing this work, He et al. (2020) used a momentum update mechanism to maintain a long queue of negative examples for contrastive learning. Chen et al. (2020a) proposed rich family of data aug- mentations on cropped images which has significantly boosted the classification accuracy. However, these methods failed further improve the performance by naively applying stronger augmentations to minimize the contrastive loss, and this motivated the proposed work.
Generative Methods The generative methods typically adopt auto-encoders (Vincent et al. (2008);
Kingma & Welling (2013)). and adversarial learning (Donahue et al. (2016); Donahue & Simonyan (2019) to train an unsupervised representation. Usually, they focused on the pixel-wise information of images to distinguish images from different classes.
Self-supervised Clustering Data clustering (Asano et al. (011 Caron et al. 20188 2019 2020);
Yan et al. (2020)) can also be used to learn visual representations by assigning pseudo cluster la- bels to individual samples. DeepCluster (Caron al. (2018)) generalized k-means by alternating between assigning pseudo-labels and updating networks. Recently, the SWAV (Caron et al. (2020) proposed to learn cluster of prototypes as the negative examples for the contrastive learning- Combined with the multi-crops of training examples, the SWAV has achieved the state-of-the-art performance on ImageNet.
Pretext Tasks In addition to the contrastive learning, there exist many alternative methods using different pretext tasks (Agrawal et al. (2015); (1:5:000000 Qi et al. (2019); Doersch et al. (2015); Kim et al. (2018); Larsson al. 2016); Zhang al. (2019) train unsupervised deep networks. For example, Doersch et al. (2015) used the relative positions of two randomly sampled patches as the supervised signal. Agrawal al. 2015): Zhang al. (0099) Gidaris al. (2018) adopted various geometric
2

---

Under review as a conference paper at ICLR 2021
View
Representation
Projection
fe
ge
V
y'
z'
Input
T
Contrastive
Loss
fΦ
V
y
Z
Negative
queue
Pool
B
B
B
B
B
B
B
B
Figure I: Contrastive instance learning framework
transformations on images and used the transformation parameters to train deep networks. For more details about these works, please refer the survey by Jynn & Tian (2020).
THE PROPOSED METHOD
In this section, will first review the preliminary work the contrastive learning, and discuss their limitations on applying stronger augmentations to explore the novel patterns of representa- tions. Then we will present new distributional divergence loss between weakly and strongly augmented images to self-train the representations over a representation bank consisting of both negative samples and positive targets. After that, the algorithm and the implementation details will also be explained.
3.1 PRELIMINARY METHODS AND LIMITATIONS
Contrastive learning (Hadsell et al. (2066) is a popular self-supervised idea and made great success in recent years with the advance of computation and various image augmentations. In contrastive learning, each image is considered as an instance, so it's sss also known as instance learning.
Fig. Tillustrates general framework of contrastive learning methods. each image in batch B, apply two transformations 7 and T to obtain two different views V and V of the same instance x. Then they go through key encoder and query encoder respectively, followed with MI.P projection layers, resulting in two embedded representations and 2 to calculate the constrastive loss. The key factor of contrastive learning is the quality and the number of negative examples. To deal with those issues, various methods have been proposed, such as Memory Bank (Wu et al. (2018)). momentum encoder (He et (2020), and online learning with bigger batch (Chen et (2020a)).
Specifically, the contrastive loss is developed to maximize the agreement of representations of differ- views of the same instance while minimizing the agreement with other negative samples. Hence, the contrastive loss for the different views of the same instance can be defined in Eq.
exp(sim(2/ 21)/T)
Lc
Σ iEB
log
B
exp(sim(2/, + Ek
with the cosine similarity
sim 24) = ( ||>)||)
where Le is the contrastive loss, and z/ are the projected representations of the different aug- mentations of the same sample Z4, the summation is taken over the samples æ; the current batch B, and 7 is the temperature parameter set to 0.2. Also, the negative pool Q {zz k 1.... K shown in Fig. [ is a queue of size K storing the embedded features from the past batches in a FIFO fashion. This will keep the examples from the most recent batches in the queue while removing these obsoleted ones from it, which has been widely adopted in previous works (Chen et al. 2020b): He et al. 20200).
As illustrated in Fig. previous contrastive learning works use two transformations and T' to generate different views V and V", in which the transformations carefully designated. Thus,
3

---

Under review as a conference paper at ICLR 2021
the two views transformed aggressively so that they can still be viewed as the same instance. However, directly adopting stronger transformations (e.g., with larger rotation angles, more aggres- sive colorjittering and cutout) in contrastive learning fails to further improve the performance or even deteriorate it for downstream tasks, which is not surprising. Stronger transformations could that strongly
distort image structures and their perceptual patterns in the learned representation augmented samples from the same instance cannot be viewed as keeping the same instance iden- tity for training the underlying network. However, stronger augmentations can expose useful clues to the novel patterns that cannot be revealed from moderately augmented images. In supervised learning (Cubuk et al. 2018); [im al. (2019) Hataya al. (2019); Cubuk al. (2020)) data aug- mentation search have been widely studied and greatly boost the performance with the novel pattern exposed by strongly augmented images. The findings in RandAugment (Cubuk et al. (2020)) have verified that strongly augmented views can provide more clues even without an explicit augmen- tation policy. We believe learning the representations from these novel patterns will pave the last mile to close the gap with the fully supervised representations. Indeed, in semi-supervisce learning and supervised learning (Cubuk et af. (2020); QQ et al. (2019); Wang et al. (2019), ), more aggressive augmentations have been adopted and achieved extraordinary performances. For example, AET et al. (2019) has adopted the parameters of augmentations as supervised signal to self-supervise the training of networks. All of these findings have inspired to explore novel ways to utilize stronger transformations in self-supervise ing them in contrastive model (Chen al. (2020a)). All of these have inspired to explore novel ways to utilize stronger transformations in self-supervised learning.
learning while avoiding deteriorated performances by naively us-
Figure 2: Comparison of the strongly weakly augmented images. The left is the original image, the middle the weakly augmented image, and the right the strongly augmented one with over- contrastive details.
However, it is not easy. As shown in Fig. [ strongly augmented image may look perceptually different from the original counterpart. Consequently, the representation of a strongly augmented image can be far apart from that of the weakly augmented one. Thus, naively using strongly aug- mented images contrastive learning can be over-optimistic since the induced distortions could dramatically ruin their image structures. To this end, in Section 3.2 we instead proposed the Distri- butional Divergence Minimization (DDM) between weakly and strongly augmented images over a representation bank to avoid overfitting the representation of strongly augmented image with that of the corresponding positive target.
3.2 DISTRIBUTIONAL DIVERGENCE MINIMIZATION BETWEEN WEAKLY AND STRONGLY AUGMENTED IMAGES
Due to the aforementioned limitation of stronger augmentations in contrastive learning, a direct retrieval strongly augmented query infeasible to self-train deep networks. Fortunately, the distribution of relative similarities of weakly augmented image from the same instance over the representation bank can provide useful information to bridge the gap. As explained below, it does not only avoid directly placing the representation of strongly augmented image too closer to that of the positive target, but also allows it to explore the novel patterns of variations exposed by the strong augmentation.

---

Under review as a conference paper at ICLR 2021
Distributional Divergence
Representation Bank
Model
Strong
1
Supervise
Original
.II.I
Model
Weak
Figure 3: Diagram of distributional divergence minimization. Here the representation bank consists of stored features žą. in the negative pool and online features from the key encoder. They will be used calculate the conditional probability of current weakly and strongly augmented images.
Formally, as shown in Fig. for an original image z; and the representation 2; of the corresponding positive target, we applied weaker and a stronger augmentation to obtain two separate views V and V." and their embeddings 2/ (weak representation) and z (strong representation), Given pool (shown in Fig. [] of K negative samples { lk 1, K} accumulated from the past iterations, we obtain a conditional distribution
exp(sim(2/, 2x)/T)
p(ze
exp(sim(2/; 4T/T) Ek=o exp(sim TT)
which encodes the likelihood of the weaker query 2 being assigned to 2k- the similar way, we can define the likelihood p(z; (x() of the query being assigned the positive target 2;, well the likelihoods p(=12²" and p==x|=z) for the stronger query z²" Here the representation bank consists of the negative pool Q and the positive query targets 2; from the current batch B.
Then, we propose to minimize the following distributional divergence between the weak and the strong queries such that
K Σ
CD Σ
log(p/w/)( p(z(22)) log(p(z(12"))
(4)
By minimizing this divergence, we assume the learned representation z!' of the strongly augmented query should inherit the representation 2! of the weakly augmented one regarding not only its belief' of the query being assigned to the corresponding positive target 2;, but also its relations with the negative samples 2І in the representation bank through the conditional distribution p(=x 2())
This will prevent a direct overfitting of the strong query representation to the positive target as well improve the generalization of the learned representation with additional clues from the other examples in the representation pool. In a more general sense, this extends the idea of knowledge distillation (Hinton et al. (201)). However, to supervise the training of student model as in the knowledge distillation. Instead, we used the distribution of the likelihoods of weak query to supervise the retrieval of strong query from pool of representations.
did not the predicted labels by teacher model
3.3 IMPLEMENTATION DETAILS
Algorithm D gives the pseudo code to implement the proposed CLSA method. the following, we will discuss the details about the applied strong and weak augmentations for distributional diver- gence minimization.
Stronger Augmentations S As explored in the previous works (Cubuk et al. (2018); Wang et al. (2019): Qier al. (2019)). strong augmentations usually have two types: geometric and non- -oo-gecmetric augmentations. Specifically, considered 14 types of augmentations: ShearX/Y, TranslateX/Y,
5

---

Under review as a conference paper at ICLR 2021
Algorithm Pseudo code for the proposed CLSA
Input: fa. fe: the query and the key encoder networks; go. 9o: the MLP projection layers for the query and the key; Q: a queue of representations K negative samples; a: momentum decay for the key network; the temperature; T and T" weak augmentation; S: strong augmentation;
3: the balancing coefficient.
1: Initialize the network fø fo- 9 go and Q:
2: for i=1 to steps do
3: Fetch from the current batch B:
4: z" "= (f (S(())))) x' = (T" -o
5:
Calculate the contrastive loss Lo:
Eq. b see Eq.
6:
Calculate distributional divergence loss LD:
7: Update the query network fe and ge with loss L Lc Lp:
8: Update the key network fa and 9s with Ø aØ + (1 - a)8;
9: Update Q with the representation output from the key network;
10: end for
Output: The representation encoder fe.
Rotate, AutoContrast, Invert, Equalize. Solarize, Posterize, Contrast, Color, Brightness, Sharpness. The magnitude of each augmentation is significant enough to produce strong augmentations as possible. More details are shown in Table [ For example, the shear is drawn from range of [-0.3,0.3], which results in aggressively transformed images that can be hard to retrieve given counterpart target. In particular, to transform an image, we randomly select an augmentation from the above 14 types of transformations, and apply it to the image with a probability of 0.5. This process is repeated five times and that will strongly augment an image as the example shown in the right panel of Fig. 2
Table 1: Various augmentations applied in experiments to strongly augment training images.
Operation ShearX(Y)
TranslateX(Y)
Rotate
AutoContrast
Invert or
Equalize 0
Mag Range [-0.3,0.3]
[-0.3,0.3]
[-30,30]
0 or 1
Operation Solarize
Posterize [4.8]
Contrast
Color
Brightness [0.05,0.95]
Sharpeness
Mag Range [0,256]
[0.05,0995]
[0.05,0.95]
[0.05,0.95]
Weaker Augmentations T Weak augmentations are drawn by following most of existing con- trastive learning methods in literature (Chen al. (2020a b): Caron et al. (2020); He et al. (2020)): an image is first cropped from input image and resized to 224x224 pixels. Then random color jittering, Gaussian Blur, grayscale conversion, horizontal flip, thannel-wise color normalization are sequentially applied to generate weakly augmented images with an example shown in the middle off
Fig. 2
Technical Details Similar the previous works (He al. 20200; Chen et al. (2020a): Caron al. (2020)), we used the ResNet-50 (He et al. (2016)) as our encoder backbones Jo and Ja and a 2- layer MLP (2048-d hidden layer with the ReL.U) the projection head go and gø- The projected representation z first 1.2-normalized (Wu et al. (2018) before calculating the cosine similarity. The temperature 7 is to 0.2, with a momentum smoothing factor a of 0.999 and fixed balancing coefficient / of 1.0. We set the size K of the queue Q to 65536 to store the negative examples used compute the conditional distribution of weakly and strongly augmented queries and minimize their divergence.
4 EXPERIMENTS
4.1 TRAINING DETAILS
For the unsupervised pretraining on ImageNet with the CLSA, we used the SGD optimizer (Bottou 2010)) with an initial learning rate of 0.03, weight decay of 0.0001 and a momentum of 0.9. We used cosine scheduler (Loshchilov & Hutter (2016) to gradually decay the learning rate to 0. Usually, the batch size is set 256. When multiple GPU cluster servers are used, the batch size will

---

Under review as a conference paper at ICLR 2021
Table 2: Top-1 accuracy under the linear evaluation on ImageNet with the ResNet-50 backbone. The left table compared methods trained over 200 epochs, and the right table compared the methods with various numbers of epochs.
Method
Top
Method
Top
InstDisc (Wu et al. (2018)
54.0
BigBiGAN Donahue & Simonyan (2019)
56.6
LocalAge (Zhuang et
(2019)
58.8
SeLa-400epoochs Asano al. (2019)
61.5
MoCo (He et al. (2020)
60.8
PIRL-800epochs (Misra & Maaten 2020
63.6
SimCLR (Chen al.
(020a
61.9
CMC (Tian al, (20T9))
66.2
CPC v2 [Hénaff et al.
2019)
63.8
SimCIRR-80eppochs (Chen et al.
[2200a)
70.0
PCL. (Li et (2020)
65.9
MoCo v2-800epochs (Chen al.
(2020b)
71.1
MoCo v2 (Chen al.
(2020b))
67.5
InfoMin Aug-800epochn (Tan cl al. (2020)
73.0
InfoMin Aug Tann el al.
(2020)
70.1
BY0L-1000epochs (Grill et al. (0200)
74.3
SWAV (Caron et al. (2020))
72.7
SWAV-800epochs (Caron al. 2020
75.3
CLSA-Single
69.4
CLSA-Single-800epchhh
72.2
CLSA-Multi
73.3
CLSA-Mulli-800epochs
76.2
Supervised
76.5
Supervised
76.5
be multiplied by the same number of servers by convention. Typically, the experiment with single strong augmentation for each training image takes roughly 70 hours to finish on 8 V100 GPUs.
For the fine-tuning TeageNet, we trained linear classifier top of the frozen feature vector (2048-D) upon the pretrained ResNet-50 with CLSSA This linear layer is trained for 100 epochs, with learning rate of 10 without weight decay. We used the cosine learning rate decay and a batch size of 256.
For the transfer learning on the VOC dataset, we trained a linear classifier upon the pretrained Resnet-50 in the similar way for ImageNet we trained 100 epochs with the SGD optimizer and learning rate of 0.05, momentum of 0.9 and no weight decay. The batch size 256 without learning rate scheduler.
Finally, for object detection, we adopted the same protocol in (He et (2200) fine-tune the pretrained Resnet-50 backbone based on detectron2 (Wu et al. (2019) for the sake of fair and straight comparison with the other methods.
4.2 LINEAR CLASSIFICATION ON IMAGENET
For the linear evaluation ImageNet, we trained the CLSA in two settings. In the first setting named CinneS we used a single stronger augmentation (see Table T) that crops each training image smaller size of 96 96, which does not incur too much computing overhead in processing these smaller augmented images. In the second setting named CLSA-Multi, we adopted five differ- stronger augmentations that crop each image into various sizes: 224 224, 192 192, 160 x 160, 128 x 128, and 96 x 96. The DDM loss in Eq. 4 is the sum over these multiple stronger augmentations. The similar multi-crop strategy has been adopted in contrastive learning literature before. For example, the SWAV (Caron et al. (2020) reached the state-of-the-at top-1 accuracy by applying such multi-crop augmentations. To ensure fair comparison with the SWAV, we chose five stronger augmentations such that the self-training with CLSA- consumed the same computing time (i.e., 166 hours with cluster V100 GPUs for 200 epochs of pre-training with a batch size of 256).
As shown Table we compared the performance with the other unsupervised methods. All the experiments are based on pretrained ResNet-50 backbone that is fine-tuned with linear classifier. The left table showed the performance of different methods pretrained over 200 epochs, and the right table reported models pretrained over epochs.
First, under the same contrastive protocol, the CLSA-Single has higher 69.4% top-1 accuracy than both MoCo v2 (67.5%) and SimCLR (61.9%) with 200 epochs training. With multiple stronger augmentations, the CLSA-Multi outperforms the Statc-of-the- SWAV model using multi-crops of training images over 200 epochs 73.3% vs. 72.7%. Moreover, shown in the right table, the CLSAA

---

Under review as a conference paper at ICLR 2021
Single outperforms MoCo v2 and SimCLR with the same training epochs. It is also noteworthy that the CLSA-Multi achieves almost the same top-1 accuracy as that the fully supervised network (76.2% vs. 76.5%).
4.3
TRANSFER LEARNING RESULTS ON DOWNSTREAM TASKS
Table 3: Transfer learning results on various downstream tasks.
Classification VOCO7
Object Detection
VOC07+12
COCO
Measurement
Accuracy
AP50
AP APs
RotNet
Gidaris al.
2018
64.6
NPID+
(Wu ct al. (2018))
76.6
79.1
MoCo He ct al. (2020)
79.8
81.5
PIRL Misra & Maaten
2020
81.1
80.7
PCL Li et al. (2020))
84.0
BoWNet (Gidans et al.
(2020)
79.3
81.3
SimCLR (Chen et al.
2020a)
86.4
MoCov2
Chen et
2020b
87.1
82.5
42,0
20.8
SWAV
Caron et al.
2020))
88.9
82.6
42.1
19.7
CLSA
93.6
83.2
42.3
24.4
Supervised
87.5
81.3
40.8
20.1
We test the generalizability of the ResNet-50 pre-trained on ImageNet to several downstream tasks. Specifically, we focused on two tasks: cross-dataset image classification and object detection. The pre-trained ResNet-50 was frozon, and fine-tuned the linear classifier the VOCO7trainval (Ev- eringham et al.] (2010)) and tested it on the VOCO7test. For object detection, we evaluated the two datasets using the detectron2 (Wu al.] (2019) used in the previous methods He et al. (020); Chen et al. (2020a). On the VOC dataset (Everingham et al. (2010)) we trained the detection head with VOCO7+12 trainval dataset and tested on VOCO7 test dataset. On the COCO dataset (Lin et al. (2014)), we fine-tuned the network on the train2017 set with 118k images and evaluate the va12017. For the sake of fair comparison, the object detection tasks are completed by detectron2 (He et al. (2017)) based on the pretrained ResNet-50.
pre-trained network
As shown in Table 3 the performances on both tasks are much better than the supervised model trained on ImageNet. This suggests that the proposed method has better generalization ability in downstream tasks. The pre-trained network on ImageNet by the CL.SA outperformed the compared models after being fine-tuned on different datasets. Among them is a top-1 accuracy of 93.6% on the VOCO7 with the linear classifier on the pretrained ResNet-50 in comparison with the previous record of 88.9% top-1 accuracy by the SWAV. On the COCO dataset, the APs for small object detection has been significantly improved to 24.4% from the previously best APs of 20.8%. As well known, it is much challenging detect small objects on the COCO dataset. Thus, the better performance of the CLSA could be attributed to the ability of involving the stronger augmentations that result in many small objects pretrain the network.
4.4 ABLATION STUDY
Table 4: Ablation study of the CLSA on ImageNet with 200 epochs of pre- pre-training.
Model
Top-1 1 Accuracy
MoCo V2
67.5
MoCo V2 with Strong query
67.7
MoCo V2 with Strong query & Strong key
67.0
CLSA-Single with contrastive loss
68.0
CinAeS
69.4
In the ablation study shown in Table we studied the role of the proposed DDM loss the CLSA. First, we naively used the stronger augmentation applied in the CLSSA-Single the query and/or
8

---

Under review as a conference paper at ICLR 2021
the key in the MoCo V2. Both results (Strong query and Strong query & Strong key) showed the performance can not be improved or even degraded. Second, we replaced the DDM loss in the CLSA-Single with the contrastive loss, and we found it can only achieved top-1 accuracy of 68.0% compared to that of 69.4% with the DDM loss. Both studies showed that the proposed CLSA and its DDM loss help learn from stronger augmentations by avoiding the performance degeneration that would be incurred by the distortions of augmented images.
5 CONCLUSION
In this paper, we present CLSA, a novel method that can utilize the distributional divergence to learn the information from strongly augmented images. The proposed method outperforms the state-of- the-art methods on all the datasets and achieved almost same performance compared to supervised ImageNet network. Meanwhile, it outperforms the previous supervised and self-supervised methods on downstream tasks, which suggests CLSA learned more reliable and fine-grained features that can contribute the development of other areas.
REFERENCES
Pulkit Agrawal, Joao Carreira, and Jitendra Malik. Learning see by moving. In Proceedings of the IEEE international conference on computer vision, pp. 37-45, 2015.
Yuki Markus Asano, Christian Rupprecht, and Andrea Vedaldi. Self-labelling via simultancous clustering and representation learning, arXiv preprint arXiv:1911.05371, 2019.
Philip Bachman, Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. In Advances in Neural Information Processing Systems, pp.
15535-15545, 2019.
Piotr Bojanowski and Armand Joulin. Unsupervised learning by predicting noise. arXiv preprint arXiv:1004.05310, 2017.
Léon Bottou. scale machine learning with stochastic gradient descent. In Proceedings of COMPSTAT pp. 177-186. Springer, 2010.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsu- pervised learning of visual features. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 132-149, 2018.
Mathilde Caron, Piotr Bojanowski, Julien Mairal, and Armand Joulin. Unsupervised pre-training of image features on non-curated data. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2959-2968, 2019.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. arXiv preprint arXiv:2006.0988) 2020.
Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint 20Xiv:2002.05709 2020a.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003. 2020b.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc Le. Autoaugment: Learning augmentation policies from data. arXiv preprint arXiv:1805.09501 2018.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with reduced search space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 702-703, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248 255. leee, 2009.

---

Under review as a conference paper at ICLR 2021
Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. In Proceedings of the IEEE international conference on computer vision, pp- 1422-1430, 2015.
Jeff Donahue and Karen Simonyan. Large scale adversarial representation learning. In Advances in Neural Information Processing Systems, pp. 10542-10552, 2019.
Jeff Donahue, Philipp Krähenbühl, and Trevor Darrell. Adversarial feature learning. arXiv preprint 1605.09970 2016.
Alexey Dosovitskiy, Philipp Fischer, Jost Tobias Springenberg, Martin Riedmiller, and Thomas Brox. Discriminative unsupervised feature learning with exemplar convolutional neural networks. IEEE Iransactions pattern analysis and machine intelligence, 18(9):1734-178 2015.
Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88(2): 303-338, 2010.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. arXiv preprint arXiv: 1803.07728 2018
Spyros Gidaris, Andrei Bursuc, Nikos Komodakis, Patrick Pérez, and Matthieu Cord. Learning representations by predicting bags of visual words. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6928-6938. 2020.
Jean-Bace Cean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, al. Bootstrap your own latent: A new approach to self-supervised learning. arXiv preprint arXiv:2006.07733, 2020.
Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning an invariant mapping. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recogni- tion (CVPR '06), volume 2. pp. 1735-1742. IEEE, 2006.
Ryuichiro Hataya, Jan Zdenek, Kazuki Yoshizoe, and Hideki Nakayama. Faster autoaugment: Learning augmentation strategies using backpropagation. arXiv preprint arXiv:1911.06988 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pp. 2961-2969, 2017.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9729-9738, 2020.
Olivier J Hénaff, Aravind Srinivas, Jeffrey De Fauw, Ali Razavi, Carl Doersch, SM Eslami, and Aaron van den Oord. Data-efficient image recognition with contrastive predictive coding. arXiv preprint 1002002000 2019.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint arXiv:1503.0253: 2015.
R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint
2018.
Longlong Jing and Yingli Tian. Self-supervised visual feature learning with deep neural networks: A survey. IEEE Transactions Pattern Analysis and Machine Intelligence, 2020.
Dahun Kim, Donghyeon Cho, Donggeun Yoo, and In So Kweon. Learning image representations by completing damaged jigsaw puzzles. In 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 793-802. IEFE, 2018.
10

---

Under review as a conference paper at ICLR 2021
Diederik P Kingma and Max Welling. arXiv:13/2.611,, 2013.
Auto-encoding variational
bayes.
arXiv preprint
Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Learning representations for auto- matic colorization. In European conference on computer vision, pp. 577-593. Springer, 2016.
Junnan Li, Pan Zhou, Caiming Xiong. Richard Socher, and Steven CH Hoi. Prototypical contrastive learning of unsupervised representations. arXiv preprint arXiv:2005.04900 2020.
Sungbin Lim, Ildoo Kim, Taesup Kim, Chiheon Kim, and Sungwoong Kim. Fast autoaugment. In Advances in Neural Information Processing Systems, pp. 6665-6675, 2019.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv: 03988, 2016.
Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invariant representa- tions. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 6707-6717, 2020.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint arXiv:1807.03744 003748880088 2018.
Guo-Jun Qi, Liheng Zhang, and Xiao Wang. Learning generalized transformation equivariant rep-
resentations via autoencoding transformations. arXiv preprint arXiv: arXiv: /906.0828, 2019.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. arXiv preprint arXiv:1906.05849 0954499960089..... 2019.
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning. arXiv preprint arXiv:2005. /0243, 2020.
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th international conference on Machine learning, pp. 1096-1103, 2008.
Xiao Wang, Daisuke Kihara, Jicbo Luo, and Guo-Jun Qi. Enaet: Self-trained ensemble autoencoding transformations for semi-supervised learning. arXiv preprint
2019.
Yuxin Wu, Alexander Kirillov, Francisco Massa, Wen-Yen Lo, and Ross Girshick. Detectron2.
2019.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3733-3742, 2018.
Xueting Yan. Ishan Misra, Abhinav Gupta, Deepti Ghadiyaram, and Dhruv Mahajan. Clusterfit: Improving generalization visual representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6509-6518. 2020.
Liheng Zhang, Guo-Jun Qi. Liqiang Wang, and Jiebo Luo. Aet vs. aed: Unsupervised represen- tation learning by auto-encoding transformations rather than data. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2547-2555, 2019.
Chengxu Zhuang, Alex Lin Zhai, and Daniel Yamins. Local aggregation for unsupervised learning of visual embeddings. In Proceedings of the IEEE International Conference Computer Vision, pp. 6002-6012, 2019.
11


Reviews:
[{"review_id": "oeNKgjl6Kv6", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper focuses on designing more effective ways for contrastive learning. The author claims that stronger augmentations are beneficial for better representation learning. Different from directly applying the stronger augmentations to minimize the contrastive loss, the author proposes to minimize the distribution divergence between the weakly and strongly augmented images. The experimental evaluations are conducted on ImageNet classification and related downstream tasks, and the results are promising.\n\nClarity:\n1. The method is very simple and straightforward. My main concern is the experimental comparisons. As we all know, contrastive learning algorithms like MOCO and SimCLR benefits from longer training epochs a lot (for example, training with 800 epochs is much better than with 400 epochs). Thus I think the comparisons in Table 2 are not convincing. From algorithm 1, we can find that the equivalent batch size of the proposed CLSA method is two times as classical MOCO method. Thus I would prefer to check the results of CLSA at epochs 100 and 400 for fair comparisons.\n2. What is the value of the balancing coefficient? It would be nice if some ablation results are provided.", "extra_scores": {}, "extra_text": {}}, {"review_id": "SYTwselb9w", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a method to incorporate stronger augmentations into the visual representation contrastive learning framework. Specifically, three correlated views of an image are first generated by using two weak and one strong augmentation operations on the same image. Then, the networks are trained to maximize the agreement between the two weak views and also to minimize the distribution divergence between a weak view and the strong view. The method is evaluated on several visual tasks including classification,  transfer learning, and object detection, with the standard evaluation protocol for self-supervised learning, and the results are promising.\n\nPros:\n\n1. This paper is well-structured and easy-to-follow.\n2. The idea of utilizing strong augmentations for contrastive learning is interesting and novel to me, and the results are promising.\n3. The proposed framework seems general which might be easily incorporated into the existing contrastive learning frameworks.\n\nCons:\n\n1. The motivation about using stronger augmentations is not well justified. Specifically, the authors propose to use stronger augmentations based on two reasons: (1) stronger augmentations can expose some novel useful patterns; (2) the effectiveness of stronger augmentations is proved in the semi-supervised learning and supervised learning field. However, no related papers are provided to support the first point, while the papers (Cubuk et al. (2018)); Qi et al. (2019); Wang et al. (2019)) that are cited to support the second point do not explicitly make relevant conclusions. (Chen et al. (2020a)) even demonstrate that when training supervised models, stronger color augmentation hurts their performance. I would like to see a more comprehensive review of related works to clarify the motivation.\n\n2. In addition, some important ablation studies are missing in the experiment. E.g.,  how does the performance change as the magnitude or usage times of stronger augmentations changes?\n\n3. The proposed DDM loss seems general for different contrastive learning frameworks. I would like to see if it still works when applied to other frameworks, e.g., SimCLR, InfoMin?\n\nOverall, given the novelty and strong results of the proposed framework, I remain positive towards this paper. I will be happy to increase my rating if my concerns are addressed in the rebuttal period.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "EUgoVglDREZ", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\\\nThis work investigate the recent popular direction of unsupervised representation learning using contrastive loss between augmented images. Authors propose to minimize the divergence between the distributions of strongly augmented vs. weakly augmented images. The method reaches competitive performance in recognition and object detection.\n-----\n+Strengths\\\n+The main idea is well motivated: that strong augmentation reveal useful cues in visual representation learning but has not been successfully exploited in unsupervised learning.\\\n+The proposed solution is novel within contrastive learning to my best knowledge.\\\n+Results are extremely strong.\n-----\n-Concerns\\\n-The divergence between the two conditional distributions can be a moving target since they are trained jointly. It is not clear if this is will result in stable learning for the unsupervised setting, and what effect that may have on the performance and quality of the representations.\\\n-Evaluation only focus on final result and lacks analysis of the proposed method, especially when compared to recent paper of similar nature published in top conferences. For example, strong augmentation is a focus of this paper, but there are no ablation regarding the augmentations. Is the performance sensitive to the choice of strong augmentation?\\\n-The paper could also use some more theoretical analysis to address some of the weaknesses stated above.\n-----\nRecommendation\\\nI like the proposed idea. It is novel and interesting and seems to achieve good results. However the lack of both theoretical and empirical analysis beyond results on performance raises many questions. As a result I am on the fence but leaning towards accept.", "extra_scores": {}, "extra_text": {}}, {"review_id": "AMoGFJTWXDk", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThis paper proposes the better utilization of strong data augmentations for contrastive loss functions in unsupervised learning. In Moco set up, typically, weaker augmentations such as color jittering, cropping is applied to construct positive pairs from the same image. In this study, by proposing a modified objective, the authors leverage stronger data augmentations to construct more challenging positives and negatives pairs to improve the quality of the representations. The paper delivers a novel objective together with leveraging existing strong augmentations to improve downstream performance. The authors can find my questions/concerns listed below.\n\n1. The paper is overall well-written, however, it is disappointing to see many typos grammar mistakes throughout the paper. Some examples are in \"Thus we proposed the CLSA (Contrastive Learning with Stronger Augementations)\", \"to train an unsupervised representation\", \"The contrastive learning (Hadsell et al. (2006)) is a popular self-supervised idea\".\n\n2. In section 3.1, the authors mention that the keys in the memory bank is managed with first in first out method. Is it not supposed to be first in last out? I would like to see some clarification on this.\n\n3.  The numerator in Equation 3 should be z_i' vs. z_i not z_k.\n\n4. The authors claim that in He et al. an input image is resized and cropped to 224×224 pixels. It should be \"an image is first cropped from an input image and resized to 224x224 pixels.\"\n\n5. In the experiments section, the authors list other methods including MoCo, SimCLR, MoCo-v2, BYOL and compare to what they propose. As a baseline, it would be nice to directly use the stronger augmentations in MoCo-v2 objective and perform comparison to their method. Throughout the paper, the authors claim that strong augmentations hurt the learned representations due to distorted images. It would be meaningful to show this experimentally as well.\n\n6. The authors explain that they choose a strong transformation randomly from the given 14 transformations and repeat it 5 times to strongly augment an image. Is the sampling done without replacement? In other words, do the authors choose 5 unique transformations with the corresponding magnitude and apply those transformations to a single image?\n\n7. I like how the authors point the similarity of their objective to knowledge distillation. In this case, strong augmentations are assigned probability of being a positive pair from the positive pair constructed with weak augmentations. It helps to understand the full picture for the proposed method.\n\n8. Finally, I think the figure 3 is confusing rather than being helpful. Both weak and strong augmentations go to the memory bank and it looks like two distributions come out of nowhere in the figure. It would be more clear to point out that there is distribution of the representations from the strong augmentations and weak augmentations and they supervise the assignment for strong augmentations given predictions on the weak augmentations.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper improves MoCo-based contrastive learning frameworks by enabling stronger views via an additional divergence loss to the standard (weaker) views. Three reviewers suggested acceptance, and one did rejection. Positive reviewers found the proposed method is novel and shows promising empirical results. However, as pointed out by the negative reviewer, the paper should have clarified about computational overheads of the method compared to the baseline (MoCo) in several aspects, e.g., their effective batch sizes or training costs, for the readers’ better understanding. As the concern was not fully resolved during the discussion phase, AC is a bit toward for reject. AC thinks the paper would be stronger if the authors include more ablations (and the respective discussions) regarding this point, e.g., CLSA-multi (and -single) vs. MoCo-v2 under the same training time, both at early epochs (~200; as reported in the author response) and longer epochs (after convergence; ~1000 and even more). 
Decision: Reject