ReSSL: Relational Self-Supervised Learning with Weak Augmentation
Authors: Mingkai Zheng
Shan You
Fei Wang
Chen Qian
Changshui Zhang
Xiaogang Wang
Chang Xu
Venue: NeurIPS 2021 Poster (2021)
Keywords: Self-Supervised Learning

Abstract: Self-supervised Learning (SSL) including the mainstream contrastive learning has achieved great success in learning visual representations without data annotations. However, most of methods mainly focus on the instance level information (\ie, the different augmented images of the same instance should have the same feature or cluster into the same class), but there is a lack of attention on the relationships between different instances. In this paper, we introduced a novel SSL paradigm, which we term as relational self-supervised learning  (ReSSL) framework that learns representations by modeling the relationship between different instances. Specifically, our proposed method employs sharpened distribution of pairwise similarities among different instances as \textit{relation} metric, which is thus utilized to match the feature embeddings of different augmentations. Moreover, to boost the performance, we argue that weak augmentations matter to represent a more reliable relation, and leverage momentum strategy for practical efficiency. Experimental results show that our proposed ReSSL significantly outperforms the previous state-of-the-art algorithms in terms of both performance and training efficiency.

ReSSL: Relational Self-Supervised Learning with Weak Augmentation
Mingkai Zheng!.2 1.2
Shan You Fei Wang
Chen Qian
Changshui Zhang'
Xiaogang Wang Chang Xu²
School of Computer Science, Faculty of Engineering, The University of Sydney 2 Sense Time Research UUiversity of Science and Technology of China Department of Automation, Tsinghua University.
Institute for Artificial Intelligence, Tsinghua University (THUAD), Beijing National Research Center for Information Science and Technology (BNRist) Thhe Chinese University of Hong Kong
Abstract
Self-supervised Learning (SSL) including the mainstream contrastive learning has achieved great success in learning visual representations without data annotations. However, most of methods mainly focus on the instance level information (i.e., the different augmented images of the same instance should have the same feature cluster into the same class), but there is a lack of attention on the relationships between different instances. In this paper, we introduced a novel SSL paradigm, which we term relational self-supervised learning (ReSSL) framework that learns representations by modeling the relationship between different instances. Specifically, proposed method employs sharpened distribution of pairwise similarities among different instances as relation metric, which thus utilized to match the feature embeddings of different augmentations. Moreover, to boost the performance, we argue that weak augmentations matter to represent a more reliable relation, and leverage momentum strategy for practical efficiency. Experimental results show that our proposed ReSSL significantly outperforms the previous state- of-the-art algorithms in terms of both performance and training efficiency. Code is available at http://ithub..oo///yleeeee
Introduction
Recently, self-supervised learning (SSL) has shown its superiority and achieved promising results for unsupervised visual representation learning in computer vision tasks [40, 27, 32, 6, 9, 47, 23, 24]. The purpose typical supervuee learning algorithm is to learn general visual representations from large amount of data without human annotations, which can be transferred or leveraged in downstream tasks (e.g., classification, detection, and segmentation). Some previous works [5, 23] even have proven that good unsupervised pretraining can lead to better downstream performance than supervised pretraining.
Among various SSL algorithms, contrastive learning [47,45, 6] serves as state-of-the the-tarr framework. which mainly focuses on learning an invariant feature from different views. For example, instance discrimination is widely adopted pre-text task as in [6. 24, 47], which utilizes the noisy contrastive estimation (NCE) to encourage two augmented views of the same image to be pulled closer on the embedding space but pushes apart all the other images away. Deep Clustering [4, 48, 5] is an alternative pre-text task that forces different augmented views of the same instance be clustered into the same class. However, instance discrimination based methods will inevitably induce class
Corresponding author senstanasenset.tt ime.eetttee.c....mmm com
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

collision problem [1, 36, 10], where similar images should be pulled closer instead of being pushed away. Deep clustering based methods cooperated with traditional clustering algorithms to assign a label for each instance, which relaxed the constraint of instance discrimination, but most of these algorithms adopt strong assumption, i.e., the labels must induce an equipartition of the data, which might introduce some noise and hurt the learned representations.
In this paper, we introduce a novel Relational Self-Supervised Learning framework (ReSSL), which does not encourage explicitly to push away different instances, but uses relation as a manner to investigate the inter-instance relationships and highlight the intra-instance invariance. Concretely, we aim to maintain the consistency of pairwise similarities among different instances for two different augmentations. For example. if we have three instances x², y and where x', x² are two different augmentations of x, y and z are different samples. Then, if x is similar to y but different to 2, we wish x² can maintain such relationship and vice versa. In this way, the relation can modelled as similarity distribution between set of augmented images, and then use it as metric to align the same images with different augmentations, so that the relationship between different instances could be maintained across different views.
However, this simple manner induces unexpectedly horrible performance if we follow the same training recipe as other contrastive learning methods [6, 24J. We argue that construction of proper relation matters for ReSSL; aggressive data augmentations as in [6. 7, 41] are usually leveraged by default to generate diverse positive pairs that increase the difficulty of the pre-text task. However, this hurts the reliability of the target relation. Views generated by aggressive augmentations might cause the loss of semantic information, so the target relation might be noisy and not that reliable. In this way, we propose to leverage weaker augmentations to represent the relation, since much lesser disturbances provide more stable and meaningful relationships between different instances. Besides, we also sharpen the target distribution to emphasize the most important relationship and utilize the memory buffer with momentum-updact network to reduce the demand of large batch size for more efficiency. Experimental results on multiple benchmark datasets show the superiority of ReSSL in terms of both performance and efficiency. For example, with 200 epochs of pre-training, our ReSSL achieved 69.9% on ImageNet [14] linear evaluation protocol, which 2.4% higher than our baseline method (MoCoV2 [8]). When working with the Multi-Crov strategy (200 epochs), ReSSL achieved new state-of-the tftth-rr- 74.7% Top-1 accuracy, which 1.4% higher than CLSA-Mull [46].
Our contributions can be summarized as follows.
We proposed a novel SSL paradigm, which we term it as relational self-supervised learning (ReSSL). ReSSL maintains the relational consistency between the instances under different augmentations instead of explicitly pushing different instances away.
Our proposed weak augmentation and sharpening distribution strategy provide a stable and high quality target similarity distribution, which makes the framework works well.
ReSSL is simple and effective SSL framework since it replaces the widely adopted contrastive loss with our proposed relational consistency loss. It achieved sfate-of-the-art performance under the same training cost.
2 Related Work
Self-Supervised Learning. Early works in self-supervised learning methods rely all sorts of pretext to learn visual representations. For example, colorizing gray-scale images [50], image jigsaw puzzle [39], image super-resolution [34], image inpainting
[19], predicting a relative offset for a pair of patches [16], predicting the rotation angle [35], and image reconstruction [2, 22, 3, 17J. Although these methods have shown their effectiveness, they lack the generality of the learned representations.
Instance Discrimination. The recent contrastive learning methods [32, 40, 6, 24, 41, 38, 29, 27, 30] have made a lot of progress in the field of self-supervised learning. Most of the previous contrastive learning methods are based on the instance discrimination [47] task in which positive pairs are defined as different views of the same image, while negative pairs are formed by sampling views from different images. SimCLR [6, 7] shows that image augmentation (e.g. Grayscale, Random Resized Cropping, Color Jittering, and Gaussian Blur), nonlinear projection head and large batch size plays a critical role in contrastive learning. Since large batch size usually requires a lot of GPU memory, which is not very friendly to most of researchers. MoCo [24, 8] proposed momentum contrast

---

mechanism that forces the query encoder to learn the representation from a slowly progressing key encoder and maintain memory buffer store large number of negative samples. InfoMin [41] proposed a set of stronger augmentation that reduces the mutual information between views while keeping task-relevant information intact. AlignUniform [45] shows that alignment and uniformity are two critical properties of contrastive learning.
Deep Clustering. In contrast instance discrimination which treats every instance as a distinct class, deep clustering [4] adopts the traditional clustering method (e.g. KMenn))) to label each image iteratively. Eventually, similar samples will be clustered into the same class. Simply apply the KMeans algorithm might lead to degenerate solution where all data points are mapped to the same cluster: SeLa [48] solved this issue by adding the constraint that the labels must induce equipartition of the data and proposed a fast version of the Sinkhorn Sinkhorn-Knopp to achieve this. SwAV [5] further extended this idea and proposed a scalable online clustering framework. PCL [36] reveals the class collision problem and simply performed instance discrimination and unsupervised clustering simultaneously; although it gets the same linear classification accuracy with MoCoV2, has better performance on downstream tasks.
Contrastive Learning Without Negatives. Most previous contrastive learning methods prevent the model collapse in an explicit manner (e.g. push different instances away from each other or force different instances to be clustered into different groups.) BYOL [23] can learn a high- nigh-quality representation without negatives. Specifically, it trains an online network to predict the target network representation of the same image under a different augmented view and using an additional predictor network on top of the online encoder to avoiding the model collapse. SimSiam [9] shows that simple Siamese networks can learn meaningful representations even without the use of negative pairs, large batch size, and momentum encoders.
3 Methodology
In this section, we will first revisit the preliminary work on contrastive learning; then, we will introduce proposed relational sele-supervised learning framework. After that, the algorithm and the implementation details will also be explained.
3.1 Preliminaries on Self-supervised Learning
Given N unlabeled samples we randomly apply a composition of augmentation functions T(-) to obtain two different views x and x² through T(x, 01) and T(x, 02) where 0 is the random seed for T. Then, a convolutional neural network based encoder F()) is employed to extract the information from these samples, i.e., h F(T(x, f)). Finally, two-layer non-linear projection head g(-) is utilized to map h into embedding space, which can be written as: g().. SimCLR [6] and MoCo [24] style framework adopt the noise contrastive estimation (NCE) objective for discriminating different instances in the dataset. Suppose z and z² are the representations of two augmented views of x; and 2µ is a different instance. The NCE objective can be expressed by Eq. (1), where the similarity function sim(-) represents the dot product between Le normalized vectors sim (u, v) = u" w//l ull |Il v/l and r is the temperature parameter.
exp(sim(2', 2²2)
LNCE = log
(1)
exp(sim (2), 2/)/)) + Σ exp(sim(z) 2x)/T)
BYOL [23] and SimSiam [9] style framework add additional non-linear predictor head g(-) which further maps z to p. The model will minimize the negative cosine similarity (equivalent to minimize the L2 distance) between to p.
p²
Loos
Lmoc = ||'' -2
||_
2
Tricks like stop-grade stop-gradient and momentum teacher often applied to avoid model collapsing.
3.2 Relational Self-Supervised Learning
In classical self-supervised learning, different instances are be pushed away from each other, and augmented views of the same instance is expected to be of exactly the same features. However, both
3

---

No Gradient
Sharper
9(h1)
SoftMax
===F(p)))
z'
Weak
x
Augmentation
Append
Target
Exponential Moving Average
Memory Buffer
X
Contrastive
=== (xe)
z2
Augmentation
x²
(u(hF)
SoftMax
Backprop
Figure I: The overall framework of our proposed method. We adopt the student-teaches framework where the student is trained predict the representation of the teacher, and the teacher is updated with a "momentum update" (exponential moving average) of the student. The relationship consistency is achieve by align the conditional distribution for student and teacher model. Please see more details in our method part.
constrains are too restricted because of the existence of similar samples and the distorted semantic information if aggressive augmentation is adopted. In this way, do not encourage explicit negative instances (those to be pushed away) for each instance; instead, we leverage the pairwise similarities as a manner to explore their relationships. And we pull the features of two different augmentations in this sense of relation metric. As result, our method relaxes both (1) and (2), where different instances do not always need to be pushed away from each other; and augmented views of the same instance only need to share the similar but not exactly the same features.
Concretely, given image x in batch of samples, , two different augmented views be obtained by x = Tx, 11), x² = T(x, 02) and calculate the corresponds embedding zt = g(F(x))) )). z² == Then, we calculate the similarities between the instances of the first augmented images. Which can be measured by sim(x', 2;). A softmax layer can be adopted to process the calculated similarities, which then produces relationship distribution:
exp(sim(z 2T)/T))
p
(3)
k=1 exp(sim(z* (z) 2T./T(.)
where Ty is the temperature parameter. At the same time, we can calculate the relationship between x² and the i-th instance as sim(z?, z.). The resulting relationship distribution can be written as:
e exp(sim(z2, (2)//T)
p Σx
exp(sim 2x )T,/Ts,)
where Tg is a different temperature parameter. We propose to push the relational consistency between p! and p by minimizing the Biclbbcck-Libbler divergence, which can formulated as:
Lrelation Dr (p' l|p²) H('', pP) H(p').
(5)
Since the pd will only be used target, we only minimize H(p p²) in our implementation.
More efficiency with Momentum targets. However, the quality of the target similarity distribution p' is crucial, to make the similarity distribution reliable and stable, usually require large batch size which is very unfriendly to GPU memorics. To resolve this issue, we utilize a "momentum update" network in [24, 8]. and maintain a large memory buffer @ of K past samples {2v/k k 1..., }} (following the FIFO principle) for storing the feature embeddings from the past batches, which can then be used for simulating the large batch size relationship and providing stable similarity distribution.
5.+mF + F.. Se mg. (1 m)g..
where Fs and 9, denote the most latest encoder and head, respectively, so we name them as the student model with subscript On the other hand, Fi and 9, stand for ensembles of the past encoder and head, respectively, so we name them as the teacher model with a subscript t. m represents the momentum coefficient which controls how fast the teacher F, will be updated.
Sharper Distribution as Target. Note, the value of T has to smaller than т. since t, will be used to generate the target distribution. A smaller will result in "sharper" distribution which can be

---

interpreted as highlight the most similar feature for z'. Align p² with p' can be regarded as pulling z² towards the features that are similar with z'.
Weak Augmentation Strategy for Teacher. To further improve the quality and stability of the target distribution, we adopt weak augmentation strategy for the teacher model since the standard contrastive augmentation is too aggressive, which introduced too many disturbances and will mislead the student network. Please refer to more details in our empirical study.
Compare with SEED and CLSA. SEED [21] follows the standard Knowledge Distillation (KD) paradigm [26, 49, 18] where it aims to distill the knowledge from larger network into smaller architecture. The knowledge transfer happens in the same view but between different models. In our framework, are trying to maintain the relational consistency between different augmentations; the knowledge transfer happens between different views but in the same network. CLSA [46] also introduced the concept of using weak augmentation guide stronger augmentation. However, the "weak" augmentation in CLSA is equivalent to the "strong" augmentation in our method (We do not use any stronger augmentations such as [12, 13]). On the other hand, CLSA still adopts the InfoNCE loss (1) for instance discrimination, where our proposed method only utilized the relational consistency loss (5). Finally, CLSA requires least one additional sample during training, which will slow down the training speed.
Algorithm 1: Relational Self supervise Learning with Weak Augmentation (ReSSL)
Input: : x: batch of samples. Tww (-): Weak augmentation function. T-(-): Contrastive
augmentation function. Fr and Fs: the teacher and student backbone network. gt. and gs :
the non-linear projection head for teacher and student. Q: the memory buffer
while network not converge do
for i=1 to step do
Fetch x from current batch B
z' Se (T f)))); g² 02])];
p p'= Soft Q" /T
p² SoftMax22 Q²"" /
// Eq. (3)(4)
Calculate Lretation loss by CrossEntropy(p! .p)::
// Eq. (5)
Update F, and gs with loss Lrelation
Update F, and gr by F₂ + mFF + +(1++111 F F. gt + mgg + (1 m)). :// Eq. (6)
Update the memory buffer Q by z!
end end
Output: : The well trained model F,
4 Empirical Study
In this section, we will empirically study our proposed method on 4 popular self-supervised learning benchmarks and compare to previous state-of-the-art art algorithms (SimCLR [6]. BYOL [23], SimSiam [9], MoCoV2 [8]..
Small Dataset. CIFAR-10 and CIFAR-100 [31]. The CIFAR-10 dataset consists of 60000 32x32 colour images in 10 classes, with 6000 images per class. There are 50000 training images and 10000 test images. CIFAR- 100 is just like the CIFAR-10, 10, except it has 100 classes containing 600 images each. There are 500 training images and 100 testing images per class.
Medium Dataset. STL-10 [11] and Tiny ImageNet [33]. STL10 [1]] dataset is composed of 96x96 resolution images of 10 classes, 5K labeled training images, 8K validation images, and 100K unlabeled images. The Tiny ImageNet dataset is composed of 64x64 resolution images of 200 classes
with 100K training images and 10k validation images.
Implementation Details We adopt the ResNet18 [25] as our backbone network. Because most of our dataset contains low-resolution images, we replace the first 7x7 Conv of stride 2 with 3x3 Conv of stride 1 and remove the first max pooling operation for a small dataset. For data augmentations, we use the random resized crops (the lower bound of random crop ratio is set to 0.2), color distortion (strength=0.55 with probability of 0.8, and Gaussian blur with a probability of 0.5. The images from the small and medium datasets will be resized to 32x32 and 64x64 resolution respectively. Our method is based on MoCoV2 [8] in order to simulate the shuffle BN trick on one GPU, we simply divide a batch of data into different groups and then calculate BN statistics within each group. The
5

---

Table 1: Compare to other SSL algorithms on small and medium dataset.
Method
BackProp
EMA
CIFAR-10
CIFAR-I00
STL-10
Tiny ImageNet
Supervised
94.22
74.66
82.55
59.26
SimCLR [6]
2x
No
84,92
59.28
85.48
44.38
BYOL [23]
2x
Yes
85.82
57.75
87.45
42.70
SimSiam [9]
2x
No
88.51
60.00
87.47
37.04
MoCoV2 [8]
1x
Yes
86.18
59.51
85.88
43.36
ReSSL (Ours)
1x
Yes
90.20
63.79
88.25
46.60
momentum value and memory buffer size are set to 0.99/0.996 and 4096/16384 for small and medium datasets respectively. Moreover, The model is trained using SGD optimizer with momentum of 0.9 and weight decay of 5e- We linear warm up the learning rate for 5 epochs until it reaches 0.06 BatchSize 256, then switch to the cosine decay scheduler [37].
Evaluation Protocol. All the models will be trained for 200 epochs. For testing the representation quality, we evaluate the pre-trained model on the widely adopted linear evaluation protocol We will freeze the encoder parameters and train a linear classifier on top of the average pooling features for 100 epochs. To test the classifier, according to predicted output. We train the classifier with a learning rate of 30, no weight decay, and momentum of 0.9. The learning rate will be times 0.1 in 60 and 80 epochs. Note, for STL-10; the pretraining will be applied on both labeled and unlabeled images. During the linear evaluation, only the labeled 5K images will be used.
the center crop of the test set and computes accuracy
Result. As we can see the result in Table I, our proposed method outperforms the previous method on all four benchmarks. Reminder, most of the previous method requires twice back-propagattoon, which results much higher training cost than MoCoV2 and our method.
4.1 A Properly Sharpened Relation is A Better Target
The temperature parameter is very crucial in most contrastive learning algorithms. To verify the effective of T, and Te for our proposed method, we fixed т, 0.1 or 0.2, and sweep over T {0.01, 0.02. ..., 0.07). The result is shown in Table 2. For the optimal value is either 0.04 or 0.05 across all different datasets. As we can see, the performance increasing when we increase T from 0 to 0.04 and 0.05. After that, the performance will start to decrease. Note, - 0 correspond to the Top-1 or argmaz operation which produce one-hot distribution as the target. On the other hand, when + 0.1, the target will be much flatter distribution that cannot highlight the most similar features for students. Hence, 7 can not be either too small or too large, but has to be smaller than To (p' has to be sharper than p²) so that the target distribution can provide effective guidance to the student model.
Table 2: Effect of different 7; and Ts for ReSSL
Dataset
T5
T:
0.01
Tr = 0.02
T: = 0.03
T 0.04
т.
0.05
T
0.06
T: 0.07
CIFAR-10
0.1
89.35
89.74
90.09
90.04
90.20
90.18
88.67
CIFAR-10
0.2
89.52
89.67
89.24
89.50
89.22
89.40
89.50
CIFAR- 100
0.1
62.34
62.79
62.71
63.79
63.46
63.20
61.31
CIFAR-100 -00
0.2
60.37
60.05
60.24
60.09
59.09
59.12
59.76
STL-10
86.65
86.96
87.16
87.32
88.25
87.83
87,08
STL-10
0.2
85.17
86.12
85.01
85.67
85.21
85.51
85.28
Tiny ImageNet
0.1
45.20
45.40
46.30
46.60
45.08
45.24
44.18
Tiny ImageNet
0.2
43,28
42.98
43.58
42.12
42,70
42.76
42,60
For it is clearly to see that the result of 0.1 can always result much higher performance
than т3 0.2, which different to MoCoV2 where T, 0.2 is the optimal value. According to [43, 44, 15], a greater temperature will result in a larger angular margin in the hypersphere. Since MoCoV2 adopts instance discrimination as the pretext task, large temperature can enhance the compactness for the same instance and discrepancy for different instances. In contrast to instance discrimination, our method can be interpreted as pulling similar instances closer on the hypersphere; when the ground truth label is not available, the large angular margin might hurt the performance.

---

Query
10-NN
Figure 2: Visualization of the 10 nearest neighbour the query image. The top half is the result when we apply the weak augmentation. The bottom half is the case when the typical contrastive augmentation adopted. Note, we use the red square to highlight the images that has different ground truth label with the query image.
4.2 Weak Augmentation Makes Better Relation
As we have mentioned, the weaker augmentation strategy for the teacher model is the key to the success of our framework. Here, We implement the weak augmentation as a random resized crop (the random ratio simply adopt the same setting as in Table 2 and report the performance of the best setting. The result shown in Table 3, as we can see that when we use the weak augmentation for the teacher model, the performance is significantly boosted across all datasets. We believe that this phenomenon because relatively small disturbances in the teacher model can provide more accurate similarity guidance to the student model. To further verify this hypothesis, from STL-10 training set as the query images, and then find the 10 nearest neighbour based on the weak contrastive augmented query. We visualized the result in Figure 2,
(0.2, 1)) and a random horizontal flip. For temperature parameter, we
random sampled three image
Table 3: Effect of weak augmentation guided ReSSL
Teacher Aug Student Aug CIFAR-10
CIFAR-100
STL-10
Tiny ImageNet
Contrastive
Contrastive
86.17
57.60
84.71
40.38
Weak
Contrastive
90.20
63.79
88.25
46.60
4.3
More Experiments on Weak Augmentation
Since the weak augmentation for the teacher model is one of the crucial points in ReSSL, further analyze the effect of applying different augmentations on the teacher model. In this experiment, we simply set T, 0.04 and report the linear evaluation performance on the Tiny ImageNet dataset. The results are shown in Table 4. The first row is the bascline, where we simply resize all images to the same resolution (no extra augmentation is applied). Then, we applied random resized crops, random flip, color jitter, grayscale, gaussian blur, and various combinations. We empirically find that if we use no augmentation (e.g., no random resized crops) for the teacher model, the performance tends to degrade. This might result from that the gap of features between two views is way too smaller, which undermines the learning representations. However, too strong augmentations of teacher model will introduce too much noise and make the target distribution inaccurate (see Figure 2). Thus mildly weak augmentations are better option for the teacher, and random resized crops with random flip the combination with the highest performance as Table shows.
7

---

Table 4: Effect of different augmentation for teacher model (Tiny ImageNet)
Random Resized Crops
Random Flip
Color Jitter
GrayScale
Gaussian Blur
Acc
31.74
46.00
30.98
29.46
29.68
30.10
46.60
44.44
42.28
44.88
43.70
42.28
44.52
4.4 Dimension of the Relation
Since we also adopt the memory buffer as in MoCo [24], the buffer size will be equivalent to the dimension of the distribution p' p2. Thus, it will be one of the crucial points in framework. To verify the effect the memory buffer size, we simply keep r, 0.1 and т; = 0.04, then varying the memory buffer size from 256 to 32768. The result is shown in Table 5, memory buffer can significantly boost the performance. However, a further increase in the buffer size can only bring marginal improvement when the buffer is large enough.
see that larger
Table 5: Effect of different memory buffer size on small and medium dataset
Dataset (Small)
256
512
1024
4096
K 8192
16384
CIFAR-I0
89.37
89.53
89.83
90.04
90.15
90.35
CIFAR-100
61.17
62.47
63.20
63.79
63.84
64,06
Dataset (Medium)
256
1024
4096
8192
16384
32768
STL.-10
85.88
87.23
87.72
87.42
87.32
87.47
Tiny ImageNet 43.08
45.32
45.78
45.42
46.60
46.48
4.5 Visualization of Learned Representations
We also show the t-SNE [42] visualizations of the representations learned by our proposed method and MoCov2 on the training set of -IFAR-10. Our proposed relational consistency loss leads to better class separation than the contrastive loss.
(a) ReSSL w/o Sharpen (b) ReSSL w/o Weak Aug
(c) ReSSL Standard
(d) MoCo V2
Figure 3: t-SNE visualizations on CIFAR-10. Classes are indicated by colors.
Results on Large-scale Datasets
also performed our algorithm on the large-scale ImageNet- dataset [14]. In the experiments, we adopt a learning rate of 0.05 Batch Size/56,, memory buffer size of 130k, and 2-layer non-linear projection head with hidden dimension 4096 and output dimension 512. For T. and T,, 0.1.
simply adopt the best setting from Table 2 where T 0.04 and

---

Linear Evaluation. For the linear evaluation of ImageNet- Ik, we strictly follow the setting in SwAV [5]. The results are shown in Table As we can see clearly that ReSSL consistently outperforms previous methods on both Ix and 2x backprop setting. (Please noted that the student network will be passed in one 224x224 augmented view and two 224x224 augmented views for 1x backprob and 2x backprob setting respectively.)
Table 6: Top-1 accuracy under the linear evaluation on ImageNet with the ResNet-50 backbone. The table compares the methods over 200 epochs of pretraining.
Method
Arch R50
Backprop
EMA
Batch Size
Param
Epochs 120
Top-1
Supervised
1x
No
256
24
76.5
Ix Backprop Methods
InstDise [47]
R50
IX
No
256
24
200
58.5
LocalAgg [52]
R50
Ix
No
128
24
200
58.8
MoCo v2 [8]
R50
1x
256
200
67.5
MoCHi [30]
R50
Ix
Yes
512
24
200)
68.0
CPC v2 [32]
R50
Ix
No
512
24
200
63.8
PCL v2 [36]
R50
1x
Yes
256
200
67.6
AdCo [28]
R50
200
68.6
ReSSI (Ours)
R50
Yes
256
24
200
69.9
2x Backprop Methods
CLSA-Single [46]
R50
2x
Yes
256
200
69.4
SimCLR [6]
R50
200
66.8
SwAV
R50
2x
No
4096
24
200
69.1
SimSiam [23]
R50
2x
No
256
24
200
70,0
BYOL [23]
R50
2x
Yes
4096
24
200
70.6
WCL [S1]
R50
2x
No
4096
24
200
70.3
ReSSL (Ours)
R50
2x
Yes
256
24
200
71.4
Working with Multi-Crop Strategy. We also performed ReSSL with Multi-Crop strategy. The result shown below Table 7. Specifically, the result of 4 crops trained with the resolution of 224 x 224, 160 x 160, 128 128, 96 x 96. For the result of 5 crops, we add an additional 192 x 192 image which exactly the same with AdCo [28]. we can see, our proposed ReSSL. is significantly better than previous state-of-He state-of-thh-arr methods.
Table 7: Working with Multi-Crop Strategy (Linear Evaluation on ImageNet)
Method
Arch
EMA
Batch Size
Param
Epochs
Top-I
SwAV [5]
R50
No
256
24
200
72.7
AdCo [28]
R50
No
256
24
200
73.2
CLSA-Multi [46]
R50
Yes
256
24
200
73.3
ReSSL (4 crops)
R50
Yes
256
24
200
73.8
ReSSL (5 crops)
R50
Yes
256
24
200
74.7
Working with Smaller Architecture. We also applied proposed method on the smaller architec- ture (ResNet-18)) The result is shown Table Following the same training recipe of the ResNet-50 in above, our proposed method has higher performance than SEED [21] without larger pretrained teacher network.
Table 8: Experiments on ResNet- 18 (Linear Evaluation on ImageNet)
Method
Epochs
Student
Teacher
Acc
MoCo
200
ResNet-18
EMA
52.2
SEED
200
ResNet-18
ResNet-50 (MoCoV2)
57.6
ReSSL (Ix backprop)
200
ResNet-18
EMA
58.1
Low-shot Classification. We further evaluate the quality of the learned representations by trans- ferring them to other datasets. Following [36], perform linear classification the PASCAL

---

VOC2007 dataset [20]. Specifically, we resize all images to 256 pixels along the shorter side and taking a 224 x 224 center crop. Then, we train linear SVM on top of corresponding global average pooled final representations. To study the transferability of the representations in few-shot scenarios, we vary the number of labeled examples K and report the mAP. Table 9 shows the comparison between our method with previous works. We report the average performance over 5 runs (except for k=full). It's clearly to see that our proposed method is consistently outperform MoCo v2 and PCL v2 across all different K.
Table 9: Transfer learning on low-shot image classification
Method
Epochs
ImageNet
K=16
K==2
K=64
Full
Random
10.10
11.34
11.96
12.42
Supervised
90
76.1
82.26
84.00
85.13
87.27
MoCo V2 [8]
200
67.5
76.14
79.16
81.52
84.60
PCL V2 [36]
200
67.5
78.34
80.72
82.67
85.43
ReSSL (Ix backprob)
200
69.9
79.17
81.96
83.81
86.31
Semi-Supervised Learning. Next, we evaluate the performance obtained when fine-tuning the model representation using small subset of labeled data. In this experiments, we adopt our 5 crops pre-trained model. The result is shown in Table 10. Notably, with just 200 epochs of pre-training, ReSSL. outperforms previous methods.
Table 10: Semi-supervised Learning
Method
Epochs
Linear Eval
1% Labels
10% Labels
SimCLR [6]
1000
69.3
48.3
65.6
BYOL [23]
1000
74.3
53.2
68.6
SwAV [5]
800
75.3
53.9
70.2
ReSSL (5 crops)
200
74.7
57.9
70.4
6 Conclusion
In this work, we propose relational self-supervised learning (ReSSL), new paradigm for unsuper- vised visual representation lcarning framework that maintains the relational consistency between instances under different augmentations. Our proposed RcSSL relaxes the typical constraints in contrastive learning where different instances do not always need to be pushed away on the embedding space, and the augmented views do not need to share exactly the same feature. An extensive empirical study shows the effect of each component in our framework. The experiments on large-scaled datasets demonstrate the efficiency and staeeoffthe--ne performance for unsupervised representation learning.
Broader Impact
This work provides technical advancement in the field of unsupervised visual representation learning. An immediate application of this work is to give a pre-trained model for the tasks where the data annotation is very hard to collect (e.g. medical images and fine-grained images.) Moreover, the most significant advantage of ReSSL is that we do not need to train the model for a long time as the previous method (generally 800 or 1000 epochs), which will cause lot of carbon dioxide emissions. We believe ReSSL is more environment-frieen -fienndyy method since it can achieve competitive performance with much lesser training costs.
Acknowledgment
This work funded by the National Key Research and Development Program of China (No. 201 A018AAA0107011 and the NSFC 61 61876095. Chang Xu was supported in part by the Australian Research Council under Projects DE1 80101438 and DP210101859, Shan You supported by Beijing Postdoctoral Research Foundation.
10

---

References
[1] S. Arora, Hrishikesh Khandeparkar, M. Khodak, Orestis Plevrakis, and Nikunj Saunshi. A theoretical analysis of contrastive unsupervised representation learning. ArXiv, abs/1902.09229, 2019. 2
[2] Pierre Baldi. Autoencoders, unsupervised learning and deep architectures. In Proceedings of the 2011 I International Conference on Unsupervised and Transfer Learning Workshop- Volume 27, UTLW"11, page
37-50. .org. 2011. 2
[3] A. Brock, J. Donahue, and K. Simonyan. Large scale gan training for high fidelity natural image synthesis.
ArXiv, abs/1 809. 1096, 2019. 2
[4] Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustcring for unsupervised learning of visual features. In European Conference on Computer Vision, 2018, 1,3
[5] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsuper-
vised learning of visual features by contrasting cluster assignments. 2020. 1, 3, 9, 10
[6] Ting Chen, Simon Komblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002.05700 2020. 1, 2, 3, 6, 9, 10
Ting Chen, Simon Kornhlith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self- supervised models are strong supersupee learners. arXiv preprint urXiv:2006. 10029, 2020. 2
[8] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.0422 2020. 2. 4, 5, 6, 9, 10
[9] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recoggiition, pages 15750-15758, 2021. 1, 5,
[10] Ching- Yao Chuang. Joshua Robinson, Yen-Chen Lin, Antonio Torralba, and Stefanie Jegelka. Debiased contrastive learning. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 8765-87777 8775. Curran Associates, Inc., 2020. 2
[11] Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. volume 15 of Proceedings of Machine Learning Research, pages 215-223, Fort Lauderdale, FL, USA, 11-13 Apr 2011. JMLR Workshop and Conference Proceedings. 5
[12] Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoe V Le. Autoaugment: Learning
augmentation policies from data. arXiv preprint arXiv: 1805 09500, 2018. 5
[13] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with reduced scarch space. In Proceedings of the IEEE/CVF Conference on Computer
Vision and Pottern Recognition Workshops, pages 202-703, 2020. 5
[14] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR09, 2009. 2,8
[15] Jiankang Deng, J. Guo, and S. 7afeiriou. Areface: Additive angular margin loss for deep face recognition. 2019 JEFF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4685-4694, 2019. 6
[16] Carl Doersch, Abhinav Gupta, and Alexei A. Efros. Unsupervised visual representation learning by context prediction. International Conference on Computer Vision (ICCV), 2015.
[17] J. Donahuc and K. Simonyan. Large scale adversarial representation learning. In NewrIPS, 2019. 2
[18] Shangehen Du, Shan You, Xiaojic Li, Jianlong Wu, Fei Wang, Chen Qian, and Changshui Zhang, Agree to disagree: Adaptive ensemble knowledge distillation in gradient space. Advances in Neural Information Processing Systems, 33, 2020.
[19] Omar ElHarrouss, Noor Almaadeed, S. Al-Mándeed, and Y. Akhari. Image inpainting: A review. Neural Processing Letters, 51: 51:2207-2008, 2019. 2
[20] Mark Everingham, Lue Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88(2)3303-338
2010. 10
[21] Zhiyuan Fang. Jianfeng Wang, Lijuan Wang. Lei Zhang, Yezhou Yang, and Zicheng Liu. (SEED): Self- supervised distillation for visual representation. International Conference on Learning Representations, 2021. 5, 9
[22] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Farde, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Z. Ghahramani, M. Welling. C. N. Lawrence, and K. Q. Weinberger, editors. Advances in Neural Information Processing Systems, vol- ume 27, pages 2672 2680. Curran Associates, Inc., 2014.
Cortes.

---

[23] Jean-Basties Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond. Elena Buchatskaya,
Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Cheshlaghi Azar, al. Bootstrap your own latent: A new approach to self-supervised learning. arXiv preprint arXiv:2006.07733, 2020. 1, 3.
5, 6, 9, 10
[24] Kaiming He, Hanqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. arXiv preprint /rXiv: 2019. 1, 2, 3, 4, 8
[25] Kaiming He, Xiangyu Zhang, Shaoqing Rea, and Jian Sun. Deep residual learning for image recognition. arXiv preprint arXiv:/512.0330
2015. 5
[26] Geoffrey Himton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv
preprint Xiv: 02.33. 2015. 5
[27] R Devon Hjelm, Alex Fedorov, Samuel Marchilnn,,, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv: 066700 2018. 1.2
(28) Qianjiang Hu, Xiao Wang. Wei Hu, and Guo-Jun Qi. Adco: Adversarial contrast for efficient learning of unsupervised representations from self-trained negative adversaries. arXiv preprint arXiv:2011.044:
2020. 9
[29] Lang Huang, Chao Zhang, and Hongyang Zhang. Self-adaptive training: Bridging the supervised and self-supervised learning. arXiv preprint 2XIi:2201.007322 2021. 2
[30] Yannis Kalantidis, Mert Bulent Sariyildiz, Noe Pion, Philippe Weinzaepfel, and Diane Larlus. Hard negative mixing for contrastive learning. In Neural Information Processing Systems (NeurIPS), 2020. 2, 9
[31] A. Krizhevsky. Learning multiple layers of features from tiny images. 2009. 5
[32] Cheng-I Lai. Contrastive predictive coding based feature for automatic speaker verification. arXiv preprint urXiv.190.01575. 2019. 1, 2, 9
[33] Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7)3, 2015. 5
[34] C. Ledig, I.. Theis, Ferene Huszár, J. Cahallero, Andrew Aitken, Alykhan Tejani, Totz, Zehan Wang. and W. Shi. Photo-realistic single image super-resolution using a generative adversarial network. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 105-114, 2017. 2
[35] Hankook Lee. Sung Hwang, and Jinwoo Shin. Self-supervised label augmentation via input transforma-
tions. In International Conference on Machine Learning, pages 9714-5724 PMLR, 2020. 2
[36] Junnan Li, Pan Zhou, Caiming Xiong, and Steven Hoi. Prototypical contrastive learning of unsupervised representations. In International Conference on Learning Representations, 2021. 2, 3, 9, 10
[37] Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint vrXiv:1608, 160800930 2016. 6
[38] Ishan Misra and Laurens van der Maaten. Self-supervised learning of inverxt-nnvariant representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR). June 2020. 2
[39] M. Noroozi and P. Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In ECCV, 2016. 2
[40] Yonglong Tian, Dilip Krishnan, and Phillip Isola. TrXiv:I906 05849, 2019. 1.2
Contrastive multiview coding. arXiv preprint
[41] Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? arXiv preprint arXiv:2005. 10243, 2020. 2, 3
[42] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008. 8
[43] Feng Wang, Xiang Xiang, Jian Cheng, and A. Yuille. Normface: 1.2 hypersphere embedding for face verification. Proceedings of the 25th ACM international conference Multimedia, 2017. 6
[44] H. Wang, Yitong Wang, Zhou, Xing Ji, Zhifeng Li. Dihong Gong, Jingchao Zhou, and Wenyu Liu. Cosface: Large margin cosine loss for deep face recognition. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5265-5274, 2018. 6
[45] Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. arXiv preprint arXiv:2005. 10242, 2020. 1,3
[46] Xiao Wang and Guo-Jun Qi. Contrastive learning with stronger augmentations. arXiv preprint
arXiv:2104.07713, 2021. 5.9
[47] Zhirong Wu, Yuanjun Xiong, X Yu Stella, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018. 1,2.9
12

---

[48] Asano YM.. Rupprecht C.. and Vedaldi A. Self-labelling via simultaneous clustering and representation learning. In International Conference Learning Representations, 2020. 1,3
[49] Shan You, Chang Xu, Chao Xu, and Dacheng Tao. Learning from multiple teacher networks. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1285 1294, 2017.
[50]
Richard Zhang. Phillip Isola, and Alexei A. Efros. Colorful image colorization. In ECCV, 2016. 2
[51] Mingkai Zheng, Fei Wang. Shan You, Chen Qian, Changshui Zhang, Xiaogang Wang, and Chang Xu. Weakly supervised contrastive learning. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 1042-10051,, October 2021. 9
[52] Chengxu Zhuang. Alex Lin Zhai, and Daniel Yamins. Local aggregation for unsupervised learning of visual embeddings. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 6002-6012, 2019.
13


Reviews:
[{"review_id": "mnzAOr5N8YU", "reviewer": "Reviewer_Wh5r", "summary": "This paper presents a loss function modification on top of the MoCoV2 framework and seem to provide consistent gain over the original MoCoV2 framework. To make it work better, the author also had to change how images are augmented during training.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Existing SSL (self-supervised learning) works usually use contrastive / consistency loss (either way it’s instance as class). One long-standing problem is the so-called collision problem where the model, during SSL, learns different instances as different classes while they may come from the same semantic class.\n\nSuppose there is an image x, and x1, x2 are two augmented version of it.\nThe authors proposed to not directly use contrastive loss on the features of two augmented versions of the same image. Instead, they propose to first draw a similarity distribution for x1/x2 over K number of recent samples. Then use KL-divergence to align these two distributions. In various experiments, this modification to the contrastive loss seems to provide benefits to the original MoCo V2 framework.\n\nQuestion 1.\nWill the proposed method work for fine grained dataset such as CelebA (faces) or Birdsnap (different birds)? I’m interested to know because the foundation of the proposed loss function is: \nGiven three samples, x,y,z, if x1 is similar to y but different to z, x2 should maintain such relationships.\n\nSuppose x, y, z are three samples from three distinct dog classes. if the x-species dog is a mix breed from y and z. Then say x1 and x2 are two different crop augmentation of the x-dog. Then depending on where the crop happens (e.g., ears, paws, tail), x1 could indeed be similar to y and x2 could be actually more similar to z.\n\nQuestion 2.\nThe SWaV number (69.1) in Table 5 seems low and I understand that the authors copied it from citation_10. But when I look at SWaV itself (https://arxiv.org/pdf/2006.09882.pdf), they report in their in Table 3with ResNet50 as 72.7 with 200 epochs. May I ask why is there such a gap?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Adequately addressed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5 hours"}, "extra_text": {}}, {"review_id": "kQoQ5tZ_l0k", "reviewer": "Reviewer_5Qa3", "summary": "This paper proposes a self-supervised learning technique, ReSSL, which makes (1) similar samples more similar and (2) dissimilar ones be more dissimilar. Since the proposed method does not assume positive and negative samples strictly, it does not suffer from such a class collision problem. Extensive experiments demonstrate the effectiveness of the proposed method from small-scale (e.g., CIFAR) to large-scale (i.e., ImageNet) datasets.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n- The proposed idea is simple and novel.\n- The proposed method demonstrates its effectiveness in both small- and large-scale benchmarks.\n\nWeaknesses & Concerns & Suggestions\n1. Few transfer learning experiments.\n   - This paper only considers the PASCAL benchmark with linear evaluation for transfer learning scenarios. However, self-supervised learning aims to transfer the learned representations or whole network parameters into various downstream tasks. This is why many SSL methods are often evaluated in various transfer learning scenarios. Could you provide more transfer learning experiments, for example, linear evaluation and fine-tuning in fine-grained classification tasks [2,3], semi-supervised learning [2,3,5], and object detection/segmentation [1,4]?\n2. Small number of training epochs\n   - The convergence rate and final accuracy highly depend on methods. For example, as reported in [4, Appendix D and Table 4], SimCLR's accuracy is more rapidly increasing than SimSiam in CIFAR-10, but SimCLR's final accuracy (91.1% at epoch 800) is lower than SimSimam (91.8%). Therefore, the reported results in this paper only show that ReSSL has a higher convergence rate (i.e., higher efficiency) than others. I think longer training experiments would further verify ReSSL's effectiveness.\n4. Ablation study\n   - Student's augmentations\n     - I agree that the weak augmentation is important for the teacher network, as shown in Table 3 and Table 9.\n     - I wonder why this paper does not conduct a similar empirical study for student's augmentations. Like CLSA, Can ReSSL use stronger augmentations?\n   - Although ReSSL achieves higher accuracy than other methods in various settings, it is hard to fully understand why the method succeed. I think more qualitative and quantitative analyses would provide a better fundamental understanding.\n     - More t-SNE comparisons with negative-free approaches (e.g., SimSiam) and clustering-based ones (e.g., SwAV) on other datasets (e.g., STL10)\n     - Could you provide the changes of the k-NN classification accuracy during pretraining and compare it with other methods, especially clustering-based one? In my opinion, the proposed method is somewhat similar to clustering. Hence, the method might have an advantage when using the k-NN accuracy instead of linear evaluation.\n5. Is this method applicable to other domains?\n   - SimCLR and MoCo have shown their effectiveness in non-vision domains such as tabular data and natural language. If ReSSL is also widely applicable, it will strengthen this paper's contribution.\n6. Some important results are provided in only the supplementary material, not the main manuscript.\n   - Multi-crop experiments (Appendix E)\n   - Ablation study on the choice of data augmentations (Appendix C)\n   - I think this paper does not use its space compactly; for example, Table 5 and Table 6 can be combined. Hence, it would strengthen this paper if the main manuscript contains all the experimental results.\n7. Some concurrent works\n   - There are concurrent works [6,7,8] that share similar concepts of the proposed method. It might be better if comparisons between them and ReSSL are provided. (since they are concurrent works, I think the information can be placed in the supplementary material).\n\n[1] He et al., Momentum Contrast for Unsupervised Visual Representation Learning, CVPR 2020 \\\n[2] Chen et al., A Simple Framework for Contrastive Learning of Visual Representations, ICML 2020 \\\n[3] Grill et al., Bootstrap Your Own Latent: A New Approach to Self-supervised Learning, 2020 \\\n[4] Chen & He, Exploring Simple Siamese Representation Learning, 2020 \\\n[5] Caron et al., Unsupervised Learning of Visual Features by Contrasting Cluster Assignments, NeurIPS 2020 \\\n[6] Dwibedi et al., With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representations, 2021 \\\n[7] Wang et al., Solving Inefficiency of Self-supervised Representation Learning, 2021 \\\n[8] Caron et al., Emerging Properties in Self-Supervised Vision Transformers, 2021\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This paper does not address limitations and potential negative societal impacts.\n\n**A suggestion for negative societal impact.**\nThe line of self-supervised learning works, including this paper, requires a large neural network, a huge training dataset, and a long training time. Hence, it could consume a lot of energy for training, i.e., result in some environmental issues. To solve the problems, we may need an energy-efficient approach. For example, curriculum learning would be one option for improving the efficiency of self-supervised learning.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "OZ6tgiKO6j2", "reviewer": "Reviewer_HMe7", "summary": "This work focuses on self-supervised learning for image classification. Previous work assumes that images and the corresponding augmentation should have similar features (or cluster into the same class) as the objective function to train the model. In contrast, this paper proposes to make the contrastive loss (positive and negative) of two different weak augmentations of a sample to be consistency. Moreover, knowledge distillation is also adopted to sharpen the target distribution for better capturing the most important relationship. Experimental results show that the proposed approach outperforms state-of-the-art approaches, e.g, SimCLR, BYOL, SimSiam, MoCoV2, with less computation requirement.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Generally speaking, this paper is well-motivated and easy to follow. The results show promising improvement on the state-of-the-art methods. Several minor issues are listed below.\n1) Some abbreviations require the full name to make them friendly for new readers, e.g., EMA.\n2) Some experimental results lack of detailed descriptions. For example, Figure 3 shows the t-sne of MoCo-v2 and the proposed ReSSL. However, it is difficult to tell which one is better. The brown ones still locate cross the green ones. Most importantly, the visualization does not show the reason why the proposed approach is better. It is encouraged to show the ablation visualization, i.e., Full model, Full model without sharpening, full model using strong augmentation.\n3) It is desirable to release the codes for reproducing the results.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No. I cannot find the statements of limitations in the supplementary materials, even though this paper states \"Please refers to our supplementary materials\" in line 402.", "review_text": "", "extra_scores": {"ethical_concerns": "No.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5 hours"}, "extra_text": {}}, {"review_id": "JdKIWawjj9B", "reviewer": "Reviewer_k9Fo", "summary": "This paper proposes a new approach to self-supervised learning of representations, wherein similar to previous works, the objective function relates to two different views–as in prior works, augmentations here–of the same data point.  However, rather than encouraging the respective two embeddings to be similar, the approach here encourages the distribution of similarities to other data points of the two embeddings to be similar, along with a momentum update approach.  On a thorough set of empirical evaluations, the proposed approach performs strongly.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, the proposed approach seems like an intuitive modification to recent self-supervised learning approaches, which performs well on a thorough range of empirical evaluations relative to recently proposed approaches, and therefore should be interesting to attendees and readers of this conference.  Some specific points:\n- The proposed approach has a number of \"tricks\" (like the momentum approach, distribution sharpening, particular type of weak augmentation), however these appear to be thoroughly ablated in the empirical results section\n- The idea of applying some objective over the matrix of pairwise distances between a data point and others in the dataset is not itself new... ideally this paper would have compared to e.g. kernel-based approaches and mapped connections there\n- The writing is somewhat unclear and has grammatical issues throughout, especially making some of the upfront exposition parts of the paper a little hard to parse.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a more nuanced take on contrastive self-supervised learning (SSL). Rather than frame the objective of SSL as a binary similar/not similar target, the authors propose to use multiple sampled augmentations as a similarity distribution. The goal of SSL becomes to embed the training data in a way that agrees with the distribution. The result can be viewed either as a relaxed form of contrastive SSL, in which multiple views of the same instance only need to be similar, not the same, or---as one reviewer pointed out---a kernel-based approach that operates on the similarities among instances.

While the idea is intuitively appealing, the authors show that simply plugging it into existing SSL setups gives very poor performance, so the authors provide a recipe for this alternative approach. In particular, only weak augmentations (as opposed to changing the instances heavily) leads to better results, in contrast with existing contrastive learning approaches where the augmentation must be strong to make the pretext task challenging. Experimental results show that the learned representations lead to significant gains on object classification tasks.

The reviewers generally agree that this paper is well written and makes a significant contribution. During the discussion phase, the authors introduced several follow up experiments, which the reviewers encourage them to include in the final version.
Decision: Accept (Poster)