RSA: Reducing Semantic Shift from Aggressive Augmentations for Self-supervised Learning
Authors: Yingbin Bai
Erkun Yang
Zhaoqing Wang
Yuxuan Du
Bo Han
Cheng Deng
Dadong Wang
Tongliang Liu
Venue: NeurIPS 2022 Accept (2022)

Abstract: Most recent self-supervised learning methods learn visual representation by contrasting different augmented views of images. Compared with supervised learning, more aggressive augmentations have been introduced to further improve the diversity of training pairs. However, aggressive augmentations may distort images' structures leading to a severe semantic shift problem that augmented views of the same image may not share the same semantics, thus degrading the transfer performance. To address this problem, we propose a new SSL paradigm, which counteracts the impact of semantic shift by balancing the role of weak and aggressively augmented pairs. Specifically, semantically inconsistent pairs are of minority, and we treat them as noisy pairs. Note that deep neural networks (DNNs) have a crucial memorization effect that DNNs tend to first memorize clean (majority) examples before overfitting to noisy (minority) examples. Therefore, we set a relatively large weight for aggressively augmented data pairs at the early learning stage. With the training going on, the model begins to overfit noisy pairs. Accordingly, we gradually reduce the weights of aggressively augmented pairs. In doing so, our method can better embrace aggressive augmentations and neutralize the semantic shift problem. Experiments show that our model achieves 73.1% top-1 accuracy on ImageNet-1K with ResNet-50 for 200 epochs, which is a 2.5% improvement over BYOL. Moreover, experiments also demonstrate that the learned representations can transfer well for various downstream tasks. Code is released at: https://github.com/tmllab/RSA.


RSA: Reducing Semantic Shift from A Aggressive Augmentations for Self-supervised Learning
Yingbin Bai
Erkun Yang²
Zhaoqing Wang'
Yuxuan Du
Bo Han"
Cheng Deng"
Dadong Wang²
Tongliang Liu
'ML Lab, The University of Sydney, Xiidian University; SJD Explore Academy; Hong Kong Baptist University; "CSIRO
Abstract
Most recent self-supervised learning methods learn visual representation by con- trasting different augmented views of images. Compared with supervised learning, more aggressive augmentations have been introduced to further improve the di- versity of training pairs. However, aggressive augmentations may distort images' structures leading to a severe semantic shift problem that augmented views the same image may not share the same semantics, thus degrading the transfer performance. To address this problem, we propose a new SSL paradigm, which counteracts the impact of semantic shift by balancing the role of weak and ag- gressively augmented pairs. Specifically, semantically inconsistent pairs are of minority, and treat them as noisy pairs. Note that deep neural networks (DNNs) have crucial memorization effect that DNNs tend to first memorize clean (ma- jority) examples before overfitting noisy (minority) examples. Therefore, we set relatively large weight for aggressively augmented data pairs at the early learning stage. With the training going on, the model begins overfit noisy pairs. Accordingly, we gradually reduce the weights of aggressively augmented pairs. In doing so, our method can better embrace aggressive augmentations and neutralize the semantic shift problem. Experiments show that our model achieves 73. %% top-1 accuracy on ImageNet-1K with ResNet-50 for 200 epochs, which 2.5% improvement over B BOL. Moreover, experiments also demonstrate that the learned representations can transfer well for various downstream tasks. Code is released at: https://githin www.coo////aoooo Rab/R/A
Introduction
golden law the context of computer vision is utilizing tremendous annotated data to learn good Unfortunately, collecting annotated data with accurate labels is generally laborious, expensive 52 56), and even infeasible (22). To this end, various approaches have been proposed to learn such representations from unlabelled visual data, usually by performing
visual representations 53 26).
visual pretext tasks. Among them, self-supervised learning methods 6 7 46 9
based on contrastive loss have recently shown great promise, achieving state-of-the-art performance.
Representative contrastive methods are generally trained by maximizing agreement between dif- ferently augmented views of the same image (positive pairs), and increasing the distance between
augmented views from different images (negative pairs) (49 6 [7]. Compared with supervised learning, these works highlight the role of data augmentation for SSL and design more aggressive augmentation operations. Here, we refer aggressive augmentations as the operations that can possibly change the semantics of images, such grayscale, color jitter, and Gaussian blur. Other used
"Correspondence to Tongliang Liu
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

(a) Noisy samples from ImageNet-1K K 25
(b) Noisy samples from CIFAR-101 100
The first row is the original images (ImageNet-IK images are resized to squares) and noisy samples from aggressive augmentation are in the second row. From the first three images of (a), we can observe that Color jitter operation makes the image too bright or too dark that covers the details of images; and in the fourth and fifth columns of (a), Grayscale and Gaussian blur operation leads images to hardly distinguished from the background; and the same augmentation strategy from ImageNet-I K leads to more vague images for CIFAR-100 shown in (b).
Figure 1:
augmentations are referred as weak augmentations. It should be noted that the random cropping operation can also change the semantic information, however, this issue can be properly addressed by exploiting object detection techniques 40 (3]. Therefore, we do not include the random cropping operation aggressive augmentation. Although the aggressive augmentations can help to further improve the model performance, they also bring severe semantic shift problem for training images. As illustrated in Figure IT, the first row shows original images from ImageNet 25 and CIFAR-100 100 [4] datasets. And the second row presents the corresponding augmented views with the widely used composition of augmentations 7 6 We see that the augmented views recognized as semantically consistent with their original versions. Pushing these images to have similar representations can adversely affect the model training, so we consider the semantically inconsistent pairs as noisy pairs. However, due to the diversity of training images and the randomness of augmentation, it is difficult to accurately measure the quality of augmented pairs. Attempting to roughly remove noisy pairs may result in decrease in performance since they are mixed with clean (semantic consistent) pairs.
be hardly
30 show that deep neural networks (DNNs) have a crucial memorization effect that DNNS tend to first memorize clean (majority/semanticllly consistent) examples before overfitting noisy (minority/semmntically inconsistent) examples. Motivated by this, propose method called Reducing Semantic shift from Aggressive augmentation (RSA) by dynamically adjusting the weights of clean and noisy pairs. Specifically, we introduce aggressive- weak augmented pairs as relatively clean pairs because weak augmentation has low probability of causing the semantic shift problem, and set a relatively large weight for the aggressively augmented data pairs at the beginning of training fully exploit all the training examples. As the training goes on, the model begins to overfit semantically inconsistent data. Therefore we gradually decrease the weight of aggressive augmented pairs and increase the weight of aggressive-wezz weak augmented pairs to reduce semantically inconsistent impact.
Fortunately, recent works 22
57 16
Empirical results multiple benchmark datasets show that method can outperform state-of-the- art methods in various settings with a large margin. For instance, with 200 epochs of pre-training, our method achieves 73.1% Top-1 accuracy on ImagetNet-Ia 1K (25) linear evaluation protocol, which is 2.5% higher than BYOL [15]. Experiments on MS COCO 22 also show that our pre-trained model can continually improve the performance for multiple downstream tasks.
2 Related Works
Self-supervised learning (SSL) has attracted great attention to capture universal representations
20
60 38 48 12 61). The core of SSL is designing agent tasks, which allow us learn representations from large-scale unlabeled data via pseudo labels instead of using any human annotations. To this
end, many proposals devise different solutions for constructing pseudo labels, including predicting the rotation of images [3), putting pieces of images together (36), or recovering color from grayscale images [59]. Particularly, Wu et [49] propose an instance-level classification, which regards images augmented from the same image positive pair and others negative examples. SimCLR [6]

---

improves performance by inserting the projection network and introducing aggressive augmentation. He et al. 7 store negative representations a queue to reduce the memory requirement. BYOL 11] enhances the power of SSL. by removing the dependence on negative examples, which also addresses the problem of false negative examples [9. Despite these methods having proved their effectiveness based on aggressive augmentation, they ignore the semantic shift problem from it.
have discovered that aggressive augmentation may generate noisy samples in the positive pairs. To alleviate this issue, ContraCAM [35] proposes a two-step approach reduce the issue of random cropping, which seeks objects first and then crops images based on their locations. In addition, Gansbeke et al. [45] conduct experiments on scene-centric datasets (e.g., COCO) containing multiple objects in images and argue that SSL can overcome the issue of random cropping. Since the issue of random cropping has been well studied, our work focuses on other types of augmentations, which can be viewed as a complement to previous studies.
Noisy samples in aggressive augmentation. Recent studies 47
40 35 39]
Learning with noisy labels. Reducing the semantic shift problem in SSL is similar to another well-studied topic in machine learning, learning with noisy labels [1 51 54 50 [53. In this field, many state-of-the-ar methods use the memorization effect to select confident examples in the early
learning phase [37 41 27]. Specifically, Co-teaching 16 uses the early stopping trick and the small-loss strategy to choose confident examples. PES finds examples with noisy labels have more detrimental effects on the latter layers, and improves the early stopping trick by progressively training each part of DNNs. However, directly using the early stopping trick difficult in SSL because noisy pairs are randomly generated by aggressive augmentations over epochs while samples with noisy labels are fixed in learning with noisy labels. Besides, roughly removing noisy pairs is more likely to delete many clean pairs by mistake, especially in the low noise rate case.
3 Methodology
Our approach aims minimize the detrimental impacts of semantically inconsistent (noisy) pairs from aggressive augmentations while taking advantage of aggressive augmentations. As such, we first revisit preliminaries on self-supervised learning. Then, we elaborate on the proposed learning algorithm that counterbalances the noise impacts by utilizing the memorization effect of DNNs.
3.1 Preliminaries on Self-supervised Learning
Self-supervised learning methods based on contrastive learning generally require learning an embed- ding space that can easily separate different examples. Let D be set of images, and an image x; is uniformly drawn from D. Denote and L' of image augmentation T. v and v' are two augmented views of the image I; with v = t(z,) and o' E' (z:). which are regarded as pair of positive examples. Then, v and v' will separately feed through an encoder fe and projector ge to embed a and 2', which are required to be close to each other via a contrastive loss function, e.g., InfoNCE [4] can be expressed as,
different instances from the same distribution
exp(2.2//1)
LNCE = log
(1)
exp(z 2//1 EmEN exp(2 n/7)'
where y is temperature parameter, and N is set of negative example vectors. The embeddings of positive and negative examples are Iz-normalized. To stabilize the training process, some state-of- the-art methods 17 61 23 employ asymmetric framework, including online network and target network. For the online and target networks, they have the same network structure but different weights of encoder fe, and projector 9s. whose parameters & are updated by the online parameters with the exponential moving average method.
Recently, BYOL [5 found negative examples are not necessary and added predictor qo in the online network to avoid collapsed solutions, e.g., all images have the same vector. And, the loss function can be simplified to,
(2, 1 2 * ||21|
Lyse = 22-2 2
(2)
3

---

online
gradient
SSL
"EST. ang.
encoder
projector
predictor
Js
weak aug. fir
2
exponential
moving average
RSA
ang.
encoder
projector
Au
ang.
Ra
target
stop gradicut
The illustration of our proposed method (RSA). We utilize an asymmetric-style framework, including an online network and a target network. The online network is optimized by gradients, and the target network updated with the exponential moving average strategy. We first adopt the weak augmentation to generate two views (Vw, ryy), then adopt the aggressive augmentations to further generate another two views (Ua, r.), Subsequently, we make aggressive-augmentee views to keep consistent with their corresponding weak- and aggressive-augmentee views in the embedding space. On the right of the image, we compare RSA with classical SSL methods. RSA forces learned representations to balance between weak- and aggressive aggressive-augmentte views. (Best viewed in color)
Figure 2:
3.2 Description of RSA
As discussed in Introduction, noisy pairs from aggressive augmentations will lead to a severe semantic shift problem, which damages the generalization on downstream tasks. However, most of the sample pairs generated from aggressive augmentations are beneficial for the model performance, and what precise degree of distortion would cause bias remains unknown, so it is hard to distinguish between clean pairs and noisy pairs in SSL. Instead of separating noise from training data, we employ weak augmentation to generate sample pairs as relatively clean pairs.
To this end, we propose an efficient method to reduce semantic shift from aggressive augmenta- tions, dubbed RSA. Envisioned by the memorization effect, DNNs will first fit clean pairs (major- ityssemantically consistent) of training data in the early learning phase and then overfit to noisy pairs ssmmatccl// inconsistent). Noisy pairs account for the minority, so we assume that the noise impacts will be small and then increases as the training process. We give different weights to clean and noisy pairs, and gradually reduce the weights of noisy pairs against raised noise impacts.
However, simply introducing clean pairs will significantly increase computation, and SSL has known that more computation will increases performance, which makes it difficult to fairly compare with other baselines. To limit computation, we meet two problems. First, generating more augmented instances, especially double instances, commonly means expensive computation, largely burdening CPU and hard disk, which leads to slow data processing down and low efficiency of GPU [ Second, more instances require more prok-propag times, which results in more processing time for GPU.
To address the first problem, we propose a novel data augmentation pipeline, called multi-stage augmentation, which can generate double instances with nearly the same resources. Specifically, data augmentation generates image variants by mixing different types of image transformation and arranging them into queue. For transformation, it receives image from the result of the previous transformation and transforms the image based on pre-defined probability, and then passes it to the next transformation, taug( (z) ta (t)))))), where t, and tur are two subsets of taug. Based on this process, we can separate it into two child processes. Namely, fnp includes weak augmentation operations while l, conducts aggressive augmentation operations based ba. This process can be described as,
Uw = (x:
Vy = (x),
(3)
Via = ta (Vw))
0/ E' (v

---

Algorithm 1: RSA: Reducing semantic shift
Input: Neural network and fe. Projector go and gg. Predictor ạo- A batch of samples An image set D. Total number of training steps K. Weak augmentation function tw- Aggressive augmentation function fa- Syyper-parameter Bbase.
for k +-11 1 to K do D vw f (x) and e t (x) Va t ta ("w) and 1/ t (1()))
Sample batch of images // Multi-stage augmentation
Fa - 20 (o So (ra))) and žn + ge ve(Je(Un) (v:))
Embedding for four views
Fo - 20 (So (ra))) and z/y + D8 ((((()))))
C t (1 B) Lmoe (zu: #'mm) + f.fmmeeFos 2/x) +
Reee (F'r Sw) + 3. (2/:: zo) // Decreasing $
3 + Prase x (COO((T *x +1
Update la, and qo with L Update [g and dg by slowly momentum updating with fo and ge Output: The trained network fo
For the second problem, send aggressive augmented views to the online network and weak augmented views to the target network, which means each instance only forwards once. Accordingly, RSA requires Back-propagation twice, which is equal to many Mate-of-the-ar SSL methods with symmetrized loss 66 g 15 5)
Za (f 2/3 =
(f(e((v z'y d6 (Fe(v)))
After obtaining four representations, we group them into four pairs and make each aggressive- augmented view keep consistent with their corresponding weak- and aggresiiv-augmented views in the embedding space. For the corresponding aggressive-augmente views, rather than encoding them on the target network, use the representations from the online network with a nop-gradienn operation. The total loss is summarized as,
(1 C 2nn)) 3L (Za: 2//) zw) 3. (Z/g za);
(5)
+(1 - +(1-3)Lmaa ()s
where B is a calculated parameter to re-weight the losses from weak- and satresssive-ugmmenett views. This loss function forces networks learn representations that achieve a balance between weak- and aggressive-augmentt simple solution. As the training goes on, the noisy impacts will increase. We further offset it by using the memorization effect, which decreases 3 with cosine decay equation,
views, which reduces overfitting to noisy pairs while avoiding
k (cos(
8 Bbase x 1
1),
(6)
K
where Bbase is given number at the training beginning, and k and K are the current training steps and the total training steps.
Experiments
4.1 Datasets and Implementation Details
Datasets: We assess the proposed method on six image datasets, from small to large. We choose CIFAR-10/100 (24 for small datasets, and STL-10 10] and Tiny ImageNet ( for medium datasets,
and ImageNet-100 [2 and ImageNet-IK 255 for large datasets. Note, ImageNet-100 contains 100
5

---

Table 1: Analysis of the multi-stage augmentation and the memorization effect. We run methods on small and medium datasets for 200 epochs, and on ImageNet-106 100 for 100 epochs. The mean and standard deviation are computed over three trials. Note that two augmented instances with aggressive augmentation are referred to as AA, instances with one aggressive and one weak augmentation as AW, and instances with multi-stage augmentation as MA.
Method
Aug.
B
CIFAR-10
CUFAR-100
STL-10
Tiny ImageNet
ImageNet-100
BYOL
AA
90.2+1.00
63.9
91.3:10.1
49.5#t..1
80.9
BYOL
AW
90.5++0.2
66.1
90.7+0000
51,1+0.3
81.2
RSA
MA
Fixed
91.3++0.2
66.5+0.00
92.8+0.2
53,9++.2
83.22
RSA
MA
Decay
92.1+-0.0
67.6++0.3
93.0++0.2
54.7++0.3
83.55
and we choose the same classes with 42] For STL-10, both 5k labeled and 100k unlabeled images are used for the pre- pre-trained model, and only 5k labeled images are used for the linear evaluation.
classes that are randomly selected from ImageNet- IK
225
Augmentation: In this paper, define "aggressive" augmentation including grayscale, color jitter, and Gaussian blur, while "weak" augmentation includes random crop and horizontal flip. The hyper- parameters of the augmentations are following MoCo v2 7 except for the size of the cropped images for the small and medium datasets, and we resize images to 32 x 32 and 64 x 64 for the small and medium datasets, respectively.
Baselines: For the comparison, we re-implement the
methods, SimCLR 6 MoCo
v2 (7). SimSiam [8], and BYOL [I5] based on the public codes. We follow [8] that implements the MoCo v2 with a symmetrized loss function, and set the exponential moving average factor to 0.99 for all experiments. For B YOL 155 the small and medium datasets, follow [15, and set the channel inner layer of 1024 in the projection and prediction MI.P and the output feature is 128. We initially set the exponential moving average factor as 0.99 and gradually enlarge it to 1.
Network structure and optimization: Our method and reproduced methods are implemented by PyTorch v1.8 and we conduct all experiments on Nvidia V100. Our method is based our reproduced BYOL [5]. For the pre-train stage on the small and medium datasets, we adopt ResNet- 18 [19] as backbone. For optimization, rate of 0.1 [62]. a momentum of 0.9, weight decay of 5 x 10 4, and a batch size of 256. We set Sbase 0.3 for CIFAR-10/100 and Boose = 0.4 for Tiny ImageNet and STL-10.
SGD optimizer with cosine-annealed learning
For the pre-train stage on large datasets, we adopt a standard ResNet-50 19 as backbone. For ImageNet-1000 the network is trained using SGD optimizer with single cycle of cosine annealing [32]. and an initial learning rate of 0.2, momentum of 0.9, weight decay of 10-4, and batch size of 256. We conduct ImageNet-IK experiments with 8x Nvidia V100 32G with Automatic Mixed Precision (AMP) package (33] Specifically, we follow [5), and train a network with a LARS optimizer [55) with single cycle of cosine annealing [32] momentum of 0.9, weight decay of 10-6,
10 6 and a batch size of 2048. The base learning rate starts from 0.9 and 0.6 for 100 and 200 epochs
respectively, linearly scaled by the times of batch size 256 14] We set Bbase 0.4 for ImageNet-100 100 and ImageNet-IK, IK.
We evaluate the representations of the pre-trained model with the linear evaluation protocol, which freezes the encoder parameters and trains linear classifier on top of the pre-trained model. For the small and medium datasets, we follow the setting in MoCo v2 ? and train linear classifier for 100 epochs with an initial learning rate of 30, no weight decay, and a momentum of 0.9. The learning rate will be multiplied by 0.1 at the 60 and 80 epochs. For the large datasets, we follow the evaluation setting in Mean Shift (33, which only requires 40 epochs and a batch size of 256. The linear classifier is trained with SGD and an initial learning rate of 0.01, weight decay of 10-4, and momentum of 0.9. The learning rate will be multiplied by 0.1 at 15, 30, and 40 epochs.
Evaluation:
4.2 Preliminary Analysis
In Table I we investigate the effectiveness of the proposed multi-stage augmentation and the memorization effect. We first compare the linear classification of BYOL with two aggressive augmentations (AA) and with one aggressive and one weak augmentation (AW) and RSA with multi-stage augmentation (MA) without memorization effect (fixed 3). From the second row, can

---

Table Performance comparison with linear classification on small and medium datasets for 200 and 800 epochs. We adopt ResNet- 18 as a backbone for all experiments. The mean and standard deviation are computed over three trials.
CIFAR- 10
CIFAR- 100
STL-I0 10
Tiny ImageNet
Method
ep
800 ep
200 ep
800 ep
200 ep
800
200 ep
800 ep
SimCLR
88.3+0.2
91.1
59.5+0.3
63.8
87.9=0.4
91.0
46.7+0.1
48.4
MoCo
87.11002
91.4
60.31104
65.0
08.4.1..3
91.3
47.8:10.4
SimSiam
86.910011
91.1
55.810.9
62,4
85.0.004
89.7
41.2.10.3
44.5
BYOL 90.2110.0
92.7
63.910.5
68.2
91.
93.4
49.
53.0
RSA (Ours)
92.310.11
93.7
08.1100.5
70.4
9..1
94.0
5..810.4
55.5
see that BYOL with AW can largely improve the performance on CIFAR-I00 and Tiny ImageNet, and mildly improve the performance on CIFAR-10 and ImageNet-100, but the network suffers from AW on STL-10. These inconsistent results suggest that although AW can help against the noise impacts from aggressive pairs, it reduces the diversity of augmented pairs compared with AA, which may result in suboptimal result. In contrast, MA not only outperforms AA and AW but also generally improves performance across five datasets, suggesting that multi-stage augmentation (MA) can take advantage of AA and AW at the same time.
According to the memorization effect of DNNs, noise impacts vary at different stages of the training process and become more apparent at the end. In order to balance the increasing noise impact, we decay 3 during the training, which results in rising the weights of aggressive- weak augmented pairs and reducing the weights of aggressive-aggress fourth verifies that
augmented pairs. Improved performance in the the memorization effect of DNNs to further reduce noise impact. Notably, we set Staee = 0.5 in preliminary experiments, and the performance can be further improved with turned Bbase in Table and
4.3 Linear Classification
Small and medium datasets. We evaluate our method on small and medium datasets, with 200 and 800 epochs. We also run three trials for a short rumning time evaluate the stability of the proposed method. Table 2 illlustrates that the proposed method significantly improves the performance across the four datasets on short training time experiments, demonstrating that RSA can accelerate convergence and the small standard deviation also shows that the proposed method has good stability. For the long train- ing time experiments, outstanding results show RSA can continue to improve the final performance.
Table 3: Performance comparison with linear clas- sification on ImageNet- for 100 and 200 epochs. All methods adopt ResNet-50 as backbone.
Method
Batch size
ep
200 ep
SimCLR
256
79.1
82,4
MoCo v2
256
80.9
83,9
SimSiam
256
79.7
82.6
BYOL
256
80.9
83.6
RSA (Ours)
256
83.7
85.5
Large datasets. We evaluate the performance of the proposed method on the large datasets, ImageNet- 100 and ImageNet- IK. We reproduce all baselines on ImageNet-100 100 with batch size 256. The results on ImageNet-100 and ImageNet-Ike IK are shown in Table 3 and Table respectively. We can see that RSA outperforms the state-of-the-an methods on ImageNet- 100 with relatively large margin across 100 and 200 epochs. For results on ImageNet- K, RSA consistently surpasses baselines, e.g., the performance of RSA for 100 epochs has already surpassed B BYOL training for 200 epochs. RSA achieves a new state-of-the-ant result for 200 epochs, exhibiting a 2.5% improvement over BYOL. Overall, empirical results on linear evaluation verify that RSA can constantly improve the generalization various settings.
4.4 Transfer Learning
We further verify the quality of representation learned by RSA more downstream tasks. For object detection and instance segmentation, we follow 43), and adopt Mask R-CNN 18] with FPN (28 to fine-tune our pre-trained ResNet-50 model on COCO train2017 with x schedule and x schedule,
7

---

Table 4: Performance comparison with linear classification on ImageNet- -eeeet-K,., IK. All methods use standard ResNet-50 as a backbone without a multi-crop strategy.
Method
Neg. pairs
Batch Size
Epochs
Top-1 Linear
Supervised
256
InstDis 49
256
200
56.5
PIRL 34
256
200
63.6
SimCLR
4096
1000
69.3
MoCo v2
256
200
67.5
JCL 4
256
200
68.7
ReSSL 61
256
200
69.6
InfoMin Aug.
43
256
200
70.1
W-MSE 4
12
4096
400
72.6
SimSiam
256
SwAV 8
4096
200
69.1
BYOL 8
4096
100
66.5
BYOL g
4096
200
70.6
BYOL
2048
300
72.4
RSA (Ours)
2048
100
71.4
RSA (Ours)
2048
200
73.1
Table 5: Transfer learning downstream tasks: object detection, instance segmentation. All models were pre-trained on ImageNet-IK for 200 epochs and fine-tuned on MS COCO with schedule. Object detection and instance segmentation results from 4
Object detection
Instance segmentation
Method
A
AP
AP
Apmk
APm
APm
random init.
32.8
50.9
35.3
29.9
47.9
32.0
supervised
39.7
59.5
43.3
35.9
56.6
38.6
InstDis 44
38.8
58.4
42.5
35.2
55.8
37.8
PIRL [34
38.6
58.2
42.1
35.1
55.5
37.7
MoCo 17
39.4
59.1
42.9
35.6
56.2
38.0
MoCo
40.1
59.8
44.1
36.3
56.9
InfoMin Aug. 43
60.6
44.6
36.7
57.7
39.4
RSA (Ours)
41.1
61.4
45.1
37.3
58.6
40,1
Table 6: Transfer learning on downstream tasks: object detection, and instance segmentation. All models were pre-trained on ImageNet-1K for 200 epochs and fine-tuned on MS COCO with 2
schedule. Baseline results from 43
Object dctection
Instance segmentation
Method
A
AP
Ap
Appa
AP
A
Random
38.4
57.5
42.0
34.7
54.8
37.2
Supervised
41.6
61.7
45.3
37.6
58.7
40.4
InstDis 49
41.3
61.0
45.3
37.3
58.3
39.9
PIRL 34
41.2
61.2
45.2
37.4
58.5
40.3
MoCo 17
41.7
61.4
45.7
37.5
58.6
40.5
MoCo 7
41.7
61.6
45.6
37.6
58.7
40.5
InfoMin Aug. [3
42.5
62.7
46.8
38.4
59.7
41.4
RSA (Ours)
42.7
62.9
47.1
38.5
60.0
41.4
and evaluate performance COCO va/2017. We set the initial learning rate 0.02 and 0.03 for the 1 x schedule and x schedule experiments respectively.
Table 5 and Table report the results of object detection and instance segmentation. We can observe that RSA outperforms baselines on object detection and instance segmentation tasks, especially, showing superior over the strong baseline InfoMin Aug. 43 that adopts more advanced augmentation, RandAugment [IT]. Strong performance on downstream tasks demonstrates that RSA can improve the general quality of learned representations.

---

Table 7: Training time comparison with BYOL on three datasets
Method
CIFAR- 100
STL-10
ImageNet- 100
BYOL
11.3h
77.6h
9.5h
RSA
11.5h
79.6h
10.4h
Top Accuracc 196
Top-1 Accuracy
Top-3 Accuracy
03
Disse
(a) CIFAR- 100
(b) STL-10 10
(c) ImageNet- 100
25
Figure 3: Sensitivity analysis for the hyper-parameter Shase- We conduct experiments on CIFAR-100
and STL-10 for 800 epochs, and ImageNet-100 for 100 epochs, respectively. Note that the green dash line is the result with 0.
4.5 Training Time Comparison
We compare the running time between our method and reproduced BYOL. We conduct experiments on CIFAR- 100 and STL-10 for 800 epochs with single Nvidia V100, and ImageNet- 100 for 200 epochs with 4x Nvidia V100, respectively. For ImageNet-1000 100, we use Automatic Mixed Precision package 33 to speed up the training process and save GPU memory. Note that since we do not have enough GPUs to run a standard BYOL with batch size 4096 on ImageNet, we use ImageNet-100 instead of ImageNet-1K K, which has similar processing efficiency but ten times the data.
Although our method uses double instances in the loss function each instance in RSA only passes into the network one time. Therefore, the number of forward and backward passes keeps the same with BYO.. In addition, thanks to the efficient multi-stan -tate augmentation, where we generate double instances using the nearly same resources. The main burden training time part of our proposed method may come from two more vector multiplication in our loss function From Table we can see that our method is as efficient as B YOL, and the differences between the two methods are less than 9% across three datasets.
4.6 Ablation Studies
In this section, we investigate the importance of the hyper-parametee Bhose in Eq which controls the initial weights of aggressive-aggressive and aggressive-weak augmented pairs Eq Figure reports the results with linear classification on three datasets. First of all, we can observe that Sbase makes remarkable contribution for the improvements compared with Bbase = O, with +0.4% (total increments +0.6%) for STL-10 and +0.7% (total increments +2.2%) for CIFAR- 1000 This suggests that removing aggressive-aggresivee augmented pairs with Brase = 0 will limit the full exploration of the diversity of aggressive augmentation. However, if Bhose is too large, aggressive-weal augmented pairs will make less contribution against noisy impacts.
4.7 Adaptability Studies
To evaluate the adaptability, we implement the proposed method based on SimSiam SimSiam RSA. Specifically, we change the same similarity function from mean square error to negative cosine similarity and adopt the same settings for all the Section 4.1 As illustrated in T we observe that the proposed Simsiam RSA achieves better transfer performance than SimSiam on three datasets. For instance, RSA significantly raises the accuracy of linear probing from 55.8% to 63.7% (+7.9%) on the CIFAR-100 100 dataset. Therefore, the proposed method does not rely the momentum network, demonstrating the adaptability of RSA.
g termed
mentioned in

---

Table Performance comparison with linear classification for 200 epochs. The mean and standard deviation are computed over three trials.
Method SimSiam Simsiam + RSA
CIFAR-I00
STL-10
ImageNet-100
55.8=0.9
85.0+0.4
82.6
63.7.10.5
89.2.10.5
84.0
Conclusion
In this paper, we first empirically demonstrate that two positive instances generated by the aggressive augmentations can cause the semantic shift issue, which introduces noisy pairs and degrades the quality of learned representation. To alleviate this issue, we propose a novel method RSA, which dynamically adjusts the weights of clean and noisy pairs based on the memorization effects. Experi- mental results show that our proposed method achieves state-of-the-ar results on various datasets and consistently improves the generalization on series of downstream tasks.
The main limitation of this paper is that, although we know there is noise from aggressive augmenta- tions, the specific conditions under which noise occurs and how much DNNs will suffer from it are still unknown. We leave it as the future research direction.
Acknowledgments and Disclosure of Funding
The authors would like to thank the anonymous reviewers and the meta-reviewer for their constructive feedback and encouraging comments this work. Yingbin Bai was supported by CSIRO Data61. Erkun Yang was supported in part by the National Natural Science Foundation of China under Grant 62202365, Guangdong Basic and Applied Basic Research Foundation (202 12111155 15151 11115100026), and Natural Science Basic Research Program of Shaanxi (Program No.2022JQ-608) Zhaoqing Wang was supported by OPPO Research Institute. Bo Han was supported by the RGC Early Career Scheme No. 22200720, NSFC Young Scientists Fund No. 62006202, and Guangdong Basic and Applied Basic Research Foundation No. 2022A15 151501 1652. Cheng Deng was supported in part by the National Natural Science Foundation of China under Grant 621 32016, Grant 62171343, and Grant 62071361, in part by Key Research and Development Program of Shaanxi under Grant 2021ZDLGY01-00 and in part by the Fundamental Research Funds for the Central Universities ZDRC2102. Tongliang Liu was partially supported by Australian Research Council Projects DP180103224, DE- DE-190101473, IC-190100031, 22222010,,,,, and FT-220100318.
References
Zohch Abai and Nishad Rajmalwar. Densenet models for tiny imagenet classification. hns/1904.10429, 2019.
CoRR,
[2] Devansh Arpit, Stanislaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S. Kanwal, Tegan Maharaj, Asja Fischer, Aaron C. Courville, Yoshua Bengio, and Simon Lacoste-Julien. A closer look memorization in deep networks. In ICML, pages 233 233-242, 2017.
[3] Yingbin Bai, Erkun Yang, Bo Han, Yanhua Yang. Jiatong Li, Yinian Mao, Gang Niu, and Tongliang Liu. Understanding and improving early stopping for learning with noisy labels. In NeurlPS, pages 24392-24403, 2021.
Qi Cai, Yu Wang, Yingwei Pan, Ting Yao, and Tao Mei. Joint contrastive learning with infinite possibilities. In NeurlPS, 2020.
[5] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsuper- vised learning of visual features by contrasting cluster assignments. pages 9912-9924, 2020.
[6] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. simple framework for contrastive learning of visual representations. In ICML, pages 1597-1607, 2020.
77] Xinlei Chen, Haoqi Fan, Ross B. Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. CoRR, abs/2003.04297, 2020.
10

---

[8] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In CVPR, pages 15750-15758, 2021.
[9] Ching-Yao Yao Chuang, Joshua Robinson, Yen-Chen Lin, Antonio Torralba, and Stefanie Jegelka. Debiased contrastive learning. NeurIPS, pages 8765-8775, 2020.
[10] Adam Coates, Andrew Y. Ng. and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In AISTATS, pages 215-223, 2011.
[11] Ekin D. Cubuk, Barret Zoph, Jonathon Shlens, and Quoe V. Randaugment: Practical data augmentation with no separate search. CoRR, 11011199911771 2019.
[12] Aleksandr Ermolov, Aliaksandr Siarohin, Enver Sangineto, and Nicu Sebe. Whitening for self-sup representation learning. ICML, pages 3015 3024,, 2021.
[13] Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In ICLR, 2018
[14] Priya Goyal, Piotr Dollár, Ross B. Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch. Yangging Jia, and Kaiming He. Accurate, large minibatch SGD: training imagenet hour. CoRR, abs/1706.026777 2017.
[15] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Ávila Pires. Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent- A new approach self- supervised learning. NeurlPS, pages 21271 112284, 2020.
[16] Bo Han, Quamming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In NeurlPS, pages 8527 0027-853,,22 2018.
[17] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum contrast for unsuper- vised visual representation learning. In CVPR, pages 9726 0006-97352222 2020.
[18] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross B. Girshick. Mask R-CNN. In ICCV, pages 2980-2988, 2017.
[19] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pages 770-778, 2016.
[20] Jyh-Jing Hwang, Stella X Yu, Jianbo Shi, Maxwell Collins, Tien-Ju Yang. Xiao Zhang, and Liang-Choe Chich Chen. Segsort: Segmentation by discriminative sorting of segments. In ICCV, pages 7334-7344, 2019.
[21] Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentomet: Learning data-driven curriculum for very deep neural networks on corrupted labels. In ICML, pages 2309-2318, 2018.
[22] Davood Karimi, Haoran Dou, Simon K. Warfield, and Ali Gholipour. Deep learning with noisy labels: Exploring techniques and remedies in medical image analysis. Medical Image Anal., 63:101759, 2020.
[23] Soroush Abbasi Koohpayegani, Ajinkya Tejankar, and Hamed Pirsiavash. Mean shift for self-supervised learning. In ICCV, pages 10306-10315 10315, 2021.
[24] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, 2009.
[25] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In NeurlPS, pages 1097-1105, 2012.
[26] Phuc H. Le-Khac, Graham Hcaly, and Alan F. Smeaton. Contrastive representation learning: A framework and review. IEEE Access, 1199907-1939344 2020.
[27] Junnan Li, Richard Socher, and Steven C.H. Hoi. Dividemix: Learning with noisy labels as semi-supervised learning. In ICLR, 2020.
[28] Tsung-Y Yi Lin, Piotr Dollár, Ross B. Girshick, Kaiming He, Bharath Hariharan, and Serge J. Belongic. Feature pyramid networks for object detection. In CVPR, pages 936-944.
[29] Tsung-Yi Yi Lin, Michael Maire. Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. In ECCV, pages 740 755, 2014.
11

---

[30] Sheng Liu, Jonathan Niles-Weed, Narges Razavian, and Carlos Fernandez-Granda. Early-learning regular- ization prevents memorization of noisy labels. In NeurIPS, pages 20331-203433 2020.
[31] Tongliang Liu and Dacheng Tao. Classification with noisy labels by importance reweighting. IEEE Transactions on pattern analysis and machine intelligence, 38(3):447-455 2016.
[32] Ilya Loshchilov and Frank Hutter. SGDR: stochastic gradient descent with warm restarts, In ICLR, 2017.
[33] Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory F. Diamos, Erich Elsen, David García, Boris Ginsburg, Michacl Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. Mixed precision training. In ICLR, 2018.
[34] Ishan Misra and Laurens van der Maaten. Self-supervised learning of gretext-invariant representations. In CVPR. pages 6706-66 -776 6716,22 2020.
[35] Sangwoo Mo, Hyunwoo Kang, Kihyuk Sohn, Chun-Liang Li, and Jinwoo Shin. Object-aware contrastive learning for debiased scene representation. In Advances in Neural Information Processing Systems, pages 12251-12264, 2021.
[36] Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In ECCV, pages 69-84, 2016.
[37] Curtis G. Northcutt, Tailin Wu, and Isaac L. Chuang. Learning with confident examples: Rank pruning for robust classification with noisy labels. In UAI, 2017.
[38] Pedro O.O Pinheiro, Amjad Almahairi, Ryan Benmalek, Florian Golemo, and Aaron C Courville. Unsu- pervised learning of dense visual representations. NeurlPS, pages
2020.
[39] Xiangyu Peng, Kai Wang, Zheng Zhu, and Yang You. Crafting better contrastive views for siamese representation learning. In CVPR, 2022.
[40] Ramprasaath Selvaraju, Karan Desai, Justin Johnson, and Nikhil Naik. Casting your model: Learning to localize improves self-supervised representations. In CVPR, pages 11058-11061 2021.
[41] Daiki Tanaka, Daiki Ikami, Toshihiko Yamasaki, and Kiyoharu Aizawa. Joint optimization framework for learning with noisy labels. In CVPR, pages 5552-5560, 2018.
[42] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In ECCV, pages 776-794, 2020.
[43] Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for gond views for contrastive learning? NeurlPS 2020, pages 6827-6839, 2020.
[44] Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. CoRR, 00/11807703748 2018
[45] Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, and Lue V Gool. Revisiting contrastive methods for unsupervised learning of visual representations. 2021.
[46] Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In ICMI., pages 9929-9939, 2020.
[47] Yu Wang, Jingyang Lin, Jingjing Zou, Yingwei Pan, Ting Yao, and Tao Mei. Improving self-supervised learning with automated unsupervised outlier arbitration. 2021.
[48] Zhaoqing Wang. Qiang Li, Guoxin Zhang, Pengfei Wan, Wen Zheng, Nannan Wang, Mingming Gong, and Tongliang Liu. Exploring set similarity for dense self-supervised representation learning. In CVPR, 2022.
[49] Zhirong Wu, Yuanjun Xiong, Stella X. Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. In CVPR, pages 3733-3742, 2018.
[50] Xiaobo Xia, Tongliang Liu, Bo Han, Nannan Wang, Mingming Gong, Haifeng Liu, Gang Niu, Dacheng Tao, and Masashi Sugiyama. Part-dependent label noise: Towards instance-dependent label noise. In NeurIPS, pages 7597-7610, 2020.
[51] Xiaobo Xia. Tongliang Liu, Nannan Wang, Bo Han, Chen Gong, Gang Niu, and Masashi Sugiyama. Are anchor points really indispensable in label-noise learning? In NeurlPS, pages 6835-6846, 2019.
[52] Yan Yan, Rómer Rosales. Glenn Fung, Subramanian Ramanathan, and Jennifer G. Dy. Learning from multiple annotators with varying expertise. Mach. Learn., 95(3):291-327, 2014.
12

---

[53] Yu Yao, Tongliang Liu, Mingming Gong, Bo Han, Gang Niu, and Kun Zhang. Instance-dependent label-noise learning under structural causal model. In NewrIPS, pages 4409-4420, 2021.
[54] Yu Yao. Tongliang Liu, Bo Han, Mingming Gong, Jiankang Deng. Gang Niu, and Masashi Sugiyama. Dual T: reducing estimation error for transition matrix in label-noise learning. In NeurIPS, pages 7260-727 2020.
[55] Yang You, Igor Gitman, and Boris Ginsburg. Scaling SGD batch size 32k for imagenet training. CoRR, abs/1 778.03888, 2017.
[56] Xiyu Yu, Tongliang Liu, Mingming Gong, and Dacheng Tao. Learning with biased complementary labels. In ECCV, pages 69 85, 2018.
[57] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In ICIRR, 2017.
[58] Daokun Zhang, Jic Yin. Xingquan Zhu, and Chenggi Zhang. Network representation learning: A survey. IEEE Trans. Big Data, 6(1):3-28, 2020.
[59] Richard Zhang, Phillip Isola, and Alexei A. Efros. Colorful image colorization. In ECCV, pages 649-666, 2016.
[60] Xiao Zhang and Michael Maire. Self-supervised visual representation learning from hierarchical grouping. In NeurlPS, pages 16579-16590, 2020.
[61] Mingkai Zheng, Shan You, Fei Wang, Chen Qian, Changshui Zhang. Xiaogang Wang, and Chang Xu. Ressl: Relational self-supervised learning with weak augmentation. 2021.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes]
(b) Did you specify all the training details (e.g.. data splits, hyperparameters, how they were chosen)? [Yes] See Section 4.1
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Section 4.5
4. If you using existing assets (e.g., code, data, models) or curating/releasing assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
13

---

5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "jSVZsS_Gjzg", "reviewer": "Reviewer_mkyA", "summary": "This paper proposes a new learning strategy that assigns different weights for aggressively augmented data pairs at different training stages, to deal with the semantic shift problem caused by aggressive data augmentation methods for SSL. ", "questions": "Some works do not have momentum encoder (e.g., SimSiam). Is that possible to generalize the proposed learning strategy to them ?", "limitations": "The novelty is limited, and some comparisons are missing. Please refer to the comments in the Weaknesses for more details.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "--Strengths：\n1. This work contains extensive experiments to verify the effectiveness of the proposed strategy. \n\n--Weaknesses：\n\n1. Although this work has a good motivation, the proposed method is quite incrmental, which does not meet the standard of NeurIPS. For example, this work claims that  aggressive augmentations are retained, which is different from ReSSL [a] (lines 39-42). However, ReSSL does not give up the aggressive augmentations. ReSSL also feeds the weakly augmented images to the teacher model ( similar to the target network) , and inputs the aggressively augmented images to the student model (similar to the online network), which is exactly the same as MSR. The proposed learning strategy is more like a trick, which cannot support a paper to be accepted by NeurIPS. \n\n2. According to Figure 3, the proposed strategy seems not to work very well on some datasets, as the increments brought by the decayed $\\beta$ seems to be negligible. \n\n3. As for the experiment, it would be great to show some results compared to ReSSL [a] and W-MSE 4 [b] with the same batch size and training epochs. Concerning the second weakness, it is not quite clear where the improvement comes from. Does it come  from the different training settings (The batch size usually has a large impact on the performance in SSL) ?\n\n\nOverall, since I have some concerns about the effectiveness of the proposed strategy, the missing comparisons (I think it is hard to supplement them during the rebuttal stage), as well as the limited novelty, I vote for a rejection for this paper, but I may increase my score after the discussion stage, especially if I misunderstand something.\n\n\n**Ref**\n\n[a] Mingkai Zheng, Shan You, Fei Wang, Chen Qian, Changshui Zhang, Xiaogang Wang, and Chang Xu. Ressl: Relational self-supervised learning with weak augmentation. 2021.\n\n[b] Aleksandr Ermolov, Aliaksandr Siarohin, Enver Sangineto, and Nicu Sebe. Whitening for self-supervised representation learning. In ICML, pages 3015–3024, 2021.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "OQCrzhMnja", "reviewer": "Reviewer_gbda", "summary": "This paper studied self-supervised learning. A semantic shift problem in the aggressive augmentations of self-supervised learning is considered. This paper inherits the memorization effect in tackling noisy labels, and gradually reduces the weights of aggressively augmented pairs. Extensive experiments verify the effectiveness of the proposed method. ", "questions": "1.\tFrom the descriptions in Line 72—79, in fact, some prior works have targeted the problem of noisy examples in aggressive augmentation. This paper is a complement of previous studies. Could the statement mean that the improvement by the proposed method comes from more general types of data augmentation? The intuition of why the proposed method is superior should be further explained. \n2.\tIn Line 80-91, this paper states that it provides an indirect way to handle noisy samples, which is different from the methods in learning with noisy labels. Intuitively, when the noise rate is low, these methods can also work very well in coping with noisy samples. More descriptions of the advantages of this indirect way should be added. \n3.\tThis paper claims that the proposed method would not bring too much computation cost. However, the descriptions in Line 127-132 are not clear to support the claim. More discussions should be added. \n4.\tFrom Eq. (5), weight decay performs on all pairs. Could we perform weight decay only on the semantic-shifted pairs? \n5.\tExpect for the classification task, it seems that the proposed method is not much superior in other tasks. More discussions are needed for this issue. \n6.\tThe experiments on ImageNet-1K are more pivotal to verify the effectiveness of the proposed method. The ablation study should include the experiments on ImageNet-1K. \n\n7.\tMinor comments:\n•\tThe definition of aggressive augmentations is somewhat confusing to me when it firstly occurs in “Abstract”. More explanations can be added. \n•\tIn Related Work, the authors mentioned self-supervised learning learn representations from large-scale unlabeled data via pseudo labels, which is unfamiliar to me. The main reason is that the definition of pseudo labels commonly occurs in semi-supervised learning. Self-supervised learning is more related to unsupervised learning. \n•\tIn learning with noisy labels, “noisy” means the combination of “clean” and “incorrect”. Thereby, the authors should use “noisy” to indicate “incorrect”. \n•\tFigure 2 could be revised to highlight the contribution of this paper. \n•\tThe full names of “AA” and “AW” can be provided in the caption of Table 1. \n•\tTypos: \n-\tLine 68 He et al -> He et al. \n-\tLine 82 using first fit examples -> using first fitted examples \n-\tLine 96 utilizes -> utilizing \n-\tLine 115 severe semantic shift problem -> a severe semantic shift problem\n", "limitations": "See above weaknesses.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. Self-supervised learning is a practical and much important research topic in the community. \n2. The motivation of this paper is clear. The semantic shift problem is common in multiple self-supervised learning methods and needs to be addressed. \n3. Experimental results on benchmark datasets and real-world datasets show the effectiveness of the proposed method. Besides, ablation studies are provided to better understand the proposed method. \n\n\nWeaknesses: \n1. The writing and organization need to be improved to enhance this paper. \n2. The intuition of the advantages of the proposed method should be supplemented. \n\nThe weaknesses of this paper are detailed below. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Ev95-efM20L", "reviewer": "Reviewer_Ckh1", "summary": "The authors find that in most self-supervised learning methods, when applying aggressive augmentations to further improve the diversity of training pairs, there would exist severe semantic shift problem, thus degrading the transfer performance.\t\n\t\nTo address this\tproblem,  the authors propose a new SSL paradigm, which counteracts the impact of semantic shift by balancing the role of weak and aggressively augmented pairs.\n\nWith the training going on, the authors  gradually reduce the weights of aggressively augmented pairs.\n\nExperiments have been done on the small datasets (CIFAR10/100) and medium datasets (STL-10  and Tiny ImageNet), and large datasets ( ImageNet-100 and ImageNet-1K), which have validated the effectiveness of the proposed method.", "questions": "see weakness.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The motivation of the proposed method is strong and clear. The authors fully consider the semantic shift problem, thus propose to minimize the negative impacts of noisy positive pairs from aggressive augmentations while taking advantage of aggressive augmentations. It has been achieved by different weights while training.\n\nBut the weakness is that, I don't think the novelty is strong enough. As for compared with the previous work BYOL, I can find that the only difference is to add an extra aggr. aug. stream. The main architecture almost keeps the same.\n\nAnd for the re-weighting strategy, the authors use equ.(6) to achieve it. It is not flexible enough, which doesn't consider the quality of each augmentation pair.\n\nAnd in Figure 2, is there a mistake? I think in the row three, v_m and v_a should be exchanged. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "34a-AEicLXc", "reviewer": "Reviewer_ofeU", "summary": "The data augmentation transformations used in some self-supervised learning models for vision can generate pairs that are not semantically consistent. For example, cropping or blurring too aggressively can produce an image that is not identifiable as its class label. This work addresses this shortcoming by decreasing the degree to which a model relies on aggressive augmentations later on in training. The authors show doing so improves Top-1 linear classification accuracy for ImageNet as well as on object detection/segmentation for COCO over other self-supervised baselines.", "questions": "What's the intuition for the claim that \"small datasets have higher probabilities of generating noisy samples”? I don’t see the connection with the size of the dataset and how augmentations act on those samples. Can you make this explicit?", "limitations": "Yes", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The authors highlight a problem arising when data augmentations are too aggressive—thus producing samples that not semantically meaningful (\"noisy samples\"). The authors propose a solution to address this problem by reducing the weight given for aggressive augmentations later on in training. The authors motivate doing so by citing recent work illustrating that deep neural networks overfit to noisier samples later on in training.\n\nThe authors define “weak” augmentation to be random crop and horizontal flip—while “aggressive” augmentations additionally include other color based augmentations such as blurring and color jitter. This delineation is 1) at odds with the original motivation since even “weak” augmentations can produce crops that are not semantically meaningful. Thus violating the motivation that only “aggressive” augmentations produce noisy samples,  2) at odds with more recent SSL methods (Masked Autoencoders, MAE) that only use “weak” augmentations. If we instead want “aggressive” to capture the extent to which the augmented sample is noisy, then a more precise approach is to control the magnitude of the cropping, degree to which gaussian noise is added, etc. Such a definition of an augmentation’s “aggressive” extent would more directly validate the authors’ claims.\n\nI find the method an unnecessarily complicated (and memory-intensive) approach to achieve the stated goal of controlling the extent to which a model relies on aggressively augmented samples. Rather than introduce two asymmetric networks with twice the memory footprint, did the authors attempt to adjust the extend to which the samples fed into the original models were “aggressively” or “weakly” augmented? Or other simpler variants before settling on this two asymmetric network approach? At the very least, I'd like to see stronger motivation for the proposed method's implementation.\n\nOverall the paper is well-organized and clearly written. I found the sentences motivating the impact of “noisy samples” on the online network around line 144 confusing. I think the wording here can be improved. Algorithm 1 would be easier to follow if it was self-contained—it doesn’t include the loss. The portion of the diagram in Figure 2 on the right with numerous arrows and boxes is not easy to follow.\n\nOverall I find the experiments well-motivated and convincing (putting aside the \"weak\" versus \"strong\" augmentation definition). Appropriate baselines are used, authors assess sensitivity to beta, and show performance across several tasks. However, several of the experimental results (Tables 3, 4, 5) are used as evidence of MSR’s superior performance, but do not include error bounds.\n\nThe authors conclude MSR is a novel method to “property utilize aggressive augmentations.” If the motivation is that “aggressive augmentations” produce semantically noise samples, then the method’s success is instead to decrease models’ reliance on aggressive augmentations. The claim that is “property utilizes” or “neutarlizes the semantic shift problem” strikes me as an exaggeration. I suggest the authors appropriately qualify these claims.\n\nThe paper’s title suggests the proposed method directly improves models’ robustness. Instead, the approach balances the extent to which models rely on “aggressively” augmented samples during training. There are no experiments directly measuring robustness to augmentations. I suggest the authors amend the title to more directly reflect the contribution. \n\nOverall, I find the problem of ensuring data augmentation produces semantically meanginful pairs for self-supervised learning important. The proposed method attempts to avoid overfitting to noisy samples shows some performance gains over existing baselines. On the other hand, I find the “aggressive” versus “weak” augmentation definition used here to miss the mark. I also find the proposed method to be needlessly complex based on the stated objective.\n\nMinor:\n- inconsistent notation: v and v' describe two augmented samples but their representations are designated using subscript z1 and z2. It's easier for readers to follow the same designation either subscript or prime is used for pairs throughout. \n- line 96 typo: “by utilizes”\n- line 157 typo: “To further against”\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept