Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper studied self-supervised learning. A semantic shift problem in the aggressive augmentations of self-supervised learning is considered. This paper inherits the memorization effect in tackling noisy labels, and gradually reduces the weights of aggressively augmented pairs. Extensive experiments verify the effectiveness of the proposed method. 
strengths_and_weaknesses: Strengths:
1. Self-supervised learning is a practical and much important research topic in the community. 
2. The motivation of this paper is clear. The semantic shift problem is common in multiple self-supervised learning methods and needs to be addressed. 
3. Experimental results on benchmark datasets and real-world datasets show the effectiveness of the proposed method. Besides, ablation studies are provided to better understand the proposed method. 


Weaknesses: 
1. The writing and organization need to be improved to enhance this paper. 
2. The intuition of the advantages of the proposed method should be supplemented. 

The weaknesses of this paper are detailed below. 

questions: 1.	From the descriptions in Line 72—79, in fact, some prior works have targeted the problem of noisy examples in aggressive augmentation. This paper is a complement of previous studies. Could the statement mean that the improvement by the proposed method comes from more general types of data augmentation? The intuition of why the proposed method is superior should be further explained. 
2.	In Line 80-91, this paper states that it provides an indirect way to handle noisy samples, which is different from the methods in learning with noisy labels. Intuitively, when the noise rate is low, these methods can also work very well in coping with noisy samples. More descriptions of the advantages of this indirect way should be added. 
3.	This paper claims that the proposed method would not bring too much computation cost. However, the descriptions in Line 127-132 are not clear to support the claim. More discussions should be added. 
4.	From Eq. (5), weight decay performs on all pairs. Could we perform weight decay only on the semantic-shifted pairs? 
5.	Expect for the classification task, it seems that the proposed method is not much superior in other tasks. More discussions are needed for this issue. 
6.	The experiments on ImageNet-1K are more pivotal to verify the effectiveness of the proposed method. The ablation study should include the experiments on ImageNet-1K. 

7.	Minor comments:
•	The definition of aggressive augmentations is somewhat confusing to me when it firstly occurs in “Abstract”. More explanations can be added. 
•	In Related Work, the authors mentioned self-supervised learning learn representations from large-scale unlabeled data via pseudo labels, which is unfamiliar to me. The main reason is that the definition of pseudo labels commonly occurs in semi-supervised learning. Self-supervised learning is more related to unsupervised learning. 
•	In learning with noisy labels, “noisy” means the combination of “clean” and “incorrect”. Thereby, the authors should use “noisy” to indicate “incorrect”. 
•	Figure 2 could be revised to highlight the contribution of this paper. 
•	The full names of “AA” and “AW” can be provided in the caption of Table 1. 
•	Typos: 
-	Line 68 He et al -> He et al. 
-	Line 82 using first fit examples -> using first fitted examples 
-	Line 96 utilizes -> utilizing 
-	Line 115 severe semantic shift problem -> a severe semantic shift problem

limitations: See above weaknesses.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The data augmentation transformations used in some self-supervised learning models for vision can generate pairs that are not semantically consistent. For example, cropping or blurring too aggressively can produce an image that is not identifiable as its class label. This work addresses this shortcoming by decreasing the degree to which a model relies on aggressive augmentations later on in training. The authors show doing so improves Top-1 linear classification accuracy for ImageNet as well as on object detection/segmentation for COCO over other self-supervised baselines.
strengths_and_weaknesses: The authors highlight a problem arising when data augmentations are too aggressive—thus producing samples that not semantically meaningful ("noisy samples"). The authors propose a solution to address this problem by reducing the weight given for aggressive augmentations later on in training. The authors motivate doing so by citing recent work illustrating that deep neural networks overfit to noisier samples later on in training.

The authors define “weak” augmentation to be random crop and horizontal flip—while “aggressive” augmentations additionally include other color based augmentations such as blurring and color jitter. This delineation is 1) at odds with the original motivation since even “weak” augmentations can produce crops that are not semantically meaningful. Thus violating the motivation that only “aggressive” augmentations produce noisy samples,  2) at odds with more recent SSL methods (Masked Autoencoders, MAE) that only use “weak” augmentations. If we instead want “aggressive” to capture the extent to which the augmented sample is noisy, then a more precise approach is to control the magnitude of the cropping, degree to which gaussian noise is added, etc. Such a definition of an augmentation’s “aggressive” extent would more directly validate the authors’ claims.

I find the method an unnecessarily complicated (and memory-intensive) approach to achieve the stated goal of controlling the extent to which a model relies on aggressively augmented samples. Rather than introduce two asymmetric networks with twice the memory footprint, did the authors attempt to adjust the extend to which the samples fed into the original models were “aggressively” or “weakly” augmented? Or other simpler variants before settling on this two asymmetric network approach? At the very least, I'd like to see stronger motivation for the proposed method's implementation.

Overall the paper is well-organized and clearly written. I found the sentences motivating the impact of “noisy samples” on the online network around line 144 confusing. I think the wording here can be improved. Algorithm 1 would be easier to follow if it was self-contained—it doesn’t include the loss. The portion of the diagram in Figure 2 on the right with numerous arrows and boxes is not easy to follow.

Overall I find the experiments well-motivated and convincing (putting aside the "weak" versus "strong" augmentation definition). Appropriate baselines are used, authors assess sensitivity to beta, and show performance across several tasks. However, several of the experimental results (Tables 3, 4, 5) are used as evidence of MSR’s superior performance, but do not include error bounds.

The authors conclude MSR is a novel method to “property utilize aggressive augmentations.” If the motivation is that “aggressive augmentations” produce semantically noise samples, then the method’s success is instead to decrease models’ reliance on aggressive augmentations. The claim that is “property utilizes” or “neutarlizes the semantic shift problem” strikes me as an exaggeration. I suggest the authors appropriately qualify these claims.

The paper’s title suggests the proposed method directly improves models’ robustness. Instead, the approach balances the extent to which models rely on “aggressively” augmented samples during training. There are no experiments directly measuring robustness to augmentations. I suggest the authors amend the title to more directly reflect the contribution. 

Overall, I find the problem of ensuring data augmentation produces semantically meanginful pairs for self-supervised learning important. The proposed method attempts to avoid overfitting to noisy samples shows some performance gains over existing baselines. On the other hand, I find the “aggressive” versus “weak” augmentation definition used here to miss the mark. I also find the proposed method to be needlessly complex based on the stated objective.

Minor:
- inconsistent notation: v and v' describe two augmented samples but their representations are designated using subscript z1 and z2. It's easier for readers to follow the same designation either subscript or prime is used for pairs throughout. 
- line 96 typo: “by utilizes”
- line 157 typo: “To further against”

questions: What's the intuition for the claim that "small datasets have higher probabilities of generating noisy samples”? I don’t see the connection with the size of the dataset and how augmentations act on those samples. Can you make this explicit?
limitations: Yes
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The authors find that in most self-supervised learning methods, when applying aggressive augmentations to further improve the diversity of training pairs, there would exist severe semantic shift problem, thus degrading the transfer performance.	
	
To address this	problem,  the authors propose a new SSL paradigm, which counteracts the impact of semantic shift by balancing the role of weak and aggressively augmented pairs.

With the training going on, the authors  gradually reduce the weights of aggressively augmented pairs.

Experiments have been done on the small datasets (CIFAR10/100) and medium datasets (STL-10  and Tiny ImageNet), and large datasets ( ImageNet-100 and ImageNet-1K), which have validated the effectiveness of the proposed method.
strengths_and_weaknesses: The motivation of the proposed method is strong and clear. The authors fully consider the semantic shift problem, thus propose to minimize the negative impacts of noisy positive pairs from aggressive augmentations while taking advantage of aggressive augmentations. It has been achieved by different weights while training.

But the weakness is that, I don't think the novelty is strong enough. As for compared with the previous work BYOL, I can find that the only difference is to add an extra aggr. aug. stream. The main architecture almost keeps the same.

And for the re-weighting strategy, the authors use equ.(6) to achieve it. It is not flexible enough, which doesn't consider the quality of each augmentation pair.

And in Figure 2, is there a mistake? I think in the row three, v_m and v_a should be exchanged. 
questions: see weakness.
limitations: yes
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper proposes a new learning strategy that assigns different weights for aggressively augmented data pairs at different training stages, to deal with the semantic shift problem caused by aggressive data augmentation methods for SSL. 
strengths_and_weaknesses: --Strengths：
1. This work contains extensive experiments to verify the effectiveness of the proposed strategy. 

--Weaknesses：

1. Although this work has a good motivation, the proposed method is quite incrmental, which does not meet the standard of NeurIPS. For example, this work claims that  aggressive augmentations are retained, which is different from ReSSL [a] (lines 39-42). However, ReSSL does not give up the aggressive augmentations. ReSSL also feeds the weakly augmented images to the teacher model ( similar to the target network) , and inputs the aggressively augmented images to the student model (similar to the online network), which is exactly the same as MSR. The proposed learning strategy is more like a trick, which cannot support a paper to be accepted by NeurIPS. 

2. According to Figure 3, the proposed strategy seems not to work very well on some datasets, as the increments brought by the decayed $\beta$ seems to be negligible. 

3. As for the experiment, it would be great to show some results compared to ReSSL [a] and W-MSE 4 [b] with the same batch size and training epochs. Concerning the second weakness, it is not quite clear where the improvement comes from. Does it come  from the different training settings (The batch size usually has a large impact on the performance in SSL) ?


Overall, since I have some concerns about the effectiveness of the proposed strategy, the missing comparisons (I think it is hard to supplement them during the rebuttal stage), as well as the limited novelty, I vote for a rejection for this paper, but I may increase my score after the discussion stage, especially if I misunderstand something.


**Ref**

[a] Mingkai Zheng, Shan You, Fei Wang, Chen Qian, Changshui Zhang, Xiaogang Wang, and Chang Xu. Ressl: Relational self-supervised learning with weak augmentation. 2021.

[b] Aleksandr Ermolov, Aliaksandr Siarohin, Enver Sangineto, and Nicu Sebe. Whitening for self-supervised representation learning. In ICML, pages 3015–3024, 2021.
questions: Some works do not have momentum encoder (e.g., SimSiam). Is that possible to generalize the proposed learning strategy to them ?
limitations: The novelty is limited, and some comparisons are missing. Please refer to the comments in the Weaknesses for more details.
review_title: null

Meta-review: metareview: This paper aims to improve SSL pretraining by adjusting the strength of augmentations applied at different points in training, providing a large number of aggressive augmentations early in training with this rate decreasing over time to prevent the model from overfitting to noisy examples. Using this approach, the authors demonstrate substantial improvements over prior methods. All reviewers recognized the soundness of the motivation and were generally convinced by the experiments, though there were some concerns about whether the approach is too incremental since it is relatively simple. I strongly agree with the authors that simplicity is not a downside of an approach, but rather a benefit, and the fact that the approach works with such a small modification makes it more likely that this result is not caused by an obscure mix of hyperparameters. I also note that the authors engaged extensively with the reviewers, providing a number of additional experiments comparing to other approaches and providing further tests of the impact of the hyperparameter they introduce. I think this is an worthwhile paper which will have impact going forward, and I recommend acceptance. 
confidence: Certain
Decision: Accept