Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: ================================================================= 

[Summary] 

This paper proposes an improved version of ReLIC, dubbed ReLICv2, by including views of varying sizes and saliency masking into the ReLIC's training loss, and presents benchmarking results using ResNet encoders in several experiments.

================================================================= 


strength_and_weaknesses: ================================================================= 

[Main Strengths] 

The key merit of this paper is that the authors' approaches for improving the ReLIC indeed produce better benchmark results than previous ResNet-based methods at a different model and dataset sizes.

================================================================= 

[Main Weaknesses] 

The fundamental shortcoming of this study is that, while the basic idea is interesting, it does not appear to have the intended impact, as the title implies, much like the various flavors of contrastive methods. Given the richness of learned visual representation, focusing primarily on linear evaluations in the classification tasks is a too narrowed contribution.  I encourage authors to explore further into specific image domains and/or paired with Vision transformers to see whether ReLICv2 is still effective. Furthermore, I don’t see the implementation GitHub repo is intended to be provided.

================================================================= 

clarity,_quality,_novelty_and_reproducibility: ================================================================= 

[Technical Comments] 

1) Does the linear evaluation setup in fact disclose learning less spurious/correlated features? Background and style features may not be useful for dominant/foreground object classifications, but they would be valuable for dense prediction, which often needs consistency of multiple objects. 
2) Furthermore, I believe the linear evaluations employ all the labels in the fine-tuning stage, thus the performances shown in the paper are not label-free as the title implies. On the other hand, studying the labeled sample complexity will have more practical implications and will be a more engaging story.
3) In Tables 4 and 5, how does the ReLICv1 perform?

================================================================= 


summary_of_the_review: Using views of varying sizes and saliency masking to improve the original ReLIC's training loss is intriguing, but, it does not appear to make a significant impact, as do the many flavors of contrastive self-supervised approaches.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a few tricks that can be added on top of ReLIC for self-supervised learning. First, saliency masking is used to explicitly reduce spurious correlations in the model. Second, augmentations of small sizes are used. This has the effect of occluding some regions of the image. Using these two tricks, the authors were able to improve the performance of ReLIC by a good margin. Moreover, the authors also achieve SOTA results on Resnet based self-supervised learning methods. Several ablation studies are also performed.


strength_and_weaknesses: Strengths:

This is a well written paper. The authors clearly state what their main contribution is. The two tricks that are proposed is intuitive and looks promising. The paper is also explained well and I was able to understand the main contributions.

The experiments are very extensively performed. Comparisons with existing Resnet based models are included. Experiments are performed with different model sizes. Results are also shown on OOD, transfer, semi-supervised leanrning and for other tasks such as segmentation. Experimental section is quite strong and I don't have any concerns.

I also apprecitate the authors for providing detailed experimental settings in appendix.

Weaknesses:

The main tricks proposed in this work seem to be general purpose. Do you think this would work on ViTs?
In this regard, the self-attentions in ViTs might alrealy be doing some sort of saliency detection. For instance, they have been shown to do semantic segmentation. So, I am curious if thesr tricks can be used with ViTs?

Why did you stick only with Resnet based architectures? Do you think ReLICv2 would work with architectures like Swin? If so, would you set SOTA results there?



clarity,_quality,_novelty_and_reproducibility: Clarity: This is a well written paper. It is very clear.

Quality and Novelty: The tricks proposed in this paper is very effective as shown in the results. These tricks seem to be general purpose and would benefit other papers too.

Reproducibility: The authors have provided extensive experiment details in the appendix which is a good sign for reproducibility.
summary_of_the_review: Overall, I think this is a very good paper in terms of ideas proposed and rigorous experimental study. I advocate for acceptance.


======================================

Post rebuttal:

After looking at other reviews and having discussion with some reviewers, I feel like the paper has the following issues:
Similarity with prior work: As pointed out by reviewer Z39N, background augmentation only has marginal improvements. So, this makes me a bit worried about the main message the paper gives. Its probably that improvements mainly comes because of multi-crop at scale. If this is the case, the authors need to make this very clear (if the paper gets accepted). I also agree with other reviewers that claiming that they beat supervised learning is bit of a overclaim and I would appreciate if the authjors address this.

But, I also feel that even if the paper proposes a few tricks, I am fine with accepting the paper as the computer vision field has made progress through a series of tricks. But the authors should make that message very clear.

Since I was not aware of these concerns at the time of giving my review, I downvote my rating to 6.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a novel self-supervised learning framework called ReLIC v2. This framework builds upon ReLIC's loss and combines multicrops and background mask. The method achieves the state-of-the-art and better linear probe accuracy than supervised ResNet50.
Extensive transfer experiments and ablation study verifies the effectiveness of the proposed method.   
strength_and_weaknesses: Strengths:
1. This method achieves state-of-the-art ImageNet linear probe accuracy. It combines a few existing effective building components including ReLIC loss function, multicrops and background mask. Even though these components are already existing in the literature, I find that some special settings including 4x big + 2x small multicrops are novel.
2. The authors conducted a scaling study on a larger dataset such as JFT300M. Even though they derived relatively poor scaling performance, I find such study helpful and insightful, i.e. how augmentation-based joint-embedding method can benefit from scaling. 

Weaknesses:
1. Overall the paper has very limited novelty.
2. From ablation study, the background mask only shows 0.3% improvements. The main advantage seems to come from the multicrops. Using 4x big views + 2x small views means effectively much higher computation cost. This method still uses 4096 batch size. As far as I can tell, the previous works limiting to 2x big views + many small views with this batch size are mainly caused by GPU memory. It is not discussed how their method can afford this. 
3. The background mask here is very hand-crafted. Figure 8 seems too good to be true for arbitrary input. Is there any failure case for the background removing?
clarity,_quality,_novelty_and_reproducibility: The paper is well written.
All experimental details are provided.
summary_of_the_review: State-of-the-art self-supervised learning framework using ReLIC loss plus multicrops and background mask. Overall limited novelty but a lot of novel engineering design that's useful for the community.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper is an extension of RELIC self supervised learning method.

In this extension, two main augmentation method is added. (1) Saliency map (2) Views of varying sizes.
The loss function has also been modified.

Extensive experiments had been perform to study the effects of these extensions.
strength_and_weaknesses: (1) Strength: a lot of experiments are being carried out with a lot of analysis
(2) Strength: Results seems to be better than previous methods.
(3) Weakness: However there is no error bars to convince readers about the robustness of the reported results. for example, multiple duplicated experiments should be perform and p-value calculated when comparing the results of various methods.
(4) Weakness: it is not clear if the reported results are due to more careful tuning of hyper-parameters. A good way to silence this criticism is to show evidence of robustness of results with respect to hyper-parameter values.
(5) Weakness: Seems that the major contributions are two more augmentation methods. 
clarity,_quality,_novelty_and_reproducibility: Clarity: paper was carefully written with good clarity
Quality: extensive experimental results, though without uncertainties reporting
Novelty: The main contribution is additional augmentation method
Reproducibility: details are given, not sure if models are easy to tune
summary_of_the_review: Well written paper, however paper can be strengthen with error bars and with experiments showing how results differs with different hyper-parameters.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: **Summary**: The authors propose a self-supervised representation learning approach RELICv2, which builds upon the previous self-supervised method RELIC [Mitrovic et al., 2021]. There are two key additions: (i) unsupervised saliency masking that decouples the foreground from background, and (ii) multiple data augmentation of varying sizes especially with small sizes. RELICv2 is claimed to be the first unsupervised representation learning approach that consistently outperforms the supervised learning baseline across a range of ResNet architectures and various learning scenarios (ImageNet evaluation, transfer, robustness, and OOD generalization).. 

**Strengths And Weaknesses**: From the reviewers’ perspectives, the strengths of the paper are clear: simple strategies, strong and state-of-the-art performance, and extensive evaluation. 

On the other hand, there are several concerns raised by the reviewers. 

(i) Over-claiming. The reviewers felt that the main statement that RELICv2 for the first time beats supervised learning is a bit over-claiming. In particular, the reviewers pointed out that the supervised learning baseline compared in this paper comes from Pytorch's official implementation some time ago. Recently, the supervised performance has been significantly increased with better augmentation and training recipes. Therefore, a fair comparison with the up-to-date supervised baseline is needed to fully support the claim in the paper.

(ii) Limited technical novelty. While the performance of RELICv2 is strong, all reviewers felt that the key techniques of background masking and multiple crops were already proposed in the previous work.

(iii) Marginal performance contribution from background masking. The reviewers pointed out that while background masking is stated as one of the key techniques, the ablation study shows that it only brings in marginal improvements; the performance improvements mainly come from multi-crop augmentation at scale. The authors’ response confirmed this on the ImageNet classification task, but mentioned that the improvements from background masking become significant on the robustness and OOD generalization downstream tasks. The reviewers were concerned about the imprecise statement in the paper.
 
(iv) The paper claims that the two strategies are general purpose. The reviewers were interested in the performance when the strategies are paired with Vision Transformers. These results were not provided in the response due to time constraint.

At the end of the discussion phase including the AC-review meeting, there is a lack of a strong champion of the paper. The area chairs found that the weaknesses raised by the reviewers are valid and weighted the paper's weaknesses over the current strengths. Therefore, the area chairs cannot recommend the acceptance of the current paper. But the area chairs believe the paper could be significantly improved by (i) revisiting several statements such as beating supervised learning and the precise contribution of background masking, and (2) including additional experiments such as comparing with up-to-date supervised baseline implementation and using Vision Transformers.
justification_for_why_not_higher_score: There is no strong champion of the paper. The area chairs found that the weaknesses raised by the reviewers are valid and weighted the paper's weaknesses over the current strengths. Therefore, the area chairs cannot recommend the acceptance of the current paper. But the area chairs believe the paper will be significantly improved by (i) revisiting several statements such as beating supervised learning and the precise contribution of background masking, and (2) including additional experiments such as comparing with up-to-date supervised baseline implementation and using Vision Transformers.
justification_for_why_not_lower_score: N/A.
Decision: Reject