Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes an alternative to multi-crop data augmentation. Multi-crop augmentation has been shown to provide consistent improvements in representation learning. This is achieved by using 2 "large crop" views that capture "global" information of the image, and a number of much smaller crops that capture the "local" information. 

The authors argue that naive small crops tend to lead to lowered mutual information between views (although they do not use the term "mutual information"), thereby leading to suboptimal performance. The authors argue that by creating synthetic backgrounds for the small crops, mutual information is reduced, thereby leading to stronger (more discriminative) features. 

A comprehensive set of experiments shows the benefits of their approach on classification and transfer learning tasks.
strength_and_weaknesses: Strengths:

1. The method is simple, although possibly details of implementations could be a little complicated (e.g. to ensure that features are re-arranged to correlate small and large crop views). 

2. Strong empirical results on various tasks such as classification and linear transfer.

Weaknesses:

There is a very limited understanding of the idea other than "diverse backgrounds". What exactly improves the features? Did the authors try ablation such as simply adding background noise to the views? 

I'm curious how this approach could be applied to vision transformers, since those architectures are increasingly more common.
clarity,_quality,_novelty_and_reproducibility: Clarify: The paper is mostly clearly written.

Quality: Empirical results are comprehensive and show definite improvements over multi-crop.

Reproducibility: It is a simple and therefore easy to reproduce method.

Novelty: The idea is straightforward, and so I may not consider it particularly "novel". However, a simple, useful idea is valuable to the community.
summary_of_the_review: The paper has a very simple idea supported by a lot of empirical evidence. Hence I lean towards acceptance. However, there is a very limited understanding of why this approach works.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Authors proposed a mosaic augmentation strategy for better representation learning in a self-supervised learning setup. The proposed MosRep strategy was applied to both MoCo-v2 and BYOL frameworks improving it's performance by a considerable margin on ImageNet-100 and ImageNet-1k datasets. The approach was further evaluated for transfer learning for both object detection and instance segmentation using MS COCO and Cityscapes with positive results.
strength_and_weaknesses: -- Strengths:
- The proposed method is reasonable and results show imprevements on performance from it
- The approach was tested on multiple benchmark datasets

-- Weaknesses:
- Novelty is limited
- Performance evaluation is focused on camparing to standard BYOL and MoCo-v2 after adding the proposed augmentation strategy. However, it does not outperform more recent SSL approaches.
clarity,_quality,_novelty_and_reproducibility: The paper is well written and easy un follow. It's novelty is limited; it build on top of the multi-crop strategy from SwAV with a few interesting improvements. Enough implementation details are provided for at least partial reproducibility.
summary_of_the_review: The proposed mosaic augmentation strategy and MosRep framework appears to be a good augmentation strategy to learn good representation in MoCo-v2 and BYOL contrastive learning frameworks. The novelty is limited, and there is no comparison with most recent SOTA SSL approaches but the results are promising for multiple tasks.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a simple modification of multi-view augmentation for use with representation learning, as well as an additional training framework to go with it. The main idea is to create a richer set of background with higher resolution than available with the multi-crop strategy. Because the augmented data is mosaiced, the author apply ROI-align to extract only relevant ROI of the feature map before transforming it to the embedding space. The contrastive loss is applied to the appropriate part of the embedding from the standard view of the input image as well as from its mosaiced version. Experiments are given and demonstrate 0.5-1.5% improvement over the multi-crop technique.
strength_and_weaknesses: Strength:
The technique is simple and is well-motivated. The multi-crop strategy leads to low-resolution negative sample, which can be of limited use if the background in the image are textured and include a lot of visual detail. By grabbing higher resolution patch and place it into the view, the author have achieve a more challenging background for the model to learn from.
The improvement over the multi-crop technique is appreciable. The difference ranges from 0.5-1.5 point, which is not large, but appreciable.
The experiments are comprehensive, and I find enough details are given to be reproducible. The comparison is fair, as the author retrain the model used for comparison. It may have been useful to actually tuned hyperparameter per augmentation strategy as well. As written, it seems like the hyperparameters were pre-selected and applied across board.

Weakness: 
I find the clarity of writing to be the main issue here. It would be much simpler to explain the augmentation algorithm as well as the representation learning methodology using a diagram. Using equations is prone to error, and also harder to understand. As written, there are some errors (index i is overloaded: i-th mosaic, and also i-th image in eq 8). This makes a text that is already hard to understand even harder.
No code was promised, following the above point, without a clearly explained method, it is hard to reimplement the work here. I believe there is no promise of code release here, so I think it can be challenging for other researcher to try to reimplement this exactly.
Lastly, the paper can be stronger if the ablation study is used to justify technical choices. For example, one main motivation for this work is to have a more complex background. It may be interesting for the author to try to vary the resolution of the background patch in the mosaic, and see if there is any change in model performance. As presented, it seems more like a tuning exercise for alpha and theta.
clarity,_quality,_novelty_and_reproducibility: Please see my strengths and weaknesses section. Overall, I think the clarity and reproducibility can be improved especially around the method section.
summary_of_the_review: I think there is enough contribution here in the paper. However, the clarity and reproducibility concern limited the impact to the research community. I find that to address this will require a major revision of the text and figure, but a code release may alleviate this somewhat.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposed a novel mosaic version of multi-view augmentation for representation learning by using 2 “large crop” views that capture “global” information of the image, and much smaller crops that capture the “local” information. The mutual information between different views is effectively reduced by creating synthetic backgrounds for the small crops. Experiments demonstrated considerable improvement (0.5~1.5%) over the multi-crop technique on classification and transfer learning tasks.

Two reviewers (R# ed1B and R# hAGo) gave very positive reviews. They both feel that the method is simple, well-motivated and effective, and strong empirical results were obtained. They expressed the weakness of clarity of writing (the authors added the pseudo-code of the mosaic augmentation strategy in the appendix), the limited understanding of the idea e.g. what exactly improves the features and what about simply adding background noise to the views (the authors addressed it with additional ablation studies and convincing results). Both reviewers were content with the rebuttal.

Reviewer #rkci was a bit negative with the concern that the novelty is limited. Unfortunately R #rkci did not respond to the author's comments nor AC’s request. The AC does not agree with the comment that the novelty is limited because “it builds on top of the multi-crop strategy from SwAV with a few interesting improvements.” The authors addressed the R #rkci’s question of “not outperforming more recent SSL approaches” with very convincing numbers on DINO.

The AC feels that the paper obtained strong results by a novel augmentation method with good motivation and justification. Both the methodology and results have broad impact to the broad ML community.

justification_for_why_not_higher_score: It's possible to boost the paper to oral but the main reason of not higher is the scope of the paper. Although the paper has impact on the broad ML community, a better data augmentation strategy is still an improvement to an existing module of the ML pipeline.
justification_for_why_not_lower_score: The findings of this paper is not only highly interesting to the self-supervised learning community, but to the broad community where data augmentation is needed.
Decision: Accept: notable-top-25%