Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: 
Summary:
This paper explores solving the masking augmentation problem in convolutional neural networks. This topic is very important. The authors' method is also very clever. However, unfortunately, its performance is not satisfactory. Moreover, it does not solve the more important problem of masking image modeling. Therefore, this article is a borderline article, leaning towards being rejected. If the authors can answer my important questions, I will consider raising the rating. If this article can solve the problem of masking image modeling, I will give it a high score.

strength_and_weaknesses: 
(Positive) The subject studied in this paper is important. Exploring the role of masking as augmentation in self-supervised learning of convolutional networks is encouraged.

(Positive) After analysis, the authors believe that convolutional networks cannot correctly ignore the null information of masked areas through the attention mechanism like VIT. This is very reasonable.


(Positive) The authors argue that masking brings artificial edges, which will hurt the learning of convolutional neural networks. This is very reasonable.


(Negative) When the authors discuss Siamese networks, their formulation ignores the commonly used and very important projectors in SSL algorithms. Please add a projector to the formulation.

(Positive) Regarding the principles that augmentation needs to adhere to in self-supervised learning, the author's conclusion is correct: Preventing easy solutions is very important; the augmentations are hoped to be class-aware.


(Positive) The authors use high-pass filtering to suppress the null information in the masked area, which is very clever.


(Positive) The authors use high-pass filtering to suppress artificial edges, which is also very clever.


(Negative) However, in the masking strategy, the author uses a noisy mask. I think this contradicts the motivation of the authors. We all know that noise masks, especially Gaussian noise, bring a lot of high-frequency signals. High-pass filtering preserves these signals. The authors claim that this kind of noisy mask is useful, isn't that slapping one's own face?


(Positive) The Channel-Independent Mask proposed by the authors is a bit interesting.

(Negative) The caption "Few-shot Semi-supervised Learning on ImageNet-1K" in Table 1 is misleading. Also included here is an evaluation of the very important linear probing in SSL. Please modify this caption.


(Negative) The proposed method has obvious gains on SimCLR. But the gain is not obvious on BYOL.

(Negative) The application of the proposed method in ViT even leads to performance degradation, which is very worrying.

(Negative) In the Object detection and instance segmentation tasks, the authors are missing one of the most important comparison objects, which is the baseline on which the authors' method is based. I'm guessing it could be BYOL or SimCLR. Given that the authors' method doesn't gain much, I'm worried that the authors' method won't outperform this baseline.


(Negative) The authors have repeatedly emphasized that masking is the most general augmentation. This is worrying. This expression is too subjective.


(Negative) In fact, the biggest challenge of Masking augmentation in convolutional networks is masked image modeling. Unfortunately, after reading the whole article, I found that the authors' method does not solve this problem. This is quite disappointing for readers.



clarity,_quality,_novelty_and_reproducibility: 
The quality and clarity are median. The originality is good.

summary_of_the_review: 
See "Summary Of The Paper." This article is a borderline article, leaning towards being rejected. If the authors can answer my important questions, I will consider raising the rating. If this article can solve the problem of masking image modeling, I will give it a high score.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposed a pre-processing method to enable the masking strategy available for siamese networks with ConvNet backbone under self-supervised settings. It is motivated by performing random masking would cause missing information and parasitic edge issues. Also, systematic masking strategies are studied, including noisy mask, channel-independent mask (RGB), focal mask, and asymmetric augmentation. Extensive experiments on various tasks are conducted, including semi-supervised classification in IN-1K, transferred image classification results on iNaturalist and Places-205, and object detection & instance segmentation on VOC & COCO. The results show consistent improvements in combining the proposed processing method, high-pass filter, and masking strategies.
strength_and_weaknesses: This paper makes the masking strategies available for convolutional-based siamese networks for SSL. The color histogram and convolutional visualization observation about the masking augmentation causing missing information and parasitic edges are interesting. The systematic studying of masking strategies can offer a way for other researchers to follow. Overall the paper is very easy to follow.

The experiments are conducted thoroughly, including various tasks (semi-supervised classification, transferred image classification, object detection, and instance segmentation) and datasets (IN-1K, iNaturalist, Places-205, VOC, and COCO). Results show the proposed method helps the standard SSL methods achieve higher performance. The related ablation studies are included to prove the effectiveness of different masking strategies. 

The proposed preprocessing method may reduce the information in the image, which will cause the performance drop. In the current manuscript, all the comparisons include both the preprocessing methods and the comprehensive masking strategies; the reviewer is not clear if the preprocessing method brings improvements or masking strategies bring improvements. It would be better to test SimCLR, SimCLR + MSCN, BYOL, and BYOL + MSCN performance with simple/standard masking strategies. 

How does the proposed processing method change the color histogram and learned convolution kernels, as shown in Figure 1? Since the proposed method is motivated by those observations, it would be more self-contained to include the updates once introducing the new methods. 
clarity,_quality,_novelty_and_reproducibility: The clarity is good due to the simplicity of the proposed method and tricks. Also, it should be easy to replicate the results. Regarding the novelty, the reviewer is unsure if all the proposed components meet the bar of ICLR and need to discuss with other reviewers and the AC. 
summary_of_the_review: The reviewer feels that all the empirical studies presented in this paper will likely benefit the relevant community. However, the proposed masking strategies still heavily involve human priors. Also, the reviewer is unsure if the improvements come from introducing the preprocessing method or the hybrid masking strategies.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents a general masking strategy for visual representation learning with siamese neural network architectures. The approach is applicable to CNN-based or ViT-based neural networks.
It consists in pre-processing the input with a high-pass filter and then doing different augmentations that mask out parts of the input images with a certain probability, inspired from the natural language processing domain.
Experimental results have been applied to different baseline models, notably to SimCLR and BYOL, and show an improved performance using the pre-trained models for image classification on ImageNet, and transfer lerning for classification, detection and instance segmentation on other datasets.

strength_and_weaknesses: Strengths:
- Good experimental results
- Genericity of the approach

Weaknesses:
- The description of the approach is unclear at some places
- Some of the results are not clearly presented


clarity,_quality,_novelty_and_reproducibility: Overall the paper lacks some clarity in the explanations, and also the results are not clearly presented.

More specifically:
- What is the effect of the high-pass filter on the performance with non-masked images?
- There may be other ways of masking without introducing spurious edges.
- Figure 4 is not clear. What do you mean by "off-the-shelf ConvNet encoder"? The joint embedding loss is not detailed. Some details are missing on how the different (positive/negative) pairs are formed (within the batches).
- The relation to Assran et al. (2022) and Baevski et al. (2022) is not clear. Can MSCN be applied to ViT architectures (and be compared to MSN)?
- The text in section 6.1. states that the approach performs slightly worse for BYOL but from Table 1 this seems not to be the case.
- For Table 2, what is the task and dataset?
- It is not obvious how to interpret and read Table 4, and what to conclude. The underlining is somewhat arbitrary and confusing.
- The standard deviations are not presented in the results tables. I presume the results correspond to only one run.


summary_of_the_review: The presented method is interesting and relatively novel although largely inspired from other papers using masking. The results show the benefit of the proposed approach for self-supervised representation learning with general siamese neural network architectures. But the presented results lack some detail and rigor.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper explores different masking augmentation strategies for self-supervised learning. The reviewers have mixed opinions on the paper, with some praising its importance and clever methods, but others noting that its performance is not satisfactory. The reviewers also raise concerns about the quality and clarity of the paper. Overall, the paper needs to be further improved and incorporated review feedback in order to meet the acceptance bar of ICLR.
justification_for_why_not_higher_score: There are several unaddressed issues in the reviews, such as the experimental results being not very satisfactory, and the paper writing should be further improved.
Decision: Reject