Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work proposes a variant of BERT-like pretraining method for computer vision (the so-called Masked Image Modeling). The method can generalize well to both vision transformers and CNNs, thus is described as architecture agnostic.

The main contributions of this work are:
- using the mean color values to fill the corrupted pixels, which makes the self-supervised pipeline suitable for different neural network architectures;
- giving a nice insight that the middle-order interactions are important for visual representation, and making two improvements to standard BERT algorithm to facilitate their learning (i. masking intermediate features and ii. introducing supervision in the frequency domain);
- and providing some empirical evidence that supports its validity.

&nbsp;
strength_and_weaknesses: **[strengths]**
- [S1] This work presents a *fresh* perspective to rethink the self-supervised learning in computer vision, say, the *middle-order* interaction. The "interaction strength" mentioned in the paper is a useful indicator to show how good the model is at modeling long-range dependency, which is actually considered to be a valuable aspect in advancing pre-training for NLP [1,2].
- [S2] The authors have put considerable effort into how to fairly compare the established algorithms, which is appreciated and allows the hypotheses presented in their paper to be fully tested. The ablation experiments also demonstrate the validity of different components in their method.


&nbsp;


**[weakness]**

There are some parts of the article that are not very easy for the reader to follow, which could be the main weakness.
I would suggest authors to explain more about the following aspects (which I find enlightening) and eventually add them to their manuscript or appendix:
  - [W1] In the Introduction, the authors claim that "it is not straightforward to directly apply mask token to CNNs". Can we replace the masked part (e.g., 16x16x3=768 pixels) with a "mask token" (e.g., 768 learnable scalars) like BERT does? This is a straightforward way to mask. Or would it be more appropriate to describe this way as "underperforming" than "not straightforward"? (considering the mediocre performance of MAE, SimMIM, etc. in Tab. 1 of the paper)
  - [W2] "MAE takes the reorganized the unmasked input patches of 112x112 as the input", can the authors explain more on this? I wonder if a substitution like the above (768 pixels to a 768-dimensional learnable vector) or some other operation has been performed.
  - [W3] According to the analysis in Sec. 3.2, middle-order interactions seem to manifest a medium- or long-range inter-patch dependency. Why the authors say "middle-order interactions could be enhanced via guiding the network to learn features of certain frequencies"? Does "certain frequencies" refer to medium or high frequencies? Why does increasing the richness of these frequencies on feature maps can promote middle-order interactions?

[W4] In addition, there appears to be some related work that has not been adequately discussed. See the "Clarity, Quality, Novelty And Reproducibility".

&nbsp;

**[open questions]**

[O1] The methodological improvements to BERT-like pre-training in this paper seems to be incremental. 
Considering that the authors propose some quantitative metrics to describe middle-order operations, is it possible to design a more principled algorithm to explicitly facilitate such middle-order interactions? I believe the insights on middle-order are valuable, but the solutions proposed in the article do not seem to fully exploit their values.

&nbsp;

------------------
[1] Jawahar, Ganesh, Benoît Sagot, and Djamé Seddah. "What Does BERT Learn about the Structure of Language?." Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 2019.

[2] Xu, Jiacheng, et al. "Discourse-Aware Neural Extractive Text Summarization." Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 2020.

&nbsp;

clarity,_quality,_novelty_and_reproducibility: **[missing discussions that may weaken the novelty]**

In the Related Work section, the authors refer to the work CIM [3].
This is an Electra-like [4] method that fills the corrupted pixels via a small network instead of mean color values, and thus [4] claimed they are the first to demonstrates that both ViT and CNN can learn rich visual representations using a unified, non-Siamese framework.
Authors are encouraged to discuss A$^2$MIM and CIM in more detail, and to update some of the corresponding descriptions.
Although this would weaken the novelty of A$^2$MIM, the related works deserve to be discussed fairly and pertinently.


**[Reproducibility]**

I believe one can easily reproduce the main results in this work given the detailed experimental configurations and source codes.


------------------
[3] Fang, Yuxin, et al. "Corrupted image modeling for self-supervised visual pre-training." arXiv preprint arXiv:2202.03382 (2022).

[4] Clark, Kevin, et al. "Electra: Pre-training text encoders as discriminators rather than generators." arXiv preprint arXiv:2003.10555 (2020).
summary_of_the_review: Overall, I find this work provides some interesting insights, but the writing and presentation of the paper leaves much to be desired.
In addition, the methodological improvements to BERT pre-training in this paper seems to be trivial and piecemeal.
I hope to understand this article better in further communication with the authors and that may influence my judgment.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper studied the problem that MIM is compatible with the Transformer family but is incompatible with CNNs. To this end, it proposed an Architecture-Agnostic Masked Image Modeling framework (A2MIM) that is compatible with both Transformers and CNNs in a unified way. Specifically, this paper used RGB Mean as masked tokens and added mask tokens on the intermediate feature maps. A frequency domain reconstruction loss is also applied to improve the ability of learned models. Experiments are conducted on ImageNet-1K, downstream COCO detection and segmentation, and ADE20K dataset with ViT models and ResNet-50.
strength_and_weaknesses: Strengths:

    - MIM is an interesting problem/framework to study, and understanding how MIM-based methods perform well in the vision domain is important for this field.

    - The proposed method is clear to understand and is easy to follow by other researchers.

    - The experiments are extensive on ImageNet-1K, COCO, and ADE20K datasets.

Weaknesses:

   - Some statements are incorrect or even a little bit over-claimed, such as:

1)	In the abstract, the authors stated: “MIM primarily works for the Transformer family but is incompatible with CNNs.” And in the introduction section, this paper stated "To the best of our knowledge, we are the first to carry out MIM on CNNs that outperforms contrastive learning counterparts." This is not true as ConvNext [1] is adequate to handle MIM learning strategy. And this paper with CNN is actually similar to it when using patchified images with CNN.

[1] Liu, Zhuang, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. "A convnet for the 2020s." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11976-11986. 2022.

2)	“Based on this fact, we propose an Architecture-Agnostic Masked Image Modeling framework (A2MIM), which is compatible with both Transformers and CNNs in a unified way.” The proposed framework is basically similar to the regular MIM approach with trivial or not significant modifications (use RGB Mean as the Masked Tokens and add mask token on the intermediate feature map). Thus, the called "a unified way" seems incorrect. The used Fourier/Frequency domain is also not related to the core framework in this paper and can straightforwardly be applied in the vanilla MIM models.

3)	The authors stated that the proposed framework has “(a) No complex or non-generic designs are adopted to ensure compatibility with all network architectures. (b) Better middle order interactions between patches for more generalized feature extraction.” I think this is because the proposed framework is too close to the original MIM architecture.

4)	The authors claimed they “delved deep into MIM and answered the question of what exactly is learned during MIM pre-training.” This statement is very strong, however, after reading this paper carefully, I still did not get the insights or intuition about what MIM pre-training learns from the input data from the paper’s descriptions. Proper explorations and explanations are necessary to support this claim.

 - The performance in this paper is not competitive. As shown in Tables 1, 2, 4, etc., the improvement is fairly marginal (0%~0.2%). Considering that this paper used extra Fourier/Frequency domain reconstruction supervision/loss, I’m not sure whether the proposed strategy is truly effective or not.

 - The writing and organization of this paper can be improved. The used Fourier/Frequency domain reconstruction loss seems not related to the key approach of the architecture-agnostic masked image modeling framework, since it can also be applied to the regular MIM frameworks. Moreover, the insights of using this additional supervision are not clearly expressed. This part seems fragmented from others in the method.

clarity,_quality,_novelty_and_reproducibility: This paper is clearly presented, however, the originality of this work is slightly insufficient.
summary_of_the_review: This paper conducted sufficient experiments. However, the method itself is not novel with marginal improvement, the novelty and originality of the method are basically not very strong. Also, many statements in this paper are slightly over-claimed (details please refer to "Strength And Weaknesses"). Thus, I tend to reject.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper inspects MIM pre-training approaches and finds MIM essentially helps the model to learn better middle-order interactions between patches. Motivated by that, this paper proposes a novel MIM-based method, dubbed A$^2$MIM, that works well for both (small-sized) ConvNets & ViTs. Extensive experiments are conducted to verify the effectiveness of the proposed approach.
strength_and_weaknesses: ### Strength 
1. This paper provides a thoroughgoing study of MIM, and highlights the essence of MIM pre-training is to help models learn middle-order interactions between patches.
2. Based on the study of middle-order interactions of MIM, this paper proposes a novel approach that works well for both ConvNets as well as ViTs. What's more, I appreciate the management of technical details in this paper, such as ``Filling Masked Tokens with RGB Mean'' is technically sound. 
3. Extensive experiments on several downstream visual recognition tasks are conducted to verify the effectiveness of the proposed approach.
   
### Weaknesses
1. The 2nd paragraph's name of Sec. 2 Related Work is inaccurate.``Autoregressive Modeling'' means that the output depends linearly on its own *previous* values (e.g, GPTs), while masked modeling is *bidirectional* modeling.
2. I don't see a strong connection between Sec. 3.1 and the proposed approach.
3. The scalability of A$^2$MIM is unknown, which is a crucial property for pre-training.
clarity,_quality,_novelty_and_reproducibility: The quality, clarity and originality are sound.

summary_of_the_review: Overall, I think this is a solid and well-written paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper analyzes the essence of MIM and introduces a universal MIM method that can be applied to both CNNs and Transformers. There are several designs that together build the introduced method, including the mean RGB replacement, the intermediate mask, the frequency reconstruction targets and the HOG targets. Experimental results on both ResNet-50 and ViTs are shown.
strength_and_weaknesses: ### Strengths
1. The paper is well-organized and easy to follow.
2. The figures, tables, and visualizations indeed help to understand the author's point of view.
3. It is new to apply MIM on convolution neural networks.

### Weaknesses
1. I have the most concern about the motivation for this work. As we know, ViTs lack the capture of inductive bias and MIM solves this issue to some extent by enforcing visual context reasoning. However, CNNs are different, they are good at capturing inductive bias. Furthermore, as observed in the authors' results (Tab. 1), a lot of effort has been done to do MIM pre-training, however, the final results are almost the same as supervised counterparts. So is it a false proposition to apply MIM to CNNs?
2. I have another concern about the authors' declaration that MIM enables the network with a better feature extraction ability. As a common observation in previous MIM works, the linearly probing results of MIM-pretrained models are unsatisfactory, worse than the contrastive methods. MIM is known to provide transferable model parameters rather than extracting out-of-the-box features.
3. In Sec. 3.1, the authors analyze different augmentation methods, however, the setup is not fair. MIM is employed for pretraining while the others (e.g., CutMix) are for fine-tuning. 
4. There are many components in the introduced method, including the mean RGB replacement, the intermediate mask, the frequency reconstruction targets and the HOG targets. How much does each component affect?
clarity,_quality,_novelty_and_reproducibility: + **Clarity**: The paper is well written, however, I do not agree with several claims (see Weaknesses).
+ **Quality**: Comprehensive experiments are conducted, however, the premises should be verified, i.e., do we need MIM on CNNs?
+ **Novelty**: Applying MIM to CNNs is new.
+ **Reproducibility**: Implementation details are provided. 
summary_of_the_review: I tend to weakly reject this paper in the initial comments due to the unfounded premises. I am looking forward to the authors' opinions in the rebuttal period on why we need MIM for CNNs.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes a pre-training approach for image models based on masked image modeling (MIM) and applies it to vision transformers and ResNets. The method works a bit better than baselines. 

The reviewers are overall negative about the paper (and, unfortunately, they have not responded to the authors' rebuttals). Based on the reviews, the authors' rebuttals, and the paper itself, here are the key points about the paper:

Pros:
1. Thorough experiments with the selected architectures, including modest performance improvements over baselines and an ablation study
2. Reasonable technical improvements to MIM
3. An interesting narrative around patch interactions
4. One of the first applications of MIM to ConvNets

Cons:
1. The paper mixes a few things - ViT/ConvNet, patch interactions, performance-improving modifications to MIM - and somewhat fails to combine them convincingly or do a great job at any of them separately.  
 1a. The name suggests the paper is largely about ViT/ConvNet comparison and making MIM work for ConvNets. Yet, only one ConvNet architecture is studied (ResNet-50; there are some additional results in the rebuttal, but incorporating them would be a pretty big change)  
 1b. The proposed improvements to MIM do not seem to affect ConvNet more than ViT, so are orthogonal to the ViT/ConvNet question.  
 1c. The analysis of patch interactions is difficult to understand, and it is difficult to see what exactly it tells us. That randomly initialized and pre-trained models have different properties is not a big surprise. Results in the appendix comparing different pre-training methods seem curious but are not discussed in the main paper. Overall, this analysis may very well be useful and interesting, but from the papers in its current form it is difficult to understand.  
 1d. In particular, the connection between the analysis and the proposed modifications is not very clear - is it just a high-level intuitive connection or is there more to it? Can it be shown that the proposed modifications affect patch interactions?
2. The improvement over the SimMIM baseline is not huge <1% and often quite a bit less, depending on the setting.
3. Experiments only with a limited set of models (ViT-S/B, ResNet-50). Not critical, but results with more (and larger) models would be nice.

Overall, the paper has some merit and proposes some interesting modifications to MIM pre-training and some potentially interesting analysis, but it's a bit of a mixed bag and fails to build a coherent narrative and deliver clear and well-founded insights. While some interesting content may already be there, presenting it properly would take some major re-work of the paper, so I recommend rejection at this point, but encourage the authors to improve the paper and resubmit to a different venue.
justification_for_why_not_higher_score: The paper is a bit of a mixed bag and fails to build a coherent narrative and deliver clear and well-founded insights. While some interesting content may already be there, presenting it properly would take some major re-work of the paper, so I recommend rejection at this point.
Decision: Reject