Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes ConvMAE, a hybrid convolution-transformer architecture that is friendly to MAE-like pre-training. MAE was originally proposed with ViT, and due to omitted mask tokens in the backbone encoder, MAE is not trivially extensible to convolutional networks. The work extends MAE by resorting to the hybrid design of first using convolutions, and then using transformers. The masking is done block-wise (at the resolution of the transformer); and masked convolutions are used to avoid potential cheating. Extensive experiments are done on ImageNet classification, object detection, semantic segmentation, video classification. Various ablation analysis is also provided. 
strengths_and_weaknesses: (+) Self-supervised learning, especially masked auto-encoding for images is an emerging topic in computer vision. A breakthrough in this direction can bear huge significance. The work aims at fixing the limitation of MAE by introducing an hybrid architecture of convolutions and transformers, which is definitely important and relevant to the NeurIPS audience.

(+) The paper is well written, and is clear enough for readers to follow through with good illustrations. 

(+) The experiments are extensive and conclusive. The downstream transfers include image classification, object detection, semantic segmentation, and even video understanding is involved (which by itself could be an independent investigation). The ablations and the conclusions are also covering most of the things I can think of -- a solid paper clearly with a lot of hard work behind the scene.

(-) I think the "Conv" part of "ConvMAE" is an over-emphasis. The architecture only has 4 conv layers in the bottom of the network, while it has 11 transformer blocks for the base model (ViT-B has 12 blocks in total). So my current understanding is that ConvMAE has a similar architecture as in:

Xiao, Tete, et al. "Early convolutions help transformers see better." Advances in Neural Information Processing Systems 34 (2021): 30392-30400.

This means the majority of the architecture is still transformers, and in this regard, the difference/significance over original MAE is not that salient. This is the biggest concern about the paper -- it has a risk of over-sell with the term "Conv" in it.

(-) One minor concern is about the scalability of ConvMAE. The paper is highly focused on the model size of the base model. It is unclear the benefit of ConvMAE can still hold when the model size further scales up, as shown in pure ViT-based MAE.

(-) Some minor typos need to be fixed with proof-reading:, e.g., should define what is FOV at page 2, and the mask ratio should be 75% instead of 25% for MAE if I recall correctly.

questions: - Please address the concerns mentioned above, especially the first point. 
limitations: I do not see potential negative societal impact concern. The paper also points this out at the end, which is adequate to me.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a self-supervised framework using a hybrid convolution-transformer architecture, to obtain multi-scale, hierarchical representations. Masked convolution is introduced to prevent information leakage in convolution blocks and block-wise masking strategy is applied to improve computational efficiency. The resulting model achieves competitive performances in image classification and dense prediction tasks such as object detection. 
strengths_and_weaknesses: strengths: 
1. This work effectively extends the self-supervised MAE framework to the hierarchical, convolution-transformer hybrid architecture. 
2. The resulting model outperforms existing self-supervised models in classification and dense prediction tasks. 
questions: How would the duration of the pre-training epochs impact the backbone performances? How much performance drop if changing to shorter training epochs, such as reducing it from 1600 to 800 epochs?
limitations: adequate
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposed a new self-supervised learning framework by integrating hybrid convolution-transformer architectures and masked convolution into the masked auto-encoders. The proposed method can achieve computational efficiency and low pretraining-finetuning gap at the same. Extensive experiments on several computer vision tasks demonstrate the effectiveness of the proposed method.
strengths_and_weaknesses: __Strengths__

- The paper is well written and easy to follow. Sufficient technique details are provided.
- The proposed method is well motivated and simple. Several key components are proposed to address heavy computational cost and pretraining-finetuning discrepancy.
- The proposed method is flexible and can be applied in both image classification and object detection.

__Weaknesses__

- It seems hybrid convolution-transformer architectures have been explored in previous works but show how very similar performance to MAE (Lines 45-47). Why the proposed method can make them work for MAE? The differences from previous work and the contribution of the paper remains vague.
- Some parts of the method are not clearly illustrated. For example, in “Block-wise Masking with Masked Convolutions”, the authors state that “Uniformly masking stage-1 input tokens would cause all tokens of stage-3 to have partially visible information and requires keeping all stage-3 tokens”. Why the proposed method can address this issue? What is the key idea of the proposed method?
- The required training epochs vary from different methods. I wonder whether the proposed method can still outperform others under the same training epochs.

__Post Rebuttal__

I thank the authors for their response. Most of my concerns have been addressed. I increased my rating and recommend acceptance for this paper.
questions: See "Weaknesses".
limitations: Yes.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper addresses the difficulty of applying MAE training with convolutional layers. The proposed ConvMAE adopts masked convolutions in the early stage of convolutional layers by applying convolution on the masked featuremaps. In this way, the information leak is prevented. With the proposed ConvMAE training, the ViT with early convolutional layers can benefit from the MAE training and achieved better transfer learning results comparing to standard ViT.  It achieves superior performance on ImageNet & MS-COCO.
strengths_and_weaknesses: 1. Novel training strategy to enable MAE training for models with convolutional layers.
2. Strong performance on various transfer learning tasks. 


questions: Would the masked featuremap introduce some bias or artifacts to the training along the mask edges? It may be helpful to find a way to alleviate the boundary issue.
limitations: The comparison with the MAE training with standard ViT backbones may be unfair, due to introducing extra computation cost with the convolutional layers. It would be helpful to further break down the improvements.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper starts with the hypothesis that a multiscale hybrid convolution-transformer can learn better representations using masked inputs than vanilla ViTs. The original masking scheme proposed in the MAE paper can be computationally prohibitive when directly applied to hybrid models. This paper presents a multiscale block-wise masking strategy with masked convolutions to efficiently train a hybrid transformer-convolutional model for representation learning. The paper shared a broad range of empirical results on classification, detection, segmentation, and video understanding tasks to show the effectiveness of the proposed technique.
strengths_and_weaknesses: Originality: The novelty of the paper lies in its proposed multi-scale hybrid convolution-transformer encoder, which can generate hierarchical representations and possibly exceed the performance of vanilla ViTs. The idea of hybrid models already exists in multiple pieces of literature (CoAtNet, Early Convolutions etc.). Masked convolutions were introduced in the PixelRNN paper (https://arxiv.org/pdf/1601.06759.pdf). The strength of this paper is in its novel combination of existing ideas to produce a very simple hybrid framework that effectively combines the strength of convolutions and transformers.

I also like the idea of performing masking at the late stage and then progressively upsampling the mask to larger resolutions to avoid the requirement of keeping all tokens in stage 3.

The proposed setup naturally generates hierarchical representations and fits nicely with Feature Pyramid Networks. It is a nice way to generate a feature pyramid with local context via convolutions and global context using transformers.

Quality: The paper primarily describes experiments using ViT-B scale networks. It covers a broad set of vision tasks but it does not cover scale. It would be nice to see whether the proposed scheme continues to outperform existing masking techniques for larger models. There is also limited runtime comparison with existing techniques.

The paper shares very informative results of ablation experiments comparing random masking, regular convolutions, multi-scale decoders etc.

Clarity: The paper is very well written, with a nice flow, and explains the concepts with ease. 
Nit: line 56 pretraing -> pretraining 

Significance: The paper proposes a simple and effective hybrid convolution-transformer encoder, which naturally generates hierarchical representations from an image and outperforms a number of existing techniques. 

questions: 1. Did you experiment with larger models of the order of ViT-L and ViT-H? How do these results scale?
2. Why did you generate limited data points for VideoConvMAE and VideoConvMAE-multiscale in figure 3?
3. Why did you not include the CoAtNet results in the experiments?
limitations: Yes, authors adequately addressed the limitations.
review_title: null

Meta-review: metareview: The reviewers are positive about this submission initially. After the authors' rebuttal, one reviewer pointed out that the name `ConvMAE' is not proper to describe the current work. The authors respond by claiming using an alternative name, which is acknowledged by the reviewer. Overall, all the reviewers stand positive for this work and AC stands with the reviewers. The authors shall take the suggestions from the reviewers to further polish the current work in the camera-ready submissions. 
confidence: Certain
Decision: Accept