Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper works on the pretext task learning a subfield of self-supervised learning. It follows the track of predicting the original image given random non-overlapped masks. To make it work on non-transformer architectures, the authors propose to use a hierarchical and sparse convolution network to pre-train the network with the reconstruction loss on the masked region. To show the advantages of the paper, the authors provide extensive comparisons with other baseline methods on several downstream tasks. The authors also conduct the ablation study to show the effectiveness empirically. 
strength_and_weaknesses: Pro:
1. This paper solves the patch mask on the ConvNet by using a "sparse" convolution operation. At the same time, the authors also incorporate the U-Net-like design, which passes the encoding mask to the decoding part to preserve the mask information.

2. This paper provides strong performance across all the tasks and architecture in a self-supervised learning setting.


Cons:

1. The authors seem to overclaim several two important contributions. 

a. Removing information of masked parts is difficult for ConvNet. This has been identified by submanifold sparse convolution, which aims to preserve the structure of the unmasked regions. Therefore, it should not be a difficulty anymore.

b. The author claims in the experiment that their work proves that the convnet is not inferior to the transformers. This seems wrong to me. I do think ConvNext has already shown that.

2. In Fig.1, since the second row is the histogram, I can understand that using zeros as masks could increase the number of zeros, but I do not understand how the sparsely dropping does not decrease the frequency of other pixels (Fig. 1c).

3. Table 5 shows that even without the hierarchical encoding, ConvNext outperforms the transform architectures. Indeed, the hierarchical encoding works, but not impressive. The performance gained mostly due to ConvNext itself.
 



clarity,_quality,_novelty_and_reproducibility: Clarity, the paper is well-organized and easy to read.


Novelty: The originality and novelty of this paper are very limited. This paper follows the way of MAE, SimMIM, and BEiT.  Compared to BEiT, MAE, and SimMIM prove that simply reconstructing the mask region (l_1 and l_2 norm) will be good enough for the pre-training. This paper transfers the same idea to the ConvNets by incorporating sparse convolution. Therefore, the contribution to the community is much less than the published ones. 

Reproducibility: It could be able to reproduce the paper, however, some details are not clear.
summary_of_the_review: Overall, I do appreciate the messages that this paper sends to us. It reinforces the claim of ConvNext that convolution networks can be better than transformers. However, the proposed framework seems very engineering oritented.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors propose a method of pre-training convolutional neural networks via masking. They mask parts of the input image, take a convolutional network as encoder, and add a U-net style decoder, with which they predict the parts which were masked out. After pre-training, the decoder is removed, and the encoder is fine-tuned for downstream tasks, similar to recent Transformer-based approaches.

The approach is experimentally validated on ImageNet classification and COCO object detection tasks and shows improvements over other contrastive and masked pre-training methods with models of similar size in terms of the number of parameters and computational complexity.
strength_and_weaknesses: Strengths:
- well written and easy to follow
- strong results on using networks pretrained with the proposed method for object detection, which could be attributed to the tested object detection architecture, Mask R-CNN, designed for convolutional networks.
- detailed ablation study. Would be interesting to see if object detection sees improvements when using zero-out masking mode.
- the provided PyTorch code is useful to have better understanding of the approach

Weaknesses
- the authors do not show results on large backbones with 86+ classification accuracy. It is not clear if the approach does not scale to larger backbones, or the authors were limited by computational resources. Including such results would be very useful for future work.
- no statistical significance evaluation, which makes validating the claims difficult, especially in the ablation study. What is the variance in the experiment?
clarity,_quality,_novelty_and_reproducibility: The approach is simple, the authors describe it in detail and provide code of the decoder. It should not be difficult to reproduce the results.
Questions to the authors:
- How well is SparK able to reconstruct images compared to Transformer? Do they reach similar MSE test error?
- How well does the method work with linear probing? An evaluation in linear probing mode is missing, even though it is common for contrastive methods.
summary_of_the_review: I suggest accept, seems like a good paper. It would be beneficial to have linear probing results in addition to fine-tuning.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a masked modeling method for convolutional vision backbones. Instead of simply zeroing out the masked regions in the input images, the method exploits the submanifold sparse convolution to handle irregular masked input. The method is evaluated on multiple popular vision tasks and backbone models.
strength_and_weaknesses: Strengths:

- Transferring the successful experiences of MIM from Transformers to convolutional models is an important and potentially impactful topic. This paper exhibits some promising results of applying MIM to ConvNeXt and ResNet.

- The idea of using the submanifold sparse convolution to handle irregular masked input is well-motivated and new. The hierarchical encoding and decoding design also look interesting.

Weaknesses:

- The motivation shown in Figure 1 may be straightforward but a bit misleading. I think the main advantage of the proposed method based on the submanifold sparse convolution is that the mask patterns are kept in all intermediate features. If we use the "vanilla sparseconv" mentioned in Sec 2.3, the model will also be difficult to generalize to raw input even if the input data distributions are close. 

- There are several missing references that should be discussed. [r1] is a widely used sparseconv framework in 3D understanding. [r2] is an existing method that uses sparseconv to accelerate 2D image recognition. 

[r1] 4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks, CVPR 2019

[r2] Dynamic convolutions: Exploiting spatial sparsity for faster inference, CVPR 2020.

- Many important implementation details are missing. Sec. 4.1 introduces the details of the proposed pre-training method. However, there is no detail about the fine-tuning process for ImageNet and downstream tasks, which is very important information to compare different pre-training methods and reproduce the results. For example, MAE and SimMIM report the results with 50/100-epoch finetuning. Can SparK models achieve competitive results on ImageNet with fewer finetuning epochs than standard training (300 epochs)? Are extra tricks used during fine-tuning?  

- The discussions in Sec 4.2 emphasize the encoding efficiency of the proposed method. From my experience, sparseconv is usually much slower than standard convolutions with similar theoretical complexity since the standard convolution is well-supported and highly optimized on GPUs. I guess 60% masked patches may not lead to significantly higher training speed than models without masked input in practice. The encoding cost shown in Table 1 is very ambiguous. I think it would be better to report the wall-clock time of the whole system and training process like the MAE paper to clearly compare different methods.

- The comparisons of small backbones are unfair.  ConvNeXt-S have doubled computational costs/parameters compared to ViT-S/Swin-T.

- Why are the detection frameworks used for small and base backbones in Table 2 different? Besides, it would be better to report the performance of the baseline supervised ConvNeXt models in both settings.

- According to Table 4, it seems ConvNeXt-L cannot outperform MAE (ViT-L). It would be better to add more discussions and analyses of this. 

Minor issues:

- How about the results if we zero out features of the masked regions (before all 7x7 convs) or replace them with the mask embeddings instead of zeroing out input? Considering sparseconv requires extra libraries/extensions and might not be easy to use, this implementation can be a competitive baseline.

- The formats of citations are different in some sentences. For example, in Section 2.2, there are both "Bao et al. (2021)" and "(Devlin et al., 2018)".
clarity,_quality,_novelty_and_reproducibility: - **Clarity:** The paper overall is easy to read and follow.

- **Quality:** Good.

- **Novelty:** Although the idea of using sparseconv to handle irregular input is not new for both 2D and 3D tasks, it seems new to use it for improving MIM of convnets.

- **Reproducibility:** Code is not attached in the submission. Many important details are missing. It would be difficult to reproduce the results based on the information provided in the current submission.

summary_of_the_review: The method presented in this paper is interesting and may be useful in future research and applications. However, there are still many important issues that require further clarification and discussion including missing comparisons, baselines and experimental details. I would upgrade my rating if the above-mentioned issues are addressed.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes an approach for masked image modeling (MIM) pre-training with ConvNets, based on sparse convolutions. The proposed method is validated on multiple tasks and model architectures and shows strong results compared to relevant baselines.

The opinions of the reviewers about the paper are somewhat divergent even after the authors' rebuttal and discussion. Based on the reviews, rebuttals, and the paper itself, the main points are as follows:

Pros:
1. New and reasonable technical approach
2. Clear presentation
3. Extensive experiments on several tasks and model architectures, showing strong results compared to relevant baselines. A good ablation study.

Cons:
1. No results for very large models
2. Doubts about wallclock computational efficiency
3. Inpainting has been done before

Con (1) is understandable since training those very large models takes an excessive amount of computational resources. Con (3) in my opinion is not really a problem - yes inpainting is not a new task, but self-supervised pre-training results using inpainting have been quite weak so far and there are no strong baselines there to compare to. Regarding con (2) - indeed would be nice to clearly report wallclock training/inference times for sparse vs usual convolution.

Overall, the paper has a clear message and supports it convincingly with extensive experiments. Therefore, I recommend acceptance.
justification_for_why_not_higher_score: The results are nice, but not that much better than baselines
justification_for_why_not_lower_score: The paper has a clear message and supports it convincingly with extensive experiments. The fact that masked modeling works for convolutional models can be of general interest, so spotlight may be justified.
Decision: Accept: notable-top-25%