Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper presents a cost-effective approach for masked image modeling (MIM) pretraining with hierarchical vision transformers. MIM has been proved effective for vision transformer pretraining, but hierarchical ViT faces a challenge in MIM pretraining that both visible and masked patches have to be involved, which greatly decreases the pretraining efficiency. This paper presents a solution which reduces the pretraining cost of Swin Transformer by around half while achieving on par performance. 
strengths_and_weaknesses: This paper addresses a relevant problem with a technically sound solution. The cost of MIM pretraining for hierarchical ViTs is indeed a painpoint in vision backbone learning. The proposed group window attention scheme is reasonable, and the evaluation on Swin Transformer is c. 

The paper claims that the proposed approach is for hierarchical ViTs, but the experiments are carried out only for Swin Transformer. It would be more convincing if the authors could show the results for at least another ViT. Or, the authors should consider not to claim the proposed approach to be generally applicable to hierarchical ViTs. 

The group window attention algorithm solves an optimization problem to determine the group size and the number of groups. The sole optimization target is the computation cost, leaving the pretrain quality completely unconsidered. Does the selection of group size affect the pretrain quality?

The authors use dynamic programming to solve the group partition problem, but does a simple and heuristic algorithm, such as g_s=max{w_i}, already suffices? The complexity of the dynamic programming algorithm is not well justified/ablated. 
questions: Please refer to the previous section. 
limitations: No potential negative societal impact is identified.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper first introduces to properly apply the MAE pre-training method on Swin Transformer backbone architecture. To adapt the visible-only encoder with local window attention in Swin, the paper proposes a new Group Window attention mechanism with Dynamic Programming to automatically tune the group size and local windows. The introduced method significantly reduces the pre-training hours of SimMIM on Swin Transformer with marginal performance drops.
strengths_and_weaknesses: Strengths:
+ Writing: The paper is generally well written and easy to follow. The story of green AI is interesting.
+ Contribution: It is significant to well apply the MAE-style pretraining method on hierarchical Vision Transformer architectures, especially benefitting dense prediction downstream tasks, e.g., detection and segmentation. It is the first work to investigate this problem so far as I know.

Weaknesses:

I am generally satisfied with this work but have several concerns about the experiments:
1. **Marginal improvements:** The improvements over training from scratch on ImageNet is marginal, i.e., 83.5 vs. 83.7. The pretraining hours are comparable (840 for training from scratch and 887 for this work), it may raise a question that if 83.7 can also be achieved by training from scratch for 887 hours. By the way, why not compare to SimMIM_224 on Swin-Base?
2. **Lack of ablation studies:** (i) I think it is important to implement this method on the vanilla MAE with ViT-Base to show the performance gaps caused by group window attention. (ii) I would like to see if the performance can be further improved with more training epochs. For example, for 800 epochs, SimMIM_192 takes double the training hours than this work but +0.3% performance gains. I want to know if this method can also achieve 84.0% with double training hours (800ep -> 1600ep).
3. **More model scales:** It is better to show the power of this method on other sizes of Swin Transformer, e.g., small, large, etc.



questions: I like the motivation and story of this work but would like to see more experiments that would make it more solid. Please see Weaknesses for details.
limitations: The authors discussed the limitation of the batch-wise masking scheme in the Appendix.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: In this paper, the authors propose a method to perform MAE style pre-training on Swin transformers. The main idea is to dynamically put the visible patches into groups and then perform attention within each group. To ensure that information doesn't "leak" out of each window, attention masking is performed in each group. This method can be efficient because of the Swin structure -- attention can only happen within a window, so the size of each group can be small and well-packed. The grouping is solved by dynamic programming to optimize FLOPs. The authors evaluate this approach on ImageNet and COCO and show that this method is efficient while being competitive in accuracy.
strengths_and_weaknesses: Strengths:
+ The 2x speedup looks promising and useful. Given the wide use of Swin transformers, an efficient way to pre-train with SSL can be quite impactful. 
+ The idea to perform grouping for fast MAE on Swin is interesting and novel. 
+ The idea to optimize grouping based on FLOPs with DP is simple but seems to work well. 
+ The accuracy on COCO suggests MIM+Swin is likely a good idea. 


Weaknesses:
- The accuracy on ImageNet is a bit disappointing: Swin+MIM trained for 900 epochs obtains 83.7, while Swin-from-scratch trained for 300 epochs already obtains 83.5. Looking at ImageNet results alone, it's not clear if we need MIM on Swin.
- The title and writing overall suggests that the method works on general hierarchical ViT models, but in reality, the authors only show that the method works on Swin transformers. It's not just lack of experiments, it's not even clear how one can apply this approach to other hierarchical models (e.g., MViT). 
- The experiments are limited to <=800 epoch training. Given that MIM is usually more useful in long-training regime, missing >800 epoch experiments make it harder to compare to MAE. 
- How much time does the DP solver take? How much time does the grouping, masking, or ungrouping take? Some more detailed analysis would be helpful. 
- (minor:) In figure 1, without accuracy, it's hard to appreciate the method by speed only.

questions: 1. I wonder if the authors have any intuition on why the accuracy of SimMIM is slightly higher than the proposed method.
2. I wonder if MIM on Swin works well with a larger backbone (e.g., Swin-L, Swin-H, etc.). I know that this might not be the main focus of this paper, but if MIM doesn't look promising on Swin, then speeding this up is less important. Conversely, if MIM on large Swin models works very well, then the importance of this work will be large.
limitations: I think the main limitation is that the method focuses on Swin Transformers only, but not other hierarchical models.
review_title: null

Meta-review: metareview: After rebuttal and discussion all reviewers recommend acceptance. The AC sees no reason to overturn this recommendation.
confidence: Certain
Decision: Accept