Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a hierarchical ViT, by analyzing the effective design in Swin compared with the plain ViT. The model contains a hierarchical patch embedding with only MLP used in early stages, using transformer blocks with global attention in the 14x14 stage, and removing the last stage. Experiments on ImageNet, COCO and ADE20K have shown quite promising results. The architecture is also compatible with MAE, which can achieve higher accuracy with the mask pretraining.
strength_and_weaknesses: Strength:
- The paper is well written, the contributions are clearly stated and supported by empirical experimental results and analysis.
- The paper gives a step-by-step analysis on the differences between ViT and Swin, which 
- The proposed model addresses the problem of incompatibility between MAE and hierarchical vision transformers, by simply removing the 7x7 stage and the self-attention in early stages.
- The authors performed rich experiments to validate the effectiveness of the proposed HiViT. The training and design details are provided, and the overall method seems reproducible.

Weaknesses:
- This paper claims that early attentions can be replaced by hierarchical patch embedding only when the model is deep enough, with an attention variance comparison on the deepened plain ViT. However, as the proposed model removed the self-attention on hierarchical ViT (Swin) instead of plain ViT, does the statement still hold on Swin?
- In the analysis section, the authors proposed a variant of Swin without stage4 and does not hurt performance. However, the model without stage4 also used global window in stage 3 (as illustrated in Fig1, bottom, 3rd & 4th figures), which may also have a significant impact on the performance and computational cost of the model. Therefore, the experiment may not well support the claim that stage4 is not helpful.
- The overall architecture of HiViT comprises an overlap patch embedding that embeds the image to 14x14 and use continuous transformer blocks like plain ViTs. Yet, using hierarchical patch embedding have been proposed by several papers, e.g., papers use overlapping patch embed [1-4], and another similar work that also removed the self-attention in early stages, and only use MLP in 56x56 and 28x28 [4]. 
[1] Xiao, Tete, et al. "Early convolutions help transformers see better." Advances in Neural Information Processing Systems 34 (2021): 30392-30400.
[2] Guo, Jianyuan, et al. "Cmt: Convolutional neural networks meet vision transformers." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.
[3] Ali, Alaaeldin, et al. "Xcit: Cross-covariance image transformers." Advances in neural information processing systems 34 (2021): 20014-20027.
[4] Pan, Zizheng, et al. "Less is more: Pay less attention in vision transformers." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 36. No. 2. 2022.
- Since an important claim of this paper is enabling hierarchical architectures to trained with MAE and save pretraining cost, a more detailed comparison of the training time or GPU usage, shall be given for a better reflection of the capacity of the architecture for pretraining, as an example, the Table2 in [5].
[5] Huang, Lang, et al. "Green Hierarchical Vision Transformer for Masked Image Modeling." arXiv preprint arXiv:2205.13515 (2022).
- The performance gain in Table 4 may not well reflect whether HiViT has “structural advantage over plain vision transformers”. ViT-B has 77.9% acc while HiViT has 83.8% acc in the supervised training, while MAE+ViT-B has 83.6%(+5.7%) acc and MAE+HiViT has 84.6%(+0.8%) in the pretraining setting. Therefore, it seems HiViT do not gain much from the pretraining, even with 1600 epochs. This phenomenon should be further explained.
- There are some other minor issues, such as mistakes in writing. For example, “As an alternative solution (used by Swin) to Step (c), it causes a significant ADE20K segmentation accuracy and Compared to Step (c)”.

clarity,_quality,_novelty_and_reproducibility: The paper presents the ideas clearly and the work is original. The quality can be improved if the concerns in the weaknesses part are addressed.
summary_of_the_review: Overall, I think this paper is well organized. It studies an important task, where the hierarchical models are not well compatible with the powerful MAE. The analysis of the plain and hierarchical ViTs are quite impressive. 

However, this proposed model is a simple plain transformer with hierarchical patch stem, which has been used in several prior works. Hence, the technical novelty is limited. The key contribution of using MAE for hierarchical model pretraining, does not give impressive results. The experiments do not present the actual training cost saved with HiViT, meanwhile the improvement with 1600epoch MAE pretraining only bring limited performance improvement compared with ViT.

Therefore, the reviewer votes for the borderline in the first round.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors study what makes hierarchical ViTs (in particular SwinTransformers) work, and based on the insights, propose a new vision transformer architecture called HiViT. The authors show that HiViT works better than Swin and vanilla ViT on ImageNet and ADE20K. Moreover, HiViT doesn't use window attention, makes it compatible to MAE style efficient SSL pre-training. The authors show that masked pre-training on HiViT obtains better accuracy than MAE on vanilla ViTs.
strength_and_weaknesses: Strengths:
+ Swins, ViTs, and MAE are state-of-the-art and widely-used tools in computer vision. The three main contributions, (1) understanding what makes Swin works well, (2) advancing architecture design, and (3) showing that HiViT works with masked-pretraining, are practically useful. I can see that many computer vision practitioners will be interested in this paper.
+ The ablations in Table 2 are clear and convincing. 

Weaknesses:
- While HiViT works well with masked-pretraining, without masked pre-training it is not very strong; For example, MViTv2 is more accurate and uses only 2/3 of FLOPs. This makes HiVIT's usefulness in practice limited.
- Many ablations are not entirely new. For example, the effectiveness of RPE and hierarchical resolutions have been shown in the original Swin paper. The finding that I find novel is that C512 x 24L works quite a lot better than C768 x 12L. However, this is technically a small change. I thus overall would not rate the technical contributions high.
- Adding runtime, flops, and params into Table 2 would makes the trade-offs clearer.

- minor points on writing:
1) why use the word "merging" instead of "pooling", which is more concretely defined?

clarity,_quality,_novelty_and_reproducibility: The clarity is good. The illustration in Figure 1 along with ablations in Table 2 makes the main message clear. The quality is reasonable, too. The authors test on state-of-the-art settings and compare with state-of-the-art methods with extensive experiments. The novelty isn't particularly high, but can be a useful reference for future researchers.
summary_of_the_review: While the technical novelty of this paper is somewhat limited, I still think this paper can be a useful reference for practitioners and future researchers on architecture design. Overall, I rate this paper as marginally above the acceptance threshold.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper investigates the improvement from plain ViT to hierarchical Swin-Transformer step-by-step and proposes a new hierarchical vision transformer that only uses hierarchical patch embedding at the beginning and keeps the global attention at 14x14 level instead of window attention. It shows similar or better performance than Swin transformers on Imagenet classification, segmentation and detection. 
strength_and_weaknesses: Strength:	
1.	The paper provides a detailed study regarding prevail Swin-Transformer architecture, identifying the key parts for its superior performance v.s. plain ViT, e.g increasing depth and decrease channels, add relative position embedding. 
2.	The paper proposes an alternative way to enjoy the performance gain from hierarchical ViT which is simpler and friendly for techniques that are designed for plain ViT, i.e. Mask Image Modeling.
Weaknesses:
1.	There are several part of descriptions not clear enough. 
a.	Table 2, no explanation for RPE, Low Att, Mid Att, etc in table caption. 
b.	Stage definition missing, though can be referred, but better have clear definition. 
c.	Can authors also add Param and Flops to Table 2, as changing components will change computation as well.
d.	Incomplete sentences: Page 5, Step (c’) ‘causes a significant accuracy’;  Sec 3.3 ‘all tokens in Stage ? are symmetric’
e.	Merging operation details? One can only infer it from figure 1, can authors add some descriptions about it? 
2.	Regarding the proposed hierarchical patch embedding, can authors give more explanation about the specifical design as 2 consecutive MLP with ratio 3, how authors reach this design. As this is a critical module proposed, can authors provide some ablation studies about it, e.g. why uses a different MLP ratio of 3 as latter stages uses 4 instead? Will allocate computations differently to 56x56, 28x28 influence the final performance? What computation percentage should be used in patch embedding v.s. latter stage (14x14) for better performance? 
 

clarity,_quality,_novelty_and_reproducibility: The paper has some minor clarity issues describing the details of the method. Overall quality is good. Novelty and contributions are enough for the community as it introduces some key findings, and the method can be widely adapted. Reproducibility is good as method is simple and code is provided. 
summary_of_the_review: 
Overall, the paper presents a good study of plain ViT and hierarchical ViT, and provides a simpler hierarchical ViT with similar or better vision task performance and is friendly to many follow-up techniques of plain ViT. The proposed method can be widely adapted with its simple design and good performance. It would be much better if authors can provides further studies regarding the proposed patch embedding module.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: 
 This paper takes a thorough look at ViT and Swin vision transformer architectures, and proposes a simpler architecture that is efficient (in terms of FLOPS/training time/etc.) while achieve strong results across a wide range of computer vision tasks. The focus is specifically on architecture designs that are easily compatible with masked image modeling (MIM) , given the importance of such self-supervised learning methods. There are a range of findings and architectural designs made in the paper, including the effectiveness network depth, relative position encoding (RPE), and hierarchical patch embeddings. Several more complex aspects of Swin are found not to be necessary, such as shifted window attention and additional stages. Overall, extensive experiments are conducted across supervised/self-supervised training regimes and several vision tasks. Importantly, other aspects such as FLOPs and training time are also shown, with strong results by the proposed method. 

  The reviewers all appreciated the clear writing and analysis of these architectures, focus on masked image modeling, and thorough results. A number of weaknesses were highlighted, including veracity of the claims, missing experimental settings/ablations, and overall contributions of the findings since some of them were already present in the literature. These points, as well as some issues of clarity, were addressed by the authors and reviewers were all satisfied that their concerns have been addressed. Overall, this paper is a strong empirical study of vision transformers, especially in the self-supervised regime, with interesting insights and an effective new architecture that is simple, compatible with MIM, efficient, and effective at a range of vision tasks. As a result, it is a strong contribution to the community and I recommend acceptance. 
justification_for_why_not_higher_score: This paper is a strong empirical paper; however, it does not necessarily make a breakthrough contribution that will significantly shift the field. As a result, I don't believe oral acceptance is warranted. 
justification_for_why_not_lower_score: Overall, the paper could be accepted as a poster as well. However, since this is a fast-moving and important field (especially in the areas of self-supervised learning) and also stands to reduce computational burden, I believe accepting as a spotlight is warranted.
Decision: Accept: notable-top-25%