Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This work attempts to address the problem of lack of inductive bias in plain ViT architectures which hinders its performance for dense prediction tasks such as semantic segmentation and object detection. Specifically, the authors propose an adapter architecture that can extract and learn representations that are useful for dense prediction tasks. The proposed methodology has been trained and tested on different down-stream tasks and purportedly achieves competitive performance on COCO test-dev dateset in terms of box AP and mask AP. 
strength_and_weaknesses: ########## Strength

1. The paper is well-written and easy to follow. Visualizations properly demonstrate the concept of the proposed architecture. 

2. The overall design of the ViT-adapter architecture is quite general and can be used in different applications. Especially, using conv-based prior modules seem to be effective and play an important role. Unlike the previous ViTDet model, the adapter seems to also make use of the input image which can be effectively leveraged along with intermediate extracted features. In addition, the idea of using both conv and transformer-based (i.e. encoders with MLP and cross attention) layers in the adapter seems to be more optimal than using conv-based pyramid-like architectures. 

3. The method has been thoroughly tested on many different tasks (both supervised and self-supervised) and achieve SOTA performance on COCO test-dev dateset --although there seems to be other approaches such as FD-SwinV2-G [1], DINO [2], etc. to have better performance, but the performance is still quite competitive. 

########## Weakness

1. Despite the effective design of the proposed architecture, the idea of the ViT-adapter is not very new. The first instance of such works was SETR [3] which simply used a CNN-based decoder for feature processing in segmentation. 

2. As the name indicates, the architecture is solely designed for ViT architectures, and this may limit its applicability as hierarchical models such as Swin Transformer have gained a lot interest. It is fair to say that with minimal changes, the adapter could be used with Swin and other similar architectures. Hence it could be beneficial to study whether it could be useful when the feature extractor is more powerful. 

3. How does the adapter affect the throughput and memory foot-print ? I am concerned that it may not be feasible to use it in practical applications due to it complex architecture. It would be nice to have comparison to SETR and ViTDet for throughput. 


[1]: Wei, Y., Hu, H., Xie, Z., Zhang, Z., Cao, Y., Bao, J., Chen, D. and Guo, B., 2022. Contrastive Learning Rivals Masked Image Modeling in Fine-tuning via Feature Distillation. arXiv preprint arXiv:2205.14141.

[2]: Zhang, H., Li, F., Liu, S., Zhang, L., Su, H., Zhu, J., Ni, L.M. and Shum, H.Y., 2022. Dino: Detr with improved denoising anchor boxes for end-to-end object detection. arXiv preprint arXiv:2203.03605.

[3]: Zheng, S., Lu, J., Zhao, H., Zhu, X., Luo, Z., Wang, Y., Fu, Y., Feng, J., Xiang, T., Torr, P.H. and Zhang, L., 2021. Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 6881-6890).
clarity,_quality,_novelty_and_reproducibility: The idea itself is not very novel, but the proposed architecture is interesting and have been optimized  for different applications. The code is supposed to be published on GitHub based on author's claims, hence it should be reproducible. 
summary_of_the_review: The proposed model addresses the problem of lack of inductive bias in ViT architecture and has been shown to be effective in many different application. The model has been testes rigorously on different dataset and achieves competitive performance on COCO test-dev dateset. As a result, I believe these contributions merit acceptance.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The application of plain ViT on dense prediction tasks suffers from unacceptable complexity. The paper proposes an injection adapter with multi-scale features to use a well-pretrained ViT model and transfer it to downstream task. The experiments are conducted on object detection, instance segmentation, and semantic segmentation. The proposed technique achieves state-of-the-art on COCO dataset, which well demonstrate the high effectiveness.
strength_and_weaknesses: Strength:
+ Applying the plain ViT to downstream tasks is an improtant and meanful topic. The motivation is well explained and makes sense. The proposed technique is simple yet effectiveness.
+ The paper is well-written and well-organized. The figures and visualizations are clear and easy to follow.
+ The experiments and ablation studies are comprehensive. The proposed adapter shows significant superiority over the previous works.
And it achieves state-of-the-art performance on several tasks.

Weakness:
- It would be better to give the computational complexity and latency in the experiments.
clarity,_quality,_novelty_and_reproducibility: The clarity is clear. The novelty and the reproducibility are good.
summary_of_the_review: This paper explores an important problem in multimodality and proposes a simple yet effective solution to transfer the ViT models to dense prediction tasks. The experiments give strong evidences to well support.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposed ViT-Adapter for dense prediction task. Essentially, Vit-Adapter utilizes additional modules to inject the image prior into ViT, encouraging ViT to learn transferable representations for the specific dense prediction tasks. The extensive experiments on several dense prediction tasks (e.g., object detection, instanve segmentation, and semantic segmentation) demonstrate the effectiveness of the proposed method.
strength_and_weaknesses: Strengths:
+ The paper is written-well and easy to follow, the idea is intuitive and clear.
+ The experiments are solid and entensive.
+ The authors have made good efforts to ablate different components of their approach.
+ The performances improvement of the proposed method are impressed.

Weaknesses:
- Adapters in ViT is an interesting research topic, there were some works focused on it  [a-e]. I strongly encourage authors to take them into deep discussion. For example, the differences and advantages between them. The additional fair experiments should be  also used for discussion.

[a] "Visual prompt tuning." In ECCV, 2022.

[b] B"Exploring visual prompts for adapting large-scale models." arXiv, 2022.

[c] "AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition." In NeurIPS, 2022.

[d]  "Neural Prompt Search." arXiv, 2022.

[e] "Convolutional bypasses are better vision transformer adapters." arXiv, 2022.

- Authors inject spatial priors and multi-scale feautres learning into ViT and achieves significant performances. I doubt that can we add the other priors or learning tricks into ViT to achieve better performances. That is, the module in Vit-Adapter can be replaced by others that is important for dense prediction tasks? Can authors provide additional ablation studies on it.
clarity,_quality,_novelty_and_reproducibility: The clarity, quality, novelty and reproducibility of this paper are good.
summary_of_the_review: This work is interesting and written-well, but some of concerns needed to be improved. I'm willing to raise my rating score if authors carefully tackle my concerns.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper designed a simple yet effective adapter module that helps plain ViT behave well for dense prediction tasks, including object detection, instance segmentation, and semantic segmentation. Benefited by the inductive bias introduced by the spatial prior module and feature interaction module, the plain ViT archives remarkable performance on dense prediction tasks.
strength_and_weaknesses: #### Strength

1. Promising performance. ViT-Adapter achieves remarkable performance on several dense prediction benchmarks, using the plain ViT model.

2. The paper is clearly written and easy to follow.

3. Extensive experiments. The experiments are solid and comprehensive.



#### Weakness

1. Only the number of parameters is reported in the table. More metrics such as FLOPs and inference latency are more important.

clarity,_quality,_novelty_and_reproducibility: 1. It is unclear about ViTDet's performance in Table 1. It seems that the original paper of ViTDet reported higher numbers.

2. The paper described introduced modules clearly. I think it is easy to implement.
summary_of_the_review: See above.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This submission receives 3 positive reviews and 1 slightly negative review.  The raised concerns are the difference between prior adapter works. In the rebuttal phase, the authors have shown the current design is for dense prediction with favorable performance. After checking all the reviews, the AC feels this is a well fit for this venue and recommends acceptance.
Decision: Accept: notable-top-25%