Vision Transformer Adapter for Dense Predictions
Authors: Zhe Chen
Yuchen Duan
Wenhai Wang
Junjun He
Tong Lu
Jifeng Dai
Yu Qiao
Venue: ICLR 2023 notable top 25% (2023)
Keywords: Plain Vision Transformer
Adapter
Dense Prediction

Abstract: This work investigates a simple yet powerful dense prediction task adapter for Vision Transformer (ViT). Unlike recently advanced variants that incorporate vision-specific inductive biases into their architectures, the plain ViT suffers inferior performance on dense predictions due to weak prior assumptions. To address this issue, we propose the ViT-Adapter, which allows plain ViT to achieve comparable performance to vision-specific transformers. Specifically, the backbone in our framework is a plain ViT that can learn powerful representations from large-scale multi-modal data. When transferring to downstream tasks, a pre-training-free adapter is used to introduce the image-related inductive biases into the model, making it suitable for these tasks. We verify ViT-Adapter on multiple dense prediction tasks, including object detection, instance segmentation, and semantic segmentation. Notably, without using extra detection data, our ViT-Adapter-L yields state-of-the-art 60.9 box AP and 53.0 mask AP on COCO test-dev. We hope that the ViT-Adapter could serve as an alternative for vision-specific transformers and facilitate future research. Code and models will be released at https://github.com/czczup/ViT-Adapter.

Published as a conference paper at ICLR 2023
VISION TRANSFORMER ADAPTER FOR DENSE PREDICTIONS
Zhe Chen 1:2, Yuchen 2.3,, Wenhai Wang 2 JJnjuun He?,
Tong Lu 12 Jifeng Dai Yu Qiao² 'Nanjing University, 2Shanghai Al Laboratory, "Tsinghua University czcz9Acz@gmail com, (duanyuchen, wangwenhai, lutongênju. edu. en, (dai feng, hunggennaaihejjjjunnnBaaa.... hejjujjnj@pjlaabboomm
www.jlla...ooooooooooooooo ong.cn
ABSTRACT
This work investigates simple yet powerful dense prediction task adapter for Vi- sion Transformer (ViT). Unlike recently advanced variants that incorporate vision- specific inductive biases into their architectures, the plain ViT suffers inferior performance on dense predictions due to weak prior assumptions. To address this issue, we propose the ViT-Adapter which allows plain ViT to achieve com- parable performance
vision-specific transformers. Specifically, the backbone our framework is a plain VIT that can learn powerful representations from large-scale multi-modal data. When transferring to downstream tasks, a pre- training-free adapter is used to introduce the image-related inductive biases into the model, making it suitable for these tasks. We verify ViT-Adapter on mul- tiple dense prediction tasks, including object detection, instance segmentation, and semantic segmentation. Notably, without using extra detection data, our ViT- Adapter-L. yields state-of-the-ae 60.9 box AP and 53.0 mask AP on COCO test- dev. We hope that the ViT-Adapter could serve an alternative for vision-specific transformers and facilitate future research. Code and models will be released be https: titbubb..... com/ wwwpc//ccccpc/ecc Vit.pp//ii-Adae.......m
Step1: Modality Pre-training
Step1: Mudti-Modal Pre-training
Image Modality
Vsion-Specific Transformer
SL/SSL
Image Video, Text
Step2: Fine-tuning
Step2: Fine-tuning with Adapter
COCO
Vision- Specific
Det/Seg
COCO
VIT
Det/Seg
Transformer
Adapter
(a) Previous Paradigm
(b) Our Paradigm
Figure Previous paradigm vs. our paradigm. (a) Previous paradigm designs vision-specific models and pre-trains large-scale image datasets via supervised self-supervised learning and then fine-tunes them on downstream tasks. (b) We propose a pre- pee-trainingg-fee free adapter to close the performance gap between plain ViT (Dosovitskiy al., 2020) and vision-specific transformers (e.g., Swin (Liu et al., 2021b)) for dense prediction tasks. Compared to the previous paradigm, our method preserves the flexibility of ViT and thus could benefit from advanced multi-moda, pre-training.
INTRODUCTION
Recently, transformers have witnessed remarkable success in a broad range of computer vision fields. Benefiting from the dynamic modeling capability and the long-range dependence of the attention mechanism, various vision transformers (Dosovitskiy et al., 2020; Chen et al., 2021; Han al., 2021; Li et al., 2021e; Wu et al., 2022b) soon rose many computer vision tasks such as object detection and semantic segmentation, surpassing CNN models and reaching state-of-the-art performance, These models are mainly divided into two families, i.e. the plain VIT (Dosovitskiy et al., 2020; Touvron et al., 2021), and its hierarchical variants (Dong al., 2021; Liu et al., 202 lb; Wang et al., 2021; 2022a). general, the latter can produce better results and is believed to intro- duce vision-specific inductive biases into their architectures by using local spatial operations.
"quual contribution. Cooresponddngg authors.

---

Published as a conference paper at ICLR 2023
Nonetheless, the plain ViT (ie.. vanilla trans- former) still has some nonnegligible advan- tages. A typical example lies in multi-modal pre-training (Zhu et al., 2021; 2022; Wang et al., 2022b). Stemming from the natural language processing (NLP) field, transformer has no assumption of input data. Equipping with different tokenizers, e.g. patch embed-
53
VIT ViT-Adapter (ours)
51
Malli-MMddl
Pre-training
coCO BBox AP (%6
49
S
Swin
PVTV2-BL
Malhed
#Param
33.7M
Aph
47
PVTv2
VIT.T
ViT--Aapten-T (ours)
26. 1M
28. 1M
46.0
40.2
VIT
PVTV2AA2
47.
47.M
45
PVT
VITES
NiT-AduplesS (ours) (ouns)
41.M
47.M
44.0
48.2
43
S
VTTBB
107-1M
48.6
ding (Dosovitskiy et al., 2020), 3D patch em-
ViT-Adapter.B (ours)
120.2M
49.6
bedding (Liu et al., 2021c), and token em-
ViT--Aaatee- (ours)
120.2M
51.2
VIT-I
137.
48.8
bedding (Vaswani et al., 2017), vanilla trans- formers such as plain VIT can use mussive multi-modal data for pre-training, including image, video, and text, which encourages the model to learn semantic-rich representations. However, the plain VIT has conclusive de- fects in dense predictions compared to vision- specific transformers. Lacking image-related prior knowledge results in slower convergence and lower performance, and thus plain ViTs are hard to compete with iesion-specific transform- ers (Huang et al., b021b; Xie et al., 2021; Wang et al., 2022a) on dense prediction tasks. Inspired by the adapters (Houlsby et al., 2019; Stickland & Murray, 2019) in the NLP field, this work aims to develop an adapter to close the performance gap between the plain VIT and vision-specific backbones for dense prediction tasks.
VIT- VIT-Adapter.. (ours)
3347.9M
52.1
0
100
150
200
250
300
350
400
#Parameter (M)
Figure 2: Object detection performance on COCO val2017 using Mask R-CNN. We see that the proposed ViT-Adapter brings significant improvements plain ViTs. * indecates us- ing multi-modal pre-trained ViT from (Zhu et al., 2021). Backbones pre-trained on ImageNet-22K are marked with 1, otherwise ImageNet-IK.
To this end, we propose the Vision Transformer Adapter (ViT-Adapter), which is a pre-training-fre additional network that can efficiently adapt the plain ViT to downstream dense prediction tasks without modifying its original architecture. Specifically, to introduce the vision-specific inductive biases into the plain ViT, design three tailored modules for ViT-Adapter, including (1) spatial prior module for capturing the local semantics (spatial prior) from input images, (2) a spatial fea- ture injector for incorporating spatial prior into the VIT, and (3) multi-scale feature extractor to reconstruct the multi-scale features required by dense prediction tasks.
As shown in Figure 1, compared the previous paradigm that pre-trains on large-scale image datasets (e.g. ImageNet (Deng et al., 2009)) then fine-tunes on other tasks, our paradigm is more flexible. In our framework, the backbone network is a general-proposs model (e.g., plain ViT) that can be pre-trained with not only images but also multi-modal data. For the transfer learning of dense prediction tasks, we use a randomly initialized adapter to introduce the image-related prior knowledge (inductive biases) into the pre-trained backbone, making the model suitable for these tasks. In this way, using ViT as the backbone, our framework achieves comparable or even better performance than vision-specific transformers such as Swin (Liu et al., 2021b).
Our main contributions are as follows:
We explore new paradigm to introduce vision-specific inductive biases into the plain VIT. It helps VIT achieve comparable performance to recent transformer variants (Liu et al., 2021b; Wang et al., 2022a) with regular ImageNet pre-training and further benefits from multi-modal pre-training.
We design a spatial prior module and two feature interaction operations, to inject the image prior without redesigning the architecture of ViT. They can supplement the missing local information and reorganize fine-graines grained multi-scale features for dense prediction tasks.
We evaluate the ViT- ViT-Adapeer multiple challenging benchmarks, including COCO (Lin al., 2014) and ADE20K (Zhou et al., 2017). As shown in Figure 2, our models consistently achieve improved performance compared to the prior arts under the fair pre-training strategy. For instance, when using only ImageNet- 1K pre-training, ViT-Adapter-B reports 49.6 box AP on COCO val, outperforming Swin-B by 1.0 points. Benefiting from multi-modal pre-training (Peng et al., 2022), our ViT-Adapter-L yields 60.9 box AP, which is the best record on COCO test-dev without training on extra detection data such as Objects365 (Shao al., 2019).
2

---

Published as a conference paper at ICLR 2023
using task prior input
Image
using task prior
Image
1/4
1/4
4x
up
178
Adapter
1/8
VIT
ViT
1x
1/16
1/16
down
1/32
1/32
2x
(a) Previous Method (Li et al,, ViTDet)
(b) ViT-Adapter (ours)
Figure 3: Overview of ViT-Adapter and two related approaches. ei al. (2021b) and ViTDet (Li al., 20226) build simple feature pyramid to adapt plain ViT for object detection, which only consider task prior. Differently, our adapter utilizes both task prior and the input image.
RELATED WORK
Transformers. In recent years, transformers have dominated various tasks across multiple modal- ities, such as natural language processing, computer vision, and speech recognition. The vanilla transformer (Vaswani et al., 2017) was initially proposed for machine translation and remains the state-of-the-ar architecture for NLP tasks today. VIT (Dosovitskiy et al., 2020) is the first work to generalize the vanilla transformer to the image classification task without much modification. PVT (Wang et al., 2021) and Swin (Liu et al., 2021b) introduce more vision-specific inductive biases by incorporating the pyramid structure from CNNs. Afterward, Conformer (Peng et al., 2021) proposed the first dual network to combine CNN with transformer. Recently, BEÏT (Bao al., 2022) and MAE (He et al., 2021) extended the scope of ViT to self-supervised learning with masked image model- ing (MIM), demonstrating the powerful potential of the plain VIT architecture. Many works (Li et al., 2021b; Zhu et al., 2021; 2022; Wang et al., 2022b) have shown that designing vision-specific models is an important direction, but the seneral-propose architectures (e.g., plain ViT) are more flexible and essential for masked data modeling and multi-modal pre-training. Therefore, we de- velop a pre- trriiinnn-nfeeee free adapter to introduce the image prior without modifying the architecture of VIT. preserving its flexibility and enjoying advanced multi-modal pre-training.
Decoders for VIT. The architecture for dense prediction commonly follows an encoder-decoder pattern, in which the encoder generates rich features and the decoder aggregates and translates them to the final predictions. Recently, illuminated by the global receptive fields of ViT, many works employ it as the encoder and design task-specific decoders. SETR (Zheng et al., 2021) is the first work to adopt ViT as the backbone and develop several CNN decoders for semantic segmentation. Segmenter (Strudel et al., 2021) also extends ViT semantic segmentation. but differs in that equips transformer- based decoder. DPT (Ranftl et al., 2021) further applies ViT to the monocular depth estimation task via CNN decoder and yields remarkable improvements. In summary, these works improve the dense prediction performance of ViT by designing modality- and task-specific decoders, but remain ViT's weakness of single-scale and low-resolution representation.
Adapters. To date, adapters have been widely used in the NLP field. PALS (Stickland & Murray, 2019) and Adapters (Houlsby et al., 2019) introduce new modules transformer encoders for task- specific fine-tuning, making the pre-trained model quickly adapt to downstream NLP tasks. In the field of computer vision, some adapters have been proposed for incremental learning (Rosenfeld & Tsotsos, 2018) and domain adaptation (Rebuffi et al., 2017; 2018). With the advent of CLIP (Radford et al,, 2021), many CL.IPbased adapters (Gao et al., 2021; Sung et al., 2021; Zhang et al., 2021) were presented to transfer tre-trained knowledge to zero-shot or few-shot downstream tasks. Recently, Li et al. (2021b) and ViTDet (Li et al., 2022b) employed some upsampling and downsampling modules to adapt the plain ViT for object detection, as shown in Figure 3(a). However, under regular training settings (i.e,, apply ImageNet supervised pre-training and fine-tune for 36 epochs), their detection performance is still inferior' to recent models (Chu al., 2021b; Dong et al., 2021; Wang et al., 2022a; Wu et al., 2022b) that well combine image prior. Therefore, it is still challenging to design a powerful dense prediction task adapter for VIT.
In ViTDet,, using regular ImageNet-22K pre-training instead of MAE (He al., 2021) drops 4.0 box AP.
3

---

Published as a conference paper at ICLR 2023
(e) Vision Transformer (VIT)
Patch Embedding
+
Block
Block
Block N
Spatial Module
Injector
Extractor
Injector
Extractor
Injector
Extractor N
VIT Adapter
Embedding
Pasition
Stem
[
Tg 1
1
xip
5%
Query
Cross-Attention
Pate
Key/Value
Cross-Attentin
FFN
Query
slement-wiss
Addition
(c) Spatial
Spatial Feature Injector
(e) Multi-Scale Extractor
Figure 4: Overall architecture of ViT-Adapter. (a) The VIT, whose encoder layers are divided into N (usually N 4) equal blocks for feature interaction. (b) Our ViT-Adapter, which contains three key designs, including (c) a spatial prior module for modeling local spatial contexts from the input image, (d) spatial feature injector for introducing spatial priors into ViT, and (e) multi-scale feature extractor for reorganizing multi-scale features from the single-scale features of VIT.
VISION TRANSFORMER ADAPTER
3.1 OVERALL ARCHITECTURE
As illustrated in Figure 4, our model can be divided into two parts. The first part is the plain VIT (Dosovitskiy al., 2020) that consists of patch embedding followed by L. transformer encoder layers (see Figure 4(a)). The second part is the proposed ViT-Adapter as shown in Figure 4(b), which contains (1) a spatial prior module to capture spatial features from the input image, (2) spatial feature injector to inject spatial priors into the VIT, and (3) a multi-scale feature extractor to extract hierarchical features from the single-scale features of ViT.
For the ViT, the input image is first fed into the patch embedding, where the image is divided into 16 x 16 non-overlapping patches. After that, these patches are flattened and projected D- dimensional tokens, and the feature resolution is reduced to 1/16 of the original image. Then, these tokens added with the position embedding, are passed through encoder layers.
For the ViT-Adapter, we first feed the input image into the spatial prior module. D-dimensional spatial features of three target resolutions (i.e., 1/8, 1/16, and 1/32) will be collected. Then, these feature maps are flattened and concatenated as the input for feature interaction. Specifically, given the number of interactions (usually N = 4), we evenly split the transformer encoders ViT into N blocks, each containing L/N encoder layers. For the i-th block, we first inject spatial priors F into the block via spatial feature injector, and then extract hierarchical features from the output of the block by multi-scale feature extractor. After N feature interactions, we obtain high-quality multi-scale features, and then we split and reshape the features into three target resolutions 1/8, 1/16, and 1/32. Finally, we build the 1/4-scale feature map by upsampling the 1/8-scale feature map using a 2 x 2 transposed convolution. In this way, we obtain a feature pyramid of similar resolutions to ResNet (He al., 2016), which can be used in various dense prediction tasks.
3.2 SPATIAL PRIOR MODULE
Recent studies (Wang et al., 2022a; Wu et al., 2021; Fang et al., 2022; Park & Kim, 2022) show convolutions can help transformers better capture the local spatial information. Inspired by this, introduce the Spatial Prior Module (SPM). It is designed to model the local spatial contexts of images parallel with the patch embedding layer, so as not to alter the original architecture of VIT.
As shown in Figure 4(c), standard convolutional stem borrowed from ResNet (He et al., 2016) is employed, which consists of three convolutions and a max-pooling layer. Then, we use stack of stride-2 3x3 convolutions to double the number of channels and reduce the size of feature maps. Finally, several Ixl convolutions are applied at the end to project the feature maps to D dimensions. In this way, we obtain a feature pyramid (Fi, F2, Fa). which contains D-dimensional feature maps
4

---

Published as a conference paper at ICLR 2023
with resolutions of 1/8, 1/16, and 1/32. Then, we flatten and concatenate these feature maps into 2w
feature tokens Fl E R2
wwy 2i )xD as the input for feature interaction.
sp
3.3 FEATURE INTERACTION
Due to weak prior assumptions, the plain ViT suffers sub-optimal performance dense prediction tasks compared to vision-specific transformers (Chu et al., 2021a; Dong et al., 2021; Liu et al,, 2021b; Wang et al., 2022a). To alleviate this issue, we propose two feature interaction modules to bridge the feature maps of our SPM and the VIT. To be specific, the two modules are mainly based on cross-attention, namely Spatial Feature Injector and Multi-Scale Feature Extractor.
Spatial Feature Injector. As shown in Figure 4(d), this module is used to inject the spatial priors into VIT. Specifically, for the i-th block of the ViT, take the input feature F IR i0 x as the query, and the spatial feature Fí E I
w. aw 16"
uw )x as the key and value. We use cross- 322
attention to inject spatial feature Fáp into the input feature F which can be written as Eqn. 1.
= Attenttionnorr norm(FF.)).
where the (-) is LayerNorm (Ba et al., 2016), and the attention layer Atdention(.) suggests using sparse attention. In addition, we apply a learnable vector 7d RDD to balance the attention layer's output and the input feature Frit, which is initialized with O. This initialization strategy ensures that the feature distribution of Feit will not be modified drastically due the injection of spatial priors, thus making better use of the pre-trained weights of ViT.
Multi-Scale Feature Extractor. After injecting the spatial priors into the VIT. we obtain the output feature Fi+1 by passing Fr through the encoder layers of the i-th block. Then, we apply a mod- ule consisting of a cross-attention layer and feed-forward network (FFN), to extract multi-scale features, as shown in Figure 4(e). This process can be formulated as:
Fi-l Fi + FFN(norm(FFtp)
f F sp
Attention(norm
sp) norm ((t)) )),
(3)
sp
in which use the spatial feature F
JxD the query, and the output feature Fitl R 162 xD as the key and value for cross attention. As same as the spatial feature injector, adopt sparse attention here to reduce computational cost. The generated spatial feature Fit1 will be used as the input of the next spatial feature injector.
E
1G
22
3.4 ARCHITECTURE CONFIGUR ATTONS
We build our ViT-Adapter for 4 different sizes of VIT. including VIT-T, VIT-S, VIT-B, and VIT- L. For these models, the parameter numbers of our adapters 2.5M. 5.8M, 14.0M, and 23.7M, respectively. We employ deformable attention (Zhu et al., 2020) as the default sparse attention in method, where the number of sampling points is fixed 4, and the number of attention heads set to 6, 6, 12, and 16. The number of interactions stack three multi-scale feature extractors. Besides, we set the FFN ratio in our adapter to 0.25 to save computational overhead, i.e. the hidden sizes of FFN are 48, 96, 192, and 256 for 4 different adapters. More details of each configuration are shown in Table 10 in Appendix
is 4, and in the last feature interaction, we
EXPERIMENTS
Previous work (Wang et al., 2021) has shown that the pyramid prior beneficial to dense prediction, but brings little gains to image classification. Therefore, in this study, we focus on how to better adapt readily available pre-trained ViTs to dense prediction tasks. We hope this method will also help decouple the model design of upstream pre-training and downstream fine-tuning.
4.1 OBJECT DETECTION AND INSTANCE SEGMENTATION
Settings, Our detection experiments are based on MMDetection (Chen et al., 2019b) and the COCO (Lin et al., 2014) dataset. We use 4 mainstream detectors to evaluate our ViT-Adapter, including
5

---

Published as a conference paper at ICLR 2023
iParam (M)
Mask R-CNN 1x schedule
Mask R-CNN
3x+ schedule
Method
AP APm 58 AP7
AP
1 AP
50 AP%
PVT-Tiny (Wang al., 2021)
32.9
36.7 59.2
39.3
35.1
56.7
37.3
39.8 62.2
43.0
37.4
59.3
39.9
PVTv2-B1 (Wang al., 2022a)
33.7
41.8
64.3
45.9
38.8
61.2
41.6
44.9
67.3
49,4
40.8
64.0
43.8
ViT-T (Li al., 2021h)
26.1
35.5
58.1
37.8
33.5
54.9
35.1
40.2
62.9
43.5
37.0
59.6
39.0
ViTDet-T (I.i ct al., 2022h)
26.6
35.7
57.7
38.4
33.5
54.7
35.2
40.4
63.3
43.9
37.1
60.1
39.3
ViT-Adapter-T (ours)
28.1
41.1
62.5
44.3
37.5
59.7
39.9
46.0
67.6
50.4
41.0
64.4
44.1
PVT-Small (Wang ct al., 2021)
44.1
40.4
62.9
43.8
37.8
60.1
40.3
443.0
65.3
46.9
39.9
62.5
42.8
PVTv2-B2 (Wang et al., 2022a)
45.0
45.3
67.1
49.6
41.2
64.2
44.4
47.8
69.7
52.6
43.1
66.8
46.7
Swin-T (I ct al., 2021h)
47.8
42.7
65.2
46.8
39.3
62.2
42.2
46.0
68.1
50.3
41.6
65.1
44.9
ConvNeXt-T (Liu et al., 2022)
48.1
44.2 66.6
48.3
40.1
63.3
42.8
46.2 67.9
50.8
41.7
65.0
44.9
Focal-T (Yang et al., 2021)
48.8
44.8
67.7
49.2
41.0
64.7
44.2
47.2 69.4
51.9
42.7
66.5
45.9
VIT-S (Li ct al., 2021b)
43.8
40.2
63.1
43.4
37.1
59.9
39.3
44.0
66.9
47.8
39,9
63.4
42.2
ViTDet-S (Li et al., 2022b)
45.7
40.6
63.3
43.5
37.1
60.0
38.8
44.5 66.9
48.4
40.1
63.6
42.5
ViT-Adapter-S (ours)
47.8
44.7
65.8
48.3
39.9
62.5
42.8
48.2
69.7
52.5
42.8
66.4
45.9
PVTv2-B5 (Wang et al., 2022a)
101.6
47.4
68.6 46.9
51.9
42.5
65.7
46.0
48.4
69.2
52,9
42,9
66.6
46.2
Swin-B (Liu et al., 2021b)
107.1
42.3
48.6
70.0
53.4
43.3
67.1
46.7
VTTBB (Li et al., 2021b)
113.6
42.9
65.7
46.8
39.4
62.6
42.0
45.8
68.2
50.1
41.3
65.1
44.4
ViTDet-B (Li et al.. 2022b)
121.3
43.2
65.8
46.9
39.2
62.7
41.4
46.3
68.6
50.5
41.6
65.3
44.5
ViT-Adapter-B (ours)
120.2
47.0
68.2
51.4
41.8
65.1
44.9
49.6
70.6
54.0
43.6
67.7
46.9
VIT-L' (Li et al., 2021b)
337.3
45.7
68.9
49.4
41.5
65.6
44.6
48.3
70.4
52.9
43.4
67.9
46.6
ViTDet-L (Li et al.. 2022b)
350.9
46.2
69.2
50.3
41.4
65.8
44.1
49.1
71.5
53.8
44.0
68.5
47.6
ViT-Adapter-L' (ours)
347.9
48.7 70.1 53.2 43.3 67.0
46.9
52.1
73.8 56.5
46.0
70.5 49.7
Table I:
Object detection and instance segmentation with Mask R-CNN on COCO va12017. For fair comparison, we initialize all ViT-T/S/B models with the regular ImageNet- IK pre- pre-training (Tou- vron et al., 2021), and ViT-L with the ImageNet-22K weights from (Steiner et al., 2021).
Method
AP APMAPtOAPA #P
Method
APh APt #P
Cascade Mask R- R-CNN
3x+MS schedule
ATSS 3X+MS schedule
Swin-T (Liu et al., 2021b)
50.5 69.3 54.9 86M
Swin-T (Liu et al., 2021b)
47.2 66.5 51.3 36M
Shuffle-T (Huang al., 2021h)
50.8 69.6 55.1 86M
Focal-T (Yang al., 2021)
49.5 68.8 53.9 37M
PVTv2-B2 (Wang al., 2022a)51.1
69.8
55.3 83M
PVTv2-B2 (Wang et al., 20.22)49.9
69.1 54.1 33M
Focal-T (Yang et al., 2021)
51.5 70.6 55.9 87M
VIT-S (Li et al., 2021b)
45.2 64.8 49.0 32M
ViT-S (Li et al., 2021b)
47.9 67.1 51.7 82M
ViT-Adapter-S (ours)
49.6 68.5 54.0 36M
ViT-Adapter-S (ours)
51.5 70.1
55.8
86M
GFL XX+ schedule
Swin-B (Liu et al., 2021b)
51.9 70.9 57.0
145M
Swin-T (Liu et al., 2021b)
47.6 66.8 51.7 36M
Shuffle-B (Huang et al., 2021b)
52.2
71.3 57.0 145M
PVTv2-B2 (Wang et al., 2022a)50.2 VYT-S (Li et al., 2021b)
69.4 54.7 33M
ViT-B (Li al., 2021b)
50.1
69.3
54.3 151M
46.0 65.5
49.7 32M
ViT-Adapter-B (ours)
52.1 70.6
56.5 158M
ViT-Adapter-S (ours)
50.0 69.1 54.3 36M
Table 2: Object detection with different frameworks COCO va12017. For fair comparison, we initialize all ViT-S/B models with the regular ImageNet-1K pre-training (Touvron et al., 2021). "#P" denotes the number of parameters. "MS" means multi-scale training.
Mask R-CNN (He et al., 2017), Cascade Mask R-CNN (Cai & Vasconcelos, 2019), ATSS (Zhang al., 2020), and GFL (Li et al., 2020). To save time and memory, we refer to (Li et al., 2021b) and modify the L-layer ViT to use 14x 14 window attention except for layers spaced at an interval of L/4. Following common practices (Wang et al., 2021), we adopt x or 3x training schedule (i.e., 12 or 36 epochs) with batch size of 16, and Adam (Loshchilov & Hutter, 2017) optimizer with an initial learning rate of 1 x 10-1 and weight decay of 0.05.
Results with ImageNet-1K 1K Pre-training. In Table and Table 2, we apply the DeiT (Touvron et al., 2021) released ImageNet-IK IK weights (without distillation) as the initialization for all ViT-T/S/B models. We compare our ViT ViT-Adapter with two related approaches (Li et al., 2021b; 2022b) and multiple representative vision- spiinn- backbones (Wang et al., 2021; 2022a; Huang et al., 202 Liu et al., b021b; Yang et al., 2021). As we can see, when using regular training settings for fair comparison. the detection performance of ViT (Li et al., 2021b) and ViTDet (Li et al., 2022b) is inferior recent vision-specific models. For example, with Mask R-CNN and 3X+MS schedule,

---

Published as a conference paper at ICLR 2023
Semantic FPN
80k
UperNet 160k
Method
Pre-train
Crop Size
#Param
mloU
+MS
#Param
mioU
+MS
PVT-Tiny (Wang et al., 2021)
IN-IK
512x512
17.0M
36.6
37.3
43.2M
38.5
39.0
VIT-T (I.i ct al., 2021h)
IN-IK
512x512
10.2M
39.4
40.5
34.1M
41.7
42.6
ViT-Adapter-T (ours)
IN-1K
512x512
12.2M
41.7
42.1
36.1M
42.6
43.6
PVT-Small (Wang al., 2021)
IN-IK
512x512
28.2M
41.9
42.3
54.5M
43.7
44.0
PVTv2-B2 (Wang et al., 2022a)
IN-IK
512x512
29.1M
45.2
45.7
Swin-T (Liu al., 2021b)
IN-IK
512x512
31.9M
41.5
59.9M
44.5
45.8
Twins-SVT-S (Chu al., 2021a)
IN-IK
512x512
28.3M
43.2
54.4M
46.2
47.1
ViT-S (Li 2021b)
IN-IK
512x512
27.8M
44.6
45.8
53.6M
44.6
45.7
ViT-Adapter-S (ours)
IN-IK
512x512
31.9M
46.1
46.6
57.6M
46.2
47.1
Swin-B (Liu al., 2021b)
IN-1K
512x512
91.2M
46.0
121.0M
48.1
49.7
Twins-SVT-L (Chu et al., 2021a)
IN-IK
512x512
103.7M
46.7
133.0M
48.8
50.2
VTT-B (Li et al., 2021b)
IN-IK
512x512
98.0M
46.4
47.6
127.3M
46.1
47.1
ViT-Adapter-B (ours)
IN-IK
512X512
104.6M
47.9
48.9
133.9M
48.8
49.7
Swin-B' (Liu et al., 2021b)
IN-22K
640x640
121.0M
50.0
51.7
Swin-L (Liu et al., 2021b)
IN-22K
640x640
234.0M
52.1
53.5
ViT-Adapter-B' (ours)
IN-22K
512x512
104.6M
50.7
51.9
133.9M
51.9
52.5
ViT-Adapter-L' (ours)
IN-22K
512x512
332.0M
52.9
53.7
363.8M
53.4
54.4
ViT-Adapter-L (ours)
MM
512x512
332.0M
54.2
54.7
363.8M
55.0
55.4
Table 3: Semantic segmentation on the ADE20K val. Semantic FPN (Kirillov al., 2019) and UperNet (Xiao et al., 2018) are used as segmentation frameworks. "IN-IK/22K" and "MM" repre- sent ImageNet-I KK/2KK and multi-modal pre-training, respectively. "MS" means multi-scale testing.
VIT-S and ViTDet-S are 3.8 APb and 3.3 APb lower than PVTv2-B2 (Wang et al., 2022a) respec- tively. Differently, our ViT- ViT-Adapter-S outperforms these two approaches by clear margins and even 0.4 APb higher than PVTv2-B2. This observation can also be seen in the experiments detectors, including Cascade Mask R-CNN, ATSS, and GFL. These results indicate that, with only the regular ImageNet- pre-training, ViT-Adapter can promote the plain ViT to attain similar or superior performance than these vision-specific transformers.
three other
Results with ImageNet-22K Pre-training. In Table 1, we employ the ImageNet-22KK pre-trained weights from AugReg (Steiner al., 2021) to initialize all ViT-L. models, including ViT (Li al., 2021b), ViTDet (Li et al., 2022b), and our ViT-Adapter. It can be seen that, when training Mask R-CNN with 3X+ schedule. our ViT-Adapter-L' brings 3.8 AP and 3.0 APh improvements over ViT-L (Li et al., 2021b) and ViTDet-L (Li et al., 2022b), respectively.
Results with Multi-Modal Pre-training. In this experiment, we study the effect of multi- modal pre-training Specifically, we fine-tune the ViT-Adapter-B with Mask R-CNN for the 3x+MS schedule using different pre-trained weights. As shown in Table 4, simply replacing the ImageNet-22K, pre-training (Steiner al., 2021) with the multi-modal pre-training (Zhu et al., 2021) gives us a significant gain of 0.7 AP and AP" These results indicate that our method can easily derive considerable benefits from advanced multi-modal pre-training, which difficult for vision-specific models like Swin.
Method
Pre-train
Aph Apm
ImageNet-IB IK
48.6
43.3
Swin-B (Mask R-CNN 3x+MS)
ImageNet-22K
49.6
44.3
Multi-Modal
N/A
N/A
ViT-Adapter-B (Mask R-CNN 3x+MS)
ImageNet- ImageNet-22K
49.6 43.6
50.5 44,6
Multi-Modal
51.2 45.3
Table Comparison of different pre-trained weights. Our method retains the flexibility of ViT and thus could benefit from advanced multi- modal pre-training (Zhu et al., 2021).
4.2 SEMANTIC SEGMENTATION
Settings. We evaluate our ViT-Adapter semantic segmentation with the ADE20K (Zhou et al., 2017) dataset and MMSegmentation (Contributors, 2020) codebase. Both Semantic FPN (Kirillov et al., 2019) and UperNet (Xiao et al., 2018) employed as the basic frameworks. For Semantic FPN, we apply the settings PVT (Wang et al., 2021) and train the models for 80k iterations. For UperNet, we follow the settings of Swin (Liu al., 2021b) train it for 160k iterations.

---

Published as a conference paper at ICLR 2023
Results with ImageNet-1I Pre-training, In Table 3, report the semantic segmentation results in terms of single-scale and multi-scale (MS) mloU. As same as Section 4.1, we initialize all ViT-T/S/B models with the DeiT (Touvron et al., 2021) released ImageNet- IK weights. It shows that, under comparable model sizes, our method surpasses the VIT (Li et al., 2021b) and many representative vision-specific transformers (Wang al., 2021; 2022a; Liu al., 2021b; Chu et al., 2021a). For instance, our ViT-Adapter-S achieves 47.1 MS mloU with UperNet, outperforming many strong counterparts such as Swin-T. Similarly, ViT-Adapter-B reports a competitive performance of 49.7 MS mloU, which is 2.6 points higher than ViT-B and on par with Swin-B and Twins-SVT-LL These fair comparisons using only regular ImageNet- IK pre-training (Touvron et al., 2021) demonstrate the effectiveness and universality of our ViT-Adapter.
Results with ImageNeet222K Pre-training. When using the ImageNNet222K pre-trained weights (Steiner et al., 2021), our ViT-Adapter-B1 attains 51.9 mloU and 52.5 MS mloU with UperNet, exceeding Swin-Bt by at least 0.8 mloU. Similarly, ViT-Adapter-L.' yields the results of 53.4 mloU and 54.4 MS mloU, which is outstanding from the counterparts like Swin-L These significant and consistent improvements over different model sizes suggest that our method can cover the shortage of plain ViT, making it more suitable for semantic segmentation.
Results with Multi- Pre-training. Here, we apply the multi-modal pre-trained weights from Uni-Perceiver (Zhu et al., 2021) for semantic segmentation. As shown in Table 3, for Semantic FPN and UperNet, replacing the ImageNet-22K pre-training with multi-modal pre-training benefits our ViT-Adapter--L with impressive gains of 1.3 mloU and 1.6 mloU, respectively.
4.3 COMPARISONS WITH STATE-OF-THE-ARTT
Settings. We conduct experiments to combine our ViT-Adapter with state-of-the-art detec- tion/segmentation frameworks, including HTC++ (Liu et al., 2021b) (without extra detection dataset) and Mask2Former (Cheng et al., 2021), and recent multi-modal pre-training BEÏTv2 (Peng al., 2022). The experimental settings are listed in Appendix A. and A.2.
Results. As shown in Table 5, our method reaches state-of-the-art performance. While these results may be partly due to the ef- fectiveness of advanced pre-training, our study demonstrates that plain backbone de- tectors/segmenters can challenge the en- trenched position of hierarchical backbones.
COCO test-dev |APb APM
ADE20K val
InloU
CB Swin-L SwinV2-L ViT-Adapter-I
60.1 52.3
FD FD-SwinV2-G
61.4
60.8 52.7
ViT-Adapter-L
61.5
66.99
BEiT3(w/ ViT-Adapter)
62.8
Table 5: Comparison with previous SOTA.
4.4 ABLATION STUDY
VIT vs. ViT-Adapter Feature. Recent works (Park & Kim, 2022; Si et al., 2022) show that ViT presents the characteristics of learning low-frequency global signals, while CNN tends to extract high-frequency information (e.g., local edges and textures). To show the difference between the features of ViT and ViT-Adapter, we first use Fourier analysis as a toolkit for visualization. As shown in Figure 5(a), the Fourier spectrum and relative log amplitudes of the Fourier transformed feature maps (average over 100 images) indicate that ViT-Adapter captures more high-frequency signals than the ViT (Li et al., 2021b) baseline. In addition, we also visualize the stride-8 feature map in Figure (b)(c) which shows that the features of ViT are blurry and coarse. In contrast, our features more fine-grann ane-ggained and have more local edges and textures. This observation demonstrates that our method grafts the merit of CNN for capturing high-frequency information to VIT.
Ablation for Components. To investigate the contribution of each key design, we gradually extend the ViT-S baseline (Li et al., 2021b) to our ViT-Adapter-S. All models are trained with Mask R-CNN for 1x schedule. As shown in the left side of Table 6, by directly resizing and adding the spatial features from SPM, our variant improves 1.4 APb and 0.9 APM over the baseline, showing that local spatial information is essential for dense prediction. From variant 2, we find that the spatial feature injector further boosts the performance by 1.0 AP and 0.8 APm. This observation illustrates that cross- attention is a more flexible way to inject spatial features. Moreover, we employ the multi- scale feature extractor to reconstruct hierarchical features, which brings 2.1 APh and 1.1 APM gains, alleviating VIT's drawber back of single-scale features. In summary, our proposed components are each necessary and collectively create 4.5 APb and 2.8 APm improvements.
8

---

Published as conference paper at ICLR 2023
0.0
VIT-Adapter
VIT AT--Aclpttr
6.00g amplitude
-0.5
VIT
1.0
-1.5
-2.0
-2.5
0.0m
0.2n 0.5mt 0.8m
1.0m
Frequency
(a) Relative Log Amplitudes
Spectrum
(b) Detection Results
(c) Stride-8 Feature
Figure 5: VIT vs. ViT-Adapter Feature. (a) Relative log amplitudes of Fourier transformed feature
maps. (b) Detection results. (c) Stride-8 feature map. Compared to the VIT baseline (Li et al., 2021b), our ViT-Adapter captures more high-frequency signals, and produces more ane-grained features with rich edges and textures, which is of great help for dense prediction.
Method
Components
Interaction Mode
Mask R R-CNN 1x APb
AP
AP"
#Param
SPM Injector Extractor
APM #Param
40.2
37.1
43.8M
ViT-S (Li al., 202
40.2 37.1
43.8M
43.2
38.9
45.5M
Variant
Add
41.6 38.0
45.1M
43.9
39.4
46.2M
Variant 2
Attention
42.6
38.8
46.6M
39,9
47.8M
ViT-Adapter-S (ours)
Attention
44.7 39,9
47.8M
44.7
39.8
49.4M
Table 6: Ablation studies of ViT-Adapter. (Left) ablation of key components. Our proposed com- ponents collectively bring 4.5 APb and 2.8 APm gains. (Right) ablation of the number of interactions N. The model gives the best performance when N = 4. SPM is short for the spatial prior module.
Attention Mechanism
Complexity
APb
APm
FLOPs
#Param
Train Time
Memory
Global Attention (Vaswani al., 2017)
Quadratic
43.7
39.3
1080G
50.3M
1.61s
#19.0G
CSwin Attention (Dong al., 2021)
Linear
43.5
39.2
456G
50.3M
0.56s
15.6G
Pale Attention (Wu et al., 2022a)
Linear
44.2
39.8
458G
50.3M
0.75s
17.4G
Deformable Attention (Zhu al., 2020)
Linear
44.7
39.9
403G
47.8M
0.36s
13.7G
Table 7: Ablation of using different attention mechanisms in our adapter. The per-iteration training time and GPU training memory measured by A100 GPUs with per-GPU batch size and FP16 training. "*" indicates using activation checkpointing to save training memory.
Number of Interactions. In the right side of Table 6, study the effect of the number of interac- tions. Specifically, we build several ViT-Adapter-S variants with different numbers of interactions. We observe that the model accuracy saturates when N goes larger, and applying more interactions cannot monotonically promote the performance. Therefore, we empirically set N to 4 by default.
Attention Type. Our method is general framework in which the attention mechanism is replace- able. To verify this, we adopt ViT-Adapter-S as the basic model and study 4 different attention mechanisms. As shown in Table 7, sparse attention with linear complexity is more suitable for our adapter than global attention with quadratic complexity. We ended up using deformable at- tention (Zhu et al., 2020) as the default configuration. Notably, it can be replaced by other more advanced attention mechanisms in the future to further boost performance.
5 CONCLUSION
This work explores new paradigm, namely ViT-Adapter, to bridge the performance gap between the plain ViT and vision-specific transformers on dense prediction tasks. Without modifying the inherent architecture, we flexibly inject image-related inductive biases into the VIT and reconstruct fine-grained multi-scale features required by dense predictions. Extensive experiments on object detection, instance segmentation, and semantic segmentation show that method can achieve comparable or even better performance than well-designed vision-specific transformers, and further derive considerable benefits from advanced multi-modal pre-training.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENT
This work is partly supported by the National Natural Science Foundation of China (Grant No. 61672273, 61832008), and the Shanghai Committee of Science and Technology (Grant No. 21DZ1100100).
REFERENCES
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey Hinton. Layer normalization. arXiv preprint
arXiv:1607.06450, 2016. 5
Hyojin Bahng, Ali Jahamian, Swami Sankaranarayanan, and Phillip Isola. Exploring visual prompts for adapting large-scale models. arXiv preprint arXiv:2203.17277, 1(3)4, 2022. 18
Hangbo Bao, Dong, and Furu Wei. Beit: Bert pre-training image transformers. ICLR, 2022. 3. 15, 16
Navaneeth Bodla, Bharat Singh, Rama Chellappa, and Larry S Davis. Soft-nms- impprving object detection with one line of code. In ICCV, pp. 5561-5569, 2017. 15
Daniel Bolya, Sean Foley, James Hays, and Judy Hoffman. Tide: general toolbox for identifying object detection errors. In ECCV, pp. 558-573, 2020. 17
Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context.
In CVPR, pp. 1209-1218, 2018. 17
Zhaowei Cai and Nuno Vasconcelos. Cascade r-cnn: high quality object detection and instance segmentation. TPAMI, 13835)1483-14443 2019. 6,
Kai Chen, Jiangmiao Pang, Jiaqi Wang, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jianping Shi, Wanli Ouyang, al. Hybrid task cascade for instance segmentation. In CVPR, pp. -1974-4883. 2019a. 15
Kai Chen, Jiaqi Wang, Jiangmiao Pang. Yuhang Cao. Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng, Chenchen Zhu, Tianheng Cheng, Qijie Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang, Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin. MMDetection: Open mmlab detection toolbox and benchmark. arXiv preprint 1rXiv:1906007155 2019b. 5
Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping Luo. Adaptformer: Adapting vision transformers for scalable visual recognition. arXiv preprint arXiv:2205.13553, 2022. 18
Wuyang Chen, Xianzi Du Fan Yang, Lucas Beyer, Xiaohua Zhai Tsung- Yi Lin, Huizhong Chen, Jing Li, Xiaodan Song, Zhangyang Wang, et al. A simple single-scale vision transformer for object localization and instance segmentation. arXiv preprint arXiv:2112.09747,, 2021.
Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked- attention mask transformer for universal image segmentation. arXiv preprint 12215:21 2021. 8, 16
Xiangxiang Chu, Zhi Tian, Yuqing Wang, Bo Zhang, Haibing Ren, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Twins: Revisiting the design of spatial attention in vision transformers. NeurIPS, 34, 2021a. 7,
Xiangxiang Chu, Zhi Tian, Bo Zhang, Xinlong Wang, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Conditional positional encodings for vision transformers. arXiv preprint 200. 2021b.
MMSegmentation Contributors. MMSegmentation: Openmmlab semantic segmentation toolbox and benchmark. https://gtt hub. com www//pto/o//eectctccccceee.e ab abbmmseemeettatee on, 2020.
10

---

Published as a conference paper at ICLR 2023
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR. pp. 248-255 2009.
Xiaoyi Dong, Jianmin Bao, Dongdong Chen, Weiming Zhang, Nenghai Yu, Lu Yuan, Dong Chen, and Baining Guo. Cswin transformer: A general vision transformer backbone with cross-shaped windows. arXiv preprini arXii:2207.000
2021. 1, 5,
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2020. 1, 3,
Hao-Shu Fang, Jianhua Sun, Runzhong Wang, Minghao Gou, Yong-Lu Li, and Cewu Lu. Instaboost: Boosting instance segmentation via probability map guided copy-pasting. In ICCV, pp. 682-691, 2019. 15
Yuxin Fang, Shusheng Yang, Shijie Wang, Yixiao Ge, Ying Shan, and Xinggang Wang. Unleash- ing vanilla vision transformer with masked image modeling for object detection. arXiv preprint arXiv:2204.02966 2022. 4
Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better vision-language models with feature adapters. arXiv preprint arXiv:21 10.04544, 2021. 3
Golnaz Ghiasi, Yin Cui, Aravind Srinivas, Rui Qian, Tsung- Lin, Ekin D Cubuk, Quoc V Le, and Barret Zoph. Simple copy-paste is strong data augmentation method for instance segmentation. In CVPR, pp. 2918-2222222 2918-2928, 2021. 18
Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, and Yunhe Wang. Transformer in transformer. NeurIPS, 34, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, pp. 770-778, 2016. 4
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-enn. In ICCV, pp. 2961- 2969, 2017. 6
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. arXiv preprint arXiv:2111.06377, 2021. 3, 15, 18
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly, Parameter-efffiient transfer learning for nlp. In ICML, pp. 2790-2799 2019. 2,
Shihua Huang, Zhichao Lu, Ran Cheng, and Cheng He. Fapn: Feature-aligned pyramid network for dense image prediction. In ICCV, pp. 864 873, 2021a. 16
Zilong Huang, Youcheng Ben, Guozhong Luo, Pei Cheng, Gang Yu, and Bin Fu. Shuffle trans- former: Rethinking spatial shuffle for vision transformer. arXiv preprint arXiv:2106.03650, 2021b. 6
Jitesh Jain, Anukriti Singh, Nikita Orlov, Zilong Huang, Jiachen Li, Steven Walton, and Humphrey Shi. Semask: Semantically masked transformers for semantic segmentation. arXiv preprint arXiv:21 2021. 16
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie. Serge Belongie, Bharath Hariharan, and Scr-Nam Lim. Visual prompt tuning. arXiv preprint arXiv:2203.12111 2022. 18
Shibo Jie and Zhi-Hong Deng, Convolutional bypasses are better vision transformer adapters. arXiv preprint arXiv:2207.07033 07709999, 2022. 18
Alexander Kirillov, Ross Girshick. Kaiming He, and Piotr Dollár, Panoptic feature pyramid net- works. In CVPR, pp.
2019.
11

---

Published as a conference paper at ICLR 2023
Feng Li, Hao Zhang, Shilong Liu, Lei Zhang, Lionel M Ni, Heung- Yeung Shum, al. Mask dino: Towards a unified transformer- based framework for object detection and segmentation. arXiv preprint arXiv:2206.002 027777777 2022a. 16
Xiang Li, Wenhai Wang. Lijun Wu, Shuo Chen, Xiaolin Hu, Jun Li, Jinhui Tang, and Jian Yang. Generalized focal loss: Learning qualified and distributed bounding boxes for dense object detec- tion. NeurlPS, 13:21002-21011 2020. 6
Yanghao Li, Chao- Yuan Wu, Haoqi Fan, Karttikeya Mangalam, Bo Xiong, Jitendra Malik, and Christoph Feichtenhofer. Improved multiscale vision transformers for classification and detection. arXiv preprint 1
2021a. 15
Yanghao Li, Saining Xie, Xinlei Chen, Piotr Dollar, Kaiming Hc, and Ross Girshick. Benchmarking detection transfer learning with vision transformers. arXiv preprint arXi::2111.11429 2021b. 3, 6, 7, 8, 9, 15, 17, 18, 19, 20
Yanghao Li, Hanzi Mao, Ross Girshick, and Kaiming He. Exploring plain vision transformer back- bones for object detection. arXiv preprint arXiv:2203.16522 2022b. 6, 7, 16, 18
Yawei Li, Kai Zhang, Jiezhang Cao, Radu Timofte, and Lue Van Gool. Localvit: Bringing locality to vision transformers. arXiv preprint arXiv:2 104,0707, 2021c. 1
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and Lawrence Zitnick. Microsoft coco: Common objects context. In ECCV, pp- 740-755, 2014. 2,5
Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, et al. Swin transformer v2: Scaling up capacity and resolution. arXiv preprint arXiv:2111. 11.98883, 2021a. 15, 16
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, pp. 10012- 10022, 2021b. 1, 2, 3, 5, 6, 7, 8, 15,
Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. Video swin transformer. arXiv preprint arXiv:2106. 13230, 2021c. 2
Zhuang Liu, Hanzi Mao, Chao- Wu, Christoph Feichtenhofer. Trevor Darrell, and Saining Xie.
A convnet for the 2020s. arXiv preprint arXiv:2201.0325 00354,, 2022. 6
Iyya Loshchilov and Frank Hutter.
Decoupled weight decay regularization. arXiv preprint
arXiv:1711.05101, 2017, 6, 15
Namuk Park and Songkuk Kim. How do vision transformers work? arXiv:2202.06709, 2022
arXiv preprint
4,8 8
Zhiliang Peng, Wei Huang, Shanzhi Gu, Lingxi Xie, Yaowei Wang, Jianbin Jiao, and Qixiang Ye. Conformer: Local features coupling global representations for visual recognition. In ICCV, pp. 367-376, 2021.
Zhiliang Peng, Li Dong, Hangho Bao, Qixiang Ye, and Furu Wei. Beit v2: Masked image modeling with vector-quantized visual tokenizers. arXiv preprint (exxix2208.06366 2022. 2, 8, 15, 16
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh. Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748-8763 PMLR, 2021. 15, 16
René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. In ICCV, pp. 12179-12188, 2021. 3
Yongming Rao, Wenliang Zhao, Yansong Tang, Jie Zhou, Ser-Nam Lim, and Jiwen Lu. Hor- net: Efficient high-order spatial interactions with recursive gated convolutions. arXiv preprint arXiv:2207. 14284, 2022. 16
12

---

Published as a conference paper at ICLR 2023
Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. Learning multiple visual domains with residual adapters. NeurIPS, 30, 2017. 3
Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. Efficient parametrization of multi- domain deep neural networks. In CVPR, pp. 8119-8127, 2018.
Amir Rosenfeld and John Tsotsos. Incremental learning through deep adaptation. TPAMI, 42(3): 651-663, 2018. 3
Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. Objects365: large-scale, high-quality dataset for object detection. In ICCV, pp. 8430- 8439, 2019. 2, 15, 16
Chenyang Si, Weihao Yu, Pan Zhou, Yichen Zhou, Xinchao Wang, and Shuicheng Yan. Inception transformer. arXiv preprint wwXi::2205.12956, 2022. 8
Andreas Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. How to train your vit? data, augmentation, and regularization in vision transformers. arXiv preprint arXiv:2106.102777 2021. 6, 7,8,
Asa Cooper Stickland and lain Murray. Bert and pals: Projected attention layers for efficient adap- tation in multi-task learning. In ICML, pp. 5986-5995, 2019. 2, 3
Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid. Segmenter: Transformer for semantic segmentation. In ICCV, pp. 7262-7272, 2021.
Yi-Lin Sung. Jaemin Cho, and Mohit Bansal. VI-adapter: Parameter-efficient transfer learning for vision-and-language tasks. arXiv preprint arXiv:21 2021.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In ICML, pp. 10347-10357, 2021. 6,
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. NeurlPS, 30, 2017. 3,9
Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In ICCV, pp. 568-578, 2021. 3, 5, 6, 7,8
Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pvtv2: Improved baselines with pyramid vision transformer. CVMJ, pp. 1-10, 2022a. 2, 4, 7,8
Wenhui Wang. Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, al. Image foreign language: Beit pretraining for all vision and vision-language tasks. arXiv preprint arXiv: 10442, 2022b.
2, 3, 15, 16, 17
Yixuan Wei, Han Hu, Zhenda Xie, Zheng Zhang, Yue Cao, Jianmin Bao, Dong Chen, and Baining Guo. Contrastive learning rivals masked image modeling in fine-tuning via feature distillation. arXiv preprint urXiv:2205. /4141, 2022. 17
Haiping Wu, Bin Xiao, Noel Codella, Mengehen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. Cvt: Introducing convolutions to vision transformers. In ICCV, pp. 22-31, 2021. 4
Sitong Wu, Tianyi Wu, Haoru Tan, and Guodong Guo. Pale transformer: general vision trans-
former backbone with pale-shaped attention. In AAAI, volume 36, pp. 2731-2739, 2022a. 9
Yu-Huan WI, Yun Liu, Xin Zhan, and Ming-Ming Cheng. P2t: Pyramid pooling transformer for scene understanding. TPAMI, 2022b. 1.3
Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In ECCV, pp. 418-434, 2018. 7
13

---

Published as a conference paper at ICLR 2023
Enze Xie, Wenhai Wang. Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Seg- former: Simple and efficient design for semantic segmentation with transformers. NeurIPS, 34, 2021. 2
Jianwei Yang, Chunyuan Li, Pengchuan Zhang, Xiyang Dai, Bin Xiao, Lu Yuan, and Jianfeng Gao. Focal self-attention for local-global interactions in vision transformers. arXiv preprint arXiv:2107.00441, 2021. 6, 15
Renrui Zhang, Rongyao Fang. Peng Gao, Wei Zhang, Kunchang Li, Jifeng Dai, Yu Qiao, and Hong- sheng Li. Tip-adapter: Training-free clip-adapter for better vision-language modeling. arXiv preprint arXiv:2111 11.22111100000 1.03930, 2021. 3
Shifeng Zhang. Cheng Chi, Yongqiang Yao, Zhen Lei, and Stan Z Li. Bridging the gap between anchor-based based and anchor-free detection via adaptive training sample selection. In CVPR, pp-
9759-9768, 2020. 6
Yuanhan Zhang, Kaiyang Zhou, and Ziwei Liu. arXiv:2206.0mm 04673, 2022. 18
Neural prompt search. arXiv preprint
Sixiao Zheng. Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip HS Torr, et al. Rethinking semantic segmentation from a sequence-to-sequenee perspective with transformers. CVPR, pp. 6881-6890, 2021. 3, 18
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In CVPR, pp. 633-641, 2017. 2,7
Jinguo Zhu, Xizhou Zhu, Wenhai Wang, Xiaohua Wang, Hongsheng Li. Xiaogang Wang, and Jifeng
Dai. Uni-perceiver-mos moe: Learning sparse generalist models with conditional moes. arXiv preprint arXiv:22066046674 046744 2022. 2, 3, 15
Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. In ICLR, 2020. 5,
Xizhou Zhu, Jinguo Zhu, Hao Li, Xiaoshi Wu, Xiaogang Wang, Hongsheng Li, Xiaohua Wang, and Jifeng Dai. Uni-perceiver: Pre-training unified architecture for generic perception for zero-shot and few-shot tasks, arXiv preprint arXiv:2112.01522 2021. 2. 15
14

---

Published as a conference paper at ICLR 2023
Backbone Pre-train
val
(+MS)
test-dey APb
test-dov (+MS)
Method
Framework
Epoch
AP
Apm
APb
Apm
Apm
APo
APm
Swin-L
HTC++
72
IN-22K, sup
$7.1
49.5
58.0
50.4
57.7 50.2
58.7
51.1
Focal-L
ITCC+
36
IN-22K, sup
57.0
49.9
58.1
50.9
58.4
51.3
MViTv2-L
Cascade
50
IN-22K, sup
56.9
48.6
58.7
50.5
MV
Cascade
50
IN-22K, sup
57.1
48.8
58.4
50.1
CBV2-Swin-L
НTC
36
IN-22K, sup
59.1
51.0
59.6
51.8
59.4
51.6
60.1
52.3
ViT-Adapter-L
HTC++
36
IN-22K, sup
49.0 56.6
57.7
49.9
57.4 50.0
58.4
50.7
Swin-L
HTC++
36
IN-IK, UM-MAE
57.4
49.8
58.7
50.9
ViTDet-L
Cascade
100
IN-IK, MAE
59.6
51.1
60.4
52.2
ViT-Adapter-
HTC++
36
IN-22K, BEIT
58.4
50.8
60.2
52.2
58.9
51.3
60.4
52.5
ViT-Adapter-L
HTC++
36
MM BEiTv2
58.8 51.1
60.5 52.5
59.5
51.8
60.9
53.0
Table 8: Comparisons with the leading results on the COCO val2017 and test-dev sets. There are three detection frameworks used, including Cascade Mask R-CNN (Cai & Vasconcelos, 2019), HTC (Chen et al., 2019: 2019a), and its extension HTC++ (Liu al., 2021b). All of these models trained without extra detection datasets, such as Objects365 (Shao et al., 2019). "IN-22K, sup" is short for ImageNet-22KK supervised pre-training. "MS" indicates multi-scale testing. "t":: Since the pre-trained CLIP (Radford et al., 2021) model is used in the training process of vE (Peng et al., 2022), regard it as a multi-modal pre-training method.
A COMPARISON WITH PREVIOUS STATE-OF-THE-ARTI
In recent years, the state-of-the-art models dense prediction benchmarks primarily vision- specific transformers, such as Swin (Liu et al., 2021b), Focal (Yang et al., MViTv2 (Li et al., 2021a), and SwinV2 (Liu et al., 2021a), while the plain ViT is rarely found. Nevertheless, we argue that the plain ViT still has the potential to reach the leading performance by leveraging our ViT-Adapter. To verify this, we conduct extensive additional experiments as follows.
MViTv2
A.1 OBJECT DETECTION AND INSTANCE SEGMENTATION
Settings. Following prior art (Li et al., 2021b), we modify the 24-layer VIT-I to use 14x 14 window attention except for layers spaced at an interval of to save training time and memory. The state- of-the-art detector HTC++ (Liu et al., 2021b) is employed for our experiments. Specifically, we rescale the shorter side of images between 400 and 1400, while the longer side is at most 1600. Instaboost (Fang et al., 2019), Soft-NMS (Bodla et al., 2017), W (Loshchilov & Hutter, 2017) optimizer (batch size of 16, initial learning rate of 1 10 and weight decay of 0.05), and 3x schedule adopted during training. We use layer-wise learning rate decay of 0.9, and drop path rate of 0.4. For a fairer comparison, here we take two initialization strategies, i.e. regular ImageNet-22K pre-training, and more advanced self-supervised multi-modal pre-training.
Results with ImageNet-22KK Pre-training. As shown in Table 8, with the ImageNet-22K super- vised pre-training from AugReg (Steiner et al., 2021), our ViT-Adapter-L reports 58.4 APb and 50.7 Apm on the COCO test-dev, which is comparable to many vision-specific transformers such as Swin-L (58.4 AP vs. 58.7 AP") and Focal-L (58.4 APb vs. 58.4 AP*). This fair comparison illustrates that, our ViT-Adapter significantly narrows the performance gap between the plain ViT and well-designed vision-specific models.
Results with More Advanced Pre-training. Since our paradigm retains the flexibility of the plain VIT, it can easily derive significant benefits from advanced pre-training techniques, such as multi- modal pre-training (Zhu et al., 2021; 2022) or self-supervised pre-training (Bao et al., 2022; Peng et al., 2022; He et al., 2021), or a combination of the both (Wang et al., 2022b), Here, we take the readily available weights from BEiT (Bao et al., 2022) and BEiTv2 (Bao et al., 2022) as examples. Due to BEiT using learnable relative position biases instead of the absolute position embeddings, replace the remaining global attention (see the settings part) with 56 56 window attention an approximation. For these layers, the relative position biases need to be interpolated to adapt to the new window size.
15

---

Published as a conference paper at ICLR 2023
Backbone Pre-train
Extra
Crop Size
ADE20K
Method
Framework
Iters
#aram
Pre-train
mloU
+MS
Swin-L
Mask2Former
IN-22K, sup
640 160k
56.1
57.3
215M
Swin-L-FaPN
Mask2Former
IN-22K, sup
640 160k
56.4
57.7
217M
ScMask-Swin-L
Mask2Former
IN-22K, sup
640
160k
57.0
58.2
HorNet-L
Mask2Former
IN-22K, sup
640
160k
57.5
57.9
ViT-Adapter-L.
Mask2Former
IN-22K, sup
640
160k
56.8
57.7
438M
BEiT-L
UperNet
IN-22K, BEiT
640
160k|
56.7
57.0
441M
ViT-Adapter-I.
UperNet
IN-22K, BEiT
640
160k
58.0
58.4
451M
BEiTv2-I.
UperNet
IN-22K, BFiTv2
512
160k
57.5
58.0
ViT-Adapter-I.
UperNet
IN-22K, BFiTv2
512 160k
58.0
58.5
451M
ConvNeXt-XL-
Mask2Former
IN-22K, sup
COCO-Stufi sup
896
80k
57.1
58.4
588M
Swin-L.*
Mask2Former
IN-22K, sup
CCofO-Sufff, sup
896
80k
57.3
58.3
434M
SwinV2-G
UperNet
IN-22K, sup
Ext-70M, sup Ext-70M, sup
896
160k
59.3
59.9
3.0B
FD-SwinV2-G
UperNet
IN-22K, sup
896 160k
61.4
3.0B
Swin-L
Mask DINO
IN-22K, sup
Objects365, sup
223M
ViT-Adapter-L
Mask2Former
IN-22K, BETT
COCO-Stuft, sup
896
80k
59.4
60.5
571M
ViT-Adapter-L
Mask2Former
MM'. BEiTv2
COCO-Stuff, sup
896
80k
61.2
61.5
571M
BETT-3 (w/ ViT-Adapter)
Mask2Former
MM, BETT-3
COCO-Stuff, sup
896 80k
62.0
62.8
Table 9: Comparison with previous state-of-the-art results on the ADE20K validation set. The results of BEiT-3 are collected from (Wang et al., 2022b). weth We follow BETT (Bao et al., 2022) to use a wider segmentation head for NeX-XeX and Swin-L to match the number of parame- ters, and apply the same training strategy to them. "IN-22K, sup": ImageNet-22K supervised pre- training. "MM": Multi-modal pre-training. "MS"; Multi-scale testing. "I"; Since the pre-trained CLIP (Radford et al., 2021) ) model is used in the training process of BEiTv2 (Peng et al., 2022), we regard it multi modal pre-traue pre-training method.
As reported in Table ViT-Adapter-L (w/ BEiT) creates 60.4 APb and 52.5 Apm the COCO test-dev, and ViT-Adapter-L (w/ BEiTv2) further sets this record to 60.9 AP and 53.0 APm.. No- tably, although it's not perfectly controlled comparison, our method attains similar performance taking fewer training epochs (36 vs. 100) than ViTDet (Li et al., 2022b). We argue that a longer training schedule such as 100 epochs may bring an added bonus, but expensive afford due to limited computing resources. In summary, from a system-level perspective, our ViT-Adapter can enjoy the dividends of various advanced pre-training techniques and help plain ViT achieve leading performance on the object detection and instance segmentation tasks.
A.2 SEMANTIC SEGMENTATION
Settings. For semantic segmentation, we employ the AdamW optimizer with an initial learning rate of 2x 10-3 batch size of 16, and weight decay of 0.05. Layer-wise learning rate decay of 0.9 and drop path rate of 0.4 are used to train the models. Other training settings, such as pre-training techniques, crop size, and the number of iterations. are listed in Table 9.
Results with ImageNet-22K Pre-training, As shown in Table 9, using Mask2Former (Cheng et al., 2021) as the segmenter, our WTT-Adapter-L-achieeee 56.8 mloU and 57.7 MS mloU on the ADE20K val, which is comparable to recent vision-specific models, such as Swin-L (Liu et al., 202 Swin- L-FaPN (Huang et al., 202 aa) SrMaakk-Sii--1 (Jain al., 2021), and HorNet-L (Rao al., 2022).
Results with More Advanced Pre-training. It can be seen from Table 9, when training with Uper- Net for 160k iterations, our ViT-Adapter-L. (w/ BEiT) yields 58.4 MS mloU, outperforming BEiT-I. by 1.4 points with only 10M additional parameters. shows that our adapter can deliver significant benefits even for a powerful suff-super pre-trained VIT.
Furthermore, we compare the performance of our method with vision-specific models that also use additional datasets. For example, SwinV2-G (Liu et al., 2021a) uses a privately collected ImageNet- 22K-ext-70M, dataset that contains 70 million images. Mask DINO (Li et al., 2022a) takes the detection pre-training on large-scale Objects365 (Shao et al., 2019) dataset as the initialization for segmentation. Due limited computing resources, explore simple and affordable transfer
16

---

Published as a conference paper at ICLR 2023
Settings ViT
Settings of Adapter
Total
Variants
Layers
Width
FFN
Heads
#Param
FFN
Heads
#Param
Param
Tiny
12
192
768
3
5.5M
48
2.5M
8.0M
Small (S)
12
384
1536
6
21.7M
96
5.8M
27.5M
Base (B)
12
768
3072
12
85.8M
4
192
12
14.0M
99.8M
Large (L)
24
1024
4096
16
303.3M
4
256
16
23.7M
327.0M
Table Configurations the ViT-Adapter. We apply our adapters on four different settings of VIT, including VIT-T, ViT-S, ViT-B, and ViT-L, covering wide range of different model sizes.
VIT-T (Ü
ViT-S
a VVT -dT-ter lours)
VIT Adapter S (ours)
ViT-Adapter- (ours)
6
41
4
2
cls
loc
both
dupe
bkg
miss
dls
loc
both
dupe
bkg
miss
ds
loc
both
dupe
bkg
miss
FN
FP
FP
FP
12
16
12
16
(a) ViT ViT-Adapter-T
(b) ViT-Adapter-S
(c) ViT-Adapter-B
TIDE error type analysis (the lower the better). We use the models listed in Table for analysis. As defined in (Bolya et al., 2020), plot the APh metric an IoU threshold of 0.5. These bars show the effect of each error type on overall detection performance). The error types include: cls: localized correctly but classified incorrectly; loc: classified correctly but localized incorrectly; both: classified incorrectly and localized incorrectly; dupe: detection would he correct not for a higher scoring detection; bkg: detected background foreground; miss: all undetected ground- truth not covered by other error types; FN: false negatives; FP: false positives, We observe that our ViT-Adapter makes fewer localization and miss errors than the VIT baseline (Li et al., 2021b), and occurs fewer false positive and negative errors.
Figure 6:
learning strategy for semantic segmentation. Specifically, we use the COCO-Stuff (Caesar et al., 2018) dataset for 80k iterations of pre-training, and then ADE20K for 80k iterations of fine-tuning. The total number of iterations is still 160k, and no additional training overhead is added.
Under this setting, our ViT-Adapter-L. (w/ BEiT) produces an exciting score of 60.5 MS mloU. Further, ViT-Adapter-L (w/ BEiTv2) creates new record of 61.5 MS mloU, which is slightly better than FD-Swin V2-G (Wei al., 2022), while the parameter number is much smaller (57IM vs. 3.0B). It's worth noting that, our ViT-Adapter is also adopted by the recently proposed BETT-3 (Wang al., 2022b), which is ViT-style foundation model that can pre-trained with multi-modal data. As described in their paper, using ViT-Adapter for the transfer learning of semantic segmentation, BEiT-3 establishes a new state-of-the-ar of 62.8 MS mIoU on ADE20K val, which is a convincing verification of the paradigm we present in Figure
B ADDITIONAL ABLATION AND DISCUSSION
Architecture Configurations. The more detailed configurations listed in Table 10.
TIDE Error Type Analysis. TIDE (Bolya al., 2020) is a toolbox for analyzing the sources of emoor in object detection algorithms. Following (Li et al., 2021b), we show the error type analysis in Figure 6. For fair comparison, the models listed in Table are adopted for analysis. These results reveal where our ViT-Adapter improves overall APh relative to the ViT baseline (Li al., 2021b). For instance, we observe that our adapter helps reduce missed and localization errors, and has a substantial effect fixing false negative and positive errors.
17

---

Published as a conference paper at ICLR 2023
Feature Visualization. We plot more visualization of feature maps produced by ViT-B (Li al., 2021b) and our ViT-Adapter-B B in Figure 7 and Figure 8, which are trained based on Mask R R-CNN for detection and UperNet for segmentation, respectively. As can be seen, the features of ViT-B are blurry and coarse, while our features are more refined and have more local edges and textures. This observation also accords with the Fourier analysis in Section 4.4, which demonstrates that VIT has the characteristics of capturing low-frequency information, and our ViT-Adapter can supplement the missing figh-frequency signals.
Comparison with SETR. Like ViTDet (Li et al., 2022b), SETR (Zheng et al., 2021) also changes the shape of features of ViT according to the task prior (see Figure 3(a)), thus allowing ViT to achieve better segmentation performance. Although this paradigm shares some similarities with our approach, e.g. combining ViT and convolutions, they have three main differences: (1) addition to the task prior, our method also takes the information of the input image (the input prior) into consid- eration when adapting ViT' dense prediction tasks; (2) The input prior will constantly interact with VIT's features, making the output features more suitable for dense prediction tasks; (3) Our method is an adapter that is general in both detection and segmentation tasks, and moreover achieves better results than segmentattion
head SETR (Zheng et al., 2021).
Comparison with other Adapters. We would like to clarify the differences between ViT-Adapter and other adapters (Jia et al., 2022; Bahng et al., 2022; Chen et al., 2022; Zhang et al., 2022; Jie & Deng, 2022) for ViT', from two aspects as follows:
(1) Different tasks. Our method is designed for dense prediction tasks, while VPT (Jia al., 2022), Visual Prompt (Bahng et al., 2022), AdaptFormer (Chen et al., 2022), NOAH (Zhang et al., 2022), and Convpass (Jie & Deng, 2022) are mainly proposed for classification tasks. By training the parameters only in input spaces, or some modules attached to the backbone, or their combination, these models perform well on classification and even obtain better results than full-tuning models.
However, when applying these methods (Jia al., 2022: Bahng et al., 2022; Chen al., 2022; Zhang et al., 2022; Jie & Deng, 2022) to
Trainable Param.
mloU (ss/ms)
Method
ViT-L (full-tuning)
318M
488.3 50.1
dense prediction tasks, they perform below ex- pectations. For example, we see from Table 11
ViT-L (w/ VPT)
16M
44.0 45.66
ViT-Adapter-L. (full-tuning)
332M
52.9 53.7
that the performance of VPT (Jia et al., 2022)
ViT-Adapter-L (frozen ViT)
30M
49.0 50.6
has a large gap with the baseline VIT-L (Zheng et al., 2021) on ADE20K.
Table 11: Comparison with VPT.
(2) Different targets. These mentioned adapters (Jia et al., 2022; Bahng et al., 2022; Chen et al., 2022: Zhang et al., 2022: Jie & Deng, 2022) aim to explore parameter-efficient transfer learning, while the goal of our ViT-Adapter is to push the performance boundaries of plain ViT downstream applications, make ViT more general for downstream tasks, and efficiently utilize large-scale ViT weights pre-trained in different ways. We argue that these two technical lines are orthogonal, as shown in the last column in Table 11. Combining ViT-Adapter with these adapters to achieve effi- cient and accurate transfer learning of dense prediction is research topic worth exploring.
ViTDet's Performance. The higher performance of the original ViTDet (Li et al., 2022b) comes from stronger training settings. Specifically, ViTDet adopts more expensive training strategy than ours, i.e., loading the MAE (He et al., 2021) pre-trained weights, and using the Large Scale Jit- ter (Ghiasi et al., 2021) augmentation to train the model for 100 epochs. This setting leads to almost 3 times the training cost compared to the commonly used 36 epochs (i.e., 3x+MS schedule). And to some extent, reveals the lack of image-related inductive biases in ViT will lead to slow conver- gence on dense prediction tasks.
For fair comparisons, we benchmark all plain ViT detectors, including ViTDet (Li et al., 2022b) and our ViT-Adapte under the commonly used 3X+MS training schedule, and use the same ImageNet- IK pre-trained weights (i.e., DeiT) as initialization. It makes sense that our ViT-Adapter achieves better performance than ViTDet under this setting, because our adapter injects image-related prior into the plain ViT, which can speed up convergence and improve performance.
18

---

Published as a conference paper at ICLR 2023
VIT-B
ViT AIT-Adapte--
Vir-B
Adapter-
VIT-B
ViT- ArT-Adapte--
ViT-B
Adapter-
VIT Adapter-B
Detection Results
Stride-4 Feature Map
Stride-8 Feature Map
Stride-16 Map
Stride-32 Map
Visualization of feature maps for object detection and instance segmentation. Com- pared to the ViT baseline (Li et al., 2021b), our ViT-Adapter yields more fine-grained multi-scale feature maps, thus improving localization quality and reducing missed detection.
Figure 7:
19

---

Published as a conference paper at ICLR 2023
VIT-B
VIT PiT-Adapter-B
ViT-B
VIT Ada-ter----
VIT-B
ViT-Adapter-B
ViT-B
VIT Adapter-
VIT-8
VIT Adaater- 8
Segmentation Results
Stride-4 Feature Map
Stride-8 Feature Map
Stride-16 Map
Stride-32 Map
Figure 8: baseline (Li et al., 2021b), our ViT-Adapter yields more fine-grained multi-scale feature maps with rich local edges and textures, thus improving the performance of semantic segmentation.
Visualization of feature maps for semantic segmentation. Compared to the ViT
20


Reviews:
[{"review_id": "izzo9aSu2QG", "reviewer": "Reviewer_hMeh", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed ViT-Adapter for dense prediction task. Essentially, Vit-Adapter utilizes additional modules to inject the image prior into ViT, encouraging ViT to learn transferable representations for the specific dense prediction tasks. The extensive experiments on several dense prediction tasks (e.g., object detection, instanve segmentation, and semantic segmentation) demonstrate the effectiveness of the proposed method.", "summary_of_the_review": "This work is interesting and written-well, but some of concerns needed to be improved. I'm willing to raise my rating score if authors carefully tackle my concerns.", "strength_and_weaknesses": "Strengths:\n+ The paper is written-well and easy to follow, the idea is intuitive and clear.\n+ The experiments are solid and entensive.\n+ The authors have made good efforts to ablate different components of their approach.\n+ The performances improvement of the proposed method are impressed.\n\nWeaknesses:\n- Adapters in ViT is an interesting research topic, there were some works focused on it  [a-e]. I strongly encourage authors to take them into deep discussion. For example, the differences and advantages between them. The additional fair experiments should be  also used for discussion.\n\n[a] \"Visual prompt tuning.\" In ECCV, 2022.\n\n[b] B\"Exploring visual prompts for adapting large-scale models.\" arXiv, 2022.\n\n[c] \"AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition.\" In NeurIPS, 2022.\n\n[d]  \"Neural Prompt Search.\" arXiv, 2022.\n\n[e] \"Convolutional bypasses are better vision transformer adapters.\" arXiv, 2022.\n\n- Authors inject spatial priors and multi-scale feautres learning into ViT and achieves significant performances. I doubt that can we add the other priors or learning tricks into ViT to achieve better performances. That is, the module in Vit-Adapter can be replaced by others that is important for dense prediction tasks? Can authors provide additional ablation studies on it.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "i4dS3DZT2j", "reviewer": "Reviewer_Ycr4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper designed a simple yet effective adapter module that helps plain ViT behave well for dense prediction tasks, including object detection, instance segmentation, and semantic segmentation. Benefited by the inductive bias introduced by the spatial prior module and feature interaction module, the plain ViT archives remarkable performance on dense prediction tasks.", "summary_of_the_review": "See above.", "strength_and_weaknesses": "#### Strength\n\n1. Promising performance. ViT-Adapter achieves remarkable performance on several dense prediction benchmarks, using the plain ViT model.\n\n2. The paper is clearly written and easy to follow.\n\n3. Extensive experiments. The experiments are solid and comprehensive.\n\n\n\n#### Weakness\n\n1. Only the number of parameters is reported in the table. More metrics such as FLOPs and inference latency are more important.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "hlsRZ8ommEb", "reviewer": "Reviewer_SVHK", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The application of plain ViT on dense prediction tasks suffers from unacceptable complexity. The paper proposes an injection adapter with multi-scale features to use a well-pretrained ViT model and transfer it to downstream task. The experiments are conducted on object detection, instance segmentation, and semantic segmentation. The proposed technique achieves state-of-the-art on COCO dataset, which well demonstrate the high effectiveness.", "summary_of_the_review": "This paper explores an important problem in multimodality and proposes a simple yet effective solution to transfer the ViT models to dense prediction tasks. The experiments give strong evidences to well support.", "strength_and_weaknesses": "Strength:\n+ Applying the plain ViT to downstream tasks is an improtant and meanful topic. The motivation is well explained and makes sense. The proposed technique is simple yet effectiveness.\n+ The paper is well-written and well-organized. The figures and visualizations are clear and easy to follow.\n+ The experiments and ablation studies are comprehensive. The proposed adapter shows significant superiority over the previous works.\nAnd it achieves state-of-the-art performance on several tasks.\n\nWeakness:\n- It would be better to give the computational complexity and latency in the experiments.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "4azeCiM-5hQ", "reviewer": "Reviewer_BCxb", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work attempts to address the problem of lack of inductive bias in plain ViT architectures which hinders its performance for dense prediction tasks such as semantic segmentation and object detection. Specifically, the authors propose an adapter architecture that can extract and learn representations that are useful for dense prediction tasks. The proposed methodology has been trained and tested on different down-stream tasks and purportedly achieves competitive performance on COCO test-dev dateset in terms of box AP and mask AP. ", "summary_of_the_review": "The proposed model addresses the problem of lack of inductive bias in ViT architecture and has been shown to be effective in many different application. The model has been testes rigorously on different dataset and achieves competitive performance on COCO test-dev dateset. As a result, I believe these contributions merit acceptance. ", "strength_and_weaknesses": "########## Strength\n\n1. The paper is well-written and easy to follow. Visualizations properly demonstrate the concept of the proposed architecture. \n\n2. The overall design of the ViT-adapter architecture is quite general and can be used in different applications. Especially, using conv-based prior modules seem to be effective and play an important role. Unlike the previous ViTDet model, the adapter seems to also make use of the input image which can be effectively leveraged along with intermediate extracted features. In addition, the idea of using both conv and transformer-based (i.e. encoders with MLP and cross attention) layers in the adapter seems to be more optimal than using conv-based pyramid-like architectures. \n\n3. The method has been thoroughly tested on many different tasks (both supervised and self-supervised) and achieve SOTA performance on COCO test-dev dateset --although there seems to be other approaches such as FD-SwinV2-G [1], DINO [2], etc. to have better performance, but the performance is still quite competitive. \n\n########## Weakness\n\n1. Despite the effective design of the proposed architecture, the idea of the ViT-adapter is not very new. The first instance of such works was SETR [3] which simply used a CNN-based decoder for feature processing in segmentation. \n\n2. As the name indicates, the architecture is solely designed for ViT architectures, and this may limit its applicability as hierarchical models such as Swin Transformer have gained a lot interest. It is fair to say that with minimal changes, the adapter could be used with Swin and other similar architectures. Hence it could be beneficial to study whether it could be useful when the feature extractor is more powerful. \n\n3. How does the adapter affect the throughput and memory foot-print ? I am concerned that it may not be feasible to use it in practical applications due to it complex architecture. It would be nice to have comparison to SETR and ViTDet for throughput. \n\n\n[1]: Wei, Y., Hu, H., Xie, Z., Zhang, Z., Cao, Y., Bao, J., Chen, D. and Guo, B., 2022. Contrastive Learning Rivals Masked Image Modeling in Fine-tuning via Feature Distillation. arXiv preprint arXiv:2205.14141.\n\n[2]: Zhang, H., Li, F., Liu, S., Zhang, L., Su, H., Zhu, J., Ni, L.M. and Shum, H.Y., 2022. Dino: Detr with improved denoising anchor boxes for end-to-end object detection. arXiv preprint arXiv:2203.03605.\n\n[3]: Zheng, S., Lu, J., Zhao, H., Zhu, X., Luo, Z., Wang, Y., Fu, Y., Feng, J., Xiang, T., Torr, P.H. and Zhang, L., 2021. Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 6881-6890).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%