ViTAE: Vision Transformer Advanced by Exploring Intrinsic Inductive Bias
Authors: Yufei Xu
Qiming ZHANG
Jing Zhang
Dacheng Tao
Venue: NeurIPS 2021 Poster (2021)
Keywords: Vision Transformer
Classification
Convolution
Inductive Bias

Abstract: Transformers have shown great potential in various computer vision tasks owing to their strong capability in modeling long-range dependency using the self-attention mechanism. Nevertheless, vision transformers treat an image as 1D sequence of visual tokens, lacking an intrinsic inductive bias (IB) in modeling local visual structures and dealing with scale variance. Alternatively, they require large-scale training data and longer training schedules to learn the IB implicitly. In this paper, we propose a new Vision Transformer Advanced by Exploring intrinsic IB from convolutions, i.e., ViTAE. Technically, ViTAE has several spatial pyramid reduction modules to downsample and embed the input image into tokens with rich multi-scale context by using multiple convolutions with different dilation rates. In this way, it acquires an intrinsic scale invariance IB and is able to learn robust feature representation for objects at various scales. Moreover, in each transformer layer, ViTAE has a convolution block in parallel to the multi-head self-attention module, whose features are fused and fed into the feed-forward network. Consequently, it has the intrinsic locality IB and is able to learn local features and global dependencies collaboratively. Experiments on ImageNet as well as downstream tasks prove the superiority of ViTAE over the baseline transformer and concurrent works. Source code and pretrained models will be available at https://github.com/Annbless/ViTAE.

ViTAE: Vision Transformer Advanced by Exploring Intrinsic Inductive Bias
Yufei Xu
Qiming Zhang
Jing Zhang'
Dacheng Tao
'Te University of Sydney, Australia, 2J Explore Academy, China
fyuxu7116, 22m.ii00..2a2200mmm..m... exydee.y .wydney. edu.au, au, jing. chaang@@yddeey.co eewprddeyyedu...eem .du.au, au, dacheng. tao0gmail.... com
Abstract
Transformers have shown great potential various computer vision tasks owing to their strong capability in modeling long-range dependency using the self-attention mechanism. Nevertheless, vision transformers treat an image as 1D sequence of visual tokens, lacking intrinsic inductive bias (IB) in modeling local visual structures and dealing with scale variance. Alternatively. they require large-scale training data and longer training schedules learn the IB implicitly. this paper, we propose a new Vision Transformer Advanced by Exploring intrinsic IB from convolutions, i.e., VITAE. Technically, ViTAE has several spatial pyramid reduction modules to downsample and embed the input image into tokens with rich multi-scale context by using multiple convolutions with different dilation rates. In this way, it acquires an intrinsic scale invariance IB and is able to learn robust feature representation for objects at various scales. Moreover, in each transformer layer, VITAE has convolution block in parallel to the multi-head self- attention module, whose features are fused and fed into the feed-forward network. Consequently, it has the intrinsic locality IB and is able to learn local features and global dependencies collaboratively. Experiments on ImageNet as well as downstream tasks prove the superiority of VITAE over the baseline transformer and concurrent works. Source code and pretrained models will be available at code
Introduction
73
T2T
75.3
72.6
have shown a domination trend in NLP studies owing to their strong ability in modeling long-range dependen-
Transformers 779 17 40
14
46
61
VITAE
Imogrhe Top-1 Actrayy (%)
71.6
71.2
71
cies by the self-attention mechanism 667 81 [51.
72.6
69
Such success and good properties of transform- ers has inspired following many works that apply
60,7
68.1
71.7
67.5
Intepeat
them in various computer vision tasks [19 100
70.8
97 80 [7]. Among them, ViT [19] is the pionecr- ing pure transformer model that embeds images into a sequence of visual tokens and models the global dependencies among them with stacked transformer blocks. Although achieves promis- ing performance image classification, quires large-scale training data and a longer train- ing schedule. One important reason is that ViT
65
61.1
68.7
63
20
60
100
68
100
200
300
Percentage
Epochs
Figure 1: Comparison of data and training effi- ciency of T2T-ViT-7 and ViTAE-T on ImageNet.
Equal Contribution. Interns JD Explore Academy.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

lacks intrinsic inductive bias (IB) in modeling local visual structures (e.g.. edges and corners) and dealing with objects various scales like convolutions. Alternatively, ViT has to learn such IB implicitly from large-scale data.
Unlike vision transformers, Convolution Neural Networks (CNNs) naturally equip with the intrinsic IBs of scale-invariance and locality and still serve as prevalent backbones in vision tasks [26, 770 62 8 96). The success of CNNs inspires us to explore intrinsic IBs in vision transformers. We start by analyzing the above two IBs of CNNs, i.e., locality and scale-invariance. Convolution that computes local correlation among neighbor pixels is good at extracting local features such as edges and corners. Consequently, CNNs can provide plentiful low-level features at the shallow layers [94], which are then aggregated into high-level features progressively by a bulk of sequential convolutions (7]. Moreover, CNNs have a bierarchy structure to extract multi-scale features at different 32 68
layers 688 38 26] Besides, intra-layer convolutions can also learn features at different scales by 70
45 96]. Consequently, scale- invariant feature representation can be obtained via intra- or inter-layer feature fusion. Nevertheless, CNNs are not well suited to model long-range dependencies ? which is the key advantage of transformers. An interesting question comes up: Can we improve vision transformers by leveraging the good properties of CNNs? Recently, DeiT (76) explores the idea of distilling knowledge from CNNs to transformers to facilitate training and improve the performance. However, it requires an off-the-shelf CNN model as the teacher and consumes extra training cost.
varying their kernel sizes and dilation rates (25
8
Different from DeiT, we explicitly introduce intrinsic IBs into vision transformers by re-designigg the network structures in this paper. Current vision transformers always obtain tokens with single- 77 and learn to adapt to objects at different scales from data. improves VIT by delicately generating tokens in a soft split manner.
scale context [19 93 80 86
69
For example, VIT
93
Specifically, it uses a series of Tokens-to- Token transformation layers to aggregate single-scale neighboring contextual information and progressively structurizes the image to tokens. Motivated by the success of CNNs in dealing with scale variance, we explore a similar design in transformers, i.., intra-layer convolutions with different receptive fields [70 [1), to embed multi-scale context into tokens. Such a design allows tokens to carry useful features of objects at various scales, thereby naturally having the intrinsic scale- invariance IB and explicitly facilitating transformers to learn scale-invariant features more efficiently from data. On the other hand, low-level local features are fundamental elements to generate high-level discriminative features. Although transformers can also learn such features at shallow layers from data, they are not skilled as convolutions by design. stack convolutions and attention layers sequentially and demonstrate that locality is a reasonable compensation of global dependency. However, this serial structure ignores the global context during locality modeling (and vice versa). To avoid such a dilemma, we follow the ddivide-and-conques idea and propose to model locality and long-range dependencies in parallel and then fuse the features to account for both. In this way, we empower transformers to learn local and long-range features within each block more effectively.
Recently, [8 43 21
Technically, propose new Vision Transformers Advanced by Exploring Intrinsic Inductive Bias (ViTAE), which is a combination of two types of basic cells, Le., reduction cell (RC) and normal cell (NC). RCs are used downsample and embed the input images into tokens with rich multi-scale context while NCs aim to jointly model locality and global dependencies in the token sequence. Moreover, these two types of cells share simple basic structure, i.e., paralleled attention module and convolutional layers followed by a feed-forward network (FFN). It is noteworthy that RC has an extra pyramid reduction module with atrous convolutions of different dilation rates embed multi-scale context into tokens. Following the setting in (93. we stack three reduction cells to reduce the spatial resolution by 1/16 and a series of NCs to learn discriminative features from data. ViTAE outperforms representative vision transformers in terms of data efficiency and training efficiency (see Figure []) as well as classification accuracy and generalization on downstream tasks.
Our contributions are threefold. First, we explore two types of intrinsic IB in transformers, i.e., scale invariance and locality, and demonstrate the effectiveness of this idea in improving the feature learning ability of transformers. Second, we design new transformer architecture named VITAE based two new reduction and normal cells intrinsically incorporate the above two IBs. The proposed ViTAE embeds multi-scale context into tokens and learns both local and long-range features
'Dspptt projection in transformer can be viewed as 1 x 1 convolution 9 the term of convolution here refers to those with larger kernels, e.g., 3 x 3, which are widely used in typical CNNs to extract spatial features.

---

effectively. Third, ViTAE outperforms representative vision transformers regarding classification accuracy, data efficiency, training efficiency, and generalization on downstream tasks. ViTAE achieves 75.3% and 82.0% top-1 accuracy on ImageNet with 4.8M and 23.6M parameters, respectively.
2 Related Work
2.1 CNNS with intrinsic IB
87 and downstream computer vision tasks. The convolution operations in CNNs extract local features from the neighbor pixels within the receptive field determined by the kernel size 42 Following the intuition that local pixels are more likely to be correlated in images [41), CNNs have the intrinsic IB in modeling locality. In addition the locality, another critical topic in visual tasks is scale-invariance, where multi-scale features are needed to represent the objects at different scales effectively [49] (90]. For example, to effectively learn features of large objects, large receptive field is needed by either using large convolution kernels 990 91 or series of convolution layers in deeper architectures [26] 32
CNNs have led to series of breakthroughs in image classification 38 94 26 95
68
77. To construct multi-scale feature representation, the classical idea is using image pyramid 39 16]. where features are
or learned from pyramid of images at different Accordingly, features from the small scale image mainly encode the large objects while features from the large scale image respond more to small objects. addition to the above inter-layer fusion way, another way to aggregate multi-scale context by using multiple convolutions with different receptive fields within single layer, i.e., intra-layer fusion 196 71 70 70 72). Either inter-layer fusion or intra-layer fusion empower CNNs an intrinsic IB in modeling scale-invariance. This paper introduces such an IB to vision transformers by following the intra-layer fusion idea and utilizing multiple convolutions with different dilation rates in the reduction cells to encode multi-scale context into each visual token.
8 55
resolutions respectively 44 52
63
35
3
2.2 Vision transformers with learned IB
VIT [9] is the pioneering work that applies a pure transformer to vision tasks and achieves promising results. However, since VIT lacks intrinsic inductive bias in modeling local visual structures, it indeed
learns the IB from amounts of data implicitly. Following works along this direction are to simplify
the model structures with fewer intrinsic IBs and directly learn them from large scale data [50 74 75
22 18 20 27 which have achieved promising results and been studied actively. Another direction is to leverage the intrinsic IB from CNNs facilitate the training of vision transformers, e.g., using less training data or shorter training schedules. For example, DeiT (76) proposes to distill knowledge from CNNs transformers during training. However, it requires an off-the-shelf CNN model teacher, introducing extra computation cost during training. Recently, some works try to introduce the intrinsic For example,
IB of CNNs into vision transformers explicitly (23 58 21
43
15
89
83
92
stack convolutions and attention layers sequentially, resulting in a serial structure and modeling the locality and global dependency accordingly. 80
43
21 3
28 design sequential stage-wise structures while 44 333 apply attention within local windows. However, these serial structure may ignore the global context during locality modeling (and vice versa). [88] establishes connection across different scales at the cost of heavy computation. Instead, we follow the "divide-and "awiid--nd-conquer" idea and propose to model locality and global dependencies simultaneously via a parallel structure within each transformer layer. Conformer (58), the most relevant concurrent work to us, employs unit to explore inter-block interactions between parallel convolution and transformer blocks. In contrast, in ViTAE, the convolution and attention modules are designed to be complementary each other within the transformer block. In addition, Conformer is not designed to have inherent scale invariance IB.
3 Methodology
3.1 Revisit vision transformer
We first give brief review of vision transformer in this part. To adapt transformers to vision lasks, VIT 19 first splits an image x E xW xC' into tokens with a reduction ratio of p (i.e.,
Ie (((Hxx)))')))) where H. W and C' denote the height, width, and channel dimensions of
3

---

B.l. M.8
8706 W710
DO0-10
000-1 00
000
Reductionn Call
Redictionn
-000-00
Norial Call
Kamman Cormeaa Call
Namall Call
Liwaxr
Dog
-0
00000
SILL
INN
Pyramid Reduccton
Lowe
Difherent Dilstion Rates
3 Position Embodding
Element-wise dd
Feature Map
Concat
Token
Class Token
The structure of the proposed VITAE. is constructed by stacking three RCs and several NCs. Both types of cells share a simple basic structure. i.e., an MHSA module and parallel convolutional module followed by an FFN. particular, RC has an extra pyramid reduction module using atrous convolutions with different dilation rates to embed multi-scale context into tokens.
Figure 2:
the input image, = Cp² denotes the token dimension. Then, an extra class token concatenated to the visual tokens before adding position embeddings in element-wise manner. The resulting tokens are fed into the following transformer layers. Each transformer layer is composed of two parts, i.., multi-head self-attention module (MHSA) and feed forward network (FFN).
MHSA Multi-head self-attention extends single-head self-attention (SHSA) by using different projection matrices for each head. Specifically, the input tokens H; are first projected to queries (Q), keys (K) and values (V) using projection matrices, i.e., Q, K, V I, We x.QK IvQv where Wo/k/V RDx denotes the projection matrix for query, key, and value, respectively. Then, the self-attention operation is calculated as:
QKT VD )V.
Attention (acettion(,RAee (Q, KVV) = softmart
This SHSA module is repeated for h times to formulate the MHSA module, where h is the number of heads. The output features the h heads are concatenated along the channel dimension and formulate the output of the MHSA module.
FFN FFN is placed on top of the MHSA module and applied each token identically and separately. consists of two linear transformations with an activation function in between. Besides, a layer normalization 2 and shortcut are added before and aside from the MHSA and FFN, respectively.
3.2 Overview architecture of VITAE
ViTAE aims to introduce the intrinsic IB in CNNs to vision transformers. As shown in Figure 2 ViTAE composed of two types of cells, i.e., RCs and NCs. RCs are responsible for embedding multi-scale context and local information into tokens, and NCs are used to further model the locality and long-range dependencies the tokens. Taken image
# xC as input, three RCs are used to gradually downsample by 4x, 2x, and 2x, respectively. Thereby, the output tokens of the token dimension (64 in our experiments). The
the RCs are of size H /1, W/16, D) where output tokens of RCs are then flattened as RHW/256XD concatenated with the class token, and added by the sinusoid position encoding. Next, the tokens are fed into the following NCs, which keep the length of the tokens. Finally, the prediction probability obtained using a linear classification layer on the class token from the last NC.

---

3.3 Reduction cell
Instead of directly splitting and flatten images into visual tokens based on a linear image patch embedding layer, we devise the reduction cell to embed multi-scale context and local information into visual tokens, which introduces the intrinsic scale-invariance and locality IBs from convolutions. Technically, RC has two parallel branches responsible for modeling locality and long-range depen- dency, respectively, followed by an FFN for feature transformation. We denote the input feature of the ieh RC as fi RBixWExDD D. The input of the first RC is the image æ. In the global dependencies branch, fi is firstly fed into a Pyramid Reduction Module (PRM) to extract multi-scale context, i.e.,
fTT & PRM((F) (a sig, rig) E Si, E RJ),
(2)
where Convey) (-) indicates the jth convolutional layer in the PRM (PRM, It uses a dilation rate Sty from the predefined dilation rate set S; corresponding to the ith RC. Note that we use stride convolution to reduce the spatial dimension of features by a ratio r; from the predefined reduction ratio set R. The conv features are concatenated along the channel dimension, i.e., f! e (( (B..s))) ((S.)) where |8:] denotes the number of dilation rates in S. f is then processed by an MHSA module to model long-range dependencies, i.e.
f = MWS.AMAAAAmm(((((.
where /mg2Seg(-) is a simple reshape operation to flatten the feature map to ID sequence. In this way, f embeds the multi-scale context each token. In addition, we use a Parallel Convolutional Module (PCM) to embed local context within the tokens, which are fused with f as follows:
(4)
Here, PCM((-) represents the PCM, which is composed of three stacked convolution layers and an (mg2Seqq-) operation. It is noteworthy that the parallel convolution branch has the same spatial downsampling ratio the PRM by using stride convolutions. In this way, the token features can carry both local and multi-scale context, implying that RC acquires the locality IB and scale-invariance IB by design. The fused tokens are then processed by the FFN, reshaped back to feature maps, and fed into the following RC or NC, i.e.,
f++ SeQ2Img(FNN,(F) + f.").
(5)
where the Seq2Img(-) is a simple reshape operation to reshape a token sequence back to feature maps. (-N,(-) represents the FFN in the ith RC. In our VITAE, three RCs are stacked sequentially to gradually reduce the input image's spatial dimension by 4x, 2x, and 2x, respectively. The feature maps generated by the last RC are of size of [H/16, W/16, D], which are then flattened into visual tokens and fed into the following NCs.
3.4 Normal cell
As shown in the bottom right part of Figure 2 except for the absence of the PRM. Due the relatively small 1 x) spatial size of feature maps after RCs, is unnecessary to use PRM in NCs. Given fa from the third RC, we first concatenate it with the class token fole, and then add it to the positional encodings to get the input tokens for the following NCs. Here we ignore the subscript for clarity since all NCs have an identical architecture but different learnable weights. fets is randomly initialized at the start of training and fixed during the inference. Similar to the RC, the tokens are fed into the MHSA module, i.e., ty = H.S.A A(t). Meanwhile, they Img2Seq
NCs share a similar structure with the reduction cell
reshaped to 2D feature maps and fed into the PCM, i.e., Note that the class token is discarded in PCM because it has no spatial connections with other visual tokens. To further reduce the parameters in NCs, from MHSA and PCM are then fused via element-wise sum, Le., hig = by + 4.- Finally, 4g are fed into the FFN to get the output features of NC, i.e., tnc FFN (tay) + tig. Similar VIT [19), we apply layer normalization to the class token generated by the last NC and feed it to the classification head get the final classification result.
group convolutions in PCM. The features
3.5 Model details
5

---

We use two variants of VITAE in our experiments for fair com- parison of other models with sim- ilar model sizes. The details of them are summarized in [ In the first RC, the default convo- lution kernel size is 7 x 7 with stride of 4 and dilation rates of S [1, 2, 4]. In the following two RCs, the convolution kernel
Table Model details of two variants of ViTAE.
Reduction Cell
Normal Cell
Params Maes
Model VITAE-T
Dilation
Cells
Heads Embed Cells
(M)
(G)
1,2,3,4
3
4
256
4.8
1.5
VITAE-S
1,2,3,4
3
6
384
14
23.6
5.6
size is 3 x with stride of 2 and dilation rates of S₂ [1, and Sy [1, 2], respectively. Since the spatial dimension of tokens decreases, there is no need to use large kernels and dilation rates.
PCM in both RCs and NCs comprises three convolutional layers with kernel size of 3 x 3.
4 Experiments
4.1
Implementation details
We train and test the proposed ViTAE model on the standard ImageNet [38] dataset, which contains about 1.3 million images and covers IK classes. Unless explicitly stated, the image size during training is set to 224 x 224. We use the Adam W [48] optimizer with the cosine learning rate scheduler and uses the data augmentation strategy exactly the same as T2T [93] for fair comparison, regarding the training strategies and the size of models. We use a batch size of 512 for training all our models and set the initial learning rate be 5e-4. The results of our models can be found in Table 2 where all
the models are trained for 300 epochs on 8 V100 GPUs. The models are built on Py PyTorch [57 and TIMM 82.
4.2 Comparison with the state-of-the-art
We compare our VITAE with both CNN models and vision transformers with similar model sizes in Table 2 Both Top-1/5 accuracy and real Top-1 accuracy on the ImageNet validation set are reported. We categorize the methods into CNN models, vision transformers with learned IB, and vision transformers with introduced intrinsic IB. Compared with CNN models, our ViTAE-T achieves 75.3% Top-1 accuracy, which is better than ResNet-18 18 with more parameters. The real Top-1 accuracy of the VITAE model is 82.9%, which is comparable to ResNet-50 that has four more times ViTAE-S achieves 82.0% Top-1 accuracy with half of the
of parameters than ours. Similarly, parameters of ResNet-101 101 and ResNet-152 showing the superiority of learning both local and long- range features from specific structures with corresponding intrinsic IBs by design. Similar phenomena can also be observed when comparing ViTAE-T with MobileNetV1 [1] and MobileNetV2 [65], where VITAE obtains better performance with fewer parameters. When compared with larger models which are searched according to NAS 773, our ViTAE-S achieves a similar performance when using 384 x 384 images input, which further shows the potential of vision transformers with intrinsic IB.
In addition, among the transformers with learned IB, ViT is the first pure transformer model for visual recognition. DeiT shares the same structure with VIT but uses different data augmentation and training strategies to facilitate the learning of transformers. Adenotts using an off-the- off-the-shelf CNN model the teacher model to train DeiT, which introduces the intrinsic IB from CNN transformer implicitly in knowledge distillation manner, showing better performance than the vanilla ViT on the ImageNet dataset. It is exciting to see that our ViTAE-T with fewer parameters even outperforms the distilled model De demonstrating the efficacy of introducing intrinsic IBs in transformers by design. Besides, compared with other transformers with explicit intrinsic IB, our ViTAE with fewer parameters also achieves comparable or better performance. For instance, VITAE-T achieves comparable performance with LocalVit-T but has IM fewer parameters, demonstrating the superiority of the proposed RCs and NCs in introducing intrinsic IBs.
4.3 Ablation study
We use T2T-ViT 933 as our baseline model in the following ablation study of our VITAE, As shown in Table 3, we investigate the hyper-parameter settings in RCs and NCs by isolating them separately. All the models are trained for 100 epochs on ImageNet and follow the same training setting and data augmentation strategy as described in Section

---

Table 2: Comparison of VITAR and SOTA methods on the ImageNet validation set.
Params
MACs
Input Size
ImageNet
Real
Type
Model
(M)
(G)
Top-1
Top-5
Top-1
ResNet-18 26
11.7
3.6
224
70.3
86.7
77.3
ResNet-50
26
25.6
7.6
224
76.7
93.3
82.5
ResNet-101
26
44.5
15.2
224
78.3
94.1
83.7
ResNet-152 26
60.2
22.6
224
78.9
94.4
84.1
EfficientNNtt
76
5.3
0.8
224
77.1
93.3
83.5
CNN
EfficiemmNNc:-B444
19.3
8.4
380
82.9
96.4
88.0
MobileNetV1
4.3
224
72.3
MobileNatV2((.44) 655
6.9
0.6
224
74.7
RegNetY-600M 62
6.1
1.2
224
75.5
RegNetY-44FF 62
20.6
8.0
224
80.0
86.4
RegNetY-8GF
39,2
16.0
224
81.7
DeiT-T 76
5.7
2.6
224
72,2
91.1
80.6
DeiT-TM 76
5.7
2.6
224
74.5
91.9
82.1
LocalViT-T 4k
5.9
2.6
224
74.8
92.6
LocaIViT-T2T
4.3
2.4
224
72,5
ConT-Ti 89
5.8
1.6
224
74.9
PiT-Ti 22
4.9
1.4
224
73.0
T2T-ViT-7 93
4.3
1.2
224
71.7
90.9
79.7
VITAE-T
4.8
1.5
224
75.3
92.7
82.9
VITAE-T 384
4.8
5.7
384
77.2
93.8
84.4
CeiT-T 922
6.4
224
76.4
93.4
83.6
ConViT-Ti
15
6.0
2.0
224
73.1
CrossViT-Ti
6
6.9
3.2
224
73.4
VITAE-6M
6.5
2.0
224
77.9
94.1
84.9
PVT-T 80
13.2
3.8
224
75.1
LocalViT-PVT
43
13.5
9.6
224
78.2
94.2
ConViT-Ti+
10.0
4.0
224
76.7
PÝT-XS
10.6
2.8
224
78.1
ConTIM 89
19.2
6.2
224
80.2
ViTAE-13M
13.2
3.4
224
81.0
95.4
86.8
DeiT-S 776
22.1
9.8
224
79.9
95.0
85.7
DeiT-S 76
22.1
9.8
224
81.2
95.4
86.8
PVT-S 80
24.5
7.6
79.8
Conformer-1
58
23.5
5.2
224
81.3
Swin-T 47
29.0
9.0
224
81.3
CeiT-S 92
24.2
9.0
224
82.0
95.9
87.3
Transformer
CvT-13
20.0
224
81.6
86.7
ConViT-S
15
27.0
10.8
224
81.3
Cio-s
26.7
11.2
224
81.0
PYT-S 20
23.5
4.8
224
80.9
TNT-S 23
23.8
10.4
224
81.3
95.6
Twins-PCPVT-S/IO
24.1
7.4
224
81.2
Twins-SVT-S 100
24.0
5.6
224
81.7
T2T-ViT-14 98]
21.5
5.2
224
81.5
95.7
86.8
VITAE-S
23.6
5.6
224
82.0
95.9
87.0
VITAE-S 384
23.6
20.2
384
83.0
96.2
87.5

---

We use v and X to denote whether or not the corresponding module is enabled during the experiments. If all columns under the RC and NC are marked x shown in the first row, the model becomes the standard TiT-ViT model. "Pre" indicates the output features of PCM and MHSA are fused before FFN while "Post" indicates late fusion strategy correspondingly. "BN" indicates whether PCM uses BN after the convolutional layer or not. "x3" in the first column denotes that the dilation rate set is the same in the three RCs, "[1, 2, 4] 1" denotes using lower dilation rates in deeper RCs, i.e.,
S [1, 2, 3, 4]. S2 [1, 2, 3). S3 [1.
As can be seen, using a pre-fusion strategy and BN achieves the best 69.9% Top-1 accuracy among other settings. It is noteworthy that all the variants of NC outperform the vanilla T2T-ViT, implying the effectiveness of PCM, which introduces the intrin- sic locality IB in transformers. It can also be observed that BN plays an im- portant role in improving the model's performance as it can help to allevi- ate the scale deviation between convo- lution's 's and attention's 's features. For the RC, we first investigate the impact of using different dilation rates in the PRM, shown in the first column. As can be seen, using larger dilation rates (e.g., 4 or 5) does not deliver better performance. We suspect that larger dilation rates may lead to plain features in the deeper RCs due to the smaller resolution of feature maps. To validate the hypothesis, we use smaller dilation rates in deeper RCs as denoted by 2,, 3, 4. As
Table 3: Ablation Study of RCs and NCs in our VITAE. "Pre" indicates the output features of PCM and MHSA are fused before FFN while "Post" indicates late fusion strategy correspondingly. "BN" indicates whether PCM uses BN or not. "[1, 2, 3, 4] 1"' denotes using smaller dilation rates in deeper RCs, i.e., S1 = [1, 2,3, 3, 4), S2 = 21 S3 [1 2].
Reduction Cell
Normal Cell
Top-1
Dilation (Si ~ Sa))
PCM
Pre
Post
BN
X
68.7
x
X
V
X
x
69.1
X
x
X
V
x
69.0
X
V
68.8
V
69.9
[1,2] x 3
X
X
X
69.5
[1,2.3
x
X
X
X
X
69.9
1,
4 x
69.2
[1,
11,2,3,4,
x3
68,9
1,
X
X
69.8
1,
X
X
71.7
1,2,3,4
72.6
can be seen, achieves comparable performance [1, 2, 3] x. However, compared with [1, 3, 4] L. [1, 2, 3] x increases the amount of parameters from 4.35M to 4.6M. Therefore, we select [1. 2, 3, 4| as the default setting. In addition, after using PCM in the RC, it introduces the intrinsic locality IB, and the performance increases to 71.7% Top-1 accuracy. Finally, the combination of RCs and NCs achieves the best accuracy at 72.6%, demonstrating the complementarity between our RCs and NCs.
4.4 Data efficiency and training efficiency
To validate the effectiveness of the introduced intrinsic IBs in improving data efficiency and training efficiency, we compare our ViTAE with T2T-ViT at different training settings: (a) training them using 20%, 60%, and 100% ImageNet training set for equivalent 100 epochs on the full ImageNet training set, e.g., we employ times epochs when using 20% data for training compared with using 100% data; and (b) training them using the full ImageNet training set for 100, 200, and 300 epochs respectively. The results are shown in Figure [ As can be seen, VYTAE consistently outperforms the T2T-ViT baseline by large margin in terms of both data efficiency and training efficiency. For example, VITAE using only 20% training data achieves comparable performance with T2T-ViT using all data. When 60% training data are used, ViTAE significantly outperforms T2T-ViT using all data by about an absolute 3% accuracy. It is also noteworthy that ViTAE trained for only 100 epochs has outperformed T2T-ViT trained for 300 epochs. After training VITAE for 300 epochs, its performance is significantly boosted to 75.3% Top-1 accuracy. With the proposed RCs and NCs, the transformer layers in our VÎTAE only need to focus on modeling aangrangee dependencies, leaving the locality and multi-scale context modeling to its convolution counterparts, ie.. PCM and PRM. Such "divide-amd aeriide-nd--oon and--andddooe conquer"" strategy facilitates the training of vision transformers, making it possible to learn more efficiently with less training data and fewer training epochs.
To further validate the data efficiency of ViTAE model, we train the VITAE model from scratch on the smaller datasets, i.e., Cifar and C0far100. The results are summarized in Table It can be viewed that with only 1/7 number of epochs, the ViTAE-T model achieves better classification
8

---

performance on Cifar10 dataset, with far fewer parameters (4.8M v.s. 86M), which further confirms ViTAE model's data efficiency.
Table 4: Results of training from scratch on Cifar
Model
Params (M)
Top-1 Acc
Epochs
Dataset
DeiT-B
86.0
97.5
7000
Cifar10
V
4.8
97.7
1000
Cifar10
VITAE-T
4.8
85.0
1000
Cifar100
4.5
Generalization on downstream tasks
Table 5: Generalization of ViTAE and SOTA methods different downstream tasks.
Model
Params (M)
Cifarlo
Cifar100
iNat19
Cars
Flowers
Pets
Grafit ResNet-50 [78
25.6
75.9
92.5
98.2
EfficientNet-B3 BB 77
30
98.1
91.1
98.5
ViT-B/16 19
86.5
98.1
87.1
89.5
93.8
VIT-L/16 19
304.3
97.9
86.4
89.7
93.6
DeïT-B 76
86.6
99.1
77.7
92.1
98.4
T2T-ViT-14 993
21.5
98.3
88.4
VITAE-T
4.8
97.3
86.0
73.3
89.5
97.5
92.6
VITAE-S
23.6
98.8
90.8
76.0
91.4
97.8
94.2
We further investigate the generalization of the proposed VITAE models on downstream tasks by fine- tuning them on the training sets of several fine-graiie fineggrained classification tasks including Flowers [33
Cars [36]. Pets [56] and INtturalist99, We also fine-tune the proposed ViTAE models Cifar10 37
and Cifar100 37 The results are shown in Table
can be seen that ViTAE achieves SOTA
performance on most of the datasets using comparable or fewer parameters. These results demonstrate that the good generalization ability of our VITAE.
4.6 Visual inspection VITAE
To further analyze the property of our VITAE, we first calculate the average attention distance of each layer in ViTAE-T and the baseline T2T-ViT-7 on the ImageNet test set, respectively. The results are shown in Figure 3 It can be observed that with the usage of PCM. which focuses modeling locality, the transformer layers in the proposed NCs can better focus on modeling long-range dependencies, especially in shallow layers. In the deep layers, the average attention distances of ViTAE-T and T2T-ViT-7 are almost the same since modeling long-range dependencies much more important. These results confirm the effectiveness of the adopted "divide-and-connuee" idea in the proposed ViTAE, i.e., introducing the intrinsic locality IB from con- volutions into vision transformers makes it possible that transformer layers only need to be responsible long-range dependencies, since locality can be well modeled by convolutions in PCM.
T2F-VIT
VITAE
Attention Distance (pkoal)
100
80
6
Besides, we apply Grad-CAM (66) on the MHSA's output in the last NC qualitatively inspect ViTAE. The visualization results are provided Figure Compared with the baseline T2T-VIT, our VITAE covers the single or multiple targets in the images more precisely and attends less to the background. Moreover, VITAE better handle the scale variance issue as shown Figure (b)). Namely, it can precisely cover the birds no matter they are in small, middle, large size. Such observations demonstrate that introducing the intrinsic IBs of locality and scale-invariance from convolutions to transformers helps ViTAE learn more discriminate features than the pure transformers.
Figure 3: The average per-layer attention dis- tance of T2T VTT-7 and our ViTAE-T.
'The performance of VITAE on dense prediction tasks such as detection, segmentation, pose estimation, can be found in the supplementary material.

---

Imput
T2I-VIT
VITAE
(8)
(b)
Figure 4: Visual inspection of T2T-ViT and ViTAE using Grad-CAM [66]. (a) Images containing multiple or single objects and the heatmaps. (b) Images containing the same class of objects at different scales and the heatmaps (Best viewed in color).
5 Limitation and discussion
In this paper, we explore two types of IBs and incorporate them into transformers through the proposed reduction and normal cells. With the collaboration these two cells, our ViTAE model achieves impressive performance on the ImageNet with fast convergence and high data efficiency, Nevertheless, due to computational resource constraints, we not scaled the ViTAE model and train it on large- size dataset, e.g., ImageNet-2IK (38) and JFT-300M [30]- Although remains unclear by now, are optimistic about its scale property from the following preliminary evidence. As illustrated in Figure our ViTAE model can be viewed an intra-cell ensemble of complementary transformer layers and convolution layers owing to the skip connection and parallel structure. According to the attention distance analysis shown Figure 3 the ensemble nature enables the transformer layers and convolution layers to focus on what they are good at, i.e., modeling long-rang dependencies and locality. Therefore, VITAE is very likely learn better feature representation from large- acage data. Besides, we only study two typical IBs in this paper. More kinds of IBs such as constituting viewpoint invariance [64] can be explored in the future study.
Conclusion
In this paper, we re-design the transformer block by proposing two basic cells (reduction cells and normal cells) incorporate two types intrinsic inductive bias (IB) into transformers, i.e., locality and scale-invariance, resulting in simple yet effective vision transformer architecture named VITAE. Extensive experiments show that ViTAE outperforms representative vision transformers in various respects including classification accuracy, data efficiency, training efficiency, and generalization ability on downstream tasks. We plan to scale ViTAE to the large or huge model size and train on large-size datasets in the future study. In addition, other kinds of IBs will also be investigated. We hope that this study will provide valuable insights to the following studies introducing intrinsic into vision transformers and understanding the impact of intrinsic and learned IBs.
Acknowledgement Dr. Jing Zhang is supported by the ARC project FL- 17010700
10

---

References
[ E. H. Adelson, C. H. Anderson, J. R. Bergen, P.J. Burt, and J. M. Ogden. Pyramid methods in image processing. RCA engineer, 29.3333-41, 1984,
[2] J.L. L. Ba, J. R. Kiros, and G. E. Hinton. Layer normalization. arXiv preprint arXiv: (6450004500 2016.
[3] H. Bay, T. Tuytelaars, and L. Van Gool. Surf: Speeded up robust features. In European conference on computer vision, pages 404 417. Springer, 2006.
[4] P. J. Burt and F. H. Adelson. The laplacian pyramid a compact image code. In Readings in computer
vision, pages 671-679. Elsevier, 1987.
[5] Z. Cai and N. Vasconcelos. Cascade r-cnn: Delving into high quality object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6154 6162, 2018.
[6] C.-F. Chen, Q. Fan, and R. Panda. Crossvit: Cross-attention multi-scale vision transformer for image classification, arXiv preprint arXiv:2103.
2021.
[7] H. Chen, Y. Wang, T. Guo, C. Xu, Y. Deng. Z. Liu, S. Ma, C. Xu, C. Xu, and W. Gao. Pre-trained image processing transformer. arXiv preprint arXiv:2012.00 2.00444444 2020.
[8] L.-C. Chen, G. Papandreou. F. Schroff, and H. Adam. Rethinking atrous convolution for semantic image segmentation, arXiv preprint
(5Gv:/770.05588 2017.
[9] Chen, S. Xie, and K. He. An empinical study of training self-supervised vision transformers. arXiv
preprint arXiv: 222.000.0077 2021.
[10] X. Chu, Z. Tian, Y. Wang. B. Zhang, H. Ren, X. Wei, H. Xia, and C. Shen. Twins: Revisiting spatial
attention design in vision transformers. arXiv preprint
(tt::2004.3388 2021.
[11] X. Chu, Z. Tian, B. Zhang, X. Wang, X. Wei, H. Xia, and C. Shen. Conditional positional encodings for vision transformers. arXiv preprint arXiv:2102. 10882, 2021.
[12] M. Contributors, MMSegmentation: Openmmlab semantic segmentation toolbox and benchmark. https
www/ //github. com open- www.mmmaambbbaaee mmmemmtttttn 2020.
[13] M. Contributors. Openmmlab pose estimation toolbox and benchmark.
htttp::/ggithb...
spen-mmlab/mee mmpose 2020.
[14] Z. Dai, Z. Yang, Y. Yang. J. Carbonell, Q V. Lc, and R. Salakhutdinov. Transformer-xl: Attentive
language models beyond fixed-length context. arXiv preprint arXiv:1901.02866 2019.
[15] S. d' Ascoli H. Touvron, M. Leavitt, Morcos, G. Biroli, and L.. Sagun. Convit: Improving vision
transformers with soft convolutional inductive biases. arXiv preprint ar3iv:2103.
2021.
[16] H. Demirel and G. Anbarjafari. Image resolution enhancement by using discrete and stationary wavelet
decomposition. JEEE transactions on image processing, 20(5):1458-14
2010.
[17] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers
for language understanding. arXiv preprint arXiv:/810.04800 2018.
[18] X. Ding. X. Zhang. J. Han, and G. Ding. Repmlp: Re-parameterizing convolutions into fully-comnected layers for image recogmition. arXiv preprint 105. 105.01883, 2021.
[19] A. Dosovitskiy, L. Beyer. A. Kolesnikov, D. Weissenborn, X. Zhai. T. Unterthiner, M. Dehghani,
M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transformers for image
recognition scale. arXiv preprint arXiv:2010. 11929, 2020.
[20] A. El-Nouby, H. Touvron, M. Caron, P. Bojanowski, M. Douze, A. Joulin, I. Laptev, N. Neverova, G. Syn- naeve, J. Verbeek, al. Xcit: Cross-covariance image transformers. arXiv preprint 17X0622/06.09681,
2021.
[21] B. Graham, A. El-Nouby. H. Touvron, P. Stock, A. Joulin. H. Jégou, and M. Douze, Levit: a vision
transformer in convnet's 's clothing for faster inference. arXiv preprint arXiv:2 136, 2021.
[22] M.-HI Guo, Z.-N. Liu, T.-J. Mu, and S.-M. Hu. Beyond self-attention: External attention using two linear layers for visual tasks. arXiv preprint arrXvv2105..0333 2021.
[23] K. Han, A. Xiao, E. Wu, J. Guo, C. Xu, and Y. Wang. Transformer in transformer. arXiv preprint
1rXiv:2103.00122 2021.
[24] K. He, G. Gkioxari, P. Dollár, and R. Girshick. Mask r-cnn. In Proceedings of the IEEE international
conference on computer vision. pages 2961-2969, 2017.
[25]
K. Hc, X. Zhang, S. Ren, and J. Sun. Spatial pyramid pooling in deep convolutional networks for visual recognition. IEEE transactions on pattern analysis and machine intelligence, 1799)1994-1116 2015. K. He, X. Zhang. S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[26]
[27]
I.. Hc, Y. Dong, Y. Wang. D. Tao, and Z. Lin. Gauge equivariant transformer. In Thirty-Fifth Conference
on Neural Information Processing Systems, 2021.
[28]
B. Heo. S. Yun, D. Han. S. Chun, J. Choe, and S. J. Oh. Rethinking spatial dimensions of vision
transformers. In International Conference on Computer Vision (ICCV), 2021.
[29] B. Heo, S. Yun, D. Han, S. Chun, J. Choe, and S. J. Oh. Rethinking spatial dimensions of vision
transformers. arXiv preprint 2 16302, 2021.
[30] Hinton, O. Vinyals.
J. Dean. Distilling the knowledge in neural network. In NIPS Deep Learning
and Representation Learning Workshop, 2015.
[31] A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang. T. Weyand, M. Andrectto, and H. Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint
(X44:1704.0466. 2017.
[32] G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger. Densely connected convolutional networks.
In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700
11

---

2017.
[33] Z. Huang. Y. Ben, G. Luo, P. Cheng, G. Yu, and B. Fu, Shuffic transformer: Rethinking spatial shuffle for vision transformer. arXiv preprint wxXiv:2106.055
2021.
[344] S. Toffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal
covariate shift. In International conference machine learning, pages 448 448-456. PMLR, 2015.
[35] Y. Ke and R. Sukthankar. Pca-sift: A more distinctive representation for local image descriptors. In Proceedings of the IEEE conference on computer vision and pattern recognition, volume 2. pages II-II. IEEE, 2004,
[36] J. Krause, M. Stark, J. Deng, and L. Fei-Fei. object representations for fine- fine-grained categorization. In
4th International IEEE Workshop on 3D Representation and Recognition (3dRR-13) 13), Sydney, Australia, 2013.
[371 A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[38] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25:1 10:1- 105, 2012.
[39] W.S. J.-B. Huang. N. Ahuja, and M. Yang. Deep laplacian pyramid networks for faas and accurate super-resolution. In Proceedings of the IEEE conference on computer vision and pattern recognation, pages 624-632, 2017.
[40] 7. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, and Soricut. Albert: A lite bert for self-supervised
learning of language representations. arXiv preprint arXiv: /909// 1009./1942, 1942, 2019.
[41] Y. LeCun, Y. Bengio, et al. Convolutional networks for images, speech, and time series. The handbook of
brain theory and neural networks, 3361(10):1955, 1995.
[42] LeCun, Y. Bengio. and G. Hinton. Deep learning. nature, 521((553):436 444, 2015,
[43] Y. Li, K. Zhang, J. Cao, R. Timafte, and I.. Van Gool. Localvit: Bringing locality to vision transformers.
arXiv preprint arXiv:2104.05700 2021.
[44] G. Lin, C. Shen, A. Van Den Hengel, and Reid, Efficient piecewise training of deep structured models for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3194-3203, 2016.
[45] T.YY. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition,
pages 2117-2125, 2017.
[46] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stooynow. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 2019.
[47] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo. Swin transformer: Hicrarchical
vision transformer using shifted windows. arXiv preprint arXiv:2103. 14030, 2021.
[48] I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2018
[49] W. Luo, Y. Li, R. Urtasun, and R. S. Zemel. Understanding the effective receptive field deep convolutional neural networks. In Proceedings of the 30th International Conference on Neural Information Processing Systems, volume 29, pages 0999-4906,75 2016.
[50] L. Melas-Kyriazi. Do you even need attention? stack of feed-forward layers does surprisingly well on imagenet. arXiv: Computer Vision and Pattern Recognition, 2021.
[51]
H. Nam, J.-W. Ha, and J. Kim. Dual attention networks for multimodal reasoning and matching. In Proceedings of the IEF.F. conference computer vision and pattern recognition, pages 299-307, 2017. [52] P. C. Ng and S. Henikoff. Sift: Predicting amino acid changes that affect protein function. Nucleic acids research, 31(13):3812 122)3312-2201 3814, 2003.
[53] M.-E. Nilsback and A. Zisserman. Automated flower classification over a large number of classes. In
Indian Conference on Computer Vision, Graphics and Image Processing, Dec 2008.
[54] S. W. J.Y Lee, N. Xu, and S. J. Kim. Video object segmentation using space-time memory networks. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9226-9235, 2019. [55] H. Olkkonen and P. Pesola. Gaussian pyramid wavelet transform for multiresolution analysis of images.
Graphical Models and Image Processing, 58(4):394-39 1996.
[56] O. M. Parkhi, A. Vedaldi, A. Zisserman, and C. V. Jawahar. Cats and dogs. In IEEE Conference on
Computer Vision and Puttern Recognition, 2012.
[57] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein,
L. Antiga, A. Desmaison, A. Kopf, E. Yang. Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner,
L. Fang, J. Bai, and S. Chintala. Pytorch: An imperative style. high- seieh--efformacce deep learning library. In
Advances in Neural Information Processing Systems, volume 32, pages 8026-8037, 2019.
[58] Z. Peng, W. Huang, S. Gu, L. Xic, Y. Wang. J. Jiao, and Q Ye. Conformer: Local features coupling global representations for visual recognition. arXiv preprint arXiv:2105 10000.....,.... 2021.
[59] F. Perazzi, J. Pont-Tuset, B. McWiliiass, L. Van Gool, M. Gross, and A. Sorkine- Horkkngg A benchmark dataset and evaluation methodology for video object segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 724-732, 2016.
[60] J. Pont-Tuset, F. Perazzi, Caelles, Arbeláez, A. Horkingg and L. Van Gool. The 2017 davis challenge on video object segmentation. arXiv preprint 1rxiiv1704.0000 2017.
[61] A. Radford, J. Wu, R. Child, D. Luan, D. Amodci, and I. Sutskever. Language models are unsupervised multitask learners. OpenAl blog, 1(8);9 2019.
[62] I. Radosavovic, R. P. Kosaraju, R. Girshick, K. He, and P. Dollár. Designing network design spaces. In
P Proceedings of the IEEE conference on computer vision and pattern recognition, pages
12

---

2020.
[63] E. Rublee, V. Rabaud, K. Konolige, and G. Bradski. Orb: An efficient alternative to sift or surf. In Proceedings of the IEEE international conference on computer vision, pages 2564-2571 2571. Iece, 2011. [64] S. Sabour, N. Frosst. and G. E. Hinton. Dynamic routing between capsules. arXiv preprint
arXiv:1710.09829, 2017.
[65] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition,
pages 4510-4520, 2018.
[66] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra. Grad-cam: Visual explanations from deep networks via gradient-based based localization. In Proceedings of the IEEE international conference on computer vision, pages 618 626, 2017.
[67] P. Shaw, J. Uszkoreit, and A. Vaswani. Self-attention with relative position representations. arXiv preprint
AXXv:1803.02150 2018.
[68] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint 1409.
2014,
[69] A. Srinivas, T.-Y. Lin, N. Parmar, J. Shlens, P. Abbeel, and A. Vaswani. Bottleneck transformers for visual recognition. arXiv preprint arXiv:2101 1605, 2021.
[70] C. Szegedy, S. Toffe, V. Vanhoucke, and A. Alemi. Inception-v4, inception-resnet and the impact of residual connections on learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.
[71] C. Szegedy. W. Liu, Jia, Sermanet, S. Reed, Anguelov, D. Erhan, Vanhoucke, and A. Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pottern recognition, pages 1 1-9, 2015.
[72] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and 7. Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEF conference on computer vision and pattern recognàtion, pages 2818-2826. 2016.
[73] M. Tan and Q Lc. Efficientnet: Rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning, pages 6105-61 6114, PMLR, 2019.
[74] I. Tolstikhin, N. Houlsby, A. Kolesnikov, L. Beyer, X. Zhai, T. Unterthiner, J. Yung, D. Keysers,
J. Uszkoreit, M. Lucie, and A. Dosovitskiy. Mip-mixer: An all-mip architecture for vision. arXiv preprint
wiXv:2105.01601 2021.
[75] H. Touvron, P. Bojanowski, M. Caron, M. Cord, A. El-Nouby, F. Grave, A. Joulin, G. Synnaeve,
J. Verbeck, and H. Jégou. Resmlp: Feedforward networks for image classification with data-efficient training. arXiv preprint arXiv:2105.0344
2021.
[76] H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jégou. Training data-efficient image transformers & distillation through attention. arXiv preprint ar1iv:2012. 128777 2020.
[77] H. Touvron, M. Cord, A. Sablayrolles, G. Synnaeve, and H. Jégou. Going deeper with image transformers.
arXiv preprint 20r::vv::3.... 17239, 2021.
[78] H. Touvron, A. Sablayrolles. M. Douze, M. Cord, and H. Jégou. Grafit: Learning fine- image
representations with coarse labels. arXiv preprint arXiv:201 20.:2.0111222, 2020.
[79] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. Proceedings of the 31st International Conference on Neural Information
Processing Systems, volume 30, pages 5998 6008, 2017.
[80] W. Wang, F.. Xic, X. Li, D.-P. Fan, K. Song. D. Liang, T. Lu, Luo, and I.. Shao. Pyramid vision trans- former: A versatile hackbone for dense prediction without convolutions. arXiv preprint arXiv:2 122, 2021.
[81] X. Wang. R. Girshick, A. Gupta, and K. He. Non-local neural networks. In Proceedings of the IEEE
conference on computer vision and pattern recognition, pages 7794-7803 2018
[82] R. Wightman. Pytorch image models. httpp://
com/ www.w/wwwwwa.pttte.eee.eee pyrtorch-imagg-mmodde
2019.
[83] H. Wu, B. Xiao, N. Codella, M. Liu, X. Dai, L. Yuan, and L. Zhang. Cvt: Introducing convolutions vision transformers. arXiv preprint arXiv:2/03. 15808, 2021.
[84] B. Xiao, H. Wu, and Y. Wei. Simple baselines for human pose estimation and tracking. In Proceedings of the European Conference on Computer Vision (ECCV), September 2018
[85] Xiao, Y. Liu, B. Zhou, Y. Jiang, and Sun. Unified perceptual parsing for scene understanding. In Proceedings of the European Conference on Computer Vision (ECCV), pages 418-44 434, 2018 [86] J. Xic, R. Zeng, Q. Wang. 7. Zhou, and P. Li. So-vit: Mind visual tokens for vision transformer. arXiv preprint arXiv:2104. 10935, 2021.
[87] S. Xic, R. Girshick, Dollár, Z. Tu, and K. Hc. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1492-1500. 2017.
[88] W. Xu, Y. Xu, T. Chang, and Z. Tu. Co-scale conv-attentional image transformers. In Proceedings of the IEEE/CVF Insernational Conference on Computer Vision (ICCV), pages 9981 19990. October 2021. [89] H. Yan, Z. Li, W. Li, C. Wang, M. Wu, and C. Zhang. Contnet: Why not use convolution and transformer
the same time? arXiv preprint arXiv: 104. 13497, 2021.
[90] F. Yu and V. Koltun. Multi-scale context aggregation by dilated convolutions. In ICLR 2016: International
Conference on Learning Representations 2016, 2016.
13

---

[91] F. Yu, V. Koltun, and T. Funkhouser. Dilated residual networks. In Proceedings of the IEEE conference computer vision and paltem recognition, pages 472-480, 2017.
[92] K. Yuan, S. Guo, Z. Liu, A. Zhou, F. Yu, and W. Wu. Incorporating convolution designs into visual
transformers. arXiv preprint arXiv:2103. 1816, 2021.
[93] L. Yuan, Y. Chen, T. Wang. W. Yu. Y. Shi, Z. Jiang. F. E. Tay, J. Feng, and Yan. Tokens-to-token vit:
Training vision transformers from scratch on imagenet. arXiv preprint arXiv:2101.1 10t::200111198, 2021.
[94] M. D. Zeiler and Fergus. Visualizing and understanding convolutional networks. European conference on computer vision, pages 818-833, 833. Springer, 2014.
[95] X. Zhang, X. Zhou, M. Lin, and J. Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6848 6856, 2018
[96] H. Zhao, J. Shi, X. Qi, X. Wang, and J. Jia. Pyramid scene parsing network. In Proceedings of the IEEF computer vision and pultern recognition, pages 2881 2881-2890, 2017.
conference
[97] S. Zhhnng, J. Lu, H. Zhao, X. Zhu, Z. Luo, Y. Wang. Y. Fu, J. Feng, T. Xiang, P. H. S. Torr, and L. Zhang. Rethinking semantic segmentation from
to-nueeee---ss perspective with transformers. arXiv
preprint arXiv:2012. 15840. 2020.
[98] B. Zhou, H. Zhao, X. Puig. S. Fidler, A. Barriuso, and A. Torralba. Scene parsing through ade20k dataset. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 633-641, 2017. [99] Zhou, H. Zhao, X. Puig, T. Xiao, S. Fidler, Barriuso, and Torralba. Semantic understanding of scenes through the adc20k dataset. International Journal of Computer Vision, 127(3)302--321, 2019.
[100] Zhu, Su, L. Lu, Li, X. Wang. and J. Dai. Deformable detr: Deformable transformers for end cod object detection. In International Conference on Learning Representations, 2021.
14


Reviews:
[{"review_id": "t6BjHxU1huP", "reviewer": "Reviewer_dFPw", "summary": "The paper propose combining convolution layers, which capture relevant intrinsic biases of images, with attention layers, which can can capture long-distance relations, to learn better models for image classification.\n\nAdditionally, the paper analyze different ways to combine lessons from CNNs (residual layers, BN, multi-scale, spatial reduction) with the latest advancements in visual transformers to obtain more powerful models, that have lower number of parameters but can compete with state-of-the-art models.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Originality: \nThe paper proposes a novel combination of convolution networks with transformer networks, that seems to get the best of both worlds. Implicit locality, multi-scale, and parameter sharing of CNNs with the long-range learnable relations of T2Ts.\n\nThis work builds of previous works, but finds a novel architecture that performs better and requires much less parameters than alternative methods.\n\nI would recommend taking a look at \"Early Convolutions Help Transformers See Better\" concurrent work which also combine convolution with transformers and obtain very good results with stable training.\n\n# Quality:\nOverall, the paper is technically sound, the experiments, comparisons and ablation studies increase the confidence that the proposed architecture provides significant improvements, and disentangle the different contributions of the design choices.\n\nA comparison with a network that replaces the Reduction part of the network with just multiple convolutions that reduce the input size in a similar fashion would helpful to understand the need of attention layers in the reduction cells.\n\n\n# Clarity: \nThe paper is clear and well organized, there are some small typos (see minor comments below), which allow the reader to get good understanding of the proposed architecture and enough information to reproduce the results.\n\nIt's unclear what real Top-1 accuracy means.\n\n# Significance: \nThe specific way of mixing CNNs and transformers seems to provide substantial value while at the same time reducing the number of parameters and maintaining the computational requirements.\n\nThe papers provided improvements on ImageNet results compared with a broad set of state-of-art methods.\n\nIf the improvements in ImageNet classification task could be translated to COCO detection task, where the multi-scale and locality properties are critical, it would significantly increase the significance of the paper.\n\n\n# Minor:\nLine 288 \"RCs and RCs\" -> \"RCs and NCs\"\n\nIn appendix Figure S2 caption there are two (c).\n\n# After further analysis and extra experiments provided by the authors updated score to 8. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paragraph about Broad impacts in section 6 is a just a generic statement, and it doesn't reflect on the possible negative consequences of having more precise visual models.", "review_text": "", "extra_scores": {"ethical_concerns": "As visual models get more and more accurate the possibilities of misuse grows, so mitigation strategies and careful design of experiments and applications are needed.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "dYB9Ugfg6o2", "reviewer": "Reviewer_D6zB", "summary": "This work incorporates two architecture changes which induce the inductive biases of 1) locality and 2) scale-invariance. It does so though a sequence of Reduction Cells followed by Normal Cells. In doing so, the authors demonstrate comparable performance to strong CNN baselines and improvements over Transformer-based ImageNet models -- the top performing ViTAE model achieves a 83.0% Top-1 accuracy. ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Notes:\n* Writing nit: if you’re proposing a method, you don’t need to also say it’s novel.\n* Writing nit: odd opening sentence (“domination trend” and an emphasis that Transformers perform well in NLP due to an ability to model long-range dependencies, not really a focus in this work).\n* Thanks for presenting the Ablation Study in Table 3. The architecture details were complicated so this helps the reader attribute significance to different pieces.\n\n\nQuestions:\n* ResNet-RS (Bello et al., 2021) notes that FLOPs can often be a poor predictor of step-time on modern accelerators due to how well certain operations are supported. Further, parameters can be a bad proxy for memory usage. Can the authors comment on the wall-clock speed of ViTAE versus other baselines? Also, Table 2 might include the ResNet-RS variants which are Pareto optimal and perform better than EfficientNet.\n* How do the authors envision the future of vision architecture design and how would they motivate this to a vision researcher? If larger-scale data is available, low-inductive bias models like ViT have shown great promise, but if the data is more limited, high-inductive bias models like ResNets/EfficientNets continue to produce Pareto-optimal models. The context of how this contribution fits in at a high level is what prevented a higher initial score. Could the authors comment?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "VCNB12_lhMH", "reviewer": "Reviewer_eo3B", "summary": "The standard vision transformer model lacks the capability to model inductive biases desirable for images e.g., local features encoding, and scale invariance. It, therefore, requires large-scale training data (300 million images), or other variants (e.g., Data Efficient ViT) distilling Knowledge from a pertained CNN. The paper proposes an architecture, which tries to take care of these inductive biases. The architecture comprises of 3 “Reduction cells” at the beginning followed by multiple “Normal Cells”. A reduction cell gets a pyramid effect using dilated convolutions at the beginning which gradually downsamples features. The normal cell doesn’t have any dilated convolutions. Each block has Convolution and attention running in parallel. \nThe assumption here is that convolution captures local relationships, pyramid structure (via diluted convolutions) captures scale invariance, and attention captures global relationships.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n- A comparison of the proposed model is presented with CNNs and recent ViT variants. The proposed model shows promising results, which are comparable with other competing methods. \n\n- Local features are captured using depth-wise convolutions (e.g., https://arxiv.org/abs/2104.06399), or local attention within a patch (e.g., SwinTransformer). Also, hierarchical feature encoding in a pyramid-like structure is common, and most recent Vision Transformer variants (which I’d categorize as concurrent works) get a spatial feature pyramid by token-merging and projecting the features to a higher dimension, progressively along the hierarchy of the model. These existing works do not nullify the contributions of this paper. The authors just need to clearly state how the existing Vision Transformer variants model scale invariance, and local features; and then justify their proposed approach. In summary, the paper is not the first attempt to introduce inductive biases in Vision Transformers. The paper needs to clearly state that, and change the relevant parts of the paper.\n\n- Capturing the inductive biases (scale invariance+local-features) enables vision transformers to translate well for dense prediction tasks (detection and segmentation). I was expecting empirical evaluations with the proposed architecture used as a backbone for Dense Prediction. For example, these works (which have local-feature-encoding + scale invariance) report results for detection and segmentation.\n\n    - see 4.2 and 4.3 of https://arxiv.org/pdf/2103.14030.pdf\n    - see 5.2 and 5.3 of https://arxiv.org/pdf/2102.12122.pdf\n    - https://arxiv.org/abs/2106.09681\n    - https://arxiv.org/abs/2106.03650\n    - https://arxiv.org/abs/2104.06399\n    - https://arxiv.org/pdf/2105.03889.pdf\n\n- I am not entirely convinced that convolution is needed in the “normal cell”. Diluted convolutions in the beginning, as well as the normal convolution operation at the initial blocks, should suffice to capture local relationships and scale invariance. The convolutions in the later parts move us away from the original transformer blocks and seem unnecessary to me. What happens if the authors replace the \"normal cells\" with standard Transformer blocks. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4.5"}, "extra_text": {}}, {"review_id": "D9ZXna5G7hs", "reviewer": "Reviewer_Zi4o", "summary": "A new vision transformer architecture VITAE, with a better use of intrinsic inductive bias IB via a new architecture using convolutional dillated layers.\nBoth scale and locality IB are enabled by that architecture.\nEmpirical results show that the new method can achieve excellent results with much less parameters than other recent methods.", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, it is a good storyline that using the right types of IB can allow more efficient architectures with less parameters.\nThe paper is very empirical, but seems that the IB does indeed help ViT types of architectures, and this is a good contribution.\nHowever, the paper can benefit from clarity in few points, and also better explanation of the experiments.\n\n- scale invariance and locality IB -- is there an ablation showing how much each contributes to the overall efficiency?\n\n- figure 2 architecture --- reduction cells have self-attention (MHSA) at a relatively large spatial resolution. Does this cost dearly in memory and computational costs -- the ViT only does attention at the short sequence of the final tokens?\n\n- normal cell -- it is not so clear to me how it is different than the usual ViT approach. Are the small changes here really necessary, or the reduction cell + normal ViT would work already very good?\n\n- 3.5 what does \"fair comparison\" mean? Are the author referring to number of parameters, memory cost, computational runtime? Experiments are done for 300 epochs on V100, but does each epoch cost the same resources for the different methods?\n\n- 4.4. data efficiency -- while the message is clear overall, i am puzzled by the discrepancy of Figure 1 and Table 2. I would expect that at 100% data,  ViT and ViTAE are the same scores as in the table 2, but this is not the case,", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "section 5 is a bit unclear. It is mentioned that the method ViTAE cannot do large size datasets yet. Please discuss better the computational bottlenecks, and best discuss this already in the whole Experiments section 4", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work proposes a new method for combining convolution layers with attention in computer vision problems by allowing the attention layers to focus on capturing long range correlation structures. In addition, this work analyzes multiple dimensions for how to combine convolutional networks with visual transformers in order to achieve favorable performance in terms of computer vs performance. All of the reviewers were impressed by the soundness of the experiments, the clarity of the exposition and the significance of the results. For all of these reasons, this paper will be accepted.

Decision: Accept (Poster)