RegionViT: Regional-to-Local Attention for Vision Transformers
Authors: Chun-Fu Chen
Rameswar Panda
Quanfu Fan
Venue: ICLR 2022 Poster (2022)
Keywords: vision transformer
image recognition
multi-scale feature

Abstract: Vision transformer (ViT) has recently shown its strong capability in achieving comparable results to convolutional neural networks (CNNs) on image classification. However, vanilla ViT simply inherits the same architecture from the natural language processing directly, which is often not optimized for vision applications. Motivated by this, in this paper, we propose a new architecture that adopts the pyramid structure and employ novel regional-to-local attention rather than global self-attention in vision transformers. More specifically, our model first generates regional tokens and local tokens from an image with different patch sizes, where each regional token is associated with a set of local tokens based on the spatial location. The regional-to-local attention includes two steps: first, the regional self-attention extracts global information among all regional tokens and then the local self-attention exchanges the information among one regional token and the associated local tokens via self-attention. Therefore, even though local self-attention confines the scope in a local region but it can still receive global information.
Extensive experiments on four vision tasks, including image classification, object and keypoint detection, semantics segmentation and action recognition, show that our approach outperforms or is on par with state-of-the-art ViT variants including many concurrent works. Our source codes and models are available at \url{https://github.com/IBM/RegionViT}.

Published as a conference paper at ICLR 2022
REGIONVILI REGIONAL-TO-LOCAL ATTENTION FOR VISION TRANSFORMERS
Chun-Fu (Richard) Chen, Rameswar Panda, Quanfu Fan MIT-IBM Watson Lab ihenrich@te b.mm com, rpanda i bmm com, qfanêus. ibm.com
ABSTRACT
Vision transformer (ViT) has recently shown its strong capability in achieving comparable results convolutional neural networks (CNNs) on image classifica- tion. However, vanilla ViT simply inherits the same architecture from the natural language processing directly, which is often not optimized for vision applications. Motivated by this, in this paper, we propose new architecture that adopts the pyramid structure and employ novel regional-to-lonn self-attention in vision transformers. More specifically, model first generates regional tokens and local tokens from an image with different patch sizes, where each regional token is associated with set of local tokens based the spatial location. The regional-to-loca attention includes two steps: first, the regional self-attention extracts global information among all regional tokens and then the local self-attention exchanges the information among one regional token and the as- sociated local tokens via self oeff-attention. Therefore, even though local self- att-ntotention confines the scope in local region but it can still receive global information. Extensive experiments on four vision tasks, including image classification, ob- ject and keypoint detection, semantics segmentation and action recognition, show that our approach outperforms or is on par with state-of ofrtrr-rtt VIT variants in- cluding many concurrent works. Our source codes and models are available at https: / ggtttb.thhhuuuggg//g//// com/ (BBB///ei//VV.
attention rather than global
INTRODUCTION
Transformers (Vaswani al., 2017) based on self-attention come naturally with the ability to learn long-range dependencies in sequential data. As important as it is to language modeling (Devlin et al., 2019), such ability is also highly desired for many vision tasks where contextual modeling plays a significant role. For this reason, self-attention and transformers have been receiving increasing attention the vision community (Bello, 2021; Srinivas et al., 2021; Zhao et al., 2020; Ramachandran et al., 2019a; Bello et al., 2019; Hu et al., 2019; Ramachandran et al., 2019b; Wang et al., 201 8). Especially, the recent Vision Transformers (ViT) (Dosovitskiy et al., 2021) demonstrates comparable image classification results against the firmly established and prevalent CNNs in computer vision (He et al., 2016; Tan & Le, 2019; Brock et al., 2021), albeit relying on a huge amount training data. has since then led to an explosion of interest further investigating its potential for a wide variety of vision applications (Wu et al., 2021; Wang et al., 2021; Heo et al., 2021; ; Zhang et al., 202 1a; Li et al., 2021; Graham et al., 2021; Liu et al., 2021; Chu al., 2021a; Yan al., 2021; Chen et al., 202
The ViT inherits the entire architecture from the vanilla transformer (Vaswani et al., 2017), which is designed for natural language processing tasks, and some of those designs thus may not meet the needs of vision tasks. For example, the transformer has an isotropic network structure with fixed number of tokens and unchanged embedding size, which loses the capability to model the context with different scales and allocates computations at different scales. As opposed to this, majority of CNNs adopt a popular pyramid architecture to compute multi-scale features efficiently. Recent vision transformers such as PVT (Wang et al., 2021) and PiT (Heo et al., 2021) also follow a similar pyramid structure as CNNs, showing improvement on both computation and memory efficiency as well as on model accuracy. Another critical bottleneck of the transformer is that the self-attention module has a quadratic cost in memory and computation with regard to the sequence length (i.e., the number of tokens). This issue is worse in ViT images are 2-D, suggesting quadratic relationship

---

Published as a conference paper at ICLR 2022
between the number of tokens and image resolution. As result, ViT indicates quadruple complexity w.I. image resolution. The highly compute- and memory- intensive self self-attention makes it challenging to train vision transformer models at fine-grained patch sizes. It also significantly undermines the applications of these models to tasks such as object detection and semantic segmentation, which benefit from require fine feature information computed from high-resolution images.
To address the aforementioned computational limitations of vision transformers, in this work, we develop memory- friendly and efficient self-attention method for transformer models to reach their promising potential for vision appli- cations. We propose a novel coarse-to-fiee toneine mechanism to compute self-attention a hierarchical way. Specifically, our approach first divides the input image into group of con-overlapping patches of large size (e.g.. 28 228. on which regional tokens are computed via linear projection. Similarly, local tokens are created for each region using standard
10r16
WIT
Id RlggreSS CCuo
Figure 1: for Vision Transformers. (a) VIT uses a fixed patch size through the whole net- work, (h) PVT adopts pyramid structure to gradually enlarge the patch size in the net- work. Both ViT and PVT uses all tokens in self-attention, which are computational- and memory-intensive (c) Our proposed ap- proach combines a pyramid structure with efficient regional-to-locat (R2L) attention mechanism to reduce computation and mem- ory usage. Our approach divides the input image into two groups of tokens, regional to- kens of large patch size (red) and local ones of small patch size (black). The two types of tokens communicate efficiently through R2L attention, which jointly attends to the local tokens in the same region and the associated regional token. Note that the numbers denote the patch sizes at cach stage of model.
Regional-to-Local Attention
smaller patch size (e.g., 4x We then transformer to process regional and local tokens separately. To enable communication between the two types of tokens, we first perform self-attention on regional tokens (regional attention) and then jointly attend to the local tokens of each region including their associated regional token (lo- cal attention). By doing so, regional tokens pass global contextual information to local tokens efficiently while be- ing able to effectively learn from local tokens themselves. For clarity, we represent this two-stage attention mecha- nism as Regional-to-
Attention, or R2L attention for short (see Figure for an illustration). Since both regional and local attention involve much fewer tokens, our R2L attention requires substantially less memory than regular global self-attention used in vision transformers. For ex- ample, in our default setting, the memory saving using R2L attention can be up to as much as 73%. We demonstrate the effectiveness of our approach on image classification and several downstream vision tasks including object detection and action recognition.
To summarize, our key contributions in this work are as follows:
1. We propose a new vision transformer (Region ViT) based on regional-to-loce attention to learn both local and global features. Our proposed regional-to-local attention alleviates the overhead of standard global attention (too many tokens) and the weakness of pure local attention (no interaction between regions) used in existing vision transformers.
2. Our regional-to-loca attention reduces the memory complexity significantly as compared to standard self-attention, leading to a savings in memory complexity by about D(N/M²) where M is the window size of region and N is the total number of tokens. This effectively allows us to train a more deep network for better performance while with comparable complexity.
3. Our models outperform on par with several concurrent works vision transformer that exploit pyramid structure for image classification. Experiments also demonstrate that our models work well on several downstream classification tasks, including object detection and action recognition.
2 RELATED WORK
CNNs with Attention. Attention has been widely used for enhancing the features in CNNs, e.g.. SENet (Hu et al., 2018) and CBAM (Woo et al., 2018). Various methods have been proposed that combine self-attention with CNNs (Bello, 2021; Srinivas et al., 2021; Zhao et al., 2020; Ramachandran et al., 2019a; Bello et al., 2019; Hu et al., 2019; Ramachandran et al., 2019b; Wang et al., 2018). E.g., SAN (Zhao et al., 2020) and SASA (Ramachandran et al., 2019a) attempts to replace all convolutional layers with local self- attenttention network. While these works show promising results, their complexity relatively high due to the self-attention. On the other hand, BoTNet (Srinivas et al., 2021) replaces
2

---

Published as a conference paper at ICLR 2022
Table 1: Comparison to related works. Most works use non-overlapped windows to group tokens, and then propose the corresponding methods to assure the information exchange among regions.
Methods
Structure
Artention Types
Local Attention
Windows of
Regions
Handling Non-ovarlappol
Global Tokens
VT (Dosavitskiy al. 2021)
Isotropic
Global
NIA
NIA
NIA
PVT (Wang et 2021)
Pyramid
Global
NIA
Swin (Liu et al., 2021)
Pyramid
Local
Strictly lon-overlapped
Shifting the window
None
Vil (Zhing at. 2021a)
Pyramid
Local Global
Overlagped
NIA
One leurnable token
Twins (Cho al., 2021a)
Pyramid
Local Global
Non-overlapped
GA
Subsampled from local tokens
RegionViT (Ours)
Pyramid
Local Regional
Non-overlappod
Regional-t--loee attention
Regional tokens
GN global ancesaon. not applicable used overlaped windows.
few convolutional layers with slightly-moiified modified self-attention to balance computation and accuracy. LambdaNet work (Bello, 2021) uses the approximated self-attention to reduce the overhead of self- attention and make the network efficient. Few works use attention approach to define the regional of interest for the fine- grained visual recognition (Zheng et al., 2020; Ding et al., 2021; Wharton et al., 2021). In contrast, our model utilizes regional-to-locat attention to alleviate the workload of global self-attention by local attention while still keeping the global information via regional attention.
Vision Transformer. VIT (Dosovitskiy et al., 2021) has recently achieved comparable results to CNNs when using& a huge amount of data, e.g., JFT300M (Sun et al., 2017). DeiT (Touvron et al., 2020; Wightman, 2019) subsequently proposes an efficient training scheme that allows vision transformer to work on par with CNNs while training only on ImageNet1K K (Deng et al., 2009). Despite promising performance of ViT, the architecture is directly borrowed from natural language processing, which might not be suitable for vision applications. Motivated by this, two main line of research works have been developed improve VIT. One is enhance different components of the vision transformer (Yuan et al., 2021; Han et al., 2021; Touvron et al., 2021; Jiang et al., 2021) while still using isotropic structure (i.e., fixed token numbers and channel dimension) like ViT, e.g., while T2T- ViT (Yuan et al., 2021) introduces a Tokens-to- structure for each token, Cross ViT propose a dual-path architecture, each with different scales, to learn multi-scale features (Chen et al., 2021a). CaiT (Touvron et al., 2021) proposes a layer-scalar to train a deeper network for better performance and LV- (Jiang et al., 2021) modified how the model is trained when CutMix (Yun et al., 2019) augmentation is applied on VIT. Another parallel thread for improving vision transformer is in incorporating CiN-like pyramid structure into VIT (Wu et al., 2021; Wang et al., 2021; Heo al., 2021; Zhang et al., 2021 la; Li et al., 2021; Graham al., 2021; Liu et al., 2021; Chu et al., 2021a; Yan et al., 2021; Chen et al., 2021b). PVT (Wang et al., 2021) and PiT (Heo et al., 2021) introduce the pyramid structure into VIT. which makes them more suitable for objection detection as it can provide multi-scale features. LocalViT (Li et al., 2021) and Con'T (Yan et al., 2021) mix convolutions with self-attention to encode locality information. Swin (Liu et al., 2021), ViL (Zhang et al., 2021a) and Twins (Chu et al., 2021a) limited the self self-attention into local region and then propose different methods allow the interaction among each local region. Our model also utilizes pyramid structure and limits the self-attention in a local region; while we propose the regional-to-lose attention to exchange the information among each region efficiently.
(T2T) transformation to encode local
Difference from PVT, Swin, ViL and Twins. Table summarizes the difference between our proposed approach, ViT with the closely related works. PVT (Wang et al., 2021) uses pyramid structure but the scope of self-attention is still global. While ViL (Zhang et al., 2021a) limits self-attention into a local region, they adopt overlapping windows as convolutions to process all the tokens. On the other hand, Swin (Liu et al., 2021). Twins (Chu et al., 2021a) and ours adopt non-overlapped windows. While Swin shifts the position of windows alternatively in the consecutive transformer encoder allow the interaction between regions, Twins subsamples local tokens the global tokens, and then use the global tokens as the keys for self-attention to achieve the interaction between regions. By contrast, our proposed method utilizes a extra set of regional tokens with regional-to-love attention to perform self-attention on regional tokens only for the global information and then each regional token is sent to the associated local tokens to pass the global information. One major difference from others is that the local self-attention in our approach includes one extra regional token to exchange the global information with local tokens.
METHOD
Our method is built on top of a vision transformer, so we first present a brief background of ViT and then describe method (Region on regional-to-local attention for vision transformers.
3

---

Published as a conference paper at ICLR 2022
C-H2B-WW2
Stgge:
DoH12800022
Blage:
Stape
Bigae
Prediction
tokans
zation
RZL
R2L
R2L
R2L Transformer Encoder
Input
Transformer
Transformer
Transformar Encodar
Ensoder
Encoder
Tekonnation
Mekess
God MieWW
2G1 MB
Figure 2: Architecture of the proposed Region VIT. Two paths are in our proposed network, including regional tokens and local tokens, and their information is exchanged in the regional-to to-call (R2L) transformer encoders. In the end, average all regional tokens and use it for the classification. The tensor shape here computed based on that regional tokens take a patch of 28 and local tokens take patch of 42.
Vision Transformer. As opposed to CNN- bas-dased approaches for image classification, ViT is a purely attention-bbsed counterpart, borrowed from NLP. It consists of stacked transformer encoders, each of which contains multihead self-attention (MSA) and feed-forward network (FFN) with layer normalization and residual shortcut. To classify an image, ViT first splits it into patches of fixed size, e.g. 16x 16, and then transforms them into tokens by linear projection. A class token is additionally prepended to the patch tokens to form the input sequence. Before the token are fed into transformer encoders, a learnable absolute positional embedding is added to each token to learn the position information. At the end of the network, the class token is used as the final feature representation for classification. Mathematically, VIT can be expressed as
X [xolel] + Xpos
Yk X;-1 +
YA + FFN()N()))),
where Xele E RICC and Xpatch E NXC are the class token and patch tokens respectively and Xpos E ((++N)xC is the position embedding, and [I]]] denotes the tensor concatenation. k. N and C are the layer index, the number of patch tokens and dimension of the embedding, respectively.
While the vanilla ViT is ideally capable of learning global interaction among all the patch tokens, the memory complexity of self-attention becomes high when there many tokens as the complexity is quadratically linear to the length of the input sequence. Moreover, the use of isotropic structure limits the capability of extending the vanilla ViT model to many vision applications that require high-resolution details, e.g., object detection. To address these issues, we propose oigional-to-local attention for vision transformer, Region VIT for short.
An Overview of Our Proposed Approach ReeiooVVTT, Figure 2 illustrates the architecture of the proposed vision transformer, which consists two tokenization processes that convert an image into regional (upper path) and local tokens (lower path). Each tokenization is a convolution with different patch sizes, e.g., in Figure 2, the patch size of regional tokens is 28 while 42 is used for local tokens with dimensions projected to C, which means that one regional token covers 72 local tokens based on the spatial locality, leading to the window size of a local region to 72. At stage I, two sets of tokens passed through the proposed regional-to-loce transformer encoders. However, for the later stages, to balance the computational load and to have feature maps at different resolutions, we deploy a downsampling process to halve the spatial resolution while doubling the channel dimension like CNN on both regional and local tokens before going to the next stage. Finally, at the end of the network, we average the remaining regional tokens as the final embedding for the classification while the detection uses all local tokens at each stage since it provides more fine-grained location information. By having the pyramid structure, the ViT can generate multi-scale features and hence it could be easily extended to more vision applications, e.g., object detection, rather than image classification only. We explain the main components of the regional-to-local transformer encoder in the next section.
Regional-to-Loch Transformer Encoder. Figure 3 shows the proposed regional-to-local (R2L) transformer encoder which includes R2L attention and feed-forward network (FFN). Specifically, within R2L attention, the regional self-attention (RSA) first involves all regional tokens to learn the global information efficiently as the number of regional tokens is few and then local self-attention (LSA) takes the local tokens with the associated regional token to learn local feature while the involved regional token could provide global information at the same time. Both RSA and LSA are multihead self-attention (MSA) but with different input tokens. Finally, the FFN is applied to enhance the features. We also add layer normalization (LN) and residual shortcuts as in standard transformer encoders. Mathematically, given the regional and local tokens, x and x as the inputs at layer d, the R2L transformer encoder can be expressed as:
d-1
4

---

Published as a conference paper at ICLR 2022
y = x,- (xx) 1), yig = -Iv", (xx d-1
h.y.m. mm.
(2)
2 = yū, + LSA((L(()).))) x 2 + (LN
where j are the spatial index with respect to regional tokens while m, n are the index of local token in the window size M Note that the input of LSA,, y includes one regional token and corresponding local tokens, and thus, the information between local and regional tokens exchanged; on the other hand, the outputs, x and x can be extracted from x like the top-right equation.
The RSA exchanges information among all tokens, which covers the context of the whole image; while the LSA combines the features among the tokens belonging to the spatial region, including both regional and local to- divided
Regional Takans
Rěi Attemtion
RSA
Feed Forword Netwark
LSA
kens. Since the regions by non-overlapped windows, the RSA also designed to exchange the in- formation among regions where the I.SA takes one regional token and then combines with it the local tokens in the same region. In such a case, all lo- cal tokens still capable of getting global information while being more focused local neighbors. It is worth noting that the weights are shared be- tween RSA and LSA except for the layer normalization; therefore, the number of parameters won't increase significantly when compared to the standard transformer encoder. With these two attentions, the R2L can effectively and efficiently exchange information among all regional and local tokens. In particular, the ooff-attention on regional tokens aims to extract high-level information and act as bridge to pass information of local tokens from one region other regions. On the other hand, the R2L attention focus on local contextual information with one regional token.
Sull-Anariion USA Local Sel-Mtterion
Local Tokens
Figure 3: Illustration of Regional-to-Local (R2L) Transformer Encoder. All regional tokens are first passed through regional self-attention (RSA) to exchange the information among regions and then local self-attention (LSA) performs parallel self-attention where each takes one regional token and corresponding local to- kens. After that, all the tokens are passed through the feed-forward network and split back to the regional and local tokens. RSA and LSA share the same weights.
Moreover, the R2L transformer encoder can further reduce the memory complexity significantly. The memory complexity of self-attention is O(N2), where N is the number of local tokens. When a region contains M local tokens, and there are N/M regions, the memory complexity is ((M++ 1)2 x (N/M)); while the complexity from the attention on regional tokens is ((NN/M) Hence, the overall complexity becomes O ((M + 1)2 x
(NVM)) E.g., the main component of Region VIT is stage 3, where M is 49 and N is 196; therefore, the memory complexity saving is ~ 73%. The saving of each model and each stage is varied based on the model configuration.
Relative Position Bias. The locality is important clue for understanding visual content; therefore, instead of adding absolute position embedding used by the vanilla ViT, we introduce the relative position bias into the attention map of R2L attention since relative position between patches (or pixels) are more important than the absolution position as objects in the images can be placed in an arbitrary way (Ramachandran et al., 201 2019a; Liu et al., 2021). We only add this bias to the attention between local tokens and not the attention between regional tokens and local tokens. Specifically, for given pair of local tokens at location, (I vm), (Izz Un), the attention value a can be expressed as
hiT (M) + b(zm - 2n Mm -w.)) -..)
(zm Mm (3 softmax
(3)
where b(s... -im y..) taken from a learnable parameter B 122--x2M-1 where Af is the window size of a region. The first term (.mm..m) 1.7' ( is the attention value based on their content.
With relative position bias, our model does not require the absolute positional embedding as vanilla VIT since this relative position bias helps encode the position information.
Downsampling. The spatial and channel dimension are changed along different stages of our network by simply applying 3 x 3 depth-wins wise convolution with stride 2 for halving the resolution and doubling the channel dimension for both regional and local tokens (weights are shared) (Heo et al., 2021). We also test with regular convolutions but it does not improve the accuracy with increased complexity.
5

---

Published as a conference paper at ICLR 2022
Table 2: Model architectures of RegionVIT. For all networks, the dimension per head is 32 and the expanding ratio in FFN is 4. The patch size of local tokens is always 4 while the patch size of regional tokens 4 M.
Model
Tokenization
Window size (M)
Dimension (C)
of Encoders each stage
Local
Regional
RegionViT: -T
3-comv
linear
7
(64, 128, 256, 512
{2. 2. 2 [2, 2,228.2)
Region V
3-conv
lineur
99, 192, 384, 768
Region ViT M
1-conv
linear
7
(96, 192, 384, 768)
(2, 2. 14
RogionViT-B
1-conv
lincar
7 {128, 256, 512, 1024]
(2, 14,2)
1-conv: Convlour- X-8, 5-4, p-3) 3-conv: Conv(out=CA4, k=3, s=2, p=1) Conviout-C22 1=3, s=2, p=1) -+
Conviout=C, s1. where C chanuei dimeusion of the first block. LayerNorm GeLU are added between Coes.
Input Tokenization. The tokenization for local tokens can be implemented by using a 4x4 convolu- tion with channel size C² and stride (i.e. non-overlapped), However, as pointed out in T2T (Yuan et al.. 2021), CrossViT (Chen et al., 2021a), and PiT (Heo et al., 2021), using a stronger but still simple subnetwork for the tokenization could further improve the performance, especially for the smaller models. Thus, we adopt two input tokenizations in our model, one still contains only one convolutional layer and another one contains three convolutional layers (see Table for details).
Regional Tokens. We adopt simple approach to generate regional tokens, that is, using linear projection with larger patch sizes in comparison to the patch size for the local tokens. E.g., as shown in Figure 2, the patch size for local tokens is 4x4, simply use patch size 28 x2 to split the image and then project each patch linearly to generate regional tokens.
Table 2 shows our Region ViT models with different configurations, By default, the window size is 7 while we also experiment with a larger window size (14), those models are annotated with +.
4 EXPERIMENTS
4.1 IMAGE CLASSIFICATION
Datasets. We ImageNetlK (Deng et al., 2009) (INIK) and ImageNet21K (Deng et al., 2009) (IN21K) to validate our method. ImageNer K contains 1 1.28 million training images and 50k val- idation images over 1k classes, and Image K is large-scale dataset that consists of around 14 million images over 21,841 classes. We use all images for training and then finetune the model on ImageNetIK. Moreover, we also perform the transfer learning from ImageNet to five down- stream datasets, including CIFARIO 10 (Krizhevsky et al., 2009), CIFAR (Krizhevsky et al., 2009), IllPets (Parkhi et al., 2012), StandfordCars (Krause et al., 2013) and ChestXRay8 (Wang et al., 2017).
Training and Evaluation. We follow DeiT (Touvron et al., 2020) to train our models on INIK except that we use batch size 4,096 with a base learning rate 0.004 and the warm-up epochs 50. We adopt the AdamW (Loshchilov & Hutter, 2019) optimizer with cosine learning rate scheduler (Loshchilov Hutter, 2017). We apply Mixup (Zhang et al., 2018), CutMix (Yun al., 2019), RandomEras- ing (Zhong et al., 2020). label smoothing (Szegedy et al., 2016), RandAugment (Cubuk al., 2020) and instance repetition (Hoffer et al., 2020). During training, we random cropped a 224 x region and take 224x 224 center crop after resizing the shorter side to 256 for evaluation. We used similar setting for IN21K and transfer learning, and more details can be found in Section A.I.
Table 3: Comparisons with recent pyramid-like structure-based ViT models ImageNetIK. The bold numbers indicate the best number within each section.
Model
Piram (M)
Acc. (%)
Model
FLOPPs (G)
(9)
PIT-XS
10.6
1.4
ConT 2020)
6-4
81.8
ConT-S
10.1
1.5
76.5
PCPVT
2021a)
PVT- (Wing al. 2021)
13.2
1.9
(Wang 2021)
ConVilTee (d' (d'Ascali 2021)
10.0
20
2021)
32.0
7.1
TVI [Chn 2021aa
340
38
Twins-SVTB (Chn
2011aa
8.3
PITS (Hea al., 2001)
23.5
299
VIL M (2hnng et 202
39.7
3.7
83.3
ConT (Yin
2021)
192
3.1
80.2
Swin (Laa et 2021)
50.0
8.7
83.0
WVS-PPPVVIS (Chu et 2021a)
24.1
3.7
81.2
PVT4 (Wang et al., 2021)
61.4
9.8
N1.7
PVT-S (Wang at. 2021)
24.5
3.8
79.8
ConViT-A (f Ascali at 2021)
46.0
10.0
82.2
RigionV/TT-i
13.8
2.4
R0.4
NisTS (Phang et al., 2021bb
35.0
10.4
83.
ReglunW/T-Ti- Ti+
14.3
2.7
$1.5
ReginViT M
41.2
7.4
83.
DefT (Tovvrnn et al. 2020)
22.1
4.6
79.9
RegionVIT-A M4
42.0
7.9
X3.4
CVT-13
2021)
20.0
4.5
81.6
DelT (Tonston 2020)
86.6
17.6
$1.8
Swis-T (Lia et 2011)
29.0
4.5
81.3
FIE-B (Hoo at. 2021)
75.8
12,5
82.0
LocatViT-S (Li al., 2021)
22.4
466
80.8
ViL-B (Thmng al. 2021g)
55.7
13.4
83.2
VIL (2hing al., 2021a)
24.
4.9
82.0
Twin--VVT-..CC (Cho 3021a)
99.2
14.8
83.7
Visformer-3
20211b)
2021b)
402
Swin (Liu 2021)
86.0
15.4
83.5
ConVIES Ascoli al., 2021)
27.0
ConViT- Ad'Asoll
2021)
86.0
H2.4
NerT.S (7hang 2011b)
17.0
NeST
2021b)
68.0
N3.8
RegionN/T
30.5
5.3
Region VIT-R
ReginnNiT-SS
31.3
5.7
ReponVil
73.8
13.6
83.8
6

---

Published as a conference paper at ICLR 2022
Table 4: Results on INIK with IN21K and transfer learning.
(a) INIK results with IN2IK.
(b) Transfer learning on downstream tasks.
Model
Params (M)
FLOPss
Acc. (%)
CIFARIO
CIFAR100
Pet
StanfordCars
ChestXRay8
V
55.7
43.7
86.0
DeiE-S
99.1
90.9
94.9
91.5
15.4
Swin-L
197.0
103.9
87.3
DeiT-B
94.4
91.7
CVT-W24
277
193.2
87.7
RegionViT-S
98.9
90.0
95.3
92.8
$7.8
RegionViT-B-
76.5
42.6
Region VIT-M
99.0
90.8
95.5
91.9
58.3
Table 5: Object detection performance on MS COCO va12017 with x and 3x schedule. The bold number indicates the best number within the section, and for MaskRCNN, both AP" and AP" are annotated.
Params
HLOPs
RetineNet
Parama
HLOPS
MaskRUNN
Backbone
(M)
(G3
AP
AP
(M)
(G)
AP
Apm
APa
AP
ResNet50
37.7
234
36,3
39.0
44.2
260.0
38.0
34.4
41.0
37,1
ConT-M (Yan al.. 2021)
PVT-S (Wang et a 202()
34.2
40.4
42.2
44.1
40.4
37.8
43.0
39.6
VIL-S (Zhang et al., 2021a)
35.7
252.2
41.6
42.9
45.0
174.3
41.8
38.5
43.4
39.6
Swin-T (L.in
2021)
38.5
245
41.5
43.9
47.8
42.2
39.1
46.0
TWins-SVT-S (w/ PEG) (Clu et al., 202
34.3
209
43.0
45.6
44.0
228
43.5
40.3
46.8
42.6
RegionV/I-S
4088
192.6
42.2
4..88
50.1
171.3
42.5
59.5
46.3
42.3
ViT-OnViT-SS
41.5
201.2
43.1
46.9
50,9
182.9
43.5
40.1
47.3
43,4
V PEG
204.3
50.9
44,2
40.8
47.6
ResNet101
56.7
315
38.5
40.9
63.2
336
40.4
36.4
42.8
38.5
ResNeXri01-5244d
56.4
319.1
39.9
41.4
62.8
340
41.9
37.5
44.00
39.2
PVT-M (Wang al. 2021)
42.0
ViL-M (Zhang et at., 2021a)
50.8
338.9
42.9
43.7
60.1
261.1 1
43.4
39.7
44.6
40.7
Swin-S (Liu ct al., 2021)
59.8
335
44.5
46.3
69.1
354
44.8
40.9
47.5
42.8
Twins-SVT-B PEG) (Chu et al., 2021a)
67.0
322
45.3
46.9
76.3
340
45.2
41.5
48.0
43.0
Region
92.2
287.9
47.2
RegionV/T--B
84.4
328.1
44.2
46.9
93.2
307.1
44.5
41.0
48.1
43.5
VogionViT--B w/ PEG
84.5
328.2
44,6
46.9
93.2
307,2
45,4
41.6
48,3
43,5
RcsNcX1101-64i
95.5
38.4
PVT-L (Wang et 2021)
71.1
345
42.6
$1.0
364
42.9
39.5
ViL-B iZhang et 2021a)
66.7
443.0
44.3
44.7
76.1
365. 1
15.1
41.0
45.7
41.3
Swin B (Lin
202.))
98.4
477
44.7
107.2
496
45.5
41.3
Twins-SVT-.. (w/ PRG) (Chu et al., 2021a)
110.9
455
119.7
45.9
41.6
RegionViT-BB VIT-B+ w/ PEGf
84.5
505.4
46.1
48.2
93.2
464.4
46.3
42.4
49.2
44.5
The reposted results of Swin are from Twins the original paper does not include resutls with ImageNettK weights. 1: iapui resolutio is 896x 1364.
Results on ImageNetIK. Table 3 shows the results on ImageNetIK where the methods listed all adopt CNN-like pyramid structure into the ViT and are all very recent and concurrent works. For the smaller models (Ti, S), RegionViT achieves a better trade-off between accuracy and complexity (parameters FL.Oss;; on the other hand, for the larger models (M and B), it obtains better accuracy while having fewer FL.PPs and parameters. The efficiency of Region VIT comes from the proposed R2L transformer encoder, and hence with such efficiency, it enables the network to be wide and deep for better accuracy with comparable complexity.
Results on ImageNet2IK. Table 4a shows that our model achieves good accuracy-efficiency tradeoff with ImageNet21K for pretraining. Region ViT-B+ only needs 25% parameters and FLOPs compared to CvT-W24 (Wu et al., 2021) and half parameters and FLOPs to Swin-L (Liu et al., 2021).
Results on Transfer Learning. Table 4b shows the transfer learning results with ImageNet pretrained weights. Our model outperforms DeiT (Touvron et al., 2020) by 2~~3% on ChestXRay8, which has a larger domain gap from ImageNet K than other four datasets. We think this is because the hierarchical feature models could provide better generalization for the domain gap compared to DeiT which uses isotropic spatial resolution throughout the whole network.
4.2 OBJECT DETECTION AND SEMANTIC SEGMENTATION
Dataset. We use MS COCO 2017 to validate our models (Lin et al., 2014) on object detection. COCO 2017 dataset contains 18K images for training and 5K images for validation across 80 categories. We also test our model on keypoint detection and results are shown in Sec. B. On the other hand, validate our model with semantic segmentation ADE20K (Zhou et al., 2017). The ADE20K dataset contains 20k images in the training set, 2k images for validation.
Training and Evaluation. For object detection, adopt RetinaNet (Lin et al., 2017) and MaskR- CNN (He et al., 2017) as our detection framework. We simply replace the backbone with Region ViT, and then output the local tokens each stages multi-scale features for the detector. The regional
7

---

Published as a conference paper at ICLR 2022
tokens not used in the detection. Before sending the features to the detector framework, we add new layer normalization layers to normalize the features. We use Region ViT-S and Region ViT-B as the backbone and the weights are initialized from ImageNetIK pretraining. The shorter side and longer side of the input image is resized to 672 and 1,120, respectively. We train our models based on the settings in 1x and 3 schedule in Detectron2 (Wu al., 2019) for object detection. More training details could be found in A.2. For semantic segmentation, we adopt Semantic FPN as the framework (Kirillov et al., 2019) and use Region ViT the backbone. We initialize models with ImageNet1K weights, and mostly follow the training receipt in Twins' paper (Chu et al., 2021a). More training details can be found in Sec. A.3.
Results on Object Detection. Table 5 compares Region ViT with other methods. For the smaller models, RegionViTs and Region ViT-S+ achieve similar accuracy while being moderately efficient in terms of FL.Pss. We find that the position encoding generator (PEG) proposed by (Chu et al., 2021 could help the detection accuracy substantially, so we also provide the results with PEG, which are also used in Twins (Chu et al., 2021a). For the middle size models with RetinaNet, Region ViT-B are slightly worse than Twins with xx schedule; however, our model catches up the performance with 3x schedule. When comparing the large models under similar FLOPs (Region ViT-B+ w/ PEGf), further train our model with the similar resolution used by other works, and observe that our models outperform all others while being more parameter efficient (models marked with in Table 5).
Results on Semantic Segmentation. Table com- pares ours to Swin (Liu et al., 2021) and Twins. As PEG is useful for object detection, we show the re- sults with PEG. Our models achieved significant im- provement with similar FLOPs for both models. This suggests that the proposed R2L attention can effec- tively model global context.
Table 6: Performance on semantic segmentation.
Model
FLOPs (G)
Params (M)
mlaU (%)
Swin.
41.5
Twins-SVTTS
28.3
43.2
Region ST (a7 PEG)
37
35.7
45.3
Swin-S*
53.2
45.2
Twins SVT-B
Region Re (w/ PRG)
78.3
47.5
Nuathers caod from the reprodased resalis Twins' paper.
4.3 ACTION RECOGNITION
Datasets and Setup. We validate our approach on Kinetics400 (K400) (Kay et al., 2017) and Something-SSomthiine V2 (SSV2) (Goyal et al., 2017). We adopt the swiied-space-ting attention in TimeSformer (Bertasius et al., 2021) as the temporal modeling perform action recognition experiments. More details could be found in Sec. A.4.
Results. Table shows that with Region ViT as the backbone, the model can be much more efficient with competitive accuracy to TimeS- former, which uses vanilla VIT as the back- bone. RegionViT-N could reduce more than 50% FLOPs and parameters than the TimeS- former. This not only shows the importance of spatial modeling in action recognition but also validates that our proposed model can also be extended for efficient action recognition.
Table 7: Performance on action recognition.
Model
FLOPP (Gi)
Params (M)
K4000 Acc. (%)
SSV2 Acc. (8)
TimeStormer
197
121.4
75.8
59.5
x TTimeStomme!
197
121.4
77.1
59.2
RegionViT-S
59.4
42.9
76.6
59.7
RegionNIT-M
83.1
57.5
77.6
59,8
stogle cop. T: retraned with the sune settnng
4.4 ABLATION STUDY
We perform the following experiments to verify the effectiveness different components in Region- VIT. All experiments are conducted based on Region ViT-S.
Table 8 shows the results with and without regional tokens three tasks, and Table A5 includes FLOPs and parameters comparison. For image classification, the re- gional tokens provide around 0.4% improve- ment with negligible overhead in both compu- tations (6%) and parameters (0.7%). On the other hand, the regional tokens clearly improve the performance of object detection and semantic segmentation with negligible overhead (<2% on both FLOPs and parameters). This is because dense prediction tasks require more multi-scale features with global contextual information (provided by the regional tokens) than image classification.
Regional Tokens.
Table 8: Ablation on regional tokens.
Dataset Regional Tokens
INIK
COCO
ADE20K
Acc.
MaskRCNN (AP"/AP")
RetinaNet (AP*)
SemanticFPN
(mloU)
N
82,2
40.5/37.8
40.7
42,3
82.6
42.5/39.5
42.2
43.7
8

---

Published as a conference paper at ICLR 2022
Downsampling. Table shows different ap- proaches for downsampling the patches. 3x3 kemel size achieves better results than 2x2 con- volution, because the kernel is non-overlapped with 2x2 convolution, which limits the interac-
Table Different downsampling approaches.
Downsampling
Params
FLOPs (G)
INIK Acc. (%)
2x2 convolution
32.1
5.5
3x3 convolution
34.1
5.7
82.5
3x3 depth-wise conv.
30.6
5.3
82.6
tion among local tokens. Moreover, using regular convolution does not improve the performance but
increases computations and parameters. Thus, we use 3x3 depthwise convolution for downsampling.
Weight sharing between RSA and LSA as well as downsampling. Table 10 shows the re- sults with and without weight sharing. As seen from the table, using separated weights for RSA and LSA only slightly improves the accuracy but significantly increases the model size. This suggests sharing parameters between RSA and LSA suffices for learning local and global information in our approach.
Table 10:
Weight sharing.
Weight sharing
Params
FLOPs (G)
INIK Acc. (%)
N
40.4
5.3
82.7
30.6
5,3
82.6
Keys LSA. Table 11 studies the number of regional tokens in LSA. When using all regional tokens in the LSA, it provides the local tokens the possibility explore all global information. Nonetheless, this model only improves 0.1% % but with 13% more computations, which suggests that when considering the regional tokens, the one
Table 11: Keys in LSA.
Regional tokens Params
FLOPs (G)
INIK Acc. (%)
30.6
6.0
82.7
Corresponding
30.6
5.3
82.6
associated with the current region is sufficient to achieve good performance.
Position information. Table 12 compares per- formance of different combinations of absolute position embedding (APE) and relative position bias (RPB). The models with RPB achieve bet- ter accuracy with similar FLOPs and parameters. Although the model with APE could slightly im-
Table 12: Different position information.
Abs. pos.
Rel. pos.
Params (M)
FLOPs (G)
INIK Acc. (%)
30.6
82.4
Y
N
30.9
5,3
82.2
Y
Y
30.9
82.7
Y
30.6
5,3
82.6
prove the performance, limits the model run at fixed resolution, which is not suitable for vision
tasks where image size could be varied. Thus, we only adopt the relative position bias in our models.
Overlapped windows. Table 13 compares the results with and without overlapped windows. The overlapping ratio of neighboring windows is 50% and hence this model allows more in- improves
Table 13: Overlapped windows.
Model
Params FLOPs (G)
INIK Acc. (%)
overlapped
30.6
18.8
82,8
non-overlapped
30.6
5.3
82.6
teractions among local tokens.
Region by 0.2% but increases the FLOPs by 3.5x. This suggests that the information exchange between the border of windows is sufficiently covered by the proposed R2L attention.
Comparison between R2L attention and shifted-window attention. Table 14 shows the comparison between winddww atten- tion (Liu et al., 2021) and proposed R2L atten- tion. R2L attention outperforms shifted-window attention by 0.7%, which shows that our proposed R2L attention can effectively model global information achieve good performance.
Table 14: Comparison of different attentions.
Model
Params
FLOPs (G)
INIK Acc. (%)
Shified-window
29.0
4.5
81.3
R2L
32.1
5.2
82.0
CONCLUSION
In this paper, we propose new ViT architecture that exploits the pyramid structure used most of CNNs to provide multi-scale features and hence, the models can be more easily extended to different vision applications, like object detection. Moreover, the proposed regional-to-local attention relaxes the memory overhead of performing self-attention on the fine-grained image tokens by limiting the scope of attention but still keeping the capability to explore global information. Extensive experiments on several standard benchmark datasets well demonstrate that our proposed models outperform on par with many concurrent ViT variants on four vision applications, including image classification, object and keypoint detection, semantic segmentation and action recognition.

---

Published as a conference paper at ICLR 2022
ETHICS STATEMENT
Our work introduces a memory-friendly and efficient self-attention method for transformer models, which have wide-ranging applications to image classification, object detection, human activity recognition, elc. By improving efficiency, our work can have positive effect on these applications in society, allowing faster processing. E.g., this could enable faster responses to detected events such as medical emergencies or detecting defects in manufacturing parts. Lower computation costs could also save energy and therefore have positive impact on the environment. Negative impacts of our research are difficult to predict, however, it shares many of the pitfalls associated with deep classification models. E.g., Image and video classification systems have negative implications on privacy and could be used by malicious actors or governments to infringe the privacy of citizens. Future research into private and ethical aspects of visual recognition is an important direction.
REPRODUCIBILITY STATEMENT
In order to reproduce our results, we describe the details the hyperparameters used in our training in Appendix A for all the tasks and also attach codes for image classification as part of the supplemental material. We will publicly release all the codes and models after acceptance. In the attached codes, our experiments with exactly the same hyperparameters. The README file also describes the requirement to build necessary Python environment for running the codes.
all the hyperparameters their default value, so that can
10

---

Published as a conference paper at ICLR 2022
REFERENCES
Irwan Bello. Lambdanetworks: Modeling long-range interactions without attention. International Conference on Learning Representations, 2021. URL cestpp::/oppnzeeee
net forum?
Irwan Bello, Barret Zoph, Ashish Vaswani, Jonathon Shlens, and Quoe V Le. Attention augmented convolutional networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3286-3295, 2019.
Gedas Bertasius, Heng Wang. and Lorenzo Torresani. Is Space-Time Time Attention All You Need for Video Understanding? arXiv.org, February 2021.
Andrew Brock, Soham De, Samuel L Smith, and Karen Simonyan. High-Performmace Large-Scale Image Recognition Without Normalization. arXiv, 2021.
Chun-Fu Chen, Quanfu Fan, and Rameswar Panda. Cross VTT: Cross-Attention Multi-Scale Vision Transformer for Image Classification. arXiv.org, March 2021a.
Zhengsu Chen, Lingxi Xie, Jianwei Niu, Xuefeng Liu, Longhui Wei, and Qi Tian. Visformer: The Vision-friendly Transformer. arxiv.org. April 2021b.
Xiangxiang Chu, Zhi Tian, Yuqing Wang, Bo Zhang, Haibing Ren, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Twins: Revisiting Spatial Attention Design in Vision Transformers. arXivorg.
April 2021a.
Xiangxiang Chu, Zhi Tian, Bo Zhang, Xinlong Wang, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Conditional Positional Encodings for Vision Transformers, arXiv, 2021b.
Ekin Dogus Cubuk, Barret Zoph, Jon Shlens, and Quoe Le. RandAugment: Practical Automated Data Augmentation with Reduced Search Space. In H Larochelle, M Ranzato, R Hadsell, M F Balcan, and H Lin (eds.), Advances in Neural Information Processing Systems, pp. 18613-18624. Curran Associates, Inc., 2020.
Stéphane d' Ascoli, Hugo Touvron, Matthew Leavitt, Ari Morcos, Giulio Biroli, and Levent Sagun. Con VIT: Improving Vision Transformers with Soft Convolutional Inductive Biases. arXiv.org. cs.CV, 03 2021. URL arxiv.orr
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 JEEE conference on computer vision and pattern recognition, pp. 248-255. leee, 2009.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chupter of the Association for Computational Linguisties: Human Language Technologies, Volume (Long and Short Papers), pp. 4171-4186, Minneapolis, Minnesota, June 2019, Association for Computational Linguistics. doi: 10.18653/11N11--42223 URL https: // //www. ac web. org/ anthooogyyy////ooooooooooooooooa / /N1 9- -19-1423.
Yifeng Ding, Zhanyu Ma, Shaoguo Wen, Jiyang Xie, Dongliang Chang, Zhongwei Si, Ming Wu, and Haibin Ling. Ap-cnn: Weakly supervised attention pyramid convolutional neural network for fine-grained visual classification. IEEE Transactions on Image Processing, 30:2826-288 2836, 2021. doi: 10.1109/TIP.201 13055617..
Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner. Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference Learning Representations, 2021. URL https: / ope.revie....eee. net / forum? i d-
Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, et al. The" something something" video database for learning and evaluating visual common sense. In ICCV, 2017.
11

---

Published as a conference paper at ICLR 2022
Ben Graham, Alaaeldin El-Nouby, Hugo Touvron, Pierre Stock, Armand Joulin, Hervé Jégou, and Matthijs Douze. LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inference. arXiv.org. April 2021.
Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, and Yunhe Wang. Transformer in transformer. arXiv preprint arXiv:2103.00113 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition, In The IEEE Conference Computer Vision and Patlern Recognition (CVPR), June 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollar, and Ross Girshick. Mask r-enn. In Proceedings the IEFE International Conference on Computer Vision (ICCV), Oct 2017.
Byeongho Heo, Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Junsuk Choe, and Seong Joon Oh. Rethinking Spatial Dimensions of Vision Transformers. arXivoorg, March 2021.
Elad Hoffer, Tal Ben-Nun, Itay Hubara, Niv Giladi, Torsten Hoefler, and Daniel Soudry. Augment your batch: Improving generalization through instance repetition. Proceedings of the JEEF/CVF Conference on Computer Vision and Puttern Recognition (CVPR), June 2020.
Han Hu, Zheng Zhang, Zhenda Xie, and Stephen Lin. Local relation networks for image recognition. In Proceedings of the IEEE/CVP International Conference on Computer Vision, pp. 3464-3473, 2019.
J. Hu, L. Shen, and G. Sun. Squeeze-and-excitation networks. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7132-7141, 2018. doi: 2..109//CPR..2088.00777
Zihang Jiang, Qibin Hou, Li Yuan, Daquan Zhou, Xiaojie Jin, Anran Wang. and Jiashi Feng. Token Labeling: Training a 85.4% Top-1 I Accuracy Vision Transformer with 56M Parameters on ImageNet. arxiv.rrg 04 2021. URL arxiv. org.
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, al. The kinetics human action video dataset. arXiv: arXiv:1705.06950 2017.
Alexander Kirillov, Ross Girshick, Kaiming He, and Piotr Dollar. Panoptic feature pyramid networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In 4th International IEEE Workshop on 3D Representation and Recognition (3dRR-13), Sydney, Australia, 2013.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Yawei Li, Kai Zhang, Jiezhang Cao, Radu Timofte, and Luc Van Gool. LicalViT: Bringing Locality to Vision Transformers. arXiv.org, April 2021.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common Objects Context. In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars (eds.), Computer Vision - ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V, pp. 740-755. Springer International Publishing, Cham, 2014.
Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollar, Focal loss for dense object detection. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), Oct 2017.
Ze Liu, Yutong Lin, Yue Cao. Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin Transformer: Hierarchical Vision Transformer using Shifted Windows, arXiv.org, March 2021.
12

---

Published as a conference paper at ICLR 2022
Iyy Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. In Interna- tional Conference on Learning Representations, 2017.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- on Learning Representations, 2019. URL htwps: /ttp:://////peeeeeew http:eviwwwweeeeetteeeeeeeeeeeee Bkg6RicqY7.
Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In JEEE Conference on Computer Vision and Pattern Recognition, 2012.
Prajit Ramachandran, Niki Parmar, Ashish Vaswani, Irwan Bello, Anselm Levskaya, and Jon Shlens. Stand-Alone Self-Attention in Vision Models. In H Wallach, H Larochelle, A Beygelzimer, F d Alch Buc, Fox, and R Garnett (eds.), Advances Neural Information Processing Systems. Curran Associates, Inc., 2019a.
Prajit Ramachandran, Niki Parmar, Ashish Vaswani, Irwan Bello, Anselm Levskaya, and Jonathon Shlens. Stand-alone self-attention vision models. arXiv preprint arXiv: 1906.05909 2019b.
Aravind Srinivas, Tsung-Ye Lin, Niki Parmar, Jonathon Shlens, Pieter Abbeel, and Ashish Vaswani. Bottleneck transformers for visual recognition. arXiv preprint arXiv:2101.11605: 2021.
C. Sun, A. Shrivastava, S. Singh, and A. Gupta. Revisiting unreasonable effectiveness of data in deep learning era. In 2017 IEEE International Conference on Computer Vision (ICCV), pp. 943-852, 2017. doi: 10.1 10.1109/CCC22017..99
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
Mingxing Tan and Quoe Le. EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, pp. 6105-6161 6114, Long Beach, California, USA, June 2019, PMLR.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers distillation through attention. arXiv preprint arXiv:2012. 12877, 2020.
Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper with Image Transformers. arXiv.org, es. Cv, 03 2021. URL arxiv.ovrog org.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is All you Need. In I Guyon, U V Luxburg, S Bengio, H Wallach, R Fergus, S Vishwanathan, and R Garnett (eds.), Advances in Neural Information Processing Systems. Curran Associates, Inc., 2017.
Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions, 2021.
Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7794-7803, 2018.
Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, Mohammadhadi Bagheri, and Ronald M Summers. Chestx-ray8; Hospital-scale chest x-ray database and benchmarks weakly-supervised classi- fication and localization of common thorax diseases. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2097-2106, 2017.
Zachary Wharton, Ardhendu Behera, and Asish Bera. An attention-driven hierarchical multi-scale representation for visual recognition, 2021.
Ross Wightman. Pytorch image models.
pytorch-
image wwwoorchh-mage-mmmoee
s, 2019.
13

---

Published as a conference paper at ICLR 2022
Sanghyun Woo, Jongchan Park, Joon- Young Lee, and In So Kweon. Cbam: Convolutional block attention module. In Proceedings of the European Conference on Computer Vision (ECCV), September 2018.
Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. CvT: Introducing Convolutions to Vision Transformers. arXiv.org, March 2021.
Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick. Detectron2. 2019.
: http:////waccccooeeaaeetetttttae
Haotian Yan, Zhe Li, Weijian Li, Changhu Wang, Ming Wu, and Chuang Zhang. ConTNet: Why not convolution and transformer at the same time? arXiv.org, April 2021.
Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Francis EH Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet, 2021.
Sangdoo Yum, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. CutMix: Regularization Strategy to Train Strong Classifiers With Localizable Features. In Proceedings of the IEEF/CVF International Conference Computer Vision (ICCV), October 2019.
Hongyi Zhang. Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference Learning Representations, 2018. URL https:
// //oppereviee www.viewwrr. www.nne/fooumm forum? d
Pengchuan Zhang, Xiyang Dai, Jianwei Yang, Bin Xiao, Lu Yuan, Lei Zhang, and Jianfeng Gao. Multi-Scale Vision Longformer: New Vision Transformer for High- Resolltisn Image Encoding. arXiv.org. March 2021a.
Zizhao Zhang, Han Zhang, Long Zhao, Ting Chen, and Tomas Pfister. Aggregating Nested Trans- formers. arXiv, 2021b.
Hengshuang Zhao, Jiaya Jia, and Vladlen Koltun. Exploring self-attention for image recognition. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), June 2020.
Heliang Zheng, Jianlong Fu, Zheng-Jun Zha, Jiebo Luo, and Tao Mei. Learning Rich Part Hierarchies With Progressive Attention Networks for Fine-Grained Image Recognition. IEEE Transactions on Image Processing, 29:476-488 2020. ISSN 1057-7149, doi: 10. 10.1109000 20109/t
Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li. and Yi Yang. Random Erasing Data Augmen- tation. Proceedings of the AAAI Conference on Artificial Intelligence, 34000): April 2020.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
14

---

Published as a conference paper at ICLR 2022
Table Al: Details of training settings for image classification on ImageNetIK and ImageNet21K.
INIK
IN21K
Finetune to INIK @384
Transfer
Batch size
4,096
4,096
1,024
768
Epochs Optimizer
300
120
30
1,000
Adam W
AdamW W
AdamW le-8
SGD
Weight Decay
0.05
0.01
le-4
Linear-rate Scheduler
Cosine (0.004)
Cosine (0.001)
Cosine (0.0002)
Cosine (0.01)
(Initial L.)
Warmup Epochs
50
5
Warmup linear-rate
Linear (1e-6)
Scheduler (Initial L.)
Data Aug.
RandAugment (m=9, n=2)
Mixup (o:)
0.8
CutMix (a)
1.0
Random Erasing
0.25
0.0
Instance
Repetition
Drop-path
0.1
0.0
Label Smoothing
0.1
disabled for Region ViT-ii
APPENDIX
Summary This appendix contains the following additional details. First, in Sec. A, we describe the training details on image classification, object and keypoint detection, semantic segmentation and ac- tion recognition separately. Then, we provide detailed results on object detection and ablation studies. The supplementary materials include our codes to reproduce our results on image classification and the README file in the attached codes (Region ViT Code.zip) also provides the detailed instructions to train and evaluate the networks.
A TRAINING DETAILS
A.1 IMAGE CLASSIFICATION
We follow DeiT (Touvron et al., 2020) to train our models on ImageNetI (Deng et al., 2009) except that we use batch size 4,096 with base learning rate 0.004 and the warm-up epochs is 50. We adopt the AdamW (Loshchilov & Hutter, 2019) optimizer with cosine learning rate scheduler (Loshchilov Hutter, 2017). We apply Mixup (Zhang et al., 2018), CutMix (Yun et al., 2019), RandomEras- ing (Zhong et al., 2020), label smoothing (Szegedy et al., 2016), RandAugment (Cubuk al., 2020) and instance repetition (Hoffer et al., 2020). During training, we randomly crop 224x 224 region and take 224 x 224 center crop after resizing the shorter side to 256 for evaluation. While pretraining on ImageNe121K (Deng et al., 2009), we use similar settings with slight modifications. We train the model using 120 epochs with 5 epochs for warmup, weight-decay of 0.01 and base learning rate of 0.001. For transfer learning experiments, we finetune the ImageNet1K pretrained models with 1,000 epochs, batch size of 768, learning rate of 0.01, SGD optimizer. weight decay of 0.0001, and using the same data augmentation in training on ImageNetIK. During evaluation, we resize the shorter side of an image to 256 and then take 224x224 region at the center and report top-1 accuracy. For finetuning experiments from ImageNet21K. we finetune the models with higher resolution of 384 384, for 30 epochs with cosine learning rate scheduler, base linear rate of 0.002, weight-decay of le-8. Moreover, as it is trained on larger resolution, we adjust the window size M to have the same number of regional tokens at the first stage, e.g., we change the window size to 12 for the the models trained at 224x with window size 7. We adopt the bicubic interpolation to upsample the weights of tokenizations and relative position bias. During evaluation, we directly resize the shorter side to 384 and then take center 384 x 384 crop. We trained the models with 32 GPUs.
15

---

Published as a conference paper at ICLR 2022
Table A2: Object detection performance on the COCO val2017 with 1x schedule. The bold number indicates the best number within the section, and for MaskRCNN, both AP and AP" are annotated.
NetioNNs
Poras (M)
PLOPs
MasARCNN
Backbure
AP%
(0)
A,7"
ReaNee30
37.7
234
56.3
55.3
38,6
19.3
40.0
45.8
46.2
260.0
38,0
58.6
41.4
34.4
55.1
36.7
CooF-M (Y6m et at. 2011
93.0
212.2
1
99.3
41.
15.1
41.1
51.9
40.1
16:1
PVTS (Wlng et cl.. 20211
34.2
40.4
61.3
43.0
25:0
42.9
53.3
48.1
40.4
62.9
43.8
37.8
60:3
40.3
V90-8 (2hang al., 202.1))
41.6
62.5
44.6
45.0
174.3
41.8
61.3
62.1
25.1
(B/PP 2021a)
44.0
228
RegionViT'S
1926
64.1
27.5
50.1
170.3
BesianViT-S+
41.5
204
43.1
64.8
46.2
20.6
46.6
56.1
50.9
182.9
43.5
65.9
47.5
40.4
63.7
43.4
RegionVVT-SS S+ W/D
41.5
204.3
43.9
65.5
47.3
28.5
47.3
$7.9
50.9
183.0
44.2
67.3
48.2
40.8
64.1
44.0
RcoNs1001
21.4
336
40.4
ResNoXr101-35
31M1
PVTM (Waaz ci... 2021)
53.9
25.0
63.9
VIl-M (Zhang etal. 20211a)
50.6
138.
42.4
17
6UL1
263.1
67.0
19.7
Swie-8 in al. 7001)
99.6
325
68.5
65.7
47.5
43.0
59.9
69.
756
64.8
05.5
689
41.9
63.4
44.3
Twiss-VVTBB PEG: (Cha etal. 2021a)
67.0
322
45.3
56.7
48.1
28.5
45.9
02.6
76.3
340
65.2
67.6
49.3
41.5
64.5
44.5
ReginnViiT-
814
1059
00.7
781-9
61.1
65.7
47.4
61.4
ResiseViT
328.1
20.2
47.5
307-1
RogionViiT-Se
ReaNccri01-6444 PVE) (Wing st 30213
95.5
473
41.0
50.9
44.0
23.9
45.2
54.0
101.
493
42.3
63.5
47.3
38.4
60.6
41.3
71.1
515
47.6
65.7
45.4
35.8
46.0
56.4
$1.0
166
42.4
65.0
66.6
19.5
61.9
43.5
VIL, B (2taag at 20216)
65.7
443.0
44.3
65.5
47.1
28.9
47.9
58,3
76.1
365
45.1
67.2
49.3
41.0
64.3
44.2
Swis-3 cl 2001)
98.4
442
167.2
400
45.5
Twiss SVT
(Chu 2021a)
110.9
119.7
Region/V BB (SO]
30.5
The regened maals Swin (1 20201 are froee TWiss et.l. 1001/0 the oegihl peper not Inclado mwrk wilt BrageNeiK wights 1 Iogutmelltte RM6
A.2 OBJECT DETECTION AND KEYPOINT DETECTION
We adopt RetinaNet (Lin et al., 2017) and MaskRCNN (He et al., 2017) as our detection framework and use KeypoinyRCNN for keypoint detection, and we simply replace the backbone with Region VIT, and then output the local tokens at each stages as multi-scale features for the detector. The regional tokens are not used in the detection. Before sending the features to the detector framework, we add new layer normalization layers to normalize the features. We use Region as the backbone and the weights are initialized from ImageNetIK pretraining. We train our models based on the settings in x schedule in Detectron2 (Wu et al., 2019), i.e., the batch size is 16 and the learning rate is set to 0.0001 which drops 10x at the 60,000-th and 80,000-th iteration. We use AdamW optimizer with weight-decay of 0.05, and resize the shorter and longer side of an image to 672 and 1,120, respectively. We also adopt drop-path when finetuning the detector, with rate set to 0.2 for both RetinaNet and Mas We trained the models with 8 GPUs. When training with longer schedule (3x), we adopt stronger data augmentation (multi-scale training) used Swin (Liu et al., 2021) or Twins (Chu et al., 2021a) rather than fixed-size training used in 1 1x schedule.
and Region ViT-B
A.3 SEMANTIC SEGMENTATION
We adopt Semantic FPN as our semantic segmentation framework (Kirillov et al., 2019) by replacing the backbone network with RigionViT like the object detection task. We train the models with ImageNetIK pretrained weights. We mostly follow the training receipt in Twins (Chu et al., 2021a) The shorter side of the image is resized to 448 and the longer side won't exceed 1792, and then crop a 448 448 region for training. During the evaluation, we take the whole image after resizing as illustrated above. We train the model with AdamW optimizer with learning rate le-4 for 80k iterations, and the learning rate is decayed with poly schedule (power is 0.9). The weight decay and drop-path rate 0.05 and 0.2, respectively, We trained all models with 8 GPUs.
A.4 ACTION RECOGNITION
We adopt the divided-space-tine attention in TimeSformer (Bertasius et al., 2021) as the temporal modeling to perform action recognition experiments. More specifically. we plugin the temporal attention before every R2L transformer encoder and finetune the model pretrained with ImageNet|K (instead of ImageNet21 K). We use the Adam W optimizer (Loshchilov & Hutter, 2019) with base linear rate of 2e-4 and weight-decay of 0.0001. We apply the same Mixup (Zhang et al., 2018), CutMix (Yun et al., 2019) and drop path (Tan & Le, 2019) in the image classification. For the input, we uniformly sample 8 frames from the whole video and the 224% x 224 region is cropped after shorter side of images is resized to the range of 256 to 320, resulting the size of the input video as 8x224 xx224x222 224..24. During evaluation, we use the same way to sample frames but take 224x 224 spatial crops (top-left, center and bottom-right) after resizing shorter side of image to 256, and then ensemble the predictions final prediction for the input video. We trained the models with 16 GPUs.
16

---

Published as a conference paper at ICLR 2022
Table A3: Object detection performance on COCO val2017 with schedule. The bold number indicates the best number within the section, and for MaskRCNN, both AP and AP" are annotated.
RetioNes
MasARCNN
Backbure
A,"5
BeaNNe50
37.7
234
39.0
58.4
41.5
22.4
42.6
51.6
46.2
260.0
41.0
61.7
44.9
37.1
55.4
40.3
PVTS (Wlong al.. 2011
$4.3
42.3
62:3
45.0
2
45.3
51-3
46.1
345
43.0
65.3
66.9
19.9
03.5
43.4
VIL (2hung et cl., 2021a)
35.7
252.2
42.9
68.8
63.6
27.8
46.4
56.3
45.0
174.3
43.4
64.9
47.0
02.1
42.4
Swis-T clare
2001a)
RegionViT-S
RegionViT-S+
204
08.3
50.
182.9
RegisnViT-S S+ FEG
41.5
204.3
46.7
68.2
30.2
30.7
50.6
62.4
50.9
183.0
47.6
70.0
52.0
43.4
67.1
47.0
ResNettM)
56.7
315
00.5
0AL1
44.0
23.7
45.0
63.2
336
62.3
63.2
00.1
41.3
RcoN:33101--2248
(Wang 00.00.200110
Swin-S Lninet 20013
99.5
335
66.3
00.33
156
42.8
Thrns-VVi V---8000 owl PHC) (Uh)))000e etal.etall 2001aa
67.0
377
66.1
06.11
90.1
7
90.3
01.6
76.
340
68.0
69.5
52.7
43.0
06.1
96.6
RestonViT
83.4
308.9
66.1
67.8
07.1
31.5
50.2
01.2
92.2
287.9
62.2
69.1
51.7
43.0
65.4
06.5
RaginnVii 1-36+
8444
101-1
RegionViT 2021a)
94.5
328
31.1
93.2
307.2
ViL
65.7
443.0
20.9
76.1
365.1
RegionViT wi PBOT
94.5
306.4
48.2
50.9
$1.5
34.3
$1.5
93.2
461.4
49.3
71.0
53.7
44.5
65.4
48.3
reporeed
regieal poperdoss
ImageNart weghts
Table A4: Performance on person keypoint detection.
Model
FLOPs (G)
Params (M)
bbox AP (%)
keypoint AP (%)
ResNet-50
137.7
59.1
53.6
64.0
Region VIT-S+ (w/ PEG)
172.2
65.7
56.0
66.1
B MORE DETAILED RESULTS
We provided more details of some Tables shown in the main paper here. Table A2 and Table A3 include more metries as compared Table 5. Table A4 shows the results on keypoint detection. Region VIT outperforms ResNet-50 with moderate increases in FLOPs and parameters. This suggests that Region VIT can model the global context keypoint detection as well. Table A5 includes complexity comparison for the ablation study on regional tokens as shown in Table 8.
17

---

Published as a conference paper at ICLR 2022
Table A5: Performance w/ and w/o regional tokens Region ViT-S.
Regional Tokens
FLOPS (G)
Params (M)
ImageNetIK Acc. (%)
5.0
30.4
82.2
Y
5.3
30.6
82.6
MaskRCNN
MS COCO (AP*/AP")
168.5
49.9
40.5/37.8
Y
171.3
50.1
42.2/39.0
RetinaNet
MS COCO (AP*)
189.8
40.5
40.7
192.6
40.8
41.7
SemanticFPN
ADE20K (mloU)
N
36.8
34.7
42.3
37.3
35.0
43.7
Table A6: Throughput comparison.
Model
Acc. (%)
Params (M)
FLOPs (G)
Throughput (images/sec)
Swin-T
81.3
29.0
4.5
1129
Swin-S
83.0
50.0
8.7
710
Region ViT-S
82.6
30.6
5.3
823
Region ViT-M
83.1
41.2
7.4
706
18


Reviews:
[{"review_id": "yH5oxAqg97a", "reviewer": "Reviewer_YwKN", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "+ives\nThe paper is well-written and easy to follow. \nThe experimental evaluation on various dataset is very impressive. However, the accuracy improvement is not convincing.\nThe accompanied source code.\n\n-ives\nThere is a significant advancement using part-hierarchies (Zheng et al TIP 2019), Attention Pyramid (Ding et al. TIP 2021), attention-driven hierarchical multi-scale (Warton et al., ArXiv 2021, I am aware that this paper is added recently)  and patches are all you need (this is also under review, https://github.com/tmp-iclr/convmixer). It is unclear how the regions and patches are novel. To me, it is how adapted in transformers.\n\nIt is unclear how memory saving is 73% when you are computing self-attention among regions followed by local self-attention between regions and local patches.\n\nIs dimension of C for features representing local patches and regional patches same?\n\nThe ablation studies is very comprehensive. However, missing ablation study without self-attention between regions (first step of R2L attention)\n\nIt is also unclear on the 2nd step of the which one is used as key for multi-head attention is it  local patches? If so what could be the outcome if we use the region as keys.\n\nThe overall accuracy of the model is good but not significantly better than the SotA approaches. The accuracy is better for + models in which the window size is 14 and the model has higher GFLOPS and parameters.", "summary_of_the_paper": "The paper presents a new ViT architecture by adopting the concepts of pyramid structure and employ regional-to-local attention instead of global self-attention as in standard vision transformers. The regional and local concept is linked to patch size i.e. region consists of larger patches in comparison to local. A regional token is associated a set of non-overlapping local tokens while computing reginal-to-local attention. This attention is computed using a regional self-attention comprising all reginal tokens to extract global information, followed by a local self- attention mechanism that pass information between the reginal token and the linked local tokens via self attention. The approach is evaluated image recognition, object and keypoints detection, semantic segmentation and action recognition. The performance is comparable to the state-of-the-art.", "summary_of_the_review": "The idea of large patches to capture high-level shape, as well as concentrate on detailed texture and parts information using smaller patches is very good. However, the justification could have been improved. The paper is well-written and the experimental evaluation is comprehensive. However, the improvement in accuracy is not significant in comparison to the way the novelty of R2L attention is described. There is an advancement to the transformer model however the impact is limited. The related work is very nicely done.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "qi-Qnn1ufM4", "reviewer": "Reviewer_iQks", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Advantages\n1) This paper is well written and easy to follow.\n2) The experiments are sufficient to evaluate the proposed method.\n\n### Disadvantage\nSince the R2L attention has two steps, I am concerned about the throughput of RegionViT. FLOPs can not directly reflect the throughput. It is better to show the throughput comparison for Table 3.  ", "summary_of_the_paper": "This paper introduces local inductive bias into vanilla vision transformer by adopting the pyramid structure and employing regional-to-local attention. The regional-to-local (R2L) attention is the main contribution of the paper. Compared to the vanilla self-attention, R2L has two steps: 1) self-attention on regional tokens; 2) self-attention on local tokens and their corresponding regional token. ", "summary_of_the_review": "The idea sounds direct and reasonable, although introducing local inductive bias into ViT is not and there have been lots of paper to achieve it. The paper is well-written and shows efficient experiments to evaluate the proposed RegionViT.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "qRK3H4iR-ry", "reviewer": "Reviewer_1u9G", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths:**\n- Overall the paper is fairly easy to follow and it's easy to understand how the approach works, including a fair amount of detail to potentially reproduce results. The provided code, albeit I did not run it, seems like it should work.\n- In many of the presented experiments, RegionViT does indeed obtain on par or state-of-the-art results when compared to similarly sized models\n- Quite some relevant and interesting ablations are presented.\n\n**Weaknesses:**\n- For me the main weakness of the paper is that it seems fairly incremental in a longer line of ideas. First there was ViT, followed by more efficient training from DeiT including many augmentations, several works then proposed a pyramid-like architectures, and finally this work is a small increment on how to best realize the pyramid architecture including some smaller tweaks. The scores indeed do improve, but sometimes there are quite some different variants (vanilla, +, + w/PEG) that are not consistently listed for the different tasks, partially raising the question how large the margin to the previous models really is. While I don't doubt that the approach has some merit, I would find the paper significantly more valuable if the focus of the experiments would be to dive deeper into how to best design the pyramid structure of the network and less on listing many different tasks with some table showing state of the art results. Now this paper presents yet another new proposal on how to change change the ViT architecture, but I gain little insight into why it really performs better. Shifting the focus to this part of the paper would be more interesting in my opinion. For example it would be interesting to discuss some of the ablation results in more details, or to do a more thorough investigation into the size of the local and global patches. One could even ask the question if two levels (local+regional) are optimal, or an additional super-regional or global attention could also be useful?\n- I was a bit surprised by the fairly low results from the ImageNet 21k transfer experiments. Of course it is clear that the FLOP count and parameter count is significantly lower than the better models in Table 4a, however, it does raise the question if RegionViT can indeed be scaled to such an extend that it matches the state of the art. Given that one big part of method is the reduced complexity, it is also not surprising that the model is computationally less heavy, but it would be a drawback if there is some performance ceiling it hits due to the specific architecture. Here it would be good to create some even bigger model to see if the larger models can be matched or outperformed. Also the other transfer experiments in Table 4 are not super convincing.\n- The keypoint detection and action recognition sections take up quite some space and especially Table 6 does not list any convincing baselines. At the same time these experiments don't add huge value to the overall story in my opinion. I would recommend to use this space to perform more relevant and interesting ablations/experiments/discussions instead and move these to the supplementary material.\n- Lastly, the writing is not the best in some parts of the paper. For a potential final version, I would recommend that the paper is proofread by a native speaker.\n\n\n**Open questions:**\n- Table 1 makes some statements about overlapping vs non-overlapping patches. The way the local tokenization works (using 1 or 3 convs)), I would say it is somewhat of a stretch so say that the patches are strictly non-overlapping.\n- I was fairly surprised about weights being shared between RSA and LSA, as well as in the downsampling convs. I don't really see a larger number of parameters as a huge issue for a model and the FLOP count/inference speed should be the same if these weights would not be shared. Actually ablating this would be very interesting. I would kind of expect that the local and regional tokens contain slightly different things and having separate weights could potentially improve the results?\n- The paper states that this model saves 73% of memory. I'm wondering if this is an empirical measure, or it was derived based on the complexity analysis? I guess this factor might also be different for the Ti/S/B/L models?\n- I find it quite surprising that the regional tokens don't have a huge effect on the overall performance (Ablation 1). Is the remaining strong performance based on the pyramidal structure of RegionViT? Given that the pure local attention never is performed on overlapping areas that is the only reason I would see why the model still remotely works.", "summary_of_the_paper": "The paper introduces a new spin on the ViT architecture. Similar to other recent developments, it proposes a pyramidal style architecture where subsequent layers process the input image at decreasing resolutions. This allows the approach to be significantly more memory and compute efficient compared to the original ViT architecture, while still obtaining solid results on several tasks and benchmarks. At the core of the method is a regional-to-local attention approach, where very coarse regional tokens are processed by a normal multi-head attention yielding global attention and each coarse regional patch is additionally represented by a set of smaller local patches, which are also processed together with the corresponding regional token to get finer-grained local attention. Each layer in the network follows this principle and between different sections of the network both the regional and local tokens are downsampled with strided conv layers. ", "summary_of_the_review": "Overall the paper proposes a fairly interesting pyramidal modification to ViT. It seems like a somewhat incremental approach though. While the performance is empirically shown, there is little that can really be learned about the underlying approach from the experiments. Nothing is really wrong with the paper, I do think it could be significantly better though. As such I recommend the paper to be accepted, but I do think it has a lot more potential that is not uncovered at the moment and I would actually be very happy to read such an improved version of the paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "med5hYQ6kb5", "reviewer": "Reviewer_rREV", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "+ The performance of this work is promising. \n-  The novelty of this work is somewhat trivial, the issue of the interaction between global and local information has been proposed by previous work, not only in image classification but also in other downstream tasks. And the proposed methods are not novel enough when compare with the previous ones.\n- Several grammatical mistakes are made. e.g. In abstract, the regional self-attention extract global information among... (extracts); \n", "summary_of_the_paper": "This paper proposes a new architecture that adopts the pyramid structure and employs a novel regional-to-local attention for vision applications. The regional-to-local attention reduces the memory complexity as compared to standard self-attention. \n", "summary_of_the_review": "This work aims to solve the interaction between global and local information in a ViT architecture and to save the computation at the same time, the performance is good. However, I am concerned with the novelty of this work and the contribution may be limited.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposed a new architecture called Regional-to-Local attention for the vision transformers. The idea is easy to understand, the model adopts the pyramid structure and adds a regional to local attention instead of using the global attention. The architecture is well-motivated and the paper is generally well written.

The main concerns from the reviewers are mostly clarification questions. The authors did a good job addressing them. Apart from those, most reviewers raise the novelty issue of such architecture, which I would think is a drawback of this paper.

I am leaning towards the acceptance of this paper mainly because of its experimental results. It is the best in my batch and I think there is a significant improvement over the previous approaches.
Decision: Accept (Poster)