Focal Attention for Long-Range Interactions in Vision Transformers
Authors: Jianwei Yang
Chunyuan Li
Pengchuan Zhang
Xiyang Dai
Bin Xiao
Lu Yuan
Jianfeng Gao
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: Focal Attention
Self-Attention
Long-range Interactions
Local-Global Interactions
Vision Transformer
Image Classification
Object Detection

Abstract: Recently, Vision Transformer and its variants have shown great promise on various computer vision tasks. The ability to capture local and global visual dependencies through self-attention is the key to its success. But it also brings challenges due to quadratic computational overhead, especially for the high-resolution vision tasks(e.g., object detection). Many recent works have attempted to reduce the cost and improve model performance by applying either coarse-grained global attention or fine-grained local attention. However, both approaches cripple the modeling power of the original self-attention mechanism of multi-layer Transformers, leading to sub-optimal solutions.  In this paper, we present focal attention, a new attention mechanism that incorporates both fine-grained local and coarse-grained global interactions.  In this new mechanism, each token attends its closest surrounding tokens at the fine granularity and the tokens far away at a coarse granularity and thus can capture both short- and long-range visual dependencies efficiently and effectively. With focal attention, we propose a new variant of Vision Transformer models, called Focal Transformers, which achieve superior performance over the state-of-the-art (SoTA) Vision Transformers on a range of public image classification and object detection benchmarks.  In particular, our Focal Transformer models with a moderate size of 51.1M and a large size of 89.8M achieve 83.6% and 84.0%Top-1 accuracy, respectively, on ImageNet classification at 224×224.  When employed as the backbones, Focal Transformers achieve consistent and substantial improvements over the current SoTA Swin Transformers [44] across 6 different object detection methods.  Our largest Focal Transformer yields58.7/59.0boxmAPs and50.9/51.3mask mAPs on COCO mini-val/test-dev, and55.4mIoU onADE20K for semantic segmentation, creating new SoTA on three of the most challenging computer vision tasks. 

Focal Attention for Long-Range Interactions in Vision Transformers
Jianwei Yang'
Chunyuan Li' Pengchuan Zhang'
Xiyang Dai Bin Xiao²
Lu Yuan² Jianfeng Gao!
Microsoft Research at Redmond, Miicrosott Cloud Al
[jianwyan, chunyl, wwwppan,nieeeeeee xidai, bixi, luyuan, wmgooommicoooootccoo
Abstract
Recently, Vision Transformer and its variants have shown great promise various computer vision tasks. The ability of capturing local and global visual dependencies through self-attention is the key to its success. However, this also brings challenges due to quadratic computational overhead, especially for the high-resolution vision tasks (e.g., object detection). Many recent works have attempted reduce the cost and improve model performance by applying either coarse-grained global attention fine-grained local attention. However, both approaches cripple the modeling power of the original self-attention mechanism of multi-layer Transformers, leading sub-optimal solutions. In this paper, we present focal attention, a new attention mechanism that incorporates both fine- tineggrinned local and coarse- sauuse-raiied global interactions. In this new mechanism, each token attends its closest surrounding tokens at fine granularity and the tokens far away at coarse granularity, and thus can capture both short- and long-range visual dependencies efficiently and effectively. With focal attention, we build new variant of Vision Transformer models, called Focal Transformers, which achieve superior performance over the state-of-the- art (SoTA) Vision Transformers object detection benchmarks. particular, our Focal Transformer models with moderate size of 51.1M and large size of 89. 99.8M achieve 83.6% and 84.0% Top-1 accuracy, respectively, on ImageNet classification at 224 x 224. When employed the backbones, Focal Transformers achieve consistent and substantial improvements over the current SOTA Swin Transformers [3 across 6 different object detection methods. Our largest Focal Transformer yields 58.7/59.0 box mAPs and 50.9/51.3 mask mAPs on COCO mini-val/test-dev, and 55.4 mloU on ADE20K for semantic segmentation, creating new SoTT on three of the most challenging computer vision tasks. Our code is available at: htttp::/gitthi comm/microsoot/Foocal htaas//Foooal--Taafrreeeeee
range of public image classification and
1 Introduction
Nowadays, Transformer cessing (NLP) 20
57 has become a prevalent model architecture in natural language pro- In the light of its success in NLP, there is an increasing effort on adapt- ing it to computer vision (CV) 47 50 Since its promise firstly demonstrated in Vision Trans- former (iT) 21], we have witnessed flourish of Frll-Transformer models for image classifica-
tion 155
60
64
43 76 (56), object detection [8
85
79
[8] and semantic segmentation [58] 62)
Beyond these static image tasks, has also been applied on various temporal understanding tasks,
such as action recognition 40 78 10
object tracking [13 59 scene flow estimation (3
The self-attention mechanism is arguably the key component that differentiates Transformers from the widely used convolutional neural networks (CNNs) (37] in computer vision. At each Transformer layer, self-attention enables global content-dependent interactions among different image regions for
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Query Patch
Attended Patches
Fine
Close
2
Pread's
Far
Figure I: Left: Visualization of the attention maps of the three heads the given query patch (blue) in the first layer of the Deiy-Tiny model [55]. Right: An illustrative depiction of focal attention mechanism. Three granularity levels are used to compose the attention region for the blue query.
modeling short- and long-range dependencies. respectively. Through the visualization of full self- attention results]] we indeed observe that self-attention learns to attend local surroundings (like CNNs) and the global contexts the same time, illustrated in Fig. | (Left). Nevertheless, when dealing with high-resolution vision tasks such as object detection or segmentation, an efficient implementation of global and fine-grained self-attention becomes non-trivial due the quadratic computational cost with respect to the number of tokens in feature maps. Recent works have alternatively exploited either marse-grained global self-attention [60 64 fine-grained local self-attention 43 76 (56), for the sake of reducing the computational cost. However, both approaches cripple the power of the original full self-attention i.e., the ability to simultancously capture local and global visual dependencies.
In this paper, we present a new attention mechanism to capture both short- and long-range interactions in Transformer layers for high-resolution input images. Considering that the visual dependencies between the nearby (local) regions are usually much stronger than the dependencies between the regions that are far away, we perform the fine-graine graiird attention only in local regions while the coarse-graine -rained attention globally. As depicted Fig. [ (Right), a query token in the feature map attends its closest local surroundings the finest granularity regions far away, it attends to summarized tokens to capture soarse-grained visual dependencies. We call this new mechanism focal attention, as each token attends the others in focal manner. We will show in this study that focal attention allows to effectively model visual dependencies among all regions covering the whole high-resolution feature maps while introducing much less number of tokens in the computation than that in the standard self-attention mechanism.
itself. However, when it goes the
Equipped with focal attention, series of Focal Transformers developed and validated via comprehensive empirical study across three core vision tasks, including image classification, object detection and segmentation. Results show that Focal Transformers consistently outperform the SoTA Vision Transformers across various settings (i.e., in model sizes and complexities). Notably, the small Focal Transformer with 51.1M parameters achieves 83.6% top-1 accuracy ImageNet- K, and the base model with 89.8M parameters obtains 84.0% top-1 accuracy. In the fine-tuning experiments for object detection, Focal Transformers consistently outperform the SoTA Swin Transformers 43 across six popular object detection methods. Our largest Focal Transformer model achieves 59.0 box mAP and 51.3 mask mAP on COCO test-dev for object detection and instance segmentation, respectively, and 55.4 mIoU on ADE20K for semantic segmentation. These results demonstrate that focal attention highly effective modeling the global interactions in Vision Transformers.
2 Related work
Vision Transformers. Vision Transformer (ViT) is first introduced in 21). It applies standard Transformer, originally developed for NL.P (77. to encode an image by analogously splitting the image into sequence of visual tokens. It has demonstrated superior performance to CNNs such as ResNet (33 on multiple image classification benchmarks, when trained with sufficient data [2]] and carefully designed data augmentation and regularization methods (55) The results thus inspire researchers to explore the applications of ViT on various vision tasks beyond image classification,
such self-supervised learning 14 g 39 object detection g 85 79 18 and semantic segmenta-
tion [58 62 There are also increasing number of studies for improving ViT via data effaceeffi
DDiiT-Tiny model, checkpoint downloaded from http::///gtabbbooo//cobbeeeeeeet

---

8
16
16
32
32
Rd
Multi-layer
Multilayer
Multhaaer
Muhislayer Percaptron
Percaptron
Perceptron
Perceptron
Patch embedding
Patch Embbedding
Patch Sebodding
Patch ambbodddigg Beodiing
LayerNorm
LayorNorm
Layerliomm
XxWx3
Attention
Focal
Self
Focal
Focal
Focal
LayerNorm
Layerhiorm
LayerNorm
LayerVormm
Focal Transformer XN
Focal Transformer
Transformer
Focal Transformer Stage 4
Stage 1
Stage 2
Stage 3
Figure 2:
Model architecture for Focal Transformers. As highlighted in light blue boxes, main innovation is the proposed focal attention in each Transformer layer.
training [55] improved patch embedding/encoding [6
311 integrating convolutional projections
into transformers [64] [70], and using multi-scale architectures and efficient self-attention mechanisms
for high-resolution vision tasks [60 64
43
76 15 Recent surveys include [36 30 36). This paper
focuses improving the self-attention mechanism of ViT for encoding high-resolution images.
Efficient global and local sell-attention. many real-world tasks, Transformers need to cope with a large number of input tokens, such as long documents in NLP and high-resolution images computer vision (CV). Recently, many efficient self-attention mechanisms have been proposed to deal with the quadratic computational and memory cost incurred by the standard self-attention mechanism. On one hand, a number of works in both NLP and CV resort to coarse-gae saarseeg global self-attention (i.e., attending the Nown-samples or summarized tokens) to capture the long-range Although this approach improves the model efficiency, loses the detailed context information surrounding the query tokens. On the other hand, to make computational cost manageable, various local fine-grained attention mechanism (i.e., attending neighboring tokens within pre-set window size) are used for both NLP K 1 and CV [56 43 76).
interactions
49
46
60
31 23
In this paper, we argue that both global and local attentions are important for model performance. This is also validated by some recent studies that aim improve CNNs by incorporating ways of modeling global attentions [35] 63 61 68) 2 51I The standard self-attention mechanism used shown in Fig. T (Left). But it often incurs
by VIT can indeed learned both types of attentions, a prohibitively high cost for high-resolution images. To the best of our knowledge, the proposed focal attention provides the first mechanism to incorporate local and global attention in single Transformer layer I can capture both short- and long-range interactions as standard self-attention but in much more efficient and effective way, especially for high-resolution images.
Method
3.1 Model architecture
To accommodate high-resolution dense prediction tasks, employ multi-scale model architecture as in (60 76 43). As shown in Fig. an image R #W xXxx3xx3 x3 is first partitioned into patches size 4 x 4, resulting in x I visual tokens with dimension 4 x 4 3. Then, we patch embedding layer, consisting of convolutional layer with filter size and stride both equal to 4, to project these patches into hidden features with dimension d. We then pass this spatial feature map to the four stages of Focal Transformer blocks. each stage i E (1, 2, 3, 4). the Focal Transformer block consists of patch embedding layer to reduce the spatial
W.
N Focal Transformer layers. After each stage, size feature map by factor 2 and increase the feature dimension by 2. For image classification tasks, take the average of the output from the last stage and send it to classification layer. For object detection, the feature maps from the last 3 or all stages are fed to particular object detector head,
A similar focal mechanism has been used in CNNs for NLP 27

---

depending on the specific detection method we choose to use. The model capacity can be customized by varying the input feature dimension d and the number of Focal Transformer layers.
Standard self-attention can capture both short- and long-range interactions at fine-grain, but suffers from high computational cost when it performs attention on high-resolution feature maps as noted in (76). Take stage 1 in Fig. 2 as an example. For a feature map of size T X T X d, the complexity H of self-attention is O(( I x Wy yd), resulting in an explosion of time and memory cost, considering that min( H, W) could be 800 or even larger for object detection. In the next section, we describe how address this issue with the proposed focal attention mechanism.
H W
3.2 Tiken-wise focal attention
Focal attention is proposed to make the Transformer lay- ers suitable for encoding high-resolution input images. Instead of attending all tokens at fine-grain, we attend the fine-grain tokens only locally, but the summarized ones (i.e., the coarse-grained tokens generated by sub-window pooling, which is illustrated in Fig. 4 and will be described later) globally. As such, focal attention can cover the same amount of image regions as standard self-attention but with much less cost. In Fig. 3 we show the size of the receptive field for standard self-attention and our focal attention as a function of the number of attended tokens. For given query position, by reducing the granularity of its surroundings based on their distance to the query, focal attention can have significantly larger receptive fields at the same cost measured by the number of visual tokens, compared to the standard self-attention mechanism.
Regular Sel-attendon
Focal Silf-Altandon
Nurmer locerss
Figure 3: The size of receptive field (y- axis) as a function of the number used visual tokens (x-axis) in regular (stan- dard) self-attention and focal attention. When plotting the curve for focal atten- tion, we increase the focal window size by 2 for each focal level up the maxi- mal window size of 8.
Theoretically, the focal attention mechanism enables global interaction with much less time and memory cost, because it attends a much smaller number of surrounding (summarized) tokens. In practice, however, extracting the surrounding tokens for each query position could incur high time cost since we need to duplicate the extraction of each token for all queries that the token surrounds. This issue had been extensively discussed in [56] 76 43) and a common solution is to partition the input feature map into windows. Thus, in our Focal Transformers, we resort to performing focal attention at the window level. We elaborate the window-wist focal attention the following.
3.2.1 Window-wise focal attention
Given a feature map of x E Nxx with spatial size M x N, we first partition it into a grid of windows of size 8p Sp- Then, we extract the surroundings for each window rather than each individual token. The proposed window-wise focal attention is illustrated in Fig. To clarify, we first define three terms:
Focal level L refers to the granularity level at which we extract the tokens for focal attention.
Focal window size s'y is the size of sub-wine window on which the summarized tokens are formed via sub-window pooling at granularity level of E {1....,L].
Focal region size s/ denotes the number of sub-windows that are filled up horizontally (or verti- cally) in an attended region at level 1.
Now, we detail how window-wise focal attention works in the following two steps. sub-window pooling and attention computing.
Sub-window pooling. Consider input feature map I E MxNxd where M N is the spatial dimension and d the feature dimension. We perform sub-window pooling for all L levels. At focal level I, we first split the input feature map into grid of hub-windows with size sy x s Then use linear projection layer It to pool the sub-windows spatially by
M N xd)x(sl,xxx))
z' = (2) E R
a = Reshape(x) E R
(l)
The pooled feature maps f2' H different levels I provide rich information at both fine- ine-grain and coarsc-grain. Since set s 1 for the first focal level which has the same granularity as the input

---

135 velero
Projetion
Quary
Figure 4: An illustration of focal attention window level. Each of the square cells represents visual token that is either from the original feature map or a summarized token formed by sub-window pooling. Suppose we have an input feature map of size 20 20. We first partition it into 5 windows of size x 4. Take the 4 x 4 blue window in the middle as the query set, we extract its surrounding tokens at three granularity levels as its keys and values. For the first level, we extract the 8 x 8 tokens which are closest to the blue window at the finest grain. At the second level, we expand the attention region and pool the surrounding results in 6 x 6 summarized tokens. At the third level, we attend a larger region covering the whole feature map and pool
2 sub-windows
form summarized tokens, which
sub-windows, which leads to 5 summarized tokens. Finally, these three levels of tokens are concatenated to compute the keys and values for the 4 x 4 = 16 tokens (queries) in the blue window.
feature map, there is need to perform any sub-window pooling. Considering that the focal window size is usually very small (7 maximally in our settings), the number of extra parameters introduced by sub-window pooling is negligible.
Atention computing. Once we obtain the pooled feature maps fa f at all levels, we compute the query the first level, and key and value for all levels using three linear projection layers fy, fx and fo, respectively, as
Q fa(x)), =
V - fo((z) "I).
To perform focal attention, we need to first extract the surrounding tokens for each query token in the feature map. As mentioned earlier, tokens inside window partition Sp 8p share the same set of surroundings. For the queries inside the i-th window Qi R2pxssxx,
extract the sf s keys and values from K and V surrounding the window which the query lies in, and then gather the keys and values from all levels to obtain K; (K},.., K²} Rexd and V. = (V?,... V.A) E Rexd where s is the sum of focal regions from all levels, i.e., = Et= (s Note that canonical implementation of focal attention following Fig. Fig- requires to exclude the overlapped regions across different levels. In our implementation, we intentionally keep them in order to capture the pyramid information for the overlapped regions. Finally, we follow [43 to include relative position bias and compute the focal attention for Q; by
Q.KT Vd
Attention Q., K. Vi) Softmax(
B)V,
where B = (B'} is the learnable relative position bias. It consists of L subsets for focal levels. Similar to 43), for the first level, we parameterize it as B (),222, the horizontal and vertical position ranges are both in [-sp + 1, 8p 1]. For the other focal levels, considering that they have different granularity with respect to the queries, treat all the queries inside a window equally and use B' E R to represent the relative position bias between the query window and each of of x s summarized tokens. Since the focal attention for each window can be performed independent of the others, we can compute Eq. (3) parallel. Once we obtain attention scores for the whole input feature map, send them to LayerNorm and the MLP block.
considering that
5

---

Output Size
Layer Name
Focal-Tiny
Focal-Sman Small
Focal-Base
56 x 56
Patch Embedding
4: = 96
Pt = 4:0 = 96
= 4:61 = 128
stage
Transformer Block
$ {1, 13)
(1. 13)
6 [1. 13)
56 x 56
si. (7.7)
x
su, {7,7)
x2
s (7.7)
x
28 x 28
Patch Embedding
p2 2; 02 192
P2 c2 192
Pa C2 256
stage?
Transformer Block
A 13)
(1.13)
fur (1. 13)
28 x 28
fu.r (7.5)
x
su. {7,5)
x
6 (7,5)
x
14 x 14
Patch Embeddiig
Pa = 2; = 384
Ps 2; 384
Ps cq 512
stage.3
Transformer Block
b {1, 13) furr (7.3)
Smrr {1, 13) fur (7.3)
Bmr (1, 13)
14 x 14
6
x 18
6u. [7.3)
x 18
7 7
Putch Embedding
Ps - 24 768
Pa 2;
768
Pi 2: 1024
stage
Transformer Block
Pe.r (1,77 Pour [7.1)
71 [7,
S (1, Sur (7.1)
7 x 7
x
x
x2
Table I: Model configurations for Focal Transformers. We use three configurations with different model capacities: Pocal-Tiny, Focal-Small and Focal-Base
3.2.2 Complexity analysis
We analyze the computational complexity for the two steps of focal attention described above. For the
input feature map x E RMxxXx xd, we have M N sub-windows at focal level I. For each sub-window, x a s
the pooling operation in Eq has the complexity of ((s Aggregating all sub-windows brings us O((MN) dd). Then for all focal levels, we have the complexity of ( NN )d) in total, which is
independent of the sub-window size at each focal level. Regarding the attention computation in Eq. 3
the computational cost for a query window Sp X Sp is ((sp)) (s,)) and (E,, for the whole input feature map. To sum up, the overall computational cost for focal attention is
(s)) (M In an extreme case, one can set s x max(M, NN) S to ensure a + O(L
global receptive field for all queries (including both corner and middle queries) in this layer.
3.3 Model configurations
For fair comparison, we consider three network configurations for Focal Transformers, follow- Specifically, we follow the design of the Tiny, Small and Base models in Swin as shown in Table Our models take 224 x 224 images inputs and the window partition size is set to 7 to make our models comparable to Swin Transformers. For the focal attention layer, introduce two levels, one for fine-grained local attention and the other for coarse-grained global attention. Except for the last stage, the focal region size is set to 13 for the window partition size of 7, which means that we expand 3 tokens for each window partition. For the last stage, since the whole feature map is 7 X 7, the focal region size at level 0 is set to 7, which is sufficient to cover the entire feature map. For the coarse- grainedr global attention, we set its focal window size the same as the window partition size 7, but gradually decrease the focal region size to get {7, 5, 3, 1} for the four stages, respectively. For the patch embedding layer, the spatial reduction ratio p; for the four stages are all (4, 2, 2, 2}. Note that Focal- -Base has a higher hidden dimension e,, compared to Focal-Tiny and Focal-Small.
ing 60
Transformer 43
4
Experiments
4.1
Image classification on ImageNet-1K
We compare different methods on ImageNet- IK [19]. For fair comparison, we follow the training recipes in [55 60) All models are trained for 300 epochs with batch size 1024. The initial learning rate is set to 10 with 20 epochs of linear warm-up starting from 10-5, For optimization, we use Adam W 44 as the optimizer with a cosine learning rate scheduler. The weight decay is set to 0.05 and the maximal gradient norm is clipped to 5.0. We use the same set of data augmentation and regularization strategies used (55] after excluding random erasing [82], repeated augmentation 4 34 and exponential moving average (EMA) 48 The stochastic depth drop rates are set to 0.2, 0.2 and 0.3 for our tiny, small and base models, respectively. During training, we crop images randomly to 224 x 224, while center crop is used during evaluation on the validation set.

---

Model
#Params. FL.OPs Top-I (%)
RetinaNet
Mask R-CNN
Backbone
ResNet-50
8+
25.0
4.1
76.2
APh
Ap
Apm
DeiT-Smal//10
55
22.1
4.6
79.9
ResNet-50
33
36.3
38.0
34.4
PVT-Small 660
24.5
3.8
79.8
PVT-Small
40.4
40.4
37.8
Vit--mall 76
24.6
5.1
82.0
ViL-Small
41.6
41.8
38.5
CvT-13 [64]
20.0
4.5
81.6
Swin-Tiny
43
42.0
43.7
39.8
Swin-Tiny
28.3
4.5
81.2
Focal-Tiny (Ours)
43.7
44.8 (+1.))
41.0 (+1.3)
Focal-Tiny (Ours)
82.2
ResNer-101 33
45.0
7.9
77.4
ResNet-101
38.5
40.4
36.4
ResNeXr101-32x4d 67
39.9
41.9
37.5
PVT-Medium 60
44.2
6.7
81.2
PVT-Medium 60
41.9
42.0
39.0
CvT-21 64
32.0
7.1
82.5
ViL-Medium 76
42.9
43.4
39.7
ViL-Medium
9.1
83.3
Swin-Small
+
49.6
8.7
83.1
Swin-Small +3
45.0
46.5
42.1
Focal-Small (Ours)
45.6 (+0.6)
47.4 (.0.9)
42.8 (+0.7)
Focal-Small (Ours)
51.1
83.6
ResNet-152 33
78.3
ResNeXt101 67
41.0
42.8
38.4
PVT-Large 60
42.6
42.9
39.5
ViT-Busc/16
86.6
17.6
77.9
DeiT-Basc/16 55
86.6
17.5
81.8
ViL-Base 776
44.3
45.1
41.0
PVT-Large 60
61.4
9.8
81.7
Swin-Base 13
45.0
46.9
42.3
ViL-Base 76
55.7
13.4
83.2
Focal-Base (Ours)
46.3 (+1.3)
47.8
(+0.9)
43.2 (+0.9)
Swin-Base 45
87.8
15.4
83.4
Table 3: Comparisons with CNN and Transformer baselines and SoTA methods on COCO object detec- tion. The box mAP (AP") and mask mAP (APm) are reported for RetinaNet and Mask R-CNN trained with 1x schedule. More detailed comparisons with 3x schedule are in Table 4
Focal-Base (Ours)
89.8
16.4
84.0
Table 2: Comparison of image classification on ImageNet-1K K for different models. Except for ViT-Base/16, all other models are trained and evaluated 224 224 resolution.
In Table 2 we summarize the results for baseline models and the state-of-the-arr models on image classification task. We can see that Focal Transformers consistently outperform other methods with similar model sizes (#Params.) and computational complexities (GFLOPs). Specifically, Focal-Tiny improves over the Transformer baseline DeiT-Small/16 by 2.3%. Meanwhile, using the same model configuration (2-2-6-2) and few extra parameters and computations, Focal-Tiny improves over Twin-Tiny by 1.0 point. For small and base models, Focal- -maall with 51.1M parameters can reach 83.6% which is better than all the counterpart small and base models using much less parameters. By increasing the model size, Focal-Bass model achieves 84.0%, surpassing all the other models with comparable parameters and FL.OPs.
To compare with the large-scale models, we further build Focal-Large Transformer by increasing the hidden dimension in Focal-Base from 128 to 196 while keeping all the other hyperparameters the same. We follow the common practice to pretrain our Focal-Large Transformer on ImageNet-22K and transfer detection and segmentation tasks 64
4.2 Object detection and instance segmentation
We benchmark our models on object detection with COCO 2017 42). The pretrained models are used as visual backbones and then plugged into two representative pipelines, RetinaNet [41] and Mask R-CNN 32.. All models are trained on the 1 188 training images and the results are reported on 5K validation set. We use the two standard training schedules, 1x with epochs and 3x with 36 epochs. For the 1 x schedule, we resize image's shorter side to 800 while keeping its longer side no more than 1,333. For the 3x schedule, we use the multi-scale training strategy by randomly resizing its shorter side to the range of [480, 800). Considering this higher input resolution, we adaptively increase the focal sizes at four stages to (15, 13, 9, 7). to ensures that the focal attention covers more than half of the image region at the first two stages, and the whole image at the last two stages. With the focal size increased, the relative position biases are accordingly up- up-sampled to the corresponding sizes using bilinear interpolation. During training. we use Adam W [44] for optimization with initial learning rate 10-4 and weight decay 0.05. Similarly, we use 0.2, 0.3 and 0.5 stochastic depth drop rates to regularize the training for our Tiny, Small and Base models, respectively. Since Swin Transformer does not report the results on RetinaNet, we obtain the results by ourselves using their official code that of Focal Transformers.
with the same
In Table 3 we show the performance for both CNN-based models and the current Transformer- based state-of-the-art models. The bbox mAP (AP") and mask mAP (AP") are reported. We see that Focal Transformers outperform the CNN-based models consistently with the gap of 4.8-7.1 points. Compared with the other methods which also use multi-scale Transformer architectures,
7

---

Backhone
#Params
FLOPs
RetinaNet 3x schedule + MS
Mask R- R-CNN 3x schedule MS
(M)
(G)
AP
AP%
AP
APs
APM
APL
AP
AP
AP
Apm
A
AP%
ResNer50 35
37.7/4.22
239/260
39.0
58.4
41.8
22.4
42.8
51.6
41.0
61.7
44,9
37.1
58.4
40.1
PVT-SmalI 60
34.2/44.1
226/245
42.2
62.7
45.0
26.2
45.2
57.2
43,0
65.3
46.9
39,9
62.5
42.8
ViL-Small 76
35.7/45.0
252/174
42.9
63.8
45.6
27.8
46.4
56.3
43.4
64.9
47.0
39.6
62.1
42.4
Swin-Tiny
38.5/47.8
245/264
45.0
65.9
48.4
29.7
48.9
58.1
46.0
68.1
50.3
41.6
65.1
44,9
Focal-Tiny (Ours)
39.4/48.8
265/291
45.5
66.3
48.8
31.2
49.2
58.7
47.2
69.4
51.9
42.7
66.5
45.9
ResNer101 33
56.7/63.2
315/336
40.9
60.1
44.0
23.7
45.0
53.8
42.8
63.2
47.1
38.5
60.1
41.3
ResNeX:101-32x46
67
56.4/62.8
319/340
41.4
61.0
44.3
23.9
45.5
53.7
44.0
64.4
48.0
39.2
61.4
41.9
PVT-Mcdium 60
53.9/63.9
283/302
43.2
63.8
46.1
27.3
46.3
58.9
44.2
66.0
48.2
40.5
63.1
43.5
ViL-Mediam 76
50.8/60.1
539/261
43.7
64.6
46.4
27.9
47.1
56.9
44.6
66.3
48.5
40.7
63.8
43.7
Swin-Small
59,8/69.1
335/354
46.4
67.0
31,0
60.3
48.5
70.2
53.5
43.3
67.3
46.6
Focal-Small (Ours)
61.7/71.2
367/401
47.3
67.8
51.0
31,6
50.9
61.1
48.8
70.5
53.6
43.8
67.7
47.2
ResNeXt101 64x--4444dd
67
95,5/102
473/493
41.8
61.5
44.4
25.2
45.4
54.6
44.4
64.9
48.8
39.7
61.9
42.6
aare- 60
71.1/81.0
345/364
43.4
63.6
46.1
26.1
46.0
59.5
44.5
66.0
48.3
40.7
63.4
43.7
ViL-Base 7G
66.7/76.1
443/365
44.7
65.5
47.6
29.9
48.0
58.1
45.7
67.2
49.9
41.3
64.4
44.5
Swin-Buse 45
98,4/107
477/496
45.8
66.4
49.1
29.9
49.4
60.3
48.5
69.8
53.2
43.4
66.8
46.9
Focal-Base (Ours)
100.8/110.0
514/533
46.9
67.8
50.3
31.9
50.3
61.5
49.0
70.1
53.6
43.7
67.6
47.0
Table 4: COCO object detection and segmentation results with RetinaNet [41 and Mask R-CNN [33
All models are trained with 3x schedule and multi-scale inputs (MS). The numbers before and after "I" at column 2 and 3 are the model size and complexity for RetinaNet and Mask R-CNN, respectively.
Focal Transformers show substantial gains across all settings and metrics. Particularly, Focal Transformers brings 0.7-1.7 points of mAP against the current best approach Swin Transformer 443 at comparable settings. Different from the other multi-scale Transformer models, Focal Transformers can simultaneously enable short-range fine-grain and long-range coarse-grain interactions for each visual token, and thus capture richer visual contexts at each layer for better dense predictions. To have more comprehensive comparisons, we train all models using the 3 schedule and show the detailed numbers for RetinaNet and Mask R-CNN in T As we can see, even with the 3 schedule, Focal Transformers can still achieve 0.3-1.1 gain over Swin Transformer models in comparable settings.
Comparison with large SoTA detection models. We follow Swin Transformers to use HTC [1]] as the detection method in that it reported SoTA performance COCO detection when using Swin Transformer as the backbone. For fair comparison, we also use soft-NMS [5]. instaboost [24] and a multi-scale training strategy with the shorter side in range [400, 1400] and the longer side no more than 1600. We train the model using Adam W [44) with base learning rate Ic-4 and weight decay 0.1. The model trained using the standard 3x schedule. The box and mask mAPs on COCO validation set and test-dev are reported in Table 5 where both single-scale evaluation and multi-scale evaluation results are presented. Our Local-Large model with multi-scal, test achieves 58.1 box mAP and 50.9 mask mAP on mini-val set, which is better than the reported numbers for Swin-Large in [4). When evaluating our model on the test-dev set, achieves 58 4 box mAP and 51.3 mask mAP, which is slightly better than Swin Transformer. Note that because our model does not include the global self-attention layer used in Swin Transformer at the last stage, it has a smaller model size and fewer FLOPs. More recently, DyHead [17 achieves new SoTA on COCO, when combined with Swin-Large. We replace the Swin-Large model with the Foral-Large model, and use the same 2 training schedule as in [17]. We report the box mAPs for both mini-val and test-dev. Focal-Large achieves 58.7 and 59.0 on mini-val and test-dev, respectively.
4.3 Semantic Segmentation
In addition to the instance segmentation results, we also evaluate our models the semantic segmentation task which usually takes high-resolution input images and requires capturing long range interactions. We benchmark our methods ADE20K [83. We use UperNet [65] as the segmentation method and Focal Transformers as the backbones. We train three models as Focal-Tiny, Focal-Small, Focal-Base, respectively. For all the models, we 512 x 512 and trains the model for 160k iterations with batch size 16. Table the comparison results. We see that Focal-Tiny, Focal-Small and Focal-Base models consistently outperform Swin Transformers of the similar size in single-scale and multi-scale mloUs.
standard recipe that sets the input size to
Comparison with large SoTA semantic segmentation models. We use the pretrained Focal-Large model as the backbone for semantic segmentation. Follow the setting in 43 we use input image size 640 640 and train the model for 160k iterations with a batch size of 16. We set the initial learning to 5 and use polynomial learning rate decay. The weight decay is set to 0.01. For
8

---

Backhone
*Param FLOPss
1MS
Method
Parrm FLOPs
mini val
lest-dev
RcsNet-10
DANet 45
69M
1119
45,3
AP
Apm
AP Apm
ResNet-101
ACNot
ResNet-101
DNL 69
69M
1249G
46.0
X101-64x48
57
155M
1033G
52.3
46.0
ResNet-101
UperNet
65
86M
10296
44,9
EllicientNet D7
77M
410G
54.4
55.1
GCNet x
1041G
51.8
44.7
52.3
45,4
HRNas-w-48
53
OCKNer
73
7IM
664G
45.7
ResNeSt-200 75
52.5
53.3
47.1
ResNeSt-200
h5
DLabv3
12
88M
1381G
48.4
Copy-poste
28
185M
1440G
55.9
47.2
56.0
47.4
Swin-T
UperNet
945G
45.8
BoTNNet-200 51
49.7
Swin-S +
UparNet
63
81M
105KG
47.6
-9..5
SpineNet-190 22
164M
18850
52.6
52.8
Swin-B 4
UperNet
63
121M
1188G
48.1
-19.7
CenterNe12 [84
56.4
Twins-VV.
UperNet
50.2
MiT-H5
M6
NegHormer
M
85M
51.0
51.8
Swin-L,
284M
14700
57.7
50.2
VTFL/16
308M
(wyn.L,(DyIlead))
1k
213M
965G
56.2
Swin-1.
[4
UperNer KS
234M
32300
52.1
53.5
Swin-L" (HIC++)
284M
58.0
50.4
58.7
51.1
VIT-L/16
Segmenter
334M
51.8
53.6
Swin-L' (DyHeadd
17
213M
58.7
Swin-L
13
K-Net
51.3
Swin-L' (QueryInst)
25
56.1
56.1
Swin [1
PutchDivense
29
234M
53.1
54.4
Focal-L. (ITCC++ (Ours)
265M
1165G
$7.0
49.9
VOLO-D5
72
UperNer
63
54.3
Focal-L (Dyal-L (Ours)
229M
1081G
Focal-T (Ours)
UperNer
9980
47.0
Focal-L (ITC++) (Ours)
265M
58.1
50.9
58.4
51.3
Focal-S (Ours)
UpatNet
63
85M
11306
48.0
50.0
Focal-L" (DyHcad) (Ours)
229M
58.7
59.0
Focal-8
UperNet
126M
13546
50.5
Focal-LL (Ours)
UparNet
65
240M
33766
54.0
55.4
Table 5: Comparison with state-of-the-ard methods on COCO object detection and instance segmen- tation. The numbers are reported on 5K val set and test-dev. Augmented HTC [I]] (denoted by HTC++) and DyHead [17 are used as the detec- tion methods. means multi-scale evaluation.
Table 6: Comparison with SoTA methods for semantic segmentation on ADE20K [83] val set. Single- and multi-scale evaluations are reported in the last two columns. means ImageNet- is used as the pretrain- ing dataset.
Model
W-Size FLOPs Top-1 (%)
Aph
Apm
Model
W-Shift Top-1 (%)
AP
APT
4.5
81.2
43.7
39.8
80.2
38.8
36.4
Swin- Tiny
Swin-Tiny
14
4.9
82.1
44.0
40.5
81.2
43.7
39.8
4.9
82.2
44.9
41.1
82.2
44.8
41.0
Focal-Tiny
Focal-Tiny
14
5.2
82.3
45.5
41.5
81.9
44.9
41.1
Table 7: Impact of different window sizes (W- Size). We alter the default size 7 to 14 and ob- serve consistent improvements for both methods.
Table Impact of window shift (W-Shift) on Swin Transformer and Focal Transformer. Tiny models are used.
multi-scale evaluation, we use the same scaling ratios [0.5, 0.75, 1.0, 1.25, 1.5, 1.75] as in previous works. The results in Table that Local-Large achieves significantly better performance than Swin-Large. In both single-scale scgle and multi-scale evaluations, Focal-Large leads to more than point mIoU improvement, creating new SoTA for semantic segmentation ADE20K.
4.4 Ablation studies
We conduct a series of ablation studies to inspect the model's capacity from different aspects. We use Focal-Tiny and the image classification and object detection tasks.
Effect of varying the window size. We have demonstrated that is crucial model both short- and long-range interactions. Thus, related question is whether increasing the window size helps as it leads to a larger receptive field. Table 7 the performance of Swin-Tiny and Focal-Tiny with window sizes 7 and 14. Clearly, a larger window size is beneficial for both methods measured in all three metrics, and Focal-Tiny consistently outperforms Swin-Timy Tiny both window sizes. Comparing the second and third row, we find that Focal-Tiny outperforms S Swin-Tiny even with a smaller window size (7 v.s. 14). We suspect that the gain is attributed to our focal attention's 's superior capability of capturing rongerange dependencies among visual tokens.
The necessity of window shift. In Swin Transformer 43), window shift is proposed capture cross- window interactions between two successive layers. In contrast, visual tokens in Focal Transformers can always communicate with each other across windows both fine- and coarse-grain. Thus, is interesting to investigate whether adding window shift to Focal Transformers can lead to any improvement. To answer the question, we remove window shift from Swin Transformer while adding it to Focal Transformers, As shown in Table 8 Swin Transformer shows a severe degradation after removing the window shift. However, adding window shift to Focal Transformer hurts classification performance. The result indicates that window shift is unnecessary for Focal Transformers. While in

---

Depths Model #Params. FLOPs
Top-1 (%)
AP AP"
Tops Arcumey (8g
44.8
2-2-2-2
Swin
21.2
3.1
78.7
38.2
35.7
21.7
3.4
79.9
40.5
37.6
60.1
bbon
2-2-4-2
Swin
24.7
3.8
80.2
41.2
38.1
38.8
Focal
25.4
4.1
81.4
43.3
39.8
Swin
28.3
4.5
81.2
43.7
39.8
2-2-6-2
Focal
29.1
4.9
82.2
44.8
41.0
+Loccl
+Siebal
-Lacol+Global
Figure 5: Ablating Focal-Tiny model by adding local, global and both interactions, respectively. Blue bars are image classification results and orange bars object detection results. This figure better viewed in color.
Table 9: Impact of the change of model depth. We gradually reduce the number of transformer layers the third stage from original to 4 and further 2. Our Focal Transformers has much slower drop rate than Swin Transformer.
Swin Transformers, there should always be even number of layers in each stage for the alternative window shift operation, Focal Transformers do not have such a constraint.
Contributions of local and global interactions. To investigate the relative contributions of capturing local fine-grain and global coarse -gaiin interactions in Focal Transformers, we have developed several variants of Focal-Ting -iny: a) Focal- Tiny- Siny-Window merely performs attention inside each window; b) Focal-Tiny-Locas attends the additional ame-grain surrounding tokens and c) Focal-Tiny -lobbal attends the extra coarse-grad -aarin summarized tokens. We train these models using the same setting as Focal-Tiny and report their performance on image classification and object detection using Mask R-CNN 1 x schedule. As shown in Fig. 5
Tiny-Window suffers from a significant performance drop on both image classification (82.2 -0.1) and object detection (44.8 +-3.3)....3...... This is expected since the communication across windows is completely cut off at each Transformer layer. After we enable either the local fine-grain or global coarse-grain interactions (middle two columns), we observe significant performance boost. When we combine short- and long-range interactions, we observe additional improvements on both tasks. This implies that these two type of interactions are complementary and both are beneficial model performance.
Model capacity against model depth. Focal attention allows Transformer model capture short- and long-range interactions at each Transformer layer. An interesting question is whether Focal Transformers need fewer layers to obtain similar modeling capacity as the Transformer models that does not use focal attention, such as Swin Transformer. To answer this question, we conduct an experiment by training a series of Swin-Tiny and Focal-Tiny models by varying the number of Transformer layers at stage 3. As shown in T Focal-Tiny outperforms Swin-Tiny consistently with the same depth. More importantly. using lewer layers, Focal-Tiny can sometimes achieve comparable even better performance than Swin Transformer. For example, Focal-Ting with (2-2-4-2) achieves 81.4 on image classification which is better than Swin-Tiny with (2-2-6-2).
5 Conclusion
In this paper, we have presented a new focal attention mechanism that enables efficient long-range interactions in Vision Transformers. Different from previous works, it performs the local attention at fine-grain and global attention at coarse-grain, providing an effective way of capturing both short- and long-range context with a manageable computational cost. By applying focal attention into multi-scale Transformer architecture, we propose Focal Transformers as general-purpose backbones for a wide range of dense vision tasks. A comprehensive empirical study shows that our Focal Transformers outperform the SoTA Vision Transformers on a range of vision tasks including image classification, object detection and segmentation.
Limitations and future work. Although our experiments show that focal attention can significantly boost the performance on image classification and dense prediction tasks, focal attention does introduce extra computational and memory cost, since each query token needs to attend more (summarized) tokens in addition to tokens inside window. A cost-effective implementation of Focal Transformer is necessary to make it more applicable to many real-world scenarios. This study focuses on incorporating focal attention into multi-scale Vision Transformers for CV tasks. However, notice that focal attention is an effective sparse attention mechanism that is widely applicable to all attention- based neural network models that are developed for processing natural language, images, videos etc. This is an exciting future direction.
10

---

References
[1] Joshua Ainslie, Santiago Ontanon, Chris Alberti, Philip Pham, Anirudh Ravula, and Sumit Sanghai. Ete: Encoding long and structured data in transformers. arXiv preprint arXiv:2004. 08483, 2020.
[2] Irwan Bello, Barret Zoph, Ashish Vaswani, Jonathon Shlens, and Quoc V Le. Attention augmented convolutional networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages
2019.
[3] Iz Beltagy, Matthew Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint XrXiv:2004.05150 2020.
[4] Maxim Berman, Hervé Jégou, Andrea Vedaldi, lasonas Kokkinos, and Mathijs Douze. Multigrain: a unified image embedding for classes and instances. arXiv preprini arXiv:1902.05509, 2019.
[5] Navanceth Bodla, Bharat Singh, Rama Chellappa, and Larry Davis. Soft-ams-improving object detection with one line of code. In Proceedings of the IEEE international conference on computer vision, pages 5561 55569, 2017.
[6] Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Ncelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are fow-shot leamerrs. arXiv preprint arXiv:2005. 14165, 2020.
[7] Yue Cao, Jiarui Xu, Stephen Lin, Fangyun Wei, and Han Hu, Genet: Non-local networks meet squeeze- excitation networks and beyond. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, pages 0-0, 2019.
[8] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European Conference on Computer Vision, pages 229. Springer, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties sclf-super vision transformers. arXiv preprint arXiv:2104, 1eeee004.444444 14294, 2021.
[10] Shuning Chang. Pichao Wang, Fan Wang. Hao Li, and Jiashi Feng. Augmented transformer with adaptive graph for temporal action proposal generation. arXiv preprint arXiv:2 103. /6024, 2021.
[11] Kai Chen, Jiangmiao Pang, Jiaqi Wang, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jianping Shi, Wanli Ouyang, et al. Hybrid task cascade for instance segmentation. In Proceedings of the IEFE/CVF Conference on Computer Vision and Puttern Recognition, pages 4974-4983, 2019.
[12] Liang- Long-Chich Chen, Yukun Zhu, George Papandrcou, Florian Schroff, and Hartwig Adam. Encoder-decode with atrous separable convolution for semantic image segmentation. In Proceedings of the European conference on computer vision (ECCV), pages 801-818, 2018
[13] Xin Chen, Bin Yan, Jiawen Zhu, Dong Wang, Xiaoyun Yang. and Huchuan Lu. Transformer tracking. arXiv preprint arXiv:2 15436, 2021.
[14] Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self sulf-super visual transformers. arXiv preprint arXiv:2104.02057. 2021.
[15] Xiangxiang Chu, Zhi Tian, Yuqing Wang, Bo Zhang, Haibing Ren, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Twins: Revisiting spatial attention design in vision transformers. arXiv preprint arXiv:2 13840, 2021.
[16] Xiangxiang Chu. Zhi Tian, Bo Zhang. Xinlong Wang, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Conditional positional encodings for vision transformers. Arxiv preprint 2102.10882. 2021.
[17] Xiyang Dai, Yinpeng Chen, Bin Xiao, Dongdong Chen, Mengchen Liu, Lu Yuan, and Lei Zhang. Dynamic head: Unifying object detection heads with attentions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7373 7382, 2021.
[18] Zhigang Dai, Bolun Cai, Yugeng Lin, and Junying Chen. Up-detr: Unsupervised pre-training for object detection with transformers. arXiv preprint arXiv:20/1.000 1,09094, 2020.
[19] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages Ieee, 2009.
11

---

[20] Jacob Devlin, Ming-Wo Wei Chang. Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. NAACL, 2019.
[21] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghami, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words:
Transformers for image recognition scale. arXiv preprint arXiv:2010. //929, 2020.
[22] Xianzhi Du, Tsung- Yi Lin, Pengchong Jin, Golnaz Ghiasi, Mingxing Tan, Yin Cui, Quoe V Le, and Xiaodan Song. Spinenet: Learning perlepermuted backbone for recognition and localization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 11592 1 11601, 2020.
[23] Haoqi Fan, Bo Xiong. Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale vision transformers. arXiv preprint arXiv:2 1/227, 2021.
[24] Hao-Shu Fang, Jianhua Sun, Runzhong Wang, Minghao Gou, Yong-Lu Li, and Cewu Lu. Instaboost: Boosting instance segmentation via probability map guided copy-pasting In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 882-691, 2019.
[25] Yuxin Fang, Shusheng Yang. Xinggang Wang. Yu Li, Chen Fang, Ying Shan, Bin Feng, and Wenyu Liu. Instances queries, 2021.
[26] Jun Fu, Jing Liu, Yuhang Wang, Yong Li, Yongjun Bao, Jinhui Tang, and Hanqing Lu. Adaptive context network for scene parsing. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 6778-6757, 2019.
[27] Jianfeng Gao, Patrick Pantel, Michael Gamon, Xiaodong Hc, and Li Deng. Modeling interestingness with deep neural networks. In Proceedings of the 2014 Conference Empirical Methods in Natural Language Processing (EMNLP), pages 2-13, Doha, Qatar, October 2014. Association for Computational Linguisties.
[28] Golnaz Ghiasi, Yin Cui, Aravind Srinivas, Rui Qian, Tsung-Yi Yi Lin, Ekin D Cubuk, Quoe V Le, and Barret Zoph. Simple copy-paste is strong data augmentation method for instance segmentation. arXiv preprint mmXiv:2012.07177. 2020.
[29] Chengyue Gong, Dilin Wang, Meng Li, Vikas Chandra, and Qiang Liu. Vision transformers with patch diversification, 2021.
[30] Kai Han, Yunhe Wang, Hanting Chen, Xinghao Chen, Jianyuan Guo, Zhenhua Liu, Yehui Tang, An Xiao, Chunjing Xu, Yixing Xu, et al. A survey visual transformer. arXiv preprint 20Xx::2012.12556, 2020.
[31] Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, and Yunhe Wang. Transformer in transformer, 2021.
[32] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 2961-2969, 2017.
[33] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[34] Elad Hoffer, Tal Ben-Nun, Itay Hubara, Niv Giladi, Torsten Hoefler, and Daniel Soudry. Augment your batch: Improving generalization through instance repetition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8129-8138, 2020.
[35] Jie Hu, Li Shen, and Gang Sun. Sgueeze-and-cxcitate networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7132-7141, 2018
[36] Salman Khan, Muzammal Nascer, Munawar Hayat, Syed Waqas Zamir, Fahad Shahbaz Khan, and Mubarak Shah. Transformers in vision: survey. arXiv preprint arXiv:2101.01100 2021.
[37] Yann LeCun, Yoshua Bengio, al. Convolutional networks for images, speech, and time series. The handbook of brain theory and neural networks, 3361 1995.
[38] Bing Li, Cheng Zheng, Silvio Giancola, and Bernard Ghanem. Setn: Sparse convolution-transfore network for scene flow estimation. arXiv preprint SXXx:2105.04444, 2021.
[39] Chunyuan Li, Jianwei Yang, Pengchuan Zhang, Mci Gao, Bin Xiao, Xiyang Dai, Lu Yuan, and Jianfeng Gao. Efficient self-supervised vision transformers for representation learning. arXiv preprint 206. 2021.
[40] Xiangyu Li. Yonghong Hou, Pichao Wang, Zhimin Gao, Mingliang Xu, and Wanqing Li. Trear: Transformer based rgb-d egocentric action recognition. arXiv preprint 2rXiv:2101.03900 2021.
12

---

[41] Tsung- Yi Lin, Priya Goyal, Ross Girshick. Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings the IEEE international conference on computer vision, pages 2980-2988. 2017.
[42] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zimick. Microsoft COCO: Common objects in context. In ECCV, 2014.
[43] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang. Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. arXiv preprint arXiv:2103.14030, 2021.
[44] Ilya Loshchilov and Frank Hutter. Decoupled 1.05:1771.05101, 2017.
weight decay regularization. arXiv preprint
[45] Hyconscob Nam, Jung-Woo Ha, and Jeonghee Kim. Dual attention networks for multimodal reasoning and matching. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 299-307, 2017.
[46] R. Pappagari, P. Zelasko, J. Villalba, Y. Carmiel, and N. Dehak. Hierarchical transformers for long document classification. In 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 338-844,22 2019.
[47] Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In International Conference Machine Learning, pages 4055-4064. PMLR, 2018
[48] Boris T Polyak and Anatoli B Juditsky. Acceleration of stochastic approximation by averaging. SIAM journal control and optimization, 30(4)8838-855, 1992.
[49] Jack W Rae, Anna Potapenko, Siddhant M Jayakumar, and Timothy P Lillicrap. Compressive transformers for long-range sequence modelling. arXiv preprint arXiv:1911 05507, 2019.
[50] Prajit Ramachandran, Niki Parmar, Ashish Vaswani, Irwan Bello, Anselm Levskaya, and Jonathon Shlens. Stand-alone self-attention in vision models. arXiv preprint arXiv: 19xiv:996.00990 2019.
[51] Aravind Srinivas, Tsung- Yi Lin, Niki Parmar, Jonathon Shlens, Pieter Abbeel, and Ashish Vaswani. Bottleneck transformers for visual recognition, 2021.
[52] Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid. Segmenter: Transformer for semantic segmentation. arXiv preprint arXiv:2105 05633, 2021.
[53] Sun, Bin Xiao, Dong Liu, and Jingdong Wang. Deep high-resolution representation learning for human pose estimation. In CVPR, 2019.
[54] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning, pages 6114. PMLR, 2019.
[55] Hugo Touvron, Matthicu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. arXiv preprint 12Xiv.2012.12877. 2020.
[56] Ashish Vaswani, Prajit Ramachandran, Aravind Srinivas. Niki Parmar, Blake Hechtman, and Jonathon Shlens. Scaling local self-attention for parameter efficient visual backbones. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12894-12904, 2021.
[57] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Fukasz Kaiser, and Illia Polosukhin. Attention all you need. NeurlPS, 2017.
[58] Huiyu Wang. Yukun Zhu, Hartwig Adam, Alan Yuille, and Liang-Chieh Chich Chen. Max Max-deeplab: End-to-end panoptic segmentation with mask transformers. arXiv preprint 20X7::20/2.00755, 2020.
[59] Ning Wang, Wengang Zhou, Jie Wang, and Houqaing Li. Transformer meets tracker: Exploiting temporal context for robust visual tracking. arXiv preprint arXiv:2103.1 1 11611, 2021.
[60] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. arXiv preprint arXiv:2102. 2021.
[61] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Nocalocal neural networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 2018.
13

---

[62] Yuqing Wang. Zhaoliang Xu. Xinlong Wang. Chunhua Shen, Baoshan Cheng, Hao Shen, and Huaxia Xia. End-to-end video instance segmentation with transformers. arXiv preprint arXiv:2011.14500 2020.
[63] Sanghyun Woo, Jongchan Park, Joon-Young Lee, and In So Kweon. Cbam: Convolutional block attention module. Proceedings of the European conference on computer vision (ECCV), pages 3 19, 2018.
[64] Haiping Wu, Bin Xiao, Noel Codella, Mengehen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. Cvt: Introducing convolutions to vision transformers. arXiv preprint arXiv:2103.15800 2021.
[65] Tete Xiao, Yingcheng Liu, Bolei Zhou. Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In Proceedings of the European Conference on Computer Vision (ECCV), pages 418 434, 2018.
[66] Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M. Alvarez, and Ping Luo, Segformer: Simple and efficient design for semantic segmentation with transformers, 2021.
[67] Saining Xie. Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transforma- tions for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recogmition, pages 1492-1500, 2017.
[68] Jianwei Yang, Zhile Ren, Chuang Gan, Hongyuan Zhu, and Devi Parikh. Cross-channed communication networks. In Proceedings of the 33rd International Conferencee on Neural Information Processing Systems, pages 1297-1306, 2019.
[69] Minghao Yin, Zhuliang Yao. Yue Cao. Xiu Li, Zheng Zhang, Stephen Lin, and Han Hu. Disentangled non-local neural networks. In European Conference on Computer Vision, pages 191-207. Springer, 2020.
[70] Kun Yuan, Shaopeng Guo, Ziwei Liu, Aojun Zhou, Fengwei Yu, and Wei Wu. Incorporating convolution designs into visual transformers. arXiv preprint arXiv:2103.11816 2021.
[71] Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Francis EH Tay, Jiashi Feng, and Shuicheng Yan.
vit: Training vision transformers from scratch imagenet. arXiv preprint arXiv:2101.1 2021.
[72] Li Yuan, Qibin Hou, Zihang Jiang. Jiashi Feng. and Shuicheng Yan. Volo: Vision outlooker for visual recognition. arXiv preprint vrXiv:2106.13111 2021.
[73] Yuhui Yuan, Xilin Chen, and Jingdong Wang. Object-contextual representations for semantic segmentation. arXiv preprint arXiv:1909.1 /009./ 1005,, 2019.
[74] Manzil Zaheer, Guru Guruganesh. Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big bird: Transformers for longer sequences. arXiv preprint arXiv:2007.1406; 2020.
[75] Hang Zhang, Chongruo Wu, Zhongyue Zhang, Yi Zhu, Haibin Lin, Zhi Zhang, Yue Sun, Tong Hc, Jonas Mueller, Manmatha, et al. Resnest: Split-attention networks. arXiv preprint arXiv:2004. 08955,, 2020.
[76] Pengehuan Zhang, Xiyang Dai, Jianwei Yang, Bin Xiao, Lu Yuan, Lei Zhang, and Jianfeng Gao. Multi- scale vision longformer: A new vision transformer for high-resolution image encoding. arXiv preprint arXiv:2103.1535, 2021.
[77] Wenwei Zhang. Jiangmiao Pang, Kai Chen, and Chen Change Loy. K-net: Towards unified image segmentation, 2021.
[78] Jiaojiao Zhao, Xinyu Li, Chunhui Liu, Shuai Bing, Hao Chen, Cees GM Snoek, and Joseph Tighe. Tuber: Tube-transformer for action detection. arXiv preprint 1.Xiv:210400090 2021.
[79] Minghang Zheng, Peng Gao, Xiaogang Wang, Hongsheng Li, and Hao Dong. End-to-end object detection with adaptive clustering transformer. arXiv preprint arXiv:201 100001,,, 2020.
[80] Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip HS Torr, Rethinking semantic segmentation from scquenee-to-sequeee perspective with transformers. arXiv preprint arXiv:2012.15880 2020.
[81] Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang. Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip HS Torr et al. Rethinking semantic segmentation from a perspective with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2021.
14

---

[82] Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmentation. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pages 13001-13008, 2020.
[83] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralha. Scene parsing through ade20k dataset. In Proceedings the IEEE conference computer vision and paitern recognition. pages 333-641,220 2017.
[84] Xingyi Zhou, Vladlen Koltun, and Philipp Krähenbühh. Probabilistic two-stage detection. arXiv preprint MrXiv:2103.07461 2021.
[85] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-ene object detection. arXiv preprint urXiv:2010.04159, 2020.
15


Reviews:
[{"review_id": "tdi1e2bnUm8", "reviewer": "Reviewer_rkny", "summary": "This paper presents an efficient self-attention mechanism for vision transformer, which simultaneously capture both short and long-range visual dependencies by making the query to attend patches at different granularity levels. The authors demonstrate its performance on image classification and object detection.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality:\n- The idea of extracting visual features using multiple granularity levels is well-established. Previous work has also tried to build a hierarchy of features within vision transformer (e.g., PVT). However, it is novel to build a single self-attention head which can attend multiple granularity levels.\n\nQuality:\n- The submission is technically sound.\n\nClarity:\n- The paper is well written.\n\nSignificance:\n- The empirical results are compelling. \n\nDetailed comment:\n\n1,  In section 3.1, a “patch embedding layer which consists of a convolutional layer with filter size and stride both equal to 4” could not transform input $\\frac{H}{4} \\times \\frac{W}{4} \\times (4\\times4\\times3)$ to output $\\frac{H}{4} \\times \\frac{W}{4} \\times d$.\n\n2,  In Table 2, Focal-Base does not achieve better result than Focal-Small. Do you think it’s due to the limited set of hyperparameters that has been tried, or it’s due to the strong induction bias within focal attention may hurt the generalization ability of big model?\n\n3, What is the criterion to organize Table 2? CvT-21 should be placed along with Focal-Tiny.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "ecpEKMw12k", "reviewer": "Reviewer_B1T8", "summary": "This paper proposes a focal version of self-attention module for vision transformers. The query is kept at high resolution, but the number of keys is reduced depending on the relative position difference. This proposed attention is implemented within multi-scale windows. It shows good performance on top of recent transformer advances, on ImageNet and COCO.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strength\n1. The proposed focal attention mechanism is original and novel.\n2. The idea is clearly presented with equations and illustrations.\n3. Training details included for fair comparison and reproducing. Comparison is fair with other methods.\n4. Various settings and tasks are explored and compared with solid baselines.\n\n## Weakness\n1. Runtime / throughput is not clear. The proposed attention requires extra subsampling layers, key value computation, and more importantly, concatenating keys and values from different resolution windows (according to the code). These might introduce extra runtime cost with the same FLOPs. But this is not sufficiently discussed in the paper.\n2. Scaling to larger models. The ImageNet performance of Focal-Base is the same as Focal-Small, and is similar to Swin with the same size. In addition, the improvement over Swin in the Small regime is limited as well. In this case, error bars may be reported to address result variances. However, Focal-Base improves consistently on detection segmentation tasks, given a limited ImageNet Top-1. Does it suggest that more context (larger focal size) is necessary for COCO than ImageNet? (Relates to the discussion in Line 281).\n3. What is the intuition behind sharing the same set of query key parameter for both small and large windows? It does save parameters and FLOPs, but the model might be extracting local edges/corners with fine detail and global semantic concepts with coarse context windows. This seems to suggest that different (not shared) parameters might be good for fine and coarse levels. It might be worth ablating that three independent attention could be used for the three levels.\n4. The training recipe seems modified slightly from DeiT/Swin, e.g. the drop path rate. Does it affect comparison fairness?\n\n------------------------------------------------------------Post Rebuttal-------------------------------------------------------------------\n\nAfter reading the author rebuttal and other reviews, I think the weakness 2 and 4 are mostly addressed. Nevertheless, the runtime/throughput is a confirmed weakness according to the authors' results in the rebuttal and is also a shared concern among reviewers. But I think this is ok given other contribution of the paper. In this case, I keep my original rating of 6.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "OaEv1g7ri8", "reviewer": "Reviewer_HEkP", "summary": "This paper proposes Focal Transformer, which captures both local and global attention based on their design. It also follows the pyramid architecture so their model can be easily extended to detection and segmentation tasks. The result on image classification detection and segmentation are very promising. ", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, the main contribution of this paper is proposing a new attention module for multi-level trasnformer, which seems efficient and can capture both local and global information. Then the architecture heavily follows PVT and Swin Transformer.  \n\nAbout performance, it is slightly better than Swin Transformer, which is reasonable because swin do not have global attention.\n\nAdvantages:\n\n1. clear paper writing, which is easy to follow.\n\n2. strong performance on several tasks.\n\n3. reasonable story to design an efficient self-attention but consider both global and local region.\n\nWeakness:\n\n1. I have some concerns about the inference speed of this method. If I have a misunderstanding please correct me. Mainly because calculating focal attention seems not very straightforward. I understand the theoretical computation complexity is efficient but it is not clear of real running speed. **It is necessary to report the speed and compare with other methods** such as Swin and PVT, especially on object detection and segmentation which need a high-resolution image as input.\n\n2. In Table2 when scaling model size from small to base, the accuracy stops to increase. So I wonder does this model's performance upper bound is lower than other methods? Because ViT can consistently improve the performance when enlarging the model size even the model size is super large.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "GskYaL9dv2F", "reviewer": "Reviewer_NEKg", "summary": "This paper proposed a new attention mechanism called focal attention to enable efficient long-range interactions in vision transformer. By plugging the proposed module into a multi-scale vision transformer model,  a new focal transformer was designed and its superiority was demonstrated over the state-of-the-art approaches on both image classification and object detection tasks.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. This paper is well written, and has no obvious mistakes.\n2. For a vision transformer model, how to reduce the computational overhead while keeping ability of capturing the long-range dependencies is still an important and interesting research topic. \n3. The experiments are conducted on various benchmarks on image classification and object detection, with extensive ablation studies.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "1. There are limited discussions on model efficiency. I suggest the authors to add more experimental comparisons with Sparse Transformer or some other efficient transformer methods. \n2. The focal windows and region partition operations do not bring any additional flops but may be time-consuming.  It should better add throughoutput or latency results in Table2.\n3. Figure 4 shows three fine to coarse attention levels with short and long range interaction.  However, in the actual model configuration the attention only has two level and mismatch Figure4. Why dose this happen?  The authors should do ablation study about the number of level including acc and latency. \n4. In table 4 stage1-2, s^1_{w} * s^1_{r} is larger than input resolution, e.g, 7x5>28. How do you handle this situation. ", "review_text": "", "extra_scores": {"ethical_concerns": "NO", "needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "2h"}, "extra_text": {}}]

Author rebuttal:

Meta-review: There is a unanimous agreement on the novelty/solid results of the paper and the AC agree with the reviewers. The proposed idea in this paper is interesting, well-motivated, and could make good contributions to the general vision and representation learning communities. In this regard, the AC recommends acceptance as spotlight.
Decision: Accept (Spotlight)