Focal Attention for Long-Range Interactions in Vision Transformers
Year: 2022

Abstract:
Recently, Vision Transformer and its variants have shown great promise on various computer vision tasks. The ability of capturing local and global visual dependencies through self-attention is the key to its success. However, this also brings challenges due to quadratic computational overhead, especially for the high-resolution vision tasks (e.g., object detection). Many recent works have attempted to reduce the cost and improve model performance by applying either coarse-grained global attention or fine-grained local attention. However, both approaches cripple the modeling power of the original self-attention mechanism of multi-layer Transformers, leading to sub-optimal solutions. In this paper, we present focal attention, a new attention mechanism that incorporates both fine-grained local and coarse-grained global interactions. In this new mechanism, each token attends its closest surrounding tokens at fine granularity and the tokens far away at coarse granularity, and thus can capture both shortand long-range visual dependencies efficiently and effectively. With focal attention, we build a new variant of Vision Transformer models, called Focal Transformers, which achieve superior performance over the state-of-theart (SoTA) Vision Transformers on a range of public image classification and object detection benchmarks. In particular, our Focal Transformer models with a moderate size of 51.1M and a large size of 89.8M achieve 83.6% and 84.0% Top-1 accuracy, respectively, on ImageNet classification at 224 × 224. When employed as the backbones, Focal Transformers achieve consistent and substantial improvements over the current SoTA Swin Transformers [43] across 6 different object detection methods. Our largest Focal Transformer yields 58.7/59.0 box mAPs and 50.9/51.3 mask mAPs on COCO mini-val/test-dev, and 55.4 mIoU on ADE20K for semantic segmentation, creating new SoTA on three of the most challenging computer vision tasks. Our code is available at: https://github. com/microsoft/Focal-Transformer.

text: Recently, Vision Transformer and its variants have shown great promise on various computer vision tasks. The ability of capturing local and global visual dependencies through self-attention is the key to its success. However, this also brings challenges due to quadratic computational overhead, especially for the high-resolution vision tasks (e.g., object detection). Many recent works have attempted to reduce the cost and improve model performance by applying either coarse-grained global attention or fine-grained local attention. However, both approaches cripple the modeling power of the original self-attention mechanism of multi-layer Transformers, leading to sub-optimal solutions. In this paper, we present focal attention, a new attention mechanism that incorporates both fine-grained local and coarse-grained global interactions. In this new mechanism, each token attends its closest surrounding tokens at fine granularity and the tokens far away at coarse granularity, and thus can capture both short- and long-range visual dependencies efficiently and effectively. With focal attention, we build a new variant of Vision Transformer models, called Focal Transformers, which achieve superior performance over the state-of-theart (SoTA) Vision Transformers on a range of public image classification and object detection benchmarks. In particular, our Focal Transformer models with a moderate size of 51.1M and a large size of 89.8M achieve 83.6% and 84.0% Top-1 accuracy, respectively, on ImageNet classification at 224 × 224. When employed as the backbones, Focal Transformers achieve consistent and substantial improvements over the current SoTA Swin Transformers [43] across 6 different object detection methods. Our largest Focal Transformer yields 58.7/59.0 box mAPs and 50.9/51.3 mask mAPs on COCO mini-val/test-dev, and 55.4 mIoU on ADE20K for semantic segmentation, creating new SoTA on three of the most challenging computer vision tasks. Our code is available at: https://github. com/microsoft/Focal-Transformer.

heading: 1 Introduction
text: Nowadays, Transformer [57] has become a prevalent model architecture in natural language processing (NLP) [20, 6]. In the light of its success in NLP, there is an increasing effort on adapting it to computer vision (CV) [47, 50]. Since its promise firstly demonstrated in Vision Transformer (ViT) [21], we have witnessed a flourish of full-Transformer models for image classification [55, 60, 64, 43, 76, 56], object detection [8, 85, 79, 18] and semantic segmentation [58, 62]. Beyond these static image tasks, it has also been applied on various temporal understanding tasks, such as action recognition [40, 78, 10], object tracking [13, 59], scene flow estimation [38].
The self-attention mechanism is arguably the key component that differentiates Transformers from the widely used convolutional neural networks (CNNs) [37] in computer vision. At each Transformer layer, self-attention enables global content-dependent interactions among different image regions for
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
modeling short- and long-range dependencies, respectively. Through the visualization of full selfattention results1, we indeed observe that self-attention learns to attend local surroundings (like CNNs) and the global contexts at the same time, as illustrated in Fig. 1 (Left). Nevertheless, when dealing with high-resolution vision tasks such as object detection or segmentation, an efficient implementation of a global and fine-grained self-attention becomes non-trivial due to the quadratic computational cost with respect to the number of tokens in feature maps. Recent works have alternatively exploited either a coarse-grained global self-attention [60, 64] or a fine-grained local self-attention [43, 76, 56], for the sake of reducing the computational cost. However, both approaches cripple the power of the original full self-attention i.e., the ability to simultaneously capture local and global visual dependencies.
In this paper, we present a new attention mechanism to capture both short- and long-range interactions in Transformer layers for high-resolution input images. Considering that the visual dependencies between the nearby (local) regions are usually much stronger than the dependencies between the regions that are far away, we perform the fine-grained attention only in local regions while the coarse-grained attention globally. As depicted in Fig. 1 (Right), a query token in the feature map attends its closest local surroundings at the finest granularity as itself. However, when it goes to the regions far away, it attends to summarized tokens to capture coarse-grained visual dependencies. We call this new mechanism focal attention, as each token attends the others in a focal manner. We will show in this study that focal attention allows to effectively model visual dependencies among all regions covering the whole high-resolution feature maps while introducing much less number of tokens in the computation than that in the standard self-attention mechanism.
Equipped with focal attention, a series of Focal Transformers are developed and validated via a comprehensive empirical study across three core vision tasks, including image classification, object detection and segmentation. Results show that Focal Transformers consistently outperform the SoTA Vision Transformers across various settings (i.e., in model sizes and complexities). Notably, the small Focal Transformer with 51.1M parameters achieves 83.6% top-1 accuracy on ImageNet-1K, and the base model with 89.8M parameters obtains 84.0% top-1 accuracy. In the fine-tuning experiments for object detection, Focal Transformers consistently outperform the SoTA Swin Transformers [43] across six popular object detection methods. Our largest Focal Transformer model achieves 59.0 box mAP and 51.3 mask mAP on COCO test-dev for object detection and instance segmentation, respectively, and 55.4 mIoU on ADE20K for semantic segmentation. These results demonstrate that focal attention is highly effective in modeling the global interactions in Vision Transformers.

heading: 2 Related work
text: Vision Transformers. Vision Transformer (ViT) is first introduced in [21]. It applies a standard Transformer, originally developed for NLP [57], to encode an image by analogously splitting the image into a sequence of visual tokens. It has demonstrated superior performance to CNNs such as ResNet [33] on multiple image classification benchmarks, when trained with sufficient data [21] and carefully designed data augmentation and regularization methods [55]. The results thus inspire researchers to explore the applications of ViT on various vision tasks beyond image classification, such as self-supervised learning [14, 9, 39], object detection [8, 85, 79, 18] and semantic segmentation [58, 62, 81]. There are also increasing number of studies for improving ViT via data-efficient
1DeiT-Tiny model, checkpoint downloaded from https://github.com/facebookresearch/deit.
training [55], improved patch embedding/encoding [16, 71, 31], integrating convolutional projections into transformers [64, 70], and using multi-scale architectures and efficient self-attention mechanisms for high-resolution vision tasks [60, 64, 43, 76, 15]. Recent surveys include [36, 30, 36]. This paper focuses on improving the self-attention mechanism of ViT for encoding high-resolution images.
Efficient global and local self-attention. In many real-world tasks, Transformers need to cope with a large number of input tokens, such as long documents in NLP and high-resolution images in computer vision (CV). Recently, many efficient self-attention mechanisms have been proposed to deal with the quadratic computational and memory cost incurred by the standard self-attention mechanism. On one hand, a number of works in both NLP and CV resort to coarse-grained global self-attention (i.e., attending the down-sampled or summarized tokens) to capture the long-range interactions [49, 46, 60, 64, 31, 23]. Although this approach improves the model efficiency, it loses the detailed context information surrounding the query tokens. On the other hand, to make the computational cost manageable, various local fine-grained attention mechanism (i.e., attending neighboring tokens within a pre-set window size) are used for both NLP [3, 74, 1] and CV [56, 43, 76]. In this paper, we argue that both global and local attentions are important for model performance. This is also validated by some recent studies that aim to improve CNNs by incorporating ways of modeling global attentions [35, 63, 61, 68, 2, 7, 51]. The standard self-attention mechanism used by ViT can indeed learned both types of attentions, as shown in Fig. 1 (Left). But it often incurs a prohibitively high cost for high-resolution images. To the best of our knowledge, the proposed focal attention provides the first mechanism to incorporate local and global attention in a single Transformer layer 2. It can capture both short- and long-range interactions as standard self-attention but in a much more efficient and effective way, especially for high-resolution images.

heading: 3 Method

heading: 3.1 Model architecture
text: To accommodate high-resolution dense prediction tasks, we employ a multi-scale model architecture as in [60, 76, 43]. As shown in Fig. 2, an image I ∈ RH×W×3 is first partitioned into patches of size 4× 4, resulting in H4 × W 4 visual tokens with dimension 4× 4× 3. Then, we use a patch embedding layer, consisting of a convolutional layer with filter size and stride both equal to 4, to project these patches into hidden features with dimension d. We then pass this spatial feature map to the four stages of Focal Transformer blocks. In each stage i ∈ {1, 2, 3, 4}, the Focal Transformer block consists of Ni Focal Transformer layers. After each stage, we use a patch embedding layer to reduce the spatial size of feature map by factor 2 and increase the feature dimension by 2. For image classification tasks, we take the average of the output from the last stage and send it to a classification layer. For object detection, the feature maps from the last 3 or all 4 stages are fed to a particular object detector head,
2A similar focal mechanism has been used in CNNs for NLP [27].
depending on the specific detection method we choose to use. The model capacity can be customized by varying the input feature dimension d and the number of Focal Transformer layers.
Standard self-attention can capture both short- and long-range interactions at fine-grain, but suffers from high computational cost when it performs attention on high-resolution feature maps as noted in [76]. Take stage 1 in Fig. 2 as an example. For a feature map of size H4 × W 4 × d, the complexity of self-attention is O((H4 × W 4 )
2d), resulting in an explosion of time and memory cost, considering that min(H,W ) could be 800 or even larger for object detection. In the next section, we describe how we address this issue with the proposed focal attention mechanism.
3.2 Token-wise focal attention
Focal attention is proposed to make the Transformer layers suitable for encoding high-resolution input images. Instead of attending all tokens at fine-grain, we attend the fine-grain tokens only locally, but the summarized ones (i.e., the coarse-grained tokens generated by sub-window pooling, which is illustrated in Fig. 4 and will be described later) globally. As such, focal attention can cover the same amount of image regions as standard self-attention but with much less cost. In Fig. 3, we show the size of the receptive field for standard self-attention and our focal attention as a function of the number of attended tokens. For a given query position, by reducing the granularity of its surroundings based on their distance to the query, focal attention can have significantly larger receptive fields at the same cost measured by the number of visual tokens, compared to the standard self-attention mechanism.
Theoretically, the focal attention mechanism enables global interaction with much less time and memory cost, because it attends a much smaller number of surrounding (summarized) tokens. In practice, however, extracting the surrounding tokens for each query position could incur high time cost since we need to duplicate the extraction of each token for all queries that the token surrounds. This issue had been extensively discussed in [56, 76, 43] and a common solution is to partition the input feature map into windows. Thus, in our Focal Transformers, we resort to performing focal attention at the window level. We elaborate the window-wise focal attention in the following.

heading: 3.2.1 Window-wise focal attention
text: Given a feature map of x ∈ RM×N×d with spatial size M × N , we first partition it into a grid of windows of size sp × sp. Then, we extract the surroundings for each window rather than each individual token. The proposed window-wise focal attention is illustrated in Fig. 4. To clarify, we first define three terms:
• Focal level L refers to the granularity level at which we extract the tokens for focal attention. • Focal window size slw is the size of sub-window on which the summarized tokens are formed via
sub-window pooling at granularity level of l ∈ {1, ..., L}. • Focal region size slr denotes the number of sub-windows that are filled up horizontally (or verti-
cally) in an attended region at level l.
Now, we detail how window-wise focal attention works in the following two steps, sub-window pooling and attention computing.
Sub-window pooling. Consider input feature map x ∈ RM×N×d, where M × N is the spatial dimension and d the feature dimension. We perform sub-window pooling for all L levels. At focal level l, we first split the input feature map x into a grid of sub-windows with size slw × slw. Then we use a linear projection layer f lp to pool the sub-windows spatially by
xl = f lp(x̂) ∈ R M slw × N slw ×d , x̂ = Reshape(x) ∈ R ( M slw × N slw ×d)×(slw×s l w) . (1)
The pooled feature maps {xl}L1 at different levels l provide rich information at both fine-grain and coarse-grain. Since we set slw = 1 for the first focal level which has the same granularity as the input
feature map, there is no need to perform any sub-window pooling. Considering that the focal window size is usually very small (7 maximally in our settings), the number of extra parameters introduced by sub-window pooling is negligible.
Attention computing. Once we obtain the pooled feature maps {xl}L1 at all L levels, we compute the query at the first level, and key and value for all levels using three linear projection layers fq , fk and fv , respectively, as
Q = fq(x 1), K = {Kl}L1 = fk({x1, ..., xL}), V = {V l}L1 = fv({x1, ..., xL}). (2)
To perform focal attention, we need to first extract the surrounding tokens for each query token in the feature map. As mentioned earlier, tokens inside a window partition sp × sp share the same set of surroundings. For the queries inside the i-th window Qi ∈ Rsp×sp×d, we extract the slr × slr keys and values from Kl and V l surrounding the window which the query lies in, and then gather the keys and values from all L levels to obtain Ki = {K1i , ...,KLi } ∈ Rs×d and Vi = {V 1i , ..., V Li } ∈ Rs×d, where s is the sum of focal regions from all levels, i.e., s = ∑L l=1(s l r)
2. Note that a canonical implementation of focal attention following Fig. 1 requires to exclude the overlapped regions across different levels. In our implementation, we intentionally keep them in order to capture the pyramid information for the overlapped regions. Finally, we follow [43] to include a relative position bias and compute the focal attention for Qi by
Attention(Qi,Ki, Vi) = Softmax( QiK T i√ d +B)Vi, (3)
where B = {Bl}L1 is the learnable relative position bias. It consists of L subsets for L focal levels. Similar to [43], for the first level, we parameterize it as B1 ∈ R(2sp−1)×(2sp−1), considering that the horizontal and vertical position ranges are both in [−sp + 1, sp − 1]. For the other focal levels, considering that they have different granularity with respect to the queries, we treat all the queries inside a window equally and use Bl ∈ Rslr×slr to represent the relative position bias between the query window and each of slr × slr summarized tokens. Since the focal attention for each window can be performed independent of the others, we can compute Eq. (3) in parallel. Once we obtain attention scores for the whole input feature map, we send them to LayerNorm and the MLP block.

heading: 3.2.2 Complexity analysis
text: We analyze the computational complexity for the two steps of focal attention described above. For the input feature map x ∈ RM×N×d, we have M
slw × N slw sub-windows at focal level l. For each sub-window,
the pooling operation in Eq.1 has the complexity of O((slw)2d). Aggregating all sub-windows brings us O((MN)d). Then for all focal levels, we have the complexity of O(L(MN)d) in total, which is independent of the sub-window size at each focal level. Regarding the attention computation in Eq. 3, the computational cost for a query window sp × sp is O((sp)2 ∑ l(s l r) 2d), and O( ∑ l(s l r)
2(MN)d) for the whole input feature map. To sum up, the overall computational cost for focal attention is O((L+ ∑ l(s l r)
2)(MN)d). In an extreme case, one can set sLr = 2×max(M,N)/sLw to ensure a global receptive field for all queries (including both corner and middle queries) in this layer.

heading: 3.3 Model configurations
text: For fair comparison, we consider three network configurations for Focal Transformers, following [60, 64, 43]. Specifically, we follow the design of the Tiny, Small and Base models in Swin Transformer [43], as shown in Table 1. Our models take 224× 224 images as inputs and the window partition size is set to 7 to make our models comparable to Swin Transformers. For the focal attention layer, we introduce two levels, one for fine-grained local attention and the other for coarse-grained global attention. Except for the last stage, the focal region size is set to 13 for the window partition size of 7, which means that we expand 3 tokens for each window partition. For the last stage, since the whole feature map is 7× 7, the focal region size at level 0 is set to 7, which is sufficient to cover the entire feature map. For the coarse-grained global attention, we set its focal window size the same as the window partition size 7, but gradually decrease the focal region size to get {7, 5, 3, 1} for the four stages, respectively. For the patch embedding layer, the spatial reduction ratio pi for the four stages are all {4, 2, 2, 2}. Note that Focal-Base has a higher hidden dimension ci, compared to Focal-Tiny and Focal-Small.

heading: 4 Experiments

heading: 4.1 Image classification on ImageNet-1K
text: We compare different methods on ImageNet-1K [19]. For fair comparison, we follow the training recipes in [55, 60]. All models are trained for 300 epochs with batch size 1024. The initial learning rate is set to 10−3 with 20 epochs of linear warm-up starting from 10−5. For optimization, we use AdamW [44] as the optimizer with a cosine learning rate scheduler. The weight decay is set to 0.05 and the maximal gradient norm is clipped to 5.0. We use the same set of data augmentation and regularization strategies used in [55] after excluding random erasing [82], repeated augmentation [4, 34] and exponential moving average (EMA) [48]. The stochastic depth drop rates are set to 0.2, 0.2 and 0.3 for our tiny, small and base models, respectively. During training, we crop images randomly to 224× 224, while a center crop is used during evaluation on the validation set.
In Table 2, we summarize the results for baseline models and the state-of-the-art models on image classification task. We can see that Focal Transformers consistently outperform other methods with similar model sizes (#Params.) and computational complexities (GFLOPs). Specifically, Focal-Tiny improves over the Transformer baseline DeiT-Small/16 by 2.3%. Meanwhile, using the same model configuration (2-2-6-2) and a few extra parameters and computations, Focal-Tiny improves over Swin-Tiny by 1.0 point. For small and base models, Focal-Small with 51.1M parameters can reach 83.6% which is better than all the counterpart small and base models using much less parameters. By increasing the model size, Focal-Base model achieves 84.0%, surpassing all the other models with comparable parameters and FLOPs.
To compare with the large-scale models, we further build Focal-Large Transformer by increasing the hidden dimension in Focal-Base from 128 to 196 while keeping all the other hyperparameters the same. We follow the common practice to pretrain our Focal-Large Transformer on ImageNet-22K and transfer it to detection and segmentation tasks [64, 43].

heading: 4.2 Object detection and instance segmentation
text: We benchmark our models on object detection with COCO 2017 [42]. The pretrained models are used as visual backbones and then plugged into two representative pipelines, RetinaNet [41] and Mask R-CNN [32]. All models are trained on the 118k training images and the results are reported on 5K validation set. We use the two standard training schedules, 1× with 12 epochs and 3× with 36 epochs. For the 1× schedule, we resize image’s shorter side to 800 while keeping its longer side no more than 1,333. For the 3× schedule, we use the multi-scale training strategy by randomly resizing its shorter side to the range of [480, 800]. Considering this higher input resolution, we adaptively increase the focal sizes at four stages to (15, 13, 9, 7), to ensures that the focal attention covers more than half of the image region at the first two stages, and the whole image at the last two stages. With the focal size increased, the relative position biases are accordingly up-sampled to the corresponding sizes using bilinear interpolation. During training, we use AdamW [44] for optimization with initial learning rate 10−4 and weight decay 0.05. Similarly, we use 0.2, 0.3 and 0.5 stochastic depth drop rates to regularize the training for our Tiny, Small and Base models, respectively. Since Swin Transformer does not report the results on RetinaNet, we obtain the results by ourselves using their official code with the same hyper-parameters as that of Focal Transformers.
In Table 3, we show the performance for both CNN-based models and the current Transformerbased state-of-the-art models. The bbox mAP (AP b) and mask mAP (APm) are reported. We see that Focal Transformers outperform the CNN-based models consistently with the gap of 4.8-7.1 points. Compared with the other methods which also use multi-scale Transformer architectures,
Focal Transformers show substantial gains across all settings and metrics. Particularly, Focal Transformers brings 0.7-1.7 points of mAP against the current best approach Swin Transformer [43] at comparable settings. Different from the other multi-scale Transformer models, Focal Transformers can simultaneously enable short-range fine-grain and long-range coarse-grain interactions for each visual token, and thus capture richer visual contexts at each layer for better dense predictions. To have more comprehensive comparisons, we train all models using the 3× schedule and show the detailed numbers for RetinaNet and Mask R-CNN in Table 4. As we can see, even with the 3× schedule, Focal Transformers can still achieve 0.3-1.1 gain over Swin Transformer models in comparable settings.
Comparison with large SoTA detection models. We follow Swin Transformers to use HTC [11] as the detection method in that it reported SoTA performance on COCO detection when using Swin Transformer as the backbone. For fair comparison, we also use soft-NMS [5], instaboost [24] and a multi-scale training strategy with the shorter side in range [400, 1400] and the longer side no more than 1600. We train the model using AdamW [44] with base learning rate 1e-4 and weight decay 0.1. The model is trained using the standard 3× schedule. The box and mask mAPs on COCO validation set and test-dev are reported in Table 5, where both single-scale evaluation and multi-scale evaluation results are presented. Our Focal-Large model with multi-scale test achieves 58.1 box mAP and 50.9 mask mAP on mini-val set, which is better than the reported numbers for Swin-Large in [43]. When evaluating our model on the test-dev set, it achieves 58.4 box mAP and 51.3 mask mAP, which is slightly better than Swin Transformer. Note that because our model does not include the global self-attention layer used in Swin Transformer at the last stage, it has a smaller model size and fewer FLOPs. More recently, DyHead [17] achieves new SoTA on COCO, when combined with Swin-Large. We replace the Swin-Large model with the Focal-Large model, and use the same 2× training schedule as in [17]. We report the box mAPs for both mini-val and test-dev. Focal-Large achieves 58.7 and 59.0 on mini-val and test-dev, respectively.

heading: 4.3 Semantic Segmentation
text: In addition to the instance segmentation results, we also evaluate our models on the semantic segmentation task which usually takes high-resolution input images and requires capturing long-range interactions. We benchmark our methods on ADE20K [83]. We use UperNet [65] as the segmentation method and Focal Transformers as the backbones. We train three models as Focal-Tiny, Focal-Small, Focal-Base, respectively. For all the models, we use a standard recipe that sets the input size to 512× 512 and trains the model for 160k iterations with batch size 16. Table 6 shows the comparison results. We see that Focal-Tiny, Focal-Small and Focal-Base models consistently outperform Swin Transformers of the similar size in single-scale and multi-scale mIoUs. Comparison with large SoTA semantic segmentation models. We use the pretrained Focal-Large model as the backbone for semantic segmentation. Follow the setting in [43], we use input image size 640 × 640 and train the model for 160k iterations with a batch size of 16. We set the initial learning to 6e-5 and use a polynomial learning rate decay. The weight decay is set to 0.01. For
multi-scale evaluation, we use the same scaling ratios [0.5, 0.75, 1.0, 1.25, 1.5, 1.75] as in previous works. The results in Table 6 show that Focal-Large achieves significantly better performance than Swin-Large. In both single-scale and multi-scale evaluations, Focal-Large leads to more than 1 point mIoU improvement, creating new SoTA for semantic segmentation on ADE20K.

heading: 4.4 Ablation studies
text: We conduct a series of ablation studies to inspect the model’s capacity from different aspects. We use Focal-Tiny and the image classification and object detection tasks.
Effect of varying the window size. We have demonstrated that it is crucial to model both short- and long-range interactions. Thus, a related question is whether increasing the window size helps as it leads to a larger receptive field. Table 7 shows the performance of Swin-Tiny and Focal-Tiny with window sizes 7 and 14. Clearly, a larger window size is beneficial for both methods measured in all three metrics, and Focal-Tiny consistently outperforms Swin-Tiny in both window sizes. Comparing the second and third row, we find that Focal-Tiny outperforms Swin-Tiny even with a smaller window size (7 v.s. 14). We suspect that the gain is attributed to our focal attention’s superior capability of capturing long-range dependencies among visual tokens.
The necessity of window shift. In Swin Transformer [43], window shift is proposed to capture crosswindow interactions between two successive layers. In contrast, visual tokens in Focal Transformers can always communicate with each other across windows at both fine- and coarse-grain. Thus, it is interesting to investigate whether adding window shift to Focal Transformers can lead to any improvement. To answer the question, we remove window shift from Swin Transformer while adding it to Focal Transformers. As shown in Table 8, Swin Transformer shows a severe degradation after removing the window shift. However, adding window shift to Focal Transformer hurts classification performance. The result indicates that window shift is unnecessary for Focal Transformers. While in
Swin Transformers, there should always be an even number of layers in each stage for the alternative window shift operation, Focal Transformers do not have such a constraint.
Contributions of local and global interactions. To investigate the relative contributions of capturing local fine-grain and global coarse-grain interactions in Focal Transformers, we have developed several variants of Focal-Tiny: a) Focal-Tiny-Window merely performs attention inside each window; b) Focal-Tiny-Local attends the additional fine-grain surrounding tokens and c) Focal-Tiny-Global attends the extra coarse-grain summarized tokens. We train these models using the same setting as Focal-Tiny and report their performance on image classification and object detection using Mask R-CNN 1× schedule. As shown in Fig. 5, Focal-Tiny-Window suffers from a significant performance drop on both image classification (82.2→80.1) and object detection (44.8→38.3). This is expected since the communication across windows is completely cut off at each Transformer layer. After we enable either the local fine-grain or global coarse-grain interactions (middle two columns), we observe significant performance boost. When we combine short- and long-range interactions, we observe additional improvements on both tasks. This implies that these two type of interactions are complementary and both are beneficial to model performance.
Model capacity against model depth. Focal attention allows a Transformer model to capture shortand long-range interactions at each Transformer layer. An interesting question is whether Focal Transformers need fewer layers to obtain a similar modeling capacity as the Transformer models that does not use focal attention, such as Swin Transformer. To answer this question, we conduct an experiment by training a series of Swin-Tiny and Focal-Tiny models by varying the number of Transformer layers at stage 3. As shown in Table 9, Focal-Tiny outperforms Swin-Tiny consistently with the same depth. More importantly, using fewer layers, Focal-Tiny can sometimes achieve comparable or even better performance than Swin Transformer. For example, Focal-Tiny with (2-2-4-2) achieves 81.4 on image classification which is better than Swin-Tiny with (2-2-6-2).

heading: 5 Conclusion
text: In this paper, we have presented a new focal attention mechanism that enables efficient long-range interactions in Vision Transformers. Different from previous works, it performs the local attention at fine-grain and global attention at coarse-grain, providing an effective way of capturing both shortand long-range context with a manageable computational cost. By applying focal attention into a multi-scale Transformer architecture, we propose Focal Transformers as general-purpose backbones for a wide range of dense vision tasks. A comprehensive empirical study shows that our Focal Transformers outperform the SoTA Vision Transformers on a range of vision tasks including image classification, object detection and segmentation.
Limitations and future work. Although our experiments show that focal attention can significantly boost the performance on image classification and dense prediction tasks, focal attention does introduce extra computational and memory cost, since each query token needs to attend more (summarized) tokens in addition to tokens inside a window. A cost-effective implementation of Focal Transformer is necessary to make it more applicable to many real-world scenarios. This study focuses on incorporating focal attention into multi-scale Vision Transformers for CV tasks. However, we notice that focal attention is an effective sparse attention mechanism that is widely applicable to all attention-based neural network models that are developed for processing natural language, images, videos etc. This is an exciting future direction.

Review summaries:
This paper proposes a focal version of self-attention module for vision transformers. The query is kept at high resolution, but the number of keys is reduced depending on the relative position difference. This proposed attention is implemented within multi-scale windows. It shows good performance on top of recent transformer advances, on ImageNet and COCO. 

This paper proposes Focal Transformer, which captures both local and global attention based on their design. It also follows the pyramid architecture so their model can be easily extended to detection and segmentation tasks. The result on image classification detection and segmentation are very promising.