Twins: Revisiting the Design of Spatial Attention in Vision Transformers
Authors: Xiangxiang Chu
Zhi Tian
Yuqing Wang
Bo Zhang
Haibing Ren
Xiaolin Wei
Huaxia Xia
Chunhua Shen
Venue: NeurIPS 2021 Poster (2021)
Keywords: Vision Transformers
image classification

Abstract: Very recently, a variety of vision transformer architectures for dense prediction tasks have been proposed and they show that the design of spatial attention is critical to their success in these tasks. In this work, we revisit the design of the spatial attention and demonstrate that a carefully devised yet simple spatial attention mechanism performs favorably against the state-of-the-art schemes. As a result, we propose two vision transformer architectures, namely, Twins- PCPVT and Twins-SVT. Our proposed architectures are highly efficient and easy to implement, only involving matrix multiplications that are highly optimized in modern deep learning frameworks. More importantly, the proposed architectures achieve excellent performance on a wide range of visual tasks including image-level classification as well as dense detection and segmentation. The simplicity and strong performance suggest that our proposed architectures may serve as stronger backbones for many vision tasks. 

Twins: Revisiting the Design of Spatial Attention in Vision Transformers
Xiangxiang Chu',
Zhi Tian!.? Yuqing Wang', Bo Zhang'
Haibing Ren', Xiaolin Wei!, Huaxia Xia', Chunhua Shen
Meituan Inc.
2 The University of Adelaide, Australia
(chuxiangxiang, wangyuqing06, zhangbo97, iaaiggoo07,rehhibbiinee renhaibing, xeixiaolin02, aaann,,xiaaiii xiaauaxxaaaa Amei tuan. com zhi. wwwaaottlooo..coo.... com, chunhuadme.com
Abstract
Very recently, a variety of vision transformer architectures for dense prediction tasks have been proposed and they show that the design of spatial attention is critical to their success in these tasks. In this work, we revisit the design of the spa- tial attention and demonstrate that carefully devised yet simple spatial attention mechanism performs favorably against the state-of state-of-the-art schemes. As a result, we propose two vision transformer architectures, namely, PCPVT and Twins- SVT. Our proposed architectures are highly efficient and easy to implement, only involving matrix multiplications that are highly optimized in modern deep learning frameworks. More importantly, the proposed architectures achieve excellent per- formance on a wide range visual tasks including image-level classification as well as dense detection and segmentation. The simplicity and strong performance suggest that our proposed architectures may serve as stronger backbones for many vision tasks. Our code is available at: https://git, Lo/wwins
Introduction
Recently, Vision Transformers [1-3] have received increasing research interest. Compared to the widely-used convolutional neural networks (CNNs) in visual perception, Vision Transformers enjoy great flexibility in modeling long-range dependencies in vision tasks, introduce less inductive bias, and can naturally process multi-m multi-modality input data including images, videos, texts, speech signals, and point clouds. Thus, they have been considered to be a strong alternative to CNNs. It is expected that vision transformers are likely to replace CNNs and serve as the most basic component in the next-generation visual perception systems.
One of the prominent problems when applying transformers vision tasks is the heavy computational complexity incurred by the spatial self-attention operation in transformers, which grows quadratically in the number of pixels of the input image. workaround is the locally-grouped self-attention (or self-attention in non-overlapped windows as in the recent Swin Transformer [4]), where the input spatially grouped into non-overlapped windows and the standard self-attention is computed only within each sub-windows Although it can significantly reduce the complexity, it lacks the connections between different windows and thus results in limited receptive field. As pointed out by many previous works [5-7], sufficiently large receptive field is crucial to the performance, particularly for dense prediction tasks such as image segmentation and object detection. Swin [4] proposes a shifted window operation to tackle the issue, where the boundaries of these local windows are gradually moved as the network proceeds. Despite being effective, the shifted windows may have uneven sizes. The uneven windows result in difficulties when the models are deployed with ONNX or TensorRT,
Corresponding author.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

which prefers the windows of equal sizes. Another solution is proposed in PVT [8]. Unlike the standard self attenttnnnion operation, where each query computes the attention weights with all the imput tokens, in PVT, each query only computes the attention with a sub-sampled version of the input tokens. Although its computational complexity in theory is still quadratic, is already manageable in practice.
From a unified perspective, the core in the aforementioned vision transformers is how the spatial attention is designed. Thus, in this work, we revisit the design of the spatial attention in vision transformers. Our first finding is that the global sub-sampled attention in PVT highly effective, and with the applicable positional encodings [9]. its performance can be on par or even better than state-of-the- art vision transformers (e.g., Swin). This results in our first proposed architecture. termed
On top of that, we further propose a carefully-desigres attention mechanism, making our architectures more efficient than PVT. Our attention mechanism is inspired by the widely-used separable depthwise convolutions and thus we name it spatially separable self-attention (SSSA). Our proposed SSSA is composed of two types of attention operations (i) locally-grousl
yer simple spatial
self-attention (LSA), and (ii) global sub-sampled attention (GSA), where LSA captures the fine-grained and short-distance information and GSA deals with the long-distance and global information. This leads to the second proposed vision transformer architecture, termed Twins-S SVT. It is worth noting that both attention operations in the architecture are efficient and easy-to- easy-to-impledt implement with matrix multiplications in few lines of code. Thus, all of our architectures here have great applicability and can be easily deployed.
We benchmark our proposed architectures on number of visual tasks, ranging from image-level classification to pixel-level semantic/instance segmentation and object detection. Extensive experi- ments show that both of our proposed architectures perform favorably against other state-of-the-at vision transformers with similar or even reduced computational complexity.
2 Related Work
Convolutional neural networks. Characterized by local connectivity, weight sharing, shift- invariance and pooling, CNNs have been the de facto standard model for computer vision tasks. The pop-performing models [10-13] in image classification also serve as the strong backbones for downstream detection and segmentation tasks.
Vision Transformers. Transformer was firstly proposed by [14] for machine translation tasks, and since then they have become the state-of- state-of-the-art art-tht-rrttt models for NLP tasks, overtaking the sequence-to sequence approach built on LSTM. Its core component is multi-head self-attention which models the relationship between input tokens and shows great flexibility.
In 2020, Transformer was introduced to computer vision for image and video processing [1-3, 9, 15- 17, 17-32]. In the image classification task, VIT [1] and DeiT [2] divide the images into patch embedding sequences and feed them into the standard transformers. Although vision transformers have been proved compelling in image classification compared with CNNs, challenge remains when it is applied to dense prediction tasks such as object detection and segmentation. These tasks often require feature pyramids for better processing objects of different scales, and take inputs the high- resolution images, which significantly increase the computational complexity of the self-attention operations.
Recently, Pyramid Vision Transformer (PVT) [8] is proposed and can output the feature pyramid [33] as in CNNs. PVT has demonstrated good performance in a number of dense prediction tasks. The recent Swin Transformer [4] introduces non non-overlapping window partitions and restricts self-attention within each local window, resulting in linear computational complexity in the number of input tokens. To interchange information among different local areas, its window partitions are particularly designed to shift between two adjacent self-attention layers. The semantic segmentation framework OCNet [34] shares some similarities with us and they also interleave the local and global attention. Here, we demonstrate this is general design paradigm in vision transformer backbones rather than merely an incremental module in semantic segmentation.
Grouped and Separable Convolutions. Grouped convolutions are originally proposed in AlexNet [35] for distributed computing. They were proved both efficient and effective in speeding up the networks. As an extreme case. depthwise convolutions [12, 36] use the number of groups that is

---

equal to the input or output channels, which is followed by point-wise convolutions to aggregate the information across different channels. Here, the proposed spatially separable self-attention shares some similarities with them.
Most vision transformers absolute/relative positional encodings, de- pending on downstream tasks, which are based on sinusoidal functions [14] or learnable [1, 2]. In CPVT [9], the authors propose the conditional positional encodings, which are dynamically conditioned on the inputs and show better performance than the absolute and relative ones.
Positional Encodings.
3 Our Method: Twins
We present two simple yet powerful spatial designs for vision transformers. The first method is built upon PVT [8] and CPVT [9], which only uses the global attention. The architecture is thus termed The second one, termed (Wins--VT, is based on the proposed SSSA which interleaves local and global attention.
3.1 Twins-PCPVT
PVT [8] introduces the pyramid multi-stage design to better tackle dense prediction tasks such as object detection and semantic segmentation. It inherits the absolute positional encoding designed in ViT [1] and DeiT [2]. All layers utilize the global attention mechanism and rely on spatial reduction cut down the computation cost of processing the whole sequence. It is surprising to that the recently-proposed Swin transformer [4], which is based on shifted local windows, can perform considerably better than PVT, even on dense prediction tasks where a sufficiently large receptive field is even more crucial to good performance.
In this work, we surprisingly found that the less favored performance of PVT is mainly due the absolute positional encodings employed in PVT [8]. As shown in CPVT [9], the absolute positional encoding encounter difficulties in processing the inputs with varying sizes (which common in dense prediction tasks). Moreover, this positional encoding also breaks the translation invariance. On the contrary, Swin transformer makes use of the relative positional encodings, which bypasses the above issues. Here, we demonstrate that this is the main cause why Swin outperforms PVT, and we show that if the appropriate positional encodings are used, PVT can actually achieve par better performance than the Swin transformer.
Here, we use the conditional position encoding (CPE) proposed in CPVT [9] to replace the absolute PE in PVT. CPE is conditioned on the inputs and can naturally avoid the above issues of the absolute encodings. The position encoding generator (PEG) [9], which generates the CPE, is placed after the first encoder block of each stage. We use the simplest form of PEG, i.e., a 2D depth-wise convolution without batch normalization. For image-level classification, following CPVT, we remove the class token and use global average pooling (GAP) at the end of the stage [9]. For other vision tasks, we follow the design of PVT. TWins-PCPVT inherits the advantages of both PVT and CPVT, which makes it easy to be implemented efficiently. Our extensive experimental results show that this simple design can match the performance of the recent state-of-the- oo-thc-art Swin transformer. We have also attempted to replace the relative PE with CPE in Swin, which however does not result in noticeable performance gains, as shown in our experiments. We conjecture that this maybe due the use of shifted windows in Swin, which might not work well with CPE.
Architecture settings We report the detailed settings of Twins-PCPVT in Table 2 (in supplemen- tary), which are similar to PVT [8]. Therefore, Twins-PCPVT has similar FLOPs and number of parameters to [8].
3.2 Twins-SVT
Vision transformers suffer severely from the heavy computational complexity in dense prediction tasks due to high-resolution inputs. Given an input of H W resolution, the complexity of self- attention with dimension d is O (d)) Here, we propose the spatially separable self self-attention (SSSA) to alleviate this challenge. SSSA is composed of locally-grouper self-attention (LSA) and global sub-sampled attention (GSA).
3

---

Stage 1
Stage 2
Stage
Stage 4
Transfemmr
X128
X512
Patch embelltig Aiog
Patch
Patch anleedding
Encolerr
Eacoder
Encoder
Tranffomer
PRG
Encoder
Transfemne
Eihender
Transfommer
LayerNomi
LaverNora
Fornee
Layonoom
LaverNorm
Figure 1 Architecture of Twins-SVT-S, "PEG" is the positional encoding generator from CPVT [9].
gocouly sell-attention (LSA). Mo- tivated by the group design in depthwise convo- lutions for efficient inference, we first equally divide the 2D feature maps into wuidds, making self-attention communications only hap- pen within each sub- window. This design also resonates with the multi-head design in self- attention, where the communications only occur within the channels of the same head. To be spe- cific, the feature maps are divided into m x n sub-windows. Without loss of generality, we assume H% = 0 and W% = Each group contains HW elements, and thus the computa- tion cost of the self-attention in this window is w2 O( n d). and the total cost is O( H" d). If
Ohibal Afratiso
Aacel Aaci Mimmio
Mim
Win
(9)
Figure 2 - (a) Twins-SVT interleaves locally-grouped attention (LSA) and global sub-sampled attention (GSA). attention
ma
(b) Schematic view of the (LSA) and global sub-sampled attention (GSA),
mn
we let k computed Ōk, HWW d). which is significantly more efficient when dr < H and K2 < W and grows linearly with HW k1 and ly are fixed.
and 2
the cost can be
Although the locally-grouped self-attention mechanism is computation friendly, the image is divided into non-overlapping sub-windows Thus, we need mechanism communicate between different sub-windows, as in Swin. Otherwise, the information would be limited to be processed locally, which makes the receptive field small and significantly degrades the performance as shown in our experiments. This resembles the fact that we cannot replace all standard convolutions by depth-wise convolutions in CNNs.
Global sub-sampled attention (GSA), A simple solution is to add extra standard global self- attention layers after each local attention block, which can enable cross-group information exchange. However, this approach would come with the computation complexity of ((H=W2)))
Here, we use single representative to summarize the important information for each of m sub- windows and the representative is used to communicate with other win-wis (serving as the key in self-attention), which can dramatically reduce the cost to 0mmHHW d) = 0 HF This is essentially equivalent to using the sub-sampled feature maps as the key in attention operations, and thus we term it global sub-sampled attention (GSA). If we alternatively use the aforementioned LSA and GSA like separable convolutions (depth-wise + wisnt- The total computation cost O H"" + ln HWd). d). We have H' k k: HWd 2HW d/WW The minimum is k2
kike
obtained when k1 l2 = / VWW. We note that H = W 224 is popular in classification. Without loss of generality, we use square sub-windows, i.e., k 1z Therefore,
= k2 = 15 close to the global minimum for H = W = 224. However, our network is designed to include several stages with variable resolutions. Stage has feature maps of 56 56, the minimum is obtained when k = 566 Theoretically, we can calibrate optimal kn and k2 for each of the stages. For simplicity, we use k k2 = 7 everywhere. As for stages with lower resolutions, we control the summarizing window-size of GSA to avoid too small amount of generated keys. Specifically, we use the size of 4, and for the last three stages respectively.

---

As for the sub sampling function, we investigate several options including average pooling, depth- wise strided convolutions, and regular strided convolutions. Empirical results show that regular strided convolutions perform best here. Formally, our spatially separable self-attention (SSSA) can be written as
8/ = LSA (LayerNorm
(2 + iij
1-1
2/ FFN (LayerNorm (21))) + 2
r = GSA (LayerNorm (z') +
(I)
2 211 FFN (LayerNorm (2(+1) + 2/111
i {1,
..... m.,,,,.,,,, E {1, n.
where LSA means locally-grouped self-attention within sub-window; GSA is the global sub-sampled attention by interacting with the representative keys (generated by the sub-sampling functions) from each window žig E R X Both LSA and GSA have multiple heads as in the standard self-attention. The PyTorch code of LSA is given in Algorithm I (in supplementary).
Again, we the PEG of CPVT [9] encode position information and process variable-length inputs on the fly. It is inserted after the first block in each stage.
Model variants. The detailed configure of Twins-SVT is shown in Table 3 (in supplementary). We try our best to use the similar settings as in Swin [4] to make sure that the good performance is due to the new design paradigm.
Comparison with PVT. PVT entirely utilizes global attentions as DeiT does while our method makes use of spatial separable-like design with LSA and GSA, which is more efficient.
Comparison with Swin. Swin utilizes the alternation of local window based attention where the window partitions in successive layers are shifted. This is used to introduce communication among different patches and to increase the receptive field. However, this procedure is relatively complicated and may not be optimized for speed on devices such as mobile devices. Swin Transformer depends on torch.roll() to perform eyclic shift and its reverse features. This operation is memory unfriendly and rarely supported by popular inference frameworks such as NVIDIA TensorRT, Google Tensorflow- Lite, and Snapdragon Neural Processing Engine SDK (SNPE), etc. This hinders the deployment of Swin either on the server-side or on end devices in a production environment. In contrast, Twins models don't 't require such an operation and only involve matrix multiplications that are already optimized well in modern deep learning frameworks. Therefore, it can further benefit from the optimization in production environment. For example, we converted Twins-SVT-S from Py Torch to TensorRT, and its throughput is boosted by x.7x. Moreover, our local-global design can better exploit the global context, which known to play an important role in many vision tasks.
Finally, one may note that the network configures (e.g. such as depths, hidden dimensions, number of heads, and the expansion ratio of MIP) of our two variants are sightly different. This is intended because we want to make fair comparisons to the two recent well-known transformers PVT and Swin. PVT prefers a slimmer and deeper design while Swin is wider and shallower. This difference makes PVT have slower training than Swin. Twins-PCPVT is designed to compare with PVT and shows that a proper positional encoding design can greatly boost the performance and make it on par with recent saate-ff-the-ar models like Swin. On the other hand, Twins-SVT demonstrates the potential of a new paradigm as spatially separable self otffaatennion is highly competitive to recent transformers.
4 Experiments
4.1
Classification on ImageNet-1K
We first present the ImageNet classification results with our proposed models. We carefully control the experiment settings to make fair comparisons against recent works [2, 8, 9]. All our models are trained for 300 epochs with a batch size of 1024 using the AdamW optimizer [37]. The learning rate is initialized to be 0.001 and decayed to zero within 300 epochs following the cosine strategy. We use a linear warm-up in the first five epochs and the same regularization setting as in [2]. Note that we do not utilize extra tricks in [26, 28] to make fair comparisons although it may further improve the
5

---

performance of our method. We use increasing stochastic depth [38] augmentation of 0.2, 0.3, 0.5 for small, base and large model respectively. Following Swin [4], we use gradient clipping with max norm of 5.0 to stabilize the training process, which is especially important for the training of large models.
We report the classification results on ImageNet- IK [39] in Table 1. PVT-small by 1.4% and obtains similar result as Swin-T with 18% fewer FLOPs. Twins-SVT-S is better than Swin-T with about 35% fewer FLOPs. Other models demonstrate similar advantages.
outperforms
It is interesting to that, without bells and whistles, Twins-PCPVT performs on par with the recent state-of- oh-tthe-rt Swin, which is based on much more sophisticated designs as mentioned above. Moreover, Twins-SVT SVT also achieves similar better results, compared to Swin, indicating that the spatial separable-like design is an effective and promising paradigm.
One may challenge our improvements are due to the use of the better positional encoding PEG. Thus, we also replace the relative PE in Swin-T with PEG [9], but the Swin-T's performance cannot be improved (being 81.2%).
4.2 Semantic Segmentation on ADE20K
We further evaluate the performance on segmentation tasks. We test on the ADE20K dataset [42], a challenging scene parsing task for semantic segmentation, which popularly evaluated by recent Transformer-basee methods. This dataset contains 20K images for training and 2K images for validation. Following the common practices, we use the training set to train our models and report the mloU on the validation set. All models are pretrained on the ImageNet- 1 dataset.
Twins-PCPVT vs. PVT. We compare our
with PVT [8] because they have similar design and computational complexity. To make fair comparisons, we use the Semantic FPN framework [43] and exactly the same training settings in PVT. Specifically, we train 80K steps with a batch size of 16 using AdamW [37]. The learning rate is initialized as I x 1x10 and scheduled by the 'poly' strategy with the power coefficient of 0.9. We apply the drop-path regularization of 0.2 for the backbone and weight decay 0.0005 for the whole network. Note that we use stronger drop-path regularization of 0.4 for the large model to avoid over-fitting For Swin, we use their official code and trained models. We report the results in Table 2. With comparable FLOPs, Prins-PCPVVT-S outperforms PVT-Small with large margin (+4.5% mloU), which also surpasses ResNet-50 by 7.6% mloU. It also outperforms Swin-T with clear margin. Besides, Twins-PCPVTT-R also achieves 3.3% higher mIoU than PVT-Medium, and Twins-PCPVTTL surpasses PVT-Large with 4.3% higher mloU.
Twins-SVT vs. Swin. We also compare our Twins-SVT with the recent state-of-thh-art model Swin [4]. With the Semantic FPN framework and the above settings, Twins-SVT-S achieves better performance (+1.7%) than Swin-T. Twins-SVT-B obtains comparable performance with Swin-S and Twins-SVT-L outperforms Swin-B by 0.7% mloU (left columns Table 2). In addition. Swin evaluates its performance using the UperNet framework [44]. We transfer our method to this framework and use exactly the same training settings as [4]. To be specific, we use the AdamW optimizer to train all models for 160k iterations with global batch size of 16. The initial learning rate is 6x 10 and linearly decayed to zero. We also utilize warm-up during the first 1500 iterations. Moreover, we apply the drop-path regularization of 0.2 for the backbone and weight decay 0.01 for the whole network. We report the mloU of both single scale and multi-scale testing (we use scales from 0.5 to 1.75 with step 0.25) in the right columns of Table 2. Both with multi-scale testing, Twins-SVT-S outperforms Swin-T by 1.3% mloU. Moreover, Twins-SVT-L achieves new state of the art result 50.2% mloU under comparable FLOPs and outperforms Swin-B by 0.5% mloU. Twins-PCPVT also achieves comparable performance to Swin [4].
4.3 Object Detection and Segmentation on COCO
We evaluate the performance of our method using two representative frameworks: RetinaNet [46] and Mask RCNN [47]. Specifically, we use our transformer models to build the backbones of these detectors. All the models are trained under the same setting as [8]. Since PVT and Swin report their results using different frameworks, we try to make fair comparison and build consistent settings for future methods. Specifically, we report standard x-schedule (12 epochs) detection results on the COCO 2017 dataset [48] in Tables 3 and 4, As for the evaluation based RetinaNet, we train

---

Table 1 Comparisons with state-of-the-ar methods for ImageNet-IK IK classification. Throughput is tested the batch size of 192 single V100 GPU. All models are trained and evaluated 224x resolution ImageNet- IK dataset. 1; CPVT's position encodings [9].
Method
Param (M)
FLOPs (G)
Throughput (Images/s)
Top-1 (%)
ConvNet
RegNetY-4G [40]
21
4.0
1157
80.0
RegNetY-8G [40]
8.0
592
81.7
RegNetY-16G [40]
84
16.0
335
82.9
Transformer
DeiT-Small/16 [2]
22.1
4.6
437
ViossViT-S [30]
26.7
5.6
81.0
T2T-ViT-14 [27]
22
S.2
81.5
TNT-S [15]
23.8
5.2
81.3
Coa'T Mini [17]
6.8
80.8
CoaT-Lite Small [17]
4.0
81.9
PVT-Small [8]
24.5
3.8
820
79.8
CPVT-SmalI-GAP [9]
23
4.6
817
81.5
Twins-PCPVT-S (ours)
24.1
3.8
815
81.2 (+1.3)
Swin-T [4]
4.5
766
81.3
Swin-T CPVT
28
4.4
766
81.2
Twins-SVT-S (ours)
24
2.9
1059
81.7 (+1.8)
T2T-ViT-19 [27]
39.2
8.9
81.9
PVT-Medium [8]
44.2
6.7
526
81.2
Twins- Twins-PCPV/I PCPVT-B(ours)
82.7 (+0.8)
Swin-S [4]
50
8.7
444
83.0
Twins-SVT-B (ours)
56
8.6
469
83.2 (+1.3)
ViT-Basc/16 [1]
86.6
17.6
86
77.9
DeiT-Basc/16 [2]
86.6
17.6
292
81.8
T2T-ViT-24 [27]
64.1
14.1
82.3
CrossViT-B [30]
104.7
21.2
TNT-B [15]]
66
14.1
82.8
CPVT-B [9]
88
17.6
292
82.3
PVT-Large
61.4
9.8
367
81.7
(OurT)
367
(+5.2)
Swin-B [4]
88
15.4
275
83.3
Twins-SVT-L. (ours)
99.2
15.1
288
83.7 (+5.8)
Hybrid
BoTNet-S1-59 [29]
33.5
7.3
81.7
BossNet-T1 [41]
7.9
81 9
CvT-13 [31]
20
4.5
81.6 61.6
BoTNet-S1-110 [29]
54.7
10.9
82.8
CvT-21 [31]
7.1
82.5
all the models using AdamW [37] optimizer for 12 epochs with batch size of 16. The initial learning rate 10-, started with 500-iteration warmup and decayed by 10x the and 11th epoch, respectively. We use stochastic drop path regularization of 0.2 and weight decay 0.0001. The implementation is based on MMDetection [49]. For the Mask R-CNN framework, we use the initial learning rate of x 10 as in [8]. All other poyper-parameee MMDetection. As for 3x experiments, we follow the common multi-scale training in [3, 4], i.e., randomly resizing the input image that its shorter side is between 480 and 800 while keeping longer one less than 1333. Moreover, for 3x training of Mask R-CNN, we use an initial learning rate of 0.0001 and weight decay of 0.05 for the whole network [4].
follow the default settings in
For x schedule object detection with RetinaNet, TWins-PCPVTS surpasses PVT-Small with 2.6% mAP and TWins-PCPVTT-B exceeds PVT-Medium by 2.4% mAP on the COCO va12017 split. Twins-SVT-S outperforms Swin-T with 1.5% mAP while using 12% fewer FLOPs. Our method outperform the others with similar advantage in 3 experiments.

---

Table 2 Performance comparisons with different backbones ADE20K validation dataset. FLOPs tested on 512x512 resolution. All backbones are pretrained on ImageNet- except SETR [45]. which is pretrained on ImageNot-216 dataset.
Semantic FPN 80k (PVT [8] setting)
Upernet 160k (Swin [4] setting)
Backbone
FLOPs
Param
mloU
FLOPs
Param
mloU/MS mloU
(G)
(M)
(%)
(G)
(M)
(%)
ResNet50 [10]
45
28.5
36.7
PVT-Small [8]
40
28.2
39.8
Twins-PCPVT-S (ours)
28.4
44.3 (+7.6)
234
54.6
46.2/47.5
Swin-T [4]
46
31.9
41.5
237
59,9
44.5/45.8
Twins-SVT-S (ours)
37
28.3
43.2 (+6.5)
228
54.4
46.2/47.1
ResNet101 [10]
66
47.5
38.8
258
86
-/44.9
PVT-Medium [8]
55
48.0
41.6
(ours)
55
48.1
44,9 (+6.1)
250
74.3
47.1/48.4
Swin-S [4]
53.2
45,2
81.3
47.6/49.5
Twins-SVT-B (ours)
67
60.4
45.3 (+6.5)
261
88.5
47.7/48.9
ResNetX(101-64x44 [13]
86.4
40.2
PVT-Large [8]
71
65.1
42.1
Twins-PCPVT-L (ours)
71
65.3
46.4 (+6.2)
269
91.5
48.6/49.8
Swin-B [4]
107
91.2
46.0
299
121
48. 1/49.7
(ours)
102
103.7
46.7 (+6.5)
297
133
48.8/50.2
Backbone
PUP (SETR [45] setting)
MLA (SETR [4S] setting)
T-Large (SETR) [45]
310
50.1
308
48.6/50.3
For 1x object segmentation with the Mask R-CNN framework, Twins-PCPVT-S brings similar improvements (+2.5% mAP) over PVT-Small. Compared with PVT-Medium, Twins-CCPVT-B obtains 2.6% higher mAP, which is also on par with that of Swin. Both Twins-SVT-S and Twins-SVT- achieve better or slightly better performance compared to the counterparts of Swin. As for large models, our results are shown in Table (in supplementary) and we also achieve better performance with comparable FL.OPs.
Table 3 Object detection performance on the COCO va12017 split using the RetinaNet framework. x is 12 epochs and 3x is 36 epochs. "MS Multi-scale training. FLOPs are evaluated on 800x600 resolution.
ParPs (G)
RetinaNet 1x
RetinaNet x + MS
Backbone
(M)
JP
APso APs AM APt/AP
APso AP3 AP APL
ResNet50 [10]
111
37.7
36.3
55.3 38.6 19.3 40.0 48.8/9.00
58.4 41,8 22.4 42.8 51.6
PVT-Small [8]
118
34.2
40.4
61.3 43.0 25,0 42.9 55.7
62.7 45,0 26.2 45,2 57.2
Twins-PCPVT: (ours)
118
34.4
43.0(+6.7)
64.1 46.0 27.5 46.3 57.345.2(+6.2)
66.5 48.6 30.0 48.8 58.9
Swin-T [4]
118
38.5
41.5
62.1 44.2 25.1 44.9 55.5
64.8 47.1 28.4 47.2 57.8
Twins-SVT-S (ours)
104
34.3
43.0(+6.7)
64.2 46.3 28.0 46.4 57.5
67.1 48.6 29.8 49.3 60.0
ResNet101 [10]
56.7
38.5
$7.8 41.2 21.4 42.6 41.1
60.1 44.0 23.7 45.0 53.8
ResNextl101-32x4d [13]
151
56.4
39,9
59.6 42.7 22.3 44,2 52.541.4
61.0 44.3 23.9 45.5 53.7
PYT-Medium [8]
151
53.9
41.9
63.1 44.3 25.0 44,9 57.6
63.8 46.1 27.3 46.3 58.9
Twins-PCPVT-B (ours)
151
54.1
44.3(+5.8)
65.6 47.3 27.9 47.9 59.646.4(455))
67.7 49.8 31.3 50.2 61.4
Swin-S
162
59.8
44.5
65.7
47.5
27.4
48.0 59.946.3
67.4 49.8
31.1 50.3 60.9
Twins-SVT-B (ours)
67.0
45.3(46.8)
66.7
48.1
28.5 48.9 60.664..((660))
68.0 50.2
31.7
50.3 61.8
4.4 Ablation Studies
Table 5 Classification performance for different combinations of LSA (L.) and GSA (G) blocks based on the small model.
Configurations of LSA and GSA blocks. We evaluate dif- ferent combinations of LSA and GSA based on our small model and present the ablation results in Table 5. The models with only gaaluee attention fail to
Function Type
Params
FLOPs
Top-1 (%)
(M)
(G)
L.
8.8
76,9
L.G.G.GLLIGGGGG
23.5
81.5
I., G)
24.1
2.8
81.7
G,
22.2
2.9
80.5
PVT-small (G. G. [8)
24.5
3.8
79.8

---

Table 4 Object detection and instance segmentation performance on the COCO va12017 dataset using the Mask R-CNN framework. FLOPs evaluated x00 600 image.
Backbone FLOPsParam
Mask R-CNN 1x
Mask R-CNN 3x MS
(G)
(M)
APh
AP
AP56AP
AP
AP% AP AP
ResNet50 [10]
174
44.2
33.00
58.6 41,4 34.4
55.1 41.0
61.7 44.9 37.1 58.4 40.1
PVT-Small [8]
178
44.1
40.4
62.9 43.8 37.8
60.1 40.3 43.0
65.3 46.9 39.9 62.5 42.8
Twins-PCPVT-S (ours)
178
44.3
42.9
65.8 47.1 (0.0
62.7 42,9 46.8
69.3 51.8 42,6 66.3 46.0
Twin-T [4]
177
47.8
42.2
64.6 46,2 39.1
61.6 42.0 46.0
68.2 50.2 41.6 65.1 44.8
Twins-SVT-S (ours) 164
44.0
43.4
66.0 47.3 (..9
63.2 43.4
69.2 51.2 42.6 66.3 45.8
ResNet101 [10] 210
63.2
40.4
61.1 44.2 36.4
57.7 38.8 42.8
63.2 47.1 38.5 60.1 41.3
ResNext101-32x4d [13] 212
62.8
41.9
62.5 37.5
59.4 40.2
44.0
64.4 48.0 39.2 61.4 41.9
PYT-Medium [8] 211
63.9
42.0
64.4
39.6
61.6 42.1 44.2
66.0 48.2 40.5 63.1 43.5
Twins-PCPVT-B (ours) 211
64.0
44.6,44)
66.7
48.9 (0.9
63.8
44.2
(7.9
70.1 52.5 43.2 67.2 46.3
Swin-S [4] 222
69.1
44.8
66.6 40.9
63.4
44.2
47,6
69.4 52.5 42,8
66.5 46.4
Twins-SVT-B (ours)
224
76.3
45.2
67.6 49.3
41.5
64.5
44.8
48.0
69.5
52.7 43.0
66.8
46.6
obtain good performance (76.9%)
because this setting has limited and small receptive field. An extra global attention layer in the last stage can improve the classification performance by 3.6%. achieves good performance (81.5%), but we do not use this design in this work.
(abbr: LLG) also
Table ImageNet classification perfor- mance of different forms of sub-sampled functions for the global sub-sampled atten- tion (GSA).
Sub-sampling functions. We further study how the dif- ferent sub-sampling functions affect the performance. Specifically, we compare the regular strided convolutions, separable convolutions and average pooling based on the 'small' model and present the results in Table The first option performs best and therefore we choose it as our default implementation.
Function Type
Top- 1(9-
2D Conv.
81.7
2D Separable Conv.
81.2
Average Pooling
81.2
Positional Encodings. We replace the relative positional encoding with CPVT for Swin-T and report the detection performance on COCO with RetinaNet and
Mask R-CNN in Table 7. The CPVT-based Swin cannot achieve improved performance with both frameworks, which indicates that our performance improvements should be owing to the paradigm of Twins-SVT instead the positional encodings.
Table 7 Object detection performance on the COCO using different positional encoding strategies.
RetinaNet
Mask RCNN
Backbone
FLOPP(() Param(M) AP
AP50
APTs
FLOPs(()
Param(M) AP
APso AP75
Swin-T [4]
245
38.5
41.5
62.1
44.2
264
47.8
42.2
64.6
46.2
Swin-T+CPVT
245
38.5
41.3
62.4
44.1
263
47.8
42.0
64.5
45.9
5 Conclusion
In this paper, we have presented two powerful vision transformer backbones for both image-level classification and a few downstream dense prediction tasks. We dub them as twin transformers: and TWins-SVT. The former variant explores the applicability of conditional positional encodings [9] in pyramid vision transformer [8], confirming its potential for improving backbones in many vision tasks. the latter variant we revisit current attention design to proffer a more efficient attention paradigm. We find that interleaving local and global attention can produce impressive results, yet comes with higher throughputs. Both transformer models set new state of the art in image classification, objection detection and semantic/instance segmentation.

---

References
[1]
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image
recognition at scale. In Proc. Int. Conf. Learn. Representations, 2021.
[2] Hugo Touvron, Matthicu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. arXiv preprint arXiv:2012. 12877, 2020.
[3] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usumier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. Proc. Eur. Conf. Comp.
Vis., pages 213-229. Springer, 2020.
[4] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proc. IEEE Int. Conf. Comp. Vis., pages 10012-10022, 2021.
[5] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille, Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Trans. Puttern Anal. Mach. Intell., 40(4):834-845, 2017.
[6] Chao Peng. Xiangyu Zhang, Gang Yu, Guiming Luo, and Jian Sun. Large kernel matters- improve semantic segmentation by global convolutional network. In Proc. IEEE Conf. Comp. 2017.
Vis. Patt. Recogn., pages
[7] Wei Liu, Andrew Rabinovich, and Alexander C Berg. Parsenet: Looking wider to see better. arXiv preprini arXiv: 051.5056664779 2015.
[8] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. arXiv preprint arXiv:2102.1 /2122, 2021.
[9] Xiangxiang Chu, Zhi Tian, Bo Zhang, Xinlong Wang. Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Conditional positional encodings for vision transformers. arXiv preprint arXiv:2102. 10882, 2021.
[10] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proc. IEEE Conf. Comp. Vis. Part. Recogn., pages 770-778, 2016.
[1]] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. Proc. Ini. Conf. Mach. Learn., pages 6105-6114 14. PMLR, 2019.
[12] François Chollet. Xception: Deep learning with depthwise separable convolutions. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pages 1251-1258, 2017.
2017.
[13] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pages
1492-1500, 2017.
[14] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proc. Advances in Neural Inf. Process. Syst., pages 6000-6010, 2017.
[15] Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, and Yunhe Wang. Transformer in transformer. arXiv preprint arXiv:2103.0 10, 2021.
[16] Prajit Ramachandran, Niki Parmar, Ashish Vaswani, Irwan Bello, Anselm Levskaya, and Jon Shlens. Stand-alone self-attention vision models. In Proc. Advances in Neural Inf. Process. Syst., volume 32, pages 68-80, 2019.
[17] Weijian Xu, Yifan Xu, Tyler Chang, and Zhuowen Tu. Co-scale conv-attentional image transformers, 2021.
[18] Yuqing Wang, Zhaoliang Xu, Xinlong Wang, Chunhua Shen, Baoshan Cheng, Hao Shen, and Huaxia Xia. End-to-end video instance segmentation with transformers. In Proc. IEFE Conf. Comp. Vis. Patt. Recogn., 2021.
[19] Hanting Chen, Yunhe Wang, Tianyu Guo, Chang Xu, Yiping Deng, Zhenhua Liu, Siwei Ma. Chunjing Xu, Chao Xu, and Wen Gao. Pre-trained image processing transformer. arXiv preprint arXiv:2012.00364, 2020.
[20] Fuzhi Yang. Huan Yang, Jianlong Fu, Hongtao Lu, and Baining Guo. Learning texture trans- former network for image super-resolution. In Proc. IEFE Conf. Comp. Vis. Putt. Recogn., pages 5791-5800, 2020.
10

---

[21]
transforma- tions for video inpainting. In Proc. Eur. Conf. Comp. Vis., pages 528- 188-53.. Springer, 2020.
Yanhong Zeng, Jianlong Fu, and Hongyang Chao. Learning joint
[22]
Zhigang Dai, Bolun Cai, Yugeng Lin, and Junying Chen. UP-DETR: Unsupervised pre-training for object detection with transformers. arXiv preprint arXiv:2011.0900
2020.
[23] Aravind Srinivas, Tsung- Yi Lin, Niki Parmar, Jonathon Shlens, Pieter Abbeel, and Ashish Vaswani. Bottleneck transformers for visual recognition. arXiv: Comp. Res. Repository, 2021. [24] Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng. Tao Xiang. Philip HS Torr, et al. Rethinking semantic segmentation from a sequence-to-scqqeneee perspective with transformers. arXiv preprint arXiv:2012.15844 2020. [25] Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In Proc. Int. Conf. Mach. Learn., volume 80. pages
4055-4064, 2018.
[26] Hugo Touvron, Matthicu Cord, Alexandre Sablayrolles, Gabriel Synnaeve. and Hervé Jégou. Going deeper with image transformers. arXiv preprint arXiv:2103. 2021.
[27] Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Francis Tay, Jiashi Feng, and Shuicheng Yan. fokens-to-toker VIT: Training vision transformers from scratch on imagenet.
arXiv preprini arXiv:2 0::::2001.111 1986, 2021.
[28] Zihang Jiang, Qibin Hou, Li Yuan. Daquan Zhou, Xiaojie Jin, Anran Wang, and Jiashi Feng. Token labeling: Training an 85.4% top-1 accuracy vision transformer with 56M parameters on ImageNet. arXiv: Comp. Res. Repository, 2021.
[29] Aravind Srinivas, Tsung- Yi Lin, Niki Parmar, Jonathon Shlens, Pieter Abbeel, and Ashish Vaswani. Bottleneck transformers for visual recognition. arXiv preprint arXiv:2101, 1605, 2021.
[30] Chun-Fu Chen, Quanfu Fan, and Rameswar Panda. Cross ViT: Cross-attention multi-scale vision transformer for image classification. arXiv preprint arXiv:2103. 14899, 2021.
[31]
Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. CvT: Introducing convolutions to vision transformers. arXiv: Comp. Res. Repository, 2021.
[32] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. In Proc. Int. Conf. Learn. Representations, 2021.
[33] Tsung-Yi Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proc. IEEE Conf. Comp. Vis. Putt. Recogn., pages 2117-2125, 2017.
[34] Yuhui Yuan, Lang Huang, Jianyuan Guo, Chao Zhang, Xilin Chen, and Jingdong Wang. OCNet: Object context network for scene parsing. arXiv: Comp. Res. Repository, 2021.
[35] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. ImageNet classification with deep convolutional neural networks. In Proc. Advances in Neural Inf. Process. Syst., volume 25, pages 1097-1105, 2012.
[36] Laurent Sifre and Stéphane Mallat. Rigid-motinn scattering for texture classification. arXiv
preprint arXiv: /403. 1687, 2014.
[37] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In Proc. Int. Conf. Learn. Representations, 2019.
[38] Gao Huang. Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Weinberger. Deep networks with stochastic depth. In Proc. Eur. Conf. Comp. Vis., pages 646-661. Springer, 64r-eerr 2016.
[39] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 1 115(3):211-252 2015.
[40] Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Design- ing network design spaces. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10428-10436, 2020.
[41] Changlin Li, Tao Tang, Guangrun Wang, Jiefeng Peng, Bing Wang. Xiaodan Liang, and Xiaojun Chang. Bossnas: Exploring hybrid cnn-transformers with block-wiselj self-supervised neural architecture search. arXiv preprint arXiv:2103.1244
2021.
[42] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ADE20k dataset. In Proc. IEEE Conf. Comp. Vis. Patt. Recogn. pages 633-641, 2017.

---

[43] Alexander Kirillov, Ross Girshick, Kaiming He, and Piotr Dollár. Panoptic feature pyramid networks. Proc. IEEE Conf. Comp. Vis. Patt. Recogn., pages 6399-6408, 2019.
[44] Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In Proceedings of the European Conference on Computer Vision (ECCV), pages 418-434, 2018.
[45] Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng. Tao Xiang, Philip H.S. Torr, and Li Zhang. Rethinking semantic segmentation from sequence-to-sequuence perspective with transformers. In CVPR, 2021.
[46] Tsung-Yi Yi Lin, Priya Goyal, Ross Girshick, Kaiming Hc, and Piotr Dollár. Focal loss for dense
object detection. In Proc. IEEE Int. Conf. Comp. Vis., pages 2980-2988, 2017.
[47] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask R-CNN. In Proc. IEEE Int. Conf. Comp. Vis., pages 2961-2969, 2017.
[48] Tsung-Yr Yi Lin, Michael Maire, Serge Belongie. James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects context. In Proc. Eur. Conf. Comp. Vis., pages 740-755, 2014.
[49] Kai Chen, Jiaqi Wang. Jiangmiao Pang. Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang. Dazhi Cheng, Chenchen Zhu, Tianheng Cheng, Qijie Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang, Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin. MMDetection: Open MMLab detection toolbox and benchmark. arXiv preprint arXiv:1906.07155, 2019.
12


Reviews:
[{"review_id": "sKewB1xWwES", "reviewer": "Reviewer_uA3x", "summary": "The paper proposes two new transformer architectures to address image classification and object detection and segmentation tasks, focusing on the attention mechanisms in each. The first method, Twins-PCPVT, combines the pyramidal subsampling of attention in PVT with conditional positional encoding of CPVT, the second, Twins-SVT, proposes to combine local grouping to manage computational complexity in attention with global subsampling to improve receptive field size, drawing analogies to separable filters.\n\nBoth methods are empirically analyzed on relevant datasets (Imagenet1k, MSCOCO, ADE20k) in comparison to state of the art methods (PVT, CPVT, Swin) and show improvements in accuracy and run-time speed.\n\n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I am inclined to rate the paper marginally below the acceptance threshold based on the following analysis:\n\nStrengths:\n- The paper addresses an important problem: improving speed and accuracy of transformer architectures by designing attention mechanisms\n- The empirical analysis appears thorough and on a wide set of relevant problems and data\n- The methods are motivated reasonably well at the coarse level (combine subsampling with conditional encoding, combine local and global grouping, but see the concerns below)\n- The empirical analysis shows numerical benefits in accuracy and speed across the tested models and data\n\nWeakness:\n- The novelty of the methods and what sets them apart from the state of the art is not clearly highlighted and (at least for me) needed several re-reads. The paper would benefit from outlining more clearly what the novelty and contribution of the paper are.\n- It is at times hard to follow the exposure: (a) Some terminology is not explained: Why are the methods termed \"Twins\", why is one PCPVT, the other SVT (l. 96ff)? The reader is left to interpret. (b) The two methods appear valid on their own, but I do not see unification (l39) beyond addressing attention mechanisms. If the authors see more arguments that the paper leads to a more coherent picture of attention mechanisms, they should point this out more clearly. As it is, the title, abstract (ll. 3-5) and introduction (ll. 39ff) left me with the impression that the paper was aiming towards coherence or unification of attention methods, but I saw little of this evidenced in the remainder of the paper.\n- While the high-level motivation appears sound (as pointed out above), the motivation for the individual choices of implementation in the Twins-SVT method (section 3.2) are not clear to me. I would like to understand why these particular choices are suitable, what are other options and why where they not chosen. I understand that the paper is primarily geared towards empirical evidence (versus theoretical motivation and analysis), but I feel that I am not learning much about why and if this is a good choice beyond the empirics.\n\nFurther comments/suggestions, these do not impact the review rating:\n- I feel that drawing the analogies to separable convolutions or separable filtering in general does not add to the paper. The motivation to bridge fine-grained, local and coarse, global information is fine by itself and does not need the motivation from separable convolutions. \n- The authors indicate deployment frameworks such as ONNX and TensorRT as a motivation from the runtime speed perspective. It would be nice to demonstrated improvements there directly (by showing feasibility and/or runtime improvements at deployment) to substantiate this motivation.\n\n\n[Discussion period update]\nI appreciate that the authors added some details based on my concerns during the review phase. I feel that my initial concerns on outlining the novelty more directly and clarifying \"unification\" and some of the terminology are reasonably well addressed. My concern on the largely empirical choices in 3.2 is dampened, but not removed. Notwithstanding this, I feel compelled to upgrade my rating. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors do not discuss societal impact, but I do not have any specific concerns.\n\nThe authors do not speculate strongly on limitations of the work. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "12"}, "extra_text": {}}, {"review_id": "dfqQLKDAmoc", "reviewer": "Reviewer_bc7u", "summary": "This paper proposes to combine CPE with PVT. In addition, local attention is combined with a subsampled global attention layer so that the model can capture both local details and global relations efficiently. The design is verified on image classification, detection and segmentation.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n## Originality\nThe Proposed Twins-PCPVT simply combines CPE with PVT in each pyramid resolution. This lacks enough novelty, except verifying the effectiveness of CPE. The Twins-SVT combines local windows attention from Swin and global attention from PVT on low resolution feature maps, but this is a relatively more interesting combination.\n\n## Strength\n1. The experiments are extensive, on three large scale datasets and tasks.\n2. The proposed method is simple and intuitive.\n\n## Weakness\n1. Table 5 lacks enough explanation. Does (L, L, L) mean three stages only? In addition, (L, LLG, LLG, G) has fewer parameters than (L, LG, LG, G). Does it mean the total number of layers is fixed?\n2. Another important baseline lacks explanation. In table 7, Swin is combined with CPVT, but is it applied to all stages as in Twins, or in the first stage only? What if the relative positional encoding is not removed?\n3. The proposed method seems to require strong regularization, e.g. a larger stochastic depth rate. The strong regularization might be due to the small spatial resolution the extra global attention layers are applied. In addition, the modified gradient clipping is claimed as especially important but lacks enough explanation. Is it also due to the extra global layer?\n4. It is not clear where the block number (in appendix table 3) comes from. For example, in Twins-SVT-S, 4 blocks are used for global attention. Is it modified from Swin for best result on ImageNet or downstream tasks?\n5. Controlled comparison (where only one module is added or removed) is not sufficient for the main Twins-SVT contribution. Although the number of FLOPS and parameters are controlled, there are always more than one difference between compared models. Maybe one could first insert all the PEGs to Swin, then add the global attention PVT layers, and then probably show that shifted window is not necessary with the global attention layers.\n\n### Typos and suggestions\n1. Line 252, 160k iterations\n2. Line 303, cannot achieve\n3. Line 163, \"key\", change to important so that it is not confused with \"attention key\"?\n4. Maybe emphasize that query is not downsampled in the global attention.\n\n------------------------ Post Rebuttal ------------------------\n\nThanks the authors for the responses. The rebuttal addresses most of my concern regarding the clarity of the paper and the fair ablation/comparison with Swin, CPE, and RPE. So I tend to keep my original rating of 6.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "1MG4v7IYL1Y", "reviewer": "Reviewer_oWv9", "summary": "This paper presents a new transformer architecture for various vision tasks. The authors revisit the spatial attention design and propose integrating LSA(local self-attention) and GSA(global self-attention) for effective feature modeling. The experiment results have demonstrated the effectiveness of the Twins on many vision tasks, including image-level classification and dense prediction.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. The paper is easy to follow.\n2. The idea of this paper is simple but effective. The author proposes to decompose global attention into two separate steps: firstly, applying local self-attention to aggregate features like the swin-transformer. Next, they propose using the summarized key to provide long-range information with low computation costs.  The most important thing of designing a transformer backbone is to alleviate the learning difficulty for the original global transformer, and the success of convolutional-based backbone gives the prior that the local interaction on features can speed up the backbone training. The idea of this paper has incorporated the design of the local window prior in convolution but uses a simple way to obtain long-range information by cross-attention. It is very cool, and I appreciate this idea.\n3. The experiments on several tasks has achieved the state-of-the-art performance, and the ablation studies give detailed analysis on the key components of this paper", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "#193~#199 has figured out the difference with Swin-transofmer. But I wonder why the authors claim the Swin-transformer is more complicated and cannot be optimized for speed on devices? Can the authors provide the evidence to support your claim? And in my opinion, the shifted window in Swin-transformer is also a straightforward idea, resulting in a simple but effective design.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This submission received 3 positive final ratings: 7, 6, 6. 
On the positive side, reviewers appreciated simplicity and effectiveness of the idea, strong empirical performance and clear presentation.
At the same time, some of them initially expressed concerns with overall novelty and motivation of certain design choices.
After an extensive discussion between the authors and the reviewers during the rebuttal period, one of the reviewers upgraded their score (from negative to positive), while others remained unchanged.
Overall, the strengths of this paper outweigh its weaknesses, so the final recommendation is to accept for poster presentation.
Decision: Accept (Poster)