Dual-windowed Vision Transformer with Angular Self- Attention
Authors: Weili Shi
Sheng Li
Venue: Accepted by TMLR ()

Abstract: Following the great success in natural language processing, transformer-based models have emerged as the competitive model against the convolutional neural networks in computer vision. Vision transformer (ViT) and its subsequent variants have exhibited promising performance in tasks such as image classification, object detection and semantic segmentation. The core  of vision transformers is the self-attention mechanism, which models the long-range dependency of different tokens. Conventionally, the attention matrix in self-attention is calculated by the scaled dot-product of \textit{query} (Q) and \textit{key} (K). In this case, the attention weight would depend on norm of Q and K as well as the angle between them. In this paper, we propose a new attention mechanism named angular self-attention, which replaces the scaled dot-product operation with the angular function in order to effectively model the relationship between tokens. In particular, we  propose two forms of functions: quadratic and cosine functions, for our angular self-attention. Based on angular self-attention, we design a new vision transformer architecture called dual-windowed angular vision transformer (\textbf{DWAViT}). DWAViT is a hierarchical-structured model characterized by the angular self-attention and a new local window mechanism. We evaluate DWAViT on multiple computer vision benchmarks, including image classification on ImageNet-1K, object detection on COCO, and semantic segmentation on ADE20K. Our experimental results also suggest that our model can achieve promising performance on the tasks while maintaining comparable computational cost with that of the baseline models (e.g., Swin Transformer).

Published in Transactions on Machine Learning Research (06/2024)
Dual-windowed Vision Transformer with Angular Self- Attention
Weili Shi
rhs@trrvinginia.ed
School of Data Science University of Virginia
Sheng
shengli Giiiggiinvigiiiia.. edu
School of Data Science
University of Virginia
Reviewed on OpenReview: https: // openreview. net/ net/forum?
forum? ? id=
oXsGN
Abstract
Following the great success in natural language processing, transformer-baset models have emerged as the competitive model against the convolutional neural networks in computer vision. Vision transformer (ViT) and its subsequent variants have exhibited promising performance tasks such as image classification, object detection and semantic segmentation. The core of vision transformers the self-attention mechanism, which models the long-range dependency of different tokens. Conventionally, the attention matrix in self-attention is calculated by the scaled dot-product of query (Q) and key (K). In this case, the attention weight would depend the norm of Q and K as well as the angle between them. In this paper, we propose new attention mechanism named angular self-attention, which replaces the scaled dot-product operation with the angular function in order to elfectively model the relationship between tokens. In particular, we propose two forms of functions: quadratic and cosine functions, for our angular self-attention. Based on angular self-attention, we design new vision transformer architecture called dinl-widdowed angular vision transformer (DWAVIT). DWAVIT is hierarchical-structurre model characterized by the angular self- attention and new local window mechanism. We evaluate DWAVIT on multiple computer vision benchmarks, including image classification on ImageNet-1K, object detection on COCO, and semantic segmentation on ADE20K. Our experimental results also suggest that our model can achieve promising performance on the tasks while maintaining comparable computational cost with that the baseline models (e.g., Swin Transformer). The source code is available https///gittab..ccoDDaaa///t///
1 Introduction
Vision transformers have received tremendous attention since its emergence. Inspired by the success of the transformer (Vaswani et al.) 2017) in the sequence modeling, Dosovitskiy et al. (Dosovitskiy et al. 2021 proposed the initial architecture of vision transformer which can be regarded the encoder part of the original transformer (Vaswani et al. 2017). Compared to convolutional neural networks (CNNs), the vision transformer is featured by its ability transform the spatial visual representation learning the image into the token-to-token learning, by partitioning the image into multiple patches. Benefited from the ability of the self-attention mechanism that can model the long-range dependence of tokcns in the image, vision transformers
exhibit on par or betler performance against. CNNs in many computer vision tasks Shi el. al. 20022 2023 ;
Zhou et al. (202bb) : Chen et al. (2023a); Wang et al. (2023a b), such as image classification Dosovitskiy
ct al. 2021 Dong al. 2022), object detection (Carion et al. 2020: He & Todorovic 2022
Zhang al.
2022a), and semantic segmentation (Zheng et al. 22021] Xie
2021). Despite the merits mentioned above,
the shortcoming of vision transformer is also obvious. The low level of the inductive bias requires more large

---

Published in Transactions on Machine Learning Research (06/2024)
datasets such as Image-21K (Deng et al. 2009) and JFT300M (Sun al. 2017 for model training. Besides, the time complexity of the self-attention computation is quadratic to the number of input tokens, which prohibits the application of vision transformers on the tasks that basically involve high-resolution images.
To deal with the excessive computation of the self-attention, the subsequent work (Liu al. 2021 Dong et al.,
2022 Huang et al. 2019 Wang et al. 2020 Xia et al., 2022) proposed different. local-window mechanisms restrict the computation of self-attention in a local window. For instance, the pioneering work Swin Transformer {Liu et al., 2021 adopts the shift windows to reduce the workload of computing self-attention
and to facilitate the interaction of local windows. CSwin (Dong et al. 2022) proposes the cross-shaped window, in which the image is split into the horizontal and vertical strips in parallel. Another work (Xia et al.
2022 presents a flexible local window, which could be implemented in a data-dependent way.
Another branch of work (Qin et al. Katharopoulos ct al., 2020: Peng et al.) Choromanski et al.) focuses on the in-depth understanding of the sell-altention mechanism and proposes new formulations to calcu- late the attention scores between different pairs of tokens. From the perspective of kernel learning, the interaction of the query and key can be modeled by specific kernel function, and the scaled dot-product operation can be replaced by the softmax-free op- eration in self-attention. Usually, the softmax-fres operation can lower the time complexity of the com- putation in self-attention.
Layer
Layert
[a]
[b)
Figure 1: The illustration of the dual window mecha- nism. The image is partitioned into (a) even number of local windows and (b) odd nunber of local windows in two layers, respectively. The size of the local window is flexible and the tokens lie on the border of one local window would reside in the interior of the local win- dow in the following layer. The connection of the local window in each layer can be bridged by the operation of the local window in the next layer.
In this paper, we present new designs on the local win- dow mechanism and the operation in self-attention. In terms of the local window, we propose a dual win- dow mechanism. As shown in Fig- [ similar to Swin Transformer (Liu et al., 2021), ; the local window is also imposed on the feature maps for the purpose of reduction of the time complexily. However, unlike the previous work in which the size of the local window is fixed, the size of our proposed local window is flexible and is adjustable according to the size of the feature maps. Besides, to mitigate the problem of lacking connections between local windows, the number of local windows is different at layer t and layer t + 1. For instance, there are even numbers of the local windows at layer but odd numbers of local windows at layer ++1 1. In this case, the tokens that lie in one local window from the first feature map would belong to another local window in the following features map. Since the feature maps are partitioned into different numbers of local windows, the coordinates of the local windows in the adjacent feature maps are different. The tokens in the overlapping area of local windows can bridge the connection of local windows since these tokens would participate in the self-attention calculation within each local window. With the interaction of the local windows, receptive fields can be enlarged implicitly, and the ability to model long range relations can also be enhanced considerably.
In traditional self-attention mechanism, the similarity of the query and key is computed by the scaled dot- product. Thus, the similarity would depend on the norm of query and key as well the angle between them. Inspired by previous work (Wang et al. 20188) Zhao et al. 2020), we notice that scaled dot-product function is not the only choice to model the relationship of tokens. In this paper, we propose the angular self-attention, in which the similarity of query and key is only dependent on the angle between them. To reduce the impact of the norm of the query and key on the relation of tokens, the query and key are L2-normalized, and query and key are distributed on the unit sphere. The relationship of the query and key would be determined by the angle between them, and a smaller angle could yield a larger attention score between a pair of query and key. In angular self-attention, we adopt two forms of functions: quadratic and cosine functions, to model this relationship, and the similarity is further enlarged by the temperature scaling. Besides, we also propose a new linear function to simplify the computation in the quadratic self-attention. Our experiments show that the angular self-attention can serve as the alternative for the traditional scaled dot-producc self-attention.
2

---

Published in Transactions on Machine Learning Research (06/2024)
Jointly combining the dual window mechanism and angular self-attention, we propose a novel hierarchical- structured vision transformer backbone called dual-windowed angular vision transformer (DWAViT). In DWAVIT, the attention score for each pair of query and key is modeled by the temperature-scaled quadratic/cosine functions, and experimental results validate that our quadratic/ /cosine functions are effective in modeling the relationship beiween tokens. Besides, the dual window mechanism is also adopted in our new backbone. The feature maps are partitioned into even/odd numbers of local windows the layers of DWAVIT alternatively. The dual window mechanism can preserve the ability model long-range relationships between tokens. With proper partition of the feature maps, our DWAVIT can be applied to the downstream tasks (e.g,, object detection, semantic segmentation) which involve high-resolution images and achieve comparable computational cost with that of baseline models (i.e., Swin Transformer (Liu et al. 2021 ) The major contributions of this paper are as follows:
We propose the dual window mechanism to split the global feature map into a couple of smaller localized feature maps. By partitioning the feature maps in even/odd numbers of local windows in an alternative way, the lack of connection between local windows can be alleviated.
We propose the angular self-attention in which the scaled dot-product operation is replaced by the temperature-scaled quadratic/ quadratic/cosine cosine functions. A linear function is also proposed to simplify the computation in quadratic self-attention. Our proposed angular self-attention can model the long range relationship of tokens and is the competitive alternative for the scaled dot-product self-attention.
The dual-windowed angular vision transformer (DWAVIT) is proposed by jointly combining the dual window and angular self-attention. The DWAVIT is evaluated on a series of dense prediction tasks and achieves competitive performance on ImageNet image classification, COCO object detection, and ADE20K semantic segmentation. With proper partition of the local windows, our model can achieve comparable computational cost as that of the baseline models.
2 Related Work
Vision Transformers. The pioneering work (Parmar et al. 2018 Wang et al. 2018 first introduced the self- attention mechanism to the computer vision field and some early work Ramachandran et al. (2019) ; Cordonnier et al. 2019) applied self-attention in the computer vision tasks. Dosovitskiy et al. (Dosovitskiy et al.) 2021 proposed the transformer-based backbone architecture called vision transformers (ViTs) (Dssovisskiy et al. 2021). With the new paradigm of representative learning, ViTs achieve on par or better performance on image classification, object detection and semantic segmentation against CNNs. Since the emergence of vision transformers, plenty of work [Touvron et al., 2021a 2022) has been done on this field and the subsequent work aims to improve the ViTs on different aspects. DeñT' [Touvron et al. 2021a 2022 proposes new training recipe to reduce the high demand of ViTs for the very large datasets. With the techniques provided by DeiT (Touvron) et al. 202 2022), ViT's can pretrained from scratch on smaller datasets such as ImageNNt-11 (Deng et al., compared to ImageNet-21K (Deng et al. 2009) and JFT300M (Sun et al.] 2017). Besides, ViTs also
2009
borrow the idea form the modern CNN archilectures (Wang et al.] 2023c He et al.) 2016 Howard et. al.
2017: Sandler et al. 2018 Tan & Le 2019;
2021 2019 Huang et al. 2017 Liu et al.) 2022b Rao et al, Wang et al. 2022a Dai et al. 2021 to improve the ability of representative learning and develop hierarchical
pyramid structure to handle the multi-scale feature maps. The pyramid- structured ViTs usually have four stage and each stage the size of the feature maps is half of that the previous stage while the dimension is doubled. Another line of work (Wu et al., 2021 Guo et al., 2022 Xíao et al. 2021 Tu al., 2022 Yuan et al. 2021 Srinivas et al. 2021 Chen et al. 2022 Mehta & Rastegari Peng al. 2021 incorporates the
convolution operation into the architecture of the vision transformers at different location. The performance of the hybrid vision transformers are further improved by fusing the local information learned by CNNs and global dependence information obtained by sell-attention. To miligate the computational cost of the global self-attention which is quadratic to the size of the input features. Some work (Tian et al.) 2023 Lee
ct al.
2022 Chen et al.) (022]) learn the contextual information from the multi-scale patch embodding.
An
extensive work (Hatamizadeh et al.) 2023 Chen et al. 2023b Hassani et al. 2023 Dong et al. 2022
Liu
et al.
2022al 2021
Xia et al. 2022
Wang et. al. 2020
Hassani et al. 2022
Han et al. 2021
H Huang al.
3

---

Published in Transactions on Machine Learning Research (06/2024)
Ren et al. 2022) proposes different local window mechanism to reduce the computational cost. The sel eol-tttention is performed within the local windows and the connection of different local window is achieved by some techniques such as shifted window (Liu et al. 2021) or cross-shaped window (Dong et al. 2022].
2019
Another line of work Shi & Li (2022 : Zhou et al. (2022a): | Mao et al. (022) suggested that vision transformers
exhibit stronger robustness to versarial atlcks that CNNs. In our paper we propose a new local window
mechanism called dual window and the connection of local windows can be achieved in a simple way.
Input image
Stage 1
Stage
Stage 3
Stage
Stem
8 Hr
" W T
42
W 32 33 KAC
Conv 3x3 sirdee2
3x3 atridem2
DWANIT Block X
DINANT Block X
DIVAVIT Block X18
DINAAIT Block 2
Pool
FC
DW Comw 2o3
Layer rorm
DWA MSA Neven (Nooa)
Layer
FFN
The illustration of our proposed dual-windowed angular vision transformer (DWAVIT). Similar to previous work our backbone adopts the hierarchical pyramid structure. The core module in our backbone is the dual-windowed angular multi-head self-attention (DWA MSA) which jointly combines the dual window mechanism and angular self-attention. In each block the feature maps are divided into even/odd numbers of local windows. Besides. The depthwise convolution provides the conditional positional embedding.
Figure 2:
Self-attention. Apart from the traditional scaled dot-product sclf-attention, different forms of self-attention mechanism are also proposed. Early work (Wang et al., 2018 Zhao al. 2020) explored the general form of the function in the self-attention and proposed several operations such as dot-product and concatenation. XeiT (Ali et al. 2021) proposed cross-covariance attention (XCA) in which the attention is performed g over channels instead of tokens. MaxVit (Tu et al.) 2022) and DaViT (Ding et al.. 2022) proposed grid attention and channel group attention, respectively. These attentions are also performed on channels dimension rather than spatial dimension. To reduce the the computational cost, the efficient sclf-attention (Han et al. 2023
Dao et al. 2022 Sun et al.] 2023 Katharopoulos el al., 2020) is proposed to approximate the traditional softmax attentotnnnon. Linear transformer (Katharopoulos et al.] 2020) suggests that softmax function can be removed and the similarity of tokens can be obtained by pure dot product of query and key. RFA (Peng et al.) and performer (Choromanski et al.) approximate the softmax attention with positive random features. CosFormer (Qin et a proposed cos-based re-weighting self-attention in which the attention score is calculated by the weighted dot-product of query and key, In SOFT (Lu et al. 2021], the dot-product similarity is replaced by the Gaussian kernel function. In our paper, we also propose a new self-attention mechanism called angular self-attention, in which the similarity of tokens calculated from a quadratic function.
3 Methodology
3.1
Dual Window
As shown Fig, [ the feature maps is partitioned into even number of local windows and odd mumber of local windows at layer t and layer t+1 alternatively. The feature maps is padded if necessary. Suppose the original size of feature map is h w. after the padding, the size of the padded feature map is h/ x w'. The number of the local window is Neven n2 side. Thus, the size of local window is Meven 2021 that bridge the connection of different local windows by complicated techniques such as cyele shilted
neven (Nodd nodd). Reven (even(Bodd)) is the number of local window per ( h X ) Compared to Swin Transformer dLiu et al. Maxen wodd
h X -
4

---

Published in Transactions on Machine Learning Research (06/2024)
local window, we solve this problem in simple way. Notice that tokens lie on the border of one local window would reside in the interior of the local window in the following layer. Therefore, the tokens on the border of the local window at one layer can participate in the self-attention calculation with the tokens from other local windows in the next layer. The dynamic interaction of the local windows can facilitate the propagation of the information between local windows, The actual size of receplive field would be larger than the size of local window and the ability to model the long-range relationship of tokens would also be enhanced.
3.2 Angular Self- Soff-Attention
Self-attention can be regarded as weighted combination of the input sequence, where the weights are determined by the similarities between tokens of the input sequence. We use O Rd to denote the generated embedding of token i from self-attention. Then the general form of self-attention could be written as:
KGQ,,K) Vj, O o. E D.-ΣE, Σ, S(Q,, K,)
where S(.) represents the similarities between Q and K and it has many forms according to the previous work [Wang et al. 2018 Zhao et al. 2020). if S(Q,, Kj) = exp(Q: K3/V /ed) Eq. would become the scaled dot-product attention as we commonly see in vision transformers. The formulation of scaled dot-product self-attention in vision transformers is Atlention(Q. K, V) Softmax( (ootmaxx(2=))1 @K-)
va.
In dot-product self-attention, the attention weight is generated from the scaled dot-product between Q and K. The dot-product of Q₁ and Kj can be expanded as Q; K, = ||Q:ll| Ky|| cos 0. indicates that the similarity would depend on the T.2 norm of Q and K well as their angles 0. In our paper, we propose angular self-attention, in which we use angular function s()) to replace the conventional scaled dot-product operation. Then the self-altention could be reformulated as:
O, O, o, -E exp(s(0e))/=)
E, exp(s(()))) V,,
where 8,₁ = arccos(Q; K2), Q and K are L2 normalized query and key, respectively. 7 is the temperature hyper-parameter that regulates the attention weight of each token.
When Q and K are normalized, they can be distributed on the surface of the unit sphere. Then the attention weight obtained from our angular self-attention is solely dependent on the angle 0. Through our training the angles 0 between different Q and K would be adjusted to model the relationship of different tokens and make the vision transformer achieve strong representative ability. Thus, we propose two alternative functions for s() in Eq. 2 They are cosine function s(0) = cos(f) and quadratic function s(0) = 1 22. In angular self-attention, the similarity of Q and K would solely depend on their angles. The matrix form of angular self-attention could be formulated as:
102
QRT
-)-)
Attention(Q, V)= Softmax
)V
T
(3)
1 10 102 V
Attention K, V) = Softmax
where θ = arccos(QRTT) Q and K are L.2 normalized query and key, respectively. To simplify the computation of Ô, a lincar function θ -- T)QRT + z is proposed to approximate arccos function, where T is the hyper-parameter that determines the closeness between the linear function and arccos function.
The intuition behind our angular-self attention is straightforward. In scaled dot-product attention the similarity score would be dependent on both the norm of the tokens as well as the angle between them. Our intuition is that the norm of the lokens may have a negalive impact on the interaction of the query and key. Thus, In our angular self-attention, we design two angular functions to model the similarity of the tokens and reduce the noisy impact from the norm of the query and key. The extensive experiments show that our angular self-attention can successfully model the relationship of the tokens and achieve promising results on different tasks such classification, object detection and segmentation.
5

---

Published in Transactions on Machine Learning Research (06/2024)
Table 1: The details of the DWAVIT variants.
Models
#Dim
# #Blocks
#Heads
4Param(MM #FLOPs(G)
DWAViT-Tiny
(66,128,266,512
[2,4,18,2]
[1,2,4,8]
22.7
4.18
DWAViT-Sni Small DWAViT-Ban
80,160,320,640
[,6,21,3
[1,2,4,8]
44.6
8.22
(96,192,384,768)
[4,8,24.4]
[1,2,4,8]
77.4
14.27
The cosine similarity and quadratic distance has common mathematical properties. Both functions are descending when 0 E [0, .T. which means that the tokens with larger angles would have less weaker relationships. 1 46 /m² when 0 E (2/2, m². - 40 / cos 0 0
Specifically, when 0 E [0, r/22, cos f = 1 - 8 /2/22 /2 which means that tokens with angles larger than π 7/2 have weaker relationships in quadratic function than that of in cos function. Our experiments suggest that on most tasks the performance of quadratic and cosine functions is comparable and the difference is very slight.
3.3 Overall Architecture
We replace the traditional scaled dot- lot-product self- attention with our angular self-attention, and inte- grate the dual window mechanism to build our dual- windowed angular vision transformer (DWAVIT). The overall illustration of our DWAVIT is illustrated in Fig. 2 Similar to the previous work (Wang et al.)
Slice
X
2021
20226
Ding et al. 2022 Fan al.) 2021 Li Liu et al. 2021 2022a), the DWAVIT
et al. 2022
Angular Attention
also adopts the hierarchical pyramid structure that take advantage of the multi-scale resolution of fea- ture maps for the dense prediction task. The size of the input image is И W x 3. Instead of adopting the convolutional layer with large kernel, we follow the work (Xiao al., 2021) and leverage the two stacked convolutional layer the stem to generate patch embedding. For each convolutional layer, the kernel size is 3 x 3 and the stride is 2 x 2. The size of the output from the stem is H W
0
Q
K
F
Localized
Localized
T T x C.
The DWAVIT consists of four stages in which the size of the feature maps is half of that from the previous stage while the dimension doubled compared to that from the previous stage. Between two adjacent stages we adopt a convolutional layer with kernel size of 2 x 2 and stride of 2 to downsample the feature maps. Each stage consists of multiple blocks which include the depthwise convolution (Chollet) 2017) that generates the conditional positional embedding (CPE) (Chu al. (0211b the dual-windowed an- gular multi-head self-attention (DWA MSA) and feed-forward network (FFN). Compared to absolute positional embedding (APE) (Vaswani et al. 2017))
Linear Projection
X
Figure 3: The illustration of the pipeline in the dual- windowed angular multi-head self-attention (DWA MSA). The Q K and V are localized by dividing them into couple of local windows. The scaled dot-product operation is replaced by the temperature-scaled angular function in the calculation of attention matrix.
that could only provide the positional information for the fixed length of sequence, The CPE can provide flexible positional information adaptive to various length of input sequence that is often seen in the down- stream tasks. Relative positional embedding (RPE) [Liu et al. [2021] [hha al. (018)) provides the relative positional information within the window. However, since the size of the window is different each stage, RPE [Liu et al. 2021 Shaw et al. 2018) would not be adopted in DWAVIT.

---

Published in Transactions on Machine Learning Research (06/2024)
The dual-windowed angular multi-head self-attention (DWA MSA) serves the core function for our backbone. It jointly combines the dual window mechanism and angular self-attention. The details of DWA MSA is illustrated in Fig. 3 Suppose the input feature is X E RixwxD, .N- = Neeen or Nodd is the number of local windows and n = VN is the number of local windows per side. After lincar projection, we obtain query, key and value Q, K, V E R.xxxDD Instead of splitting the X into smaller local windows, we split Q and K into N local windows. The size of the local window is k' w'. The N' and w' are the height and width of the x
n n.
padded feature maps. In each stage, the partition with even/odd number of windows take turns and the value of Neven Neven(Noda) (Nods) vary according to the size of the feature maps. After the partition, for tth head, we obtain the localized query Q* = (Q. Qa...., 2% and key K* = [K7 K2 K ). the localized query and key are L2 normalized and the angle matrix @ is calculated from the normalized query and key. Next the new embeddinn is calculated from the localized query, key and value by Eq. 3 The new feature maps of each local window is obtained by the concatenation of the embedding from each head X, = concat , .., x. K is the number of heads. We concatenate the feature maps of each local window form the complete feature map X- = concat(X1, (X1, X2, ..., Xx). The size of X is larger than that of the original feature map. To restore it, the final feature map is obtained from X with slice operation:
X = [topp= h bottom, left : w - right],
(4)
where top, bottom, left and right are the size of padding on the top, bottom, and right of the feature maps, respectively. With all the components aforementioned above, the pipeline of the block in our DWAVIT can be formulated as:
Z' DWConn(X*-)+
X₁ = DWA-attenniion(NN(()) + +z²
(5)
X₂ = FFN(LN(())) +X,
where Xt-1 denote the output feature from the 4t block in the backbone.
In our DWAVIT, there are strong connections between the two key components: angular self-attention and dual local window mechanism. On one hand, The dual window mechanism can confine the operation of self- attention in localized areas. On the other hand, our empirical study suggests that our angular self-attention can achieve better performance with our dual local window techniques than that with previous local window techniques (i.c., Swin Transformer (Liu et al. 2021 for two reasons. First, the partition of the dual local window is implemented on the queries and keys, not on the feature maps directly, which would be helpful for the direct interaction of the tokens. Second, the size of our local window is flexible. And we can choose the optimal window size for tasks with different input resolutions such as object detection and sogmentation. To wrap up, the two proposed techniques have strong mutual connections and are two indispensable components.
3.4 Architecture Variants
We build three variants of DWAVIT with different numbers of parameters and FLOPs, namely DWAVIT-Timg DWAliT-Small and DWAViT-Base. For all the variants, the number of local windows in each stage is set to 4100,49), (49,16), (4.1), (1,1) in image classification, respectively. In stage one, the size of the local window would be 6 x 6 and 8 8 in two consecutive blocks. For the downstream tasks such as object detection and semantic segmentation, since the size of the input image is larger, the number of local windows in DWAVIT is also different. The details of three DWAVIT variants are illustrated in T
3.5 Time Complexity Analysis
Suppose the original size of the feature map is h x w and the dimension is C. After padding the size of feature maps becomes h' x w'. The total number of local windows is N. The time complexity of the linear projection is 4 Since the self-attention is performed on the padded feature maps, the time complexity for the self-attention calculation is 2(1/ C//N. Thus, the total time complexity of our DWA MSA is:
Ω(DWA MSA) 4hwwO2 2(1 C//N.
(6)
7

---

Published in Transactions on Machine Learning Research (06/2024)
As illustrated in Eq. 6 In order to reduce the computational cost, should choose large value of N while keeping the / and w as close as to the original value. Note that though angular sell-attention includes some operations like arceos and L2 normalization, they do not increase the time complexity. However, the memory demand of angular self-attention is larger than that of traditional self-attention.
Shifted Window Attention (Swin)
Crose-shaped Window (CSwin)
Window//Grd Attention (MaxViT)
Dual Window Attention (DAWVIT)
Figure 4: The comparison our dual local window mechanism with the prior work such as the shifted local window and the cross-shaped window.
3.6 Theoretical Analysis
As aforementioned, propose quadratic and cosine functions to model the relationships of tokens. Compared to the scale dot-product function, the difference of our method is that we map the Q and K fcatures on the unit sphere and the relationship of tokens is only dependent on the angles between them. To better understand the angular self-attention, essential to investigate the relationship of tokens in our method. Thus, we provided the Proposition to analyze this problem.
Proposition 1 Suppose the angles between the query of a token and the keys of all the tokens match 8o s 01 S ... - 0111. J is the total number of the tokens. In angular self-attention, the embedding of the token O, Vo + w1 V1 + +.. wJ-VV-- where wi exp( exp() (0, 1). s(O) = 40º self-allention and s0) = cos 0 for cosine self-allention.
x2 for quadratic
The proposition states that the embedding of a token can be regarded the combination of the value vectors with the relative weight denoted by wa- The relative weight of the value vector with the smallest. angle to the target query is normalized to 1. And the relative weight of value vector is smaller if the corresponding angle is larger. The proposition suggests the larger contribution of the value vector to the embedding of the target token with smaller angles between them. The proof can be found in Appendix.
3.7 Comparison with Prior Work
The local window mechanism has been widely adopted in various vision transformers. In different vision transformers, the local window mechanism is featured by their unique partition of the feature maps well the dynamic interaction of the tokens from different local windows. Fig. 4 illustrates our dual local window well as the previous local window mechanisms.. Swin Transformers (Liu et al. 2021) first proposed the shifted local window to reduce the time complexity of the vision transformer. To tackle the lack of the connection of the local windows, some techniques such as cyclic shift adopted adjust the partition of the local windows and the connection can be bridged in the alternating layers of the model. Another local window mechanism called Cross-Shaped Window (Dong al. 2022) is proposed by CSwin Transformers (Dong et al.

---

Published in Transactions on Machine Learning Research (06/2024)
Different from the local window partition in Swin Transformers (Luu et al. 2021 the feature map is splitted into vertical and horizontal strips in parallel and the width of the strips would determine the level of connection of tokens. Recently, Neighborhood Attention (Hassani et al. 2023 proposed the efficient and scalable sliding local window mechanism in which the attention operation is confined in a small sliding local window. Besides, MaxViT (Tu el al. 2022) proposed block attention and grid attention, respectively. The block attention performs the self-attention in the local window while the grid attention would attend to tokens globally. Furthermore, DaV (Ding et al. 2022) also introduced channel group attention remedy the lack of connection between the spatial local windows.
2022
In our dual local window, the feature map is partitioned into local windows of different sizcs at cach layer. Compared to the prior local window mechanism such MaxViT (Tu et al. 2022) that deals with connection of local windows by grid attention, the lack of connection of local windows is mitigated by the overlapping part of the local windows in the alternating layer of the model. Besides, our dual local window enjoys three merits that are not possessed by prior work such as Swin Transformers (Liu et al., 2021) and CSwin Transformers (Dong et. al., 2022. First, the partition of the local window is implemented on the queries and keys, not on the feature maps directly, which would be helpful for the direct interaction of the tokens. Second, the size of our local window is flexible. And we can choose the optimal window size for tasks with different input resolutions such as segmentation. Last but not least, our dual local window is easy to implement.
Self-attention mechanism is the core of transformers and previous work (Wang et al. 2018 Zhao et al. 2020) has shown that in the attention mechanism the similarity can be modeled by different functions and scaled dot-product function not the only choice to model the relationship of tokens. To alleviate the large computation complexity in scaled dot product self-attention, XciT (Ali et al. 2021) proposed cross-variance attention in which the attention scores are obtained from the cross-covariance matrix computed over the key and query. Besides, Qin et. al. (Qin et al.) proposed a new variant linear transformer called Cosformer (Qin lt al.). The core of Cosformer iQin et al.) is the linearized self-attention in which the similarity score is modeled by the linear kernel function (c.g., ReLU). Furthermore, Lu ct.al. (Lu et al.] 2021) proposod a softmax-free transformer (SOFT) (Lu et al. 2021). In SOFT (Lu el al. 2021) the Gaussian kernel is adopted to model the interaction of the tokens and a novel low-rank matrix decomposition algorithm is proposed for approximation to reduce the computation overhead. In our angular self-attention, we suggest that the norm of the tokens play an insignificant role in the self-attention and the relationship of the tokens can be solely determined by the angles between them. As a consequence, we propose the angular self-attention in which the attention scores are obtained from our angular function and the impact from the norm of the tokens can be diminishod. In order to reduce the computational cost, the linear function is adopted to approximate the time-consuming operation the angular self-attention. The extensive experiments suggest that our angular self-attention can also successfully model the relationships of the tokens and serve as the competitor for the scaled dot product attention and linearized attention.
4 Experiments
In this section, we evaluate our proposed DWAVIT on ImageNet-1K, 1K (Deng et al. 2009g classification, COCO (Lin et al. 2014) object detection, and ADE20K (Zhou et al.] (017) semantic segmentation. Besides, we also implement the ablation study to investigate the effectiveness of angular self-attention and compare the results of angular sell-attention against that of traditional scaled dot-product self-attention on benchnnrrks
4.1
ImageNet-1K 1K Classification
In this experiment we adopt the same training recipe as previous work (oovvron et al., 2021a Li et al. 2022 Lee et al. 2022 Dong et al. 2022) for fair comparison. The training strategies include repeated data augmentation methods and the EMA (Polyak & Juditsky 1992). The total training epoch is 300 with the first 20 epochs as warm-up. We adopt the Adamn <Kingma & Ba 2014] algorithm to optimize the model. The initial learning rate is 1.2e-3 and the weight decay is 0.05. The learning rate is adjusted according to the cosine learning rate schedule. The drop path rate is 0.1 and the input image resized to 224 x 224. The mlp ratio for all the DWAVIT variants is set to 4. The number of windows in each stage is (100,49), 169, (4,1), (1,1). The temperature in angular self-attention is 0.1 for DWAVIT-T and DWAVIT-S
9

---

Published in Transactions on Machine Learning Research (06/2024)
Table 2: The performance of our proposed DWAVIT and the baseline models on the ImageNet- 1K classification. The DWAVIT consists of the angular self-attention and dual local windows. The resolution of the image is 224 x 224. The inference time is evaluated NVIDIA A100 when the batch size is fixed at 100. [cos] and (quad] denote cosine and quadratic function, respectively.
Model
#Param(M)
#FLOPs(G)
#Inference(S)
CNN VIT
Top-I
Ace
ResNet-50 (He et al.
2016
25.0
4.1
205.01
76.2
DeiT-8
Touvron el al
2021a
22.1
4.5
201.16
79.8
PVT-S
Wang
2021
24.5
3.8
217.51
79.8
R-gNetY-44G Radosavovic
2020
21.0
4.0
194.54
80.0
CrossViT-3 CChe al..
202
208.11
81.0
TNT-S
Han et al.
2021
23.8
5.2
213.60
81.3
Swin-T
Liu ot al.
2021
28.3
4.5
211.91
81.2
CoAtNet-O
Dai et al 202
25.0
4.0
214.64
81.6
CvT-13
Wu al.
202
20.0
4.5
213.26
81.6
Cafl
XS-24
Touvron
2021b
26.6
5.4
213.11
81.8
ViL-S
Zhang cl
2021
5.1
82.0
PVTv2-B2
Wang et al
2022bb
25.4
4.0
218.47
82.0
ConvNeXt-1
Liu ct al
2022b
29.0
5.0
203.89
82.1
HiVTT-T Zhang et a...
2T22h
19.0
4.6
213.71
82.1
DaVIT-T
Ding et
2022
28.3
4.5
218.67
82.8
MViTv2-1 Li
2122
24.0
4.7
217.27
82.3
CSWin-T Dong
2022
23.0
4.3
210.46
82.7
DWAViT=T coos] (Ours]
22.7
4.2
216.24
82.7
DWAVIT-T(quaa (Ours)
22.7
4.2
216.90
82.8
RusNect-101 (He et al.
2016
45.0
7.9
210.12
77.4
PVT-M WWan et
2021
6.7
222.54
81.2
RegNetY-SG
Radosavvviic
al
2020
201.64
81.7
Swin-S
[Tu et al.
2021
49.6
8.7
216.56
83.1
CoAtNet-1
Dai et
2021
42.0
8.0
218.24
83.3
CvT-21 Wu al.
121
32.0
7.1
218.60
82.5
ViL-M Zhany
2021
39.7
9.1
208.04
83.3
PVTv2-B3 Wang
2022b
45.2
6.9
226.60
83.2
ConvNeXi-S Liu
20226
207.71
Focal-S Yang et al.
2021
9.1
230.70
83.5
HiVIT-S Zhang ct ml
2022b
38.0
9.1
221.46
83.5
MViTv2-S [Li ct
7222
35.0
7.0
224.71
83.6
CSWin-S Dong
2022
35.0
6.9
213.72
83.6
DWAVi1-S|
Jurs]
44.6
8.2
218.81
83.5
DWAVIT-SS DWAVIT-S[quad (Ours)
8.2
219.38
83.6
RusNet-152 (He el al.
2016
60.0
11.0
216.37
78.3
PVT-L Waan al.
2021
61.4
9.8
227.47
81.7
DeiT-B Touvron
202
86.7
17.4
206.01
81.8
Swin-H
87.8
15.4
219.12
83.4
ViL-B Zhang et al.
2021
13.4
210.63
83.22
Foral-B
Yang et al.
2021
89.8
16.0
235.18
83.8
HiVIT-B (Zhang et al
2022h
66.0
15.9
225.62
83.8
DWAVIT- (Ours
77.4
14.3
220.36
83.9
DWAViT-Blqud (Ours
77.4
14.3
223.21
83.8
10

---

Published in Transactions on Machine Learning Research (06/2024)
Table Object detection and instance segmentation performance of our model and the baseline models with Mask R-CNN framework. The DWAVIT consists of the angular self-attention and dual local windows. The model is trained with a 3x scheme. The FLOPs are measured at resolution 800 x 1280. The inference time is evaluated on NVIDIA A100 when the batch size is fixed at 100. [cos] and [quad] denote cosine and quadratic function, respectively.
Backbone
(Mrami(M)
#FLOPs(G)
#Inferenan(s)
AP
AP%
A/
APm
AL
ResNet-50
2016
44
260
138.23
41.0
61.7
44.9
33..1
58,4
40.1
ConvNeXt-TT
Lit
2022b
48
262
163.01
16.2
67.9
50.8
-11.7
05
14.9
PVT-S
WWing
2021
41
215
177.20
13.0
65.3
46.9
39.9
62.5
12.8
Vil-8
Taang et al
2021
-55
218
47.1
68.7
51.5
42.7
65.9
46.2
TwinsP-5
Jhu ml
20721
41
245
16.8
69.3
51.8
42.6
06.3
16.0
Twins-S S
44
228
46.8
51.2
42.6
66.3
45.8
Swin-T
Lit ot al
2021
48
264
194.50
46.0
68 2
50.2
41..6
65.1
44.8
Focal-T
Yang e5 6.
2022
19
291
17.2
69: 1
51.9
12.7
06.5
15.9
PVTV2-112
Wang
21/22h
300
210.43
47.8
69.7
43.1
06.8
46.7
XeTT-S12 All
43
050
233.54
47.0
51.7
-2.33
06.0
45.4
DaVIT-T
Ding pi
21122
48
263
244.08
47.4
69.5
52.0
42.9
66.8
46.4
DAT-T
Nia et al.
2122
48
272
47.1
69.2
51.6
42.4
66.1
45.5
MVTTv2
206.42
67.9
NAT-T
Haseani
22123
48
258
157.
47.7
52.6
42.6
66.1
45.9
GC VTT-T
Enttmmzddee 16 m..
20123
48
201
240.19
47.9
70.1
52..8
43.2
67.0
46.7
DWAVIL"T pDs (Ohms)
42
255
223.46
48.4
70.4
533.1
43.5
67.7
47.1
DWAviT-T/quadl
(Ouns)
42
255
235.47
20.7
68.1
47.1
Rest0l
He
2016
63
336
155.50
42.8
63.2
38.5
00.1
41.33
ConyNeXi-S
Lau ot
2122h
70
348
225.28
47.9
70.0
52.7
42.9
66.9
46.2
PVT-M
Wang et al.
2021
64
302
222.18
44.2
66.0
48.2
40.5
63.1
43.5
ViL-M
Maang
ml 21/21
60
261
44.6
40.7
63.8
43.7
TwinsP-15
2021
64
302
47.9
70.1
52.5
43.2
67.2
46.3
Twins-B
C mm ai.
21/2
76
340
48.0
69.5
52.7
43.0
60.8
46.6
Swin-S
Lau ml.
2122
69
354
222.31
48.5
70.2
533.5
43.3
67.3
46.6
Focal-8
Yang ml.
20121
401
43.8
67.T
47.2
PVTv2-B3 B3 Wann
21722b
65
3997
265.997
48.4
69.8
533.3
43.2
66.99
46.7
X227-M2478 Aii ct al
21/1
99
1448
467.60
48.5
70.3
53.4
43.7
67.5
46.9
NAT-S
Hassaut ml
2023
70
330
201.14
18.4
69.8
53.2
43.2
06.9
16.5
D/A [Ours]
DWAVII--[quad] (Ours)
61
338
282.52
19.1
70.8
53.5
44.0
68.2
17.4
X101-64
Xie et al
2017
101
493
207.68
44.4
64.9
48.8
39.7
61.9
42.6
PVT-L
Wang DE a.
2021
81
364
286.52
44.5
66.0
48.3
40.7
63.4
43.7
Swin-B
Lau al
107
496
303.74
48.5
43.4
06.8
46.9
DaVIT-B
Ding
107
491
461.84
49.9
71.5
54.6
44.6
68.8
47.8
DWAVI
pos
(0irs)
97
462
423.62
40.8
71.2
54.88
44.5
68.6
47.8
DWAVTT- B[qudd]
(Ours)
97
462
440.69
49.4
71.1
54.6
44.4
68.6
47.8
and 0.25 for DWAVIT-B, respectively. The linear function is adopted simplify the computation of the quadratic self-attention and is set to 0.4. All the experiments running NVIDIA A100. To evaluate the time efficiency of our proposed model, the inference time is also reported. Since the model is trained TmageNet-1K for many training epochs (300 epochs), the variance could be diminished considerably and become insignificant compared to the main results. Thus, only the major results are reported.
The results are illustrated in Table 2 Our proposed DWAVIT is compared against the previous state-of- the-art vision transformers and CNNs including the CSwin Dong et al., 2022), MViTv2 (Li et al., 2022). ) DaVIT (Ding al. 2022] and ConvNeXt (Liu al. 2022b). Specifically, Swin Transformer (Liu al., 2021), Focal Transformer Yang et al. 2021), ,, Da VIT (Ding et al. 2022). MVTTv2 (Li et al. 2022 and CSWin Transformer (Dong et al.) 2022 are baselines for the exact comparison. The experimental results show that under the similar amount of parameters, the DWAViT-T can outperform the latest vision transformers and CNNs, The top-1 accuracy of the DWAViT-T can achieve 82.8%, which is even 0. 0.1% higher than that of CSwin-T (Dong et al. 2022). As for the small-sized model, our DWAVIT-S can achieve the top-1 accuracy of 83.6% in the classification task, which on par with that of CSWin (Dong al. (022) and MV1Tv2 [Li et al. 2022). For baso-sized models, our DWAViT-B with cosine self-attention can achieve 83.9% accuracy the ImageNet-1K classification task.
11

---

Published in Transactions on Machine Learning Research (06/2024)
Table 4: The semantic segmentation performance of DWAVIT-T and baselines on ADE20K, The FLOPs are calculated with resolution 512
2048. [cos] and [quad] denote cosine and quadratic function, respectively.
Backbone
#Parrm(M) #FLOPs(G)
mloU
Swin-T Liu et al.
2021
59
945
44.5
Focal-T Yang et al.
2021
62
998
45.8
XcfT-S12/ 16 [Ali et
2021
54
966
45.9
XciT-S12/8 Ali et al.
2021
53
1237
46.6
DaViT-T [Ding et al.
2022
60
940
46.3
DAT-T Xia et al.
2022
60
957
45.5
GC ViT-T Hatamizadeli et al.
2023
58
947
47.0
NAT-T Hassani et al.
2023
58
931
47.1
DWAVIT-T (Ours)
52
930
47.5
DWAViT-Tiquad (Ours)
52
930
45.4
4.2 COCO Object Detection
Next, evaluate model on the COCO object detection task. The COCO dataset has 118K images for training and 5K images for validation. We adopt the Mask R-CNN (He et al. 2017) ) and Cascade Mask R-CNN (Cai & Vasconcelos 2018) as the framework and our DWAVIT serves as the backbone. For fair comparison, we follow the same training recipe as the previous work (Touvron et al. 2021a Li et al. 2022 Lee
iet al. 2022 Dong et al. 2022) ) and perform the experiment with MMDeteectin toolbox (Chen et al.) 2019
In order tackle the images with high resolution. The number of local windows in the object dctection task is different from that in the image classification task. The number of windows in each stage is (256,225), 464,49), (16,9), (4,1), respectively. In both frameworks, the size of the local window in each stage is half of that in the previous stage. We use the model pretrained on ImageNet-1K3 and fine-tune it the COCO dataset with 1x and 3 x schedule with 12 and 36 epochs, respectively.
The results on object detection and instance segmentation of our our model and the baseline models with Mask R-CNN (He et al. 2017) framework with 3x schedule are illustrated in Table The baseline methods include the latest ViT models such as MViIv2 (Li et al. 2022). DAT (Xia et al.) 2022) and DAVIT (Xial et al. 2022) ). SSpecifically, Swin Transformer (Liu et al. 2021). Focal Transformer (Yang et al. 2021). XciT (Ali ct al. 2021), DAT (Xia et al., 2022) and CSwin (Dong et al.) 2022) are basclines for exact comparison. The experimental results show that the DWAVIT can achieve methods. For DWAViT-T, the AP can achieve 48.8%, which is 0.4% higher than that of MViTv2-T (Li ct al. 2022). the APm of DWAViT-T is 43.8%, which is on par with that of MViTv2-T (Li ct al. 2022). The DWAViT-S achieves 49.1% on APh and 44.4% on APm, which outperforms all the baseline methods. And the experimental results suggest that our DWAVIT-B can outperform the Swin-B [Liu et al. 2021 on this task. More results can be found in Appendix.
par better result with latest baseline
4.3 ADE20K Semantic Segmentation
In this section, we further investigate the performance of our proposed model on semantic segmentation task. 2018) framework is adopted. Our model and the baseline methods are evaluated
The Upernet (Xiao et al.
on benchmark ADE20K Zhou et al. 2017). For fair comparison, we follow the training procedure from previous works (Ding et al.
2022 Dong al. 2022) and perform the experiment with MMScgmentation toolbox (Contributors] 2020). The image is resized to 512 x
512 and train the model with 160K iterations. the mloU is adopted as the metric. The results of experiment
illustrated in Table 4 and Table 15 (see Appendix). Swin Transformer (Luu et al. 2021), Focal Transformer (Yang et al. 2021 XciT (Ali et al. 2021), ViT (Xia et al. 2022) and DAT (Xi et al.) 2022) are the basclines for exact comparison. Those methods are also implereenedd with MMSegmenlation Loolbox (Contributors)) 2020). Besides, Upernet (Xiao et al. 2018) is adopted as the framework and models pre-trained on ImageNett IK only are used as the feature extractor for the basclines and our method. The results show that our DWAVIT with cosine self-attention function can outperform the basclines. Besides, the performance of our model with cosine function is also much better than that of our model with quadratic function. The mIoU of DWAViT-T reach 47.5%.
12

---

Published in Transactions on Machine Learning Research (06/2024)
Table The performance of our proposed DWAVIT with angular sclf-attention, scaled dot-product self- Table 6: The performance of our proposed DWAVIT attention and the linearized self-attention the with the dual local window, window /rrid attention ImageNet- 1K classification. The resolution of the and the shifted local window the ImageNet- image is 224 x 224.
classification. We adopt the quadratic self-attention
and the resolution of the image is 224 x 224.
Model
Param(M)
FLOPs(G)
Top-1 Ace
Bockbooue
Param/M) FLOPs(G)
Top- Are
Tdot product]
22.7
4.2
82.5
DWAVIT-T[linedl
22.7
4.2
82.2
DWAVEI-T[ssaite (shhifted window]
DWAViT-T window/gri grid)
22.7
22.7
4.2
82.5
DWAVIT-TIoar (Ours)
22.7
4.2
82.7
DWAVIT-T ''ddull window] (Ours)
22.7
4.2
82.8
DWAVIT-T[quad] (Ours)
22.7
4.2
82.8
Table 7: Object detection and instance segmentation performance of our DWAVIT with angular self-attention, scaled dot-product self-attention and linearized self-attention in Mask R-CNN framework. The model is
trained with a 3x scheme. The FLOPs are measured at resolution 800 x 1280.
Backbone
# #Param(M)
#FLOPs(G)
AP
AP%
AP%
APm
AP
A Pm
DWAVIT-T product]
255
70.3
53.2
43.5
67.3
47.1
DWAViT-T"Hl [linear)
42
255
48.0
70.0
52.8
43.2
67.0
46.5
DWAVIT-T (cco) (Ours)
255
70.4
53.1
43.5
67.7
47.1
DWAVIT-T [quad]] (Ours)
42
255
48.8
70.7
53.6
43.8
68.1
47.1
Table Object detection and instance segmentation performance our DWAVIT with the dual local window, shifted local window and window /grid allention Mask R-CNN framework. The model is trained with a
scheme. We adopt the quadratic self-attention and the FLOPs are measured resolution 800
1280.
Backbone
# #Param(M)
#FLOPs(G)
AP
AP
AP
APm
AP
AP
TWAViT-T/shifter window]
259
47.5
69.1
52.3
42.3
66.3
45.9
(TWAV/-Twwwidww///// DWAViT-T(aa window] (Ours)
259
47.8
70.2
52.7
42.9
66.7
46.3
42
255
48.8
70.7
53.6
43.8
68.1
47.1
Table The performance of Swin-T, DeiT'S, CSwin-T and MaxViT-T with angular self-attention and scaled dot-product self-attention the ImageNet-1K classification. The resolution of the image is 224 x 224. The dot product, cos and quad in the square bracket denote scaled dot-product, cosine and quadratic function, respectively.
Model
Param(M)
FLOPss()
Top-1 Acc
Swin-T
LLuu et al.
2021
dot product]
28
4.3
81.2
Swin-T
Liu et al.
2021
[cos] (Ours)
28
4.3
81.4
Swin-T
Liu et al.
2021
quad] (Ours)
28
4.3
81.3
DeiT-S
Touvron et al. 2021 g
dot product]
22
4.6
79.8
DeiTT-
Touvron et a..
202 &
[cos] (Ours)
22
4.6
80.0
DeiT-S
louvron et al.
2021a
quaad] (Ours)
22
4.6
80.2
CSwin-1 T
Dong et al.
2022
dot product
23
4.3
82.7
CSwin-T Dong et al.
2022
[cos] (Ours)
23
4.3
82.9
CSwin-T (Dong et al. 2022
quad] (Ours)
23
4.3
83.0
Max ViT-T
(TTu et al. 2022
dot product
31
5.6
83.6
MaxViT-T
TTu et al. 2022
[cos] (Ours)
31
5.6
83.8
MaxViTs (Tu al.
2022
quad] (Ours)
31
5.6
83.8
which outperforms other bascline methods like DAT-T Xia et al. XciT-S (Ali et al. 2021
2022 3 DaVIT-T (Ding et al.
2022 and
13

---

Published in Transactions on Machine Learning Research (06/2024)
40
2
mHOU
0.2
0.4
0.6
9.8
100
0.2
0.4
0.6
0.8
100
02
0.4
0.6
0.8
100
0.2
0.4
0.6
0.6
1.0
(a) classification
(b) object detection
(c) object detection
(d) segmentation
Figure 5: The performance of our model with different values of X on classification, object detection and segmentation lasks.
Table 10: The actual rumning time of Swin Transformer and our DWAVIT during the training ImageNet- -K We implement the testing on single A100 GPU and the batch size is fixed to 100 for all the models. The average time per iteration is reported.
Model
# Param(M)
#FLOPs(G)
Inferencee (x 10-2 8)
Training (s)
Swin-TT [Liu et al.
2021
28.3
4.5
4.6
0.15
DWAViT-1 coos|
Ours)
22.7
4.2
3.6
0.20
DWAVIT-TE T [quad]
(Ours)
22.7
4.2
4.0
0.20
Swin-S (Liu et al.
2021
49.6
8.7
7.1
0.23
DWAVIT-S/coss
urs
44.6
8.2
6.9
0.29
DWAViT-S/quad
(Ours)
44.6
8.2
7.4
0.31
Swin-B (Liu et al.
2021
87.8
15.4
9.2
0.30
DWAVIT-B coos [Ours]
77.4
14.3
9.6
0.40
DWAdiT-B[quad (Ours)
77.4
14.3
10.3
0.42
4.4 Running Time Analysis
In this section we quantitatively evaluate the actual running time our model in both training and inforence stages on the image classification lask. Specifically, we fix the batch to 100 and compare the running time of our model with that of Swin Transformer (Liu al.] 2021). All the experiments are implemented on a single A100 GPU and report the average time per iteration and the results illustrated in Table 10 The experimental results suggest that our model could achieve comparable inference time with that of Swin Transformer [Liu et al.) 2021). Specifically, compared to Swin-T (Liu et al., 2021), our DWAViT-T(cos) can achieve 1.5% higher accuracy on image classification with lower time cost during the inference stage. Our DWAViT-B(cos) can achieve an improvement of 0.5% accuracy on the classification task with almost the same inference time Swin-B Liu et al. (2021). Basically, the time cost of our model with quadratic self-attention would be slightly higher than that of our model with cosine self-attention. Besides, the training time of model is slightly higher than that of Swin Transformer (Liu et al. 2021) due to the new formulation in self-attention. However, the actual running time does not blow up when the size of our model scales up.
4.5 Ablation Study
In the ablation study we first. compare the performance our model with traditional scaled dot-product self-attention and our proposed angular self-attention. In the experiment we replace the angular self-attention with traditional scaled dot-product self-attention linearized self-attention and evaluate the performance of the model ImageNet-IK classification, COCO object detection and ADE20K semantic segmentation. We adopt the DWAViT-T as the backbone in three tasks. The results of the image classification and objection delection are illustrated in Table and Table 7 respectively. More results on other lasks can be found in Appendix. When our model adopts the traditional scaled dot-product self-attention, the top-1 accuracy would be 0.2% and 0.3% lower than that of cosine function and quadratic function. On the object detection task our angular self-attention also achieves better performance than that of the baseline methods, We further investigate the performance of angular self-attention by replacing the scaled dot-product self-attention
14

---

Published in Transactions on Machine Learning Research (06/2024)
with our angular self-attention in other vision transformer models. Table 9 shows the performance of Swin-T (Liu et al. 2021), DeiT-S (Touvron et al. 2021a), CSWin-T (Dong et al. 2022 and MaxViT-T (Tul et al. 2022) ImageNet-1I image classification with scaled dot-product self-attention and our proposed angular self-attention. The experimental results suggest that when the existing vision transformer models are equipped with our angular self-attention, the performance on the image classification can be improved. Tt further validates that our angular self-attention is a competitive alternative for the scaled dot-product self-attention.
We also compare the performance of our model with the proposed dual local window with the shifted window proposed by Swin Transformer (Liu et al., 2021) and the window /grid attention from MaxViT (Tu et al. 2022). In the experiment we replace the dual local window with the shifted local window and window/st /grid attention and evaluate the performance of the model on ImageNet--IK classification, COCO object detection and ADE20K semantic segmentation. We adopt the DWAViI-T(quad) as the backbone in three tasks, The results of the image classification and object detection are illustrated in Table ( and Table g More results on other tasks can be found in Appendix. When our model is equipped with the shifted local window, the top-1 accuracy would be 0.4% lower than that of the model with our proposed dual local window. On the object, detection task, our dual local window also achieves better performance than that of baseline methods, which suggests our dual local window is strong competitor to the shifted local window and window/grid attention.
At last, We investigate the effect of the hyper-parameter on the performance of our model. The temperature scaling coefficient À is the primary hyper-parameter in our angular self-attention and it determines the discriminativeness of the similarity score of tokens. We adopt DWAVIT-T(quad) as the target model and choose 0.1, 0.5,1.0 as the value of X. The performance of the model is evaluated on image classification, object detection and semantic segmentation and the results are shown in Fig 5 The results suggest that the performance of our model would be reduced with larger value of X. Compared to the classification task, the performance of our model is more sensitive to the change of the value of X. Since the value of À become larger, the similarity score of the tokens would become less discriminative. And the ability of our angular self-attention to model the interaction of tokens would be undermined. As a consequence, the performance of the model would drop, especially on object detection and segmentation lasks.
4.6 Statistical Significance
To validate the statistical significance of our results on various tasks (classification, object detection and segmentation), we perform the significant test on our proposed DWAVIT model and choose one important baseline model (e.g., Swin Trans- former (Luu et al.) ) for the comparison. In image classifica- tion the Swin-T (Liu et al. 2021] is adopted in our test. The av- crage accuracy of Swin-T (Liu et al. 2021) is 81.20% as reported in the original paper. To demonstrale rate that the performance improvement of our proposed model is statistically significant over that of Swin Transformer. We first choose DWAViT-T as the target model since it has similar size with that of Swin-T. Our mull hypothesis is (cc(DWAVTT-T) acc(Swin-T) and the alternative hypothesis is (DWAVITTT) > acc(Swin-T). Next, the evaluation of our model on image classification is performed multiple times (e.g., 20) under different settings. At last, the t-score and p-value are calculated according to the one-tail T-test and the result is shown in Table II The p-value is 7.65e-50, which is significantly smaller than the threshold 0.05. For object detection and segmentation tasks, we follow the same routine and the results are also shown in Table [1 The results suggest the rejection of the null hypothesis and the improvements of our proposed model are statistically significant.
Table 11: The results of the significant test on classification, objection detection and segmentation ion tasks.
Task
(-score
p-value
Classification Object detection
1477.27
7.65e-50
245.76
4.81e-35
Segmentation
369.16
2.11e-38
Our experimental results show that when the scaled dot-product self-attention in CSwin (Dong et al.] 2022) and MaxVIT (Tu et al. (022) is placed by our angu- lar self-attention. There is a slight performance im-
Table 12: The results of the significant test on CSwin and MaxViT.
Model
t-score
p-value
provement the image classification, as illustrated in Table g To further validate the statistical significance of the improvement, we perform the significant. test.
CSwin-T Dong et al. 2022
144.47
1.16e-30
MaxViT-T Tu et al.
2022
96.55
2.42o-27
15

---

Published in Transactions on Machine Learning Research (06/2024)
CSwin (Dong et al. 2022; and MaxViT Tu et al. 2022 with our angular self-attention, respectively, In the test, the CSwin-T T (Dong et al. 2022) is chosen as the target model and the null hypothessi would be that the performance of CSwin (Dong et al. 2022 with our angular self-attention (quad) is less than that of CSwin (Dong et al. 2022) with scaled dot-product self-attention, namely acc(CSwin(quad)) BaccCSi(ddoo-proddicc
& 82.7% Next, the evaluation of CSwin with our angular self-attention on image classification performed multiple times (e.g., 20) under different scttings (e.g., different sceds). At last, the I-score and p-value are also caleulated according to the one-tail T-test and the results of CSwin (Dong et al,, 2022) are shown in Table 12 The p-value for CSwin (Dong al. 2022) is 1.16e-39, which is significantly smaller than the threshold 0.05. The results suggest that the mull hypothesis should be rejected. We follow the same routine to perform the hypothesis test on MaxViT (Tu et al.) (022) and the results are also illustrated in Table 12] Both results suggest that the improvement on the image classification is indeed statistically significant and our angular self-attention can serve as a strong competitor for the traditional scaled dot-product sclf-attention.
5 Conclusions
In this paper, we propose the dual-window mechanism and angular self-attention. The dual-window mechanism divides the feature maps into even/odd numbers of local windows in each stage alternatively for the information exchange of the local windows. In angular self-attention, the traditional scaled dot-product operation is replaced by our proposed quadratic and cosine functions. The proposed angular function can also model the relationship of tokens in the long range. Based on the dual-window mechanism and angular self- attention, we propose a new vision transformer backbone called dual-windowed angular vision transformer. Extensive experiments show that our backbone can achieve competitive performance on the lasks such image classification, object detection and semantic segmentation and while maintaining the comparable computational cost with that of the bascline models.
Broader Impact. This work proposed a new architect of vision transformer called DWAVIT featured by angular self-attention and dual local window mechanism. Our model is proven to achieve competitive performance in downstream tasks such as object detection and semantic segmentation and has the enormous potential be used various practical scenarios. In particular, object detection is one of the most promising applications of vision transformers in the real world and it is often used in systems which require extensive interaction with the surrounding environment visually. For instance, autonomous vehicles require a large number of object detectors to identify the pedestrians and other vehicles nearby and the safety and the trustworthiness of vision transformers are critical in this area. Though our proposed model can achieve promising results on object detection and other tasks, some critical issucs such as adversarial robustness and trustworthiness are quile under-explored and further investigation is necessarily required.
Acknowledgement
The work is in part supported by the U.S. Army Research Office Award under Grant Number W911NF-21-1- 0109 and the National Science Foundation under Grant IIS-2316306.
References
Alaaeldin Ali, Hugo Touvron, Mathilde Caron, Piotr Bojanowski, Matthijs Douze, Armand Joulin, Tvan Laptev, Natalia Neverova, Gabriel Synnaeve, Jakob Verbeek, et al. Xcit: Cross-covariance image transformers. Advances in neural information processing systems, 34:20014-20027, 2021.
Zhaowei Cai and Nuno Vasconcelos. Cascade r-cmm: Delving into high quality object detection. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognátion (CVPR), June 2018.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In Computer Vision-ECCo 2020: 16th Europeon Conference, Glasgow, UK, August 23-28, 28, 2020, Proceedings, Part I 16, pp. 213 229. Springer, 2020.
16

---

Published in Transactions on Machine Learning Research (06/2024)
Chun-Fu Richard Chen, Quanfu Fan, and Rameswar Panda. Crossvit: transformer for image classification. In Proceedings of the IEEE/CVE international conference on compuler vision, pp. 357 366, 2021.
multi-scale vision
Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, et al. Mudetecciio: Open minlab delection toolbox and benchmark. arXie preprinl arXiv: 1901109666..5, 2019.
Liyan Chen, Weihan Wang, and Philippos Mordohai. Learning the distribution of errors in stereo matching for joint disparity and uncertainty estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 17235-17244, June 2023a.
Mengzhao Chen, Mingbao Lin, Ke Li, Yunhang Shen, Yongjian Wu, Fei Chao, and Rongrong Ji. Cf-vit: A general coarse-ta-fine method for vision transformer. In Proceedings of the AAAT Conference on Artificial Intelligence, volume 37, pp. 7042 7052, 2023b.
Yinpeng Chen, Xiyang Dai, Dongdong Chen, Mengchen Liu, Xiaoyi Dong, Lu Yuan, and Zicheng Liu. Mobile-former: Bridging mobilenet and transformer. In Proceedings of the IEEE/CVE Conference Computer Vision and Pattern Recognition, pp. 1270-5279) 2022.
François Chollet. Xception: Deep learning with depthwise separable convolutions. In Proceedings of the IEFE conference on computer vision and pattern recognition, pp. 1251 1258, 2017.
Krzysztof Marcin Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. In International Conference Learning Representations.
Xiangxiang Chu, Zhi Tian, Yuqing Wang, Bo Zhang, Haibing Ren, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Twins: Revisiting the design of spatial attention in vision transformers. In NewrIPS, pp. 9355-9366, 2021a.
Xiangxiang Chu, Zhi Tian, Bo Zhang, Xinlong Wang, Xiaolin Wei, Huaxia Xia, and Chunhua Shen, Conditional positional encodings for vision transformers. arXiv preprint 2rXiu:2102.10822 2021b.
MMSeggmenation Contributors. Mmssgmentatioo:: Openmmlab semantic segmentation toolbox and bench- mark, 2020.
Jcan-Baptiste Cordonnier, Andreas Loukas, and Martin Jaggi. On the relationship between self-attention and convolutional layers. Tn International Conference on Learning Represenlations, 2019.
Zihang Dai, Hanxiao Liu, Quoe V Le, and Mingxing Tan. Coatnet: Marrying convolution and attention for all data sizes. Advances in Neural Information Processing Systems, 34:3965-3977, 2021.
Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Newral Information Processing Systems, 35:16344 16359, 2022.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Lú, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248 255. leee, 2009.
Mingyu Ding, Bin Xiao, Noel Codella, Ping Luo, Jingdong Wang, and Lu Yuan. Davil: Dual atlention 2022: 17th European Conference, Tel Aviv, Israel, October
vision transformers. In Computer 23-27, 2022, Proceedings, Part XXIV, pp. 74-92. Springer, 2022.
Xiaoyi Dong, Jianmin Bao, Dongdong Chen, Weiming Zhang, Nenghai Yu, Lu Yuan, Dong Chen, and Baining Guo. Cswin transformer: A general vision transformer backbone with cross-shaped windows. In Proceedings of the IEEF/CVF Conference on Computer Vision and Patlern Recognition, pp. 12124-12133 2022.
17

---

Published in Transactions on Machine Learning Research (06/2024)
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In JCLR. OpenReeviw.net, 2021.
Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1824-6835, 2021.
Jianyuan Guo, Kai Han, Han Wu, Yehui Tang, Xinghao Chen, Yunhe Wang, and Chang Xu. Cmt: Convolutional neural networks meet vision transformers. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 12175-12185, 2022.
Dongchen Han, Xuran Pan, Yizeng Han, Shiji Song, and Gao Huang, Flatten transformer: Vision transformer using focused linear attention. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5961-5971, 2023.
Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, and Yunhe Wang. Transformer in transformer. Advances in Newral Information Processing Systems, 11:15908-15919) 2021.
Ali Hassani, Steven Walton, Jiachen Li, Shen Li, and Humphrey Shi. Neighborhood attention transformer. arXiv preprint arXic:2204.07143, 2022.
Ali Hassani, Steven Walton, Jiachen Li, Shen Li, and Humphrey Shi. Neighborhood attention transformer. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pollern Recognition, pp. 6185 6194, 2023.
Ali Hatamizadeh, Hongxu Yin, Greg Heinrich, Jan Kautz, and Pavlo Molchanov. Global context vision transformers. International Conference on Machine Learning, pp. 12633 12646. PMLR, 2023.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-777 170-778, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollar, and Ross Girshick. Mask r-cmn, In Proceedings of the IEEE International Conference on Computer Vision (ICCV), Oct 2017.
Liqiang He and Sinisa Todorovic. Destr: Object detection with split transformer. In Proceedings of the IREF/CVF conference on computer vision and pattern recognition, pp. 9377-9386, 2022.
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. Xix preprint or XrXiu: 770.00486... 2017.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700-4708, 2017.
Zilong Huang, Xinggang Wang, Lichao Huang, Chang Huang, Yunchao Wei, and Wenyu Liu. Cenet: Criss- cross attention for semantic segmentation. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 603 612, 2019.
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are rmns: Fast autoregressive transformers with linear attention. In International Conference on Machine Learning, pp- 5156 5165. PMLR, 2020.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arxiv preprint 1412:112.600 2014.
18

---

Published in Transactions on Machine Learning Research (06/2024)
Youngwan Lee, Jonghee Kim, Jeffrey Willette, and Sung Ju Hwang. Mpvit: Multi-path vision transformer for dense prediction. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pallern Recognition, pp. 7287 7296, 2022.
Yanghao Li, Chao- Yuan Wu, Haoqi Fan, Karttikeya Mangalam, Bo Xiong, Jitendra Malik, and Christoph Feichtenhofer. Mvitv2: Improved multiscale vision transformers for classification and detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4804-4814, 2022.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 18, pp. 740-755. Springer, 2014.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the TEEE/CVF international conference on computer vision, pp. 10012 10022, 2021.
Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, et al. Swin transformer v2: Scaling up capacity and resolution. In Proceedings of the IEEE/CY CVF conference on computer vision and pattern recognition, pp. 12009 120199, 2022a.
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 11976 11986, 2022b.
Jiachen Lu, Jinghan Yao, Junge Zhang, Xiatian Zhu, Hang Xu, Weiguo Gao, Chunjing Xu, Tao Xiang, and Li Zhang. Soft: Softmax-free transformer with linear complexity. Advances in Neural Informalion Processing Systems, 34:21297-2 1421199722309, 2021.
Xiaofeng Mao, Gege Qi, Yuefeng Chen, Xiaodan Li, Ranjie Duan, Shaokai Ye, Yuan He, and Hui Xue. Towards robust vision transformer. In Proceedings of the IEEE/CVE conference on Computer Vision and Pollern Recognition, pp. 12042 12051, 2022.
Sachin Mchta and Mohammad Rastegari. Mobilevit: Light-weight, pencral-purpose, and mobile-friendly vision transformer. In International Conference on Learning Regresentations.
Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In International conference machine learning, pp. 4055 4055-064. PMLR, 2018.
Hao Peng, Nikolaos Pappas, Dani Yogatama, Roy Schwartz, Noah Smith, and Lingpeng Kong. Random feature attention. In International Conference on Learning Representations.
Zhiliang Peng, Wei Huang, Shanzhi Gu, Lingxi Xie, Yaowei Wang, Jianbin Jiao, and Qixiang Ye. Conformer: Local features coupling global representations for visual recognition. In Proceedings of the IEEE/CVF internalional conference on compuler vision, pp. 367 167-376, 2021.
Boris T Polyak and Anatoli B Juditsky. Acceleration of stochastic approximation by averaging, SIAM journal on control and optimization, 30(4):838 885, 1992.
Zhen Qin, Weixuan Sun, Hui Deng, Dongxu Li, Yunshen Wei, Baohong Lv, Junjie Yan, Lingpeng Kong, and Yiran Zhong. cosformer: Rethinking softmax in attention. In International Conference on Learning Representations.
Iliaa Radosavovic, Raj Prateck Kosaraju, Ross B. Girshick, Kaiming He, and Piotr Dollár. Designing network design spaces. In CVPR, pp. 10433. Computer Vision Foundation IEEE, 2020.
Prajit Ramachandran, Niki Parmar, Ashish Vaswani, Irwan Bello, Anselm Levskaya, and Jon Shlens. Stand-alone self-atenntion in vision models. Advances in neural information processing systems, 32, 2019.
19

---

Published in Transactions on Machine Learning Research (06/2024)
Yongming Rao, Wenliang Zhao, Yansong Tang, Jie Zhou, Ser-Nam Lim, and Jiwen Lu. Hornet: Efficient high-order spatial interactions with recursive gated convolutions. In Advances in Neural Information Processing Systems.
Pengzhen Ren, Changlin Li, Guangrum Wang, Yun Xiao, Qing Du, Xiaodan Liang, and Xiaojun Chang. Beyond fixation: Dynamic window visual transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Paltern Recognition, pp. 11987 11997, 2022.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference compuler vision and pattern recognition, pp.
2018.
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representations. ar preprint Xiv: 1803: 02155, 2018.
Weili Shi and Sheng Li. Improving robustness of vision transformers via data-augmentted virtual adversarial training. In 2022 IEEE International Conference Big Data (Big Data), pp. 135-140. IEEE, 2022.
Weili Shi, Ronghang Zhu, and Sheng Li. Pairwise adversarial training for unsupervised class-imbalanced domain adaptation. In Proceedings of the 28th ACM SIGKDD conference knowledge discovery and data mining, pp. 1598 1606, 2022.
Weili Shi, Zhongliang Zhou, Benjamin H Letcher, Nathaniel Hitt, Yoichiro Kanno, Ryo Futamura, Osamu Kishida, Kentaro Morita, and Sheng Li. Aging contrast: A contrastive learning framework for fish identification across seasons and years. In Australasian Joint Conference on Artificial Intelligence, pp. 252-264. Springer, 2023.
Aravind Srinivas, Tsung-Yi Lin, Niki Parmar, Jonathon Shlens, Pieter Abbeel, and Ashish Vaswani. Bottleneck transformers for visual recognition. In Proceedings of the IEEE/CVE /CVF conference computer vision and pottern recognition, pp. 16519-16529, 2021.
Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unrcasonable effectiveness of data in deep learning era. In Proceedings of the IEEE international conference computer vision, pp. 2017.
Weixuan Sun, Zhen Qin, Hui Deng, Jianyuan Wang, Yi Zhang, Kaihao Zhang, Nick Barnes, Stan Birchfield, Lingpeng Kong, and Yiran Zhong. Vicinity vision transformer. IEEE Transactions Pattern Analysis and Machine Intelligence, 2023.
Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. Tn International conference on machine learning, pp. 6105 6114. PMLR, 2019.
Mingxing Tan and Quoe Le. Efficientnetv2: Smaller models and faster training. In International conference machine learning, pp. 10096-10106. PMLR, 2021.
Rui Tian, Zuxuan Wu, Qi Dai, Han Hu, Yu Qiao, and Yu-Gang Jiang. Resformer: Scaling vits with multi-resolution training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern 2023.
Recognition, pp.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International conference on machine learning, pp. 10347-10357. PMLR, 2021a.
Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 32 42, 2021b.
Hugo Touvron, Matthieu Cord, and Hervé Jégou. Deit ii:: Revenge of the vit. In Computer Vision-ECCV 2022: 17th European Conference, Tel Avio, Israel, October 23-27, 2022, Proceedings, Part XXIV, pp. 516 533. Springer, 2022.
20

---

Published in Transactions on Machine Learning Research (06/2024)
Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, Alan Bovik, and Yinxiao Li. ECCV 2022: 17th European Conference, Tel
Maxvit: Multi-axis vision transformer. In Compuler Aviv, Israel, October 23 27, 2022, Proceedings, Part XXIV, pp. 459 479. Springer, 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Huiyu Wang, Yukun Zhu, Bradley Green, Hartwig Adam, Alan Yuille, and Liang-Chieh Chen. Axial-deeplab: Stand-alone axial-attention for panoptic segmentation. In Computer EitionECCVV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part IV, pp. 108-126. Springer, 2020.
Weihan Wang. Bharat Joshi, Nathanicl Burgdorfer, Konstantinos Batsosc, Alberto Quattrini Lid, Philippos Mordohaia, and Toannis Rekleitish. Real-time dense 3d mapping of underwater environments. In 2023 IEEE International Conference Robotics and Automation (ICRA), pp. 5184-5191, 2023a. doi: 10.1109/
Weihan Wang, Jiani Li, Yuhang Ming, and Philippos Mordohai. Edi: Eskf-based disjoint initialization for visual-inertial slam systems. In 2023 LEEE/RS! International Conference on Intelligent Robots and Systems (TROS), pp. 1472, 2023b. doi: 10.1 20000/TRO0555522223.33. 10342100..
Wenhai Wang, Eaze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 568-578, 2021.
Wenhai Wang, Jifeng Dai, Zhe Chen, Zhenhang Huang, Zhiqi Li, Xizhou Zhu, Xiaowei Hu, Tong Lu, Lewei Lu, Hongshong Lú, et al. Internimage: Exploring large-scale vision foundation models with deformable convolutions. Xixiv preprint or XrXv:2211.05777, 2022a.
Wenhai Wang, Enze Xie, Xiang Li, Deng-Pine Long-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pvt v2: Improved baselines with pyramid vision transformer. Compulational Visual Media, 8(3): 415 26-, 2022b.
Wenhai Wang, Jifeng Dai, Zhe Chen, Zhenhang Huang, Zhiqi Li, Xizhou Zhu, Xiaowei Hu, Tong Lu, Lewei Lu, Hongsheng Li, et al. Internimage: Exploring large-scale vision foundation models with deformable convolutions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pollern Recognition, pp. 14409, 2023c.
Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks, In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7794-7803, 2018.
Haiping Wu, Bin Xiao, Nocl Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. Cvt: Introducing convolutions to vision transformers. In Proceedings of the IEEE/CVF International Conference on Compuler Vision, pp. 22-31, 2021.
Zhuofan Xia, Xuran Pan, Shiji Song, Li Erran Li, and Gao Huang. Vision transformer with deformable attention. In Proceedings of the IEEE/CU CCV conference on computer vision and pattern recognition, pp. 4794-4803, 2022.
Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In Proceedings of the European conference computer vision (ECCV), pp- 418 418-434, 2018.
Tete Xiao, Mannat Singh, Eric Mintun, Trevor Darrell, Piotr Dollár, and Ross Girshick. Early convolutions help transformers see better. Advances in Neural Information Processing Systems, 34:30392 30400, 2021.
Enze Xic, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. Advances in Neural Information Processing Systems, 10112077-120000 2021.
21

---

Published in Transactions on Machine Learning Research (06/2024)
Saining Xie, Ross Girshick, Piotr Dollar, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEFE Conference on Computer Vision and Pallern Recognition (CVPR), July 2017.
Jianwei Yang, Chunyuan Li, Pengchuan Zhang, Xiyang Dai, Bin Xiao, Lu Yuan, and Jianfeng Gao. Focal attention for long-range interactions in vision transformers. In NewrIPS, pp. 30008 30022, 2021.
Kun Yuan, Shaopeng Guo, Ziwei Liu, Aojun Zhou, Fengwei Yu, and Wei Wu. Incorporating convolution designs into visual transformers. In Proceedings of the IEEE/CVE International Conference Compuler Vision, pp. 579 379-588, 2021.
Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel Ni, and Harry Shum. Dino: Detr with improved denoising anchor boxes for end-to-end object detection. In International Conference on Learning Representations, 2022a.
Pengchuan Zhang, Xiyang Dai, Jianwei Yang, Bin Xiao, Lu Yuan, Lei Zhang, and Jianseng Gao. Multi-scale vision longformer: A new vision transformer for high-resolution image encoding. Proceedings of the IEEE/CVF international conference on computer vision, pp. 2998 3008, 2021.
Xiaosong Zhang, Yunjie Tian, Lingxi Xie, Wei Huang, Qi Dai, Qixiang Ye, and Qi Tian. Hivit: simpler and more eflicient design of hierarchical vision transformer. In The Eleventh International Conference on Learning Representations, 2022b.
Hengshuang Zhao, Jiaya Jia, and Vladlen Koltun. Exploring self-attention for image recognition. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 10076 10085, 2020.
Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip HS Torr, et al. Rethinking semantic segmentation from a sequence-lo-sequeeee perspective with transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6881-6890 2021.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through adc20k dataset. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 633 641, 2017.
Daquan Zhou, Zhiding Yu, Enze Xie, Chaowei Xiao, Animashree Anandkumar, Jiashi Feng, and Jose M Alvarez. Understanding the robustness in vision transformers. In International Conference on Machine Learning, pp. 27378 27394. PMLR, 2022a.
Zhongliang Zhou, Nathanicl P Hitt, Benjamin H Letcher, Weili Shi, and Sheng Li. Pigmentation-based visual learning for salvelinus fontinalis individual re-identification In 2022 IEEE International Conference on Big IEEE, 2022b.
Data (Big Data), pp.
A Appendix
A.1 Theoretical Analysis
The proof of Proposition is provided below.
Proof 1 Assume du S 01 <... 0J-1 for angles between the target query and all the keys. J is the total number of the tokens. V is the walue vector. According to our angular self-attention, the embedding of target
22

---

Published in Transactions on Machine Learning Research (06/2024)
Table 13: Object detection and instance segmentation performance of our model and the baseline models with Mask R-CNN framework with 1x schedule training scheme. The FLOPs mcasured at resolution 800 x 1280. [cos] and [quad] denote cosine and quadratic function, respectively.
Backbone
#Param(M) #FLOPs(G)
AP
AP
AP7
AP
AP
AP
ResNet-50
(He
2016
260
38.0
58.6
41.4
55.1
36.7
PVT-S
Wang et al.
2021
44
245
40.4
62.9
43.8
37.8
60.1
40.3
Vil-S
Zhang et a..
2121
45
218
44.9
67.1
49.3
41.0
64.2
44.1
TwinsP-S
Chu al.
2021a
245
42.9
65.8
47.1
40.0
62.7
42.9
Twins-S
CChu et al.
2021
44
228
43.4
66.0
47.3
40.3
63.2
43.4
Swin-T Liu
2021
48
264
42.2
64.6
46.2
39.1
61.6
42.0
DAT-T
Xia all
2022
272
41.4
67.6
48.5
40.4
64.2
43.1
CSWin-T
Dong ct a..
2022
42
279
46.7
68.6
51.3
42.2
65.6
45.4
DWAViT-1 [cos]
(Ours)
42
255
46.2
69.2
50.8
41.7
65.9
44.7
DWAViT-TIquad
(Ours)
255
16.6
69.6
51.3
42.2
66.3
45.7
Res101
(He et al.
2016
63
336
40.4
61.1
41.2
36.4
57.7
38.8
PVT-M
(Wang ct
202
302
42.0
64.4
45.6
39.0
42.1
ViL-M
Zhang et sl.
2021
60
261
41.6
66.3
48.5
10.7
63.8
43.7
TwinsP-B
Chu
2021a
64
302
44.6
66.7
48.9
40.9
63.8
44.2
Twins-B
Chu al.
2021a
340
45.2
49.3
41.5
44.8
Swin-$
Liu et al.
2021
69
354
44.8
66.6
48.9
40.9
63.4
41.2
CSWin-S
Dong
1022
54
342
47.9
70.1
52.6
43.2
67.1
46.2
DAT-S Xia et
2022
378
17.1
69.9
51.5
42.5
66.7
45.4
DWAVIT
(Ours)
64
338
48.0
70.7
52.6
43.3
67.6
46.5
DWAViT-S| quad] (Ours)
338
48.2
70.8
52.8
43.3
67.7
46.6
X101-64 (Xie et al.
2017
101
493
42.8
63.8
47.3
38.4
60.6
41.3
PVT-L Wang et al.
212T
81
364
42.9
65.0
46.6
39.5
61.9
42.5
CSWin-BB Dong et
2022
526
48.7
70.4
53.9
43.9
67.8
47.3
DWAVIT-B cos] (Ours)
97
162
48.6
71.1
33,7
43.6
68.0
46.9
DWAVIT- BBouud] (Ours)
97
462
48.6
71.2
53.6
43.6
67.9
47.1
tokens can be expressed as:
= C Σ
exp(s(0))/T)V,
C expl s(bb)
1
s(01) - ss(b))
(Vo exp(
V
7
+ exp( s(0,-1) s(0o)
7 V
(7)
- expl s(00)
1(Vo + wil ... -WJ W/-1V
x Vo wi V +... -w--Vss11
where wj exp( (00(())))0000))) and C is normalization coefficient. Since 0z On and s(0,) < s(0o), wj E (0.
A.2 Experiment
The results on object detection and instance segmentation of our model and the baseline models with Mask
R-CNN He et al.) 2017) framework with 1x schedule are illustrated in Table 13 The baseline methods include the latest VIT models such CSwin (Dong et al.] 2022) and DAT (Xia al. 2022). Specifically, Swin Transformer [Liu et al.) 2021). DAT (Xia et al.) 2022) and CSwin (Dong et al.
2022 are basclines
for exact comparison. These baselines and our method adopt MMDetection toolbox (Chen el. al. 2019) to perform the experiment and use models pre-trained on ImageNet- 1K only as the feature extractor. For tiny-sized models the experimental results show that the DWAVIT-T can achieve on par or better result against that of CSWin (Dong et al. 2022). For instance, the APt. of DWAViT-T(quadl can achieve 69.6%, which is 1.0% higher than that of CSWin (Dong et al. 2022 And the Apm of DWAViT-T(quad) is 42. 42.2%,
23

---

Published in Transactions on Machine Learning Research (06/2024)
Table 14: Object detection and instance segmentation performance of our model and the baseline models with Cascade Mask R-CNN framework. The model is trained with 3x scheme. The FLOPs are mcasured at resolution 800 x 1280. [cos] and [quad] denote cosine and quadratic function, respectively.
Backboue
(Param(M)
#FLOPs(G)
AP
AP
Af%
AAm
AP
AFT
Res50
He et al.
2016
82
739
46.3
64.3
50.5
40.1
61.7
43.4
Swin-T
Liu et al.
202T
86
745
50.5
69.3
54.9
43.7
66.6
47.1
DAT-T
Xia et al.
2022
86
750
51.3
70.1
55.8
44.5
67.5
48.1
NAT-T
Hassani ct
2023
85
737
51.4
70.0
55.9
44.5
67.6
47.9
GC VIT-T Hatamizzdeh
2023
770
51.6
70.4
56.1
44.6
67.8
48.3
[WAV (Ours)
734
52.2
57.0
45.1
68.3
DWAdil-Tlquad (Ours)
734
51.0
70.6
56.7
44.9
68.5
48.8
X101-32 Xie el al.
2017
101
819
48.1
66.5
52.4
41.6
63.9
45.2
Swin-S
Liu et al.
2021
107
838
51.8
70.4
56.3
44.7
67.9
48.5
DAT-S Xia
2022
107
807
52.7
71.7
57.2
45.5
69.1
49.3
NAT-S
Hassami
2023
108
809
52.0
70.4
56.3
44.9
68.1
48.6
GC VIT-S (Hatamizaehh al.
2023
108
866
52.4
71.0
57.1
45.4
68.5
DWAVIT-S poos (Ours)
102
817
52.5
71.3
57.0
45.6
68.9
49.6
UNVAViT-S(qual (Ours)
102
817
52.5
71.4
57.2
45.6
68.9
49.9
X101-64 Xie et al.
2017
140
972
48.3
66.4
52.3
41.7
64.0
45.1
Swin-B Lau al.
202T
145
982
51.9
70.9
56.5
45.0
68.4
48.7
NAT-B (Hsssaai et
2023
147
931
52.5
71.1
57.1
45.2
68.6
49.0
GC VIT-B (Hatamizadeh
2023
146
1018
52.9
57.8
DWAVTT-B(cos (Ours)
134
940
52.5
71.3
57.0
45.6
69.0
49.6
DWAui (Ours)
134
940
52.8
71.6
57.3
45.7
69.1
49.5
which is on par with that of CSWin Dong et al. 2022 outperform all the baseline methods on all the metrics. The APb and APm can reach 48.2% and 43.3%, respectively. Furthermore, DWAViT-B can achieve best results on AP and AP
The DWAVII-S with quadratic self-attention
Table 14 show the performance of our model and the baseline models with Cascade Mask R-CNN (Cai & Vasconcelos 2018) framework on object detection and instance segmentation. Swin Transformer (Liu et al., 2021) and DAT (Xia al. 2022) are the major baselines for the exact comparison. The experimental results show that our DWAVIT outperforms baseline methods. DWAViT-T can achieve 52.2% on APb and 45.1% Apm, and DWAVIT-S can achieves 52.5% Ap and 45.6% on APm, The DWAVIT-S with quadratic self-attention can achieve 45.6% and 49.9% on APm and APM. respectively, which is 0.1% and 0.3% higher than that of DAT-S (Xia et al. 2022
The performance of DWAVIT-S and DWAVIT-B on semantic segmentation task are illustrated in Table 15 The bascline models include XCIT (Ali et al. 2021), Swin Transformer (Liu et al. 2021). DaV (Ding et al. 2022). Focal Transformer (Yang et al., 2021 and DAT (Xia et al., 2022). The experimental results suggest. that our DWAVIT with cosine self-attention can achieve the best performance and the mloU can reach 49.3% and 49.8% for DWAVIT-S and DWAViT-B, respectively.
A.3 Ablation Study
In the ablation study we first compare the performance of our model with traditional scaled dot-product self- attention and our proposed angular self-attention and evaluate the performance on ImageNet-1K classification, COCO object detection and ADE20K semantic segmentation. We adopt the DWAViT-T as the backbone in three tasks. In object detection, Mask R-CNN is adopted the framework and the model trained with 36 epochs. In semantic segmentation Upernet is adopted the framework and the model is trained with 160K iteration. The performance of model semantic segmentation performance of our model with scaled dot-product solf-attention is lower than that of our model with the proposed angular self-attention. On some tasks the gap of the performance is more obvious. For instance, on semantic segmentation our model of scaled dot-prodduct self-attention only achieves 44.7% on mloU, approximately 3% lower than that of our model with cosine self-attention. The experimental results suggest our angular self-attention can model the relationship of the tokens successfully and the angular self-attention is a powerful alternative to the traditional scaled dot-product self-attention.
illustrated in Table [6] The
24

---

Published in Transactions on Machine Learning Research (06/2024)
Table 15: The semantic segmentation performance of DWAVIT-S, DWAVIT-B and baselines on ADE20K. The FLOPs are calculated with resolution 512 x 2048. [cos] and [quad] denote cosine and quadratic function, respectively.
Backbone
# (Param(M)
#FLOPs(G)
mloU
ResNet-101 (He
2016
86
1029
44.9
XCIT-S24/16. [ATi et al.
2021
76
1053
46.9
TwinsP-B (Chu et al.
2
2021a
74
977
47.1
XCYT-M24/
16 [Alii et al. 2021
112
1213
47.6
Swin-S
Liu et al.
202T
81
1038
47.6
Twins-B
Chu et al.
2021a
89
1020
47.7
Focal-S
Yang et al.
2021
1130
48.0
DaVir-T Ding
2022
81
1030
48.8
DAT-T Xia et al.
2022
83
1079
48.3
GC ViT-S (atammizaee et al.
2023
84
1163
48.3
NAT-S (assanii et al. 2023
82
1010
48.0
DWAVIT-S (Ours)
75
1015
49.3
DWAViT-S -S S[quad] (Ours)
75
1015
47.8
XCIT-M24/8 (Ali et al.
2021
110
2161
48.4
Swin-B Liu et al. 2021
121
1841
48.1
Focal-B B Yang et al. 2021
126
1354
49.0
DaViT-B Ding et al 2022)
121
1175
49.4
DAT-B (iaa et al. 2022
121
1212
49.4
GC ViT-B (Hatamizadch
2023
125
1348
49.2
NAT-B (asssani al. 2023
123
1137
48.5
DWAVIT-B cos] (Ours)
108
1143
49.8
DWAViT-B(quus (Ours)
108
1143
49.1
Table 16: The performance of our proposed DWAVIT with angular self-attention, scaled dot-product self- attention and lincarized self-attention the semantic sogmentation. FLOPs calculated with resolution 512 x 2048. The (dot product], [cos] and [quad] in the square bracket denote scaled dot-product, cosine and quadratic function, respectively.
Backbone
Param(M)
FLOPss(G
mloU
DWAViT-T(dot product]
52
930
44.7
DWAVIT-T[hiee [linear]
52
930
44.1
DWAViT-T(con (Ours)
52
930
47.5
DWAVIT-T(quad] (Ours)
52
930
45.4
We also compare the performance of our model with the proposed dual local window with the shifted window proposed by Swin Transformer (Liu al.) 2021). In the experiment we replace the dual local window with the shifted local window and evaluate the performance of the model on ImageNet-IK classification, COCO object detection and ADE20K semantic segmentation. We adopt the DWAVIT-T(quad) as the backbone in three tasks. The results on ADE20K semantic segmentation are illustrated in Table [7 respectively. On semantic segmentation, our model with shifted local window can achieve better performance than that of our model with dual local window. However, on the object detection task, the performance of our model with shifted window lower that of our model with dual local window on all metrics. To wrap up, our dual local window is effective in the modeling of the relationship of tokens and it can achieve better performance than that of the shifted local window on some tasks.
25

---

Published in Transactions on Machine Learning Research (06/2024)
Table 17: The performance our proposed DWAVIT with the dual local window and shifted local window on the semantic segmentation. We adopt the quadratic self-attention and the FLOPs are calculated with resolution 512 2048.
Backbone
Param(M)
FLOPs(G)
mloU
DWAviT-T[shiftedd window]
52
933
46.2
DWAViT-T[dual window] (Ours)
52
930
45.4
26


Reviews:
[{"review_id": "bMPpxobdHG", "reviewer": "Reviewer_ouca", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths:\n- The paper introduces two components, the dual window mechanism and angular self-attention, to enhance the original vision transformer architecture.\n- The methodology is well-explained and easy to understand.\n\n### Weaknesses:\nLimited novelty as the proposed components bear similarities to existing works, with limited demonstrated benefits over prior approaches.\n1. Regarding the dual window mechanism, several previous works (MaxVit, CSwin, DaViT, etc.) have explored alternatives to the global self-attention in the original vision transformer. However, this paper does not provide a comprehensive comparison or theoretical analysis to highlight the advantages of their approach over these existing methods. The empirical results (Table 6) show only marginal improvements compared to the Swin transformer.\n2. The proposed angular self-attention mechanism also fails to demonstrate significant benefits over existing approaches like CosFormer or SOFT. Even when compared to the original dot-product self-attention (Table 5), the improvements are relatively minor.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a new vision transformer architecture DWAViT. It introduces two main components compared to existing works: (1) dual window mechanism for local window interactions and (2) angular self-attention mechanism to replace dot-product in original self-attention. DWAViT achieves comparable performance compared to existing works on various computer vision tasks. The ablations studies show that the introduced components can slightly improve the overall performance.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. A more comprehensive presentation and analysis of the advantages offered by the proposed method, particularly in comparison to advanced architectures such as MaxVit, CosFormer, and other relevant state-of-the-art approaches. \n2. A more thorough empirical evaluation against the latest architectures with analysis or insights to demonstrate the significance of the proposed method.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "hKitrDXdxD", "reviewer": "Reviewer_Y9sP", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n-The proposed angle operation is reasonable. It measure the relationship between different tokens only with the angle, instead of the norm. It naturally reduces some noise.\n\n-The authors conduct extensive experiments to validate the effectiveness of the proposed methods. It compares image classification methods on ImageNet-1K，detection methods on COCO and semantic segmentation methods on ADE20K. The results show that the proposed new architecture achieves better performance than the existing methods.\n\nWeaknesses:\n\n-Compared to the standard attention, the angle computation seem more complex. Will it introduce extra latency. The authors are required to measure the practical inference speed, and add the results in the main tables (such as table 2 and table 3).\n\n-The architecture introduces extra depthwise convolution, which has been proven effective to improve model's performance. For a fair comparison, could the authors use the architecture configure of swin transformer and only replace the attention module? More discussions are also required.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a new  attention computation method, that  replaces the  dot-product operation into  angular computation. It reckons that the angular computation can represent the relationship between different tokens better than the conventional dot-product operation. Based on the new attention, the authors construct a new transformer model, called DWAViT. Similar to Swin-Transformer, it also has a hierarchical-structure which extract information gradually.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See the weaknesses.", "broader_impact_concerns": "No concern on the ethical implications.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "hjFhGK2FKY", "reviewer": "Reviewer_twmT", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Paper is well written and easy to follow\n- The idea of using odd/even windows is interesting and can help bridge connections across layers at least due to the changes in border areas. \n- The gains are positive but small for the most part\n- The approach is simple which makes easier for the community to adopt it. \n\nWeaknesses:\n- Performance improvements : My key worry is that : are the improvements over various prior works statistically significant ? In 4.1, what do you mean \"variance diminished considerably\" ? \"major results are reported\" ? \n- Intuitions: Authors should provide some intuitions on why they think angular attention is better than scaled dot product attention. Do the authors have some intuitions for why the angular attention works better with the dual local window techniques? \n- Make differences more obvious: In the key result tables, make it clear what DWAViT was built on. Ie, if you do not use the dual window and angular attention contributions, which row in the table will be your baseline. I also suggest that the authors include second-best results for the tables. \n- Hyperparameters: How sensitive are the results to various choice of hyper parameters ? Is the proposed approach less or more sensitive to such changes compared to prior works ? This will be interesting and might help strengthen the paper. \n\nMinor:\n- Some typos : 3.1, ood or odd ?, 3.2 \"tokes\"", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "- The paper proposes a new transformer variant for vision tasks called DWaViT. (dual-windowed angular vision transformer). \n- Two key modifications are proposed over standard ViT and Swin transformers : a dual window mechanism is proposed which partitions alternate feature maps in odd/even local windows to allow for connections between different local windows. Second, use of angular attention is proposed which depends only on the angle between query and key as opposed to scaled dot product attention which is also a function of norm of the vectors. \n- The proposed model is trained on a variety of pre-training and downstream tasks and shows improvements over prior works.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "A discussion/experiment on statistical significance will be helpful in securing my recommendation for this paper.", "broader_impact_concerns": "No concern", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new vision transformer architecture DWAViT. It introduces two main components: (1) dual window mechanism for local window interactions and (2) angular self-attention mechanism to replace dot-product in original self-attention. DWAViT achieves comparable performance compared to existing works on various computer vision tasks.

After the rebuttal, the three reviewers gave two Leaning Accept and one Reject. The major concern of the Reject is the slightly lower performance in image classification. The authors demonstrated only comparable performance of their proposed DWAViT in Tables 12-14. The AE understands the concern of the reviewer. The following question shall also be raised:
- When adding the proposed angular attention and dot product attention to CSwin and MaxViT, the gains are only ~0.3%. Are the gains statistically significant?
Decision: Accept (poster)