Orthogonal Transformer: An Efficient Vision Transformer Backbone with Token Orthogonalization
Authors: Huaibo Huang
Xiaoqiang Zhou
Ran He
Venue: NeurIPS 2022 Accept (2022)
Keywords: Vision Transformer
Efficient Self-attention
Orthogonality

Abstract: We present a general vision transformer backbone, called as Orthogonal Transformer, in pursuit of both efficiency and effectiveness. A major challenge for vision transformer is that self-attention, as the key element in capturing long-range dependency, is very computationally expensive for dense prediction tasks (e.g., object detection). Coarse global self-attention and local self-attention are then designed to reduce the cost, but they suffer from either neglecting local correlations or hurting global modeling. We present an orthogonal self-attention mechanism to alleviate these issues. Specifically, self-attention is computed in the orthogonal space that is reversible to the spatial domain but has much lower resolution. The capabilities of learning global dependency and exploring local correlations are maintained because every orthogonal token in self-attention can attend to the entire visual tokens. Remarkably, orthogonality is realized by constructing an endogenously orthogonal matrix that is friendly to neural networks and can be optimized as arbitrary orthogonal matrices. We also introduce Positional MLP to incorporate position information for arbitrary input resolutions as well as enhance the capacity of MLP. Finally, we develop a hierarchical architecture for Orthogonal Transformer. Extensive experiments demonstrate its strong performance on a broad range of vision tasks, including image classification, object detection, instance segmentation and semantic segmentation.

Orthogonal Transformer: An Efficient Vision Transformer Backbone with Token Orthogonalization
Huaībo Huang Xiaoqiang Zhou 1,2,4 Ran He 2233.
CCetter for Research on Intelligent Perception and Computing, CASIA, Beijing, China National Laboratory of Pattern Recognition, CASIA, Beijing, China
School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China *University of Science and Technology of China, Hefei, China huaibo. huangecripac. ia. ac. cn, xq5250mail ustc. edu. cn, rhe@nlpr. iaa.iiia..... ac cn
Abstract
We present general vision transformer backbone, called as Orthogonal Trans- former, in pursuit of both efficiency and effectiveness. A major challenge for vision transformer is that self-attention, as the key element in capturing long-range dependency, very computationally expensive for dense prediction tasks (e.g., object detection). Coarse global self-attention and local self-attention are then designed reduce the cost, but they suffer from either neglecting local correlations or hurting global modeling. We present an orthogonal self- attenttnnnion mechanism alleviate these issues. Specifically, self-attention computed in the orthogonal space that is reversible to the spatial domain but has much lower resolution. The capabilities learning global dependency and exploring local correlations are maintained because every orthogonal token in self-attention can attend to the entire visual tokens. Remarkably, orthogonality nously orthogonal matrix that is friendly to neural networks and can be optimized as arbitrary orthogonal matrices. We also introduce Positional MLP to incorporate position information for arbitrary input resolutions well enhance the capacity MLP. Finally, we develop a hierarchical architecture for Orthogonal Transformer. Extensive experiments demonstrate its strong performance broad range of vision tasks, including image classification, object detection, instance segmentation and semantic segmentation.
realized by constructing an endoge-
Introduction
Recently, transformer [46] has made tremendous success in the field of natural language processing (NLP). Benefiting from the self-attention (SA) mechanism. it has strong capacity in building long-range dependencies in sequential data. Since long-range modeling is also essential for wide range of vision tasks, transformer has been adapted into computer vision by converting an image into sequence of patches [13] (called tokens) and achieved competitive performance compared to Moreover, -tteeettoio has a quadratic computational complexity to the number of tokens, resulting in intolerably expensive computational cost for dense prediction tasks, e.g., object detection and segmentation.
CNN 43 44
Many efforts have been made design efficient self-attention mechanisms for vision transformer. PVT 50) employs a pyramid architecture with downsampled key and value tokens to reduce the computation of global attention. Swin [35] performs self-attention in a local region with shifted window to allow cross-region connection. GG-Transformer [63] and CrossFormer
present dilated
Ran He is the corresponding author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

target feature
atfention region
correspondane befure/afler
Downsampled BelF-Attention
(b) Local Self-Atteatin
(c) Diluted Self-Atteation
orthogonalization
AZ
Permute
Visual Tokens
Token Orthogomnlizatin
Orthogonal Tokens
A²
A A (AZ)
(usA(AZ)
Permute
Visual Tokens
Token Reverse
Orthogonal Tokens
Self-Attention in Orthogonal Space
(d) Orthogonal Self-Attemtion
Figure Illustration of different kinds of efficient self-attention mechanisms.
self-attention learn large-scale features efficiently. As shown in Fig. | (a-c), the aforementioned attention mechanisms can reduce the number of tokens conveniently. However, the penalty is losing fine-level details for coarse global self-attention (including downsampled and dialted ones) or hurting
long-range modeling for local self-attention
41 52
42
631
This paper presents orthogonal self-attention (OSA) mechanism to capture global dependency without losing fine-level details. As shown in Fig. (d). we orthogonalize tokens within local regions, permute them into token groups and perform group-wise wise self-attention the orthogonal space. We apply Orthogonal Transformation (OT) for the tokens for the following reasons. Firstly, tokens have a lower resolution in the orthogonal space than the original visual space, thus reducing the computational cost (details see Sec. 3.2.2.. Secondly, the orthogonal space is reversible to the visual space without information loss (thus outperforming downsampled self self-attention), and can be easily reversed back since the inverse matrix A' is just the transpose of the orthogonal matrix A, i.e., A'=A" AT, Thirdly, OT can separate tokens into linearly independent groups. Thereby computing self-attention in such groups helps to explore different properties in representation. Lastly, OT builds connections among adjacent tokens explicitly and is more capable of modeling local correlations than dilated partition.
Despite of these benefits, often needs complex optimization algorithms []] extra penalties on the loss functions [58] (88 to enforce orthogonality. In this work, we construct an endogenously orthogonal matrix that is friendly to neural networks with gradient optimizers. We build an orthogonal matrix A E R Rax by the product of Householder transformations (H, =I=I-- 2u,u," 0, n 1}. where u, R learnable unit vector. In this way. A is naturally orthogonal and can be optimized as arbitrary orthogonal matrices. In particular, the dilated self-attention can viewed as degeneration of OSA when
imposing
The strong capacity exploring diverse transformations makes OSA more essential for the model to achieve competitive performance.
Based on the orthogonal self-attention mechanism, we propose a new Orthogonal Transformer in this work. As shown in Fig. 2 it follows the hierarchical design [50) 35 and serves as general- purpose backbone for computer vision. We stack window self-attention (WSA) and orthogonal self-attention (OSA) alternatively to capture both global and local dependencies. Note that although our OSA superior preserving local details, combing it with WSA can further enhance Orthogonal Transformer. To improve the flexibility for arbitrary resolutions, we adopt convolutional position embedding to incorporate position information. Specifically, Positional MLP (PMLP) introduced by equipping MLP with depth-wise convolution (DConv) after the non-linear activation. Such simple design not only enables the network generate position information flexibly, but also enhance the capacity of MLP. Besides, also allows for downsampling within the transformer block

---

with strided DConv. We empirically show that downsampling within the block can achieve better performance than outside.
Extensive experiments demonstrate the strong performance of Orthogonal Transformer on a wide range of vision tasks. For example, without extra training data or supervision (such as token label- ing [26) and distillation [43)), our large model Ortho-L achieves 85.4 top-1 accuracy on ImageNet- K image classification, surpassing the previous state-of-the-art Swin Transformer [35] by +1.2 with similar model size and FLOPs. Our base model Ortho-B achieves 53.0 box AP and 45,9 mask AP on the COCO detection task, 49.8 8 mIOU on the ADE20K semantic segmentation task, surpassing the Swin Transformer counterpart by +1.2, +1.2 and +2.2, respectively. Under a smaller setting of FLOPs, our Ortho-S even obtain larger performance gains, i.e., +2.1 on ImageNet classification with 224 x 224 resolution, +1.8 box AP, +1.6 mask AP on COCO detection and +4.0 mIoU on ADE20K segmentation.
2 Related Work
Vision Transformers. Transformer network is firstly proposed in the natural language processing (NLP) field and achieves superior performance in many NLP tasks 46 27 Recently, there emerges a trend towards incorporating the transformer into computer vision tasks, which are previously dominated by CNNs [19
VIT [13] is the pioneering work of vision transformers. After that, there are a large number of works focusing on designing a general vision Transformer backbone [g]
40
35
5
50
66
63 61
12
52
60
26 for different vision tasks. To adapt transformer to the
image inputs, hierarchical architectures
35 50) efficient self-attentions [35] 63 25 and diverse
positional encodings [46, 38
10 12 are proposed. Vision transformers are also applied to different
downstream vision tasks, such as object detection 4 [69, semantic segmentation [39 67 53). image
restoration [6] [7]. and video processing [33] []. We propose new vision transformer backbone to tackle various vision tasks efficiently and effectively.
37 have been proposed in the field of NLP to efficiently handle long sequences. Since the image resolution usually high in many vision tasks, the global self-attention can be applied only for a few times and in the low resolution feature space. Early approaches, such as ViT [3 and DeiT (43), employ large image patches to restrict the number of tokens. PVT (50) and Twins [9] use downsampled tokens to compute self-attention. Many works adopt local self-attention to limit the token number by attending only sub- regions of the input, such as the window attention (35] axis attention (21. and criss-cross attention [25]. Inspired by dilated convolution 62). GG Transformer (63) and CrossFormer [52) introduce dilated self-attention to capture long-distance dependency. Unlike existing methods, orthogonal self-attention can attend to the entire tokens at a low computational cost. Global dependency can be
Efficient Self-Attentions. Many efficient self-attention mechanisms 30 29
captured without losing local details.
Positional Encodings. Recently, researchers propose to add different kinds of position information to the token feature or self-attention process. Absolute positional encoding (APE) (46) is the first work to add position encodings to the Transformer inputs. Relative positional encoding (RPE) 35 R8
considers the relative distance between two tokens instead of the absolute position. Conditional positional encoding (CPE) [10] takes feature as inputs and generates position information via a convolution layer. We follow CPE but compensate position information inside the MLP module. This design enables MLP to explore local correlations, thus enhancing the model performance.
Orthogonality. Orthogonality has been widely used in neural networks to regularize neurons 58 24 48 or learning disentangled representations [54]. In contrast to existing works, we employ orthogonality to group visual tokens for efficient self-attention. Besides, most of previous works depend on complex optimization algorithms, such as singular value decomposition (3 and iterative
approximation 32), or imposing extra regularization terms on the loss functions [58 48 Unlike
them, we construct an endogenously orthogonal matrix that is more friendly to neural networks with gradient optimizers.
3

---

T TT TTTC
#
16 16
X
32 "12
X
OTB Trmmition $)
X44
PC
Canvalutionl Embeddinng
Transiion
OTH with Framition
PAELP
Coanj-d)
PHELP
OTB with Transition
OTB withen Trimstio
107!
WAA OSA
BUA OMA
WhaT
Trunsition
Stage
Stage
Stage
Stuge
Figure 2: The architecture of Orthogonal Transformer. Orthogonal self-attention (OSA) and window self-attention (WSA) are used successively, where the tokens are grouped via window partition (WP) or orthogonal transformation (OT). WP-1 and OT denote the respective reverse processes.
3 Method
3.1 Overall Architecture
An overview of the Orthogonal Transformer architecture illustrated Fig. 2 For an input image #x333 we follow [57] and adopt several stacked convolution layers obtain H W patch tokens. Similar to Swin (35], the transformer blocks are divided into 4 stages to produce hierarchical representations. We stack blocks of window self-attention (WSA) and orthogonal self-attention (OSA) alternatively for better capacity of global and local modeling. For the Orthogonal Transformer blocks (OTBs), introduce Positional MIP (PMLP) with additional depth- wise convolution (DConv) after GELU in MI.P to provide position information. Based on PMLP, we design two kinds of OTBs, one with transition and one without. OTBs with transition aim reduce the spatial resolution by strided DConv and a residual strided convolution. The first three stages are composed of stacked OTBs without transition following by single OTB with transition, while the last stage only consists of OTBs without transition.
3.2 Orthogonal Self-Attention
Orthogonal self-attention (OSA) is proposed to enable the transformer layers encode high-resolution images efficiently. Unlike previous works [35] [0 63]. OSA performs self-attention in the orthogonal space to capture global dependency without neglecting local correlations. It can cover the same global receptive field with much less cost than standard self-attention.
3.2.1 Orthogonal Transformation
Orthogonal transformation (OT) corresponds I is the identity matrix, As shown in Fig. T we employ orthogonal transformation to convert visual tokens into orthogonal tokens for self-attention computation. Since it hard to enforce orthogonality in neural networks [3] [2], we first introduce simple yet effective way to construct an endogenously orthogonal matrix that can be optimized as arbitrary orthogonal matrices. It is based on the following linear algebra theorem 45 (Proof is provided in the appendix).
orthogonal matrix A satisfying ATA I, where
Theorem 1. Every real orthogonal x matrix is the product of most real orthogonal Householder transformations.
Given a tokens, inspired from Theorem 1, construct an orthogonal matrix A E RRXX as:
A HoH, Hn-1
(I)

---

where H; = u is Householder transformation, u; is unit column vector. Providing a set of inddam-inttialized vectors {vili O.... n 1}. the orthogonal matrix can be reformulated as:
VoVo Vo
VIVI M
(I-2 Vn--Vnn1
A
......II-
1
We set (v. ..... 1} as learnable parameters, and hence A can be optimized as arbitrary orthogonal matrices in neural networks with gradient optimizers. Remarkably, can maintain orthogonality endogenously throughout the training process without extra regularizer.
3.2.2 OSA Block
To clarify, we first define two terms:
Orthogonal window size m, is the size of the sub-window on which OT is performed. We perform OT separately for local windows to control its computation complexity.
Orthogonal groups n. is the number of groups into which the tokens separated OT. Since the orthogonal matrix is square, Yo, m
Now we elaborate how OSA works in the following three steps, token orthogonalization, self-attention for orthogonal tokens, token reverse.
Token Orthogonalization. Given the input feature Z E R (exx)xx (viewed as h x w tokens with the dimension c), it firstly divided into grid of non-overlapped windows of size m, X mo i.e., Z E X", where fow m. m. and n. m Z can be viewed fow sub-windows, of which each consists of n, tokens, i.e., OT separately for Z, by multiplying the orthogonal matrix A E R and get orthogonalized tokens 2, A E Re Finally, we combine the The sub-windows and permute the feature Ž R Th Xn, Xe into n. groups of orthogonal tokens 2) E R (where = ....., To 1).
(Z, R = 0, ....... Tlw - 1}. Then we perform
Self-attention for orthogonal tokens. We perform standard multi-head self-attention (MSA) group- wisely for 23. Since 2 has smaller token number than the input Z (actually m. m. against w), with an appropriate mo- performing self-attention the orthogonal space can significantly reduce the computation cost. Besides, for OSA, local correlations can be modeled because 25 is computed from the adjacent tokens in Z, and local details are preserved due to the fact that Ž and Z are reversible each other.
A_ ar
Token reverse. After the self-attention computation, we employ A as the inverse of A, to recover the visual tokens from the enhanced orthogonal representations. The token reverse process is not repeated herein because is exactly the inverse process of token orthogonalization except that the orthogonal matrix is A
The complete process of OSA can be defined as:
(Z) (MMMA +
simplify the process of token orthogonalization as Z for description convenience. Note that when m. 1, OSA equals to the global self-attention.
Complexity Analysis. The computation complexity of the standard global self-attention is
Q(GS A) = 4hwC2 + 2(hw))
(4)
For our OSA, the computational complexity is
1 2(hw)) 2nchwC. ((OSA) = 2 + no
(5)
The third term is produced by OT but can be ignored when & Vhw (this is usually true for high-resolution vision tasks). Compared to the standard global self-attention, OSA's computation
cost is reduced significantly from (((w))² to O 1 (hw)²) for high-resolution vision tasks. n.
5

---

Table 1: Configurations of Orthogonal Transformer. The FLOPs are measured at resolution 224 x 224.
Model
Blocks
Channels
Heads
Ratio
Params (M)
FI.OPs (G)
Ortho-T
[2, 6,
[32, 64, 160, 256]
[1, 2,
3
3.9
0.7
Ortho-S
[3,5, 13, 3]
[64, 128, 256, 512]
12,4, 8, 16]
4
24
4.5
Ortho-B
[3.5. 19, 4]
[80. 160, 320, 640]
[2.4, 8. 16]
4
50
8,6
Ortho-L
[4, 6, 24,
196, 192. 384, 768]
[3, 6. 12, 24]
4
88
15,4
3.3
Window Self-Attention
Since Orthogonal Transformer is composed of stacked window self-attentions (WSAs) and OSAs, we give brief review of WSA here. WSA first employ window partition (WP) to split the input feature Z R. (hxxw))e into my mh non-overlapped windows of size mhe mhe, then performs MSA within each window, and finally reconstruct the tokens from the enhanced representation. The complete process of WSA is defined as:
Z. where fwP and fwep denote window partition and its reverse. respectively. WSA has a linear computational complexity to token number hw:
= = WWsss
(
Q(W SA) = 4hwC2 + hwCC
3.4
Positional MLP
relative position embedding (RPE) 35 38) to handle the position information but they cannot be well adapted for arbitrary input resolutions [[O]. Recently, convolution position embedding (CPE) [10] is proposed to generate position information by convolutional layers. In detail, CPE adds position encodings into the input tokens outside the transformer blocks. We follow CPE TO but apply it in different way. As shown in Fig. 2 we introduce Positional MLP (PMLP) to tackle position information in the MLP module. We equip PMLP with depth-wise convolution (DConv) after the non-linear GELU. This design can not only provide position information, but also enhance the capacity of MLP in exploring local correlations. Based on it, we further develop PMLP with transition by setting the stride of DConv as 2 and adding residual strided convolution. PMLP without transition is defined as:
Earlier VITs adopt absolute position embedding (APH) 46
fPMLr(Z) (reeerffiiiaaaaaaooo/CCocaaaaaaa + Z,
(8)
while PMLP with transition is defined
(PMLP (fcu (fF(((( + (s2) &Con
f(a1) where f JCCone and f(s2) DCon denote depth-wise convolutions with stride and stride respectively, and flo)) denote a convolution with stride 2.
Discussions. Note that depth-wise convolution (DConv) has been applied in previous vision transformers. The major difference between the proposed Positional MLP and others is that we use strided DConv perform downsampling within the transformer block. In this way, do need extra patch merging layer to perform feature downsampling. In Table we empirically show that can achieve better performance than outside the transformer block. We also conduct extensive experiments to investigate the best location of DConv in Appendix D.4. There exist various manners to employ DConv in vision transformers (e.g., PVT [51], CeiT (64) CMT [16) and ours adopt DConv in different locations). We believe our work is suggestive to find appropriate manner of incorporating depth-wise convolution into transformers.
3.5 Architecture Variants
As shown in Table we build four different Orthogonal Transformer backbones by changing the block number and channel number in each stage. Specifically, Ortho-S, Ortho-B and Ortho-I. are designed to have a similar setting of FLOPs and model size with their Swin Transformer counterparts.

---

Table 2: Comparison with the state-of-the-ar on ImageNet-IK classification.
#Param
FLOPs
Acc.
#Param
FLOPs (G)
Acc. (%)
Model
Model
(M)
(G)
(%)
(M)
DeiT-Ti 433
5.7
1.3
72.2
PVT-L [50
61
9.8
81.7
PVTv2-b0 51
3.7
0.6
70.5
T2T-24
65
64
13.2
82.2
T2T-7 65
4.3
1.1
71.7
Swin-S 35
50
8.7
83.0
QuadTree-B-hhe
0.7
72.0
CvT-21
[55]
32
7.1
82.5
Ortho-T
3.9
0.7
74.0
CaiT-s24 [44
47
9.4
82.7
Focal-S [6
51
9.1
83.5
RegNetY
36
21
4.0
80.0
CrossFormer-B [22
52
9.2
83.4
DeiT-S 43
22
4.6
79.9
Region [5]
41
7.4
83.1
PVT-S 50
25
3.8
79.8
Ortho-B
50
8.6
84.0
T2T-
5.2
80.7
Swin-T [35
29
4.5
81.3
CvT-21 384 65
32
25.0
84.9
Twins-SVT-S
24
2.9
81.7
CaiT-s241 384 [44
47
32.2
84.3
CvT-13 [55]
4.5
81.6
Ortho- Bf
50
26.6
85.2
CaiT-xs24 44
27
5.4
81.8
DeiT-B 43
86
17.5
81.8
Focal-T [6]
29
4.9
82.2
Swin-B 35
88
15.4
83.3
CrossFormer-S 152
31
4.9
82.5
CaiT-s48 44
90
18.6
83.5
RegionViT-S
5.3
82.6
Focal-B [6]
90
16.0
83.8
Container 15
22
8.1
82.7
CrossFormer-L 52
92
16.1
84.0
QuadTree-B-b2 B-22 [4]
24
4.5
82.7
Region ViT-B 5
73
13.0
83.2
Ortho-S
24
4.5
83.4
Ortho-L
88
15.4
84.2
CvT-13f 384 [55
20
16.3
83.0
Swin-Bt 384 35
88
47.0
84.2
CaiT-xs241 384 44
27
19.3
83.8
CaiT-s481 384
90
63.8
85.1
Ortho-S
14.3
Ortho- Lf 384
88
47.4
85.4
For all the models, we set the orthogonal window sizes m, for OSA in the four stages as 8, 4, 2, respectively, and the window size for WSA 7. The expansion ratios in MLP are Ortho-T and 4 for the other three variants, For the convolutional patch embedding, we borrow early convolutions from (57) and apply convolutions with the same setting of [57). For the transformer blocks, we set the last one in each of the first 3 stages as OTB with transition. The second FC and residual convolution change the feature channels. The convolutions reduce the spatial resolution with the stride of 2. The kernel sizes of DConv and the residual convolution are set to be 5 x 5 and 3 x 3, the Appendix.
3 for
respectively. More details
Experiments
We conduct experiments wide range of vision tasks: image classification ImageNet-Ie 11]. object detection and instance segmentation on COCO 2017 (34), and semantic segmentation on ADE20K [68]. We also take ablation studies to validate the importance of each component. More details about experimental settings and extra ablation studies are in the Appendix.
4.1 Image Classification on ImageNet-1KK
Experimental Settings. We benchmark our Orthogonal Transformer on ImageNet-1K 11 image classification. We follow the same training strategy DeiT (43 and adopt the strong data augmenta- tion and regularization, except for repeated augmentation 22 that does not improve performance. For fair comparison, we do use training data or extra supervision techniques, such token labeling (26) and distillation [4) because most of previous works didn't 't use them. All our models are trained from scratch for 300 epochs with the input size of 224 x 224. An Adam W optimizer with cosine decay learning rate scheduler and epochs of linear warm-up up is employed. The initial learning rate, weight decay, and batch-size are 0.001, 0.05, and 1024, respectively. For Ortho-T, we use smaller weight decay of 0.01. The maximum rates of increasing stochastic depth [[3] are set as 1/0.2/0.4/00. for the models from tiny to large. For the results of 384 x 384, we fine-tune models for 30 epochs with learning rate of le-5, weight decay of 1e-8 and bach-size of 512.

---

Table 3: Object detection and instance segmentation with Mask R-CNN on COCO val2017. FLOPs are measured resolution 800x 1280. All the models are pre-trained on ImageNet-IK.
Mask B-CNN schedule APm
Mask AP
MS schedule
Backhome
(M)
(G)
AP
A.
AP
AP
AP
AP
AP%
APm
.A
APE
Rea50
20
44
260
58.6
41.4
34.4
55.1
41,0
61.7
44.9
37.1
58.4
40.1
PVT-S 500
44
245
40.4
62.9
43,8
37.8
60.1
40.3
43.0
65.3
46.9
39.9
62.5
42.8
Twins-S
63
46.8
69.2
66.3
45.8
Swin-T [5]
48
264
42.2
64.6
46.2
39.1
61.6
42.0
46.0
68.2
50.2
41.6
65.1
44,8
ViL-S 66
45
218
44.9
67.1
49.3
41.0
64.2
44.1
47.1
68.7
51.5
42.7
65.9
46.2
Focal-T [
49
291
67.7
49.2
41.0
64.7
47.2
69.4
51.9
42.7
66.5
45.9
RigS
S
51
183
44.2
40.8
47.6
43.4
Ortho-S
44
42.5
48.7
70.5
47.3
Re:101 20
63
336
40.4
61.1
44.2
36.4
57.7
38.8
42.8
63.2
47.1
38.5
60.1
41.3
X101-32
PVT-M 50
64
302
42.0
64.4
45.6
39.0
61.6
42.1
44.2
66.0
48.2
40.5
63.1
43.5
Twins-B
76
340
45.2
67.6
49.3
41.5
64.5
44.8
48.0
69.5
52.7
43,0
66.8
46.6
Swin-$ 135
69
354
44.8
66.6
48.9
40.9
63. 4
48.5
70.2
53.5
43.3
67.3
46.6
Focal-S 61
71
401
47.4
69.8
51.9
42.8
66.6
46.1
48.8
70.5
53.6
43.8
67.7
47.2
RigionViT-B
93
Orthe-B
69
372
48.3
70.5
53.0
43.3
67.3
46.5
49.9
71.4
$4.8
44.3
68.6
47.9
Table 4: Object detection and instance segmentation with Cascade Mask R-CNN on COCO val2017.
#Params (M)
FLOPs
3x
MS schedule Apm
Method
(G)
Ap
AP 30
AP
AP
AP
DeiT 43
48.0
67.2
51.7
41.4
64.2
44.3
Swin-T
35
86
745
50.5
69.3
54.9
43.7
66.6
47.1
Focal-T
61
87
770
51.5
70.6
55.9
Ortho-S
81
755
52.3
71.3
56.8
45.3
68.6
49.2
X101-32 (59
101
819
48.1
66.5
52.4
41.6
63.9
45.2
Swin-S 35
107
838
51.8
70.4
56.3
44.7
67.9
48.5
Ortho-B
107
851
53.0
72.0
57.4
45.9
69.4
49.9
Results. Table repports the image classification results on ImageNet-IK, It clearly shows that our Orthogonal Transformer has stronger performance than previous models under similar settings of FLOPs and model size. Specifically, the tiny model Ortho-T achieves a 74.0% Top-1 accuracy with only 0.7G FLOPs, surpassing QuadTree-B-bol T2T-7 and PVTv2-bO by 2%, 2.3% and 3.5%, respectively. The small model Ortho-S achieves the same accuracy as base model CrossFormer-B with 51 % fewer FLOPs. As for 384 x 384 input size, the large model Ortho-L achieves accuracy of 85.4%, surpassing Swin-B by 1.2% with similar model size, and outperforming CaiT--448 with 26% fewer FLOPs. These quantitative comparisons with SOTA methods demonstrate the efficiency and effectiveness of Orthogonal Transformer.
4.2 Object Detection on COCO
Experimental Settings. Experiments object detection and instance segmentation are conducted on COCO 2017 dataset (3 Following [35], we use our transformer models as the backbone network, and use Mask-RCNN 18 and Cascaded Mask R-CNN 3 as the detection and segmentation heads. For both tasks, the backbones are pretrained on ImageNet- and fine-tuned on the COCO training set with AdamW optimizer. We take experiments two common settings: "I x (12 training epochs) and "3 x +MS" (36 training epochs with multi-scale training). The configurations follow the setting of 35 and are implemented with MMDetection 7
Results. Table and Table show the results with Mask R-CNN and Cascade Mask R-CNN, respectively. The results show that our method achieves the best performance in all comparisons. For vision transformers with Mask R-CNN framework, our Ortho-S outperforms Focal-T (having similar model size and computational cost) by +2.2 box AP, +1.5 mask AP with the 1 x schedule and +1.5 box AP, +0.9 mask AP with the multi-scale learning schedule. The results with Cascade Mask R-CNN also show that Orthogonal Transformer exceeds the counterparts by evident margins.

---

Table 5: Semantic segmentation with Semantic FPN and Upernet on ADE20K, The FLOPs are
measured at resolution 512x2048.
Semantic FPN 80k
Upernet 160k
Backbone
#Param
FLOPs
mloU
Backbone
#Param (M)
FLOPs
mloU
MS mloU
(M)
(G)
(%)
(G)
(%)
(%)
Res50 20
29
183
36.7
TwinsP-S 9
55
919
46.2
47.5
PVT-S 50
28
161
39,8
Twins-S 9
54
901
46.2
47.1
TwinsP-S
9
28
162
44.3
Swin-T
35
60
945
44.5
45.8
Swin-T
35
32
182
41.5
Focal-T
61
62
998
45.8
47.0
Ortho-S
28
48.2
Ortho-S
54
956
48.5
49.9
Res101 20
48
260
38.8
Res101
20
86
1029
44.9
PVT-L 50
65
283
42.1
Twins-B 9
89
1020
47.7
48.9
TwinsP-L
283
Swin-S 35
1038
47.6
49.5
Swin-S
53
274
45.2
Focal-T 61
85
1130
48.0
50.0
Ortho-B
53
297
49.0
Ortho-B
81
1057
49.8
51.2
Table 6: Ablation Study on three benchmarks using the Ortho-S backbone.
ImageNet-IK
COCO
ADE20K
#Param (M)
FLOPs (G)
Acc. (%)
AP
APm
mloU
window sa
24.0
4.5
46.2
42.0
47.5
shifted window
24.0
4.5
82.5
46.4
42.1
47.6
dilated sa ortho. sa window/ortho.
24.0
4.5
82.9
46.2
41.7
46.6
24.0
4.6
83.2
47.0
42.3
48.0
24.0
4.5
83.4
47.0
42.5
48.2
pos.
23.4
4.4
82.2
44.0
40.4
44.6
abs. pos.
23.4
4.4
82.2
44.1
40.6
44.4
rel. pos.
23.4
4.4
82.4
43.4
40.1
44.6
conv. pos.
24.0
4.5
83.4
47.0
42.5
48.2
w/o early convs
23.9
4.2
82.6
46.5
41.9
47.1
early convs
24.0
4.5
83.4
47.0
42.5
48.2
outside transition
24.0
4.6
83.1
46.7
42.4
48.0
inside transition
24.0
4.5
83.4
47.0
42.5
48.2
4.3 Semantic Segmentation ADE20K
Experimental Settings. Experiments on semantic segmentation are conducted on the ADE20K (6 We employ Semantic FPN (8 and Upernet [56) as the segmentation heads and replace the backbones with our Orthogonal Transformer. For fair comparison, we follow the same setting of PVT (50) to train Semantic FPN 80k iterations and apply the setting of Swin (35) train Upernet for 160k iterations. More details are provided in the Appendix.
Results. The results on semantic segmentation are listed in Table Our method obtains large performance gain over other vision transformers. For methods that using Semantic FPN as the segmentation head, Orthogonal Transformer surpasses Twins by +3.9, +2.6 on mIOU. The results with Upernet indicate that Orthogonal Transformer achieves +2.9, +1.2 absolutely higher Multi- Scale(MS) mIOU than Focal Transformer, and reports 51.2 MS mIOU with 81M parameters. The comparisons on downstream tasks further validate the superiority of our method.
4.4 Ablation Study
In this section, conduct ablation study inspect the respective roles of each part in Orthogonal Transformer, using ImageNet-1K 1K image classification, Mask R-CNN (1x) x on COCO object detection, and Semantic FPN on ADE20K semantic segmentation.
Self-Attention Mechanism. Ablations of the orthogonal self-attention (SA) are reported Table 6 Orthogonal SA outperforms window SA and dilated SA consistently the three tasks. Since window

---

SA and dilated SA belong to fine- -eee-ggrand local SAs and coarse- -erained global SAs respectively. the results may validate the superiority of orthogonal SA in capturing global dependency while preserving local details. The shifted window SA achieves similar performance with window SA. It is designed to allow for cross-window connection. the Ortho-S backbone, DConv in MLP allows for cross- cross-window interaction. That may be why shifted window SA is not superior to window SA herein. Besides, replacing half of orthogonal SA with window SA can lead to slight performance gain with 0.1G decrease of FLOPs, justifying the design of Orthogonal Transformer.
Position Encoding, Comparisons of different position embedding methods are reported in Table g Convolution position embedding (CPE) in PMLP performs better than those without position encoding and those with APE or RPE. The gains are specifically large on downstream nigh-resolution tasks, indicating that the design of our CPE is more suitable for arbitrary large input resolutions.
Patch Embedding. We borrow the idea of early convolutions from (57] to perform overlapped patch embedding. The results in Table 6 indicate that Ortho-S with early convolutions outperforms the counterpart with non-overlapped patch embedding.
Transition. Most of previous works using hierarchical architectures [35 50
perform the transition of token number and feature dimension between stages. In this work, we adopt a different manner and employ the strided convolutions in PMLP to perform transition in the transformer block. The results in Table 6 imply that Ortho-S with inside transition can yield performance gain as well as computation reduction compared to the counterpart with outside transition.
5 Conclusion
We present a new vision transformer, called as Orthogonal Transformer, to serve as a strong general backbone for computer vision. The orthogonal self-attention mechanism is introduced to capture global information while exploring local correlations. Positional MLP developed to handle position information within the transformer block for arbitrary input resolutions. Orthogonal Transformer achieves state-of-the-an performance in various vision tasks, including image classification, ob- ject detection, instance segmentation and semantic segmentation. We expect to apply Orthogonal Transformer for more vision tasks, such as image processing and video prediction.
Acknowledgment
This work is partially funded by National Key Research and Development Program China (Grant No. 2020AAA0140000),, National Natural Science Foundation of China (Grant No. 62006228, U21B2045), and Youth Innovation Promotion Association CAS (Grant No. 2022132).
References
[1] Anurag Arnah, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučié, and Cordelia Schmid. Vivit: video vision transformer. In ICCV, pages 6836-6846, 2021.
[2] Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. In EMNLP, 2020.
[3] Zhaowei Cai and Nuno Vasconcelos. Cascade r-cnn: Delving into high quality object detection. In CVPR,
pages 6154-6162, 2018.
[4] Nicolas Carion, Francisco Massa, Gabriel Synnaeve. Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, pages 213-229, 2020.
[5] Chun-Fu Chen, Rameswar Panda. and Quanfu Fan. Regionvit: Regional-to-local attention for vision transformers. In /CLR, 2022.
[6] Hanting Chen, Yunhe Wang. Tianyu Guo, Chang Xu, Yiping Deng, Zhenhua Liu, Siwei Ma, Chunjing Xu, Chao Xu, and Wen Gao. Pre-trained image processing transformer. In CVPR, pages 12299-12310, 2021. [7] Kai Chen, Jiaqi Wang, Jiangmiao Pang. Yuhang Cao, Yu Xiong. Xiaoxiao Li, Shuyang Sun, Wansen Feng. Ziwei Liu, Jiarui Xu, et al. Mmdetection: Open mmlab detection toolbox and benchmark. arXiv preprint
arXiv:1906.00 00155.0115.11111 2019.
[8] Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, David Belanger, Lucy Colwell, et al. Masked language modeling for proteins via linearly scalable long-context transformers. In ICLR, 2020.
[9] Xiangxiang Chu, Zhi Tian, Yuqing Wang, Bo Zhang, Haibing Ren, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Twins: Revisiting the design of spatial attention in vision transformets. In NeurIPS, 2021.
10

---

[10] Xiangxiang Chu. Zhi Tian, Bo Zhang. Xinlong Wang, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Conditional positional encodings for vision transformers. arXiv preprini arXiv:2102. 108.2 2021.
[11] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hicrarchical image database. In CVPR, pages 248-255, 2009.
[12] Xiaoyi Dong. Jianmin Bao, Dongdong Chen, Weiming Zhang, Nenghai Yu, Lu Yuan, Dong Chen, and Baining Guo. Cswin transformer: A general vision transformer backbone with cross-shaped windows. In
CVPR, 2022.
[13] Alexey Dosovitskiy, Lucas Boyer, Alexander Kolesnikov, Dirk Weissenbom, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold. Sylvain Gelly, et al. An image is worth
16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
[14] Haoqi Fan, Bo Xiong. Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph
Feichtenhofer. Multiscale vision transformers. In ICCV, pages 6824-688 6835, 2021.
[15] Peng Gao. Jiasen Lu, Hongsheng Li. Roozbeh Mottaghi, and Aniruddha Kembhavi. Container: Context aggregation networks. In NewrlPS, 2021.
[16] Jianyuan Guo, Kai Han, Han Wu, Ychui Tang, Xinghao Chen, Yunhe Wang, and Chang Xu. Cmt: Convolutional neural networks meet vision transformers. In CVPR, pages 12175-12185, 2022. [17] Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, and Yunhe Wang. Transformer in transformer. NewrIPS, 34, 2021.
[18] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In ICCV, pages 2961-2969,
2017.
[19] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition.
In CVPR, pages 770-778 2016.
[20] Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CVPR,
pages 770-778, 2016.
[21] Jonathan Ho, Nal Kalchbrenner, Dirk Weissenborn, and Tim Salimans. Axial attention in multidimensional transformers. arXiv preprint arXiv: 1el:. 12180, 2019.
[22] Elad Hoffer, Tal Ben-Nun, Itay Hubara, Niv Giladi, Torsten Hoefler, and Daniel Soudry. Augment your batch: Improving generalization through instance repetition. In CVPR, pages 8129-8138 2020. [23] Gao Huang. Yu Sun. Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In ECCV, pages 646-661, 661, 2016.
[24] Lei Huang, Xianglong Liu, Bo Lang, Adams Wei Yu, Yongliang Wang, and Bo Li. Orthogonal weight normalization: Solution to optimization over multiple dependent stiefel manifolds in deep neural networks.
In AAAI,
[25] Zilong Huang. Xinggang Wang. Lichao Huang, Chang Huang, Yunchao Wei, and Wenyu Liu. Cenet:
Criss-cross attention for semantic segmentation. In ICCV, pages 603 603-612, 2019.
[26] Zi- Zi-Hang Jiang. Qibin Hou, Li Yuan. Daquan Zhou, Yujun Shi, Xiaojie Jin, Anran Wang, and Jiashi Feng.
All tokens matter: Token labeling for training better vision transformers. In NeurIPS.
[27] Jacob Devlin Ming- Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, pages 4171-4186, 2019.
[28] Alexander Kirillov, Ross Girshick, Kaiming He, and Piotr Dollár. Panoptic feature pyramid networks. In
CVPR, pages 6399-6408, 2019.
[29] Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In ICLR, 2019.
[30] Juho Lee, Yoonho Lee. Jungtack Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. Set transformer:
permutation-invarinn neural networks. In ICML, pages 3744-3753, 2019. [31] José Lezama, Qiang Qiu, Pablo Musé, and Guillermo Sapiro. Ole: Orthogonal low-rank embedding- a plug and play geometric loss for deep learning. In CVPR, pages 8109-81 18, 2018.
A framework for attention-based
[32] Jun Li. Fuxin Li, and Sinisa Todorovic. Efficient riemannian optimization on the stiefel manifold via the cayley transform. In ICLR, 2019.
[33] Kunchang Li, Yali Wang, Peng Gao, Guanglu Song, Yu Liu, Hongsheng Li, and Yu Qiao. Uniformer:
Unified transformer for efficient spatiotemporal representation learning. In ICLR, 2022.
[34] Tsung- Lin, Michael Maire, Serge Belongic, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects context. In ECCV, pages 740-755, 2014. [35] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hicrarchical vision transformer using shifted windows. In ICCV, pages 10012-10022, 2021. [36] Iljaa Radosavovic, Raj Prateek Kosaraju. Ross Girshick. Kaiming He, and Piotr Dollár. Designing network design spaces. In CVPR, pages 10436, 2020.
[37] Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier. Efficient content-basse sparse
attention with routing transformers. TACL, 9:53-68, 2021.
[38] Peter Shaw. Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representations. In
NAACL-HLTT (2), 2018
[39] Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid. Segmenter: Transformer for semantic segmentation. In /CCV. pages 7262-7272, 2021.
[40] Mingxing Tan and Quoc Lc. Efficientnet: Rethinking model scaling for convolutional neural networks. In
ICML, pages 6105 6114. PMLR, 2019.
[41] Shitao Tang, Jiahui Zhang, Siyu Zhu, and Ping Tan. Quadtree attention for vision transformers. In ICLR, 2022.
[42] Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. Efficient transformers: A survey, arXiv preprint arxiv:2009.000 06732, 2020.
11

---

[43]
Hugo Touvron. Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In ICML, pages 10347-10357
2021.
[44] Hugo Touvron, Matthiuu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. In ICCV, pages 32-42, 2021.
[45] Frank Uhlig. Constructive ways for generating (generalized) real orthogonal matrices as products of (generalized) symmetries. S
Linear Algebra and its Applications, 132:459-467 2001.
[46] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NewrIPS, pages 5998-6008, 2017.
[47] Ziyu Wan, Jingbo Zhang. Dongdong Chen, and Jing Liao. High-fidelity pluralistic image completion with transformers. In ICCV, pages 4692 -4701, 2021.
[48] Jiayun Wang. Yuhei Chen, Rudrasis Chakraborty, and Stella X Yu. Orthogomal convolutional neural networks. In CVPR, pages 11505-11511 2020.
[49] Sinong Wang. Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with lincar complexity. arXiv preprint arXiv: 04768, 2020.
[50] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: versatile backbone for dense prediction without convolutions.
In pages 568-578. 2021.
[51]
Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping -ing Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pvt v2: Improved baselines with pyramid vision transformer. CVM, pages 1-10, 2022.
[52] Wenxiao Wang, Lu Yao, Long Chen, Binbin Lin, Deng Cai, Xiaofei He, and Wei Liu. Crossformer:
versatile vision transformer hinging on cross-scale attention. In ICLR, 2022.
[53] Yuqing Wang. Zhaoliang Xu, Xinlong Wang, Chunhua Shen, Baoshan Cheng. Hao Shen, and Huaxia Xia. End-to-end video instance segmentation with transformers. In CVPR, pages 8741-8750, 2021. [54] Yuxiang Wei, Yupeng Shi, Xiao Liu, Zhilong Ji, Yuan Gao, Zhongqin Wu, and Wangmeng Zuo. Orthogonal jacobian regularization for unsupervised disentanglement in image generation. In ICCV, pages 6721-6730. 2021.
[55] Haiping Wu, Bin Xiao, Noel Codella, Mengehen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. Cvt:
Introducing convolutions vision transformers. In ICCV, pages 22-31, 2021.
[56] Tete Xiao, Yingcheng Liu, Bolei Zhou. Yuning Jiang, and Jian Sun. Unitied perceptual parsing for scene understanding. In ECCV, pages 418-434, 2018.
[57] Tete Xiao, Mannat Singh, Eric Mintun, Trevor Darrell, Piotr Dollár, and Ross Girshick. Early convolutions
help transformers see better. NewrIPS, 34:30392-30400 2021.
[58] Di Xie, Jiang Xiong, and Shiliang Pu. All you need beyond a good init: Exploring better solution for training extremely deep convolutional neural networks with orthonormality and modulation. In CVPR, pages 6176-6185, 2017.
[59] Saining Xie, Ross B. Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transfor-
mations for deep neural networks. CVPR, pages 5987-5995, 2017.
[60] Weijian Xu, Yifan Xu, Tyler Chang, and Zhuowen Tu. Co-scale conv-attentional image transformers. In ICCV, pages 9981-9990, 2021.
[61]
Jianwei Yang, Chunyuan Li, Pengchuan Zhang, Xiyang Dai, Bin Xiao, Lu Yuan, and Jianfeng Gao. Focal celf-attention for local-global interactions in vision transformers. In NeurlPS, 2021. Fisher Yu and Vladlen Koltun. Multi-scale context aggregation by dilated convolutions. arXiv preprint
[62]
arXiv: 1007 2015.
[63] Qihang Yu, Yingda Xia, Yutong Bai, Yongyi Lu, Alan L Yuille, and Wei Shen. Glance-and -aze vision
transformer. NeurIPS, 34, 2021.
[64]
Kun Yuan, Shaopeng Guo, Ziwei Liu, Aojun Zhou, Fengwei Yu, and Wei Wu. Incorporating convolution designs into visual transformers. In ICCV, pages 579-588. 2021.
[65] Li Yuan, Yumpeng Chen, Tao Wang, Weihao Yu. Yujun Shi. Zi- Zi-Hang Jiang, Francis EH Tay, Jiashi Feng. and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenct. In ICCV, pages 558-567, 2021.
[66] Pengchuan Zhang, Xiyang Dai, Jianwei Yang, Bin Xiao, Lu Yuan, Lei Zhang, and Jianfeng Gao. Multi- scale vision longformer: A new vision transformer for high-resolution image encoding. In ICCV, pages 2998 3008, 2021.
[67] Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, 7ekun Luo, Yahiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip HS Torr, al. Rethinking semantic segmentation from sequence-to-sequenee perspective with transformers. In CVPR, pages 6881-6890, 2021.
[68] Bolei Zhou, Hang Zhao, Xavier Puig. Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Toralba. Semantic undcrstanding of scenes through the adc20k dataset. IJC,, 127(3):00--321, 2019.
[69] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. In ICLR, 2020.
12


Reviews:
[{"review_id": "nBLDZOpLEK", "reviewer": "Reviewer_GmQX", "summary": "This paper introduces orthogonal transformation of tokens and combines it with the idea of interleaving window self attention (WSA) and dilated self-attention (DSA) to capture both local and global interactions without incurring the prohibitive cost of full global self attention. They showed that initialization and optimization of orthogonal matrices can be simplified by using Houholder transformations and leveraging already established optimization techniques to learn these matrices. In addition, the paper also make use of Positional MLPs where MLPs are equipped with depth-wise convolutions to allow for downsampling within the transformer block. \n\nAuthors individually ablated different techniques proposed in the paper and combined the best configurations to achieve top performance on wide range of image tasks.", "questions": "line 44: Authors claim that one of the advantages of orthogonal transformations is that the tokens can be separated into linearly independent groups. But, there is no further explanation as to why this linear independency is important or how this contributes to the overall performance of the model.\n\nequation 5: Authors present complexity analysis in comparison with the global self-attention. But, it is only fair to compare against dilated self attention complexity. ", "limitations": "Given that the paper is just concatenation of already existing ideas with minimal novelty of newly introduced techniques, I am not inclined to accept the paper. ", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The main differentiating contribution of the paper is using the Householder transformation trick to introduce orthogonal transformations. This idea is coupled with already existing techniques of window self-attention and dilated self-attention. Authors present extensive experiments ablating each individual technique introduced. The paper is also well written and easy to read and understand. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "XvY-nWgGhUQ", "reviewer": "Reviewer_hQkC", "summary": "This paper proposed an efficient Transformer backbone, Orthogonal Transformer, for efficient processing of visual inputs. The backbone is mainly built on the proposed efficient self-attention mechanism, orthogonal self-attention (OSA), to model dependencies between tokens in the orthogonal space. The proposed backbone achieves excellent performance-FLOPs tradeoff on visual tasks.", "questions": "The questions about the paper are listed in the Weaknesses.", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\nThe performances of the proposed backbone are superior.\nThe paper is well written and easy to follow, and the illustration is clear to me.\n\n### Weaknesses\nThe motivation is not clear to me. Why do other efficient self-attention approaches fail? The author claimed other efficient self-attention mechanisms \"lose fine-level details for coarse global self-attention or hurting long-range modeling for local self-attention\" (L35-36). However, I cannot find any pieces of evidence to support the claim. In addition, how does the proposed orthogonal self-attention (OSA) work? The author claimed OSA \"captures global dependency without losing fine-level details\" (L37-38). I have found no evidence to support the claim, either theoretically or empirically.\n\nThe contribution is not significant. I believe the only novel part of this paper is the proposed efficient self-attention approach.  The proposed Positional MLP in Sec 3.4 has been studied in PVT v2 [CVMJ 22], CeiT [ICCV 21], and CMT [CVPR 22]. So I would not say this is the novel part. \n\nThe ablation is not sufficient. In Tab 6, the authors compared the proposed OSA with window SA and dilated SA. How about the window SA with shified window like Swin?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "OpeZGExH5jL", "reviewer": "Reviewer_SBMR", "summary": "The paper proposed an token orthogonalization layer, there self-attention is performed on groups of orthogonalized tokens. The paper then designs the Orthogonal Transformer (OT) which combines the orthogonal self-attention layer with a pyramid Transformer architecture, positional MLPs (adding a depthwise-convolution into the MLP layer), early convolutions, and a novel downsampling method between stages. OT performs well on ImageNet-from-scratch, COCO detection and instance segmentation, and ADE20k semantic segmentation.", "questions": "Please address my main concern about the orthogonalization layer. Given that the orthogonal self-attention layer is marketed the main contribution of the paper it would be useful to have an experiment that demonstrates the value of this layer in isolation. I think the best way to do this would be to run a vanilla ViT/DINO setup with just the OT layer replacing self-attention, but without any of the additional changes/tricks. This would help determine with the OT layer is useful in isolation, or it requires the other network modifications in addition to be a useful component.\n\nSecond, you answered \"Yes\" to \"Did you include the code, data, and instructions needed to reproduce the main experimental result?\". I did not see any mention of opensourced code, will code be made available?", "limitations": "Section F in the appendix dedicated to limitations and societal impact. The societal impact is adequately addressed. The limitations section is brief, but I think it is adequate, and i believe captures the main limitation that the study is restricted only to image-based tasks.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n* I like the proposal of constructing a trainable orthogonalization module from the product of Householder matrices. The orthogonal self-attention layer is fairly simple and general, and I think is a nice contribution.\n\n* The experiment cover multiple tasks, appear thorough, and compare to many state-of-the-art alternatives.\n\n* There is an ablation of the four components of the orthogonal Transformer.\n\nWeaknesses\n\n* I think the main weakness is the large number of moving parts in OT: the combination of both windowed attention and orthogonal attention, addition convolutions at the start and in the middle of the network, and a new downsampling mechanism. The combination of these factors significantly increases the complexity of the network over the original Transformer/Vision Transformer, potentially limiting adoption. The ablation study shows that some of the \"minor\" components have an equal-or-greater impact than the orthogonal attention layer which is the main \"selling point\" for OT. E.g. Conv position embeddings improve the scores on all tasks over absolute position embeddings more than orthogonal attention. Therefore, it feels like the orthogonal self-attention layer is not the key driver of performance in OT.\n\n* Given that the network contains many vision-specific components (convolutions), I feel that \"Orthogonal Transformer\" is over-selling or over-generalizing the network. A name like \"Orthogonal Vision/Image Transformer\" would make it more clear that this is a vision-specific variant.\n\n* There are a few typos, such as:\n\n\"tokens has a lower resolution\" -> have\n\n\"singe OTB\" -> single OTB", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "CDjIURn-UzT", "reviewer": "Reviewer_2jJ5", "summary": "The paper proposes a new efficient self-attention form  (Orthogonal Self-Attention, OSA) that conducts self-attention within groups, where orthogonalization of window tokens are performed first before forming the groups. The final architecture consists of alternative OSA and window self-attention (WSA) as attention, with FFN equipped with depth-wise convolutions. It shows competitive results on ImageNet classification, COCO object detection, ADE20k semantic segmentation. ", "questions": "1.\tAs mentioned in weakness, The reversible operation is the novel part, but no ablation on that. Experiments do not show the essentialness of using orthogonal matrix instead of regular one. What if we keep the form (i.e. using non-orthogonal matrix to transform tokens within local windows, and then group them in same way as current strategy to perform self-attention), then no token reverse as inverse of weights is hard to get, maybe replacing multiplying the inverse of A with another new weights A’. \n\n2.\tLine 41 mentioned that tokens have lower resolution in the orthogonal space, can the authors provide more details regarding this? From my understanding, the new tokens obtained from multiplying the orthogonal matrix will not change dimensions, Z’ = AZ, Z’ is (n x c), A is (n x n), Z is (n x c), n = h x w.  The computation saving for attention here is from conducting group-wise attention, which does not seem to be brought from orthogonalization of tokens, but from window and group partitions of tokens. \n3.\tHow to choose window size for WSA and OTB? Will different size affect the final performance? \n\n\n\n-----post-rebuttal---\nThanks for the authors presenting the ablation study results on the essentialness of orthogonalization during rebuttal. Still, considering that the gain from ortho. sa is limited, 0.3% compared to dilated sa as in Table 6, I'll keep my initial rating as borderline accept. ", "limitations": "NA", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n \tThe way to construct orthogonal matrix endogenously by multiplying householder reflectors is novel and interesting. The OSA form proposed enabled the token connections within local window and links across windows. \n\tThe experiments results show competitive performance across different vision tasks, including image classification, object detection, semantic segmentation.\n\nWeaknesses:\n\tThe essentialness of using orthogonal matrix is not studied. \nThe whole OSA process \n1.\tconnects tokens within local windows with local window token orthogonalization, is serves as MLP layer within local windows, except the weight matrix of MLP is naturally orthogonal.\n2.\tConnects tokens beyond local windows by forming new groups across previous local window. \n3.\tToken reverse as the inverse of orthogonal matrix is easy to get, just the transpose of the matrix. \n            Step 2 can be done regardless of the weight matrix of this local window MLP is orthogonal or not. \n\tStep 3 is the vital part that only orthogonal matrix weight can perform, I believe this should be studied, which is not presented, for validating the essentialness of using orthogonal matrix rather than just following the form that connects local and connects beyond local windows. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept