CrossFormer: A Versatile Vision Transformer Hinging on Cross-scale Attention
Authors: Wenxiao Wang
Lu Yao
Long Chen
Binbin Lin
Deng Cai
Xiaofei He
Wei Liu
Venue: ICLR 2022 Poster (2022)
Keywords: vision transformers
architecture

Abstract: Transformers have made great progress in dealing with computer vision tasks. However, existing vision transformers have not yet possessed the ability of building the interactions among features of different scales, which is perceptually important to visual inputs. The reasons are two-fold: (1) Input embeddings of each layer are equal-scale, so no cross-scale feature can be extracted; (2) to lower the computational cost, some vision transformers merge adjacent embeddings inside the self-attention module, thus sacrificing small-scale (fine-grained) features of the embeddings and also disabling the cross-scale interactions. To this end, we propose Cross-scale Embedding Layer (CEL) and Long Short Distance Attention (LSDA). On the one hand, CEL blends each embedding with multiple patches of different scales, providing the self-attention module itself with cross-scale features. On the other hand, LSDA splits the self-attention module into a short-distance one and a long-distance counterpart, which not only reduces the computational burden but also keeps both small-scale and large-scale features in the embeddings. Through the above two designs, we achieve cross-scale attention. Besides, we put forward a dynamic position bias for vision transformers to make the popular relative position bias apply to variable-sized images. Hinging on the cross-scale attention module, we construct a versatile vision architecture, dubbed CrossFormer, which accommodates variable-sized inputs. Extensive experiments show that CrossFormer outperforms the other vision transformers on image classification, object detection, instance segmentation, and semantic segmentation tasks.

CROSSFORMER: A VERSATILE VISION TRANSFORMER HINGING ON CROSS-SCALE ATTENTION
Wenxiao Wang 1,2 Lu Yao', Long Chen², Binbin Lin Deng Cai', Xiaofei He' & Wei Liu 1 State Key Lab of CAD & CG, Zhejiang University
Data Platform, Tencent
Columbia University
"School of Software Technology, Zhejiang University
ABSTRACT
Transformers have made great progress in dealing with computer vision tasks. However, existing vision transformers have not yet possessed the ability of build- ing the interactions among features of different scales, which is perceptually im- portant visual inputs. The reasons are two-fold: (1) Input embeddings of each layer are equal-scale, so no cross-scale feature can be extracted; (2) to lower the computational cost, some vision transformers merge adjacent embeddings inside the self-attention module, thus sacrificing small-scale (fine-grained) fea- tures of the embeddings and also disabling the cross-scale interactions. To this end, propose Cross-scale Embedding Layer (CEL) and Long Short Distance Attention (LSDA). On the one hand, CEL blends each embedding with multiple patches of different scales, providing the self-attention module itself with cross- scale features. On the other hand, LSDA splits the self-attention module into a short-distance and long-distance counterpart, which not only reduces the computational burden but also keeps both small-scale and large-scale features the embeddings. Through the above designs, we achieve cross-scale atten- tion. Besides, we put forward dynamic position bias for vision transformers to make the popular relative position bias apply variable-sized images. Hinging on the cross-scale attention module, we construct versatile vision architecture, dubbed CrossFormer, which accommodates variable-sized inputs. Extensive ex- periments show that CrossFormer outperforms the other vision transformers on image classification, object detection, instance segmentation, and semantic seg- mentation tasks.
INTRODUCTION
It turns out that transformer Vaswani et al., 2017: Devlin et al., 2019: Brown et al.) 2020) has achieved great success in the field of natural language processing (NLP). Benefitting from its self- attention module, transformer is born with the key ability to build long-distance dependencies. Since long-distance dependencies are also needed by a number of vision tasks (Zhang & Yang 2021 Chu
et al. 2021), surge of research work (Dosovitskiy al. 2021 Touvron et al. 2021 Wang et al. 2021 has been conducted to explore various transformer- based vision architectures.
A transformer requires sequence of embedding Fe e.g., word embeddings) input. To adapt this requirement to typical vision tasks, most existing vision transformers (Dosovitskiy et al.) 202T Touvron et al. 2021 Wang et al. 2021 Liu et al., 2021b) produce embeddings by splitting an input image into equal-sized patches. For example, a 224 x 224 image can be split into 56 x 56 patches of size 4 x 4, and these patches are projected through linear layer to yield embedding sequence. Inside a certain transformer, self-attention is engaged to build the interactions between any two embeddings. Thus, the computational or memory cost of the self-attention module is O(N²), where N is the length of an embedding sequence. Such cost is too big for visual input because its embedding sequence much longer than that of NLP. Therefore, the recently proposed vision
* TThe corresponding authors. 1 'The code has been released: httpggihhhhoooo/cceess//oooooooe
In this paper, we also use "embeddings" to represent the input of each layer.

---

transformers (Wang et al. 2021 Liu et al. 2021b Lin al. 2021) develop multiple substitutes to approximate the vanilla self attenntton module with a lower cost.
Though the aforementioned vision transformers have made some progress, they do not explicitly utilize features of different scales, whereas multi-scale features are very vital for a lot of vision tasks. For example, image often contains many objects of different scales, and to fully understand the image, building the interactions among those objects is helpful. Besides, some particular tasks such as instance segmentation need the interactions between large-scale (coarse-grained) features and small-scale (fine-grained) ones. Existing vision transformers fail to deal with the above cases due two reasons: (1) The embeddings are generated from equal-sized patches. Though these patches theoretically have a chance to extract any scale features if only the receptive field is large enough, it difficult to promise that they can learn appropriate multi-scale features automatically in practice. (2) Inside the self-attention module, adjacent embeddings are often grouped together and merged (Wang al., 202T). Since the number of groups is smaller than that of embeddings, such a behavior can reduce the computational budget of the self-attention. In this case, however, even if embeddings have both small-scale and large-scale features, merging operations will lose the small-scale features of each individual embedding, thereby disabling the cross- scale attention.
To enable the building of cross-scale interactions, we co-design a novel embedding layer and a self- attention module as follows. 1) Cross-scale Embedding Layer FFollwwnn) Wang al. (2021), we also employ a pyramid structure for our transformer, which naturally splits the vision transformer model into multiple stages. CEL appears the start of each stage, which receives last stage's output (or an input image) as input and samples patches with multiple kernels of different scales (e.g.. 4 x 4 or 8 x 8). Then, each embedding Through this way, we enforce some dimensions (e.g., dimensions from 4 x 4 patches) to focus on small-scat scale features only, while others (e.g., those from 8 x patches) have a chance to learn large- scale features, leading to an embedding with explicitly cross-scale features. 2) Long Short Distance Attention (LSSDA) We propose a substitute of the vanilla self-attention, but to preserve small-scale features, the embeddings will not be merged. In contrast, we split the self-aftention module into Short Distance Attention (SDA) and Long Distance Attention (LDA). SDA builds the dependencies among neighboring embeddings, while LDA takes charge of the dependencies among embeddings far away from each other. The proposed LSDA can also reduce the cost of the self-attention module like previous studies (Wang et al. 2021), but different from them, LSDA does not undermine either small-scale or large-scale features. As a result, attention with cross scale interactions is enabled.
-
constructed by projecting and concatenating these patches.
Besides, following prior work (Shaw al. 2018: Liu al. 2021b), we employ a relative position bias for embeddings' position representations. The Relative Position Bias (RPB) only supports fixed image/group size However, image size for many vision tasks such as object detection is variable, so does group size for many architectures, including ours. To make the RPB more flexible, we further introduce a trainable module called Dynamic Position Bias (DPB), which receives two embeddings' relative distance as input and outputs their position bias. The DPB module is optimized end-to-end in the training phase, inducing an ignorable cost but making RPB apply to variable image/group size.
All our proposed modules can be implemented with about ten lines of code. Based on them, we construct four versatile vision transformers of different sizes, dubbed CrossFormers. Other than im- age classification, the proposed CrossFormer can handle a variety of tasks with variable-sized inputs such as object detection. Experiments on four representative vision tasks (i.e., image classification, object detection, instance segmentation, and semantic segmentation) demonstrate that CrossFormer outperforms the other state-of-the-art vision transformers on all the tasks. Remarkably, the perfor- mance gains brought by CrossFormer are substantially significant dense prediction tasks, e.g., object detection and instance/semantic segmentation.
It is worth highlighting our contributions as follows:
We propose cross-scale embedding layer (CEL) and long short distance attention (LSDA), which together compensate for existing transformers' incapability of building cross-scale attention.
The dynamic position bias module (DPB) is further proposed to make the relative position bias more flexible, i.e., accommodating variable image size or group size.
SSoe vision transformers split input embeddings into several groups. Group size means the number of emheddings in group
2

---

Ho x xD
We
Ho Wo x x20
Mo Wo x4D
Ho. We 32 32 x80
16
(4x4 20d, 10-10 10-16,,3200 32x53)
Coossoomer Black Xn,
Crossoomeer Block Xh2
Croad CrosFoomer Fomner Block Xity
Coesueemee Blockk Xnu
Classification Head
Ho
Wa
12x2 4x4)
12x2 (x4)
12x2, 4x4)
CEL
CEL
CRL
CEL
SDA
LDD
DPB
DPR
LN
Stage-1 (a) The architecture of CrossFormer for image classification.
Stage-2
Stage-3
Stage-4
(b) Two consecutive CrossFommr blocks.
Figure 1: (a) The architecture of CrossFormer for classification. The input size is Ho Wo, and the size of feature maps in each stage is shown on the top. Stage-i consists blocks. Numbers in CELs represent kernels' sizes used for sampling patches. (b) The inner structure of two consecutive CrossFormer blocks. SDA and LDA appear alternately in different blocks.
Consess of a CEL and n, CrossFormer
Multiple CrossFormers with different sizes are constructed, and we corroborate their effective- ness through sufficient experiments on four representative vision tasks.
BACKGROUND
Vision Transformers. Motivated by the transformers developed for NLP, researchers design specific visual transformers for vision tasks take full advantage of their powerful attention mechanism. In particular, VIT and DeiT transfer the original transformer Vaswani et al. (017)) to vision tasks (Tou- vron al. [201 Dosovitskiy et al., 2021). achieving impressive performance. Later, PVT (Wang et al. 2021). HVT (Pan et al.) 2021). Swin (Liu et al. (021b), and VITAE (Xu et al.) Ib) intro- duce a pyramid structure into the visual transformers, greatly decreasing the number of patches in the later layers of respective model. They also extend the visual transformers to other vision tasks like object detection and segmentation (Wang et al.
2021
Liu et al. 2021b).
Cross-scale Feature Extraction. SZzgedy et al. (2015): Tan & Le (2019) multi-scale convo- lutional kernels at every layer of models for cross- scale features, while the largest kernel size can only be 7 x and larger kernel size will induce unaccpetable computational budget. For vision transformers, CoaT (Xu et al. 2021a) uses multi-scale features at later layers of models by mixing features from different layers: CViT (Chen et al.) 2021a) keeps embeddings of different scales in different branches, and introduces cross-scale interaction through self-attention between branches.
More discussions can be seen in the appendix C.I
Sparse Self-attention. Many substitutes have been proposed Liu et al. 202 Wang et al. 2021 Chu et al. 2021 Child et al., 2019 Beltagy al. 2020 Zaheer et al. 2020) to alleviate the cost of the vanila sell- -atteentinoon module. Instead of going sell-attention among all embeddings, each embedding only interacts with part of other embeddings, thus changing dense self-attention sparse attention. More discussions can be seen in the appendix C.2
Position Representations. To make the respective model aware of position information of embed- dings, many different position representations (Vaswani et al. 2017) are proposed. For example, Dosovitskiy et al.] (202T) directly added the embeddings with the vectors that contain absolute po- sition information. In contrast, Relative Position Bias (RPB) (Shaw et al. 2018) resorts to position information indicating the relative distance of two embeddings.
CROSSFORMER
The overall architecture of CrossFormer is plotted in Figure Following (Wang al. 2021 Liu
et al. 2222bb] Lin et al. 2021), CrossFormer also employs a pyramid structure, which naturally splits the transformer model into four stages. Each stage consists of cross- embedding layer (CEL, Section 3.J) and several CrossFormer blocks (Section 3.2h. A CEL receives last stage's output (or an input image) input and generates cross-scale embeddings. In this process, CEL (except that in Stage-/) reduces the number of embeddings to a quarter while doubles their dimensions for a pyramid structure. Then, several CrossFormer blocks, each of which involves long short distance attention (LSDA) and dynamic position bias (DPB), are set up after CEL. A specialized head (e.g., the classification head in Figure TT follows after the final stage accounting for specific task.
3

---

3.1 CROSS-SCALE EMBEDDING LAYER (CEL)
Cross-scale embedding layer (CEL) is leveraged to generate input embeddings for each stage. Fig- ure 2 takes the first CHL, which is ahead of Stage-l, /, as an example. It receives an image as input, then sampling patches using four kernels of different sizes. The stride of four kernels is kept the same so that they generate the same number of embed- dings" As we can observe in Figure every four corresponding patches have the same center but different scales, and all these four patches will be projected and concatenated as one embedding. In practice, the process of sampling and projecting can be fulfilled through four convolutional layers.
Embedding
Concarenate
64 dims
32 dims
16 LG
dims dims
For cross-scale embedding, one problem is how to set the projected dimension of each scale. The computational budget of a convolutional layer is proportional to D²D where K and D represent kernel size and input/output dimension, respec- tively (assuming the input dimension equals to the output dimension). Therefore, given the same dimension, large kernel consumes more budget than a smaller one. To control the total budget the CEL, we use lower dimension for large ker- nels while a higher dimension for small kernels. Figure [ provides the specific allocation rule in its subtable, and a 128-dimensional example is given. Compared with allocating the dimension equally, our scheme saves much computational cost but does not explicitly affect the model's per- formance. The cross-scale embedding layers in other stages work Figure [] CELS for Stage-2/3/4 use two different kernels (2 x 2 and 4 x 4). Further, to form a pyramid structure, the strides of CELs for Stage-2/3/4 are set as 2 x 2 reduce the number of embeddings to a quarter.
Embedding Layer
Type
Kernel
Stride
Dim
Conv,
4 x
4 x 4
Dx
Conv,
4 4
22
Conv. 16 16
4 4
Conv.
32 x
x
9.
Figure 2: Illustration of the CEL layer. The in- put image is sampled by four different kernels (i.e., 4 x 4,8 8 x 8, 16 x 16, 32 x 32) with same stride 4 4. Each embedding constructed by projecting and concatenating the four patches. D; means the total dimension of the embedding.
similar way. As shown in
3.2 CROSSFORMER BLOCK
Each CrossFormer block consists of a long short distance attention module (i.e., LSDA, which in- volves a short distance attention (SDA) module or a long distance attention (LDA) module) and a multilayer perceptron (MLP). As shown in Figure SDD and LDA appear alternately in different blocks, and the dynamic position bias (DPB) module works in both SDA and LDA for obtaining embeddings" position representations. Following the prior vision transformers, residual connections are used in each block.
3.2. LONG SHORT DISTANCE ATTENTION (LSDA)
We split the self-attention module into parts: short distance attention (SDA) and long distance attention (LDA). For SDA, every G x G adjacent embeddings are grouped together. Figure gives an example where G 3. For LDA with input of size S, the embeddings are sampled with fixed interval I. For example in Figure 3p (I 3). all embeddings with red border belong to group, and those with yellow border comprise another group. The group's 's height width for LDA is computed as G = 5/T (i.e., G = 3 in this example). After grouping embeddings, both SDA and IDA employ the vanilla self-attention within each group. As a result, the memorylcomputations cost of the self-attention module is reduced from O(8*) to OSS and G < S in most cases.
is worth noting that the effectiveness of LDA also benefits from cross-scale embeddings. Specif- ically, we draw all the patches comprising two embeddings in Figure 3b. As we can see, the small- scale patches of two embeddings are non-adjacent, so is difficult to judge their relationship without the help of the context. In other words, it will be hard build the dependency between these two embeddings if they are only constructed by small-scale patches (i.e., single-scale feature). On the contrary, adjacent large-scale patches provide sufficient context to link these embeddings, which makes long-dicg ding-distance cross-scale attention easier and more meaningful. *The image will be padded necessary.
4

---

G=3
(Axty. Ayr)
Ayr))
Linear, D/4
G-3
LN
x3
ReLUU
Linear, 1
BLI
(a) SDA
(b) LDA
(c) DPB
Figure 3: (a) Short distance attention (SDA). Embeddings (blue cubes) are grouped by red boxes. (b) Long distance attention (LDA). Embeddings with the same color borders belong to the same group. Large patches of embeddings in the same group are adjacent. (c) Dynamic position bias (DBP). The dimensions of intermediate layers are D 4. and the output a scalar.
We provide the pseudo-code of LSDA in the appendix A. Based on the vanilla multi- self- attention, LSDA can be implemented with only ten lines of code. Further, only reshape and permute operations are used, so no extra computational cost is introduced.
3.2.2 DYNAMIC POSITION BIAS (DPB)
Relative position bias (RPB) indicates embeddings' relative position by adding a bias to their atten- tion. Formally, the LSDA's attention map with RPB becomes:
Attention = Sofftma /Va V B)
(1)
where Q, K, V E G xD represent query, key, value in the self-attention module. respectively. vd constant normalizer, and B E p x2² is the RPB matrix. In previous work (Liu et al. 2021b), Br, B Avy. where B fixed-sized matrix, and (Aziy, ABtr) is the coordinate distance between the ish and Jen embeddings. It is obvious that the image/group size is restricted in case that (Amij. Avú;) exceeds the size of B. In contrast, we propose an MLP-based module called DPB to generate the relative position bias dynamically, i.e.
B DPB(Ar AViz).
(2)
The structure of DPB is displayed in Figure Se. Its non linear transformation consists of three fully- connected layers with layer normalization input dimension of DPB is 2, i.e., (Arij, Aw;;), ). and intermediate layers' dimension set D D/4, where is the dimension of embeddings. The output Bi, is a scalar, encoding the relative position feature between the ith and jen embeddings. DPB is trainable module optimized along with the whole transformer model. It can deal with any image/group size without worrying about the bound of (Arij. Ajrr). In the appendix (..2)
al. 2016) and ReLU (Nair & Hinton 2010). The
prove that DPB is equivalent to RPB if the image/group size is fixed. In this case, we can transform a trained DPB to RPB the test phase. We also provide efficient O(G?) implementation of DPB when image/group size is variable (the complexity is O(G)) normal case because E G²²x)
3.3 VARIANTS OF CROSSFORMER
Table lists the detailed configurations of CrossFormer's four variants (-T. -S, -B, and -L for tiny, small, base, and large, respectively) for image classification. To re-use the pre-trained weights, the models for other tasks (e.g., object detection) employ the same backbones as classification except that they may use different G and /. Specifically, besides the configurations same to classification, we also test with G 22 14, I 16, and /2 8 for the detection (and segmentation) models' first two stages to adapt to larger images. The specific configurations are described in the appendix A.3). Notably, the group size the interval (i.e.. G I) does not affect the shape of weight tensors, so the backbones pre-trained on ImageNet can be readily fine-tuned on other tasks even though they different G or T.
5

---

Table 1: Variants of CrossFormer for image classification. The example input size 224 224. S represents the feature maps' height (and width) of each stage. D and H mean embedding dimensions and the number of heads in the multi-head self-attention module, respectively. and I are group size and interval for SDA and LDA, respectively.
Ouspot Size
Layer Name
CrossFormer-T
CrossForme-SS
CrossFormer-B
CrossFormerLL
Cross Embed.
Kernel size: x 4,8 x.8.8 16 16, x 32, Stride=1
56 36
[D 64
[. 967
125
Stage-1
(81 56)
SDALDA MI.P
Gi
Hi
x1
G1
H
x2
Hj
G: X2
G1
H1
x
4
4 =
Cross Embed.
Kernel 2, 4 4, Stride=2
28 28
D2 = 128
D2
1921
D2 1927
D2 256
Stage
SDA/LDA MLP
H,
H1 8
(82 28)
G2 7
C2
X
G2
X
G2
X
12
1/2
Cross Embed.
Kernel size: 2 x 2, x Strid=22
Stage-3
14 14 (Sg 14)
SDA/LDA MLP
D2
H1 8
Dg
Hg 12
D 3847
Hg 12
D2
Hy 16
G 2
Ga
==
1
49 =
Fimbed.
Kernel 2, 4, Stride22
Stage-4
x (8, 7)
SDA/LDA MI.P
D2 5121
Hg
D: 7681
H₂
D; 768
D4 1021)
32
G4 -
x
G4 7
x
C4 7
x
G.
x2
4-1
14
14
Head
Avg Pooling
Kernel size: T
Linear
Classes: 1000)
EXPERIMENTS
The experiments carried out four challenging tasks: image classification, object detection, in- stance segmentation, and semantic segmentation. To entail a fair comparison, we keep the same data augmentation and training settings as the other vision transformers far possible. The competi- tors are all competitive vision transformers, including DeiT (Touvron et al. [202I). PVT (Wang et al.,
2021) T2T-ViT (Yuan et al.)
2021). TNT (Han al.) (202T). CVIT (Chen et al. 2021a). Twins (Chu et al. 2021), Swin (Liu et al.. al. 2021a), CAT (Lin et al.
2021b), NesT (Zhang et 20.1 CVT et al. [2021] ViL (Zhang
(021), ) ResT (Zhang RZeTnZZaag & Yang 2021), TransCNN (Liu et al. (021a),
Shuffle (Huang et al. 2021). BoTNee (Srinivas et al.
202 and RegionViT (Chen et al. 2021b).
4.1 IMAGE CLASSIFICATION
Experimental Settings. The experiments on image classification are done with the ImageNet (Rus- sakovsky et al. 2015) dataset. The models are trained on 1.28M training images and tested on 50K validation images. The same training settings as the other vision transformers ticular, we use an AdamW (Kingma & Ba 2015) optimizer training for 300 epochs with cosine decay learning rate scheduler, and 20 epochs of linear warm-up are used. The batch size is 1,024 split on 8 V100 GPUs. An initial learning rate of 0.001 and a weight decay of 0.05 are used. Be- sides, we use drop path rate of 0. 0.2, 0.3, 0.5 for CrossFormer--,, CrossFormer-S, CrossFormer--. CrossFormer-L. respectively. Further, Similar to Swin (Liu et al. 2021b), RandAugment (Cubuk al. 2020), Mixup (Zhang et al. (018a). Cutmix (Yun et al. 2219), ) random erasing (Zhong al.
adopted. In par-
2020 and stochastic depth (Huang et al. 2016) are used for data augmentation.
Results. The results are shown in Table 2 As we can see, CrossFormer achieves the highest accu- racy with parameters and FLOPs comparable to other state-of-the-an vision transformer structures. In specific, compared against strong baselines DeiT, PVT, and Swin, our CrossFormer outperforms them at least absolute 1.2% in accuracy on small models. Further, though Region ViT achieves the same accuracy (82.5%) as ours on small model, it is 0.7% (84.0% vs. 83.3%) absolutely lower than ours on the large model.
4.2 OBJECT DETECTION AND INSTANCE SEGMENTATION
Experimental Settings. The experiments on object detection and instance segmentation are both done on the COCO 2017 dataset (Lin et al. 2014). which contains 1 188 training and 5K val images.

---

Table Results the ImageNet validation set. The imput size is 224 224 for most models, while is 384 x 384 for the model with a Results of other architectures are drawn from original papers.
Architectures
#Params
FLOPs
Acc.
Architectures
Paramss
FLOPs
Acc.
PVT-S
24.5M
3.8G
79.8%
BoTNet-SI-59
33.5M
7.3G
81.7%
RegionViT-T
13.8M
2.4G
80.4%
PVT-L
61.4M
9.8G
81.7%
Twins-SVT-S
24.0M
2.8G
81.3%
CvT-21
32.0M
7.1G
82.5%
CrossFormer-T
27.8M
2,9G
81.5%
CAT-B
52.0M
8.9G
82.8%
Swin-S
S0.0M
8.7G
83.0%
DeîT-S
22.1M
4.6G
79.8%
TIT-VTT
21.5M
5.2G
80,7%
Region
Twins-SVT-B
41.2M
56.0M
8.3G
7.4G
83.1%
83.1%
CVIT-S
26.7M
5.6G
81.0%
NesT-S
38.0M
10.4G
83.3%
PVT-M
44.2M
6.7G
81.2%
CrossFormer-B
52.0M
9.2G
83.4%
TNT-S
23.8M
5.2G
81.3%
Swin-T
29.0M
4.5G
81.3%
DeiT-B
86.0M
17.5G
81.8%
NesT-T
17.0M
5.8G
81.5%
DeiT-Bt
86.0M
55.4G
83.1%
CvT-13
20.0M
4.5G
81.6%
ViL-B
55.7M
13.4G
83.2%
ResT
30.2M
4.3G
81.6%
RegionViT-B
72.0M
13.3G
83.3%
CAT-S
37.0M
5.9G
81.8%
Twins-SVTHL
99.2M
14.8G
83.3%
ViL-S
24.6M
4.9G
81.8%
Swin-B
88.0M
15.4G
83.3%
RegionViT-S
30.6M
5.3G
82.5%
NesT-B
68.0M
17.9G
83.8%
CrossFormer-S 30.7M
4.9G
82.5%
Crossorrmer-T.
92.0M
16.1G
84.0%
Table 3: Object detection results on COCO 2017 val set with RetinaNets as detectors. Results for
Swin drawn from Twins Swin does report results RetinaNet. Results in blue fonts the second-placed ones. CrossFormers with f use different group sizes from classification models. (More details are put in the appendix (A.3)
Method
Backbone
#Params
FLOPs
Apé
AP
AP
AP
APt
AP}
16
ResNet-50
37.7M
234.0G
36.3
55.3
38.6
19.3
40.0
48.8
CAT-B
62.0M
337.0G
41.4
62.9
43.8
24.9
44.6
55.2
Swin-T
38.5M
245.0G
41.5
62.1
44.2
25.1
44.9
55.5
PVT-M
53.9M
63.1
25.0
44.9
57.6
ViI.M
50.8M
338.9G
64.0
45.4
46.1
57.2
RegionViT-B
83.4M
308.9G
43.3
65.2
46.4
29.2
46.4
$7.0
TransCNN-B
36.5M
43.4
64.2
46.5
27.0
47.4
56.7
CrossFormer-S
40.8M
282.0G
44.4 (+1.0)
65.8
47.4
28.2
48.4
$9.4
RetinaNct
1x schedule
CrossFormer-S
40.8M
272.1G
44.2 (+0.8)
65.7
47.2
28.0
48.0
59.1
ResNet101
56.7M
315.0G
38.5
57.8
41.2
21.4
42.6
51.1
PVT-L
71.1M
345.UG
42.6
63.7
45.4
25.8
46.0
58.4
Twins-SVT-B
67.0M
322.0G
44.4
66.7
48.1
28.5
48.9
60.6
Region ViT-B+
84.5M
328
66.4
29.6
47.6
59.0
Swin-B
98.4M
477.0G
44.7
65.9
Twins-SVT-L.
110.9M
455.0G
44.8
66.1
48.1
28.4
48.3
60.1
CrossFormer-B
62.1M
389.0G
(+.24)
67.8
49.5
30.1
49,9
61.8
CrossFormer-BB
62.1M
379.1G
46.1(+1.3)
67.7
49.0
29.5
49.9
61.5
We use MMDetection-hasee (Chen et al. 2019) RetinaNet (Lin et al.
2020 and Mask R-CNN (He
jct al. 2017) as the object detection and instance segmentation head, respectively. For both tasks, the backbones trained with batch size 16 on V100 GPUs, and an Adam optimizer with an initial learning rate of 1 10-4 is used. Following previous works, adopt 1x training schedule (ie.,, the models trained for 12 epochs) when taking RetinaNets as detectors, and images are resized to 800 pixels for the short side. While for Mask R-CNN, both 1x and 3x training schedules used. It is noted that multi-scale training (Carion et al.) 2020 is also employed when taking 3x training schedules.
initialized with the weights pre-trained on ImageNet. Then the whole models are
Results. The results RetinaNet and Mask R-CNN shown in Table Table respectively. As we can see, the second-placed architecture changes along with the experiment, that is, these architectures may perform well on one task but poorly on another task. In contrast, our CrossFormer outperforms all the others on both tasks (detection and segmentation) with both model sizes (small and base). Further, CrossFormer's s performance gain over the other architectures gets sharper when enlarging the model, indicating that CrossFormer enjoys greater potentials.
4.3 SEMANTIC SEGMENTATION
Experimental Settings. ADE20K (Zhou et al., (017) is used as the benchmark for semantic seg- mentation. It covers broad range of 150 semantic categories, including 20K images for training

---

Table Object detection and instance segmentation results on COCO val 2017 with Mask R-CNNS as detectors. AP and APm are box average precision and mask average precision, respectively.
Method
Backbone
#Params
FLOPs
Ap
AP
AP75
Apm
AP
AP
PVT-M
63.9M
42.0
64.4
45.6
39.0
61.6
42.0
Swin-T
47.8M
264.0G
42.2
64.6
46.2
61.6
42,0
Wwins-PCPVT-S
44.3M
245.0G
42.9
65.8
47.1
40.0
62.7
42.9
TransCNN-B
46.4M
44.0
66.4
48.5
40.2
63.3
43.2
Vil-M
60.1M
261.1G
43.3
65.9
47.0
39.7
62.8
42.0
RegionViTE ViT-B
92.2M
287.9G
43.5
66.7
47.4
40.1
63.4
43.0
Region
93.2M
307.1G
44.5
67.6
48.7
41.0
64.4
43,9
CrossFormer-S
50.2M
301,G
45.4 (+0.9)
68.0
49.7
41.4 (+0.4)
64.8
44.6
Mask R-CNN
Crossoormr-SS
50.2M
291.IG
45.0 (+0.5)
67.9
49.1
41.2 (+0.2)
64.6
44.3
1x schedule
CAT-B
71.MM
356.0G
41.8
45.2
38.7
62.3
41.4
PVT-L
81.M
364.0G
42.9
65.0
46.6
39.5
61.9
42.5
Twins-SVT-B
76.3M
340.0G
45.1
67.0
49.4
41.1
64.1
44.4
ViL-B
76.1M
365.IG
45.1
67.2
49.3
41.0
64.3
44.2
TWins-SVT-L.
119.7M
474.0G
45,2
67.5
49.4
41.2
64.5
44.5
Swin-S
69.1M
354.0G
44.8
66.6
48.9
40.9
63.4
44.2
Swin-B
107.2M
49
45.5
CrossFormer-B
71.5M
407.9G
(+1.7)
51.8
42.7
66.6
46.2
CrossForme--B
71.5M
398.1G
47.1 (+1.6)
69.9
$2.0
42.7 (+1.4)
66.5
PVT-M
63.9M
66.0
45.0
63.1
43.5
ViL-M
60.IM
261.GG
66.3
48.5
40.7
63.8
43.7
Swin-T
47.8M
264.0G
46.0
68.2
50.2
41.6
65.1
44.8
Shuffle-T
48.0M
268.0G
46.8
68.9
51.5
42.3
66.0
45.6
CrossForme--S*
50.2M
291.1G
48.7 (+1.)
70.7
53.7
43,9 (+16)
67.9
47.3
Mask R-CNN
3x schedule
PVT-L
81.0M
364.0G
44.5
66.0
48.3
40.7
63.4
43.7
ViL-B
76.1M
365.1G
45.7
67.2
49.9
41.3
64.4
44,5
Shuffle-S
69.0M
359.0G
48.4
70.1
53.5
43.3
67.3
46.7
Swin-S
69.1M
354.0G
48.5
70.2
53.5
43.3
67.3
46.6
CrossFormer-BB
71.5M
398.1G
49.8(+1.3)
71.6
54.9
(4.5(+
68.8
47.9
Table 5: Semantic segmentation results on the ADE20K validation set. "MS IOU" means testing with variable input size.
Semantic FPN (80K iteration))
UPerNet (160K iterations)
Backbone
#Params
FLOPs
IOU
Backbone
#Params
FLOPs
IOU
MS IOU
PVT-M
48.0M
219.0G
41.6
Swin-T
60.0M
945.0G
44.5
45.8
Twins-SVT-B
60.4M
261.0G
45.0
Shuffle-T
60.0M
949.0G
46.6
47.6
Swin-S
53,2M
274.0G
CrossFormer-S
62.3M
979.5G
47.6(+1.0)
48.4
CrossFormer-S
34.3M
220.7G
(6.0()
CrossFormer-S
62.3M
968.5G
47.4 (40.8)
CrossFormer-S
34.3M
209.8G
(6.1()
Swin-S
81.0M
1038.0G
47.6
49.5
PVT-L
65.1M
283.0G
42.1
Shuffle-S
81.0M
1044.0G
48.4
49.6
CAT-B
55.0M
276.0G
43.6
CrossFormer-B
83.6M
1089.7G
49.7 (+1.3)
50.6
CrossFoorme-BB
55.6M
331.0G
47.7 (+4.1)
CrossFormme-BB
83.6M
1078.8G
49.2 (+0.8)
50.1
CrossFormer--B'
55.6M
320.1G
48.0 (+4.4)
Swin-B
121.0M
1088.0G
48.1
49.7
Twins-SVT-L
70
397.0G
45.8
Shuffle-B
121.0M
1096.0G
49.0
CrossFormer-L
95.4M
497.0G
48.7 (+2.9)
CrossFormer-L
125.5M
1257.8G
(0.44)
51.4
CrossFormer-LL
95.4M
482.7G
49.1 (43.3)
Crossrormer-L
125.5M
1243.5G
(+1.5)
and 2K for validation. Similar to models for detection, we initialize the backbones with weights pre- trained on ImageNet, and MMSegmentation based (Contributors 2020 semantic FPN and UPer- Net (Xiao et al.) 2018) are taken as the segmentation head. For FPN (Kirilov et al.) 2019). we use AdamW optimizer with learning iterations with batch size 16. For UPerNet, an Adam optimizer with an initial learning rate of 6 x 10- and weight decay of 0.01 is used, and models are trained for 160K iterations.
and weight deacy of 1 10- Models are trained for 80K
Results. All results are shown in Table Similar to object detection, CrossFormer exhibits a greater performance gain over the others when enlarging the model. For example, CrossFormer-T achieves 1.4% absolutely higher on IOU than Twins-SVT-B, but CrossFormer-B achieves 3.1% absolutely higher on IOU than Twins-SVT-L Totally, CrossFormer shows a more significant advantage over the others on dense prediction tasks (e.g., detection and segmentation) than on classification, implying that cross-scale interactions the attention module are more important for dense prediction tasks than for classification.
8

---

Table Results the ImageNet validation set. The baseline model is CrossFormer-S (82.5%). We test with different kernel sizes of CELs.
CEL's Kernel Size Stage-2
#Params/FLOO
Acc.
APm
Stage-1
Stage=3
Stage-4
4 X 8 x 8
2 X
2
2
28.3M 4.5G
81.5%
39.7
2
2
28.3M 4,5G
81.9%
40.2
4,8 X
2 x 4
2 2x2,4x4
2,4
30.6M 4.8G
82.3%
x
4.8 x
32
32
x
4
2
2,4
x 2x2,4
30.7M 4.9G
82.5%
41.4
x
4.
x
16, 32
2,
4,8
2 2x2,4x
2
29.4M 5.0G
82.4%
Table 7: Experimental results of ablation studies.
(a) Ablation studies cross-scale embeddings (CHL) and long short distance attention (LSDA). tions. The base model is CrossFormer-S. Throughput is
(b) Comparisons between different position representa-
The base model is CrossFormer-S (82.5%).
tested V100 GPU.
PVT-like
Swin-like
LSDA
CEL
Acc.
Method
#Params/FLOO0
Thruughuut
Acc.
81.3%
APE
30,9342M/4. 9061G
686 imgs/sec
82.1%
81.9%
RPB
30.6159M1/
684 imgs/sec
82.5%
82.5%
DPB
N0.6573M//4909980
672 imgs/sco
82.5%
81.5%
DPB-residual
30.6573M/4.90966
672 imgs/sec
82.4%
4.4 ABLATION STUDIES
Cross-scale Embeddings vs. Single-scale Embeddings. We conduct the experiments by replac- ing cross-scale embedding layers with single-scale ones. As can in Table 6 when using single-scale embeddings, the x 8 kernel in Stage-/ brings 0.4% (81.9% vs. 81.5%) absolute im- provement compared with the x 4 kernel. It tells that overlapping receptive fields help improve the model's performance. Besides, all models with cross-scale embeddings perform better than those with single-scale embeddings. particular,
CrossFormer achieves 1% (82.5% 81.5%) abso- lute performance gain compared with using angle-sccal embeddings for all stages. For cross- cross-scale embeddings, we also try several different combinations of kernel sizes, and they all show similar performance (82.3% ~ 82.5%). In summary, cross-scale embeddings can bring a large performance gain, yet the model is relatively robust to different choices kernel size.
LSDA vs. Other Self-attentions. Two self-attention modules used in PVT and Swin are compared. Specifically, PVT sacrifices the small-scale features when computing the self-attention, while Swin restricts the self-attention in local region, giving the long-distance attention. As Table 7a| compared against the PVT-like and Swin-like self-attention mechanisms, our Cross- Former outperforms them least absolute 0.6% accuracy (82.5% vs. 81.9%). The results show that performing the self-attention in a long-short distance manner is most conducive to improving the model's performance.
observe
DPB vs. Other Position Representations. We compare the parameters, FLOPs, throughputs, and accuracies of the models among absolute position embedding (APE), relative position bias (RPB), and DPB. The results are shown in Table (7b DPB-residual means DPB with residual connections. Both DPB and RPB outperform APE for absolute 0.4% accuracy, which indicates that relative po- sition representations are more beneficial than absolute ones. Further, DPB achieves the same accu- racy (82.5%) as RPB with an ignorable extra cost; however, as we described Section 3.2.2 is more flexible than RPB and applies to variable image size group size. The results also show that residual connection DPB does not help improve or even degrades the model's performance.
CONCLUSIONS
We proposed a novel transformer-based vision architecture, namely CrossFormer. Its core ingredi- ents are Cross-scale Embedding Layer (CEL) and Long Short Distance Attention (LSDA), thereby yielding the cross-attentio module. We further proposed dynamic position bias, making the rel- ative position bias apply to any input size. Extensive experiments show that CrossFormer achieves superior performance over other state-of-the-an vision transformers several representative vision tasks. Particularly, CrossFormer demonstrated to gain great improvements on object detection and segmentation, which indicates that CEL and LSDA together essential for dense prediction tasks.

---

REFERENCES
Lei Jimmy Ba, Jamie Ryan Kiros, and Geoffre E. Hinton. abs/1607.06450, 2016.
Layer normalization.
CoRR,
Iz Beltagy, Matthew Peters, and Arman Cohan, Longformer: The don-mmetmnntt transformer. CoRR, abs/2004.05150, 2020.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Vosse Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Bemner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Neural Informa- fion Processing Systems, NeurlPS, 2020.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. Computer Vision Euro- pean Conference, ECCV, volume 12346, pp. 213-229, 2020.
Chun-Fu Chen, Quanfu Fan, and Rameswar Panda. Crossvit: Cross-attention multi-scale vision transformer for image classification. CoRR, Abs/2103.14899, 2021a.
Chun-Fu Chen, Rameswar Panda, and Quanfu Fan. Regionvit: Regional-to-lowe attention for vision 2021b.
transformers. CoRR,
Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng, Chenchen Zhu, Tianheng Cheng, Qijic Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang, Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin. MMDetection: Open mmlab detection toolbox and benchmark. arXiv preprint
2019.
Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. CoRR,
2019.
Xiangxiang Chu. Zhi Tian, Yuging Wang, Bo Zhang, Haibing Ren, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Twins: Revisiting spatial attention design in vision transformers. CoRR, abs/2104. 13840, 2021.
MMSSegmenattion Contributors. MMSegmentation: Openmmlab semantic segmentation toolbox and benchmark, 2020.
Ekin Dogus Cubuk, Barret Zoph, Jon Shlens, and Quoe Le. Randaugment: Practical automated data augmentation with a reduced search space. Neural Information Processing Systems, NeurIPS, 2020.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Conference of the North American Chapter of the Association for Computational Linguistics, NAACL, pp. 4171-4186, 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, ICLR, 2021.
Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, and Yunhe Wang. Transformer in transformer. CoRR, 0bs/2103.00012, 2021.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross B. Girshick. Mask R-CNN, In International Conference Computer Vision, ICCV, pp. 2980-290099 2980, 2017.
Gao Huang. Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q. Weinberger. Deep networks with stochastic depth. In Bastian Leibe, Jiri Matas, Nicu Sebe. and Max Welling (eds.), European Conference Computer Vision, ECCV, volume 9908, pp. 646-661, 2016.
10

---

Zilong Huang. Youcheng Ben, Guozhong Luo, Pei Cheng. Gang Yu, and Bin Fu. Shuffle trans- former: Rethinking spatial shuffle for vision transformer CoRR, abs/2 100o/2006.,3650, 2021.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, ICLR, 2015.
Alexander Kirillov, Ross B. Girshick, Kaiming He, and Piotr Dollár. Panoptic feature pyramid networks. In Conference on Computer Vision and Pattern Recognition, CVPR, pp. 6399-6408, 2019.
Hezheng Lin, Xing Cheng, Xiangyu Wu, Fan Yang, Dong Shen, Zhongyuan Wang, Qing Song, and Wei Yuan. CAT: cross attention in vision transformer. CoRR, abs/2200.005886 2021.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. In European Conference on Computer Vision, ECCV, volume 8693, pp. 740-755, 2014.
Tsung-Yi Yi Lin, Priya Goyal, Ross B. Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. Transactions on Puttern Analysis and Machine Intelligence, PAMI, 42(2))318- 327, 2020.
Yun Liu, Guolei Sun, Yu Qiu, Le Zhang, Ajad Chhatkuli, and Luc Van Gool. Transformer in convolutional neural networks. CoRR, abs/2106.03180, 2021a.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Bain- ing Guo. Swin transformer: Hierarchical vision transformer using shifted windows. CoRR, Abs/2103.14030, 2021b.
Vinod Nair and Geoffrey E. Hinton. Rectified linear units improve restricted boltzmann machines. In Johannes Fürnkranz and Thorsten Joachims (eds.), International Conference on Machine Learn- ing, ICML, pp. 807-8 814, 2010.
Zizheng Pan, Bohan Zhuang, Jing Liu, Haoyu He, and Jianfci Cai. Scalable visual transformers with hierarchical pooling. CORR, abs/2103. 1001/220..0619, 2021.
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Fei-Fei Li. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, IJCV, 2015.
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position represen- tations. In Conference of the North American Chapter of the Association for Computational Linguistics; Human Language Technologies, NAACL, pp. 464-468, 2018.
Aravind Srinivas, Tsung-Yi Yi Lin, Niki Parmar, Jonathon Shlens, Pieter Abbeel, and Ashish Vaswani. Bottleneck transformers for visual recognition. In Conference Computer Vision and Pattern Recognition, CVPR, 2021.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott E. Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR, pp. 1-9, 2015.
Mingxing Tan and Quoe Le. Mixconv: Mixed depthwise convolutional kernels. In British Ma-
chine Vision Conference, BMVC, pp. 74, 2019.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning, ICML, volume 139, pp. 10347-10357, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In Neural Information Processing Systems, NeurIPS, pp. 5998 6608, 2017.
11

---

Wenhai Wang. Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. CoRR, abs/2102.12122, 2021.
Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. Cvt: Introducing convolutions to vision transformers. CoRR, abs/2103.15808 2021.
Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In European Conference on Computer Vision, ECCV, volume 11209, pp- 432-448, 2018.
Weijian Xu, Yifan Xu, Tyler A. Chang, and Zhuowen Tu. Co-scale conv-attentional image trans- formers. CoRR, abs/2104.06399, 2021a.
Yufei Xu, Qiming Zhang, Jing Zhang, and Dacheng Tao. Vitae: Vision transformer advanced by exploring intrinsic inductive bias. CoRR, abs/2200.00334,, 2021b.
Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Francis Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. CoRR, abs/2101. 11966, 2021.
Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Seong Joon Oh, Youngjoon Yoo, and Junsuk Choe. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Interna- tional Conference on Computer Vision, ICCV, pp. 6022-6031, 2019.
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santi- ago Ontañón, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang. and Amr Ahmed, Big bird: Transformers for longer sequences. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances Neural Information Processing Systems, NewrIPS, 2020.
Hongyi Zhang, Moustapha Cissé, Yann N. Dauphin, and David Lopez-Paz. mixup: cal risk minimization. In International Conference on Learning Representations, ICLR, 2018a.
Beyond empiri-
Pengchuan Zhang, Xiyang Dai, Jianwei Yang, Bin Xiao, Lu Yuan, Lei Zhang, and Jianfeng Gao. Multi-scale vision longformer: A new vision transformer for high-resolution image encoding. CoRR,
2021a.
Qing-Long Zhang and Yubin Yang. Rest: An efficient transformer for visual recognition. CoRR, abs/2105.13677 2021.
Ting Zhang, Guo-Jun Qi, Bin Xiao, and Jingdong Wang. Interleaved group convolutions for deep neural networks. CoRR, abs/1707.02725, 2017.
Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR, pp. 6848-6888668 68566 2018b.
Zizhao Zhang, Han Zhang, Long Zhao, Ting Chen, and Tomas Pfister. Aggregating nested trans- formers. CoRR, Abs/2105.1272., 2021
Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data aug- mentation. In Association for the Advancement of Artificial Intelligence, AAAI, pp. 13001-13008, 2020.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ADE20K dataset. In Conference on Computer Vision and Pattern Recognition, CVPR, pp. 2017.
12

---

A CROSSFORMER
A.1 PSEUDO CODE OF LSDA
The pseudo code for LSDA is shown in Algorithm 1 attention module, both SDA and LDA are implemented with only ten lines of code, and only reshape and permule operations are used.
As we can see, based on the vanilla self-
Algorithm LSDA code (PyTorch-like)
height, N: width, G: group size of SDA/IDA input tenser (8, N,
cless LSDA
def forward type):
44 group is type "SDA":
embeddings
x reshaspp // C, G, # // C, c, DJ D1.ppmmute10, 10, 2, 1, 3, 4)
elit type "LDA": // G, 6, 8 G, DJ D1.permute 3, 0, 2, 4)
x x.reshaspe x.reshape (H /7 (G *. 2), G .. 2, D)
# the vanilla self atteation module Atzention(x)
# un group the embeddings
k.rreshagpeli !i // 6, // G, G, G, DJ
15 type "SDA":
x.permute (0, 2, 1, 3, .reshappe (H, 9, allf type == "LNA":
x.permute (2, 0, 3, 1, 4) rreshaspe 8, return
A.2 DYNAMIC POSITION BIAS (DPB)
Figure gives an example of computing (Arty, Ayú with = 5 in the DPB module. For group of size G G, it is easy deduce that:
0 <<
1- 1- Azo -
(3)
G Aμy S G-
Thus, motivated by the relative position bias, we construct a matrix
B E (2G (222-11xx226 ()) where
AX = -1
Bi, = D G G 0 2G 1. (4)
Ayij = -
X
The complexity of computing B O(G)). Then, the bias matrix B in DPB can be drawn from B, i.e.,
Figure 4: An example of com- puting (Ariy. Aytj).
Bij B BAry
(5)
When the image/group size (i.e., G) is fixed, both B and B will be also unchanged in the test phase. Therefore, we only need to compute B and B once, and DPB equivalent to relative position bias in this case.
A.3 VARIANTS OF CROSSFORMER FOR DETECTION AND SEGMENTATION
We test two different backbones for dense prediction tasks. The variants CrossFormer for dense prediction (object detection, instance segmentation, and semantic segmentation) are in Table The architectures are the same as those for image classification except that different G and the first two stages used. Notably, group size (i.e., G and I) does not affect the shape of weight tensors, so backbones pre-trained on ImageNet can be fine-tuned directly on other tasks even they use different G and T.
13

---

Table 8: CrossFormer-basee backbones for object detection and semantic/instance segmentation. The example input size is 1280 x 800. D and H mean embedding dimension and the number of heads in the multi-heas head self-attention module, respectively. G and / group size and interval for SDA and LDA, respectively.
Outpot Size
Layer Name
CrossForner-T
CrossFormer-S
CrossFormer-B
CrossForme-LL
Cross Embed.
Kernel size:
4,8 16 16, x 32. Stride=t
64
[1 967
125
Stage-1
3200 200
SDALDA MI.P
Hi
H
Hj
11
x1
x
Ci 14 x
Gr 14
x
16
16
16,
4
Cross Embed.
Kernel 2,
Stride=2
Dg 128
Do =
1921
D2 192
D2 256
Stage
160 100
SDA/LDA MLP
H;
H.
G2
G2
X
G2 14
X
G2 14
X
12
12
Cross Embed.
Kernel size: 2 x 2, 2.41 Strid==2
Dg
Dg
Dj
Stage-3
80 50
SDA/LDA MLP
H1 8
Hg 12
Hg 12
Hy 16
==
Is
1
49 =
Kernel 2, 4, Stride22
D; 5121
D; 7681
D; 768
D4 1021
Stage-4
SDA/LDA MI.P
Hg
Hg
32
64 -
x
G4 7
X2
C4 -7
x
x2
4-1
4-
1
4
Table 9: Object detection results on COCO val 2017. "Memory" means the allocated memory per GPU reported by (G, I) from classification models.
indicates that models use different
Method
Backhone
G1
T
G₂
T2
Memory
#Params
FLOPS
Apb
AP
AP
CrossFormer-S
7
4
14.7G
40.8M
282.0G
44.4
65.8
47.4
RetinaNet
CrossFormer-Stt
14.
16
14
8
11.9
40.8M
272.1G
44.2
65.7
47.2
1x schedule
CrossFormer-B
7
4
22.8G
62.1M
389.0G
46.2
67.8
49.5
CrossFormer-Bi
16
14
8
20.2G
62.
379.0G
46.1
67.7
49.0
CrossHormmr-S CrossFormer-S'
7
4
15.5G
50.2M
301.0G
45.4
68.0
49.7
Mask-RCNN
14
16
14
8
12.7G
50.2M
291.1G
45.0
67.9
49.1
1x schedule
CrossFormer-B B
7
4
23.8G
71.5M
407.9G
47.2
69.9
51.8
CrossFormer-Bi
14
16
14
8
21.0G
71.5M
398.1G
47.1
69.9
$2.0
Table 10: Semantic segmentation results on ADE20K validation set with semantic FPN or UPerNet
as heads.
Semantic FPN (8UK itcrations)
UPerNet (160K itcrations)
Backbone
G
I
G2
Iz
Memory
4Params
FLOPs
IOU
Memory
#Params
FLOP
IOU
MS IOU
CrossFomme-S
7
8
7
4
20.9G
34.3M
220.7G
46.0
62.3M
979.5G
47.6
48.4
CrossFomee-SSt
14
8
20.9G
34,3M
209.8G
46.4
14.6G
62.3M
968.5G
47.4
48.2
CrossFomer-B
7
4
14.6G
55.6M
331.0G
47.7
15.8G
83.6M
1089.7G
49.7
50.6
Cr
14.
16
14
8
14.6G
55.6M
320.1G
48.0
15.8G
83.6M
1078.8G
49.2
50.1
Crosr-o
7
8
4
25.3G
95. 25.4M
497.0G
48.7
18.1G
125. 5M
1257,8G
50.4
51.4
Frosr-
14
25.36
18.1G
125.5M
51.4
B EXPERIMENTS
B. OBJECT DETECTION
Table 9 provides more results object detection with RetinaNet and Mask-RCNN as detection heads. As we can see, smaller (G, achieves a higher AP than larger one, but the performance gain is marginal. Considering that larger (G, I) can save memory cost, think (G1 14, I1 = 16, G2 = 14 = 8), which accords with configurations Table g achieves a better trade-off between the performance and cost.
14

---

Table 11: Classification results on ImageNet dataset after plugging CEL into other vision transform- ers.
Modcls
#Params
FLOPs
Accuracy
Swin
29.0M
4.5G
81.3%
Swin + CEL
29.2M
4.8G
81.9% (+0.6%)
NesT
38.4M
10.4G
83.3%
NesT CEL
38.5M
10.6G
83.8% (+0.5%)
B.2 SEMANTIC SEGMENTATION
Similar to object detection, we test two different configurations of (G, I) for semantic segmenta- tion's backbones. The results are shown in Table I0 As we can see, the memory costs of the two configurations are almost the same, which different from experiments on object detection. Further, when taking semantic FPN as the detection head, CrossFormers* show advantages over CrossForm- ers on both IOU (e.g., g., 46.4 vs. 46.0) and FLOPs (e.g., 209.8G vs. 22). When taking UPerNet the segmentation head, a smaller (G, I) achieves higher performance like object detection.
B.3 CLASSIFICATION WITH CEL
Table TT shows results ImageNet dataset after plugging CEL into other vision transformers. As we can see, CEL brings about 0.5% performance gain for both Swin Transformer and NesT, which further shows the effectiveness of CEL.
C DISCUSSION
C.1 CROSS-SCALE FEATURE EXTRACTION
Since GoogLeNet (Szegedy et al., 2013) and MixConv (Tan & Le 2019) use multi-scale convolutional kernels at every layer of models, the largest kernel size can only be 7 x 7, and larger kernel size will induce unaccpetable computational budget. In contrast, CEI only introduces cross-scale features at the start of each stage (at most 4 layers), kernels with very large size (c.g., 32 32) can also be used for diversified
CEL vs. GongLeNet/MixCony
objects.
CEL vs. CoaT/CViT. Some vision transformers also take advantage cross-scale features. For example, Coa'T (Xu et al. 2021a) only uses multi-scale features at later layers by mixing features from front layers, while CHI introduces multi-scale features the start of each stage, thus tokens embeddings all layers (not only those at later layers) could be seen cross-scale. Besides, CViT (Chen et al., 2021a) keeps embeddings of different scales different branches and introduces cross-scale interaction through self-attention between branches. This method well preserve the information of each scale, but multiple branches take more cost than single-branch model. As result, for controllable cost, CVTT can only introduce two different scales (4 x 4 and 8 x 8). In contrast, CEL only appears at the start of each stage (at most layers). So, CEL can introduce more different scales (e.g., x 4, 8, 16 16, 32 x 32) than CViT with ignorable extra cost (< 0.1M parameters and < 0.3G FLOPs in most cases).
C.2 SPARSE SELF-ATTENTION AND GROUP CONVOLUTION
LSDA vs. ShuffleNet/IGCNet. LSDA, ShuffleNet (Zhang et al. 2018b) and IGCNets (Zhang al., 2017) devote reducing the computational cost by approximating the original operations (self-attention and convolution) in lower cost. Nevertheless, they are essentially different: (1) Different target and dimension: ShuffleNe/IGCNe devotes to reducing the budget caused by the large channels of convolution. So, both ShuffleNet and IGCNet do muuffle-and-permuaalie channel dimension. While for the self-attention module, the budget mainly attributes to the large spatial size of feature maps (i.e., the large number of embeddings). Thus, LSDA is mainly operating along the spatial dimension. (2) Different motivation: During the process of designing LSDA, we first consider the locality of images and propose extract short-distance attention through SDA.
along
15

---

Meanwhile, to preserve the ability of modeling oong-ditaance dependency, LDA is further proposed to fill this gap. In contrast, ShufftNNerICGNets do not need to consider these aspects because all channels equivalent, and locality does not exist along channel dimmension. Though they also perform convolution among adjacent channels first, is for the sake of fast memory access.
LSDA vs. Other Sparse Self-attention. Previous works proposed some sparse self-attention, such as Big Bird (Zaheer et al., 2000)), Longformer (Beltagy et al. 2020). Sparse Transformer (Child al., 2 etc. Wherein, generative model, Sparse Transformer proposed strided attention in the one-dimensional case, and every pixel can see the pixel generated ahead of itself. Instead, LSDA works two-dimensional case. Further, L.SDA has co-design with CEL., As we described in Section 3.2.1 and Figure 3 in long-distance attention (LDA), the small-scale patches of two embeddings are non-adjacent, so it is difficult to judge their relationship without the help of the context. In other words, it will be hard to build the dependency between these two embeddings they are only constructed by small-scale patches (i.e., single-scale feature). On the contrary, adjacent large-scale patches provide sufficient context link these two embeddings, which makes long-distance cross-scale attention easier and more meaningful.
16


Reviews:
[{"review_id": "nDb-DDcMZuE", "reviewer": "Reviewer_K7wS", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n- How to build the interaction among different scale features is an interesting problem in computer vision. The proposed method provides an easy way to build the multi-scale feature interaction for transformer-based architecture. \n- Comprehensive experiments are conducted, and the results showed the effectiveness of the proposed architecture.\n- The idea of the paper is clear and easy to follow. \n\nCons:\n- The short-distance attention is essentially the local window attention in the Swin-transformer, which is not new.\n- The results in Table 6(b) showed that the proposed dynamic positional embedding has a similar performance with relative positional embedding. It's hard to say that the proposed dynamic positional embedding (DPE) is a contribution. The original relative positional embedding is also learned and dynamic and is also the function of relative coordinates. So, the problem becomes how to define the function.  However, the authors did not show how and why the function in the proposed DPE is superior to original relative positional embedding. \n- The way to generate the multi-scale features is not novel. In the previous works [3, 4], the mixed convolution, which is especially the same as the proposed Cross-scale Embedding Layer. \n- The long-distance attention is like the method of interleaved group or shuffle, which are proposed in the paper of [1,2] over the spatial features. I think the authors should carefully discuss the relation between them. \n\n[1] ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices\n\n[2] Interleaved Group Convolutions\n\n[3] Going Deeper with Convolutions\n\n[4] MixConv: Mixed Depthwise Convolutional Kernels\n", "summary_of_the_paper": "The paper proposed a novel multi-stage vision transformer architecture, named CrossFormer, which aimed at building interaction among features from different scales. The proposed architecture follows the layout of Swin transformer but replaces the shifted windows transformer block with the proposed short-distance attention and long-distance attention. A cross-scale module is proposed to replace the patch embedding or patch merging layer in the Swin transformer. Comprehensive experiments are conducted on image classification, objection detection, and segmentation tasks. Results shows the effectiveness of the proposed architecture. ", "summary_of_the_review": "The idea of the paper is clear. Comprehensive experiments are performed, and superior results over several benchmarks demonstrated the effectiveness of the proposed architecture. My main concern is that the idea of the proposed components is not new in the CNN-based works. The overall novelty of the paper is limited. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "VtQLoIn7OB8", "reviewer": "Reviewer_BmmR", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\n* Overall simplicity and elegance of the introduced components. I think the paper strikes a good balance between introduced complexity and achieved performance.\n* Results are very strong on a number of vision tasks, and in particular on dense prediction tasks\n* Well presented and easy to understand.\n\n## Weaknesses\n* Context to prior work:\n  * In principle a fully connected layer over a large patch can learn to represent patches of multiple smaller scales as well. The Cross-scale embedding layer in this paper is essentially just a more constraint version of it. Taking 32x32 patches (4x4 stride) and projecting those to D is strictly more expressive than the procedure described in the paper. This is not to say that it is not useful as it clearly provides inductive bias, but it is less expressive which is the opposite of what the paper says. Hence I don't agree with the blank statement: \"They fail to build the interactions among features of different scales, whereas such an ability is very vital for a lot of vision tasks.\" A more subtle discussion on inductive biases would be more appropriate and I would ask the authors to change that. Personal side-note: A more interesting take on scale (IMO) would be to introduce a mechanism that enforces scale invariance (at least in some range).\n  * Long-short distance attention is basically a version of sparse attention (strided attention) that was introduced for instance in [1]. There are also other papers, e.g., [2, 3], that introduced very similar sparse attention mechanisms, however, mostly with a focus on text but they easily generalize to images. So the novelty of the presented mechanism here is limited (at best) and the absence of proper references to similar work is a bit concerning. A more thorough study and comparison with prior work should be added.\n* Novelty (Framing): The individual components are not very novel (see points before), and it should be made clear in the paper. It's, however, not a big issue for me because finding the right combination of these components in order to strike the right balance between complexity and performance is more important and as such also novel.\n\n## Questions\n* Ablations on a dense prediction task to see where the improvements really come from. The authors hypothesize they are due to cross scale patches. These ablations are important to also verify that it isn't the overall training setup or other subtle differences in the architecture that give the advantage.\n* A strange choice is that smaller patches project to larger dimensions, leading to larger compression. Was this a deliberate choice and ablated?\n* Setting G and I to different values and not making G divisible by I is a bit odd, I wonder why such values were chosen.\n\n* [1] https://arxiv.org/abs/1904.10509\n* [2] https://arxiv.org/abs/2007.14062\n* [3] https://arxiv.org/abs/2004.05150\n* [4] https://arxiv.org/abs/2103.15358", "summary_of_the_paper": "The paper describes a pyramidal vision transformer that introduces cross-scale patch embeddings, dynamic relative position biases and \"Long-Short Attention\". The model shows superior performance on imagenet classification and especially on dense prediction tasks. Ablations on classification are provided hinting that the multi-scale embeddings are the most important factor contributing to the overall superior results to similar, previous architectures.", "summary_of_the_review": "The paper is overall well written and easy to understand. I think there are some strong contributions that are a bit oversold (in my view). However, I am leaning towards accepting the paper given that some of the concerns and questions are properly addressed. I am willing to raise my score in case those concerns are addressed.\n\nUPDATE after rebuttal. I am willing to raise my score after reading the rebuttal and the updates to the paper. I think the paper is a good and clean contribution as it is conceptually simpler (in my view) than most of the previous works while showing stronger performance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "F7n5ZGSxIJ8", "reviewer": "Reviewer_euuQ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n1. The idea of CEL is sound and its design is neat. As this layer can be plugged in other structures easily, it can also boost other models.\n2. Thorough experiments verify the effectiveness of the proposed techniques. And CrossFormer achieve good performance on several vision taks.\n\nCons:\n1. Some comparisons and discussions with related works are missed. For example, CrossViT[1] also combine features from patches of different sizes and Shuffle Transformer[2] proposes a spatial shuffle mechanism for transformer layer, which is similar with the long attention module in this paper.\n\n[1]  Chen, C. F., Fan, Q., & Panda, R. (2021). Crossvit: Cross-attention multi-scale vision transformer for image classification. arXiv preprint arXiv:2103.14899.\n\n[2]  Huang, Z., Ben, Y., Luo, G., Cheng, P., Yu, G., & Fu, B. (2021). Shuffle Transformer: Rethinking Spatial Shuffle for Vision Transformer. arXiv preprint arXiv:2106.03650\n", "summary_of_the_paper": "This paper proposes two new techniques: Cross-scale Embedding Layer (CEL) and Long Short Distance Attention (LSDA). Cross-scale Embedding Layer samples patches with convolution kernels of different sizes and concatenates them as one embedding, so that patch embeddings contain information of multiple scales. Long Short Distance Attention includes a short attention module and a long attention module. The short attention is computed in a local region. The long one covers the whole image but is computed sparsely. These two new designs enable the network to learn cross-scale features. Also, a dynamic position bias is designed to make the network suitable for inputs of different resolutions.", "summary_of_the_review": "Overall, the ideas of CEL and LSDA are sound and solid experiments support most claims in the paper. However, some designs are not new and some related works are missed. I would rate this as a borderline paper (marginally above the acceptance threshold).", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "AgSemm0NAix", "reviewer": "Reviewer_GtDH", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n1. The Cross-scale Embedding Layer (CEL) extends the single-scale feature maps used in prior works of ViTs (e.g. ViT, DeiT, PVT, Swin)  to multi-scale feature maps. The mixture of multi-scale features enable a richer context, especially for dense prediction tasks like object detection, instance segmentation and semantic segmentation. In addition, the CEL employs a series of kernel with the same stride and decreasing number of output dimensions, which is able to control the total computation budgets.\n\n2. In the self-attention mechanism, the proposed CrossFormer introduces a long short distance attention (LSDA), which includes a long distance attention (LDA) and a short distance attention (SDA). The SDA is very similar to the shifted window attention in Swin, which splits feature maps into windows (denoted by groups in this paper) and performs self-attention in each window (group). Different from Swin, the LDA module has a dilated window where each embedding in the window has a large interval to its adjacent embeddings. The proposed LDA can model long range interactions while still keeping a linear complexity in computation.\n\n3. In the experiment section, the proposed approach achieves state-of-the-art performance on all tasks. Specifically, on dense prediction task like detection and segmentation, the proposed CrossFormer obtains decent improvement over prior works with similar computation budget.\n\nCons / Questions:\n1. The proposed CrossFormer focuses on better modeling of cross-scale information for ViTs: CEL enables multi-scale features and LSDA enables long distance interaction (which implies interaction among large-scale features). However, there are already some works on the cross-scale information such as [1]. Although the prior works may not use the identical way, it is still needed to have some discussion in the manuscript to compare the alternatives to model cross-scale information and the benefits from the proposed CrossFormer.\n\n    [1] Co-Scale Conv-Attentional Image Transformers, Xu et al, ICCV 2021\n\n2. The proposed LSDA may have some limitations in its design: Table 1 shows that each stage in CrossFormer has the same choice of (G, I) and Appendix A.1 shows that G*I is the size of the feature map. Thus, when the size is large (e.g. stage 1, size=56), since G is alway 7, interval I will be large (e.g. stage 1, I=7), which causes the elements in the window (group) are very scattered, and the behavior of the LDA could be very different from SDA in the same stage. I wonder if such large gap between LDA and SDA could lead to some potential issues. In addition, perhaps ablation study on LDA with lower interval is needed, since it may work better together with SDA.\n\n3. This paper also proposes a dynamic position bias (DPB), which aims to replace the relative position bias (RPB) used in Swin and early works in NLP (Shaw et al). However, Table 6(b) does not show that the proposed DPB is able to outperform RPB, which limits the use of DPB. In addition, it is claimed that DPB has the capability to support different window (group) size. It would be better to have a case that demonstrates this benefit.\n\n4. In the empirical experiments, on the image classification task, the proposed CrossFormer obtains similar performance as the best of prior works. Thus, I wonder if it is possible to apply the proposed CEL and LSDA to prior works (e.g. RegionViT or NesT) to further improve the performance?\n", "summary_of_the_paper": "This paper proposes a novel vision transformer architecture called CrossFormer which focuses on the cross-scale ability in the attention module. Specifically, the proposed CrossFormer introduce Cross-scale Embedding Layer (CEL) and Long Short Distance Attention (LSDA) to model cross-scale interactions. CEL has the capability to merge multi-scale features with various kernel sizes, while LSDA enables the self-attention to capture feature interactions of both short distance and long distance. Experiments demonstrate that the proposed CrossFormer achieves performance improvement on image classification, object detection, instance segmentation and semantic segmentation tasks.", "summary_of_the_review": "This paper proposed CrossFormer that aims to model cross-scale information through Cross-scale Embedding Layer (CEL) and Long Short Distance Attention (LSDA). The design of both modules is reasonable, and it demonstrates performance improvement on dense prediction tasks. However, due the potential limits of LSDA and DPB and missing comparison with some prior works, I would like to give a \"marginally above the acceptance threshold\" rating.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes several modifications to vision transformers: multiscale features, a variant of factorized attention, and "dynamic position bias". The proposed architecture with these modifications achieves strong results on classification, detection, and segmentation.

After considering the authors' responses, all reviewers are positive about the paper (reviewer K7wS mentioned upgrading to weak accept, but apparently forgot to do so). Main pros include clean architecture and strong empirical results. The main con is the somewhat limited novelty.

Overall, I recommend acceptance. While each of the proposed modifications might not be that unique, they are reasonably new in the context of transformers and their combination makes for a clean architecture that performs very well in practice.
Decision: Accept (Poster)