CROSSFORMER: A VERSATILE VISION TRANSFORMER HINGING ON CROSS-SCALE ATTENTION
Year: 2021

Abstract:
Transformers have made great progress in dealing with computer vision tasks. However, existing vision transformers do not yet possess the ability of building the interactions among features of different scales, which is perceptually important to visual inputs. The reasons are two-fold: (1) Input embeddings of each layer are equal-scale, so no cross-scale feature can be extracted; (2) to lower the computational cost, some vision transformers merge adjacent embeddings inside the selfattention module, thus sacrificing small-scale (fine-grained) features of the embeddings and also disabling the cross-scale interactions. To this end, we propose Cross-scale Embedding Layer (CEL) and Long Short Distance Attention (LSDA). On the one hand, CEL blends each embedding with multiple patches of different scales, providing the self-attention module itself with cross-scale features. On the other hand, LSDA splits the self-attention module into a short-distance one and a long-distance counterpart, which not only reduces the computational burden but also keeps both small-scale and large-scale features in the embeddings. Through the above two designs, we achieve cross-scale attention. Besides, we put forward a dynamic position bias for vision transformers to make the popular relative position bias apply to variable-sized images. Hinging on the cross-scale attention module, we construct a versatile vision architecture, dubbed CrossFormer, which accommodates variable-sized inputs. Extensive experiments show that CrossFormer outperforms the other vision transformers on image classification, object detection, instance segmentation, and semantic segmentation tasks.1

heading: 1 INTRODUCTION
text: It turns out that transformer (Vaswani et al., 2017; Devlin et al., 2019; Brown et al., 2020) has achieved great success in the field of natural language processing (NLP). Benefitting from its selfattention module, transformer is born with the key ability to build long-distance dependencies. Since long-distance dependencies are also needed by a number of vision tasks (Zhang & Yang, 2021; Chu et al., 2021), a surge of research work (Dosovitskiy et al., 2021; Touvron et al., 2021; Wang et al., 2021) has been conducted to explore various transformer-based vision architectures.
A transformer requires a sequence of embeddings2(e.g., word embeddings) as input. To adapt this requirement to typical vision tasks, most existing vision transformers (Dosovitskiy et al., 2021; Touvron et al., 2021; Wang et al., 2021; Liu et al., 2021b) produce embeddings by splitting an input image into equal-sized patches. For example, a 224 × 224 image can be split into 56 × 56 patches of size 4 × 4, and these patches are projected through a linear layer to yield an embedding sequence. Inside a certain transformer, self-attention is engaged to build the interactions between any two embeddings. Thus, the computational or memory cost of the self-attention module is O(N2), where N is the length of an embedding sequence. Such a cost is too big for a visual input because its embedding sequence is much longer than that of NLP. Therefore, the recently proposed vision transformers (Wang et al., 2021; Liu et al., 2021b; Lin et al., 2021) develop multiple substitutes to approximate the vanilla self-attention module with a lower cost.
Though the aforementioned vision transformers have made some progress, they do not explicitly utilize features of different scales, whereas multi-scale features are very vital for a lot of vision tasks. For example, an image often contains many objects of different scales, and to fully understand
1The code has been submitted along with our supplementary material. 2In this paper, we also use “embeddings” to represent the input of each layer.
the image, building the interactions among those objects is helpful. Besides, some particular tasks such as instance segmentation need the interactions between large-scale (coarse-grained) features and small-scale (fine-grained) ones. Existing vision transformers fail to deal with the above cases due to two reasons: (1) The embeddings are generated from equal-sized patches. Though these patches theoretically have a chance to extract any scale features if only the receptive field is large enough, it is difficult to promise that they can learn appropriate multi-scale features automatically in practice. (2) Inside the self-attention module, adjacent embeddings are often grouped together and merged (Wang et al., 2021). Since the number of groups is smaller than that of embeddings, such behavior can reduce the computational budget of the self-attention. In this case, however, even if embeddings have both small-scale and large-scale features, merging operations will lose the small-scale features of each individual embedding, thereby disabling the cross-scale attention.
To enable the building of cross-scale interactions, we co-design a novel embedding layer and selfattention module as follows. 1) Cross-scale Embedding Layer (CEL) – Following Wang et al. (2021), we also employ a pyramid structure for our transformer, which naturally splits the vision transformer model into multiple stages. CEL appears at the start of each stage, which receives last stage’s output (or an input image) as input and samples patches with multiple kernels of different scales (e.g., 4×4 or 8 × 8). Then, each embedding is constructed by projecting and concatenating these patches. Through this way, we enforce some dimensions (e.g., dimensions from 4 × 4 patches) to focus on small-scale features only, while others (e.g., those from 8× 8 patches) have a chance to learn largescale features, leading to an embedding with explicitly cross-scale features. 2) Long Short Distance Attention (LSDA) – We propose a substitute of the vanilla self-attention, but to preserve small-scale features, the embeddings will not be merged. In contrast, we split the self-attention module into Short Distance Attention (SDA) and Long Distance Attention (LDA). SDA builds the dependencies among neighboring embeddings, while LDA takes charge of the dependencies among embeddings far away from each other. The proposed LSDA can also reduce the cost of the self-attention module like previous studies (Wang et al., 2021), but different from them, LSDA does not undermine either small-scale or large-scale features. As a result, attention with cross-scale interactions is enabled.
Besides, following prior work (Shaw et al., 2018; Liu et al., 2021b), we employ a relative position bias for embeddings’ position representations. The Relative Position Bias (RPB) only supports fixed image/group size3. However, image size for many vision tasks such as object detection is variable, so does group size for many architectures, including ours. To make the RPB more flexible, we further introduce a trainable module called Dynamic Position Bias (DPB), which receives two embeddings’ relative distance as input and outputs their position bias. The DPB module is optimized end-to-end in the training phase, inducing an ignorable cost but making RPB apply to variable image/group size.
All our proposed modules can be implemented with about ten lines of code. Based on them, we construct four versatile vision transformers of different sizes, dubbed CrossFormers. Other than image classification, the proposed CrossFormer can handle a variety of tasks with variable-sized inputs such as object detection. Experiments on four representative vision tasks (i.e., image classification, object detection, instance segmentation, and semantic segmentation) demonstrate that CrossFormer outperforms the other state-of-the-art vision transformers on all the tasks. Remarkably, the performance gains brought by CrossFormer are substantially significant on dense prediction tasks, e.g., object detection and instance/semantic segmentation.
It is worth highlighting our contributions as follows: • We propose cross-scale embedding layer (CEL) and long short distance attention (LSDA), which
together compensate for existing transformers’ incapability of building cross-scale attention. • The dynamic position bias module (DPB) is further proposed to make the relative position bias
more flexible, i.e., accommodating variable image size or group size. • Multiple CrossFormers with different sizes are constructed, and we corroborate their effective-
ness through sufficient experiments on four representative vision tasks.

heading: 2 BACKGROUND
text: Vision Transformers. Motivated by the transformers developed for NLP, researchers design specific visual transformers for vision tasks to take full advantage of their powerful attention mechanism. In
3Some vision transformers split input embeddings into several groups. Group size means the number of embeddings in a group.
particular, ViT and DeiT transfer the original transformer Vaswani et al. (2017) to vision tasks (Touvron et al., 2021; Dosovitskiy et al., 2021), achieving impressive performance. Later, PVT (Wang et al., 2021), HVT (Pan et al., 2021), Swin (Liu et al., 2021b), and ViTAE (Xu et al., 2021b) introduce a pyramid structure into the visual transformers, greatly decreasing the number of patches in the later layers of a respective model. They also extend the visual transformers to other vision tasks like object detection and segmentation (Wang et al., 2021; Liu et al., 2021b).
Cross-scale Feature Extraction. Szegedy et al. (2015); Tan & Le (2019) use multi-scale convolutional kernels at every layer of models for cross-scale features, while the largest kernel size can only be 7 × 7, and larger kernel size will induce unaccpetable computational budget. For vision transformers, CoaT (Xu et al., 2021a) uses multi-scale features at later layers of models by mixing features from different layers; CViT (Chen et al., 2021a) keeps embeddings of different scales in different branch, and introduces cross-scale interaction through self-attention between branches. More discussions can be seen in the appendix C.1.
Sparse Self-attention. Many substitutes have been proposed (Liu et al., 2021b; Wang et al., 2021; Chu et al., 2021; Child et al., 2019; Beltagy et al., 2020; Zaheer et al., 2020) to alleviate the cost of the vanilla self-attention module. Instead of going self-attention among all embeddings, each embedding only interact with part of other embeddings, which change dense self-attention to a sparse one. More discussions can be seen in the appendix C.2.
Position Representations. To make the respective model aware of position information of embeddings, many different position representations (Vaswani et al., 2017) are proposed. For example, Dosovitskiy et al. (2021) directly add the embeddings with the vectors that contain absolute position information. In contrast, Relative Position Bias (RPB) (Shaw et al., 2018) resorts to position information indicating the relative distance of two embeddings.

heading: 3 CROSSFORMER
text: The overall architecture of CrossFormer is plotted in Figure 1. Following (Wang et al., 2021; Liu et al., 2021b; Lin et al., 2021), CrossFormer also employs a pyramid structure, which naturally splits the transformer model into four stages. Each stage consists of a cross-scale embedding layer (CEL, Section 3.1) and several CrossFormer blocks (Section 3.2). A CEL receives last stage’s output (or an input image) as input and generates cross-scale embeddings. In this process, CEL (except that in Stage-1) reduces the number of embeddings to a quarter while doubles their dimensions for a pyramid structure. Then, several CrossFormer blocks, each of which involves long short distance attention (LSDA) and dynamic position bias (DPB), are set up after CEL. A specialized head (e.g., the classification head in Figure 1) follows after the final stage accounting for a specific task.

heading: 3.1 CROSS-SCALE EMBEDDING LAYER (CEL)
text: Cross-scale embedding layer (CEL) is leveraged to generate input embeddings for each stage. Figure 2 takes the first CEL, which is ahead of Stage-1, as an example. It receives an image as input, then sampling patches using four kernels of different sizes. The stride of four kernels is kept the same
so that they generate the same number of embeddings4. As we can observe in Figure 2, every four corresponding patches have the same center but different scales, and all these four patches will be projected and concatenated as one embedding. In practice, the process of sampling and projecting can be fulfilled through four convolutional layers.
For a cross-scale embedding, one problem is how to set the projected dimension of each scale. The computational budget of a convolutional layer is proportional to K2D2, where K and D represent kernel size and input/output dimension, respectively (assuming the input dimension equals to the output dimension). Therefore, given the same dimension, a large kernel consumes more budget than a smaller one. To control the total budget of the CEL, we use a lower dimension for large kernels while a higher dimension for small kernels. Figure 2 provides the specific allocation rule in its subtable, and a 128 dimensional example is
given. Compared with allocating the dimension equally, our scheme saves much computational cost but does not explicitly affect the model’s performance. The cross-scale embedding layers in other stages work in a similar way. As shown in Figure 1, CELs for Stage-2/3/4 use two different kernels (2 × 2 and 4 × 4). Further, to form a pyramid structure, the strides of CELs for Stage-2/3/4 are set as 2× 2 to reduce the number of embeddings to a quarter.

heading: 3.2 CROSSFORMER BLOCK
text: Each CrossFormer block consists of a long short distance attention module (i.e., LSDA, which involves a short distance attention (SDA) module or a long distance attention (LDA) module) and a multilayer perceptron (MLP). As shown in Figure 1b, SDA and LDA appear alternately in different blocks, and the dynamic position bias (DPB) module works in both SDA and LDA for obtaining embeddings’ position representations. Following the prior vision transformers, residual connections are used in each block.

heading: 3.2.1 LONG SHORT DISTANCE ATTENTION (LSDA)
text: We split the self-attention module into two parts: short distance attention (SDA) and long distance attention (LDA). For SDA, every G×G adjacent embeddings are grouped together. Figure 3a gives an example where G = 3. For LDA with input of size S × S, the embeddings are sampled with a fixed interval I . For example in Figure 3b (I = 3), all embeddings with a red border belong to a group, and those with a yellow border comprise another group. The group’s height or width for LDA is computed as G = S/I (i.e., G = 3 in this example). After grouping embeddings, both SDA and LDA employ the vanilla self-attention within each group. As a result, the memory/computational cost of the self-attention module is reduced from O(S4) to O(S2G2), and G S in most cases. It is worth noting that the effectiveness of LDA also benefits from cross-scale embeddings. Specifically, we draw all the patches comprising two embeddings in Figure 3b. As we can see, the smallscale patches of two embeddings are non-adjacent, so it is difficult to judge their relationship without the help of the context. In other words, it will be hard to build the dependency between these two embeddings if they are only constructed by small-scale patches (i.e., single-scale feature). On the contrary, adjacent large-scale patches provide sufficient context to link these two embeddings, which makes long-distance cross-scale attention easier and more meaningful.
We provide the pseudo-code of LSDA in the appendix (A.1). Based on the vanilla multi-head selfattention, LSDA can be implemented with only ten lines of code. Further, only reshape and permute operations are used, so no extra computational cost is introduced.
4The image will be padded if necessary.

heading: 3.2.2 DYNAMIC POSITION BIAS (DPB)
text: Relative position bias (RPB) indicates embeddings’ relative position by adding a bias to their attention. Formally, the LSDA’s attention map with RPB becomes:
Attention = Softmax(QKT / √ d + B)V , (1)
where Q,K,V ∈ RG2×D represent query, key, value in the self-attention module, respectively. √ d is a constant normalizer, and B ∈ RG2×G2 is the RPB matrix. In previous works (Liu et al., 2021b), Bi,j = B̂∆xij ,∆yij , where B̂ is a fixed-sized matrix, and (∆xij ,∆yij) is the coordinate distance between the ith and jth embeddings. It is obvious that the image/group size is restricted in case that (∆xij ,∆yij) exceeds the size of B̂. In contrast, we propose an MLP-based module called DPB to generate the relative position bias dynamically, i.e.,
Bi,j = DPB(∆xij ,∆yij). (2)
The structure of DPB is displayed in Figure 3c. Its non-linear transformation consists of three fullyconnected layers with layer normalization (Ba et al., 2016) and ReLU (Nair & Hinton, 2010). The input dimension of DPB is 2, i.e., (∆xij ,∆yij), and intermediate layers’ dimension is set as D/4, where D is the dimension of embeddings. The output Bij is a scalar, encoding the relative position feature between the ith and jth embeddings. DPB is a trainable module optimized along with the whole transformer model. It can deal with any image/group size without worrying about the bound of (∆xij ,∆yij). In the appendix (A.2), we prove that DPB is equivalent to RPB if the image/group size is fixed. In this case, we can transform a trained DPB to RPB in the test phase. We also provide an efficient O(G2) implementation of DPB when image/group size is variable (the complexity is O(G4) in a normal case because B ∈ RG2×G2 ).

heading: 3.3 VARIANTS OF CROSSFORMER
text: Table 1 lists the detailed configurations of CrossFormer’s four variants (-T, -S, -B, and -L for tiny, small, base, and large, respectively) for image classification. To re-use the pre-trained weights, the models for other tasks (e.g., object detection) employ the same backbones as classification except that they may use different G and I . Specifically, besides the configurations same to classification, we also test with G1 = G2 = 14, I1 = 16, and I2 = 8 for the detection (and segmentation) models’ first two stages to adapt to larger images. The specific configurations are described in the appendix (A.3). Notably, the group size or the interval (i.e., G or I) does not affect the shape of weight tensors, so the backbones pre-trained on ImageNet can be readily fine-tuned on other tasks even though they use different G or I .

heading: 4 EXPERIMENTS
text: The experiments are carried out on four challenging tasks: image classification, object detection, instance segmentation, and semantic segmentation. To entail a fair comparison, we keep the same data
augmentation and training settings as the other vision transformers as far as possible. The competitors are all competitive vision transformers, including DeiT (Touvron et al., 2021), PVT (Wang et al., 2021), T2T-ViT (Yuan et al., 2021), TNT (Han et al., 2021), CViT (Chen et al., 2021a), Twins (Chu et al., 2021), Swin (Liu et al., 2021b), NesT (Zhang et al., 2021b), CvT (Wu et al., 2021), ViL (Zhang et al., 2021a), CAT (Lin et al., 2021), ResT (Zhang & Yang, 2021), TransCNN (Liu et al., 2021a), Shuffle (Huang et al., 2021), BoTNet (Srinivas et al., 2021), and RegionViT (Chen et al., 2021b).

heading: 4.1 IMAGE CLASSIFICATION
text: Experimental Settings. The experiments on image classification are done with the ImageNet (Russakovsky et al., 2015) dataset. The models are trained on 1.28M training images and tested on 50K validation images. The same training settings as the other vision transformers are adopted. In particular, we use an AdamW (Kingma & Ba, 2015) optimizer training for 300 epochs with a cosine decay learning rate scheduler, and 20 epochs of linear warm-up are used. The batch size is 1,024 split on 8 V100 GPUs. An initial learning rate of 0.001 and a weight decay of 0.05 are used. Besides, we use drop path rate of 0.1, 0.2, 0.3, 0.5 for CrossFormer-T, CrossFormer-S, CrossFormer-B, CrossFormer-L, respectively. Further, Similar to Swin (Liu et al., 2021b), RandAugment (Cubuk et al., 2020), Mixup (Zhang et al., 2018a), Cutmix (Yun et al., 2019), random erasing (Zhong et al., 2020), and stochastic depth (Huang et al., 2016) are used for data augmentation.
Results. The results are shown in Table 2. As we can see, CrossFormer achieves the highest accuracy with parameters and FLOPs comparable to other state-of-the-art vision transformer structures. In specific, compared against strong baselines DeiT, PVT, and Swin, our CrossFormer outperforms them at least absolute 1.2% in accuracy on small models. Further, though RegionViT achieves the same accuracy (82.5%) as ours on a small model, it is 0.7% (84.0% vs. 83.3%) absolutely lower than ours on the large model.

heading: 4.2 OBJECT DETECTION AND INSTANCE SEGMENTATION
text: Experimental Settings. The experiments on object detection and instance segmentation are both done on the COCO 2017 dataset (Lin et al., 2014), which contains 118K training and 5K val images. We use MMDetection-based (Chen et al., 2019) RetinaNet (Lin et al., 2020) and Mask R-CNN (He et al., 2017) as the object detection and instance segmentation head, respectively. For both tasks, the backbones are initialized with the weights pre-trained on ImageNet. Then the whole models are
trained with batch size 16 on 8 V100 GPUs, and an AdamW optimizer with an initial learning rate of 1× 10−4 is used. Following previous works, we adopt 1× training schedule (i.e., the models are trained for 12 epochs) when taking RetinaNets as detectors, and images are resized to 800 pixels for the short side. While for Mask R-CNN, both 1× and 3× training schedules are used. It is noted that multi-scale training (Carion et al., 2020) is also employed when taking 3× training schedules. Results. The results on RetinaNet and Mask R-CNN are shown in Table 3 and Table 4, respectively. As we can see, the second-placed architecture changes along with the experiment, that is, these architectures may perform well on one task but poorly on another task. In contrast, our CrossFormer outperforms all the others on both tasks (detection and segmentation) with both model sizes (small and base). Further, CrossFormer’s performance gain over the other architectures gets sharper when enlarging the model, indicating that CrossFormer enjoys greater potentials.

heading: 4.3 SEMANTIC SEGMENTATION
text: Experimental Settings. ADE20K (Zhou et al., 2017) is used as the benchmark for semantic segmentation. It covers a broad range of 150 semantic categories, including 20K images for training and 2K for validation. Similar to models for detection, we initialize the backbones with weights pre-
trained on ImageNet, and MMSegmentation-based (Contributors, 2020) semantic FPN and UPerNet (Xiao et al., 2018) are taken as the segmentation head. For FPN (Kirillov et al., 2019), we use an AdamW optimizer with learning rate and weight deacy of 1× 10−4. Models are trained for 80K iterations with batch size 16. For UPerNet, an AdamW optimizer with an initial learning rate of 6× 10−5 and a weight decay of 0.01 is used, and models are trained for 160K iterations. Results. All results are shown in Table 5. Similar to object detection, CrossFormer exhibits a greater performance gain over the others when enlarging the model. For example, CrossFormer-T achieves 1.4% absolutely higher on IOU than Twins-SVT-B, but CrossFormer-B achieves 3.1% absolutely higher on IOU than Twins-SVT-L. Totally, CrossFormer shows a more significant advantage over the others on dense prediction tasks (e.g., detection and segmentation) than on classification, implying that cross-scale interactions in the attention module are more important for dense prediction tasks than for classification.

heading: 4.4 ABLATION STUDIES
text: Cross-scale Embeddings vs. Single-scale Embeddings. We conduct the experiments by replacing cross-scale embedding layers with single-scale ones. As we can see in Table 6, when using single-scale embeddings, the 8 × 8 kernel in Stage-1 brings 0.4% (81.9% vs. 81.5%) absolute improvement compared with the 4× 4 kernel. It tells us that overlapping receptive fields help improve the model’s performance. Besides, all models with cross-scale embeddings perform better than those with single-scale embeddings. In particular, our CrossFormer achieves 1% (82.5% vs. 81.5%) absolute performance gain compared with using single-scale embeddings for all stages. For cross-scale embeddings, we also try several different combinations of kernel sizes, and they all show similar performance (82.3% ∼ 82.5%). In summary, cross-scale embeddings can bring a large performance gain, yet the model is relatively robust to different choices of kernel size.
LSDA vs. Other Self-attentions. Two self-attention modules used in PVT and Swin are compared. Specifically, PVT sacrifices the small-scale features when computing the self-attention, while Swin restricts the self-attention in a local region, giving up the long-distance attention. As we can observe in Table 7a, compared against the PVT-like and Swin-like self-attention mechanisms, our CrossFormer outperforms them at least absolute 0.6% accuracy (82.5% vs. 81.9%). The results show that performing the self-attention in a long-short distance manner is most conducive to improving the model’s performance.
DPB vs. Other Position Representations. We compare the parameters, FLOPs, throughputs, and accuracies of the models among absolute position embedding (APE), relative position bias (RPB), and DPB. The results are shown in Table 7b. DPB-residual means DPB with residual connections. Both DPB and RPB outperform APE for absolute 0.4% accuracy, which indicates that relative position representations are more beneficial than absolute ones. Further, DPB achieves the same accuracy (82.5%) as RPB with an ignorable extra cost; however, as we described in Section 3.2.2, it is more flexible than RPB and applies to variable image size or group size. The results also show that residual connection in DPB does not help improve or even degrades the model’s performance.

heading: 5 CONCLUSIONS
text: We proposed a novel transformer-based vision architecture, namely CrossFormer. Its core ingredients are Cross-scale Embedding Layer (CEL) and Long Short Distance Attention (LSDA), thereby yielding the cross-attention module. We further proposed a dynamic position bias, making the relative position bias apply to any input size. Extensive experiments show that CrossFormer achieves superior performance over other state-of-the-art vision transformers on several representative vision tasks. Particularly, CrossFormer is demonstrated to gain great improvements on object detection and segmentation, which indicates that CEL and LSDA are together essential for dense prediction tasks.

heading: A CROSSFORMER
text: A.1 PSEUDO CODE OF LSDA
The pseudo code for LSDA is shown in Algorithm 1. As we can see, based on the vanilla selfattention module, both SDA and LDA are implemented with only ten lines of code, and only reshape and permute operations are used.
Algorithm 1 LSDA code (PyTorch-like)
# H: height, W: width, G: group size of SDA/LDA # x: input tensor (H, W, D) class LSDA():
def forward(x, type): ## group the embeddings if type == "SDA":
x = x.reshaspe(H // G, G, W // G, G, D).permute(0, 2, 1, 3, 4) elif type == "LDA":
x = x.reshaspe(G, H // G, G, W // G, D).permute(1, 3, 0, 2, 4) x = x.reshape(H * W // (G ** 2), G ** 2, D)
## the vanilla self-attention module x = Attention(x)
## un-group the embeddings x = x.reshaspe(H // G, W // G, G, G, D) if type == "SDA":
x = x.permute(0, 2, 1, 3, 4).reshaspe(H, W, D) elif type == "LDA":
x = x.permute(2, 0, 3, 1, 4).reshaspe(H, W, D) return x
A.2 DYNAMIC POSITION BIAS (DPB)
Figure 4 gives an example of computing (∆xij ,∆yij) with G = 5 in the DPB module. For a group of size G×G, it is easy to deduce that:
0 ≤ x, y < G 1−G ≤ ∆xij ≤ G− 1 1−G ≤ ∆yij ≤ G− 1.
(3)
Thus, motivated by the relative position bias, we construct a matrix B̂ ∈ R(2G−1)×(2G−1), where
B̂i,j = DPB(1−G + i, 1−G + j), 0 ≤ i, j < 2G− 1. (4)
The complexity of computing B̂ is O(G2). Then, the bias matrix B in DPB can be drawn from B̂, i.e.,
Bi,j = B̂∆xij ,∆yij . (5)
When the image/group size (i.e., G) is fixed, both B̂ and B will be also unchanged in the test phase. Therefore, we only need to compute B̂ and B once, and DPB is equivalent to relative position bias in this case.
A.3 VARIANTS OF CROSSFORMER FOR DETECTION AND SEGMENTATION
We test two different backbones for dense prediction tasks. The variants of CrossFormer for dense prediction (object detection, instance segmentation, and semantic segmentation) are in Table 8. The architectures are the same as those for image classification except that different G and I in the first two stages are used. Notably, group size (i.e., G and I) does not affect the shape of weight tensors, so backbones pre-trained on ImageNet can be fine-tuned directly on other tasks even if they use different G and I .

heading: B EXPERIMENTS
text: B.1 OBJECT DETECTION
Table 9 provides more results on object detection with RetinaNet and Mask-RCNN as detection heads. As we can see, a smaller (G, I) achieves a higher AP than a larger one, but the performance gain is marginal. Considering that a larger (G, I) can save more memory cost, we think (G1 = 14, I1 = 16, G2 = 14, I2 = 8), which accords with configurations in Table 8, achieves a better trade-off between the performance and cost.
B.2 SEMANTIC SEGMENTATION
Similar to object detection, we test two different configurations of (G, I) for semantic segmentation’s backbones. The results are shown in Table 10. As we can see, the memory costs of the two configurations are almost the same, which is different from experiments on object detection. Further, when taking semantic FPN as the detection head, CrossFormers‡ show advantages over CrossFormers on both IOU (e.g., 46.4 vs. 46.0) and FLOPs (e.g., 209.8G vs. 220.7G). When taking UPerNet as the segmentation head, a smaller (G, I) achieves higher performance like object detection.
B.3 CLASSIFICATION WITH CEL
Table 11 shows results on ImageNet dataset after plugging CEL into other vision transformers. As we can see, CEL brings about 0.5% performance gain for both Swin Transformer and NesT, which further shows the effectiveness of CEL.

heading: C DISCUSSION
text: C.1 CROSS-SCALE FEATURE EXTRACTION
CEL vs. GoogLeNet/MixConv. Since GoogLeNet (Szegedy et al., 2015) and MixConv (Tan & Le, 2019) use multi-scale convolutional kernels at every layer of models, the largest kernel size can only be 7 × 7, and larger kernel size will induce unaccpetable computational budget. In contrast, CEL only introduces cross-scale features at the start of each stage (at most 4 layers), so kernels with very large size (e.g., 32× 32) can also be used for diversified-size objects.
CEL vs. CoaT/CViT. Some vision transformers also take advantage of cross-scale features. For example, CoaT (Xu et al., 2021a) only uses multi-scale features at later layers by mixing features from front layers, while CEL introduces multi-scale features at the start of each stage, thus tokens embeddings at all layers (not only those at later layers) could be seen as cross-scale. Besides, CViT (Chen et al., 2021a) keeps embeddings of different scales in different branches and introduces cross-scale interaction through self-attention between branches. This method can well preserve the information of each scale, but multiple branches take more memory/computational/parameters cost than a single-branch model. As a result, for a controllable cost, CViT can only introduce two different scales (4× 4 and 8× 8). In contrast, CEL only appears at the start of each stage (at most 4 layers). So, CEL can introduce more different scales (e.g., 4×4, 8×8, 16×16, 32×32) than CViT with ignorable extra cost (< 0.1M parameters and < 0.3G FLOPs in most cases).
C.2 SPARSE SELF-ATTENTION AND GROUP CONVOLUTION
LSDA vs. ShuffleNet/IGCNet. LSDA, ShuffleNet (Zhang et al., 2018b) and IGCNets (Zhang et al., 2017) devote to reducing the computational cost by approximating the original operations (self-attention and convolution) in a lower cost. Nevertheless, they are essentially different: (1) Different target and dimension: ShuffleNet/IGCNet devotes to reducing the budget caused by the large channels of convolution. So, both ShuffleNet and IGCNet do shuffle-and-permutation along channel dimension. While for the self-attention module, the budget mainly attributes to the large spatial size of feature maps (i.e., the large number of embeddings). Thus, LSDA is mainly operating along the spatial dimension. (2) Different motivation: During the process of designing LSDA, we first consider the locality of images and propose to extract short-distance attention through SDA.
Meanwhile, to preserve the ability of modeling long-distance dependency, LDA is further proposed to fill this gap. In contrast, ShuffleNet/ICGNets do not need to consider these aspects because all channels are equivalent, and locality does not exist along channel dimmension. Though they also perform convolution among adjacent channels first, it is for the sake of fast memory access.
LSDA vs. Other Sparse Self-attention. Previous works proposed some sparse self-attention, such as Big Bird (Zaheer et al., 2020), Longformer (Beltagy et al., 2020), Sparse Transformer (Child et al., 2019), etc. Wherein, as a generative model, Sparse Transformer proposed a strided attention in the one-dimensional case, and every pixel can see the pixel generated ahead of itself. Instead, LSDA works in two-dimensional case. Further, LSDA has a co-design with CEL. As we described in Section 3.2.1 and Figure 3, in long-distance attention (LDA), the small-scale patches of two embeddings are non-adjacent, so it is difficult to judge their relationship without the help of the context. In other words, it will be hard to build the dependency between these two embeddings if they are only constructed by small-scale patches (i.e., single-scale feature). On the contrary, adjacent large-scale patches provide sufficient context to link these two embeddings, which makes long-distance cross-scale attention easier and more meaningful.

Review summaries:
The paper describes a pyramidal vision transformer that introduces cross-scale patch embeddings, dynamic relative position biases and "Long-Short Attention". The model shows superior performance on imagenet classification and especially on dense prediction tasks. Ablations on classification are provided hinting that the multi-scale embeddings are the most important factor contributing to the overall superior results to similar, previous architectures.

This paper proposes a novel vision transformer architecture called CrossFormer which focuses on the cross-scale ability in the attention module. Specifically, the proposed CrossFormer introduce Cross-scale Embedding Layer (CEL) and Long Short Distance Attention (LSDA) to model cross-scale interactions. CEL has the capability to merge multi-scale features with various kernel sizes, while LSDA enables the self-attention to capture feature interactions of both short distance and long distance. Experiments demonstrate that the proposed CrossFormer achieves performance improvement on image classification, object detection, instance segmentation and semantic segmentation tasks.

This paper proposes two new techniques: Cross-scale Embedding Layer (CEL) and Long Short Distance Attention (LSDA). Cross-scale Embedding Layer samples patches with convolution kernels of different sizes and concatenates them as one embedding, so that patch embeddings contain information of multiple scales. Long Short Distance Attention includes a short attention module and a long attention module. The short attention is computed in a local region. The long one covers the whole image but is computed sparsely. These two new designs enable the network to learn cross-scale features. Also, a dynamic position bias is designed to make the network suitable for inputs of different resolutions.

The paper proposed a novel multi-stage vision transformer architecture, named CrossFormer, which aimed at building interaction among features from different scales. The proposed architecture follows the layout of Swin transformer but replaces the shifted windows transformer block with the proposed short-distance attention and long-distance attention. A cross-scale module is proposed to replace the patch embedding or patch merging layer in the Swin transformer. Comprehensive experiments are conducted on image classification, objection detection, and segmentation tasks. Results shows the effectiveness of the proposed architecture.