XCiT: Cross-Covariance Image Transformers
Authors: Alaaeldin El-Nouby
Hugo Touvron
Mathilde Caron
Piotr Bojanowski
Matthijs Douze
Armand Joulin
Ivan Laptev
Natalia Neverova
Gabriel Synnaeve
Jakob Verbeek
Herve Jegou
Venue: NeurIPS 2021 Poster (2021)
Keywords: deep learning
neural networks
image classification
image segmentation
object detection
transformer

Abstract: Following their success in natural language processing, transformers have recently shown much promise for computer vision. The self-attention operation underlying transformers yields global interactions between all tokens ,i.e. words or image patches, and enables flexible modelling of image data beyond the local interactions of convolutions. This flexibility, however, comes with a quadratic complexity in time and memory, hindering application to long sequences and high-resolution images. We propose a “transposed” version of self-attention that operates across feature channels rather than tokens, where the interactions are based on the cross-covariance matrix between keys and queries. The resulting cross-covariance attention (XCA) has linear complexity in the number of tokens, and allows efficient processing of high-resolution images.
Our cross-covariance image transformer (XCiT) is built upon XCA. It combines the accuracy of conventional transformers with the scalability of convolutional architectures. We validate the effectiveness and generality of XCiT by reporting excellent results on multiple vision benchmarks, including image classification and self-supervised feature learning on ImageNet-1k, object detection and instance segmentation on COCO, and semantic segmentation on ADE20k.
We will opensource our code and trained models to reproduce the reported results.

XCiT: Cross-Covariance Image Transformers
Alaaeldin El-Nouby!.2
Hugo Touvron' 1.3
Mathilde Caron 1:2
Piotr Bojanowski
Matthijs Douze!
Armand Joulin'
Ivan Laptev²
Natalia Neverova
Gabriel Synnaeve'
Jakob Verbeek
Hervé Jégou'
Facebook Al
2 Inria
Sorbonne University
Abstract
Following tremendous success in natural language processing. transformers have recently shown much promise for computer vision. The self-attention operation underlying transformers yields global interactions between all tokens, i.e. words image patches, and enables flexible modelling of image data beyond the local interactions of convolutions. This flexibility, however, comes with a quadratic complexity in time and memory, hindering application to long sequences and high- resolution images. We propose a "transposed" version of self-attention that operates across feature channels rather than tokens, where the interactions are based the cross-covariance matrix between keys and queries. The resulting covariane attention (XCA) has linear complexity in the number of tokens, and allows efficient processing of righ-resolution images. Our cross-covariance image transformer (XCIT) - built upon XCA combines the accuracy of conventional transformers with the scalability of convolutional architectures. We validate the effectiveness and generality of XCiT by reporting excellent results on multiple vision benchmarks, including (self-supervised) image classification on ImageNet-Il 1k, object detection and instance segmentation on COCO, and semantic segmentation on ADE20k.
Introduction
Transformers architectures
G have provided quantitative and qualitative breakthroughs in speech and natural language processing (NLP). After few attempts incorporate wide-range self-attention Dosovitskiy et al. 21] established transformers as a viable architecture for learning visual representations, reportiin competitive results for image classification while relying on large-scale pre-training. Touvron et al. [ have shown par or better accuracy/thooggpuut compared strong convolutional baselines such as EfficientNets (58) when training transformers on ImageNet- k using extensive data augmentation and improved training schemes. Promising
in vision architectures 7Z1 82
results have been obtained for other vision tasks, including image retrieval 22 object detection and
semantic segmentation 44 70
SJ
well as video understanding 2 23
One major drawback of transformers the time and memory complexity of the core self-attention operation, that increases quadratically with the number of input tokens, or similarly number of patches in computer vision. For h images, this translates prohibitive for most tasks involving high-resolution images, such as object detection and segmentation. Various strategies have been proposed to alleviate this complexity, for instance using approximate forms of self-attention feature maps [70] However, none the existing solutions are fully satisfactory, as they either trade complexity for accuracy, or their complexity remains excessive for processing very large images.
complexity Owwh?²) which is
81, or pyramidal architectures which progressively downsample the
We replace the self-attention, as originally introduced by Vaswani al. [68 with "transposed" attention that we denote as "cross-covariance attention" (XCA). Cross-covariance attention substi-
Code:
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

XCIT layer
Lx
Self-attention (Vaswani et al.)
Foed Forward Network (FFN)
KT/ /vdl
A(K. Q) = Softmax
LayerNorm
AER
Local Patch Interaction (LPI)
Attention (XCA)
Lore
RT
Cross- Conss-Cvvrriane Attention (XCA)
Axo(K, Q) Softmax
NayerNoru
Axc Red xd,
input tokens
K E Nxd, Q Rxd,,
Figure 1: Our XCTT layer consists of three main blocks, each preceded by LayerNorm and followed by residual connection: (i) the and (iii) a geed-forward network (FFN). By transposing the query-key interaction, the computational complexity of XCA is linear the number data elements N, rather than quadratic as in conventional self- attention.
covariacve attention (XCA) operation, (ii) the local patch interaction (LPI) module,
tutes the explicit full pairwise interaction between tokens by self-attention among features, where the attention map is derived from the cross-covariance matrix computed over the key and query projections of the token features. Importantly, XCA has linear complexity in the number of patches. To construct our Pross-Covariance Image Transformers (XCIT), we combine XCA with local patch interaction modules that rely efficient depth-wise convolutions and point-wise feedforward net- works commonly used in transformers, see Figure [ XCA can be regarded as a form of a dynamic 1 convolution, which multiplies all tokens with the same data-dependent weight matrix. We find that the performance of our XCA layer can be further improved by applying it on blocks of channels, rather than directly mixing all channels together. This "block-diagona" shape of XCA further reduces the computational complexity with a factor linear in the number of blocks.
Given its linear complexity in the number of tokens, XCIT can efficiently process images with more than thousand pixels in each dimension. Notably, our experiments show that XCIT does not compromise the accuracy and achieves similar results to DeiT [64 and CaiT 67 in comparable settings. Moreover, for dense prediction tasks such as object detection and image segmentation, models outperform popular ResNet [3] backbones well the recent transformer-basee models 44 70, 8 Finally, we also successfully apply XCIT to the self-supervised feature learning
using DINO 12 and demonstrate improved performance compared to DeiT-based backbone
(4
Overall, we summarize our contributions as follows:
We introduce cross-covariance attention (XCA), which provides a "transposed" alternative to conventional self-attention, attending over channels instead of tokens. Its complexity is linear in the number of tokens, allowing for efficient processing of high-resolution images, Figure 2
XCA attends to fixed number of channels, irrespective of the number of tokens. As result, our models are significantly more robust to changes image resolution at test time, and are therefore more amenable to process variable-size images.
For image classification, we demonstrate that our models are on par with state-of-the-ar vision transformers for multiple model sizes using simple columnar architecture, i.e., in which we keep the resolution constant across layers. In particular, our XCTT-L24 model achieves 86.0% top-1
accuracy on ImageNet, outperforming its CaiT-M24 67 and NFNet-F2 (10] counterparts with comparable numbers of parameters.
For dense prediction tasks with high-resolution images, our models outperform ResNet and multiple transformer-basee backbones. On the COCO benchmark, achieve strong performance of 48.5% and 43.7% mAP for object detection and instance segmentation respectively. Moreover, we report 48.4% mloU for semantic segmentation the ADE20k benchmark, outperforming the state-of-the-aat Swin Transformer 44 backbones across all comparable model sizes.
Finally, our XCTT model highly effective in self-supervised learning setups, achieving 80.9% top-1 accuracy ImageNet- Ik using DINO [2]

---

2
Related work
Deep vision transformers. Training deep vision transformers can be challenging due to instabilities and optimization issues. Touvron et al. [67] successfully train models with up to 48 layers using LayerScale, which weighs contributions of residual blocks across layers and improves optimization. Additionally, the authors introduce class attention layers which decouple the learning of patch features and the feature aggregation stage for classification.
Spatial structure in vision transformers. Yuan et al. 79 propose applying a soft split for patch projection with overlapping patches which is applied repeatedly across model layers, reducing the number of patches progressively. Han et al. [27] introduce a transformer module for intra-patch structure, exploiting pixel-level information and integrating with an inter-patch transformer to attain higher representation power. d' Ascoli et al. [ consider the initialization of elff-attention blocks as a convolutional operator, and demonstrate that such initialization improves the performance of vision transformers in low-data regimes. Graham et al. [26 introduce ViT, which adopts multi- stage architecture with progressively reduced feature resolution similar to popular convolutional architectures, allowing for models with high inference speed while retaining strong performance. Moreover, the authors adopt a convolution-ban based module for extracting patch descriptors. Yuan et al. 7 improve both the performance and the convergence speed of vision transformers by replacing the linear patch projection with convolutional layers and max-pooling, as well as modifying the feed-forward networks in each transformer layer to incorporate depth-wise convolutions.
Efficient attention. Numerous methods for efficient self-attention have been proposed in the literature to address the quadratic complexity of self-attention in the number of input tokens. These include restricting the span of the self-attention to local windows 4% [50, strided patterns [14), axial patterns [30, an adaptive computation across layers [57]. Other methods provide an approximation of the self-attention matrix which can be achieved by a projection across the token dimension (69], or through a factorization of the softmax-attention kernel D55 3r
56 77], which avoids explicit computation of the attention matrix. While conceptually different, our XCA performs similar computations without being sensitive to the choice of the kernel. Similarly, Lee-Thorp et al. [41] achieve faster training by substituting self attf-attention with unparametrized Fourier Transform. Other efficient attention methods rely on local attention and adding small number of global tokens, thus allowing interaction among all tokens only by hopping through the global tokens [I] 5 34] 80 Similarly, Goyal et al. 25 use global workspace though which items interact, albeit one that is shared across layers.
Transformers for high-resolution images. Several works adopt visual transformers to high- resolution image tasks beyond image classification, such as object detection and image segmentation. Wang et al. [70] design model with pyramidal architecture and address complexity by gradually reducing the spatial resolution of keys and values. Similarly, for video recognition Fan et al. [23 utilize pooling reduce the resolution across the spatial and temporal dimensions allow for an efficient computation of the attention matrix. Zhang et al. [ adopt global tokens and local attention reduce the model complexity, while Liu et al. [4 provide an efficient method for local attention with shifted windows. In addition, Zheng et al. S and Ranftl et al. [54] study problems like semantic segmentation and monocular depth estimation with the quadratic self-attention operation.
Data-dependent layers. Our XCIT layer can be regarded as a "dynamic" 1x1 convolution, which multiplies all token features with the same data-dependent weight matrix, derived from the key and query cross-covariance matrix. In the context of convolutional networks, Dynamic Filter Networks O explore a related idea, using filter generating subnetwork to produce convolutional filters based on features in previous layers. gruuezz--and-Excien convolutions in convolutional architectures. Spatially average-pooled features are fed to 2-layer MLP which produces per channel scaling parameters. Closer in spirit to our work, Lambda layers propose a way to ensure global interaction in ResNet models Their "content-base" lambda function" is computing a similar term as our cross-covariance attention, but differing in how the softmax and & normalizations are applied. Moreover, Lambda layers also include specific position- based lambda functions, and LanbdaNetworks are based on ResNets while XCTT follows the VIT architecture. Recently fata-independent analogues of self-attention have also been found to be an These
networks [32)] use data dependent 1 x
effective alternative convolutional and self-attention layers for vision tasks 20 46 62
66
3

---

methods treat entries in the attention map as learnable parameters, rather than deriving the attention map dynamically from queries and keys, but their complexity remains quadratic in the number of tokens. Zhao et al. X2 consider alternative attention forms in computer vision.
3 Method
In this section, we first recall the self-attention mechanism, and the connection between the Gram and covariance matrices, which motivated our work. We then propose our cross-covariance attention operation (XCA) which operates along the feature dimension instead of token dimension in conventional transformers and combine with local patch interaction and feedforward layers to construct our Cross-Covariance Image Transformer (XCiT), See Figure for an overview.
3.1 Background
Token self-attention. Self-attention, as introduced by Vaswani et al. 63 operates on an input matrix Rxxx where N the number of tokens, each of dimensionality d. The input X is linearly projected to queries, keys and values, using the weight matrices Wg E Rdxd, W₁ E Rdxds and W. E Rdxd, such that Q-XW; g. K-XW& and V-XW,, where dg = dg. Keys and values are used to compute attention map A(K, Q) = Softmax(QKT /vdx), and the output of the self-attention operation is defined as the weighted sum of N token features V with the weights corresponding to the attention map: Attention(4 Q, K, V) A(K, Q)V. The computational complexity of self-attention scales quadratically in N, due to pairwise interactions between all N elements.
Relationship between Gram and covariance matrices. To motivate our cross-covariance atten- tion operation, we recall the relation between Gram and covariance matrices. The unnormalised d d covariance matrix is obtained C=X X. The N N Gram matrix contains all pairwise innerproducts: G-XX The non-zero part of the eigenspectrum of the Gram and covariance matrix are equivalent, and the eigenvectors of C and G can computed in terms of each other. If V are the eigenvectors of G. then the eigenvectors of C' are given by U-XV. To minimise the computational cost, the eigendecomposition either the Gram or covariance matrix can be obtained in terms the decomposition of the other, depending which of the two matrices is the smallest
We draw upon this strong connection between the Gram and covariance matrices consider whether it is possible to avoid the quadratic cost to compute the N x attention matrix, which is computed from the analogue of the N N Gram matrix QK T XW, W,T X Below we consider how we can use the dy x dg cross-covariance matrix, K S'Q-,, T X XW, which can be computed in linear time in the number of elements N, to define an attention mechanism.
3.2 Cross-covariance attention
We propose cross- covsrcooveie based self-attention function that operates along the feature dimension, rather than along the token dimension as in token self-attention. Using the definitions of queries, keys and values from above, the cross-covariance attention function is defined as:
XC-AAtention(() (Q, K, V) = V Ax,
Axc (K, Q) Sooftmx R Q/T//))
(1)
where each output token embedding dimension is a convex combination of the d, features of its corresponding token embedding in V. The attention weights A are computed based on the cross- covariance matrix.
l2- G-Noomaaization and temperature scaling. addition to building our attention operation on the cross-covariance matrix, we make second modification compared to token self-attention. We restrict the magnitude the query and key matrices by lz normoaalisngg them, such that each column length N the normalised matrices Q and Æ has unit norm, and every element in dxd
matrix R T Q is in the range [-1, 1]. We observed that controlling the norm strongly enhances the
For C' to represent the covariance, X should be centered, i.e. X1-0. For the relation between C' and G. however, centering is required.

---

25
(G9)
20
80
ImageNet-vo Top-3
DeiT-S
Cat-S12
(Longformer)
VIL (Performar)
Swin-T
ResNet50
ResNen-S0
XCIT XCIT-S12/8
XCIT 512/8
Image
Image Resolution
Figure 2: Inference memory usage of vision trans- former variants. Our XCIT models scale linearly in the number of tokens, which makes it possible scale much larger image sizes, even comparison approaches employing approximate self-attention or a pyramidal design. All measurements are performed with batch size of 64 on a single V100-32GB GPU.
Figure 3: Performance when changing the resolution at test-time for models with similar number of pa- rameters. All networks trained at resolution 224, w/o distillation. XCIT more tolerant changes resolution than Gram-based more from the "FixRes" effect (63 when inference performed at larger resolution than at train-time.
benefit
stability of training, especially when trained with variable numbers of tokens. However, restricting the norm reduces the representational power of the operation by removing a degree of freedom. Therefore, we introduce a learnable temperature parameter which scales the inner products before the Softmax, allowing for sharper or more uniform distribution of attention weights.
Block-diagonal cross-covariance attention. Instead of allowing all features to interact among each other, we divide them into h groups, or "heads", in similar fashion multi-head token self-attention. We apply the cross poos-covariiaee attention separately per head where for each head, we learn separate weight matrices to project to queries, keys and values, and collect the corresponding weight matrices in the tensors W, hxdxx,, W₁ E ghxdxdd and W, E Raxdxd, where we set de ddd//h, Restricting the attention within heads has two advantages: (i) the complexity of aggregating the values with the attention weights is reduced by a factor h; (ii) more importantly, we empirically observe that the block diagonal version is easier optimize, and typically leads to improved results. This observation is in line with observations made for Group Normalization [A). which normalizes groups of channels separately based on their statistics, and achieves favorable results for computer vision tasks compared to Layer Normalization K which combines all channels single group. Figure ! shows that each head learns to focus on semantically coherent parts the image, while being flexible change what type of features it attends to based on the image content.
Complexity analysis. The usual token self-attention with h heads has a time complexity O(N"d) and memory complexity of O(hN²+Nol Due to the quadratic complexity, it is problematic to scale token self-attention to images with a large number of tokens. Our cross-covariance attention overcomes this drawback as its computational cost of O(Nd /h) scales linearly with the number of (dd /oh+/N)) Therefore, our model scales much better
tokens, as does the memory complexity to cases where the number of tokens N is large, and the feature dimension d is relatively small, as is typically the case, in particularly when splitting the features into á heads.
3.3 Cross-covariance image transformers
To construct our cross- cross-cov image transformers (XCIT), we adopt a columnar architecture
which maintains the same spatial resolution across layers, similarly to 22 64 67]. We combine our cross-covariance attention (XCA) block with the following additional modules, each one being preceded by a LayerNorm 3.. See Figure [ for an overview. Since in this section we specifically design the model for computer vision tasks, tokens correspond to image patches in this context.
5

---

Table 1: XCIT models. Design choices include model depth, patch embeddings dimensionality and the number of heads h used XCA. By default our models are trained and tested at resolution 224 with patch sizes of 16x 16, We also train with distillation using a convolutional teacher (denoted T) as proposed by Touvron et al. Finally, we report performance of our strongest models obtained with 8x8 patch size, fine-tuned (f) and tested at resolution 384 384 (column @ 384/8), using distillation with teacher that also fine-tuned @ 384.
[
Model
Depth
d
#heads
#params
GFLOPs
ImageNet-Ile Ik-val top-1 1 acc. (%)
@224/16
384/8
224/16
@224/16T
@ 3844/81
XCYT-NI2
12
128
4
3M
0.5
6.4
69.9
72.2
77.8
XCTT-T12
12
192
4
7M
1.2
14.3
77.1
78.6
82.4
XCTT-124
24
192
4
12M
2.3
27.3
79.4
80.4
83.7
XCIT-S12
12
384
8
26M
4.8
55.6
82.0
83.3
85.
XCTT-S24
24
384
8
48M
9.1
106.0
82.6
83.9
85.6
XCYT-M24
24
512
8
84M
16.2
188.0
82.7
84.3
85.8
XCTT-L24
768
16
189M
36.1
417.9
82.9
84.9
86.0
Local patch interaction. the XCA block communication between patches is only implicit through the shared statistics. To enable explicit communication across patches add simple Local Patch Interaction (LPI) block after each XCA block. LPI consists of two depth-wise 3x3 convolutional layers with Batch Normalization and GELU non-linearity in between. Due to its depth-wise structure, the LPI block has a negligible overhead in terms of parameters, as well as a very limited overhead in terms of throughput and memory usage during inference.
Feed-forward network. As is common in transformer models, we add a point-wise feedforward network (FFN), which has single hidden layer with 4d hidden units. While interaction between features is confined within groups in the XCA block, and no feature interaction takes place in the LPI block, the FFN allows for interaction across all features.
Global aggregation with class attention. When training our models for image classification, we utilize the class attention layers proposed by Touvron et al. [67] These layers aggregate the patch embeddings of the last XCiT layer through writing to a CLS token by one-way attention between the CLS tokens and the patch embeddings. The class attention is also applied per head, i.e. feature group.
Handling images of varying resolution. contrast to the attention map involved in token self- attention, in case the covariance blocks are of fixed size independent of the input image resolution. The softmax always operates over the same number of elements, which may explain why our models behave better when dealing with images of varying resolutions (see Figure B). In XCTT we include additive sinusoidal positional encoding [68] with the input tokens. We generate them in 64 dimensions from the 2d patch coordinates and then linearly project to the transformer working dimension d. This choice is orthogonal to the use of learned positional encoding, as ViT [21] However, it is more flexible since there is need interpolate fine-tune the network when changing the image size.
Model configurations. In Table I we list different variants of our model which we use in our experiments, with different choices for model width and depth. For the patch encoding layer, unless mentioned otherwise, we adopt the alternative used by Graham et al. (26 with convolutional patch projection layers. We also experimented with linear patch projection
described in (21),
ablation in Table Our default patch size is 16 x 16, as in other vision transformer models including ViT 211, DeiT 64 and CaiT (67). We also experiment with smaller 8 8 patches, which has been observed to improve performance [12.. Note that this is efficient with XCIT as its complexity scales linearly which the number of patches, while ViT, DeiT and CaiT scale quadratically.
4 Experimental evaluation
In this section we demonstrate the effectiveness and versatility of XCIT on multiple computer vision benchmarks, and present ablations providing insight the importance of its different components. In the supplementary material we provide additional analysis, including the impact on performance of image resolution in Section A.I and of multiple approximate attention baselines in Section A.2

---

They are trigger by different concepts when such Table 2: ImageNet classification. Number of parame- ters, FLOPs, image resolution, and top-1 accuracy ImageNet-Ik and ImageNet- V2. Training strategies vary across models, transformer-based models and the re- ported RegNet mostly follow recipes from DeiT 64
Model
#params
FLOPs
Res.
InNett
V2
EfficientNet-B5 RA
30M
9.9B
456
RegNerY-44F 5
21M
4.0B
224
80.0
72.4
DeiT-ST [4]
22M
4.68
224
81.2
68.5
Swin-T (44
29M
224
CarT-XS245
67
26M
19.38
384
84.1
74.1
XCIT-S12/16T
26M
4.8B
224
83.3
72.5
XCIT-S12/16T
26M
14.38
74.1
XCIT-S128TT
26M
55.6B
384
85.1
74.8
EfficieniNet-B7 RA 1v
66M
37.0B
600
84.7
NFNet-FO [0]]
72M
12.4B
250
83.6
72.6
RegNerY-8GP 5
39M
8.0B
72.4
TNT-B 10
66M
14.1B
224
82.8
Swin-S [5]
50M
8,7B
224
CarT-S24Y 57
47M
32.28
384
85.1
75.4
XCIT-S24/16T
48M
9.1B
224
83.9
73.3
XCIT-S24/16T
48M
26.9B
384
85.1
74.6
XCTT-S248T
48M
105.98
388
85.6
75.7
Fix-EficieeNNee-BB
35
87M
89.5B
800
85.7
75.9
RegNetY-
53
84M
16.08
224
82.9
72.4
Swin-Br [
88M
47.0B
DeiT-B
64
87M
55.58
384
85.2
75.2
CaiT-S48Y
89M
63.8B
384
76.2
attention map be- tween the CLS token and individual patches in the class-attention stage. For each column, each row represents the attention map w.r.t. one head, corre- sponding the image in the first Each head appears sensitive to semantically coherent regions. 76.1
Figure Visualization
XCIT-M24/16T
84M
16.2B
84.3
73.6
XCTT-M24/16Y
84M
47.78
384
85.4
75.1
XCIT-M24/8T
84M
187.9B
384
76.1
N#Net-F2 10
62.6B
NFNet-F3 I0
255M
114.8B
416
85.7
75.2
CaTT-M24T
67
186M
116.18
384
85.8
XCIT-L24/16T
36.1B
Hcads are sensitive to similar features within the 74.6
XCIT-L24/16T
189M
106.0B
384
85.8
same or across images (e.g. people bird faces). 75.8
XCIT-L24/ST
189M
417.8B
384
86.0
76.6
features are missing (e.g., cockpit for race ears).
4.1 Image classification
We use ImageNet-1k 1 19 to train and evaluate our models for image classification. It consists of 1.28M training images and 50k validation images, labeled across 1,000 semantic categories. Our training setup follows the DeiT recipe [64] We train our model for 400 epochs with the AdamW optimizer 4 using cosine learning rate decay. order to enhance the training of larger models, we utilize LayerScale [67] and adjust the stochastic depth [33] for each of our models accordingly (see the supplementary material for details). Following [67] images are cropped with crop ratio 1.0 for evaluation. In addition to the ImageNet-1k validation set, we report results for ImageNet-V2 55
which has a distinct test set. Our implementation is based on the Timm library
Results on ImageNet. We present family of seven models in Table with different operating points in terms of parameters and FLOPs. We observe that the performance of the XCIT models
benefits from increased capacity both in depth and width. Additionally, consistent with 64 67
we
find that using hard distillation with convolutional teacher improves the performance. Because of its linear complexity in the number of tokens, it is feasible to train XCIT at 384 x 384 resolution with small 8 x 8 patches, i.e. 2304 tokens, which provides strong boost in performance across all configurations.
We compare to the ofate-of-the-ar convolutional and transformer-base architectures 10
58 [7 in Table 2 By varying the input image resolution and/or patch size, our models provide competitive or superior performance across model sizes and FLOP budgets. First, the models operating on 224 224 and 16x 16 (e.g. XCIT-S12/16) enjoy high accuracy at relatively few FLOPs compared to their counterparts with comparable parameter count and FLOPs. Second, our models with 16 and 384 x 384 resolution images (e.g. XCIT-S12/16t) yield an improved accuracy the expense of higher FLOPs, and provide superior on-par performance compared to of-ic----theharr
7

---

Table Sell-supervised learning. Top-1
Table 4: Ablations of various architectural design choices on the task of ImageNet- I classification using the XCiT-S12 model. Our baseline model uses the convolutional projection adopted from Ve
ImageNet-Ik. We report with a crop-ratio 0.875 for consis- tency with DINO. For the last row it set to 1.0 (improves from 80.7% to 80.9%). All models are trained for 300 epochs.
SSL Method
Model
Aparams
Fl.0Ps
Linear
I-NN
Model
Ablation
ImNet top-1 acc.
MoBY
N01
Swin-T T
[5]
29M
4,58
75.0
XCYT-S12/16
82.0
DINO
19
ResNet-50
23M
4.18
74.5
65.6
XCTT-S128
Hase
83.4
DINO
12
VIT-S/I6 23]
22M
76.1
72.8
DINO
12
VIT-S/8 20
22M
22.4R
79.2
77.2
XCIT-S12/16
81.1
77.8
76.0
XCTT-S128
Linear patch proj.
83.1
DINO
1P4
XCIT-S128
26M
18.98
79.2
77.1
w/o LPI layer w/o XCA layer
80.8
DINO
VIT-R/16
17.5B
78.2
76.1
XCYT-S12/16
75.9
DINO
1R
BVS 21
87M
78.28
80.1
77.4
XCII-M2416
84M
78.8
76.4
w/o Eg-normal. learned temp.
failed
DINO
12
XCIT-M24/8
84M
64.08
80.3
77.9
XCYT-S12/16
DINO
IP]
(3TT-M24881384
84M
188.08
80.9
78.3
models with comparable computational requirements. Finally, the linear complexity of XCiT allows scale to process 384 384 images with 8 8 patch sizes (e.g. XCIT-S12/81), achieving the highest accuracy across the board, albeit at a relatively high FLOPs count.
Class attention visualization. In Figure we show the class attention map obtained in the feature aggregation stage. Each head focuses on different semantically coherent regions in the image (e.g. faces or umbrellas). Furthermore, heads tend focus on similar patterns across images (e.g. bird head or human face), but adapts by focusing on other salient regions when such patterns are absent.
Robustness to resolution changes. In Figure 3 we report the accuracy of XCIT-S12, DeiT-S and ResNet-50 trained on 224x224 images and evaluated at different image resolutions. While DeiT outperforms ResNet-50 when train and test resolutions are similar, it suffers from a larger drop in performance as the image resolution deviates farther from the training resolution. XCIT displays substantially increased accuracy when train and test resolutions are similar, while also being robust to resolution changes, in particular for the model with 8 8x8 8 patches.
Self-supervised learning. We train XCIT in a self-supervised manner using DINO 12 on ImageNet-Ik. Table 3 we report performance using the linear and k-NN protocols as Across model sizes XCIT obtains excellent accuracy with both protocols, substantially improving DINO with ResNet-50 or ViT architectures, as well as over those reported for Swin-Transformen trained with MoBY 76. Comparing the larger models to ViT, we also observed improved perfor- mance for XCIT achieving a strong 80.3% accuracy. For fair comparison, all reported models have been trained for 300 epochs. Further improved performance of small models is reported by Caron et al. [12 when training for 800 epochs, which we expect to carryover to XCiT based on the results presented here.
Analysis and ablations. In Table 4 we provide ablation experiments to analyse the impact of different design choices for our XCIT-S12 model. First, we observe the positive effect of using the convolutional patch projection as compared to using linear patch projection, for both x and 16x x 16 patches. Second, while removing the LPI layer reduces the accuracy by only 1.2% (from 82.0 to 80.8), removing the XCA layer results in a large drop of 6.1%, underlining the effectiveness of XCA. We noticed that the inclusion of two convolutional components convolutional patch projection and LPI not only brings improvements in accuracy, but also accelerates training. Third, although we were able to ensure proper convergence without Ennormalizattin of queries and keys by tweaking the syper-parameten,, we found that it provides stability across model size (depth and width) and other syyper-paramete Finally, while the learnable softmax temperature parameter is not critical, removing it drops accuracy by 0.2%. Additional ablations are provided in the supplementary material.
8

---

Table 5: COCO object detection and instance segmen- tation performance on the mini-val set. All backbones are pre-trained on ImageNet-Ik, use Mask R-CNN model 2 and are trained with the same 3x schedule.
Table 6: ADE20K semantic segmentation perfor- mance using Semantic FPN RS] and UperNet [74] (in comparable settings). We do not include com- parisons with other state- of the art models that are
pre trained on larger datasets 44 54 33
Backhone
#params
AP
AP
AP
AP
AP
ResNer18
31,2M
36.9
57.1
40.0
336
53.9
35.7
Backbone
Somantic FPN
UperNet
ViI-Tiny 8
26.9M
41.2
64.0
62.2
44.7
43.0
37.9
37.4
59.8
40.6
#params
mloU
#perams
mloU
XCIT-T12/16
26.1M
42.7
64.3
46.4
38.5
61.2
41.1
ResNet18
29
15.5M
32.9
XCIT-T12/8
25.8M
44.5
66.4
48.8
40.3
PVT-Tiny
17.0M
35.7M
ResNet50 [X]
PVT:Small
44.2M
41.0
61.7
65.3
44.9
37,1
39,9
58.4
40.1
XCVT -T28
XCII-T12/16
8.4M
8.4M
39,9
38.1
33.7M
33.7
43.5
41.5
ViL-Small XD
45.0M
43.4
64.9
47.0
39.6
62.1
42.4
ResNet50 25
28.5M
36.7
66.5M
42.0
Swin-T 49
47.8M
46.0
68.1
50.3
41.6
65.1
44.9
PVT-Small 70
28.2M
39.8
XCIT-S12/16
44.3M
45.3
67.0
49.5
40.8
64.0
43.8
Swin:T
XCT--1128
43.1M
47.0
68.9
$1.7
42.3
66.0
45.4
XCII-S12/16
30.4M
43.9
52.4M
45.9
ResNet101 25
42.8
63.2
38.5
41.3
XCTT S12/8
30.4M
44.2
52.3M
46.6
RcsNcXt101-32
62.8M
44.0
64.4
48.0
39.2
61.4
41.9
ResNer101 23
47.5M
38.8
85.5M
43.8
PVT.Madium
7D
63.9M
44.2
66.0
48.2
40.5
63.1
43.5
ResNeX:101-3 M6
47.1M
39.7
ViL-Mediam
S1
60.1M
44.6
66.3
48.5
40.7
63.8
43.7
PVT-Medium [0
48.0M
41.6
Swin-S [4
69. IM
48.5
70.2
53.5
43.3
67.3
46.6
Swin-S -4
81.0M
47.6
XCIT-S24/16
68.0
50.9
41.8
XCIT-S24/16
5188
XCIT-S24/8
64.5M
48.1
69.5
53.0
43.0
66.5
46.1
XCIT-S24/8
51.8M
47.1
73.8M
48.1
ResNcXt101-64
7S
101 9M
44.4
64.9
48.8
39.7
42.6
ResNeX:101-63 -4 h61
86.4M
40.2
PVT-Large R0
81.0M
44.5
66.0
48.3
40.7
63.4
43.7
PVT-Large ND
65.1M
42.1
ViL-Large 80
76.1M
45.7
67.2
49.9
41.3
64.4
44.5
Swin 24
121.0M
48.1
XCTT-M2416
46.7
68.2
42.0
44.9
XCII-M2416
90.8M
109.
XCIT-M24/8
98.9M
48.5
70.3
53.4
43,7
67.5
46.9
XCIT M24/8
90.8M
46.9
108.9M
48.4
4.2 Object detection and instance segmentation
Our XCIT models can efficiently process high-resolution images (see Figure Additionally, XCIT has better adaptability to varying image resolutions compared ViT models (see Figure 3 These two properties make XCIT a good fit for dense prediction tasks including detection and segmentation.
2
We evalutate XCIT for object detection and instance segmentation using the COCO benchmark 42] which consists of 118k training and 5k validation images including bounding boxes and mask labels for 80 categories. We integrate XCIT as backbone in the Mask R-CNN 22 detector with FPN 43
Since the XCTT architecture is inherently columnar, we make it FPN FPN-compatible by extracting features from different layers, e.g., layers 4, 6, 8, and 12 for XCIT-S12 S12. All features have constant stride of 8 or 16 based on the patch size, and the feature resolutions are adjusted to have strides of 4, 8, 16, and 32, similar ResNet- backbones, where the downsampling is achieved by max pooling and the upsampling is obtained using single transposed convolution layer (see the supplementary material for details). The model is trained for 36 epochs (3x schedule) using the AdamW optimizer with learning rate of 10-1, 0.05 weight decay and 16 batch size. We adopt the multiscale training and augmentation strategy of DETR Our implementation is based on the mmdetection library
Results on COCO. In Table 5 we report object detection and instance segmentation results of four variants of XCiT using 16x and 8x8 patches. We compare to ResNets 28 and concurrent efficient All models are trained using the 3x schedule after ImageNet- pre- training. Note that other results with higher absolute numbers have been achieved when pre-training on larger datasets 44 or with longer schedules 4], and are therefore not directly comparable to the reported results. First, across all model sizes XCiT outperforms the convolutional ResNet [28] and RtsNeXt 75 by a large margin with either patch size. Second, we observe a similar increase in accuracy compared to PVT 770 and ViL [81] backbones. Finally, XCIT provides competitive performance with Swin 44 For relatively small models, XCIT-S12/8 outperforms its Swin-T counterpart with a decent margin. On the other hand, Swin-S provides slightly stronger results compared to XCIT-S24/8. Utilizing smaller 8x8 patches leads to a consistent gain across models.
vision transformers 4
S.
We use report the results provided by the authors in their open-sourced code /ttpp://githubt /ttp. SvinTraanffrmmrr/Stiti Twwpeee//swwwetnaaforaaoaeee
Trans Objecttt-r--oeeeeeeeeeeeee Detection

---

4.3 Semantic segmentation
We further show transferability of our models with semantic segmentation experiments on the ADE20k dataset [84]. which consists of 20k training and 5k validation images with labels over 150 semantic categories. We integrate our backbones in two segmentation methods: Semantic FPN ES and UperNet (74]. We train for 80k and 160k iterations for Semantic FPN and UperNet respectively. Following 44. the models are trained using batch size 16 and an Adam W optimizer with learning rate of 6 x 10 and 0.01 weight decay. We apply the same method of extracting FPN features as
explained in Section 4.2 We report the performance using the standard single scale protocol (without
multi-scale and flipping). Our implementation is based on the mmsegmentation library [6
Results on ADE20K. We present the semantic segmentation performance using XCiT backbones in Table 6 First, for Semantic FPN [8). XCIT provides a superior performance compared to ResNet, ResNeXt and PVT backbones using either option of patch size. Second, compared Swin Transformers using the same UperNet decoder [74], XCTT with 8x8 patches consistently achieves higher mloU for different models. XCIT with 16x 16 patches provides a strong performance especially for smaller models where XCTT-S12/16 outperforms Swin-T.
5 Conclusion
Contributions. We present an alternative to token self-attention which operates on the feature dimension, eliminating the need for expensive computation of quadratic attention maps. We build our XCIT models with the cross-covariance attention as its core component and demonstrate the effectiveness and generality of our models on various computer vision tasks. In particular, it exhibits a strong image classification performance on par with state-of-the-aan transformer models while similarly robust to changing image resolutions as convnets. XCIT is effective as backbone for dense prediction tasks, providing excellent performance on object detection, instance and semantic segmentation. Finally, we showed that XCTT can be a strong backbone for self- seff-superviee learning, matching the state-of-the-ar results with less compute. XCTT a generic architecture that can readily be deployed in other research domains where self-attention has shown success.
Limitations. Our models enable training with smaller patches and on higher-resolution images, which leads to clear performance gains. However, for tasks like image classification this gain comes cost of relatively high number of FLOPs. In order to address this issue, other components, like FFN, could also be re-examined. Another point is that XCIT models seem to overfit more than their CaiT counterparts, see Table They are more similar to some convnets in that respect.
References
[1] Joshua Ainslie, Santiago Ontanon, Chris Alherti, Vaclav Cvicek, Zachary Fisher, Philip Pham, Anirudh Ravula. Sumit Sanghai. Qifan Wang, and Li Yang. Etc: Encoding long and structured inputs in transformers. In Conference on Empirical Methods in Natural Language Processing. 2020.
[2] Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučić, and Cordelia Schmid. Vivit: A video vision transformer. arXiv preprint arXiv:2103. 156911 2021.
[3] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey Hinton. Layer normalization. arXiv preprint
1607006600 2016.
[4] Irwan Bello. LambdaNetworks: Modeling long-range interactions without attention. arXiv preprint arXiv:2102.00 082.00022, 2021.
[5] Iz Beltagy, Matthew Peters, and Arman Cohan. Longformer: The long-documen transformer. arXiv preprint wwXiv:2004.055555 2020.
[6] Maxim Berman, Hervé Jégou, Andrea Vedaldi, lasonas Kokkinos, and Matthijs Douze. MultiGrain: unified image embedding for classes and instances. arXiv preprint
2019.
[7] Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding? arXiv preprint 2:xiv:2/02.05000 2021.
[8] Y-Lan Boureau, Jcan Ponce, and Yann LeCun. thcoretical analysis of feature pooling in visual recognition.
In International Conference on Machine Learning, 2010.
[9] B. De Brabandere, X. Jia, T. Tuytelaars, and L. Van Gool. Dynamic filter networks. In Advances in Neural Information Processing Systems, 2016.
10

---

[10] Andrew Brock, Soham De, Samuel L Smith, and Karen Simonyan. Higg-peeffrmance large-scale image recognition without normalization. arXiv preprini arXiv:2102.06171, 2021.
[11] Nicolas Carion, Francisco Massa, Gabriel Synnaeve. Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European Conference Computer Vision, 2020.
[12] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties self-supervised vision transformers. arXiv preprint arXiv:2104.1 14294, 2021.
[13] Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang. Dazhi Cheng, Chenchen Zhu, Tianheng Cheng, Qijie Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang, Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin. MMDetection: Open mmlab detection toolbox and benchmark. arXiv preprint arXiv:1906. 0006007155.220 2019.
[14] Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv: 190:::904400000 10509, 2019.
[15] Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. arXiv preprint arXiv:2009. 14794, 2020.
[16]
MMSegmentation Contributors. MMSegmentatiin: Openmmlab semantic segmentation toolbox and benchmark. hattpa://gitaei com hptp//oennnn-eee mm1 ab/ mbm.mmmmmmmmttttiiionn 2020.
[17] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoe V Le. Randaugment: Practical automated data augmentation with reduced search space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Partern Recognition Workshops, 2020.
[18] Stéphane d' Ascoli, Hugo Touvron, Matthew Leavitt, Ari Morcos, Giulio Biroli, and Levent Sagun. Convit: Improving vision transformers with soft convolutional inductive biases. arXiv preprint arXiv:2133.1669, 2021.
[19] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: large-scale hierarchical image database. Computer Vision and Pattern Recognition, 2009,
[20] Xiaohan Ding, Xiangyu Zhang, Jungong Han, and Guiguang Ding. RepMLP: Re-parameterizing convolu- tions into fully-connected layers for image recognition. arXiv preprint arXiv:2105.00 0183, 2021.
[21] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenhomn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghami, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
[22] Alaacldin El-Nouby, Natalia Neverova, Ivan Laptev, and Hervé Jégou. Training vision transformers for image retrieval. arXiv preprint arXiv: 202.0200,,, 2021.
[23] Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale vision transformers. arXiv preprint arXiv:2104. 1/207 2021.
[24] Albert Gordo, Jon Almazán, Jérôme Revaud, and Diane Larlus. Enndtt-end learning of deep visual representations for image retrieval. International journal of Computer Vision, 124, 2017.
[25] Anirudh Goyal, Aniket Didolkar, Alex Lamb, Kartikeya Badola, Nan Rosemary Ke, Nasim Rahaman, Jonathan Binas, Charles Blundell, Michael Mozer, and Yoshua Bengio. Coordination among neural modules through a shared global workspace. arXiv preprint arXiv:2103.01197, 2021. URL https
serxiv..org/bee http://bb22110..0rm
[26] Ben Graham, Alaaeldin El-Nouby, Hugo Touvron, Pierre Stock, Armand Joulin. Hervé Jégou, and Matthijs Douze. Levit: a vision transformer in convnet's clothing for faster inference. arXiv preprint
arXiv:2104.01136 2021.
[27] Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, and Yunhe Wang. Transformer in transformer. arXiv preprint arXiv:2103.00112 2021.
[28] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition.
In Computer Vision and Pattern Recognition, 2016.
[29] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In International Conference Computer Vision, 2017.
[30] Jomathan Ho. Nal Kalchbrenner, Dirk Weissenborn, and Tim Salimans. Axial attention in multidimensional transformers. arXiv preprint 1111 2019.

---

[31] Grant Van Hom, Oisin Mac Aodha, Yang Song. Alexander Shepard, Hartwig Adam, Pietro Perona. and Serge J. Belongie. The iNaturalist species classification and detection dataset. arXiv preprint vrXiv:1707.06G0
2017.
[32] Jie Hu, Li Shen, and Gang Sun. Squeezz-and-exxitation networks. In Computer Vision and Pattern Recognition, 2018
[33] Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinherger. Deep networks with stochastic depth. In European Conference on Computer Vision, 2016.
[34] Andrew Jaegle, Felix Gimeno, Andrew Brock, Andrew Zisserman, Oriol Vinyals, and Joao Carreira. Perceiver: General perception with iterative attention. arXiv preprint arXiv:2103.0222 2021.
[35] Hervé Jégou, Matthijs Douze, and Cordelia Schmid. Hamming embedding and weak geometric consisteney for large scale image search. In European Conference on Computer Vision, 2008.
[36] Hervé Jégou, Florent Perronnin, Matthijs Douze, Jorge Sánchez, Patrick Perez, and Cordelia Schmid. Aggregating local image descriptors into compact codes, TEEE Transactions on pottern Analysis and Machine Intelligence, 34(9), 2012.
[37] Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are RNNs:
Fast autoregressive transformers with linear attention. In International Conference on Machine Learning, 2020.
[38] Alexander Kirillov. Ross Girshick, Kaiming He, and Piotr Dollár. Panoptic feature pyramid networks. In Computer Vision and Pattern Recognition, 2019.
[39] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In 4th International IEEE Workshop on 3D Representation and Recognition (3dRR- 13), 2013.
[40] Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, CIFAR, 2009.
[41] James Lee- Lhorp, Joshua Ainslie, Ilya Eckstein, and Santiago Ontanon. Fnet: Mixing tokens with fourier transforms. arXiv preprint arXiv:2105 03324, 2021.
[42] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects context. In European Conference on Computer Vision, 2014.
[43] Tsung-Yi Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Computer Vision and Pattern Recognition, 2017.
[44] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hicrarchical vision transformer using shifted windows. arXiv preprint 103.v2210..140300 2021.
[45] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 1051v 2017.
[46] Luke Keras-Kyriazi. Do you even need attention? a stack of foed-forward layers does surprisingly well on imagenet. arXiv preprint arXiv:2 2021.
[47] M-E. Nilsback and A. Zisserman. Automated flower classification over a large number classes. In Proceedings of the Indian Conference on Computer Vision, Graphics and Image Processing, 2008
[48] Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In International Conference on Machine Learning. 2018
[49] J. Philbin. O. Chum, M. Isard, J. Sivic, and A. Zisserman. Object retrieval with large vocabularies and fast spatial matching. In Computer Vision and Pattern Recognition, 2007.
[50] Jiezhong Qiu, Hao Ma. Omer Levy, Scott Wen-tau Yih, Sinong Wang, and Jie Tang. Blockwise self- attention for long document understanding. arXiv preprint arXiv:1911 1.02972, 2019.
[51] Filip Radenović, Ahmet Iscen, Giorgos Tolias, Yannis Avrithis, and Ondiej Chum. Revisiting oxford and paris: Large-scale image retrieval benchmarking. In Computer Vision and Pattern Recognition, 2018.
[52] Filip Radenović, Giorgos Tolias, and Ondrej Chum. Fine-tuning CNN image retrieval with no human annotation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018.
[53] Ilija Radosavovic, Raj Prateck Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Designing network design spaces. In Computer Vision and Pattern Recognition, 2020.
[54] René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. arXiv preprint arXiv21033 13413, 2021.
[55] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize imagenet? In International Conference on Machine Learning, 2019.
12

---

[56] Zhuoran Shen. Mingyuan Zhang, Haiyu Zhao, Shuai Yi, and Hongsheng Li. Efficient attention: Attention with linear complexities. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2021.
[57] Sainbayar Sukhbaatar, Edouard Grave, Piotr Bojanowski, and Armand Joulin. Adaptive attention span in transformers. arXiv preprint urXiv:1905.07799 2019.
[58] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning PMLR, 2019.
[59] Giorgos Tolias, Yannis Avrithis, and Hervé Jégou. Image search with selective match kemels: aggregation across single and multiple images. International journal of Computer Vision. 116(3), 2016.
[60] Giorgos Tolias, Ronan Siere, and Hervé Jégou. Particular object retrieval with integral max- max-pooling of enn activations. In International Conference on Learning Representations, 2016.
[61] Giorgos Tolias, Tomas Jenicek, and Ondiej Chum. Learning and aggregating deep local descriptors for instance-level recognition. In European Conference on Computer Vision, 2020.
[62] Ilya Tolstikhin, Neil Houlsby, Alexander Kolesnikov. Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung. Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, Mario Lucie, and Alexey Dosovitskiy, MLP-Mixer: An alI-MLP architecture for vision. arXiv preprini wxXv:2105.01601 2021.
[63] H Touvron, A Vedaldi, M Douze, and Jégou. Fixing the train-test resolution discrepaney. Advances in Neural Information Processing Systems, 2019.
[64] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data- efft-effic image transformers and distillation through attention. arXiv preprint 1287, 2020.
[65] Hugo Touvron, Andrea Vedaldi. Matthijs Douze, and Hervé Jégou. Fixing the train-test resolution discrepancy: Fixefficientnet. arXiv preprint arXiv:2003.082:: 082337, 2020.
[66] Hugo Touvron, Piotr Bojanowski. Mathilde Caron, Matthieu Cord, AlaacIdin El-Nouby, Edouard Grave, Armand Joulin, Gabriel Synnaeve, Jakob Verbeek, and Hervé Jégou. ResMLP: Feedforward networks for image classification with data-efficient training. arXiv preprint arXiv:2 105.020050 2021.
[67] Hugo Touvron, Matthiuu Cord, Alexandre Sablayrolles. Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. arXiv preprint arXiv: 17239, 2021.
[68] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, 2017.
[69] Sinong Wang, Belinda Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXi::2006.04766 2020.
[70] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. arXiv preprint arXiv: 102. 12 /2122, 2021.
[71] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In Computer Vision and Pattern Recognition, 2018.
[72] Ross Wightman. Pytorch image models. 2019.
[73] Yuxin Wu and Kaiming He. Group normalization. In European Conference on Computer Vision, 2018.
[74] Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In Eurupean Conference on Computer Vision. 2018
[75] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transforma- tions for deep neural networks. In Computer Vision and Pattern Recognition, 2017.
[76] Zhenda Xie, Yutong Lin, Zhuliang Yao, Zheng Zhang, Qi Dai, Yue Cao, and Han Hu. Self sulf-super learning with swin transformers arXiv preprint arXiv:2105.045. 2021.
(77] Yunyang Xiong, Zhanpeng Zeng, Rudrasis Chakraborty, Mingxing Tan, Glenn Fung, Yin Li, and Vikas Singh. Nyströmformer: A nyström-based algorithm for approximating self-attention. arXiv preprint
mrXiv:2102.03902. 2021.
[78] Kun Yuan, Shaopeng Guo, Ziwei Liu, Aojun Zhou, Fengwei Yu, and Wei Wu. Incorporating convolution designs into visual transformers. arXiv preprint arXiv:2 /1816, 2021.
[79] Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zihang Jiang, Francis EH Tay, Jiashi Feng. and Shuicheng Yan. Tokens-to to-en VIT: Training vision transformers from scratch on ImageNet. arXiv preprint arX 1188,, 2021.
13

---

[80] Manzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang. Li Yang, al. Big bird: Transformers for longer sequences. arXiv preprint arXiv:2007.1440 14062, 2020.
[81] Pengchuan Zhang. Xiyang Dai, Jianwei Yang. Bin Xiao, Lu Yuan, Lei Zhang, and Jianfeng Gao. Multi- scale vision longformer: new vision transformer for high-resolution image encoding. arXiv preprint arXiv:2103.15358, 2021.
[82] Hengshuang Zhao, Jiaya Jia, and Vladlen Koltun. Exploring self-attention for image recognition. In Computer Vision and Pattern Recognition, 2020.
[83] Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip HS Torr, et al. Rethinking semantic segmentation from a sequence-to-seqqeeeee perspective with transformers. arXiv preprint 2012::2012.18400 2020.
[84] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Computer Vision and Pattern Recognition, 2017.
14


Reviews:
[{"review_id": "blwOBRlchjC", "reviewer": "Reviewer_oXoV", "summary": "This paper proposes the idea of bypassing the calculation of NxN attention scores and directly deriving the weighted average of values in a self-attention by first calculating DxD cross-covariance matrix of K^TQ and then averaging over the D features of V instead of its tokens. Since this method of attention does not involve cross-token communication, the paper further proposes to use a Local Patch Interaction module in which a depth-wise 3x3 convolution enables cross-token communication.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Manuscript: The paper is very well-written and easy to follow. While the paper provides results on multiple tasks and datasets, the comparison to state-of-the-art is not complete and very little ablation is provided.\n\nNovelty: The core idea of the paper is interesting. However, intuitive justification for some ideas is missing in the paper. It is important to cover the following questions in the paper:\n1. The intuition behind a conventional attention layer is clear. What is the equivalent intuition behind XCA attention layer?\n2. The paper claims that adding LPI helps with accuracy and convergence. This might be due to the Batch Normalization in this layer. Have the authors tried only a Batch Normalization (which also implicitly brings cross-sample stats in) after the XCA modules?\n3. The paper argues against MLP-Mixer for deriving attention scores by learnable weights. Related to this argument, an interesting ablation to consider is applying a channel-wise MLP instead of LPI after the attention module to observe the effect of such projection. This, while still quadratic in computation, is necessary to support such argument. It essentially shows what happens if we replace the MLP-2 module in MLP-Mixer with an XCA.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The societal impact has not been discussed in the paper. However, the limitations are mentioned and discussed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "a7i3t3oXqyW", "reviewer": "Reviewer_nWMF", "summary": "This paper proposes a “transposed” self-attention module, it has linear complexity in the token side, which is more efficient on high-resolution images. The performance of XCiT on ImageNet classification  is promising, it also shows strong results on detection/segmentation and self-supervised pre-training.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper has clear paper writing, nice reference/related work, and very solid experiments. The novelty is also good for me.\n\nHere are some suggestions: \n\nThis paper seems like a new version of SENet[1], the proposed Cross-Covariance Attention (XCA) actually does a self-attention at channel level, which is more complex than SENet's way (and the results is better).  It achieves spatial interaction with 3x3 dw-convs. It makes sense.\n\nThe idea of XCA is good. It can reduce the complexity of self-attention. But I have several questions. If the authors can solve my concerns in rebuttal, I am happy to increase my score. \n\n(1) How about the results on detection/segmentation of combining XCA with general-purpose Vision Transformers such as PVT[2] and Swin Transformer[3]? \nBecause XCiT can only return features with the same scales, and manually resize them to multiple scales, which is not elegant and may hurt performance.\nThis paper has a strong potential to handle high-resolution images, which is important for object detection and semantic segmentation tasks. Choosing PVT or Swin as a baseline is a better choice.\n\n(2) How about using regular self-attention for patch interaction instead of using 3x3 convs? Can it further improve the performance? Or use efficient attention in PVT(sparse manner) or Swin(local manner).\n\n(3) Miss the ablation study about comparison XCA and Squeeze-and-Excitation. \n\n\nReference\n\n[1] Squeeze-and-Excitation Networks\n\n[2] Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions\n\n[3] Swin Transformer: Hierarchical Vision Transformer using Shifted Windows", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "-UZBr5BHuRK", "reviewer": "Reviewer_d8h9", "summary": "This paper presents a modification of vision transformers by introducing Cross-Covariance Attention (XCA) and Local Patch Interaction (LPI) which substantially reduce the computational overhead and enhances the stability of training. The authors also adopt the DeiT training strategy with distillation using a convolutional teacher to improve the performance. Experiments on Imagenet classification, self-supervised training with DINO and many downstream tasks (COCO object detection and instance segmentation, ADE20K semantic segmentation) show the superior performance. ", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The core idea of Cross-Covariance Attention (XCA) to make transformer linearity in this paper was originally proposed in [56]. This submission use it to make the whole transformer model work with L2-Normalization and temperature scaling.\nOn the other hand, the Local Patch Interaction (LPI) is good but do not bring any theoratical contribution or give exciting intuition despite the 1.2% improvement on ImageNet (Table 4).\nAnother contribution is that the final model performs better than original ViT on various tasks including self-supervised training with DINO and many downstream tasks.\nIt also achieves state-of-the-art performance by using the DeiT training strategy.\nHowever, this paper does not bring any new idea but incorporates several different things together. I vote for rejection.\n\n[56] Zhuoran Shen, Mingyuan Zhang, Haiyu Zhao, Shuai Yi, and Hongsheng Li. Efficient attention: Attention with linear complexities. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2021.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "See above", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "-H_4SJN__Rt", "reviewer": "Reviewer_DFVp", "summary": "This paper proposes an efficient method to compute self-attention in Transformers without sacrificing accuracy. The idea is to compute the covariance self-attention over the feature dimension followed by a 3x3 conv to model the patch interaction. With additional techniques such as feature grouping and L2 norm, the method shows competitive accuracy on ImageNet compared to the recent Vision Transformer models. This method has linear complexity to the input patches and thus can be used for high-resolution images on the detection and segmentation tasks.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality**: High the method is new in vision transformers.\n\n**Quality**: Good. The method is well motivated and explained. The main claim appears to be supported by strong empirical results.\n\n**Clarity**: Great.\n\n**Significance**: High. This fundamental method enables linear computation of self-attention over high-resolution images which may impact many works in object detection and segmentation.\n\nStrengths:\n+ Novel and efficient method that is well-motivated by covariance computation and 1x1 conv operator.\n+ The problem is non-trivial for which the authors develop additional techniques in order to obtain promising results.\n+ Extensive experiments are conducted over multiple tasks to support its efficiency and accuracy.\n\nWeaknesses:\n- The performance on ImageNet is not state-of-the-art without distillation. It is okay but is it fair to include the distillation numbers in Table 2 as many of the baselines do not? What is the performance for the teacher conv model for the models in Table 1?\n\n- The comparison to other efficient attention computation methods (in Table A of Appendix) should be included in the main paper. Would the results still be consistent on other popular Vision Transformers other than DeiT? \n\n- As the advantage of the method is on high-resolution images, why not compare with other baselines in Table A on object detection or segmentation?\n\nMy preliminary rating is 7 but I hope the authors can address the above questions.\n\n\n=====\nPost-rebuttal\n\nThank you for answering the questions. My questions are mainly about clarity and presentation, and the author made them clearer in the rebuttal. I keep my rating and hope the authors would make the changes stated in the rebuttal.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3.5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Initially, the paper received three acceptance and one rejection. One reviewer had concern on the novelty of the paper given its similarity with EA. Upon rebuttal and discussion, the reviewer converged with the other reviewers on the novelty of this paper. The AC agrees with the reviewers and recommends to accept the paper. The authors are encouraged to improve final version of the paper following the suggestions from the reviewers.
Decision: Accept (Poster)