Augmented Shortcuts for Vision Transformers
Authors: Yehui Tang
Kai Han
Chang Xu
An Xiao
Yiping Deng
Chao Xu
Yunhe Wang
Venue: NeurIPS 2021 Poster (2021)
Keywords: Vision transformer
computer vision
ImageNet

Abstract: Transformer models have achieved great progress on computer vision tasks recently. The rapid development of vision transformers is mainly contributed by their high representation ability for extracting informative features from input images. However, the mainstream transformer models are designed with deep architectures, and the feature diversity will be continuously reduced as the depth increases, \ie, feature collapse. In this paper, we theoretically analyze the feature collapse phenomenon and study the relationship between shortcuts and feature diversity in these transformer models. Then, we present an augmented shortcut scheme, which inserts additional paths with learnable parameters in parallel on the original shortcuts. To save the computational costs, we further explore an efficient approach that uses the block-circulant projection to implement augmented shortcuts. Extensive experiments conducted on benchmark datasets demonstrate the effectiveness of the proposed method, which brings about 1% accuracy increase of the state-of-the-art visual transformers without obviously increasing their parameters and FLOPs.

Augmented Shortcuts for Vision Transformers
Yehui Tang Kai Han', Chang Xu', An Xiao*, Viping Deng'. Chao Xu', Yunhe Wang" Key Lab of Machine Perception (MOE), Dept. of Machine Intelligence, Peking University. Huawei Noah's Ark Lab. *Central Software Institution, Huawei Technologies. SSchool of Computer Science, Faculty of Engineering, University of Sydney. yhtang @ pku.edu.cn, (kai.han, yunhe. wang, an xiao, yiping.deng humawei.com, c.xu suu@sydney.edu.ag xuchao@cis wiw..uu.edu..cc..
Abstract
Transformer models have achieved great progress on computer vision tasks re- cently. The rapid development of vision transformers is mainly contributed by their high representation ability for extracting informative features from input images. However, the mainstream transformer models are designed with deep architec- tures, and the feature diversity will be continuously reduced as the depth increases, i.e. feature collapse. In this paper, we theoretically analyze the feature collapse phenomenon and study the relationship between shortcuts and feature diversity in these transformer models. Then, we present an augmented shorteut scheme, which inserts additional paths with learnable parameters in parallel on the original short- cuts. To save the computational costs, we further explore an efficient approach that uses the block-circulan projection to implement augmented shortcuts. Extensive experiments conducted on benchmark datasets demonstrate the effectiveness of the proposed method, which brings about 1 % accuracy increase of the satee-o-thhe--e visual transformers without obviously increasing their parameters and FLOPs.
1 Introduction
Originating from the natural language processing field, transformer models 33 9 have recently made great progress in various computer vision tasks such image classification 9 34 23 object detection 3 and image processing 4.. Wherein, the ViT model [9] divides the input images as visual sequences and obtains an 88.36% top-1 accuracy which is competitive the SOTA convolutional neural network (CNN) models (e.g.. EfficientNet (3I). Compared to CNNs which are usually customized for vision tasks with prior knowledge (e.g., translation equivalence and locality), vision transformers introduce less inductive bias and have a larger potential to achieve better performance on different visual tasks. Besides, considering the high performance of transformers various fields (e.g., natural language processing and computer vision), we may only need to support transformer for processing different tasks which can significantly simplify the softwareehhrdware design.
Besides the self-attention layers in vision transformers, a shortcut is often included to directly connect multiple layers with identity projection [5
The introduce of shortcut is motivated by the architecture designs in CNNs, and has been demonstrated to be beneficial for a stable convergence and 39 38). There is series of works on interpreting and understanding the role of shortcut. For example, Balduzzi et al. [ analyze the gradients of deep networks and show that shorteut connection can effectively alleviate the problem
35 9
better performance [16] 33 22
gradient vanishing and exploding.
Veit er al. (36] reckon ResNet [15) as the ensemble of a collection of paths with different lengths, and the gradient vanishing problem is addressed by short paths. From theoretical perspective, Liu et al. (22) prove that shortcut connection can avoid network parameters trapped by spurious local optimum and help them converge to global optimum. Apart from CNNs, shortcut connection is
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

83.5
Multi head Self-Attention (MSA)
83.0
Input patches
Accuray (%)
82.5
Shortcut
82.0
81.5
ViT(DeTT
Aug-VmDeeT)
PVT
Aug-S
Output patches
Aug-PVT
O0
T2T
Aug-TZT
Criculant Projection
12
14
16
18
FLOPs (G)
The diagram of MSA module equipped with augmented shortcuts, where different patterns (rectan- gle, triangle, etc.) denote different features from various patches. The original identity shorteut copies input feature while the augmented shortcuts (Aug-S) project features of each input patch to diverse representations.
Figure 1:
Figure 2: Performance improvement for using the proposed augmented shortcuts on the state-of-the-arr vi- sion transformer models. The perfor- mances of different models on Ima- geNet are provided.
consequently widely-used in other deep neural networks such transformer 35 9. LSTM and AlphaGo Zero 229..
In vision transformers, the shortcut connection bypasses the multihead self-attention (MSA) and multilayer perceptron (MLP) modules, which plays a critical role towards the success of vision transformers. A transformer without shortcut suffer extremely low performance (Table Empirically, removing the shortcut results in features from different patches becoming indistinguishable as the network going deeper (shown in Figure 3 a)), and such features have limited representation capacity for the downstream prediction. We name this phenomenon as feature collapse. Fortunately, adding shorteut in transformers diverse. However, the conventional shortcut simply copies the input feature to the output, limiting its ability for enhancing the feature's 's diversity.
alleviate the phenomenon (Figure 3]b) and make the generated features
In this paper, we introduce novel augmented shorteut scheme for improving feature diversity in vision transformers (Figure []. Besides the conventional identity shortcut, we propose to parallel the MSA module with multiple parameterized shortcuts, which provide more alternative paths to bypass the attention mechanism. In particular, augmented shortcut connection is constructed as the sequence of a linear projection with learnable parameters and nonlinear activation function. We theoretically demonstrate that the transformer model equipped with augmented shorteuts avoid feature collapse and produce more diverse features. For the efficiency reason, we further replace the original dense matrices with block-cireulant matrices, which have lower computational complexity in the Fourier frequency domain while high representation ability in the spatial domain. Owing to the compactness of circulant projection, the parameters and computational costs introduced by the augmented shortcuts are negligible compared to those of the MSA and MI.P modules. We empirically evaluate the effectiveness of augmented shorteut with the VIT model ( 34 and its SOTA variants (PVT (37). T2T [40)) on ImageNet dataset. Equipped with our augmented shortcut, the performances (top-1 accuracies) of these models can be enhanced by about 1% with comparable computational costs (e.g., Figure
2 Preliminaries and Motivation
The vision transformer (VIT) [9 adopts the typical architecture proposed in [35], with MSA and MLP modules alternatively stacked. For an image, VIT splits it into patches and projects each patch into d-dimension vector. Thus the input of vision transformer can be seen matrix Zy R Nxd An MSA module with heads is defined as
MSA(Z) (on
where 7, E Nxd is the features of the I-th layer, Ala E NNNN and W E (dx(d/H) are the corresponding attention map and value projection matrix in the h-th head, respectively. Concat(-)

---

denotes the concatenating for features of the H heads and W/ E Raxd is the output projection matrix. The attention matrix An is calculated by the self-attention mechanism, i.e.,
(Z.W ( (Z th Vd
Auh softmax
(2)
(ZWW (Z,WW. Vd
Ay softmax
(3)
where W E (d/(d/H)) and W (d(((/ are the query and value projection matrices, respec- tively. Attention An reflects the relation between different patches, and a larger value Al indicate that patch and patch have stronger relationship.
The MLP module extracts features from each patch independently, which usually constructed by stacking two linear projection layers with weights Wa E Rddddaadenn W E Rdiiddeo xd and non-linear activation function a, i.e,, (Z) = (Z,W"))W' vision transformer model
constructed by stacking the MLP and MSA modules alternatively.
Depth
Topl accuracy (%)
Layer 12
0.8
18.3
2
43.4
0.6
4
0.13
Layer6
6
0.12
00.4
Aug-ViT
0.15
shortout
10
0.14
0.2
ViT w/o shortout
12
0.15
Va/VIT w/o (b) VIT with Aup-VIT
Table 1: Performance of the VIT model without shortcut ImageNet.
Figure 3: The similarity ma- trices over patches in the ViT- Base model.
Layer
Figure r (Zi) varies w.E.. layers.
Diversity metric
Besides MSA and MLP, the shortcut connection also plays a vital role to achieve high performance. Empirically, removing shorteut severely harms the accuracies of VIT models (as shown in Table To explore the reason behind, we propose to analyze the intermediate features in the ViT-like models. Specially, calculate the cosine similarity between different patch features and show the similarity matrices of low (Layer 1), middle (Layer 6) and deep layers (Layer 12) in Figure 3 Features from different patches in a layer quickly become indistinguishable as the network depth increasing. We call this phenomenon feature collapse, which greatly restrict the representation capacity and then prevent high performance.
The feature collapse mainly results from the attention mechanism, which aggregates features from different patches layer-by-layer. Denoting Zin as the output feature after the attention map, the attention mechanism can be formulated as:
N s.t. Σ A 1
Zin
A Σ j=1
[1,2, ........ .N],
(4)
j=1
where the feature 7 from the i-th patch is the weighted average of features from all other patches, and the weights values in attention map An. Though attention mechanisms capture global relationship, the diversity patch features are reduced as well. Theoretically. the diversity a feature Zi can be measured by the difference between the feature and rank-1 matrix, iee
r r(4) 2 1z, [|. where 2; = argmin
1z
(5)
where || denotes the matrix norn |
Z1, 2/ E d are vectors and is an all-ones vector. The rank of matrix 1z, 1. Intuitively, 2, E Rd can represent Z, E R Nxd of N patches, the generated
Here || is defined the dx.e-oscoooisc for the convenience of theoretical derivation.

---

feature will be redundant. A larger r r(z) implies stronger diversity of the given feature Z,. We use r(Z,) as the diversity metric in the following. Because MSA module incurs the feature collapse, we focus on the model stacked by attention modules and have the following theorem 8 i.e.
Theorem Given model stacked by the MSA modules, the diversity r(X,) of feature the 1-th layer can be bounded by that of input data Zo. i.c.,
H
r(Z)) S
r(Zo)
(6)
where H is number of heads, d is feature dimension and y is a constant related to the norms of weight matrices the MSA module.
Hy/ Vd and r( r())) are usually smaller than 1, so the feature diversity r( r(Zi) will decrease rapidly as the network depth increases [8. We also empirically show how r(Zi) varies w.z.f the network depth in the ViT models (Figure 4 and the empirical results are accordant to the Theorem
Fortunately, adding shortcut connection parallel the MSA module can empirically preserve the feature diversity especially in deep layers (see Figure 3)) and Figure 4). The MSA module with shortcut can be formulated as:
ShortcutMSS((2) MSA(Zi) Ze.
where the identity projection (i.e., Zí) is parallel to the MSA module. Intuitively, the shortcut connection bypasses the MSA module and provides another alternative path, where features can be directly delivered to the next layer without interference of other patches. Adding the shortcut connections can also theoretically improve the bound of feature diversity r(Zi) (as discussed in Section 3.I). The success of shortcut shows that bypassing the attention layers with extra paths is an effective way to enhance feature diversity and improve the performance of transformer-lit models.
23 there is only a single shorteut connection with identity projection for each MSA module. which only copies the input features to the outputs. This simple formulation may not have enough representation capacity to improve the feature diversity maximally. In the following chapters, aim refine the existing shortcut connections in vision transformers and explore efficient but powerful augmented shortcuts to produce visual features with higher diversity.
However, in general vision transformer models 9 13| 37 14
Approach
3.1
Augmented Shorteuts
We propose augmented shortcuts to alleviate the feature collapse by paralleling the original identity shortcut with more parameterized projections. The MSA module equipped with T augmented shortcuts can be formulated as:
AugMSA(Z) MSA(Z) + Zi Tu On.) [1,2, ,,,
(8)
where (.)(.) is the i-th augmented shortcut connection of the 1-th layer and Ou denotes its parameters. Besides the original shortcut, the augmented shortcuts provide more alternative paths to bypass the attention mechanism. Different from the identity projection directly copying the input patches the corresponding outputs, the parameterized projection T..(.) can transform input features into another feature space. Actually, projections Ti: will make different transformations the input feature as long as their weight matrices θ₁ are different, and thus paralleling more augmented shortcuts has potential to enrich the feature space.
simple formulation for T..(-) is the sequence of linear projection and an activation function i.e.,
Tx(Z; On)) or (Z),),,, [1, (1, L], i e [1,2, 21. ,T],
where Ory Raxdd the weight matrix and the non-linear activation function (e.g., GELU). In Eq. 9 T-((-) tackles each patch independently and preserves their specificity, which is complement to

---

the MSA modules aggregating different patches. Note that the identity mapping is a special case of Eq. i.e. (xx) = x and O₁ is the identity matrix.
Recall that in a transformer-like model without shorteut, the upper bound of feature diversity r (i) decreases dramatically as the increase of network depth(Theorem]] []. In the following, we analyze how the diversity r r(z)) changes w.r.f. the layer in the model stacked by the AugMSA modules, and we has the follow theorem.
Theorem Given model stacked by the AugMSA modules, the diversity r(Xi) of feature in the 1-th layer can be bounded by that of input data 7o, i.e.,
3m
HT
r(z))= S
max USMS!
(2Ham))
13th (1-m))
(10)
where Omm = -1+E OllO mmi is the weight matrix in the i-th augmented shortcut of the m-th layer, and X is the Lipschitz constant of activation finctiin (.))
Compared with Theorem 1 the augmented shortcuts introduce an extra term (2Ham) (00mmm)("(((0))) (1-mm) which will increase doubly exponentially as 2Ham is usually larger than 1. This tends to suppress the diversity decay incurred by attention mechanism. The term O'm (0 S m S 1) is determined by the norms of weight matrices mm of the augmented shortcuts in the m-th layer, and then bound of diversity r( r(Zi) in the 1-th layer can be affected by all the augmented shortcuts in the previous layers. For the ShorteutMSA module (Eq. | with only identity shortcut, we have O'm 1. Adding more augmented shortcuts can increase the magnitude of O.m which further improves the bound. Detailed proof for Theorem 2 is represented the supplemental material.
Considering that shortcut connections exist in both MSA and MLP modules, the proposed augmented shortcuts can also be embedded into MLP similarly, i.e.,
AugMLP(Z) MLP(Z)) Z Σ Ta(Zi LE [1,2, ,.,,
(11)
where Zi is the input feature of the MLP module in the 1-th layer and 0% denotes the parameters in augmented shortcuts. Paralleling the MLP module with the augmented shortcuts can further improve the diversity, which is analyzed detailedly in the supplemental material. Stacking the AugMSA and AugMLP modules constructs the Aug-Vi7 ViT model, whose feature has stronger diversity as shown in Figure and Figure 4
3.2 Efficient Implementation via Circulant Projection
As discussed above, Paralleling multiple augmented shortcuts with the MSA and MLP modules in a vision transformer can improve the feature diversity for higher performance. However, directly implementing T
(Eq. 9 involves a lot of matrix multiplications, which are computationally
expensive 999 matrix multiplication Z) consumes nd FLOPs, where d is usually large in vision transformers (e.g., 798 in ViT-B). Therefore, we propose to implement the augmented shortcuts with block- circulant matrices, whose computational costs are negligible compared to other modules in a vision transformer. In the following, we omit the subscripts and indicating layers and paths for brevity.
4 2
For example, given feature Zi RRad and weight matrix O₁ dxd the
Circulant matrix 19 is a typical structured matrix with extremely few parameters and low computational complexity on the Fourier domain. A circulant matrix C E Rd' xd' only has d' parameters and the product between C and a vector only has Odd log d') computational complexity via the fast Fourier transformation (FFT). The projection with circulant matrix also has theoretical guarantee to be good approximation of unstructured projection and preserve the critical projection properties such as & distance and angles [17] [10. Consequently, we take advantage of circulant matrices to implement augmented shortcuts. Specially, the original matrix θ is split into 6² sub- matrices O E Rdd i.e.
C
1
2
2
26
(12)
Ob
0
G6
5

---

where d' is the size of sub-matrices and bd' = d. Each sub-matris C* is circulant matrix generated by circulating the elements in a d' di-dimnnsion vector cº [cp o
20 Le.
=+ circ(()))
(13)
4 -''1
For efficient implementation of the projection T(z) o (70), first split the input Z into b slices Z E Nxd' i.e., [Z': zz:...: 1. and then multiply each slice Z by a circulant matrix C The product between circulant matrix and vector in the original domain is equivalent to element-wise multiplication in the Fourier domain, ie., F(Z)CO) = F(Z2) O F(c")), where F(-) is the discrete Fourier transformation and @) denotes that each row in Z is multiplied by the according element in e²,, The discrete Fourier transform and inverse transformation both can be efficiently calculated with fast Fourier transform FFT and its inverse IFFT with only Odd log d') computational complexity. The output calculated as:
b Σ 20S = Σ LFTT(FFTZZ)
b
T(Z) =
(14)
where T(Z)' RN a slice of TZ)). Finally, T(Z) obtained by concatenating slices T(ZY'. :(())]]
ie., [T))
Complexity Analysis. For a x d matrix θ split into multiple 6² sub-matrices, the numbers of parameters is only bd, which has linear complexity with matrix size d. Note that the learnable parameters c can be stored as its Fourier form, and both the FFT and IFFT operations in Eq. 14 are conducted one time. Thus the computational cost (i.e., FLOPs) of Eq. 14 about (d log(d/b) where d dlog(d/b) /b) comes from the FFT and IFFT transformation and 26d is consumed by the element- wisely operation in the Fourier domain [26 2
26d),
The size of matrix d in transformer is usually large while the number of sub-matrices is small, and thus the parameters and computation costs incurred by the augmented shortcuts be negligible. For example, in the ViT-S model with d 384, b is set to 4 in our implementation. The augmented shortcuts only adds 0.07 M parameters and 0.08 G FLOPs, which is negligible considering the ViT-S model has 22.1 1 M parameters and 4.6 G FLOPs.
Experiments
In this section, conduct extensive experiments to demonstrate the effectiveness of the proposed augmented shortcuts. The vision transformer [9] and its SOTA variants are equipped with the augmented shortcuts for performance improving. We first compare the performances of different models on the ImageNet dataset for the image classification task. Then ablation studies are conducted analyze the algorithm. To validate its generalization ability, further test the proposed Aug-ViT model on the object detection and transfer learning tasks.
4.1 Experiments on ImageNet
Dataset. ImageNet (ILSVRC-2012) dataset O contains 1.3 M training images and 50k validation images from 1000 classes, which is widely used image classification benchmark.
Implementation details. We the same training strategy with DeiT [34] for fair comparison. Specifically, the model is trained with AdamW 24 optimizer for 300 epochs with batchsize 1024. The learning rate is initialized to 10 -3 and then decayed with the cosine schedule. Label smoothing (30), DropPath [21] and repeated augmentation (18) are also implemented following DeiT (34). The data
augmentation strategy contains Band-Augment (5) Mixup 42) and CutMix 4T). Besides the original
"The constants are approximated by considering the relation between complex operation and real operation. well the symmetry property 2

---

shortcuts, one or two augmented shortcuts are added, where the papee-prrrmeter b for partitioning matrices is set to empirically. The models are trained from scratch on ImageNet and no extra data are used. All experiments are conducted with PyTorch (28) and MindSpore on NVIDIA V100 GPUs. The model architecture can be found in the MindSpore model zoo
Backbones. We apply the augmented shorteuts on multiple vision transformer models. ViT [ the typical transformer model for the vision tasks, which splits an image multiple 16 16 patches. Deit [34] adopts the same architecture with VIT but improves the training strategy for better performance. T2T 40 and PVT [77 are two recently proposed SOTA variants of ViT. T2T [40 improves the process of producing patches by considering the structured information in images. PVT [37] designs a pyramid like structure by partitioning the model into multiple stages. In Table [ "ViT(DeiT)-S" 'ViT(DeiT)-B' and 'PVT-S', 'PVT-B' denote the DeiT and PVT models with different size. 'T2T-14', 'T2T-19' and 'T2T-24' are the T2T models with different depths.
Experimental Results. The validation accuracies of different models on ImageNet are shown in Table 2 We firstly equip the ViT models with the augmented shortcuts and get Aug-ViT, which show large superiority to the plain counterparts, i.e., more than 1% accuracy improvement without noticeable parameter and computational complexity increasing. For example, the proposed -iT--ViT- S' achieves 80.9% top-1 accuracy with 4.6G FLOPs, which suppresses the baseline ('ViT(DeiT)-S" with 4.6G FLOPs) by 1.2% top-1 accuracy while the computational cost is barely changed. For a large model with higher input image resolution (e.g., ViT-B with input resolution 384 x 384) and high performance. equipping with the augmented shortcuts can still further improve the performance (e.g., 83.1% - 84.2%).
Besides the typical VIT model, the augmented shorteuts be embedded into multiple vision transformers flexibly and improve their performance as well. For example, equipping the PVT-M with augmented shorteut can improve its accuracy form 81.2% 82.3%. For the T2T-14 model, the performance improvement is even more obvious, i.e., 1.3% accuracy improvement from 82.3% to 83.6%.
Performance Improvement w.n.f. Network Depth. It is interesting to see that the augmented shortcuts improve the performance of deeper models more obviously. 'T2T-14', 'T2T-19' and 'T2T- 24' compose of the same blocks but have different depths. For the baseline, increasing the depth of T2T model from 14 to 24, the accuracy is only improved by 0.7% (from 81.5% to 82.3%). While with the augmented shortcuts, the Aug-T2T model with 24 layers can achieve 83.6%, which achieves more obvious performance improvement than models with 14 layers. Deeper models tend to suffer more serve feature collapse suffer more serve feature collapse as features from different patches aggregated with more attention layers. The proposed augmented shortcuts alleviate the feature collapse and release the potential of deep models, which results in higher performance.
4.2 Ablation Studies
To better understand the proposed augmented shortcuts for vision transformers, we conduct extensive experiments to investigate the impact of each component. All the ablation experiments are conducted based on ViT(DeiT)-S model on the ImageNet dataset.
The number of augmented shortcuts. The performance varies w.z.t. the number of augmented shorteuts shown in Table 3 Besides the original identity shortcut, adding only one augmented shortcut can significantly improves the performance of the ViT model (e.g., 0.8% top-1 accuracy improvement compared to the baseline). Further increasing the number of augmented shortcuts will further improve the performance, and the improvement margin will be saturated gradually. We empirically find that two augmented shorteuts are enough achieve obvious performance improvement.
Location for implementing the augmented shorteuts. As discussed before, the augmented short- cuts can be paralleled with both MSA and MLP modules to increase the feature diversity. Table shows how the implementation location affects the final performance. Paralleling MSA with the
https://wwe wwwp::/wwwnnww.oe//m//m
https://gitee wtt.e/ooooommmnieeeeeeeeeeemeeee
'Note that DeiT [34] adopts the same model architecture with VIT [9] but achieves higher performance by adjusting the training strategy, which is used as the baseline model.
7

---

Table
Performance of different models on ImageNet.
Model
Resolution
Topl-Aceuracy (%)
Params (M)
FLOPs (G)
ViT(DeiT)-S
9
224x224
79.8
22.1
4.6
Aug-ViT-,
224x224
80.9 (+1.1)
22.2
4.7
ViT(DeiT)-B
224x224
81.8
86.4
17.6
Aug-Vitt ViT-ViT-B
224x224
82.4 (+0.6)
86.5
17.7
ViI(DeiT)-B1
9
384x 384
83.1
86.4
55.6
Aug- V
384x 384
84.2 (+1.1)
86.5
55.8
PVT-S 37
224x224
79.8
24.5
3.8
Aug-PVT-S
224x224
80.5 (+0.7)
24.6
3.9
PVT-M 37
224x224
81.2
44.2
6.7
Aug-PVT-M 37
224x224
82.3 (+1.1)
44.3
6.8
T2T-14 40
224x224
81.5
21.5
5.2
Aug-T2T-14
224x224
82.1 (+0.6)
21.6
5.3
T2T-19 40
224x224
81.9
39.2
Aug-T2T-L1 12T-19
224x224
82.9 (+1.0)
39.3
9.0
T2T-24 40
224x224
82.3
64.1
14.1
Aug-T2T-24
224x224
83.6 (+1.3)
64.2
14.3
Table Location for implementing the aug-
Table 3: The number of augmented shortcuts.
mented shortcuts.
Topl-Acc. (%)
Params
FLOPs
Path
Topl-Ace.
Params (M)
FLOPs
(M)
(G)
Location
(%)
(G)
79.8 (+0.0)
22.1
4.6
Baseline
79,8 (+0.0)
22.1
4.6
80.6 (+0.8)
+0.04
+0.04
MSA
80.6 (+0.8)
+0.035
+0.04
80.9 (+1.1)
+0.07
MLP
(+0.6)
+0.035
+0.04
80.9 (+1.1)
+0.12
+0.12
MSA&MLP
80.9 (+1.1)
+0.07
+0.08
augmented shortcuts significantly improves the performance (e.g.. 0.8% top-1 accuracy), which attribute it to the increasing feature diversity. Enhancing MLP module also bring the performance improvement, which is accordant analysis in supplementary materials. Combining them together can achieve the highest performance (1.1 1% accuracy improvement), which we adopt in our implementation.
Efficiency the block-circulant projection. In the block circulant projection, hyper- parameter controls the number of sub-matrices C* partitioned by original matrix O. Table shows how the performance varies w.r.f the parameter A larger implies the matrix O will be partitioned into more circulant matrix with smaller sizes, which brings more parameters and higher computational cost, as well as the performance improvement. The unstructured projection can also be used augmented shortcuts and brings performance improvement, but it incurs obvious increasing of parameters and computational cost. Using the very similar performance with the unstructured projection but has much few parameters (e.g., 0.07M 7.M). implying that the block-circulant projection is an efficient and effective formulation to transform features in the augmented shortcuts.
projection with b = can achieve
Formulation of the augmented shortcuts. The augmented shortcuts are implemented sequentially with the block-circulant projection and activation function (e.g., GeLUU. Table 6 shows the impact of each component. Only using the activation function introduces no learnable parameters, which only achieves similar accuracy with the baseline model. implying that the learnable parameters vital to produce diverse features. If only the linear circulant projection is kept, multiple augmented shorteut can be merged to single one. Swapping the sequential order of the circulant projection and activation function has negligible influence on the final performance.
Drop rates of stochastic depth and dropout. We change the parameters of stochastic depth/dropout for both ViT(DeiT)-S model and AiT-V and the results are shown Table 7n Table We empirically find the default parameters for DeiT (drop rates of stochastic depth/Dropout are set as 1/0.0) also work well for the Aug- VTT models. In addition, the models equipped with augmented shortcuts tend to have higher tolerance to over-large drop rates. For example, with drop rate=0.3 for

---

Table 5: Efficiency of the block cifculant projection.
Table 6: Formulation of the augmented
Tocl-Acc.
Params
FLOPs
shortcuts. 'Act' denotes the activation
Type
b
(%)
(M)
(G)
function.
Baseline
22.1
Form
Topl-Ace (%)
Block-Circulant
80.5 (+0.7)
+0.02
+0.07
Baseline
79.8
Block-Circulan
2
80.7 (+0.9)
+0.03
+0.07
Act.
79.9
Circulircul
4
80.9 (+1.1)
+0.07
+0.08
Circulant
Soock-Circulant
80,9 (+1.1)
+0.15
+0.10
Act. Circulant
80.8
Unstructured
81.0 (+1.2)
+7.1
+1.4
Circulant Act.
80,9
Table 7: Top-1 accuracies (%) on ImageNet
Table 8: Top-1 accuracies (%) on ImageNet w.r.f. drop rates of dropout.
w.t. drop rates stochastic depth.
Drop rate
DeiT-S
Aug-ViT-S
Drop rate DeiT-S
Aug-ViT-S
0
79.7
80.9
79.8
80.9
0.05
79.8
80.9
0.05
78.9
80.3
0.1
79.8
80.9
0.1
77.3
79.1
0.2
79.3
80.4
0.2
74.0
76.5
0.3
78.6
80.0
0.3
69.8
75.1
dropout, the accuracy of Aug-Vit ViT-V can still achieve 75.1% accuracy, while the accuracy of DeiT-S decreases to 69.8%.
Block-1
Black-6
image
Block-11 -1
(a) Shortout
(b) with Shortcut
Aug-VVI
Figure 5: Feature Visualization of different models.
Feature Visualization. We intuitively show features of different models Figure 5 From top to bottom features in low, middle and deep layers of the ViT-Small model. The input image is scaled to 1024 x 1024 for better visualization and the patch embeddings are reshaped to their spatial positions to construct the features maps. Without shortcut, the feature maps in deep layers conveys no effective information (Figure (a)), and adding a shortcut connection make the feature maps informative (())). Compared with them, the features in the Aug-Vi ViT model are further enriched, especially for the deep layers.
4.3 Object Detection with Pure Transformer
We also validate the effectiveness of the augmented shortcuts on the object detection task. A pure transformer detector can be constructed by combining the vision transformer backbone and the DETR 3 head, and we equip the backbones with the augmented shortcuts. For a fair comparison, we follow the training strategy in PVT 337 and fine-tune the models for 50 epochs on the COCO train2017 dataset. Random flip and random scale are used as the data augmentation strategy. The results on COCO val2017 are shown in Table The detector equipped with augmented shortcut achieve better performance than the base model. For the DeïT-S model with 33.9 AP, the augmented shortcut can improve 1.8% AP and achieve 35.7%.
4.4 Transfer Learning
To validate the generalization ability of vision transformers equipped with the projection, we conduct experiments on the transfer learning tasks. Specifically, the models trained on ImageNet is further

---

Table 9: Results of pure transformer object detection on COCO val2017 dataset.
Backbone
Parmas(M)
Epochs
AP
AP50
A7
APs
APM
APL
Backbone
41
50
32.3
53.9
32.3
10.7
33.8
53.0
DeïT-S 34
38
50
33.9
54.7
34.3
11.0
35.4
56.6
Aug-ViT-
38
35.7
56.7
36.5
12.2
38,0
58.6
PVT-S [37
40
50
34.7
55.7
35.4
12.0
36.4
56.7
Aug-PVT-S
40
50
35.1
55.8
35,9
12.5
36.5
57.3
fine-tuned the downstream tasks, containing superordinate-lveel image recognition dataset (CIFAR- 10 (20), CIFAR- (20] and fine-grained image recognition dataset( Oxford 102 Flowers 25 and Oxford-IIIT Pets 27 Following [34], we fine-tune the models images with resolution 384 384, and use the same fine-tuning strategy as [4]. Table TO shows the performances of different models
on the downstream tasks, and the model equipped with the augmented shortcut always achieves higher accuracies than the baseline on different tasks.
Table 10: Results downstream tasks with ImageNet pre-training. All the models fine-tuned with the image resolution 384 x 384.
Model
Parmas(M)
ImageNet
CIFAR-I0
CIFAR-100
Flowers
Pets
ViT-B/16, 384
86.4
77.9
98.1
87.1
89.5
93.8
DeiT-Br BeiT-Brass 384
86.4
83.1
99.1
90.8
98.4
Aug- See-VIT-BB BTT-Braa
86.4
84.1
99.2
91.3
98.8
95.1
Conclusion
We presented augmented shortcuts for resolving the feature collapse issue in vision transformers. The augmented shortcuts are parallel with the original identity shorteuts, and each connection has its own learnable parameters to make diverse transformations on the input features. Efficient circulant projections are used implement the augmented shortcuts, whose memory and computational cost are negligible compared with other components in the vision transformer. Similar to the widely used identity shortcuts. the augmented shortcuts do depend on the specific architecture design either, which can be flexibly embedded into various variants of vision transformers (e.g., VIT 19) T2T 40). PVT 377) for enhancing their performance on different tasks such image classification, object detection and transfer learning. In the future, we plan to research designing deeper vision transformers with the help augmented shortcuts.
Acknowledgment
This work is supported by National Natural Science Foundation of China under Grant No. 61876007, and Australian Research Council under Project DE180101438 and DP210101859.
References
[1] David Balduzzi, Marcus Frean, Lennox Leary, JP Lewis, Kurt Wan-Duo Ma, and Brian McWilliams. The shattered gradients problem: If resnets the answer, then what the question? In International Conference on Machine Learning, pages 342-350. PMLR, 2017.
[2] E Oran Brigham. The fast Fourier transform and its applications. Prentice- Prenticc-Hall, Inc., 1988.
[3] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European Conference on Computer Vision, pages 213 219. Springer, 2020.
[4] Hanting Chen, Yunhe Wang. Tianyu Guo, Chang Xu, Yiping Deng. Zhenhua Liu, Siwei Ma. Chunjing Xu, Chao Xu, and Wen Gao. Pre- Pre-trrined image processing transformer. arXiv preprint arXi::2012.0036 2020.
10

---

[5] Ekin D Cubuk. Barret Zoph. Jonathon Shlens, and Quoe V Le. Randaugment: Practical automated data augmentation with reduced search space. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition Workshops, pages 702-703, 2020.
[6] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image datahase. In 2009 IEFE conference on computer vision and pattern recognition, pages 248-255. Iece, 2009.
[7] Claude Dietrich and Garry N Newsam. Fast and exact simulation of stationary gaussian processes through circulant embedding of the covariance matrix. SIAM Journal on Scientific Computing. 18(4):1088 1107, 1997.
[8] Yihe Dong, Lean-Baptiste Cordonnier, and Andreas Loukas, Attention is not all you need: Pure attention loses rank doubly exponentially with depth. arXiv preprint arXiv:2103.034004 2021.
[9] Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 2020.
[10] Rohert M Gray. Toeplitz and circulant matrices: A review. 2006.
[11] Klaus Greff, Rupesh K Srivastava, Jan Koutník, Bas R Steunebrink, and Jürgen Schmidhuber. Lstm: A search space odyssey. IEEE transactions neural networks and learning systems, 28(10):222 2016.
[12] Jianyuan Guo, Kai Han, Han Wu, Chang Xu, Yehui Tang, Chunjing Xu, and Yunhe Wang. Cmt: Convolu- tional neural networks meet vision transformers. arXiv preprint arXiv:2107.06263 2021.
[13] Kai Han, Yunhe Wang, Hanting Chen, Xinghao Chen, Jianyuan Guo. Zhenhua Liu, Yehui Tang, An Xiao, Chunjing Xu, Yixing Xu, et al. A survey visual transformer. arXiv preprint 2xXi:2002..12556 2020.
[14] Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, and Yunhe Wang. Transformer in transformer. arXiv preprint (xXix:1003.00110 2021.
[15] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 770-778, 2016.
[16] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference computer vision, pages 630-645. Springer. 2016.
[17] Aicke Hinrichs and Jan Vybíral. lahnson-lindenstrauss lemma for circulant matrices. Random Structures & Algorithms, 39(3):391-398, 2011.
[18] Elad Hoffer, Tal Ben-Nun, Itay Hubara, Niv Giladi, Torsten Hoefler, and Daniel Soudry. Augment your batch: Improving generalization through instance repetition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 8129-8138, 2020.
[19] Irwin Kra and Santiago R Simanca. On circulant matrices. Notices of the AMS, 59(3):36-3377, 2012.
[20] Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
[21] Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Fractalnet: Ultra-deep neural networks without residuals. arXiv preprint arXiv:1605.07648. 2016.
[22] Tianyi Liu, Minshuo Chen, Mo Zhou, Simon S Du. Enlu Zhou, and Tuo Zhao. Towards understanding the importance of shortcut connections in residual networks. In Advances in Neural Information Processing Systems, volume 32, 2019.
[23] Zhenhua Liu, Yunhe Wang, Kai Han, Siwei Ma, and Wen Gao. Post-training quantization for vision transformer. arXiv preprint arXiv:2106.1413 14156, 2021.
[24] Ilya Loshchilov and Frank Hutter. 17Xiv:1711.05101, 2017.
Decoupled weight decay regularization. arXiv preprint
[25] Maria-Flena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, pages 722-729. IEEE, 2008.
[26] Henri J Nussbaumer. The fast fourier transform. In Fast Fourier Transform and Convolution Algorithms. pages 111. Springer, 1981.

---

[27] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference compuder vision and paltern recognition, pages 3498-3505. IECE, 2012.
[28]
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary Ve Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
[29] David Silver, Thomas Hubert, Julian Schrittwieser, loannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel. al. Mastering chess and shogi by self-play with general reinforcement learning algorithm. arXiv preprint arXiv:1712.00
2017.
[30] Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojua. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and paitern recognition, pages 2818-2826, 2016.
[31] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning, pages 6105-6114, 14. PMLR, 2019.
[32] Yehui Tang, Kai Han, Yunhe Wang. Chang Xu, Jianyuan Guo, Chao Xu, and Dacheng Tao. Patch slimming for efficient vision transformers. arXiv preprint arXiv:2106.02833 2021.
[33] Yehui Tang, Yunhe Wang, Yixing Xu, Hanting Chen, Boxin Shi, Chao Xu, Chunjing Xu, Qi Tian, and Chang Xu. A supe-super assessor of neural architectures. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 1810-1819, 2020.
[34] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou, Training data-efficient image transformers & distillation through attention. arXiv preprint 1mXiv:2012.12877. 2020.
[35] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. arXiv preprint arXiv:1706. 17060377. 2017.
[36] Andreas Veit, Michael Wilber, and Serge Belongie. Residual networks behave like ensembles of relatively shallow networks. arXiv preprint arXiv:1605.06433 06431, 2016.
[37] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transforme: versatile backbone for dense prediction without convolutions. arXiv preprint arXiv:2102.12122 2021.
[38] Yixing Xu, Yunhe Wang. Kai Han, Yehui Tang, Shangling Jui, Chunjing Xu, and Chang Xu. Renas: Relativistic evaluation neural architecture search. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2021.
[39] Yiding Yang, Jiayan Qiu, Mingli Song. Dacheng Tao, and Xinchao Wang. Distilling knowledge from graph convolutional networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 7074-7083, 2020.
[40] Li Yuan, Yunpeng Chen, Tao Wang. Weihao Yu, Yujun Shi, Francis EH Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-toker vit: Training vision transformers from scratch on imagenet. arXiv preprint arXiv:2101.11986, 2021.
[41] Sangdoo Yun, Dongyoon Han, Scong Joon Oh, Sanghyuk Chun, Junsuk Choc, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6023 0-336002, 2019.
[42] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv: 1710.09412, 2017.
[43] Mingjian Zhu, Kai Han, Yehui Tang, and Yunhe Wang. Visual transformer pruning. arXiv preprint arXiv:2104.0 08500, 2021.
12


Reviews:
[{"review_id": "rHKbnaSHYcd", "reviewer": "Reviewer_E6ty", "summary": "This paper introduces a simple, effective and efficient way to address the feature collapse issue in current Vision Transformer models, which is add multiple parametrized shortcut paths in parallel to the original identity shortcut path, with an efficient approximation implementation. Experiments show the proposed method consistently improves over the baseline and the authors show transfer to detection task as well. ", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nThe idea to use multiple parameterized shortcut to reduce the feature collapse is straightforward and effective, shown both theoretically and empirically. Actually, it is a bit supervising to see it works so well, e.g. 1.0% improvement on ImageNet by simply adding more shortcuts. \n\nThe paper is extremely well structured, and easy to follow. The experiments are extensive and ablation studies are comprehensive to understand the proposed method.\n\nThe paper proposes to add additional shortcut paths, I’m wondering what’s the performance like if simply replace the original identify shortcut to parameterized one (1x1 shortcut)? \n\nThere is a value projection inside the attention calculation, what if we simply use this projection output as augmented shortcut, what would the performance be like? in this way, no extra params would be introduced, and we get a parametrized shortcut as well. \n\n---------------Post Rebuttal Feedback------------- \n\nThe authors' response make sense to me and solved my questions. I'll keep my rating. \n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have properly addressed the limitations. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "cXG6_kmhSx", "reviewer": "Reviewer_UjMK", "summary": "This paper alleviates feature collapse in vision transformers by introducing an efficient module, named augmented shortcut. As shown in the theoretical and empirical results, the feature diversity is enhanced and obvious performance improvement is achieved. The authors conduct experiments with various SOTA vision transformer models and validate the effectiveness of the proposed augmented shortcuts. ", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Positive:\n\nThe proposed method is interesting and has a clear motivation. The authors observe that massive features in vision transformers are very similar, which may restrict their representation capacity. It is interesting to solve the issue by improving the shortcuts. Multiple paths with parameters have stronger power to preserve the feature diversity. \n\nThe block-wise circulant projection makes the augmented shortcut can be calculated very efficiently. The extra computation cost brought by the augmented shortcuts can be neglected. Thus the augmented shortcut can be flexibly implemented various models, just as the identity shortcut. \n\nExtensive experiments are conducted with various models (including ViT, PVT, T2T-ViT) on different mainstream tasks. For the classification task on ImageNet, the augmented shortcuts improve the performance by 1% top-1 accuracy, which is significant. For the detection task, all the metrics are improved. These experimental results validate the superiority of the proposed method.  \n\nThe components and hyper-parameters of the proposed method have been well analyzed with the ablation studies. It is glad to see that the proposed method is robust to these hyper-parameters. \n\nNegative:\n\nMaking more discussions about the experimental results in Table 2 will make the paper stronger. For example, it is interesting to see that the augmented shortcut can improve the performance of the deeper model more significantly. Could the authors make more analyses about the reason behind it?\n\nThe captions in Figure 3 are too small to read. The authors are required to refine them.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "These aspects have been well discussed.", "review_text": "", "extra_scores": {"ethical_concerns": "These aspects have been well discussed.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hours"}, "extra_text": {}}, {"review_id": "9dL3Tbj8FH1", "reviewer": "Reviewer_KRcF", "summary": "This paper observes the problem of feature collapse in transformer, where the tokens in deep layers collapse to the same representation. It then proposes to add additional augmented shortcuts in parallel to the original shortcut connection in the multihead self-attention modules and MLP modules, based on the observation that simple identity shortcut alleviates the problem of feature collapse. In the experiments, the proposed method brings 1.1% performance gain in the ImageNet classification task compared to the baseline model. Performance gains can also be seen from COCO detection task and other downstream finegrained classification tasks, as compared to the baseline models.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In this paper, a simple add-on approach is proposed to encourage the feature diversity in deep transformer layers. It introduces additional shortcut connections, comprising of a linear layer and an activation non-linearity, on top of the existing shortcuts. The feature diversity is measured by the matrix norm of the difference with the best rank-1 approximation matrix to the feature matrix. This diversity measurement is improved compared to both the setting without any shortcut or with identity shortcut(Figure 3 and Figure 4).\n\nCirculant martices are used as the weights of the linear layers introduced in the additional shortcuts to save computation and storage. As compared to using the full randomly initialized weight matrices, this approach uses roughly 1/200 the storage and 1/20 the G-FLOPs, with minimum performance drop.\n\nThe main modification centers around the shortcut connections in transformer architecture. Therefore it would be more convincing if more ablations could be done around the theme of shortcuts. Specifically, how does the choices of stochastic depth/dropout parameters influence the performance? How does the architecture of the shortcuts(e.g. multiple linear layers, choices of non-linearity, etc.) affects the diversity of the token feature outputs?\n\nMoreover, the paper observes the decrease in feature diversity as the depth of the model increases, and that adding more shortcut connections can effectively solve this issue. An experiment with deeper transformer model, such as the CaiT family, is therefore much sought for. It would be more convincing if the paper can show improved diversity in token features of deeper transformer model.\n\n------------------------------------------------Post Rebuttal--------------------------------------------------------------------------\n\nThe responses have addressed the issues listed in the previous comment, so I will  I will increase the final score to 6.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "See above. The societal impact is shown one the last page of the manuscript.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3.5 hours"}, "extra_text": {}}, {"review_id": "46yB4II4sw7", "reviewer": "Reviewer_pv6P", "summary": "This paper introduce a universal module to improve vision transformers. They show that vision transformers suffer feature collapse, which mean that features in deep layers will become similar. To address, this paper presents an augmented shortcut module to  be flexibly embedded into various transformer models. Extensive experiments are conducted on multiple tasks (image classification, object detection, and transfer learning), which show that augmented shortcuts can bring obvious performance improvements. That is a good paper to me because of the new and effective module which is a new investigation to the field.", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n+ The proposed method is novel and reasonable and extend the vision transformer from a new viewpoint and new investigation into the it. The augmented shortcuts can effectively alleviate feature collapse and enhance the representational ability, extensively validated both theoretically and empirically. The introduced block-wise circulant matrix can effectively improve computational efficiency.\n\n+ The proposed augmented shortcut module has favorable flexibility and can be implemented on different transformer models for various tasks. Their module is generic and can be compatible with different transformer architectures on various models (e.g., ViT and SOTA variants PVT, T2T), which show obvious accuracy improvements. The commonality of the augmented shortcut make it have a lot of application scenarios.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "- Figure 1 illustrates how to equip the MSA module with the proposed augmented shortcuts. What do patterns (rectangle, triangle, etc.) denote? Do the patterns with different colors denote different features from various patches? It will be more intuitive to add more detailed captions.\n\n- The parameter b in block-wise circulant matrix can affect its computational cost. How do you determine its value? Is the parameter b sensitive to different architectures of vision transformers?  \n\n[1] Deep Residual Learning for Image Recognition, CVPR 2016\n[2] Densely Connected Convolutional Networks, CVPR 2017 \n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "11"}, "extra_text": {}}, {"review_id": "1HIdo7jYc50", "reviewer": "Reviewer_1wm5", "summary": "This work proposes to boost the performance of vision transformers from shortcuts. As analyzed, the feature diversity of tokens declines as the layers go deeper while the shortcut connection provide a fix towards to potential ``feature collapse'' issue. To this end, additional branches of augmented shortcut is proposed to further improve the feature diversity as well as the final recognition power.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- The authors propose to augment the shortcuts in vision transformers. This technique is independent from the specific architecture designs, being able to be plugged into different vision transformers. \n\n- The demonstration of feature diversity is clear and interesting. The motivation of augmenting the shortcut is well explained through the figures and tables in Section 2. \n\n- This paper is well-written. All detailed techniques are clearly clarified. Extensive experiments are conducted to validate the significance of the proposed simple module.\n\n- Weakness and Limitation: see next section.\n\n---------------Post Rebuttal Feedback-------------\nAfter reading the authors' rebuttal, some of my concerns are solved. Thus I increase my rating to 6.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations:\n- The motivation of this work is straightforward, from the observation that the rank, i.e. diversity, is lost when layers go deeper. However, this phenomenon does not appear merely in vision transformers. Instead, this issue has also be explored in modern CNNs structures. Based on this observation, lots of progress has been made, such as Channel Pruning [1], compression [2] and semantic segmentation [3]. Meanwhile, the lost-rank phenomenon of self-attention layer is originally investigated in [4], which this work is based on. Thus the contribution of this paper might not be adequate. \n\n- The technical novelty of the proposed module is also limited. There have been some prior works that transform/enhance the shortcut pathway. It's quite obvious that adding a re-parameterized shortcut branch is able to improve the performance. Furthermore, another concern is about the memory cost brought by additional shortcuts.\n\n[1] Channel Pruning for Accelerating Very Deep Neural Networks. ICCV 2017\n\n[2] On compressing deep models by low rank and sparse decomposition. CVPR 2017\n\n[3] Tensor Low-Rank Reconstruction for Semantic Segmentation. ECCV 2020\n\n[4] Attention is not all you need: Pure attention loses rank doubly exponentially with depth. arxiv preprint", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper got mixed ratings initially. All the reviewers agree with the motivation and effectiveness of the proposed method, but they also raise the concerns on the novelty and technique contributions of this work. In particular, the feature collapse issue has been identified by a prior work [8]. The theorem listed in the paper was not developed by this work either. The reviewers also suggested to add more ablation experiments on this work. After the response, the authors address these concerns and all the reviewers give positive ratings. AC has read the submission, reviews and authors' response. Although the finding of feature collapse is not firstly identified by this work, the proposed shortcut augmentation is simple and the performance improvement is significant. Thus, AC agrees with reviewers that this work would be valuable for the community and recommend acceptance.  
Decision: Accept (Poster)