Anti-Oversmoothing in Deep Vision Transformers via the Fourier Domain Analysis: From Theory to Practice
Authors: Peihao Wang
Wenqing Zheng
Tianlong Chen
Zhangyang Wang
Venue: ICLR 2022 Poster (2022)
Keywords: Deep ViT
Spectral Analysis
Attention Collapse
Patch Diversity

Abstract: Vision Transformer (ViT) has recently demonstrated promise in computer vision problems. However, unlike Convolutional Neural Networks (CNN), it is known that the performance of ViT saturates quickly with depth increasing, due to the observed attention collapse or patch uniformity. Despite a couple of empirical solutions, a rigorous framework studying on this scalability issue remains elusive. In this paper, we first establish a  rigorous theory framework to analyze ViT features from the Fourier spectrum domain. We show that the self-attention mechanism inherently amounts to a low-pass filter, which indicates when ViT scales up its depth, excessive low-pass filtering will cause feature maps to only preserve their Direct-Current (DC) component. We then propose two straightforward yet effective techniques to mitigate the undesirable low-pass limitation. The first technique, termed AttnScale, decomposes a self-attention block into low-pass and high-pass components, then rescales and combines these two filters to produce an all-pass self-attention matrix. The second technique, termed FeatScale, re-weights feature maps on separate frequency bands to amplify the high-frequency signals. Both techniques are efficient and hyperparameter-free, while effectively overcoming relevant ViT training artifacts such as attention collapse and patch uniformity. By seamlessly plugging in our techniques to multiple ViT variants, we demonstrate that they consistently help ViTs benefit from deeper architectures, bringing up to 1.1% performance gains "for free" (e.g., with little parameter overhead). We publicly release our codes and pre-trained models at https://github.com/VITA-Group/ViT-Anti-Oversmoothing.

Published as a conference paper at ICLR 2022
ANTI-OVERSMOOTHING IN DEEP VISION TRANS- FORMERS VIA THE FOURIER DOMAIN ANALYSIS: FROM THEORY TO PRACTICE
Peihao Wang, Wenqing Zheng, Tianlong Chen & Zhangyang Wang Department of Electrical and Computer Engineering, The University of Texas at Austin (peihaowang, w. zheng, t tianlong... chen, atlaswang] @utexas edu
ABSTRACT
Vision Transformer (ViT) has recently demonstrated promise in computer vision problems. However, unlike Convolutional Neural Networks (CNN), it is known that the performance of ViT saturates quickly with depth increasing, due to the observed attention collapse or patch uniformity. Despite a couple of empirical solutions, rigorous framework studying this scalability issue remains clu- sive. In this paper, we first establish a rigorous theory framework to analyze VIT features from the Fourier spectrum domain. We show that the self-attention mechanism inherently amounts to a low-pass filter, which indicates when ViT scales up its depth, excessive low-pass filtering will cause feature maps to only preserve their Direct-Current (DC) component. We then propose two straight- forward yet effective techniques mitigate the undesirable low-pass limitation. The first technique, termed AttnScale, decomposes a self-attention block into low-pass and high-pass components, then rescales and combines these two fil- ters to produce an all-pass self-attention matrix. The second technique, termed FeatScale, re-weights feature maps on separate frequency bands to amplify the high-frequency signals. Both techniques are efficient and hyperparameter-free while effectively overcoming relevant ViT training artifacts such as attention col- lapse and patch uniformity. By seamlessly plugging in our techniques to multiple ViT variants, we demonstrate that they consistently help ViTs benefit from deeper architectures, bringing up to 1.1 % performance gains "for free" (e.g., with little parameter overhead). We publicly release our codes and pre-trained models at attps://githus: www.//m//ic/c.. /VITA http://VIA-GGoupp///a /iT Ant weatttivverrmoottooo
INTRODUCTION
Transformers have achieved phenomenal success in Natural Language Processing (NLP) /Vaswani
ct al. [017]/Devii et al. 20188 | Dai et al. 2019 Brown et al. 2020). and recently a wide range
of computer vision applications too (Dosovitskiy et al. 2020 Liu et al. 2021 Arnab al.) 2021 1
Carion et al. 2020 Jiang et al., 202Ta). One representative advance, the Vision Transformer (VIT) (Dosovitskiy al. 2020), stacks Multi-head Self-Attention (MSA) blocks, by treating each local image patch as semantic tokens and modeling their interactions globally. Unlike Convolutional
Neural Networks (CNNs) that hierarchically enlarge the receptive from local global, even shallow ViT is able to effectively capture the global contexts, leading to their very competitive performance on image classification and other tasks (Liu al. 2021 Jiang et al. 2021a).
Going deep has always been a trend in deep learning (LeCun et al. 2015 Krizhevsky et al. 201 and VIT was expected to make no exception. One might reasonably conjecture that deeper ViT with more MSA blocks significantly outperform its shallower baseline. Unfortunately, building deeper ViTs face practical challenges. Empirically, Zhou et al. (2021a) shows a vanilla VIT of 32 layers under-performs the 24-layer one. Gong al. (2021) demonstrates downgraded patch diversity in deeper layers, and Dong et al. (2021) mathematically reveals the rank collapse phenomenon when Transformer goes deeper. Despite efforts towards deep ViT through patch diversification (Gong ct al. 2021] Zhou et al. (2211)) rank collapse alleviation (Zhou et al., 202 la Zhang et al. 2021). and training stabilization (Touvron al. 202Tb Zhang al. (019). most of them are restricted to

---

We begin by introducing our notations. Let X E RRad denote the feature matrix, where n the number of samples, and d is the feature dimension. Let x; E Hd Vi 1,... n. the i-th row of X, denote the feature vector of the i-th sample, and æ; E R", Vj = 1, ,d, the j-th column of X, represent signals of the j-th channel. In the context of ViT, X denotes set (sequence) of image Published as a conference paper at ICLR 2022
empirical studies. Rethinking the problem with deep ViT from more principled angle pends further efforts.
In this paper, present the first nigorous analysis of stacking self-attention mechanism in the Fourier space, We mathematically show that cascading self-attention blocks is equivalent to repeatedly applying a low-pass filter, regardless of the input key query tokens (Section 2.2) As consequence, going deeper with vanilla ViT blocks only preserves Direct Component (DC) of the signal at the output layer. This theoretical finding explains the observations of patch uniformity and rank collapse, and is also inherently related to the over-smoothing phenomenon in Graph Convolutional Networks (GCNs) (Kipf Welling 2017: IT & Machara [02T] Oono & Suzuki 2019 Cai & Wang 2020). Moreover, we also reveal the role of other transformer modules (e.g., MLP and residual connection)
in preventing this undesirable low-pass filtering (Section 2.3
Built on the aforementioned analysis framework in the Fourier domain, we propose two novel techniques, to mitigate the low-pass filtering effect of self-attention and effectively scale up the depth of ViTs. The first technique, termed Attention Scaling (AttnScale). directly manipulates on the calculated attention map to enforce an all-pass filter (Section 3.T). It decomposes the self-attention matrix into a low-pass filter plus a high-pass filter, then adopts learnable weight to adaptively amplify the effect of high-pass filter. The second technique, termed Feature Scaling (FeatScale), hinges feature maps re-weight different frequency bands separately (Section 3.2) It employs trainable coefficients to re-mix the DC and high-frequency components, hence selectively enhancing the high-frequency portion of the MSA output. Both AttnScale and FeatScale extremely memory and computationally friendly. Neither runs Fourier transformation explicitly, bringing little extra complexity to the original ViTs.
Our contributions can be summarized as follows:
We establish the first rigorous theoretical analysis of ViT from the spectral domain. We characterize the low-pass filtering effect of cascading MSAs, which connects the recent empirical findings of ViT patch diversity loss or rank collapse.
We present two theoretically grounded Fourier-domain scaling techniques, named AttnScale and FeatScale. They operating on re-adjusting the low- and nighfrreuuency components of the attention maps and feature maps, respectively. Both are efficient, hyperparameter-fieee easy-to-use, and able to generalize across different ViT variants.
We conduct extensive experiments by integrating AttnScale and FeatScale with different ViT backbones. Both of our approaches substantially boost DeiT, CaiT, and Swin- Transformer with up to 1.1%, 0.6% and 0.5% performance gains, without whistles and bells.
2 WHY VIT CANNOT Go DEEPER?
2.1 NOTATION AND PRELIMINARIES
patches, x; denotes the flatten version the i-th patch embedding (d patch width x patch height), and zj denotes whole image signal of the j-th channel.
Transformer Architecture Vision Transformer (ViT) consists of three main components: a patch embedding and position encoding part, stack of transformer encoder block with Self-Attention (MSA) and Feed-Forward Network (FFN), and a score readout function for image classification. We depict a transformer block in Fig. 3a The key ingredient here is the Self-Attenti Attention (SA) module, which takes in the token representation of the last layer, and encodes each image token by aggregating information from other patches with respect to the computed attention value, formulated as below /Vaswani et al. 2017):
XWo XWo(XWR) XWv,
SA(X) softmax
Vd
2

---

Published as a conference paper at ICLR 2022
MHA
MHA
MHA
Log Relative Distance
Actual Theorem
Actual Theorem
-5
Actual
-5
Theorem
-10
-15
-15
12
12
12
Layer Index
Layer Index
Layer Index
Figure 1: Visualize the intensity of high-frequency component and their theoretical upper bounds under different transformer blocks. The blue line is defined by log HC [x,]] [X pr) and the red line is estimated using the results in Section 2.2
2.3
See details in Appendix
where WK E Rdxd, Wi E Rdxd, Wv E Rdxd are the key, query, and value weight matrices, respectively, v here denotes a scaling factor, and softmax(-) operates on X row-wisely. Multi-Head Self-Attention (MSA) involves a group of SA heads and combines their outputs through a linear projection Vaswani et al., 2017):
(2) where the subscripts denote the SA head number, // is the total number of SA heads, and Wo IR dddd projects multi-head ouiputs to the hidden dimension. Besides MSA module, each transformer block is equipped with a normalization layer, forw-forward network, and skip connections to cooperate with MSA. Formally, a transformer block can be written as follows:
(SA(X) = (X)
SAn(X)) Wo:
X' = MSA(LayyrNorm (X)) + X,
(3)
Y = FFN (X))) + X'.
(4)
Fourier Analysis The main analytic tool in this paper is Fourier transform. Denote F : R" C" be the Discrete Fourier Transform (DFT) with the Inverse Discrete Fourier Transform (IFT) F-1: C" + RT. Applying F to a flatten image signal
equivalent to left multiplying DFT
matrix, whose rows are the Fourier basis fx = [=2m)(k-1)00 where denotes the k-th row of DFT matrix, and j is the imaginary unit. Let ă
e2mj(k- (n(((--))(nn1))) / E R" Fz be the
spectrum of 2, and Zdc. E C. hhc E Ca-1 take the first element and the rest elements of z, respectively. Define DC [z] = Eiefi E C as the Direct-Current (DC) component of signal z, and HC [=]= f2
fn] Tne E C" the complementary higu-freyuency component.
In signal processing, low-pass filter is a system that suppresses the high-frequency component of signals and retain the low- fow-frequency component. In this paper, we refer to low-pass filter particular type of filters that only preserve the DC component, while diminishing the remaining fighffrequency component. To be more precise, we define low low-pass filters in Definition Definition 1. Given an endomorphism f : + R" with f denoting applying f fort times, f is a low-pass filter if and only for all E R":
|HC (x))ll DC (=)()]] = 0.
lim
(5)
Definition D reveals the nature of low-pass filters: they will produce a dominant response on DC component, while imposing an inhibition effect on the high-frequency band. We refer interested readers to Appendix A for more useful backgrounds.
2.2 SELF-ATTENTION IS A LOW-PASS FILTER
In this subsection, we will give theoretical justification on self-attention terms its spectral-domain effect. Our main result is that self-attention is constantly a low-pass filter, which continuously erases high-frequency information, thus causing ViT to lose features expressiveness at deep layers.
Formally, we have the following theorem that shows attention matrix produced by softmax function (c.g. Eqn. D is a low-pass filter independent of the input token features or key/query matrices.
Theorem 1. (SA matrix is low-pass filter) Let A = softmax P), where P E RRax,, Then A must
be a low-pass filter. For all z E R", lim HC [A' xx] / DC
[A' 2122|22 2 = 0.
3

---

Published as a conference paper at ICLR 2022
Theorem I is straightforward result of Ferron-Frobenius theorem. See Appendix [8.1 for a proof. Theorem allso reveals that no matter how attention is computed inside the softmax function, including dot product (Vaswani et al.
Layer 4
Layer 8
Magnitude
Layer
2017), linear combination (Veličković et al.] 2018), 1.2
distance (Kim et al., 2021). the resulting attention matrix always low-pass filter. One can see consecutively applying self-attention matrix simulates the process of ViT's forward propagation. As the layer number increases infinitely, the final output will only keep the DC bias, and ViT loses all the feature expressive power.
100
-50
50
100
Frequency
Figure 2: Visualize the spectral response of attention map. We randomly pick a sample and depict its first head of 4/8/12-th layer. Refer to Appendix F2
Corollary 2. Let PI, P2....... Pn be sequence of ma- tix in n xn, and each Px. vk = 1, 1.... LL has A₁ = softmax Pa). Then II 1 Aş is also low-pass filter.
In fact, ViT re-computes self-attention matrices per layer, which seems avoid the consecutive power of an identical self-attention matrix. However, we also provide Corollary 2 which suggests even the VIT consists of distinctive self-attention matrix at each layer, their composition turns out to act like low-pass filter as well. We also visualize the spectrum of attention maps (Fig. 2 and more on Appendix F22 to support our theoretical conclusions.
Knowing that self-attention matrices amount to low-pass filters, we are also interested in to which extent an MSA layer would suppress the high-frequency component. Thereby, we also provide a convergence rate to illustrate this speed that the figh-frreqence component are being annihilated.
Theorem 3. (smoothening rate of SA) Let A = softmax P) and a max Payl. where P E Raxn, Define SA(X) AXW, Wv the output of self-attention module, then
20
I/CC |HC X S
Wv |W [X]][-
(6)
2a + n
In particular, when P = XW X /dd and assume tokens distributed inside ball with radius 7 i.e., || || S 7, Vi 1,... n, then a S W W 1 Vd
The proof of Theorem 3 can be found in Appendix
B.3
Theorem
3 says the high-frequency
intensity ratio to the pre- and post- attention aggregation is upper bounded by WV lly
neRe
2 4n-
< 1, HC [SA(X)] converges to zero exponentially. We note that, When WV 4n--11
ne
matter how attention is computed or signals are initialized, since V 62 ---1111 bounded by Vn,
|Wvll2 < 1/vn will definitely cause monotonically decreasing high-frequency component. When dot-product attention is adopted, sufficient condition that || [X]]| decreases to zero within logarithmic time is >² W| W / vd + log/ Wv I log 1/.
2.3 EXISTING MECHANISMS THAT COUNTERACT LOw-PASS FILTERING
In this section, we take other VIT building blocks into consideration. We will justify whether Multi- Head Self-Attention (MSA), Feed-Forward Network (FFN), and residual connections can effectively alleviate the low-pass filtering drawbacks. All the derivations follow from Theorem 3 and some proof ideas are borrowed from Dong et al. (022I) We further present Fig. to justify our results.
Does multi-head help? MSA employs weights to combine the results of multiple self-attention blocks. We can rewrite it as MSA(X) D" SA(X) We show by Proposition in Appendix C.I that the convergence rate turns to 010211
2 42-1 where Hl the number of heads, One can see MSA can only slow down the
ne2a
σ1 max W and 02 = max W/ convergence up constant ogHl, which does not root out the problem.
Does residual connection benefit? In addition to MSA, a transformer block also leverages a skip connection, which can be formulated Res X) MSA (ISA(X)) +X. X We show that residual connection
4

---

Published as a conference paper at ICLR 2022
can effectively prevent high- freeueeyyy component from diminishing to by promoting the rate
0102l1y
to 1 + 010211 +n+
1. Refer to Proposition 5 in Appendix C.2,
A feed-forward network is appended to MSA module. We Our Proposition Suggests that a FFN with Lipschitz constant convergence rate, which does not improve the original one. However, if skip connection adopted over FFN, 0g can guarantee the upper bound of the high-frequency component is non-contractive.
Does FEN make any difference? characterize its effect in Appendix C.3
33 contributes a 03 1 + H
ne
620
+
Although multi-head, FFN, and skip connection all help preserve the high-frequency signals, none would change the fact that MSA block as a whole only possesses the representational power of low-pass filters. Our Proposition 4 6 states multi-head, FFN, skip connections can only slow down the convergence by indistinguishably amplifying low- and frggh-freuency components with the same factor. However, since they are incapable of promoting figh-frreuency information separately, is inevitable that high-freques frequency components are continuously diluted as ViT goes deeper. This restricts the expressiveness of ViT, resulting in the performance saturation in deeper ViT.
2.4 CONNECTIONS TO EXISTING THEORETIC UNDERSTANDING WORKS
It is known that Graph Convolutional Networks (GCN) are not more than low-pass filters NT & Maehara 2021). In the meanwhile, Oono & Suzuki 2019); Cai & Wang (2020) pointed out GCN's node Teatures will exponentially trapped into the nullspace of the graph Laplacian matrix. Similarly, our work concludes that self- attention module is yet another low-pass pass filter. Combining with our theoretical derivation, one can see the root reason is that both graph Laplacian matrices and self-attention matrices consistently own fixed leading eigenvector, namely the DC basis. This makes aggregating information via such matrices inherently project the token representation onto these invariant eigenspaces. And we note that smotthno rank collapse, and patch uniformity are all the manifestation of excessive low-pass filtering. See Appendix D.I for more discussion.
In Dong et al. (021), the authors proved that ViT's feature maps will doubly exponentially collapses to a rank-1 I matrix, which reveals ViT loses feature expressiveness at deep layers. Besides, they also gave systematic study on other building blocks of transformer. While they share the similar insights with us, our work further specifies which rank-1 matrix the feature activation will converge to, namely the subspace spanned by the DC basis. That makes theory to be better grounded with peene--pooesssing and geometric interpretations, via directly measuring the intensity of the high- frequency residual, instead of examining a composite norm distance an agnostic rank-1 I matrix. Although Dong et al. (2021) presented a faster convergence speed, we respectfully suggest that the current proof of [Dog et al. (2021) might be deficient, or at least incomplete in the assumptions (see Appendix (D.2). Moreover, our theory can be generalized to other attention mechanisms such as logistic attention (Veličković et al., 20188) and L2 distance (Kim et al. 2021 See Appendix DD.
3 ATTNSCALE & FEATSCALE: SCALING FROM THE FOURIER-DOMANT
3.1 ATTNSCALE: MAKE ATTENTION AN ALL-PASS FILTER
As we discussed in Section 2.2 self-attention matrix can only perform low-pass filtering, which narrows the filter space ViT can express. Inspired by this, we propose scaling techniques directly manipulating the attention map, termed Attention Scaling (AttnScale), to balance the effects of low- and high-past pass filtering and produce all- filters. AttnScale decomposes the self-attention matrix to a low-pass filter plus high-pass filter, and introduces a trainable parameter to rescale the high-pass filter to match the magnitude with the low-pass component.
Formally, let A denote self-attention matrix. To decompose low-pass filter from A, we find the largest possible low-pass filter that can be extracted from A. We use Lemma in Appendix D to justify our solution. By Lemma 8 we can simply extract L = F diag(1, 0, .0FF= 11 from A and take the complementary part as the high-pass filter. Afterward, we can rescale the high-pas -ass component of the filter, and combine low-pan -w-pass and high-pass together to form new self-attention matrix. We illustrate this scaling trick in Fig, [bb To be more precise, for the 1-th layer and h-th head, we recompute the self- self-attention map as follows:
5

---

Published as a conference paper at ICLR 2022
cosine Similority
Deil-S Deft.5 AttnScale
Ap
Xat
21
24
Layer Index
Mall--Baa
Cosine simtarity
Dels5
---
Dell-S FeatScale
8.4
MaeMul
Layer 15
21
24
(a)
(b)
Figure 3: Illustration of our proposed techniques. (a) recalls the standard ViT block. (b) and (c) illustrate our proposed AttnScale and FeatScale, which scaling high-pass filter com- ponent and figh-frequency signals, respectively.
Figure 4: Visualize cosine similar- ity of attention and feature maps with/without our proposed methods. Refer to Appendix F33 for details.
A (1,b)
LP -11T
1
A (1h)
HP A (L.A)
A
(L.N) LP
(8)
A LP (Wi, + HP
(1.A)
(L.N)
where WI,A is trainable parameter, and different layers and heads adopt separate WI,h- During training time, wi,s's 's are initialized with 0, and jointly tuned with the other network parameters. By adjusting wt. (1.h) can simulate any type of filters: low-pass, high-pass, band- pass, or all-pass. Note that our AttnScale is extremely lightweight, as it only brings O(HL) extra parameters, where H is the number of heads, and the number of ViT blocks.
3.2 FEATSCALE: REWEIGHT HIGU--RRQQEENY SIGNALS
According to our analysis in Section 2.2 MSA module will indiscriminately suppress high- frequency signals, which leads to severe information loss. Even though residual connection can retrieve lost information through the skip path, the high-frequency portion will be inevitably diluted (Theorem D.. To this end, propose another scaling technique that operates feature maps, named Feature Scaling (FeatScale). FeatScale processes the output of MSA by mixing the information from varying frequency bands discriminatively. FeatScale first decomposes the resultant signals into their DC and high-frequency components. Then it introduces two groups of parameters to re-weight the two components for each channel, respectively. The pipeline of this scaling technique is depicted in Fig. 3c To be more precise, we re-weight the output of the 1-th MSA by
(1)
DC = DC (MSA(X)) (s: I),
(10)
X (
HC = HC [MSA(X)] (diag(ti) + I),
(11)
(1) x x HC.
(12)
where E Rd and t; E Rd are learnable parameters to perform channel-wise re-weighting. We initialize 81 and t with zeros and tune them with gradient descent. After adjusting the proportion of different frequency signals, FeatScale can prevent the dominance of the DC component. DC [-] and HC [:] are cheap to compute without explicit Fourier transform. Calculating DC [X] is as simple as running the column average of matrix X, and HC [X] can be efficiently computed by X DC [X].
3.3 DISCUSSION
We have proposed two methods to facilitate the deeper stacking of ViT MSA modules, and discussed their motivations and strengths from our perspective of filtering and signal processing. In this section, we will connect these two techniques with commonly mentioned problems with ViTs.
How does AttnScale prevent attention collapse? Deep ViT suffers from the attention collapse issue /hou et al. 2021a). When transformer goes deepr, the attention maps gradually become similar

---

Published as a conference paper at ICLR 2022
and even much the same after certain layers. Combining with our thoery, one see collapsed attention maps turn out to be pure low-pass filter, which wipes off all the high-frequency information in one shot. Zhou al. (2021a) la) proposed the re-attention trick, which blends attention map across different heads. By doing this, modified attention maps aggregate high-pass components from other heads and is endowed with richer filtering property. We note that our AttnScale is akin to a more lightweight re-attention mechanism with better interpretability. We rewrite the attention map as a sum of an already-collapsed attention (11" 2 /n) with the complementary residual map that encodes diverse patterns in self-attention matrix. By re-weighting the residual map, the diversified patterns can be amplified, which prevents it from degenerating to rank-1 matrix. We further verify this argument using cosine similarity metric (Zhou et al. 202 in the upper sub-figure of Fig. 4
How does FeatScale conserve patch diversity? Self-attention blocks tend to map different patches into similar latent representations, yielding information loss and performance degradation (Gong et al. 2021). By our theory, this asymptotic smoothness of feature map is caused by excessive low-pass filterin, and the remaining DC bias signifies uniform patch representations. Conventional approaches to addressing this problem include incorporating convolutional layers (Wu et al. 2021 Jiang et al., 2021b) and enforcing patch diversity regularizations (Gong et al.] (021). As diverse features are often characterized by high- nigh-frequency signals, our FeatScale instead elevating the righ-frequeccy component via a Icarnable scaling factor, can be regarded as a more straightforward way to reconstruct the patch richness. Compared with LayerScale (Touvron et al. 2021b), in which each frequency band is equally scaled, our FeatScale treating DC and high-frequency components differently, not only perform a per-channel normalization, but also perform spectral-domain calibration with high- frequency details and low-frequency characteristics. The lower sub-plot of Fig. shows ViT with our FeatScale has lower feature similarity at deep layer.
4 RELATED WORK
Transformers in Vision. Transformer (Vaswani al. 2017) entirely relies on self-attention mech-
anism to capture correlation and exchange information globally among the input. It has achieved a
remarkable performance in natural language processing (Devlin et al., 2018 Dai et al.,
2019 Brown 2020) and many cross-disciplinary applications (Jumper et al. 2021 Ying et al.
et al.
2021 Zheng
et al. T02 Recent advances have also successfully applied Transformer to computer vision tasks.
Dosovitskiy et al. (2020) first adopts pure transformer architecture (ViT) for image classification.
The follow-up works (Chen et al. (021b) extend ViT to various vision tasks, such as object detection
(Carion al. 2020 Zhu al. 20
al., 2021a Sun al. (020), segmentation (Chen Wang et al. (022)) ) image generation Parmarr et al. 2018: Jijng et al.] [ilT][22aaa. video
cet al. 202
processing (Zhou al.] [018 Arnab et al., 2021), and 3D instance processing (Guo al. 2021 Lin
et al. 2021 To capture multi-scale non-local contexts, Zhang et al. (2020) designs transformers
in lelel, top-down, and bottom-up interaction fashion. Liu al. (202)) presents hierarchical
ViTs with shifted window based attention that can efficiently extract multi-scale scale features. To dismiss
ViT from the heavy reliance on large-scale dataset pre-training, Touvron et al. (2021a): Yuan et al.)
(202T) propose knowledge distillation and progressive tokenization for data-efficient training. Despite impressive effectiveness, most of these model are only based on relatively shallow VIT backbones with dozen of MSA blocks.
Advances in deep ViTs, Building deeper ViT's has arisen many interests. Zhou et al. (022a)) first investigated the depth scalability of ViT. The authors found that the attention collapse hinders VIT from scaling up, and propose two methods to conquer this problem i) increasing the embedding dimension, and ii) a cross-head re-attention trick to regenerate attention map. A concurrent work Touvron et al. (b02 came up with LayerScale layer that performs per-channel multiplication for each residual block. More importantly, they make explicit separation of transformer layers involving self-attention between patches, from class-attention layers that are devoted to extract the global content into a single embedding to be decoded. Gong et al. (202T) further proposed a series of losses that can enforce patch diversity in VIT. Such regularizations include penalty on cosine similarity, patch-wise contrastive loss, and mixing loss. (Tan et al. (2021) presented a shorteut augmentation scheme with biock-cireulant projection improve feature diversity. Although these existing solutions manage to deepen ViTs, most of them are empirical works and bring no principled theory.
Role of depth in NNs. Discussing the importance of deep structures in Neural Networks (NNs) is an overly broad topic. Here we only focus subset of works that scaling up transformer could
7

---

Published as a conference paper at ICLR 2022
Table 1: Experimental evalutation of AttnScale & FeatScale plugged into DeiT and CaiT. The number
inside the ( represents the performance gain compared with the baseline model, and accuracies within/out of parenthesis are the reported/neroddeed performance.
Backbone
Method
Input size
Layer
# Param
FLOPs
Throughput
Top-1 Aec (%²
Deil-S
224
12
22.0M
4,37G
1589.4
(79.9
DeiT-S AttnScale
224
12
22.0M
4.57G
1416.7
80.7 (1 0.9)
Deil-S FeatScale
224
12
22.0M
4,57G
1309.9
80.9 (1 1.1)
DeiT
DeiT.S
224
9.09G
K16.4
DeîT-S AttnScale
224
24
43,3M
9.10G
722.0
81.1 0.6)
DeiT-S + FeatScale
224
43.4M
9.10G
772.5
81.3 0.8)
CaiF-S
224
46.9M
9,33G
371.9
(2.67
CarT
CaiT-S + AttnScale
224
24
46.9M
9.34G
339.0
83.2 0.6)
CaiT-S + FeatScale
224
46.9M
9.34G
358.2
83.2 0.6)
Swin-S
224
49.6M
8.74G
593.2
(8.0)
Swin
Swin-S AttnScale
224
24
49.6M
8,75G
553.4
83.4 (1 0.4)
Swin-S FeurScale
224
49.6M
8.75G
550.3
83,5 0.5)
relate to. For ordinal deep learning models, such as FFNs and CNNs, deep architecture immediately benefits from the universal approximation power and expressive capacity (Cybenko 1989 Hornik
1991 Telgarsky 2016 Lu et al. 2017; Petersen & Voigtlaender 2020 Zhou 2020). In contrast, several studies in graph learning domain have reported severe performance degradation due to over- smoothing when stacking many layers Kipf & Welling 2017 Wu et al. 2019 Li et al. 2018). The subsequent studies (NT & Maehara 2021 Oono & Suzuki 2019 Cai & Wang 2020) gave theoretical explanations of the over-smoothing phenomena from the views of graph signal filtering and feature dynamics. Likewise, VIT have been witnessed performance saturation when going deeper. However, to our best knowledge. [Dong et al. (222) is the sole work in the literature that systematically and rigorously analyzes this issue with deep VIT. The main idea of this work is showing the self-attention block will downgrade the rank of the feature maps. Our work takes step forward by studying ViT on spectral domain, and manages to reveal the signals will ultimately fall into the one-dimension DC subspace. We see our theory and techniques are also applicable NLP transformers. However, we only focus on ViT because empirical observations indicate NLP modeling (including Transformer) does not require deep structure /Vaswani et al. 2017 Brown et al. 2020). while vision tasks always demand one (LeCun et al. 2015).
EXPERIMENTS
In this section, we report experiment results to validate our proposed methods. First, we validate the effectiveness of our AttnScale and FeatScale when integrated with different deep ViT backbones (Section 5.1). Second, we compare our best models with state-of-the-art (SOTA) results (Section (.2). All of our experiments are conducted on the ImageNet dataset (Russakovsky al., 2015) with around 1.3M images in the training set and 50k images in the validation set. Our implementations are based on Timm (Wightman 2019) and DeiT (Touvron et al. 2021a) repositories.
5.1 How CAN ATTNSCALE & FEATSCALE BENEFIT DEEP VIT?
Experiment Settings. In this subsection, we intend to testify our models are beneficial to various ViT backbones with different depth settings and training modes. We choose DeiT (Touvron et al.) 2021ah as our first backbone in order to train from scratch. When training 12-layer DeiT. we follow the same training recipe, byper-parameters, and data augmentation with Touvron al. (2021a). When training 24-layer DeiT, we follow the setting in Gong et al. (202T). Specially, we set dropout rate to 0.2 when training 24-layer DeiT (Touvron et al. 20211). Our second backbone is CaiT (Touvron) et al. 2021b). We only apply our techniques to the patch embedding layers. The third backbone is the SOTA model Twin-Transformer (Liu al. 2021). All experimental settings share the same with Liu et al, (2021). In addition to training from scratch, we also investigate the fine-tuning setting. We defer this part Appendix G.I
The results suggest our proposed AttnScale and FeatScale successfully facilitate both DeïT, CaïT, Swin-Transformer under different depth settings and training modes. Specifically, AttnScale brings less than 100/150 extra parameters for 12/24-layer DeiT while boosting the top-1 accuracy by 0.9% for 12-layer DeiT and 0.6% for layer DeiT. Our FeatScale substantially improves top-1 accuracy by 1% for 12-layer DeiT and 0.8% for 24-layer DeiT. Compared with existing techniques, the improvements
Results. All of our experimental evaluations are summarized in Table [
8

---

Published as a conference paper at ICLR 2022
Table 2: Compared with state-of-the-ar models on ImageNet dataset. Accuracies with superscript (*) are reported by Gong et al. (202T) with superseript (f) are reported by Yuan et al. (202), and others are reported by the original papers. Bold accuracies signifies best models among pure transformers.
Category
Method
# Purum
Inpat size
FLayer
Top-1 (cc
CNN
ResNet 152 el. 2016
DenseNet-200
210
77M
124
152
75.1
CNN+
CVT21 Wa all A144
12M
224
21
82.5
Transformer
IN-VT--
ar 2:21
26M
224
83.3
VIT-S716
Drsavitskyy al 2020
49M
224
12
78.1
VIT-R/16 (Usaviiddyy
86M
224
79.8
DeiT-S
Toavzon a 20410
22M
224
79.8
Deil-S Dionet Enovrrn
2021a
22M
224
12
81.2
Transfomer
(Li
V1- Yuan cl [041
64M
224
82.3
DeepViT-24B Zioe AuAla
36M
224
80.1
[Touvrio
Dell-S DivansePach (Gong 2021
44M
224
K2.2
Detl-S AttSScale
43M
224
81.1
FeatSeale
224
Qurs
CaiT-S AttnScale
CarlSS PeatScale
47M
47M
224
224
24
83.2
83.2
Swin-S ArmSScal
50M
Swin-S + FentScale
50M
224
24
83.5
of AttnScale and FeatScale already surpass re-attention (0.6%) Zhou al., 2021a), LayerScale
(0.7%) (Touvron et al. 2021b), and late class token insertion (0.6%) (Tovrro et al. (2021b). We also observe a consistent 0.6% performance gain when AttnScale and FeatScale plugged into CaiT. Under fine-tuning setting. promote their performance by 2 0.2% (see Table 3). On Trin-Trrasformer, both our AttnScale and FeatScale bring around 0.5% accuracy gain. This makes Swin-S with 50M parameters comparable to Swin-B B (83.5% top-1 accuracy on ImageNetlk) with 88M parameters. We defer more model interpretation and visualization Appendix G For brief summary, we observe that both shallow and deep ViT enjoy from AttnScale and FeatScale that: 1) the attention maps can simulate richer filtering properties (compare Fig. 9 with Fig. 10). and 2) more high-frequenc data can be preserved (refer to Fig.
will show in Appendix G.1 only tens of epoch fine-tuning can further
5.2 COMPARISON WITH SOTA MODELS
In this subsection, we compare our best models with state-of-the-arr models on ImageNet benchmark. We choose SOTA models from three classes: CNN only, CNN transformer, and pure transformer. For transformer domain, we only conduct experiments with those lightweight models with comparable number of parameters, such VIT-S and DeiTT-. All the results are presented in Table
Among all methods, Trin-Tranfformer combined with our methods achieves the state-of-the-wee performance. Our CaiT-S AttnScale and CaiT-S + FeatScale 24-layer CaïT-S also attain superior results over all other pure transformers, while keeping low parameter cost. That our performance surpasses some CNN-based models (e.g., ResNet-152 1.2 and CVTT) indicates by increasing depth, ViT will be endowed with higher potential to surpass CNNs that have been dominating computer vision domain so far. Our DeiT-S+FeatScale result also outperforms ViT-B/16 and DeiT-S Distilled, which suggests deepening network can bring more considerable accuracy gain than increasing model width or employing teacher model.
6 CONCLUSION
In this paper, we investigate the scalability issue with ViT and propose two practical solutions via Fourier domain analysis. Our theoretical findings indicate Multi-Head Self-Attention (MSA) inherently performs low-pass filtering on image signals, thus causes rank collapse and patch uniformity problems deep VIT. To this end, we proposed two techniques, AttnScale and FeatScale, that can effectively break such low-pass filtering bottleneck by adaptively scaling high-pass filter component and high-frequency signals, respectively. Our experiments also validate the effectiveness of our methods. Both techniques Grounded with our theoretical framework, interesting directions for further work include designing parameter regularizations and spectrum-speciffe normalization layers.
boost various ViT backbones by significant performance gain.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGMENTS
Z.W. in part supported by an NSF SCALE MoDL project (#2133861).
REFERENCES
Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučié, and Cordelia Schmid. Vivit: A video vision transformer. In IEEE International Conference on Computer Vision (ICCV), 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems (NeurIPS), 2020.
Chen Cai and Yusu Wang. A note on over-smoothing for graph neural networks. In International Conference Machine Learning Workshop (ICMLW), 2020.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European Conference Computer Vision (ECCV), 2020.
Hanting Chen, Yunhe Wang, Tianyu Guo, Chang Xu, Yiping Deng, Zhenhua Liu, Siwei Ma, Chunjing Xu, Chao Xu, and Wen Gao. Pre-trained image processing transformer. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021a.
Tianlong Chen, Yu Cheng, Zhe Gan, Lu Yuan, Lei Zhang, and Zhangyang Wang. Chasing sparsity in vision transformers: An end-to-end exploration. In Advances in Neural Information Processing Systems (NeurIPS), 2021b.
George Cybenko. Approximation by superpositions of a sigmoidal function. Mathematics of Control, Signals and Systems, 1989.
Zihang Dai, Zhilin Yang, Yiming Yang. Jaime Carbonell, Quoc V Le. and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond fixed-length context. In Annual Meeting of the Association for Computational Linguistics (ACL), 2019.
Jacob Devlin, Ming-Was Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv:1810004805, 2018
Yihe Dong, Jean- Can--Batistte Cordonnier, and Andreas Loukas, Attention is not all you need: Pure attention loses rank doubly exponentially with depth. In International Conference Machine Learning (ICML), 2021. 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR), 2020.
Chengyue Gong, Dilin Wang, Meng Li, Vikas Chandra, and Qiang Liu. Vision transformers with patch diversification. arXiv: 2104. 12753, 2021.
Meng-Hao Guo, Jun-Xiong Cai, Zheng-Ning Liu, Tai-Jiang Mu, Ralph R Martin, and Shi-Min Hu. Pet: Point cloud transformer. Computational Visual Media, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2016.
Kurt Homik. Approximation capabilities of multilayer feedforward networks. Neural Networks, 1991.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
10

---

Published as a conference paper at ICLR 2022
Yifan Jiang, Shiyu Chang, and Zhangyang Wang. Transgan: Two pure transformers can make one strong gan, and that can scale up. In Advances in Neural Information Processing Systems (NeurIPS), 2021a.
Zihang Jiang, Qibin Hou, Yuan, Daquan Zhou, Xiaojie Jin, Anran Wang, and Jiashi Feng. All tokens matter: Token labeling for training better vision transformers. In Advances in Neural Information Processing Systems (NeurIPS), 2021b.
John Jumper. Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool Russ Bates, Augustin Zidek. Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature, 2021.
Hyunjik Kim, George Papamakarios, and Andriy Mnih. The lipschitz constant of self-attention. In International Conference Machine Learning (ICMI.), 2021.
Thomas Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations (ICLR), 2017.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep con- volutional neural networks. In Advances in Neural Information Processing Systems (NeurIPS), 2012.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 2015.
Qimai Li, Zhichao Han, and Xiao-Ming Wu. Deeper insights into graph convolutional networks for semi-supervised learning. In AAAI Conference on Artificial Intelligence (AAAI), 2018.
Kevin Lin, Lijuan Wang, and Zicheng Liu. End-to-end human pose and mesh reconstruction with transformers. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows, In International Conference on Computer Vision (ICCV), 2021.
Zhou Lu, Hongming Pu, Feicheng Wang. Zhiqiang Hu, and Liwei Wang. The expressive power of neural networks: A view from the width. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
Carl D Meyer. Matrix analysis and applied linear algebra, volume 71. SIAM, 2000.
Hoang NT and Takanori Maehara. Revisiting graph neural networks: All we have is low-pass filters. In International Conference on Pattern Recognition (ICPR), 2021.
Kenta Oono and Taiji Suzuki. Graph neural networks exponentially lose expressive power for node classification, In International Conference on Learning Representations (ICLR), 2019.
Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In International Conference on Machine Learning (ICMI.), 2018.
Philipp Petersen and Felix Voigtlaender. Equivalence of approximation by convolutional neural networks and fully-connected networks. Proceedings of the American Mathematical Society, 2020.
Dana Randall. Rapidly mixing markov chains with applications in computer science and physics. Computing in Science & Engineering, 2006.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 2015.
Zhiqing Sun, Shengcao Cao, Yiming Yang, and Kris Kitani. Rethinking transformer-basee set prediction for object detection. In IEEE International Conference on Computer Vision (ICCV), 2020.
11

---

Published as a conference paper at ICLR 2022
Yehui Tang, Kai Han, Chang Xu, An Xiao, Yiping Deng, Chao Xu, and Yunhe Wang. Augmented shortcuts for vision transformers. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
Matus Telgarsky. Benefits of depth in neural networks. In Conference on Learning Theory, 2016.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning (CML), 2021a.
Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. In International Conference on Computer Vision (ICCV), 2021b.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems (NeurIPS), 2017.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. In International Conference on Learning Representations (ICLR), 2018.
Nitika Verma, Edmond Boyer, and Jakob Verbeek. Feastnet: Feature-steered graph convolutions for 3d shape analysis. In JEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2018.
Yuqing Wang. Zhaoliang Xu, Xinlong Wang, Chunhua Shen, Baoshan Cheng, Hao Shen, and Huaxia Xia. End-to-end video instance segmentation with transformers, In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
Ross Wightman. Pytorch image models.
com
pytorch--mage-
2019.
Felix Wu, Amauri Souza, Tianyi Zhang, Christopher Fifty, Tao Yu, and Kilian Weinberger. Simpli- fying graph convolutional networks. In International Conference on Machine Learning (ICML), 2019.
Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. Cvt: Introducing convolutions to vision transformers. In IEEE International Conference on Computer Vision (ICCV), 2021.
Chengxuan Ying. Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, and Yan Liu. Do transformers really perform bad for graph representation? In Advances in Neural Information Processing Systems (NeurIPS), 2021.
Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zihang Jiang, Francis EH Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. In International Conference on Computer Vision (ICCV), 2021.
Aston Zhang, Alvin Chan, Yi Tay, Jie Fu, Shuohang Wang, Shuai Zhang, Huajie Shao, Shuochao Yao, and Roy Wei Lee. On orthogonality constraints for transformers. In Annual Meeting of the Association for Computational Linguistics (ACL), 2021.
Biao Zhang, Ivan Titov, and Rico Sennrich. Improving deep transformer with depth-scaled initializa- tion and merged attention. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2019.
Dong Zhang, Hanwang Zhang, Jinhui Tang, Meng Wang. Xiansheng Hua, and Qianru Sun. Feature pyramid transformer. In European Conference on Computer Vision (ECCV), 2020.
Minghang Zheng, Peng Gao, Xiaogang Wang, Hongsheng Li, and Hao Dong. End-to-end object detection with adaptive clustering transformer. In British Machine Vision Conference (BMVC), 2021a.
12

---

Published as a conference paper at ICLR 2022
Wenqing Zheng, Qiangqiang Guo, Hao Yang, Peihao Wang, and Zhangyang Wang. Delayed propagation transformer: universal computation engine towards practical control in cyber- physical systems. In Advances in Neural Information Processing Systems (NeurIPS), 2021b.
Daquan Zhou, Bingyi Kang, Xiaojie Jin, Linjie Yang. Xiaochen Lian, Zihang Jiang. Qibin Hou, and Jiashi Feng. Deepvit: Towards deeper vision transformer. arXiv:2 1886, 2021a.
Daquan Zhou, Yujun Shi, Bingyi Kang, Weihao Yu, Zihang Jiang, Yuan Li, Xiaojie Jin, Qibin Hou, and Jiashi Feng. Refiner: Refining self-attention for vision transformers. arXiv:2106.0377 2021b.
Bing-Xuan Zhou. Universality of deep convolutional neural networks. Applied and computational harmonic analysis, 2020.
Luowei Zhou, Yingbo Zhou, Jason Corso, Richard Socher, and Caiming Xiong. End-to-end dense video captioning with masked transformer. In JEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. In International Conference on Learning Representations (ICLR), 2021.
A MORE PRELIMINARIES ON FOURIER ANALYSIS
In this appendix, provide preliminary knowledge about Fourier analysis. Here, only consider discrete Fourier transform on real-value domain F RR + C". A Discrete Fourier transform (DFT) can be written in matrix form as below
2mj
2xj(n-1)
1 DFT
2mj(k-1)-1 1).1
2xj(k- 10-1-1))n-111
(13)
Vn
2mj(n-1)
2nj(nn11) 172
and its inverse discrete Fourier transform is DFT DFT. In this paper, regard matrices as multi-channe signals. For example, X E Rnxd means d d-channel n-length signals. When DFT and inverse DFT are applies to multi-channel signals, each channel is transformed independently. i.c. F(æa)) = DFT. X.
F(X) = (F(x))
Hereby, we can simply operators DC [+] and HC using the matrices in Eqn. 13 can write 'DC [-] as below:
By definition, we
DC [ DFT diag(1,0, (1,0,110,--0.. 1,0,
DFDT DFF.
(14)
11
(15)
namely DC [ = 117/n. Conversely, we can write HC as:
HC [x]
DFT diag (0, 1, DFDFT -DFT-14 (I (1,0, DF)
(16)
(17)
11 x.
(18)
which indicates HC []
11T /n. We will frequently use these derivations later in the proofs.
Without loss of generality, only consider 1D Fourier transformer, since the DC components invariant the dimension of signals.
13

---

Published as a conference paper at ICLR 2022
B DEFERRED PROOFS
B.1 PROOF OF THEOREM 1
Theorem 1. (SA matrix is low-pass filter) Let softmax(P), where P RR Then A must be a low-pass filter. For all & E R",
|CC [A [A'2]l22 |CC [A'z]]e 0. 21000
lim
Proof. Let Å1, A2,.... A, C be the eigenvalues of A with ordering |^1 A2 Px Notice that, A is positive matrix, each of whose element is strictly greater than zero (Ass > 0, Vi. Besides, for all 1, n, Σÿ= A.j 1. Therefore, A1 implies A must have an eigenvalue and its corresponding eigenvector the all-one vector 1.
By Perron-Frobenius Theorem (Meyer 2000), eigenvalue corresponds to a all-positive eigenvector implies A 1 should be the largest eigenvalue without multiplicity, and the absolute value of other cigenvalues λ2, .- ' must be less than 1. Let us rewrite A in the Jordan canonical form A PJP
A1
((22)
u
A 01
Vn
(19)
J J(x.)
where the Jordan block J() can be written as
J(A)
(20)
A
1
Applying A to for i times be written A'z which is equivalent to:
A
J(22)
A'z = PJPP-1
p-12 2
(21)
J J(xs)
T
J(M2)*
=
P-1z
(22)
J(A.)
Let f(x xt, then A' = Pf(J)p-'= P diag(f(()), (()), (- a Jordan block with shape k. then
Suppose
((()) ((()) f(())
(0)
(k-
I)!
S(A)
f'(A)
-()()))
(23)
("
f())
Therefore, on the diagonal number m I min(t, & 1) above the main diagonal stands:
t(t - 1)... m + 1) m
(24)
m!!
14

---

Published as a conference paper at ICLR 2022
For arbitrary m k and
< 1,
t(t
- 1)... (t m + 1) Xmm 0
lim 61000
(25)
m!
Recall that \ and according the definition of Jordan canonical form, 1i 1. By Eqn.
25
lim A'z
P lim
(((2))) (. (()
(26)
t-000
P diag(Ni; (1, 0..... .......0)Pp-2
(27)
(28)
luf=
(29)
Plug the result from Eqn.
29 into the original limit:
HC [A'z]l2
HC [A'=]]13
lim
lim
(30)
DC
[A'z]]|
1-000
| HC [A'z]]B
| | [A'z]]3
lim
(31)
||HC| [A'=113
11(I 11 >A
lim
(32)
E-+00
- 11 - A
11 3. 111 211
(33)
1 ||(I - 1--11,111 1, 1u
| [TIuf luf 2113
(34)
[[[ 1u 21
(35)
where Eqn. 31 is due to the orthogonality of DC and HC terms.
L
B.2 PROOF OF COROLLARY
Corollary 2. Let P1, P2,.... Pn. be sequence of matrix in Raxn, and each Pk, Vk 1,,.. L has A softmax(PM, Then I As is also low-pass filter.
Proof. Let A- = [ As, then we show A satisfies the following conditions, so that A can be regarded as another self-attention matrix. Then we can conclude the proof by Theorem !
1) For every = 1, 1,... Σド= Aij =. 1.
Suppose Σラ=1 Bi = 1 for every = 1.... n. then for every is == 1, L and i= 1,,,, n,
n
n
Σ
()
An.im Bmj
(36)
j=l
Ag.im Σ Bmj
(37)
m=1
n
Σ
Ag.im 1.
(38)
m=1
By induction, for every i, Ex= A = 1 + (A2Ail) = =1 = an E A =1
2) For every i,j = 1.... n, Aij > 0.
Suppose B₁ 0, Vi then for every ho = 1.... ,L (A₂ >E= Ak.im B Bmj. Since
Ak.im > 0, B Buj > 0, then (A > 0. By induction, for every i,j, A1,i5 > - (A₂Ar)s )as >
0 Aij > 0.
15

---

Published as a conference paper at ICLR 2022
B.3 PROOF OF THEOREM! 3
Theorem 3. (convergence rate of SA) Lel A = softmax P) and a = maxij Pijl. where P E
Raxn, Define SA(X) AXW. Wv the oulput of self-altention module, then
T1e 20x
(SA())( S
W WWvl22lHCC [X]]]r-
22 +
In particular, when P XW (XWR) /Vd, and assume tokens are distributed inside ball with radius y > O, i.e. | 2 - 7 vi = 1, ... n, then a S ( WWWWWR W2ll///
Proof. First, we write X = DC X + HC X 1 z H. where DC [X] = 1z" equals to the
orthogonal projection of X to subspace span(1) (1), and H HC [X] represents the remaining part of the original signals.
HC [SA(X)] (I 11 AXWr
(39)
1 11
A(1=" + H)Wy
(40)
n
1 11"
Al=T Wv + 1 11- AHW
(41)
n
1
11T
AHWv
(42)
n
Therefore,
1 11T
[HCISA(X) )[ll
1
AHWV
(43)
11
F
11T
(44)
n
||sftmaxx(P) (P) [tll//otmaxxpP]]lammm
[|sottmax(P||///sttmxx>)) W, H|
(45)
/|ssflmaxx PP) Wrlla||H||
(46)
The Eqn. 45 leverages special case of Hölder's inequality, and the Eqn. 46 be yielded from
||sottmax Pa = 1. Now we need to upper bound [|softmax(P) P)] Suppose a max;, Piyll
then for each I, n, have the following inequality for the element with the largest value (say the j-th column):
PO
20
Aij
I
(47)
E Pi. ea + + e
22a +
(n
Hence, we have [|softmax( P)]| S E. max, Aij S can conclude the proof. In particular, when P XW XW K) IT //d
no
Insert this result to Eqn. 46 we
1n=
We a x
a = max P = max
(48)
Vd
Since |22;||2, 11, I y, Vi,j, < 1| We w lall l ll/Vd s >2llooWElb//a Vd.
C EXTENSION OF THEOREM
C.I MULTI-HEAD ATTENTION
Proposition (smoothening rate with MSA) Let A = softmax(Ph), where ph E Raxa with
h = 1,... H. Let max output of multi-head self-atiention module, then
H maxs,3 PP Define MSA(X) EH A X w/ the
ne2a
HHC where H is the number of heads, as max W 1|2 and max [= Wh v
50102
||HC [X]][r)
220 + -
16

---

Published as a conference paper at ICLR 2022
Proof. For the h-th head, according to Theorem
ne2as
||C [SAA S
|WB1121174C [X]] r:
(49)
c + n
where oA = maxij] p.ll Then we have:
| (MSA(
HC
Σ SAn(X)
(50)
H
HC [SAA(X)Wo]]
(51)
ne2an
W 1121111 [X]]l2
(52)
200
+
H
ne
[x]]|e
(53)
22an
+
h=1
ne2a
S0102 22 + +n-1 ||C [X]|
(54)
where Eqn. 51 from the linearity of HC [t] and triangle inequality. Eqn. 54 can be obtained by relaxing on. [ 22, and |ll to oi and 02 (Note that with a).
meda +n- is monotonically increasing
C.2 RESIDUAL CONNECTION
Proposition 5. (smoothening rate with skip connection) Let A = softmax(Ph), where ph E Raxn with h 1.....,,........ ,H. .H. Let a maxf max,, P.y. Define X' MSA(X) + X as the output of multi-head self-attention module with skip connection, then
[H']I1 S + 110002H
| [M|[X]] |
2a + n
where is the number of heads, σi max
W |12 and 02 |
Proof. By Proposition
4
|| [X'Ile = ||HC (MSA(X) + X]IFF
(55)
HC [MSA X ||1| ||HC [X] |F
(56)
ne2n 2n
F + |HC [X]lIP
(57)
20
ne
+ 1100g
||HC
(58)
c2a +
Again, Eqn. 56 follows from the linearity of MC [-] and triangle inequality.
C.3 FEED-FORWARD NETWORK
Proposition (smoothening rate with FFN) Let A softmax( (aammax(P))..... where Ph E Raxx with h 1. H. Let a max h=1 maxi, P.ll. Define Y FFN(MSA(XX)) + X) as the output of transformer block, then
H
no2o
|HC [YIlF S S3 1+ 22
|HC [X]]
where FEN Rd + Rd represents fordard network, H is the number of heads, max W 02 = max m W and σ3 = Lips(FFN) is the Lipschitz constant of FFN. In
particular, d3 1 Lips(FFN) when residual connection considered in FFN.
17

---

Published as a conference paper at ICLR 2022
Proof. Let X' MSA(X) X and 2' = (X(( R., then we have
HC [FFN(()]]]] - [IFFN(X') -FFFN())) I|
(59)
=|IFF((X') FF )]F
(60)
S oal| X' - -1="11p
(61)
( (1-
03
X
=03l|HC [X']]]
(62)
n
F
ne22
1+0
(63)
2
+
where Eqn. 59 follows from Lemma 7 Eqn. 60 holds because FFN operates row-wisely on feature matrix, and Eqn. is due to the definition of Lipschitz constant. Finally, Eqn. 633 is yielded from Proposition
Lemma 7. Given X E RRdd [X]|l X 1z for all E RR.
Proof. We prove the Lemma by showing that x" = x²1 1/n achieves the minimum of the optimiza- tion problem arg min X 1z 1=="p 1|
X
12 Tr(x" =11) -1=²)
(64)
= =Tr(X" TT(=11 x) Tr(X"1=²) + 1r(x2191z)
(65)
= =n = - 12
Tr(x x)
(66)
is easy to show the derivative in terms of 2:
V 12 1 2nz 2x"
(67)
Therefore, x" 1 x achieves the minimum.
D DEFERRED REMARKS ON SECTION 2.4
D.1 CONNECTION WITH RANDOM WALK THEORY
We add that the asymptotic evolution of feature representations can be interpreted through the lens of random walk theory. We can regard self-attention map A as probability transition matrices for Markov chain. Since each entry is larger than zero, the Markov chain should be irreducible. This implies the Markov chain will converge into a unique stationary distribution π (Randall 2006). Let a; denote the i-th row of A, then for all i 1,...
where E (0, 1) is the mixing rate of the transition matrix A. As a consequence, lim A' = 1TT yields pure low- -asspsss filter. When repeatedly applying this self-attention matrix to feature maps, limp- A'X X > 1TT" X only preserves the rank-1/DC portion of the signals, which is consistent with our Theorem Nevertheless, this interpretation does not bring other transformer components into consideration. And our theory further provides concrete convergence rate with respect to the network parameters (Theorem 3).
have (I(a,)" T²"l < a.
D.2 REMARKS ON DONG ET AL.
2021
Here we respectfully elaborate on the hidden assumptions in the proof of currentt preprint of Dong ct al. (021I).
1) In the proof of Lemma A.3, Taylor expansion used to approximate and upper bound an exponentiation. However, to let the right-hand side upper bound satisfied, we conjectured that the authors implicitly assumed E,j Eig is bounded around zero. After directly communicating with the authors, they confirmed that a missed assumption here is max;,; EE; Egy)) < 1.
2) In the proof of Lemma A. 1, to let Eqn. (8)-(9) hold, the authors may have assumed R, Wv 2 0, where 2 denotes entry-wise inequality. As the authors suggested, an entry- absolute value can be imposed to R and wy as simple fix, without influencing their e and for norm. However,
18

---

Published as a conference paper at ICLR 2022
after those changes, still have difficulty walking through Eqn. (6)-(8), and currently communicating with the authors on this matter.
3) In the proof of Lemma A.I, find Eqn. (12) may not be satisfied in general. We can raise the following counterexample: Since E, r, R, Wv can be any matrices, we simply let = d2ag(2, 3), softmax(() = [0.8 0.2 R Wv I. Then ||D1softmax(r)* RWv = 4 while D1| sc. |||||]|| = 3, which disproves the claim. We conjecture that some additional prerequisite constraints E, might be needed here to proceed the derivation, and currently communicating with the authors on this matter.
D.3 GENERALIZE TO OTHER ATTENTION MECHANISM
Our theorizing can be smoothly generalized to other attention mechanisms because our Theorem | and do not require any prior knowledge on pre-softmax pairwise correlation P.
Logistic Attention. We refer logistic attention to the attention mechanism used in Veličković et al. 2018): Verma et al. 20188, where attention is calculated via a linear combination:
exp (x uq + x u + b)
A.j Σ, exp (z u.uk + b)
(68)
where us and ug are query/key parameters, b is the bias term. With the same condition in Theorem 3 we can upper bound by [[ ||2 + llu |2)7 + b).
L2 Distance Attention. L2 distance based attention (Kim et al. 2021) Lipschitz formulation of self-attention. The pair-wise attention can be written as follows:
exp - Wa x 113TT)
A.
(69)
Σ, exp /
Wq x W wxl//r)
where We and W₁ are query/key weights, and a scaling factor. Similar Theorem we can upper bound by ([[Wklla + //22
E AN AUXILIARY LEMMA FOR ATTNSCALE
Lemma 8. Let = be the spectral response of attention matrix A, and parameterize low-filter by L = diag(3. 3, 3.0.... O.... Then 3 = 1 is the optimal solution of the following optimization problem: arg mingl | L||F.
Proof. First we make simplification L = F- diag(3,0, ... ..,..FFFFFFFFFF.. - f_11 Then have:
(refer to Appendix
A
A - - = ||A $111 1 = TT(
311 " (A- --117)
(70)
f² 11
=T
ATA
11
11 A+
(71)
n
82
28
Tr11
A T
(72)
n
23 m
R
82
A +
A
(73)
1=1 1=1
- 23 + Tr ATA From Eqn. 74 3+ achieves the minimum the objective function.
(74)
F
MORE ON VISUALIZATION
F.1
DETAILS ON FIGURE
To verify our Theorem 3 we depict the high-free high-frequency intensity of each layer's output and its theo- retical upper bound. Our visualization is based the official checkpoint of 12-layer De Since
19

---

Published as a conference paper at ICLR 2022
training ViT without either FFN or residual connection will certainly cause failure, remove these components directly from the pre-trained model to illustrate the effects of different components. We logarithmic scale for the purpose of better view. Let X; denote the output of the I-th layer, and Xo be the initial inputs. For red line, we directly calculate log( ( HC
||ll////// Xollr> at each layer. For blue line, we first obtain the coefficient in Section 2.2 and 2.3 with respect to network parameters
(e-g-, compute "Y
neza
Wvll2 for attention only architecture). Then estimate
[ool / To summarize, one can see without residual connection, the first two sub-figures imply an exponential convergence rate, which is consistent with our Theorem
| the upper bound by n1 HC [X [t-]]]l and apply the logarithm by log(?) HC [x,=]]llr/
F.2 MORE VISCALIZATION SPECTRUM
In this appendix, provide more visualization the spectrum of attention map to validate Theorem [ We compute the spectrum of attention map for both Fig. 2 and Fig. 5 in the following way. By regarding A as linear filter, its Fourier-domain response is another linear kernel = FAF- When A applied to a spectrum ã = Fx of signals æ, the i-th frequency response will be A, where A; the i-th row of A. Hence, we use ||i]|2 evaluate the spectral response intensity of the i-th frequency band. Below we provide a complete spectral visualization of attention maps computed from random sample in ImageNet validation set.
SHETT
1000000
SEEE
Figure 5: Visualize the spectrum of attention maps. Each row demonstrates every head at a same layer, and from top bottom, the 12 rows correspond 12-th layer, for left to right, the columns correspond to - 6-th head, respectively. Best view in zoomable electronic copy.
20

---

Published as a conference paper at ICLR 2022
F.3 DETAILS ON SIMILARITY CURVES (FIGURE
In Fig. we visualize the cosine similarity of attention maps and feature maps to show the effectiveness of our AttnScale and FeatScale 24-layer DeiT, respectively. We follow the definition Zhou et al. (2021a) to compute the cosine similarity metric for attention maps. Instead of measuring cross-layer similarity, we calculate average cross-patch similarity the same layer. Given the layer index I and corresponding attention maps A (I.A) RRxx, the cosine similarity be computed by:
2 n(n - 1)H
H
n
B
A
(LA)T A
(LA) 5
M
Σ
Σ
(75)
(M
where (1.h) denotes the i-th column of A (1.A), and H is the number of heads. The cosine similarity between i-th and j-th column of (1.A) measures how the contribution of one token (say the i-th token) varies from the other (say the j-th token). We average the similarity between every pair
of tokens' attention map (excluding the self-to-self similarity) and every attention head. We refer interested readers to /houu et al. (2021a) for more details.
We use the similar metric to compute similarity for feature maps. Following Gong al. (2021), , we compute pair-wise cosine similarity between every two different tokens. Formally, given the layer index l, and its output X (!) E Rx the cosine similarity is estimated by:
2
M
(76)
n(n 1)
1=1 j=++1
where _ denotes the i-th row ( The cosine similarity between between i-th and j-th row of X (1) measures how similar the feature representations of two tokens are. Likewise, we average the similarity between every pair of tokens' features except for the self-to-self similarity. More details can found in Gong et al. 2021). We additionally provide visualization of these two metrics for
12-layer DeiT in Fig. 12
G DEFERRED EXPERIMENTS AND MODEL INTERPRETATION
G.1 FINE-TUNING TUNING EXPERIMENTS
Our deferred fine-tuning experiment with CaiT (Touvron al.) 2021b)
results are presented in
T Different from trining scratch, we fine-tune CaiT with AttnScale and FeatScale parameters from the pre-trained models for 60 epochs following Gong al. (2021). For a fair comparison, simultancously train a plain CaiT for another 60 epochs. During fine-tuning, we reduce learning rate to 10-$ and weight decay x 10-4, All other pprremprrretters and training recipe are kept consistent with the original paper Touvron et al. 20216).
Table 3: Experimental evaluation of finetuning AttnScale & FeatScale with CaiT. The number inside the (t represents the performance gain compared with the baseline model, and accuracies within/out of parenthesis are the reported/reproduced performance.
Backbone
Method
Input size
Laver
FLOPs
Throughput
Top-I Ace (%)
CaTTXXX
224
24
12.0M
2,33G
589.3
(77.6
CXX
AutScale
24
77.8 0.3)
CaiT-XXS
FeatScale
224
24
12.0M
2.53G
573.5
77.8 0.3)
CaiT
CarT-S
24
46.9M
8.74G
371.9
82.6182.7)
Cail-S AttaScale
224
24
46.9M
8.75G
339.0
82.8 (1 0.2)
CaiT FeaiScale
224
24
46.9M
8.75G
358.2
82.9 0.3)
G.2 VISUALIZATION AND INTERPRETATION OF ATTNSCALE
In this subsection, provide visualization to interpret AttnScale and further support experiments. In Fig. 6 we visualize the learned weights our AttnScale, We observe conclude our AttnScale successfully trained to amplify the high-pass component. We also find when layer
21

---

Published as a conference paper at ICLR 2022
index goes larger, the scaling weights turns larger to prevent attention collapse at deeper layer. ? We also compare the attention map produced by AttnScale with those produced by original DeiT. extract more salient and higher contrastive attention
We observe from Fig. 8 that AttnScale than vanilla DeiT, which indicates our AttnScale possesses higher capability to distinguish tokens from larger variety of attention schemes. @ To be
objective, we plot the spectrum of 24-layer DeiT's attention maps with/withour our AttnScale in Fig. 9 and [0] The visualization procedure has been elaborated Sec. F2 We find that attention maps from AttnScale enjoy richer filtering diversity, capable of performing high-pass nigh-pass (row 2, column 3) and band-pass (row 12, column 5)
filtering, instead of only low-pass filtering (see Fig. 9
2
Magnitude
10
10
0
4
6 8 9 1011
6 9 1011 Layer Index
0 0
2
Figure 6: Visualize the learned weights of DeiT-S AunScale. Each sub-plot depicts the scaling weights of the same head for different layers. For left to right, top to bottom, six sub-figures correspond 6-th head, respectively. Best view in color.
Param
Param
Param
2
Magnitude
0
3
5
9 1011
8
9
1011
0
0
10
Param
Param
Param
2
1011
9 1011
Layer Index
Figure 7: Visualize the learned weights of DeiT-S + FeatScale. Each sub-plot depicts two groups of scaling weights of the same head for different layers. For left to right, top to bottom, six sub-figures correspond to 6-th head, respectively. Best view in color.
G.3 VISUALIZATION AND INTERPRETATION OF FEATSCALE
In this subsection, we provide visualization to interpret our FeatScale. We plot the scaling weights of FeatScale in Fig. 7 We observe that the re-weighting factors learned for nigh-frequenne components & is consistently larger than the weights for the DC term 8, which indicates our FeatScale successfully trained elevate high-frequency features against the dominance of DC component. Similarly, the gap between & and t becomes huger when going deeper. @ We also demonstrate the proportion of feature maps' geggffreyeency component in Fig. 11 for both 12 (lower one) 24(upe one) -layer DeiT. The proportion value is calculated by |[HC [ X ||.. We find high-frequency signals diminish quickly deeper layer, and 24-layer DeiT suffers from faster pace. Our FeatScale is effective to keep the high- figh-frequency signals stand for both 12-layer and 24-lr DeiT.
22

---

Published as a conference paper at ICLR 2022
DeT'
0e'' AtsSclc
0.0
Figure Visualize attention map DeïT-S with/without AutnScale. 4 max pooling has been applied. The first visualizes attention maps without AttnScale, and the second row visualizes attention maps with AttnScale. Each column corresponds to the layer noted by its sub-title. The attention map computed from random sample in ImageNet validation set. We only demonstrate the first head of each layer. Best view in zoomable electronic copy.
Figure Visualize the spectrum of attention maps without AttnScale, Each row demonstrates every head at same layer, and from top bottom, the 24 rows correspond to 1 24-th layer, for left to right, the 6 columns correspond to 1 6-th head, respectively. Best view zoomable electronic copy.
23

---

Published as a conference paper at ICLR 2022
Whans Marar
Miphha
kmmm
abblsonn..
Lotend Aad Antuni
Allcoom
Mitiied Mandee
A Maian
moace Manuto
Misan
s Munnns
Jone Gay A Mynaom
Sodd Wull Now
byode Welh Hahmel
A/l Aol
L
Yaghh Vin Audeol
Loughte Manauwl
Vaueed be M
Figure 10: Visualize the spectrum of attention maps with AttnScale. Each row demonstrates every head at same layer, and from top to bottom, the 24 rows correspond to 1 24-th layer, for left to right, the 6 columns correspond to 1 6-th head, respectively. Best view a zoomable electronic copy.
1.0
DelT-S DelT-S FeatScale
Cosine Similarity
0.8
DeiT-S AttnScale
0.9
0.6
0.8
0.4
0.2
15
18
21
24
0
10
Layer Index
Layer Index
0.40
DeiT-S
Cosine Similarity
DelT-S Featšcale
0.9
0.35
0.30
0.8
DelT-S DelT-S FeatScale
0.25
0.7
12
10
12
Layer Index
Layer Index
Figure 11: Visualize the proportion of the high-frequency component of feature maps with/without our FeatScale 12/24 layer DeiT. Refer to Appendix G.3 for details.
Figure 12: Visualize cosine similarity of at- tention and feature maps hith/without our pro- posed methods on 12-layer DeiT. Refer to Ap- pendix F3 for details.
24


Reviews:
[{"review_id": "xSNkltTbb4u", "reviewer": "Reviewer_kYZM", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: \t\n1.\tThe paper provides theoretical analysis of ViT from spectral domain, which explains the empirically findings of patch feature collapse of ViTs. \n2.\tReweight techniques of low-pass and high-pass components are proposed, which is sound according to the theory part.  \n\nWeaknesses:\nExperiments part does not strongly support the proposed method.\n1.\tThe gain is limited, 0.2 ~ 0.3 improvements for 24-layer models. \n2.\tWith FeatScale, 12-layer DeiT has 1.0 improvement v.s. 24-layer DeiT has 0.3 improvement, this is kind of opposite to the paper’s claim, as layers increases, the collapse should be more severe, where one would expect the proposed techniques should be more helpful,  however the experiments results indicates the opposite. It questions the proposed methods’ ability to scaling depths further beyond 24 layers. \n\n\n\n\n------post-rebuttal-----\nWith the new provided experiments results, I raised my rating towards accept. ", "summary_of_the_paper": "This paper provides a viewpoint of spectrum domain to show self-attention amounts to a low-pass filter and will cause feature maps to only preserve direct-current components as depth increases. Based on this viewpoint, it proposes two technique named AttnScale, FeatScale to reweight low-pass, high-pass components with minimal parameters. Experiments are conducted on ImageNet. ", "summary_of_the_review": "\nThe paper address the feature collapse problem of ViT from spectral prospective, decomposing attention to low-pass and high-pass components. The therotical part is nice, while the experiments does not show adequate evidence that the proposed method would actually help scale depths of ViTs (e.g. beyond 24 layers) as title indicates. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "_Wm4lSwD8Nn", "reviewer": "Reviewer_DHWs", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "However, some problems remain:\n1.The article proves that the self-attention is a low-pass filter, thus neglecting the high-frequency information. The conclusion comes to the stage that pure attention leads to rank collapse in the network. FFN, skip connections and mutli-head self-attention mechanism which are used in ViTs help to hold high-frequency components. The current clarification in the paper are not very convincing to verify why the ViTs cannot go deeper, even though with FFN, skip connections and mutli-head.\n2. The visualization of the modified attention is about the learnable parameters. However, the spectrum of the modified attention seems more convincing to show the change on high-frequency components. And it is more persuasive to show the situation when the depth is 24.\n", "summary_of_the_paper": "This paper explores the reasons why ViTs cannot go deeper. The article provides clear and solid proofs, clarifying the rank collapse in attention matrix via Fourier analysis. Meanwhile, the authors suggest two types of possible solutions, AttnScale and FeatScale, which are useful when scaling up the ViTs architecture. The experiments on DeiT and CaiT show that the adjustment on attention helps avoiding rank collapse when going deeper, holding high-frequency information. \n", "summary_of_the_review": "The overall presentation is good. The main concerns are in above. \nThe models (CaiT-S and DeiT-S) are fairly small. Can the authors provide more experiments on big models to further verify the effectiveness of the proposed method?", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "UnPKsmIaqTg", "reviewer": "Reviewer_D5xX", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, this paper proposes a novel idea of thinking self-attention weight matrix as a low pass filter. This explains the difficulty of convergence for deeper tranformer models. The proposed methods alleviates the low-passing representation of the transformers. The experiments also prove the superior performance using proposed modules. However, the experiments are not sufficient, the improvement is relatively minor, and it doesn't show whether the proposed approach can help deeper transformers converging. It would be very interesting to see it is helpful for deep models.", "summary_of_the_paper": "This paper first analysis the difficulty of going deeper using transformer models. The challenge is that the mechanism of self-attention is applying a low-pass filter.  Keep stacking self-attention layers will lose the feature expressive power and only preserve the DC bias.  To alleviate this issue, this paper propose two modules to make the MHA to be an all pass filter. First module AttenScale combines the high-pass component and FeatScale reweights the featuremap. The experimental results demonstrate the improvement on accuracy for DeiT and CaiT when adopting the proposed method.", "summary_of_the_review": "Interesting and novel idea and good mathematical analysis. May need more experiments to demonstrate the claims in the paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper analyses the frequency filtering properties of self-attention in vision architectures, shows that it mainly acts as a low-pass filter, and proposes fixes that allow to better preserve the higher frequencies. These fixes yield moderate classification accuracy gains (~0.5-1%) for several existing attention-based architectures.

The reviewers are quite borderline about the paper, but after considering the authors' responses lean towards acceptance. Pros include interesting and novel analysis and sound model improvements leading to non-trivial empirical gains. The main con is that the experimental results are fine, but not outstanding.

Overall, I recommend acceptance. Empirical results are indeed good but not outstanding, but the theoretical analysis is interesting and it is good to see that it leads to actionable insights on the model design side that actually help in practice - even is not by a huge amount. One part that in my opinion is confusing (and might have been confusing to the reviewers too) is that the title seems to suggest the paper will present very deep vision transformers while it does not. Adding deeper models or adjusting the title would help here.
Decision: Accept (Poster)