How Do Vision Transformers Work?
Authors: Namuk Park
Songkuk Kim
Venue: ICLR 2022 Spotlight (2022)
Keywords: vision transformer
self-attention
multi-head self-attention
loss landscape

Abstract: The success of multi-head self-attentions (MSAs) for computer vision is now indisputable. However, little is known about how MSAs work. We present fundamental explanations to help better understand the nature of MSAs. In particular, we demonstrate the following properties of MSAs and Vision Transformers (ViTs): (1) MSAs improve not only accuracy but also generalization by flattening the loss landscapes. Such improvement is primarily attributable to their data specificity, not long-range dependency. On the other hand, ViTs suffer from non-convex losses. Large datasets and loss landscape smoothing methods alleviate this problem; (2) MSAs and Convs exhibit opposite behaviors. For example, MSAs are low-pass filters, but Convs are high-pass filters. Therefore, MSAs and Convs are complementary; (3) Multi-stage neural networks behave like a series connection of small individual models. In addition, MSAs at the end of a stage play a key role in prediction. Based on these insights, we propose AlterNet, a model in which Conv blocks at the end of a stage are replaced with MSA blocks. AlterNet outperforms CNNs not only in large data regimes but also in small data regimes. The code is available at https://github.com/xxxnell/how-do-vits-work.

Published as a conference paper at ICLR 2022
How Do VISION TRANSFORMERS WORK?
Namuk Park 1.2 Songkuk Kim'
Yonsei University, "NAVER AI Lab
Seamukpprkee songkuk ac kr
ABSTRACT
The success of multi-head self-attentions (MSAs) for computer vision is now indisputable. However, little is known about how MSAs work. We present funda- mental explanations to help better understand the nature of MSAs. In particular, demonstrate the following properties of MSAs and Vision Transformers (ViTs):
MSAs improve not only accuracy but also generalization by flattening the loss landscapes. Such improvement is primarily attributable to their data specificity, not long-range dependency. On the other hand, ViTs suffer from non-convex losses. Large datasets and loss landscape smoothing methods alleviate this problem; MSAs and Convs exhibit opposite behaviors. For example, MSAs are low-pass filters, but Convs are high-pass filters. Therefore, MSAs and Convs are comple- mentary; 3 Multi-stage neural networks behave like a series connection of small individual models. addition, MSAs the end of a stage play a key role prediction. Based on these insights, we propose AlterNet, model in which Conv blocks the end of a stage are replaced with MSA blocks. AlterNet outperforms CNNs not only in large data regimes but also in small data regimes.
INTRODUCTION
There is limited understanding of multi-head self-attentions (MSAs), although they are now ubiquitous in computer vision. The most widely accepted explanation for the success of MSAs their weak inductive bias and capture of longerange dependencies (See, c.g., (Dosovitskiy et al., 2021; ;; Naseer al., 2021; Tuli et al., 2021; Yu et al., 2021a; Mao al., 2021; Chu et al., 2021)). Yet because of their over- wle-iiixx Vision Transformers ( (ViTs) neural networks (NNs) consisting of MSAs have been known to have a tendency to overfit training datasets, consequently leading poor predictive performance in small data regimes, e.g., image classification CIFAR. However, we show that the explanation is poorly supported.
1.1 RELATED WORK
Self-attentions (Vaswani al., 2017; Dosovitskiy et al., 2021) aggregate (spatial) tokens with normalized importances:
QK
2j = Σ Softmax
V
where Q, K, and V are query, key, and value, respectively. d the dimension of query and key, and zj is the j-th output token. From the perspective of convolutional neural networks (CNNs), MSAs are a transformation of all feature map points with large-sized and data-specific kernels. Therefore, MSAs are at least expressive as convolutional layers (Convs) (Cordonnier et al., 2020), although this does not guarantee that MSAs will behave like Convs.
Is the weak inductive bias of MSA, such as modeling long- rong-range dependencies, beneficial for the predictive performance? To the contrary, appropriate constraints may actually help a model learn strong representations. For example, local MSAs (Yang et al., 2019; Liu et al., 2021; Chu et al., 2021), which calculate self-attention only within small windows, achieve better performance than global MSAS not only small datasets but also on large datasets, e.g., ImageNet-21K.
In addition, prior works observed that MSAs have the following intriguing properties: ① MSAs improve the predictive performance of CNNs (Wang al., 2018; Bello al., 2019; Dai et al., 2021;

---

Published as a conference paper at ICLR 2022
ResNet
100
Density
Density
200
3h0
ViT
Density
Density
1000
1000
ResNet
(a) Loss landscape visualizations
(b) Hessian max eigenvalue speetra
Figure 1: Two different aspects consistently show that MSAs flatten loss landscapes. Left: Loss landscape visualizations show that VIT has a flatter loss (NLL + Eg regularization) than ResNet. Right: Hessian eigenvalue spectra show that the magnitude of the Hessian eigenvalues of VIT is smaller than that of ResNet during training phases. We report the Hessian spectra at the end of the warmup phases, 100th 2h0 and 300 epochs. See Fig. for a more detailed analysis.
Guo et al., 2021; Srinivas et al., 2021), and ViTs predict well-calibrated uncertainty (Minderer et al., 2021). 22 ViTs are robust against data corruptions, image occlusions (Naseer et al., 2021), and Mao et al., 2021).
adversarial attacks (Shao et al., 2021; Bhojanapalli et al., 2021; Paul & Chen, 2022;
They are particularly robust against high-frequency noises (Shao et al., 2021). (3) MSAs closer to the last layer significantly improve predictive performance (Graham et al., 2021; Dai al., 2021).
These empirical observations raise immediate questions:
What properties of MSAs do we need to better optimize NNs? Do the long- long-range dependencies of MSAS help NNs learn? Do MSAs act like Convs? If not, how they different? How harmonize MSAs with Convs? Can we just leverage their advantages?
We provide an explanation of how MSAs work by addressing them a trainable spatial smoothing of feature maps, because Eq. (1) also suggests that MSAs average feature map values with the positive importance-weights, Even non-trainable spatial smoothings, such a small 2 x 2 box blur, help CNNs see better (Zhang, 2019; Park & Kim, 2021). These simple spatial smoothings not only improve accuracy but also robustness by spatially ensembling feature map points and flattening the loss landscapes (Park & Kim, 2021). Remarkably, spatial smoothings have the properties of MSAs See Appendix B for detailed explanations of MSAs spatial smoothing.
1.2 CONTRIBUTION
We address the three key questions:
What properties of MSAS do we need to improve optimization? We present various evidences to support that MSA is generalized spatial smoothing. It means that MSAs improve performance because their formulation Eq. (1)-is an appropriate inductive bias. Their weak inductive bias disrupts NN training. In particular, key feature of MSAS is their data specificity, not long-range dependency. As an extreme example, local MSAs with a 3 x 3 receptive field outperforms global MSA because they reduce unnecessary degrees of freedom.
How do MSAs improve performance? MSAs have their advantages and disadvantages. On the one hand, they flatten loss landscapes shown in Fig. 1. The flatter the loss landscape, the better the
2

---

Published as a conference paper at ICLR 2022
ResNet
VIT
-5.0
RecNie
A Log amplitude
A Log amplitude
1.00
No N depth
Vir
-2.0-
-1.0
-.73
A Accuracy
-7.5
0.50
-10.0
-4.0
Exb
-2.0
0.25
-12.5
-6.0
0.00
-15.0
0.0%t
0.5xt Frequency
1.0
0.0%
0.5xt
1.0xt
0.1xt 0,3 0.5x 0.7x 0.9
Frequency
Noise frequency
(a) Relative log amplitudes Fourier transformed feature maps.
(b) Robustness for noise frequency
The Fourier analysis shows that MSAs do not act like Convs. Left: Relative log ampli- tudes of Fourier transformed feature map show that VIT tends to reduce high-frequency signals, while ResNet amplifies them. A Log amplitude is the difference between the log amplitude at normalized frequency 0.0w (center) and at 1.0w (boundary). See Fig. 8 for more detailed analysis. Right: We measure the decrease in accuracy against frequency-based random noise. ResNet is vulnerable to high-frequency noise, while ViT is robust against them. We use frequency window size of 0.1m.
Figure 2:
performance and generalization (Li et al., 2018; Keskar et al., 2017; Santurkar et al., 2018; Foret et al., 2021; Chen et al., 2022). Thus, they improve not only accuracy but also robustness in large data regimes. On the other hand, MSAs allow negative Hessian eigenvalues in small data regimes. This means that the loss landscapes of MS As are non-convex, and this non-convexity disturbs NN optimization (Dauphin et al., 2014). Large amounts of training data suppress negative eigenvalues and convexify losses.
Do MSAs act like Convs? We show that MSAs and Convs exhibit opposite behaviors. MSAs aggregate feature maps, but Convs diversify them. Moreover, as shown in Fig. 2a, the Fourier analysis of feature maps shows that MSAs reduce high-frequency signals, while Convs, conversely, amplifies high-frequency components. other words, MSAs are low-pass filters, but Convs are high-pass filters. In addition, Fig. 2b indicates that Convs are vulnerable to high-frequency noise but that MSAs are not. Therefore, MSAs and Convs are complementary.
How can we harmonize MSAS with Convs? We reveal that multi-stage NNs behave like series connection of small individual models. Thus, applying spatial smoothing at the end of stage improves accuracy by ensembling transformed feature map outputs from each stage (Park & Kim, 2021) shown in Fig. 3a. Based on this finding, we propose an alternating paltern of Convs and MSAs. NN stages using this design pattern consists a number of CNN blocks and one (or a few) MSA block as shown in Fig. 3c. The design pattern naturally derives the structure of canonical Transformer, which has one MSA block per MLP block as shown in Fig. 3b. It also provides an explanation of how adding Convs to Transformer's MLP block improves accuracy and robustness (Yuan et al., 2021; Guo et al., 2021; Mao et al., 2021).
Surprisingly, models using this alternating pattern of Convs and MSAs outperform CNNs not only on large datasets but also on small datasets, such as CIFAR. This contrasts with canonical ViTs, models that perform poorly small amount of data. It implies that MSAs are generalized spatial smoothings that complement Convs, not simply generalized Convs.
2 WHAT PROPERTIES OF MSAS Do WE NEED To IMPROVE OPTIMIZATION?
To understand the underlying nature of MSAs, we investigate the properties of the ViT family: e.g.. vanilla ViT (Dosovitskiy et al., 2021); PiT (Heo et al., 2021), which is "VIT multi-stage"; and Swin (Liu et al., 2021), which is "ViT multi-stage + local MSA". This section shows that these additional inductive biases enable ViTs to learn strong representations. We also use ResNet (He et al., 2016a) for comparison. NNs are trained from scratch with DeiT-style data augmentation (Touvron al., 2021) for 300 epochs. The NN training begins with a gradual warmup (Goyal et al., 2017) for 5 epochs. Appendix A provides more detailed configurations and background information for experiments.
3

---

Published as a conference paper at ICLR 2022
ResNet Blocks
MLP Block
ResNet Blocks L.
2848, 1x1,
2848, 1-1,
512, 1-1, 2048
512, 3x3, 512
512, 3x3, 512
2848, 1-1, 512
512, 1s1, 2048
512, 3x3, 2048
Attention Block
Smoothing Block
Attention Block
2048, 1+1,
MS4, DNH
AvgPool
512, NSA, DKH
DHH, 1x1, 512
DHH, 1w1, 2048
Subsampling +
Suhsampling
(a) Spatial smoothing
(b) Canonical Transformer (c) Alternating pattern (ours)
Figure 3: Comparison of three different repeating patterns. Left: Spatial smoothings are located at the end of CNN stages, Middle: The stages of ViTs consist of repetitions of canonical Transformers. "D" is the hidden dimension and "H" is the number of heads. Right: The stages using alternating pattern consists of number of CNN blocks and an MSA block. For more details, see Fig. 11.
The stronger the inductive biases, the stronger the representations (not regularizations). Do models with weak inductive biases overfit training datasets? To address this question, we provide two criteria on CIFAR-100: the error of the test dataset and the cross-entropy, or the negative log-likelihood, of the training dataset (NLLtaiin the lower the better). See Fig. 5a for the results.
Contrary to our expectations, experimental results show that the stronger the inductive bias, the lower both the test error and the training NLL. This indicates that ViT does not overfit training datasets. In addition, appropriate inductive biases, such as locality constraints for MSAs, helps NNs learn strong representations. We also observe these phenomena on CIFAR- 10 and ImageNet shown in Fig. C.1. Figure C.2 also supports that weak inductive biases disrupt NN training. In this experiment, extremely small patch sizes for the embedding hurt the predictive performance of VIT.
VIT does not overfit small training datasets. We observe that VIT does not overfit even on smaller datasets. Figure 5b shows the test error and the training NLL of VIT on subsampled datasets. In this experiment, the size of the dataset decreases, the error increases expected, but surprisingly, NLLusin also increases. Thanks to the strong data augmentation, ViT does not overfit even on a dataset size of 2%. This suggests that ViT's poor performance in small data regimes is not due to overfitting.
BeiBll
warmup th
WIT
(6R)
VIT's non-convex losses lead to poor performance.
Density
Density
How do weak inductive biases of MSAs disturb the optimization? A loss landscape perspective provides an explanation: the loss function of VIT is non-convex, while that of ResNet is strongly neeare)convex. This poor loss disrupts NN training (Dauphin al., 2014), especially in the early phase of training (Jastrzebski et al., 2020; 2021). Figure b and Fig. provide t largest Hessian eigenvalue densities (Park& & Kim, 2021) with batch size of 16. The figures show that ViT has a number of negative Hessian cigenvalues, while ResNet only has few.
-100
Figure 4: Hessian max eigenvalue spec- tra show that MSAS have their advan- tages and disadvantages. The dotted line is the spectrum of VIT using 6% dataset for training. Left: ViT has number of neg- ative Hessian eigenvalues, while ResNet only has few. Right: The magnitude of VIT's positive Hessian eigenvalues is small. See also Fig. lb for more results.
Figure 4 also shows that large datasets suppress nega- tive Hessian eigenvalues in the early phase of training. Therefore, large datasets tend to help ViT learn strong representations by convexifying the loss. ResNet enjoys little benefit from large datasets because its loss is convex even on small datasets.
4

---

Published as a conference paper at ICLR 2022
Ertor NLLmvi
80-
30
2.4
NLLaaa
Error (%)
3.5
NLaaa
20
2.2
RX
R
Twins Swin PiT
VIT Mixer
10
10
CNN
Local MSA Global MSA
Dataset size (img/ class)
(a) Error and NLLmin each model.
(b) Performance of VIT dataset size.
Figure 5: VIT does not overfit training datasets. "R" is ResNet and "RX" is ResNeXt. Left: Weak inductive bias disturbs NN optimization. The lower the NLLuai the lower the error. Right: The lack of dataset also disturbs NN optimization.
Loss landscape smoothing methods aids in VIT train- ing. Loss landscape smoothing methods can also help VíT learn strong representations. In classification tasks, global average pooling (GAP) smoothens the loss land- scape by strongly ensembling feature map points (Park Kim, 2021). We demonstrate how the loss smoothing method can help ViT improve performance by analyz- ing VIT with GAP classifier instead of CLS token on CIFAR-100.
VIT GLS
w warmup
VIT GAP
Density
Density
-100
100
200
Figure 6 shows the Hessian max eigenvalue spectrum of the ViT with GAP. As expected, the result shows that GAP classifier suppresses negative Hessian max eigen- values, suggesting that GAP convexify the loss. Since negative eigenvalues disturb NN optimization, GAP clas- sifier improve the accuracy by +2.7 percent point.
Figure 6: GAP classifier suppresses neg- ative Hessian max eigenvalues in an early phase of training. We present Hes- sian max eigenvalue spectrum of ViT with GAP classifier instead of CLS token.
Likewise, Sharpness- Aware Minimization (SAM) (Foret et al., 2021), an optimizer that relies on the local smoothness of the loss function, also helps NNs seek out smooth minima. Chen et (2022) showed that SAM improves the predictive performance of VIT.
MSAS flatten the loss landscape. Another property of MSAs is that they reduces the magnitude of Hessian eigenvalues. Figure 1b and Fig. 4 shows that the eigenvalues of VIT significantly smaller than that of CNNs. While large eigenvalues impede NN training (Ghorbani et al.. 2019), MSAs can help NNs learn better representations by suppressing large Hessian eigenvalues. Figure la also support this claim. In Fig. 1a, we visualize the loss landscapes by using filter normalization (Li al., 2018), and the loss landscape of ViT is flatter than that of ResNet. In large data regimes, the negative Hessian eigenvalues the disadvantage of MSAs disapeisapperrs, and only their advantages remain. As a result, ViTs outperform CNNs on large datasets, such as ImageNet and JFT (Sun et al., 2017). PiT and Swin also flatten the loss landscapes. For more details, see Fig. C.4.
A key feature of MSAs is data specificity (not long-range dependency). The two distinguishing features of MSAs long-range dependency and data specificity, also known as data dependency, as discussed in Section 1.1. Contrary to popular belief. the long-range dependency hinders NN optimization. To demonstrate this, we analyze convolutional VIT. which consists of voo-dimensional convolutional MSAs (Yang et al., 2019) instead of global MSAs. Convolutional MSAs calculates self-attention only between feature map points in convolutional receptive fields after unfolding the feature maps in the same way convolutions.
Figure 7a shows the error and NLLarann convolutional ViTs with kernel sizes of 3 x 3, 5 5, and 8 x 8 (global MSA) on CIFAR-I00. In this experiment, x 5 kernel outperforms 8 x 8 kernel both the training and the test datasets. NLLaqain of 3 kernel is worse than that x kernel, but better than that of global MSA. Although the test accuracies of x 3 and 5 kernels comparable, the robustness of x kernel significantly better than that 3 x 3 kernel on CIFAR-100 (Hendrycks & Dietterich, 2019).
5

---

Published as a conference paper at ICLR 2022
29.0
watmuqp
2.46
Error (%)
28.6
Density
Density
2.44
27.8
-240
-100
Kernel
Max Eigenvalue
Max Eigenvalue
(a) Error and a of with local MSA for kernel size
(h) Hessian negative and positive max eigen- value spectra in early phase of training
Figure 7: Locality constraint improves the performance of VIT. We analyze the VIT with con- volutional MSAs. Convolutional MSA with 8 kernel is global MSA. Left: Local MSAs learn stronger representations than global MSA. Right: Locality inductive bias suppresses the negative Hessian eigenvalues, i.e. local MSAs have convex losses.
Figure 7b shows that the strong locality inductive bias not only reduce computational complexity as originally proposed (Liu et al., 2021), but also aid in optimization by convexifying the loss landscape. x kernel has fewer negative eigenvalues than global MSA because it restricts unnecessary degrees of freedom. 5 x kernel also has fewer negative eigenvalues than 3 x 3 kernel because it ensembles larger number of feature map points (See also Fig. 6). The amount of negative eigenvalues is minimized when these two effects are balanced.
is clear that data specificity improves NNs. MLP-Mixer (Tolstikhin et al., 2021; Yu al., 2021a), a model with an MLP kernel that does not depend on input data, underperforms compared to ViTs. Data specificity without self self-attention (Bello, 2021) improves performance.
3 Do MSAs ACT LIKE CONVS?
Convs data-agnostic and channel-specifie. In contrast, MSAs data-specific and channel- agnostic. This section shows that these differences lead to large behavioral differences. suggests that MSAs and Convs are complementary.
MSAs are low-pass filters, but Convs are high- pass filters. As explained in Section 1.1, MSAs spatially smoothen feature maps with self-attention importances. Therefore, we expect that MSAs will tend to reduce high-frequency signals. See Ap- pendix B for more detailed discussion.
Log amplitude
-2.55
-2.50
Figure shows the relative log amplitude (A log am- plitude) of VIT's Fourier transformed feature map high-frequency (1.0w) on ImageNet. In this figure, MSAs almost always decrease the high-frequency amplitude, and MLPs-ccrresponding to Convs- increase The only exception the early stages of the model. In these stages, MSAs behave like Convs, i... they increase the amplitude. This could serve as evidence for hybrid model that uses Convs in early stages and MŠAs late stages (Guo et al., 2021: Graham et al., 2021; Dai et al., 2021; Xiao al., 2021; Srinivas al., 2021).
2.75
0.2
0.4
10
Normalized depth
Figure 8: MSAs (gray area) generally duce the high-frequency component of fea- ture map, and MLPs (white area) amplify it. This figure provides A log amplitude of VIT figh-frequency (1.0z). See also Fig, 2a and Fig. D.2 for more results.
Based on this, we can infer that low-frequency signals and high-frequency signals are informative to MSAs and Convs, respectively. In support of this argument, report the robustness of VIT and ResNet against frequency- based random noise. Following Shao et al. (2021) and Park Kim random noise
(2021). measure the decrease in accuracy with respect to data with frequency- Enoise = Zo + J-1 (F(6) O M,). where &p is clean data, F(.) and (.) are Fourier transform and inverse Fourier transform, ó is Gaussian random noise, and M, is frequency mask.

---

Published as a conference paper at ICLR 2022
ResNet
V77
Feature map variance
ueature map variance
0.10
0.6
0.06
0.4
0.04
0.2
0.02
0.0
0.00
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Normalized depth
Normalized depth
Figure 9: MSAs (gray area) reduce the variance of feature map points, but Convs (white area) increase the variance. The blue area subsampling layer. This result implies that MSAs ensemble feature maps, but Convs do not.
As expected, the result in Fig. 2b reveals that ViT and ResNet are vulnerable to low-frequency noise and high-frequency noise, respectively. Low- Low-frequency signals and the high-frequency signals each correspond to the shape and the texture of images. The results thus suggests that MSAs are shape-biased (Naseer et al., 2021), whereas Convs are texture-biased (Geirhos et al., 2019).
MSAs aggregate feature maps, but Convs do not. Since MSAs average feature maps, they will reduce variance of feature map points. This suggests that MSAs ensemble feature maps (Park & Kim, 2021). To demonstrate this claim, we measure the variance of feature maps from NN layers.
Figure 9 shows the experimental results of ResNet and VIT. This figure indicates that MSAs in ViT tend to reduce the variance; conversely, Convs in ResNet and MLPs in ViT increase it. In conclusion, MSAs ensemble feature map predictions, but Convs do not. As Park & Kim (2021) figured out, reducing the feature map uncertainty helps optimization by ensembling and stabilizing the transformed feature maps. See Fig. D.I for more results on PiT and Swin.
We observe two additional patterns for feature map variance. First, the variance accumulates in every NN layer and tends to increase as the depth increases. Second, the feature map variance ResNet peaks at the ends of each stage. Therefore, we can improve the predictive performance of ResNet by inserting MSAs the end of each stage. Furthermore, we also can improve the performance by using MSAs with large number of heads in late stages.
4 How CAN WE HARMONIZE MSAS WITH CONVS?
Since MSAs and Convs are complementary, this section seeks to design a model that leverages only the advantages of the two modules. To this end, we propose the design rules described Fig. 3c, and demonstrate that the models using these rules outperforms CNNs, not only in the large data regimes but also the small data regimes, such as CIFAR.
4.1 DESIGNING ARCHITECTURE
We first investigate the properties of multi-stage NN architectures. Based this investigation, come to propose an alternating pattern, i.c., a principle for stacking MSAs based on CNNs.
Multi-stage NNs behave like individual models. Fig. we observe that the pattern of feature map variance repeats itself at every stages. This behavior is also observed in feature map similarities and lesion studies.
Figure 10a shows the representational similarities of ResNet and Swin on CIFAR-100. In this experiment, we use mini-batch CKA (Nguyen et al., 2021) to measure the similarities. As Nguyen et al. (2021) figured out, the feature map similarities of CNNs have a block structure. Likewise, observe that the feature map similarities of multi-stage ViTs, such as PiT and Swin, also have a block structure. Since vanilla ViT does not have this structure (Bhojanapalli et al., 2021; Raghu et al., 2021), the structure is an intrinsic characteristic multi-stage architectures. See Fig. D.3 for more detailed results of VIT and PIT.
7

---

Published as a conference paper at ICLR 2022
ResNet
ResNet
Figure 10: Multi-stage CNNs and VITs behave like a series connection of small individual models. Left: The feature map similar- ities show the block structure of ResNet and Swin. "E" stands for stem/embedding and "P" for pooling (sub- sampling) layer. Right: We measure decrease in accuracy after removing one unit from the trained model. Accuracy changes periodically, and this period is one stage. White, gray, and blue areas are Conv/MLP, MSA, and subsampling layers, respectively.
P
A Accuracy (5)
-10-
0.75
1.00
P
Normalized depth
Swin
Swin
Accuracy (8)
P
E
E
0.00
0.25
Normalized depth
0.50
0.75
1.00
(a) Feature map similarity
(b) Accuracy of one-unit-removed model.
Figure 10b shows the results of lesion study (Bhojanapalli et al., 2021), where one NN unit is removed from already trained ResNet and Swin during the testing phase. In this experiment, remove one 3 x 3 Conv layer from the bottleneck block of ResNet, and one MSA or MLP block from Swin. In ResNet, removing an early stage layers hurts accuracy more than removing late stage layers. More importantly, removing layer at the beginning of a stage impairs accuracy more than removing layer at the end of stage. The case of Swin even more interesting. At the beginning of stage, removing an MLP hurts accuracy. At the end of stage, removing an MSA seriously impairs the accuracy. These results are consistent with Fig. 8. See Fig. D.4 for the results on VIT and PÝT.
Based on these findings, we expect MSAs closer to the end of a stage to significantly improve the predictive performance. This is contrary the popular belief that MSAs closer the end of a model improve the performance (Srinivas et al., 2021; d' Ascoli et al., 2021; Graham et al., 2021; Dai et al., 2021).
Build-up rule.
Considering all the insights, we propose the following design rules:
Alternately replace Conv blocks with MSA blocks from the end baseline CNN model.
If the added MSA block does not improve predictive performance, replace a Conv block located the end of earlier stage with an MSA block
Use more heads and higher hidden dimensions for MSA blocks in late stages.
We call the model that follows these rules AlterNer. AlterNet unifies ViTs and CNNs by adjusting the ratio of MSAs and Convs as shown in Fig. 3. Figure 11 shows AlterNet based on pre-activation ResNet-50 (He al., 201 for CIFAR-100 an example. Figure D.5 shows AlterNet for ImageNet.
Figure 12a reports the accuracy of Sier-RRssNet-O0 which replaces the Conv blocks in ResNet-50 with local MSAs (Liu al., 2021) according to the aforementioned rules, on CIFAR- 1IFAR-100.. As expected, MSAs in the last stage (c4) significantly improve the accuracy, Surprisingly, an MSA in 2nd stage (c2) improves the accuracy, while two or more MSAs the 3rd stage (c3) reduce it. In conclusion, MSAs at the end of a stage play an important role in prediction.
Figure 12c demonstrates that MSAs suppress large eigenvalues while allowing only few negative eigenvalues. As explained in Fig. 4, large datasets compensate for the shortcomings of MSAs. Therefore, more data allows more MSAs for a models.
8

---

Published as a conference paper at ICLR 2022
Stage 1 (ct)
Stage 2 (e2)
Stage 3 (c3)
Stage (c4)
Classifier
Sreee block
Conv block
Conv black
Conv black
Conv black
Conv block
Conv black
MSA block
Conv black
Conv black
Conv block
Conv black
Carv black
MSA block
Conv black
MSA block
Conw block
MSA block
Input
Gol
MLP
Figure 11: Detailed architecture of Aiter-ReNNet-50 for CIFAR-100. White, gray, and blue blocks mean Conv, MSA, and subsampling blocks. All stages (except stage 1) end with MSA blocks. This model based on pre-activation ResNet-50. Following Swin, MSAs in stages to have 3, 6, 12, and 24 heads, respectively.
Accuracy (%)
84.0
c4
65.0
Alter ResNet
Twins
ResNet XXX
RoaNe
AltetNet
warmup
83.5
Robustness
60.0
Swin
Density
83.0
55.0
VT
50.0
82.5
Marer
70.0
80.0
2000
MSA number
Accuracy (%)
Max cigenvalue
(a) Accuracy of AlterNet for MSA number
(b) Accuracy and robustness in (c) Hessian eigenvalue spectra small data regime (CIFAR-100)
an early phase of training
Figure 12: AlterNet outperforms CNNs and ViTs. Left: MSAs the late of the stages improve accuracy. We replace Convs of ResNet with MSAs one by one according to the build-up rules, c1 c4 stands for the stages. Several MSAs in c3 harm the accuracy, but the MSA the end of c2 improves it. Center: AlterNet outperforms CNNs even in small data regime. Robustness is mean accuracy on
"RX" ResNeXt. Right: MSAs AlterNet suppress the large eigenvalues; i.e., AlterNet has flatter loss landscape than ResNet in the early phase of training.
4.2 PERFORMANCE
Figure 12b shows the accuracy and corruption robustness of Alter-ResNet-50 and other baselines on CIFAR- and CIFAR-100-C. Since CIFAR small dataset, CNNs outperforms canonical ViTs. Surprisingly, Alter-ResNet
model with MSAs following the appropriate build-up rule- outperforms CNNs even in the small data regimes. This suggests that MSAs complement Convs. In the same manner, this simple modification shows competitive performance on larger datasets, such as ImageNet. See Fig. E. for more details.
5 DISCUSSION
Our present work demonstrates that MSAs are not merely generalized Convs, but rather general- ized spatial smoothings that complement Convs. MSAs help NNs learn strong representations by ensembling feature map points and flattening the loss landscape.
Since the main objective of this work is investigate the nature of MSA for computer vision, preserve the architectures of Cony and MSA blocks in AlterNet. Thus, AltenNet has a strong potential for future improvements. In addition, AlterNet can conveniently replace the backbone for other vision tasks such as dense prediction (Carion et al., 2020). As Park & Kim (2021) pointed out, global average pooling (GAP) for simple classification tasks has strong tendency ensemble feature maps, but NNs for dense prediction do not use GAP. Therefore. we believe that MSA to be able to significantly improve the results in dense prediction tasks by ensembling feature maps. Lastly, strong data augmentation for MSA training harms uncertainty calibration as shown in Fig. F.Ia. We leave detailed investigation for future work.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGEMENT
We thank the reviewers, Tacoh Kim, and Pilhyeon Lee for valuable feedback. This work was supported by the Samsung Science and Technology Foundation under Project Number SSTF-BA1501-52.
REPRODUCIBILITY STATEMENT
To ensure reproducibility, we provide comprehensive resources, such as code and experimental details. Appendix A.
The code is available at https:/ //t::///iiii giggtttt....... com/xxxnell htwp:/hwwwwitteeeeeeeeeeeeeeeeee provides the specifications of all models used in this work. Detailed experimental setup including hyperparameters and the structure of AlterNet are also available in Appendix A. and Appendix De-facto image datasets are used for all experiments as described in Appendix A.1.
REFERENCES
Irwan Bello. Lambdanetworks: Modeling long-range interactions without attention. In International Conference on Learning Representations, 2021.
Irwan Bello, Barret Zoph, Ashish Vaswani, Jonathon Shlens, and Quoe V Le. Attention augmented convolutional networks, In International Conference on Computer Vision, 2019.
Irwan Bello, William Fedus, Xianzhi Du, Ekin Dogus Cubuk, Aravind Srinivas, Tsung- Yi Lin, Jonathon Shlens, and Barret Zoph. Revisiting resnets: Improved training and scaling strategies. In Advances in Neural Information Processing Systems, 2021.
Srinadh Bhojanapalli, Ayan Chakrabarti, Daniel Glasner, Daliang Li, Thomas Unterthiner, and Andreas Veit. Understanding robustness of transformers for image classification. International Conference on Computer Vision, 2021.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European Conference Computer Vision, 2020.
Xiangning Chen. Cho-Jui Hsich, and Boqing Gong. When vision transformers outperform resnets without pretraining or strong data augmentations. In International Conference on Learning Representations, 2022.
Xiangxiang Chu, Zhi Tian, Yuqing Wang, Bo Zhang, Haibing Ren, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Twins: Revisiting the design of spatial attention in vision transformers. In Advances in Neural Information Processing Systems, 2021.
Jean-Baptiste Cordonnier, Andreas Loukas, and Martin Jaggi. On the relationship between self- attention and convolutional layers. In International Conference on Learning Representations, 2020.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with a reduced search space. Advances Neural Information Processing Systems, 2020.
Zihang Dai, Hanxiao Liu, Quoc Le, and Mingxing Tan. Coatnet: Marrying convolution and attention for all data sizes. In Advances in Neural Information Processing Systems, 2021.
Stéphane Ascoli, Hugo Touvron, Matthew Leavitt, Ari Morcos, Giulio Biroli, and Levent Sagun. Convit: Improving vision transformers with soft convolutional inductive biases. In International Conference on Machine Learning, 2021.
Yann N Dauphin, Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, Surya Ganguli, and Yoshua Bengio. Identifying and attacking the saddle point problem in nigh-dimensional non-convex optimization. In Advances in Neural Information Processing Systems, 2014.
10

---

Published as a conference paper at ICLR 2022
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization. International Conference on Learning Representations.
2021.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and
Wieland Brendel. Imagenet-trained enns are biased towards texture; increasing shape bias improves accuracy and robustness. In International Conference on Learning Representations, 2019.
Behrooz Ghorbani, Shankar Krishnan, and Ying Xiao. An investigation into neural net optimization via hessian eigenvalue density, In International Conference on Machine Learning, 2019.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in hour. arXiv preprint arXiv: 700..006777777 2017.
Ben Graham, Alaaeldin El-Nouby, Hugo Touvron, Pierre Stock, Armand Joulin, Hervé Jégou, and Mathhis Douze. Levit: a vision transformer in convnet's clothing for faster inference. In International Conference on Computer Vision, 2021.
Jianyuan Guo, Kai Han, Han Wu, Chang Xu, Yehui Tang, Chunjing Xu, and Yunhe Wang. Cmt: Convolutional neural networks meet vision transformers, arXiv preprint arXiv:2107.06263. 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference Computer Vision and Puttern Recognition, 2016a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European Conference on Computer Vision, 2016b.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In International Conference on Learning Representations, 2019.
Bycongho Heo. Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Junsuk Choe, and Scong Joon Oh. Rethinking spatial dimensions of vision transformers. In International Conference on Computer Vision, 2021.
Jiri Hron, Yasaman Bahri, Jascha Sohl-Dickstein, and Roman Novak. Infinite attention: Nugp and ntk for deep attention networks. In International Conference on Machine Learning, 2020.
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. European Conference on Computer Vision, 2016.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. In Advances in Neural Information Processing Systems, 2018.
Stanislaw Jastrzebski, Maciej Szymezak, Stanislav Fort, Devansh Arpit, Jacek Tabor, Kyunghyun Cho, and Krzysztof Geras. The break-even point on optimization trajectories of deep neural networks. In International Conference on Learning Representations, 2020.
Stanislaw Jastrzebski, Devansh Arpit, Oliver Astrand, Giancarlo B Kerg, Huan Wang. Caiming Xiong, Richard Socher, Kyunghyun Cho, and Krzysztof Geras, Catastrophic fisher explosion: Early phase fisher matrix impacts generalization. In International Conference on Machine Learning. 2021.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. In International Conference Learning Representations, 2017.
Alex Krizhevsky, Geoffrey Hinton, al. Leaming multiple layers of features from tiny images. 2009.

---

Published as a conference paper at ICLR 2022
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. In Advances in Neural Information Processing Systems, 2018.
Chaoyue Liu, Libin Zhu, and Mikhail Belkin. On the linearity of large non non-linear models: when and why the tangent kernel is constant. In Advances in Neural Information Processing Systems, 2020.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In International Conference on Computer Vision, 2021.
Ilya Loshchilov and Frank Hutter. Sgdr: fional Conference on Learning Representations, 2017.
Stochastic gradient descent with warm restarts. In Interna-
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2019.
Xiaofeng Mao, Gege Qi, Yuefeng Chen, Xiaodan Li, Ranjie Duan, Shaokai Ye, Yuan He, and Hui Xue. Towards robust vision transformer. arXiv preprint 1XXx::2005.079266, 2021.
Paul Michel, Omer Levy, and Graham Neubig. Are sixteen heads really better than one? In Advances in Neural Information Processing Systems, 2019.
Matthias Minderer, Josip Djolonga, Rob Romijnders, Frances Hubis, Xiaohua Zhai, Neil Houlsby, Dustin Tran, and Mario Lucic. Revisiting the calibration of modern neural networks. In Advances in Neural Information Processing Systems, 2021.
Muzammal Naseer, Kanchana Ranasinghe, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming- Ming-Hsuan Yang. Intriguing properties of vision transformers. In Advances in Neural Information Processing Systems, 2021.
Thao Nguyen. Maithra Raghu, and Simon Kornblith. Do wide and deep networks learn the same things? uncovering how neural network representations vary with width and depth. In International Conference Learning Representations, 2021.
Namuk Park and Songkuk Kim. Blurs behave like ensembles: Spatial smoothings to improve accuracy, uncertainty, and robustness. arXiv preprint arXiv:2 / 2021.
Namuk Park, Taekyu Lee, and Songkuk Kim. Vector quantized bayesian neural network inference for data streams. Proceedings of the AAAI Conference on Artificial Intelligence, 2021.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems, 2019.
Sayak Paul and Pin- Yu Chen. Vision transformers are robust learners. In Proceedings of the AAAI Conference on Artificial Intelligence, 2022.
Maithra Raghu, Thomas Unterthiner, Simon Korblithh, Chiyuan Zhang, and Alexey Dosovitskiy. Do vision transformers see like convolutional neural networks? In Advances in Neural Information Processing Systems, 2021.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 2015.
Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, and Aleksander Mądry. How does batch normal- ization help optimization? In Advances in Neural Information Processing Systems, 2018.
Rulin Shao, Zhouxing Shi, Jinfeng Yi, Pin-Yr Yu Chen, and Cho-Jui Hsich. On the adversarial robustness of visual transformers. arXiv preprint arXiv: /5670, 2021.
12

---

Published as a conference paper at ICLR 2022
Aravind Srinivas, Tsung- Yi Lin, Niki Parmar, Jonathon Shlens, Pieter Abbeel, and Ashish Vaswani. Bottleneck transformers for visual recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2021.
Andreas Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. How to train your vit? data, augmentation, and regularization in vision transformers. arXiv preprint arXiv:2 10270, 2021.
Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable effectiveness of data in deep learning era. In International Conference on Computer Vision, 2017.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016.
Ilya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung. Daniel Keysers, Jakob Uszkoreit, Mario Lucic, et al. Mip-mixer: An all-mip architecture for vision. In Advances in Neural Information Processing Systems, 2021.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning, 2021.
Shikhar Tuli, Ishita Dasgupta, Erin Grant, and Thomas L Griffiths. Are convolutional neural networks or transformers more like human vision? In Proceedings of the Annual Meeting of the Cognitive Science Society, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, 2017.
Peihao Wang, Wenqing Zheng, Tianlong Chen, and Zhangyang Wang. Scaling the depth of vi- sion transformers via the fourier domain analysis. In International Conference on Learning Representations, 2022.
Phil Wang. Implementation of vision transformer. vit-pytorch, 2021.
https://ggttubb..om/ucidad
Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018.
Yeming Wen, Ghassen Jerfel, Rafael Muller, Michael W Dusenberry, Jasper Snock, Balaji Laksh- minarayanan, and Dustin Tran. Combining ensembles and data augmentation can harm your calibration. In International Conference on Learning Representations, 2021.
Ross Wightman. Pytorch image models.
pytorch- image-models. 2019.
Tete Xiao, Mannat Singh, Eric Mintun, Trevor Darrell, Piotr Dollár, and Ross Girshick. Early convolutions help transformers see better. In Advances in Neural Information Processing Systems, 2021.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017.
Weijian Xu, Yifan Xu, Tyler Chang, and Zhuowen Tu. Co-scale conv-attentional image transformers. In International Conference on Computer Vision, 2021.
Baosong Yang, Longyue Wang, Derek Wong, Lidia Chao, and Zhaopeng Tu. Convolutional self-attention networks. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), 2019.
13

---

Published as a conference paper at ICLR 2022
Zhewei Yao, Amir Gholami, Kurt Keutzer, and Michael W Mahoney. Pyhessian: Neural networks through the lens of the hessian. 2020 IEEE International Conference on Big Data (Big Data), 2020.
Dong Yin, Raphael Gontijo Lopes, Jon Shlens, Ekin Dogus Cubuk, and Justin Gilmer. A fourier perspective on model robustness in computer vision. In Advances in Neural Information Processing Systems, 2019.
Tan Yu, Xu Li, Yunfeng Cai, Mingming Sun, and Ping Li. Rethinking token- loken-mixing mip for mip-based vision backbone. In BMVC, 2021a.
Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang, Jiashi Feng, and Shuicheng Yan. Metaformer is actually what you need for vision. arXiv preprint arXiv:2111.1418, 2021b.
Kun Yuan, Shaopeng Guo, Ziwei Liu, Aojun Zhou, Fengwei Yu, and Wei Wu. Incorporating convolution designs into visual transformers. In International Conference on Computer Vision, 2021.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In International Conference on Computer Vision, 2019.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference Learning Representations, 2018.
Richard Zhang. Making convolutional networks shift-invariant again. International Conference on Machine Learning, 2019.
Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmenta- tion. In Proceedings of the AAAI Conference on Artificial Intelligence, 2020.
14

---

Published as a conference paper at ICLR 2022
A EXPERIMENTAL DETAILS
This section provides experimental details, e.g., setups and background information.
A.1 SETUPS
We obtain the main experimental results from two sets of machines for CIFAR (Krizhevsky et al., 2009). The first set consists of an Intel Xeon W-2123 Processor, 32GB memory, and a single GeForce RTX 2080 Ti, and the other set of four Intel Intel Broadwell CPUs, 15GB memory, and a single NVIDIA T4. For ImageNet (Russakovsky et al., 2015), we use AMD Ryzen Threadripper 3960X 24-Core Processor, 256GB memory, and four GeForce RTX 2080 Ti. NN models are implemented in PyTorch (Paszke et al., 2019).
We train NNs using categorical cross-entropy (NLL) loss and AdamW optimizer (Loshchilov & Hutter, 2019) with initial learning rate of 1.25 x 10-4 and weight decay of 5 x 10-2 We also use cosine annealing scheduler (Loshchilov & Hutter, 2017). NNs trained for 300 epochs with a batch size of 96 on CIFAR, and a batch size of 128 on ImageNet. The learning rate is gradually increased (Goyal et al., 2017) for epochs. Following Touvron al. (2021), strong data augmentations -uuh as RandAugmen (Cubuk et al., 2020), Random Erasing (Zhong et al., 2020), label smoothing (Szegedy al., 2016), mixup (Zhang et al., 2018), and CutMix (Yun et al.. 2019) are used for training. Stochastic depth (Huang et al., 2016) is also used to regularize NNs. This DeiT-style configuration, which significantly improves the performance (Steiner et al., 2021; Bello et al., 2021), is the de facto standard in VIT training (See, e.g., (Heo et al., 2021; Liu et al., 2021)). Therefore, we believe the insights presented in this paper can be used widely, See source code (https://gitbe
com/xxxnelll.. h/www-oo--ooooo vir--work)) for detailed configurations.
We mainly report the performances of ResNet-50, ViT-Ti, PiT-Ti, and Swin-Ti. Their training throughputs on 110 are 320, 434, 364, and 469 image/sec, respectively, which are comparable to each other. Figures 5a and C. report the predictive performance of ResNeXt-50 (Xie et al., 2017), Twins-S (Chu et al., 2021), and MLP-Mixer-Ti (Tolstikhin et al., 2021). Figure E. additionally reports the performance of ViT-IT (d' Ascoli et al., 2021), LeV----28S (Graham et al., 2021), and CoaT-Lite-Ti (Xu et al., 2021). We use patch size of 2 for ViT and PiT on CIFAR; for Swin, patch size of x 1 and a window size of 4 x 4. We use patch size of 4 X 4 for ViT only in Fig. 7. We halve the depth of the ViT in Fig. C.5 and Fig. C.6 due to the memory limitation.
All models for CIFAR, and ResNet, ViT, and AlterNet for ImageNet are trained from scratch. We use pertained PiT and Swin from Wightman (2019) for ImageNet. The implementations of Vision Transformers are based on Wightman (2019) and Wang (2021).
For Hessian max eigenvalue spectrum (Park & Kim, 2021), 10% of the training dataset is used. We also use power iteration with a batch size of 16 to produce the top-5 largest eigenvalues. To this end, we use the implementation of Yao al. (2020). We modify the algorithm to calculate the eigenvalues with respect to &2 regularized NLL on augmented training datasets. In the strict sense, the weight decay is not l regularization, but we neglect the difference.
For the Fourier analysis and the feature map variance experiment, the entire test dataset is used. We report the amplitudes and the variances averaged over the channels.
A.2 BACKGROUND INFORMATION
Below are the preliminaries and terms of our experiments.
Test error and training NL... We report test errors on clean test datasets and training NLL.s on augmented training datasets in experiments, c.g., Fig. 5 and Fig. NLL is an appropriate metric for evaluating convergence on a training dataset because an NN optimizes NLL. In addition, it is the most widely used as a proper scoring rule indicating both accuracy and uncertainty. To represent predictive performance on a test dataset, we use a well-known metric: error. Although NLL can also serve the same purpose, results are consistent even when NLL is employed.
If an additional inductive bias or a learning technique improves the performance of an NN, this is either method to help the NNs leam "strong representations", or method to "regularize" it.
15

---

Published as a conference paper at ICLR 2022
An improved-ice lower-traiie NLL suggests that this bias technique helps the NN learn strong representations. Conversely, a compromised training NLL indicates that the bias or technique regularizes the NN. Likewise, say that "an NN overfits training dataset" when a test error is compromised as the training NLL is improved.
Hessian max eigenvalue spectrum. Park & Kim (2021) proposed " "Hessian max eigenvalue spec- tra", feasible method for visualizing Hessian eigenvalues of large-sized NNs for real-world prob- lems. It calculates and gathers top-k Hessian cigenvalues by using power iteration mini-batch wisely, Ghorbani al. (2019) visualized the Hessian eigenvalue spectrum by using the Lanczos quadrature algorithm for full batch. However, this is not feasible for practical NNs because the algorithm requires a lot of memory and computing resources.
A good loss landscape is a flat and convex loss landscape. Hessian eigenvalues indicate the flatness and convexity of losses. The magnitude of Hessian eigenvalues shows sharpness, and the presence of negative Hessian eigenvalues shows non-convexity. Based on these insights, we introduce negative max eigenvalue proportion (NEP, the lower the better) and an average of positive max eigenvalues (APE, the lower the better) to quantitatively measure the non-convexity and the sharpness, respec- tively. For a Hessian max eigenvalue spectrum p()), NEP is the proportion of negative eigenvalues [ P(A) dλ, and APE is the expected value of positive eigenvalues Jo Xp(A) dx So p(A) dX. We use these metrics in Fig. C.5 and Fig. C.6.
Note that measuring loss landscapes and Hessian eigenvalues without considering regularization on clean datasets would lead to incorrect results, since NN training optimizes l regularized NLL on augmented training datasets-not NLL on clean training datasets. We visualize loss landscapes and Hessian eigenvalues with respect to "Eg regularized NLL loss" "augmented training datasets".
Fourier analysis of feature maps. We analyze feature maps in Fourier space to demonstrate that MSA is a low-pass filter as shown in Fig. 2, Fig. 8, and Fig. D.2. Fourier transform converts feature maps into frequency domain. We represent these converted feature maps normalized frequency domain, so that the highest frequency components are at f = { (- ग, +T). and the lowest frequency components are at f = O. We mainly report the amplitude ratio of figh-frreqeecy components and low-frequency components by using A log amplitude, the difference in log amplitude at f = π and O. Yin et al. (2019) also analyzed the robustness of NNs from a Fourier perspective, but their research focused on input images not feature maps in Fourier spaces.
B MSAS BEHAVE LIKE SPATIAL SMOOTHINGS
As mentioned in Section 1.1, spatial smoothings before subsampling layers help CNNs see better (Zhang, 2019; Park & Kim, 2021). Park & Kim (2021) showed that such improvement in performance is possible due to spatial ensembles of feature map points. To this end, they used the (Bayesian) ensemble average of predictions for proximate data points (Park al., 2021), which exploits data uncertainty (i.e., a distribution of feature maps) as well as model uncertainty (i.e., a posterior probability distribution of NN weights):
p(zy|z,, D) Σ (xx (z)) p(z)) w.)
where T ((:((zz) ) is the normalized importance weight of a feature map point with respect to another feature map point æj. i.e., Σ, (22 1. This importance is defined as the similarity between æ; and zy- pl p(=))1() pa,, w)) and p( |.; D) stand for NN prediction and output predictive distribution, respectively. an; the NN weight sample from the posterior pDw|D) with respect to the training dataset D. Put shortly, Eq. (2) spatially complements prediction with other predictions based on similaritics between data points, For instance, a x 2 box blur spatially ensembles four neighboring feature map points, each with 1/4 of the same importance.
We note that the formulations for self-atte attention and the ensemble averaging for proximate data points identical. The Softmax term and V in Eq. (1) exactly correspond to л (mx (x)) and [z w;) in Eq. (2). The weight samples in Eq. (2) is correspond to the al., 2020)).
of MSAs (See also (Hron
16

---

Published as a conference paper at ICLR 2022
Eazor NLlaua
30
10-
Error (%)
NL tnin
Error (%)
1.3
25-
3.0
20
RX
R
Twins Swin PiT VTT
Mixer
R
Twins
Swin
PiT
VIT
CNN
Local MSA
Global MSA
CNN
Local MSA
Global MSA
(a) CIFAR-10
(h) ImageNet
The lower the training NLL, the lower the test error. "R" is ResNet and "RX" is ResNeXt. Left: In small data regimes, such as CIFAR-10 and CIFAR-100 (Fig. 5a), the cons of MSAS outweigh their pros; i.e., the non-convex losses disturb ViT optimization. Right: Large datasets convexify the loss functions. Therefore, the pros of MSAs outweigh their cons in large data regimes; i.e., MSAs help NNs learn strong representations by flattening the loss landscapes.
Figure C.I:
Likewise, the properties of spatial smoothing are the same as those of MSAs (Park & Kim, 2021): Spatial smoothing improves the accuracy of CNNs. In addition, spatial smoothing predicts well-calibrated uncertainty.
2 Spatial smoothing robust against MC dropout (which equivalent to image occlusion), data corruption, and adversarial attacks, and particularly robust against high- frequency noise. 3 Spatial smoothing layers closer to the output layer significantly improves the predictive performance. In addition, concurrent works suggest that MSA blocks behave like spatial smoothing. Wang et al. (2022) provided a proof that Softmax-normalinn although this does not guarantee that MSA blocks will behave like pw-sass filters. Yu al. (2021b) demonstrated that the MSA layers of VIT can be replaced with average pooling layers.
matrix is low-pass filter,
Taking all these observations together. we provide an explanation of how MSAs work by addressing themselves as general form of spatial smoothing or an implementation of ensemble averaging for proximate data points. Spatial smoothing improves performance in the following ways (Park Spatial smoothing helps in NN optimization by flattening the loss landscapes.
& Kim, 2021):
Even small 2x2 box blur filter significantly improves performance. low-pass filter. CNNs are vulnerable to high-frequency noises, but spatial smoothing improves the robustness against such noises by significantly reducing these noises. Spatial smoothing is effective when applied at the end of stage because it aggregates all transformed feature maps. This paper empirically shows that these mechanisms also apply MSAs.
Spatial smoothing is
C
VITS FROM A LOSs LANDSCAPE PERSPECTIVE
This section provides further explanations of the analysis in Section
The lower the NLL on the training dataset, the lower the error on the test dataset. Figure 5a demonstrates that low training NLLs result in low test errors on CIFAR-100. The same pattern can be observed on CIFAR-I0 and ImageNet as shown in Fig. C.I.
In small data regimes, such as CIFAR-10 (Fig. C. and CIFAR-100 (Fig. 5a), both the envor and the NLLanin of ViTs are inferior to those of CNNs. This suggests that the cons MSAs outweigh their pros. As discussed in Fig. 4, ViTs suffers from the non-convex losses, and these non-convex losses disturb VIT optimization.
In large data regimes, such as ImageNet (Fig. C.1b), both the error and the NLLumin of ViTs with local MSAS are superior to those of CNNs. Since large datasets convexify the loss functions as discussed in Fig. 4, the pros of MSAs outweigh their cons. Therefore, MSAs help NNs learn strong representations by flattening the loss landscapes.
Rigorous discussion on the regularization of CNN's inductive bias. In Fig. 5a, we compare models of similar sizes, such ResNet-50 and ViT-Ti. Through such comparison, show that weak inductive bias hinders NN training, and that inductive biases of IONNs-inductive bias of Convs and multi-stage architecture- help NNs learn strong representations. However, inductive biases of
17

---

Published as a conference paper at ICLR 2022
Ercee
2.
Error (1%)
ALtera
Density
Density
15.0
2.7
32.5
30.0
2x2
4+4
8-3
Patch size
Max Bigenvalue
Max Eigenvalue
(a) Error and NLLaai of VIT for patch size
(b) Negative and positive Hessian max eigen- value spectra in early phase of training
Figure C.2: A small patch size does not guarantee better performance. We analyze ViTs with three embedded patch sizes: 2 x 2, 4 x 4, and 8. Note that every MSA has global receptive fields. Left: As expected, a large patch size harms the performance, but surprisingly, the same is observed from small patch size. Right: A small patch size, or a weak inductive bias, produces negative cigenvalues. This is another evidence that weak inductive bias hinders NN optimization. On the other hand, MSAs with small patch size reduce the magnitude of eigenvalues because they ensemble a large number of feature map points. Performance is optimized when these two effects are balanced.
CNNs produce better test accuracy for the same training NLL, i.e., Convs somewhat regularize NNs. We analyze two comparable models in terms of NLLuain CIFAR- CIFAR-100. The NLLquin of ResNet-18. a model smaller than ResNet-50, is 2.31 with an error 22.0%. The NLLurin of ViT-S, model larger than ViT-Ti, is 2.17 with error of 30.4%. In summary, the inductive biases of CNNs improve accuracy for similar training NLLs.
Most of the improvements come from the multi-stage architecture, not the inductive bias of Convs. The NLLrain of the PiT-Ti, a multi-stage ViT-Ti, is 2.29 with an error of 24.1 %. The accuracy of PiT is only 1.9 percent point lower than that of ResNet. In addition, the small receptive field also regularizes ViT. See Fig. 7.
VIT does not overfit a small training dataset even with large number of epochs. Figure 5b shows that VIT does not overfit small training datasets, such as CIFAR. The same phenomenon be observed in VIT training with a large number of epochs.
Eres
3.0
Error (%)
40
NLL NLL
In Fig. C.3, we train ViT and ResNet for 75, 150, 300, 600, and 1200 epochs. Results show that both NLLasin and error decrease as the number of epochs increases. The predictive performances of ViT are inferior to those of ResNet across all ranges of epochs.
30-
20
75
150
300
600
1200
Epochs
A smaller patch size does not always imply better results. ViT splits image into multiple patches. The smaller the patch size, the greater the flcxibility of expression and the weaker the inductive bias. By analyzing VIT with three patch sizes- 2 x 2, 4 4, and 8 x 8 wewe demonstrate once again that weak inductive bias disturbs NN optimization.
Figure C.3: A large number of epochs does not make ViT overfit the training dataset of CIFAR. Solid line is the pre- dictive performance of ViT and dashed line is that of ResNet.
Figure C.2a shows the еnror on the test dataset and the NLL the training dataset CIFAR-100. expected, a large patch size harms the performance on both datasets. Surprisingly, however, small patch size also shows the same result. As such, appropriate patch sizes help ViT learn strong representations and do not regularize VIT.
The Hessian max eigenvalue spectra in Fig. C.2b explain this observation. Results reveal that a small patch size reduces the magnitude of Hessian eigenvalues but produces negative Hessian eigenvalues. In other words, the weak inductive bias makes loss landscapes flat yet non-convex. A large patch size suppresses negative cigenvalues. On the other hand, it not only limits the model expression but also sharpens loss landscapes. Performance is optimized when these effects are balanced.
18

---

Published as a conference paper at ICLR 2022
ResNet
VIT
PiT
Swin
(a) Loss (NLL ++2) landscape visualizations
warmupph
300
Density
Sedo
Density
Density
Swits
Density
-750
-500
-250
1000
-750
-500
-2.50
1000
(b) Negative and positive Hessian max cigenvalue spectra in early phase (left) and late phase (right) of training
Figure C.4: multi-stage architecture (in PiT) and a local MSA (in Swin) also flatten the loss landscapes. Top: PiT has a flatter loss landscape than VIT near the optimum. Swin has an almost perfectly smooth parabolic loss landscape, which leads to better NN optimization. Bottom: A multi- stage architecture in PiT suppresses negative Hessian eigenvalues. A local MSA in Swin produces negative eigenvalues, but significantly reduces the magnitude of cigenvalues.
A multi-stage architecture in PiT and local MSA in Swin also flatten loss landscapes. As explained in Fig. 1, an MSA smoothens loss landscapes. Similarly, multi-stage architecture in PiT and local MSA in Swin also help NN learn strong representations by smoothing the loss landscapes.
Figure C.4 provides loss landscape visualizations and Hessian eigenvalue spectra of ResNet, ViT, PiT, and Swin. Figure C.4a visualizes the global geometry the loss functions. The loss landscapes of PiT is flatter than that of VIT near the optimum. Since Swin has more parameters than ViT and PiT, l2 regularization determines the loss landscapes. All the loss surfaces of ViTs are smoother than that of ResNet. Figure C.4b shows the local geometry of the loss functions by using Hessian eigenvalues. In the early phase of training, a multi-stage architecture in PiT helps training by suppressing negative Hessian eigenvalues. A local MSA in Swin produces negative eigenvalues, but significantly reduces the magnitude of eigenvalues. Moreover, the magnitude of Swin's Hessian eigenvalue does not significantly increases in the late phase of learning.
lack of heads may lead to non-convex losses. Neural tangent kernel (NTK) (Jacot al., 2018) theoretically implies that the loss landscape of a VIT is convex and flat when the number of heads or the number of embedding dimensions per head goes to infinity (Hron al., 2020; Liu et al., 2020). In particular, Liu et al. (2020) suggests that ||||| ~ O(1/m) where ||/|| is the Hessian spectral norm and m is the number of heads or the number of embedding dimensions per head. Therefore, in practical situations, insufficient heads may cause non-convex and sharp losses.
Fig. C.5 empirically show that lot of heads MSA convexify and flatten the loss landscapes (cf. Michel et al. (2019)). In this experiment, we use NEP and APE to measure the non-convexity and the sharpness as introduced in Appendix A.2. Results show that both NEP and APE decrease as the number of heads increases. Likewise, Fig. C.6 shows that high embedding dimensions per head also convexify and flatten losses. The exponents of APE are -0.562 for the number of heads and 79.796 for the number of embedding dimensions, which are in close agreement with the value predicted by the theory -1/2.
Large models have a flat loss in the early phase of training. Figure C.7 analyzes the loss land- scapes of large models, such as ResNet-101 and VIT-S. As shown in Fig. C.7a, large models explore
19

---

Published as a conference paper at ICLR 2022
100
Warmuup
11
9.0
Density
Density
APE
NEP
7.0
6.0
-10
5.0
5 6899
-100
100
200
Head number
Max Eigenvalue
Max Eigenvalue
(a) NEP (non-convexityy and APE (sharpness) of head number
(b) Hessian negative and positive max eigen- value spectra in early phase of training
Figure C.5: Multi-heads convexify and flatten loss landscapes. Left: We use negative max eigen- value proportion (NEP) and average of positive max eigenvalues (APE) quantify, respectively, the non-convexity and sharpness of loss landscapes. As the number of heads increases, loss landscapes become more convex and flatter. Right: Hessian max eigenvalue spectra also show that multi-head suppress negative eigenvalues and reduce the magnitude of eigenvalues.
10.0
NEP
100
Emiied +
APE
9.0
Density
Exthed 148
Density
APE
NEP
8.0
7.0
22
110
198
-100
100
200
Embed dim Head
Max Eigenvalue
Max Eigenvalue
(a) NEP (non-comve convexxtt)) and APE (sharpness) of embedding dim
(b) Hessian negative and positive eigen- value spectra in early phase of training
Figure C.6: High embedding dimensions per head convexify and flatten the loss landscape. Left: As the number of embedding dimensions per head increases, loss landscapes become more convex and flat Right: Hessian max eigenvalue spectra also show that high embedding dimensions suppress negative eigenvalues and reduce the magnitude of eigenvalues as shown in Fig. C.5.
low NLLs. This can be a surprising because loss landscapes of large models are globally sharp as shown in Fig. C.7b.
The Hessian eigenvalue spectra in Fig. C.7c provide a solution to the problem: Hessian eigenvalues of large models are smaller than those of small models in the early phase of training. This indicates that large models have flat loss functions locally.
D VITS FROM A FEATURE MAP PERSPECTIVE
This section provides further explanations of the analysis in Section 3 and Section 4.1.
MSAs in PiT and Swin also ensemble feature maps. In Fig. 9, we show that MSAs in ViT reduce feature map variances. The same pattern can be observed in PiT and Swin. Figure D.I demonstrates that MSAs in PiT and Swin also reduce the feature map variances, suggesting that they also ensemble feature maps. One exception is the 3rd stage of Swin. MSAs suppresses the increase in variance at the beginning of the stage, but not the end of the stage.
MSAs in PiT and Swin are also low-pass filters. As discussed in Fig. 8, MSAs in ViTs are low- pass filters, while MLPs in ViT and Convs in ResNet are high-pass filters. Likewise, we demonstrate that MSAs in PiT and Swin also low-pass filters.
20

---

Published as a conference paper at ICLR 2022
ResNet-50
ResNet-101
VI--Ii
VI7-8
(a) NLL landscape visualizations
ResNet-50
ResNet-101
ViT-Ti
VIT-S
(b) Loss (NLL +62 landscape visualizations
warmup"
Sedres Ractee-00 V/T//
300
VITT:
Density
VIT-5
Density
Density
Density
(c) Negative and positive Hessian max cigenvalue spectra in carly phase (left) and late phase (right) of training
Figure C.7: Loss landscapes of large models. ResNet-50 and ResNet- 101 comparable to ViT-Ti and ViT-S, respectively. Top: Large models explore low NLLs. Middle: Loss landscape visualizations show that the global geometry of large models is sharp. Bottom: The Hessian eigenvalues of large models are smaller than those of small models. This suggests that large models have a flat local geometry in the early phase of training, and that this flat loss helps NNs learn strong representations. In the late phase of training, large ViTs have flat minima while large ResNet has a sharp minimum.
21

---

Published as a conference paper at ICLR 2022
ResNet
VI7
Feature map variance
Feature map Variance
0.10
0.08
0.6
0.04
0.2
0.0
0.00
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Normalized depth
Normalized depth
PiT
Swin
Feature map variance
Feature map variance
1.5
0.2
1.0
0.1
0.5
0.0
0.4
0.8
1.0
0.2
0.6
0.8
1.0
Normalized depth
Normalized depth
Figure D.1: MSAs in PiT and Swin also reduce feature map variance except in 3rd stage of Swin. White, gray, and blue areas are Conv/MLP, MSA, and subsampling layers, respectively.
Figure D.2 shows the relative log amplitude of Fourier transformed feature maps. As in the case of ViT, MSAs in PiT and Swin generally decrease the amplitude of high-frequency signals; in contrast, MLPs increases the amplitude.
Multi-stage ViTs have a block structures. Feature map similarities CNNs shows block struc- ture (Nguyen et al., 2021), As Raghu et al. (2021) pointed out, ViTs have uniform representations across all layers. By investigating multi-stage ViTs, we demonstrate that subsampling layers create characteristic block structure of the representation. See Fig. D.3.
Convs at the beginning of a stage and MSAs at the end of stage play an important role. Figure D.4 shows the results of lesion study for ResNet and ViTs. this experiment, we remove one 3 3 Conv layer from the bottleneck block of ResNet, and one MSA or MLP block from ViTs. Consistent results can be observed for all models: Removing Convs at the beginning of a stage and MSAs at the end of stage significantly harm accuracy. As result, the accuracy varies periodically.
E EXTENDED INFORMATION OF ALTERNET
This section provides further informations on AlterNet.
Detailed architecture AlterNet. Section introduces AlterNet to harmonize Convs with MSAs. Since most MSAS take pre-activation arrangements, pre-activation ResNet is used baseline for consistency. We add one CNN block to the last stage of ResNet to make the number of blocks even. A local MSA with relative positional encoding from Swin is used for AlterNet. However, for simplicity of implementation, we do not implement detailed techniques, such as cyclic shift and layer-specific initialization. For CIFAR, the patch size of the MSA is Conv blocks are alternately replaced with MSA, AlterNet becomes a Swin-like model.
and the window size is 4. If all
In order to achieve better performance, NNs should strongly aggregate feature maps at the end of models as discussed in Section 3 and Section 4. To this end, AlterNet use 3, 6, 12, 24 heads for MSAs each stage.
22

---

Published as a conference paper at ICLR 2022
ResNet
0.0
A Log amplitude
A Log amplitude
-2.00
-2.0
-4.00
-4.0
-6.00
-6.0
3c4m
0.0%
0.5x
1.0x
0.0
0.2
0.4
0.6
0.8
1.0
Frequency
Normalized depth
WT
0.0
-1.75
A Log amplitude
A Log amplitude
-2.00
-1.0
-2.25
Eah
-2.0
-2.50
-2.55
0.0m
0.5x
10a
0.0
0.2
0.4
0.6
0.8
1.0
Frequency
Normalized depth
PiT
0.0
A Log amplitude
A Log amplitude
-2.00
-1.0
-2.0
-3.0
-3.00
0.0%
1.0x
Frequency
Normalized depth
Swin
0.0
A Log amplitude
A Log atulitude
-2.00
-2.0
-3.00
-4.00
-4.0
Emb
-5.00
0.0
0.5
1.0g
0.0
0.2
0.4
0.6
0.8
1.0
Frequency
Normalized depth
Normalized depth
Figure D.2: MSAS PiT and Swin also reduce tigh-freeuency signals. Left: log amplitude of Fourier transformed feature map. We only provide the diagonal components. Right: The high- frequency (1.0T) A log amplitude. White, gray, and blue areas are Conv/MLP, MSA, and subsampling layers, respectively.
23

---

Published as a conference paper at ICLR 2022
ResNet
VIT
PiT
Swin
P
P
P
P
P
E
E
E
E
E
E
P
E
P
Figure D.3: Multi-stage ViTs have block structures in representational similarities. Block structures can be observed in all multi-stage NNs, namely, ResNet, PiT, and Swin. "E" is the stem/embedding and "p" the pooling (subsampling) layer.
ResNet
VIT
Accuracy (%)
A Accuracy (%)
-5-
-5
-10
-10-
-15
-55
20
0.00
0.25
0.50
0.75
1.00
0.75 1.00
Normalized depth
Normalized depth
PiT
Swin
Accuracy (%)
4 Accuracy (%)
-0.5
-1.0
-1.5
-2.
-15
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Normalized depth
Normalized depth
Figure D.4: Lesion study shows that Conys at the beginning of stage and MSAs at the end of a stage are important for prediction. We measure the decrease in accuracy after removing one unit from the trained model. In this experiment, we can observe that accuracy changes periodically. The white, gray, and blue areas are Convs/MLPs, MSAs, and subsampling layers, respectively.
Stage 1 (c1)
Stage 2 (c2)
Stage 3 (c3)
Stage (c4)
Classifier
Inpor
Stee bock
Cone blook
Come block
Cone block
Cone blaok
Cone black
Cone block
MSA block
Cone block
MSA blcck
Conv lock
MSA black
Cone Mlok
MSA black
Canv blook
MSA black
Cone Block
MSA block
Outpurt
GAP
MLP
Figure D.5: Detailed architecture of -eer-ResNet-50 for ImageNet-IK. The white, gray, and blue
blocks each represent Convs, MSAs, and subsampling blocks. This model alternately replaces Conv blocks with MSA blocks from the end of stage. Following Swin, MSAs in stages to 4 use 6, 12, and 24 heads, respectively. We use MSA blocks for ImageNet since large amounts of data alleviates the drawbacks of MSA. See Fig. 11 for comparison with the model for CIFAR-100, which uses MSA blocks.
24

---

Published as a conference paper at ICLR 2022
Redlet
Rodle VIT
Warmapp d4
Accuracy (%)
80
Jug
Aug
-a-
VIT
60-
VIT
Density
Density
40
20-
60
80
200
100
1000
Confidence (%)
Max Eigenvalue
Max Eigenvalue
(a) Reliability diagram
(b) Hessian eigenvalue spectrum
Figure F.I: Distinctive properties of strong data augmentation. "Aug" stands for strong data augmentation. Left: Strong data augmentation makes predictions underconfident on CIFAR-100 -000. The same phenomenon can be observed on ImageNet-IK IK. Right: Strong data augmentation significantly reduces the magnitude of Hessian max eigenvalues. This means that the data augmentation helps NNs converge better optima by flattening the loss landscapes. On the other hand, strong data augmentation produces a lot of negative Hessian eigenvalues, i.e., it makes the losses non-convex.
The computational costs of Conv blocks and MSA blocks are almost identical. The training throughput of Alter-ResNet-50 is 473 image/sec CIFAR-100, which is 20% faster than that of pre-activation ResNet-50.
Toiza
Robustness (%)
55.0
50.0
Alter RetNet
RX
The optimal number of MSAs depends the model and dataset, so we empirically determine the number of MSAs as shown in
CanViT LeCT
45.0
PIT
Fig. 12a. A large dataset allows large number of MSAs. For ImageNet, we use MSAs as shown in Fig. D.5, because a large datasets alleviates the shortcomings of MSAs.
40.0
WT
70.0
75.0
80.0
Accuracy (%)
MSAS improve the performance of CNNs on ImageNet. Since MSAs complement Convs, MSAs improve the predictive performance of CNNs when appropriate build-up rules are applied as shown in Section 4.1. Figure E illustrates the accuracy and robustness mean accuracy on ImageNet-C of CNNs and ViTs on ImageNet- K. Since ImageNet a large dataset, a number of VITs outperform CNNs. MSAs with the appropriate build-up rules significantly improves ResNet, and the predictive performance of AlterNet is on par with that of Swin in terms of accuracy without heavy modifications, c.g., the shifted windowing scheme (Liu et al., 2021). ). AlterNet is -ooyple-ml and has a strong potential for future improvements. addition, the build-up up rules not only improve ResNet, but also other NNs, e.g., vanilla post-activation ResNet and Rt: but we do not report this observation in order to keep the visualization simple.
Figure E.I: MSA with the ap- propriate build-up rules sig- nificantly improves ResNet on ImageNet. Robustness is mean accuracy on ImageNet-C. "RX" ResNeXt.
F DISTINCTIVE PROPERTIES OF DATA AUGMENTATION
This section empirically demonstrates that NN training with data augmentation is different from train- ing on large datasets. We compare DeiT-style strong data augmentation with weak data augmentation, ie.. resize and crop. In this section, "a result without data augmentation" stands for "a result only with weak data augmentation".
F.I DATA AUGMENTATION CAN HARM UNCERTAINTY CALIBRATION
Figure F.la shows a reliability diagram of NNs with and without strong augmentation on CIFAR- 100. Here, both ResNet and VIT without data augmentation (i.e., only with weak data augmentation) predict overconfident results. We show that strong data augmentation makes the predictive results under- confident (cf. Wen et al. (2021)). These are unexpected results because the predictions without data augmentation on large datasets, such as ImageNet, are not under-confident. A detailed investigation remains for future work.
25

---

Published as a conference paper at ICLR 2022
F.2 DATA AUGMENTATION REDUCES THE MAGNITUDE OF HESSIAN EIGENVALUES
How does data augmentation help an MSA avoid overfitting on a training dataset and achieve better accuracy test dataset? Figure F.1b shows the Hessian and without strong data augmentation. First of all, strong data augmentation reduces the magnitude of Hessian cigenvalues, i.c., data augmentation flattens the loss landscapes in the early phase of training. These flat losses leads to better generalization. On the other hand, strong data augmentation produces lot of negative Hessian eigenvalues, i.e., data augmentation makes the losses non-convex. This prevents NNs from converging to low losses on training datasets. It is clearly different from the effects of large datasets discussed in Fig. 4-large datasets convexify the loss landscapes. A detailed investigation remains for future work.
eigenvalue spectrum of NNs with
26


Reviews:
[{"review_id": "wBY2fBpyoyd", "reviewer": "Reviewer_aDUi", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strenghts:\n\n- Studies to better understand the behaviour of these models are valuable. The presented study is comprehensive in the sense that it includes many recent variants of ViTs and the explanations seem plausible.\n\n\nWeaknesses:\n\n- It is not clear to me how the construction rule for AlterNet is implemented in practice. The paper states \"... does not improve performance sufficiently ...\": on what dataset is this measured? Is this on a separate validation set or was this applied to the reported test set? \n\n- It would have been great to see if the AlterNet modification can be universally applied, by showing thi on some additional base CNN models beyond ResNet-50.\n\n- There are a couple of interesting findings and explanations, however, it seems hard to get to actionable design rules from them (beyond the proposed AlterNet). For example, the insight that loss smoothing methods aid in training seems to be generally true, not only for ViT, so it is not clear how much this insight really helps.\n\n- It would be great to see a discussion on how the number of parameters influence the findings. Are the tested ViT variants and ResNets comparable in this respect? The main results are reported on \"Tiny\" variants of transformer architectures. Would some of the insights change with larger models?\n\nMinor:\n\n- Figure 10 (b): it seems that the ResNet figure is cut off from below.", "summary_of_the_paper": "This paper attempts to explain the behaviour of multi-headed self-attention in vision transformers using a series of empirical observation. Some main observation are that the poor performance of ViT in the small data regime is not due to overfitting but to issues induced into the optimization procedure and that the behaviour of MSA is somewhat complementary to convolutions. Based on the observations a new model is proposed that combines convolution and MSAs. The experiments show that the new architecture improves results on smaller dataset.", "summary_of_the_review": "I overall think that this paper shows some interesting phenomena, even though many of the observations are post-hoc and don't seem to lead to immediate actionable design rules. However, follow-up work may still benefit from these insights.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "tlmeUqmF5U_", "reviewer": "Reviewer_Kvf7", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n- Paper focuses on addressing important problem and draws interesting insights\n- Paper provides extensive empirical results and evidence\n- Paper proposed a new Alter-ResNet architecture that is both more accurate and robust\n\nCons: \n- Paper is generally hard to read and understand. Imperial results are reported in a variety of different forms and entirety of observations is hard to grasp. I read the paper carefully twice and I am still not 100% sure I understand ALL the intricacies of every experiment and how those unequivocally lead to insights drawn. The paper is just very dense. I believe the observations and insights are correct, the issue is manly expositions which could benefit from substantial improvements. \n- The form of analysis conducted isn't well motivated or explained. This is perhaps the most negative aspect of the paper. Many different aspects of models are analyzed, e.g., NLL, Hassian max eigenvalue spectra, Frequency / Log Amplitude, etc. It is very hard to keep track of these different measures and little intuition is given for why these are ultimately \"the right\" measures to look at for a given empirical evaluation. Most experiments take the form (and I am simplifying of course) of \"We did X, which means Y.\" However, no explanation is given for why X is an appropriate measure for Y; or explanation of potential alternative measures for Y and why those were not assess. Finally, no discussion of what optimal value or behavior of X would be appropriate for a good model and so on. I would suggest re-structured each of the empirical experiments to have clearer flow and argumentation. \n", "summary_of_the_paper": "Paper presents empirical analysis Multi-headed Self-Attention (MSA) as part of Vision Transformer (ViT) and its variants. It identifies the following properties of MSA: (1) ViT with MSA improves both accuracy and generalization by flattening the loss landscape; (2) ViT with MSA behaves differently from Convolutional Neural Nets (CNNs). Specifically, ViT with MSA behaves as a low-pass and CNNs as high-pass filters. And (3) multi-stage neural nets behave like small independent models connected in series. Based on these observations, paper proposes a new architecture -- AlterNet, where CNN block is combined with MSA block and this \"base\" structure is replicated to produce deeper models. AlterNet is shown to outperform pure CNNs in both large and small data regimes. ", "summary_of_the_review": "Overall, the paper is empirically addressing a series of important questions about the (relatively) new class of architectures (ViT). The paper serves as both: (1) effectively a survey of recent papers and observations made in prior works and (2) empirical evaluation of these models. Empirical evaluations point to some interesting insights that are ultimately leveraged to build a new form of the model which is a (relatively simple) combination of the MSA and CNN. \n\nThe main concern with the paper is density of the text and the general lack of motivation for the various experimental designs, as well as sometimes somewhat tenuous (and not well explained) connections between empirical results and high-level insights. Despite these issues with exposition, which I think would benefit from fixing, I think the paper does address an important topic and drawn insights can be broadly useful. For these reasons I am in favor of acceptance. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "IuuQkDKDman", "reviewer": "Reviewer_GbX6", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper shows that MSA is low-pass while convs are high-pass, thus they complement each other. The paper interprets deep models as a series of independent blocks, and then proposes a new architecture, in which the last conversation block is replaced with multi-head self-attention. The architecture called AlterNet, shows better performance and robustness on ImageNet and CIFAR-100 datasets.\n\nPaper says “local MSAs with a 3 × 3 receptive field outperforms global MSA”. Can the authors please provide a relevant reference to a ViT work, which does not encapsulate patch-level interactions (with MSAs) altogether, and only relies on local within-patch attention? I would be amused to see a deep architecture with a receptive field as small as 3*3.\n\nWhile the paper makes a good effort in explaining how ViTs work. Some of the findings and explanations might not be in line with existing literature e.g., Fig.1 of the paper https://arxiv.org/pdf/2106.01548.pdf shows that ViT landscape is not flattened, and is sharper. \n\nThe authors mention that \"Multi-stage architecture in PiT and local MSA in Swin also flatten the loss landscapes” I think it is the hierarchial features, captured in PiT and Swin, that help them compared with vanilla ViTs where feature hierarchies are missing. CNNs also have feature hierarchies with subsequent pooling. I guess vanilla ViTs, PiT, Swin or CNNs, are all multi-stage. It is multi-scale feature hierarchies that is absent in vanilla ViTs, and is present in PiT, Swin, ResNets.  Also, Vanilla ViTs (based upon the standard attention is all you need architecture) do not capture local relationships within small image patches. Recent variants of ViTs which capture such local cues with dilated convolutions, or local self-attention, basically, introduce better inductive biases, since local cues are important for images. ", "summary_of_the_paper": "The authors show that the robustness and better performance of ViTs is attributed to their property of flattening the loss surface. Insights are taken from a parallel ICLR submission, where spatial smoothing has been shown to smooth loss landscape, and help improve robustness. The paper shows that the properties of spatial smoothing are exhibited by MSA as well.\n\n", "summary_of_the_review": "- The visualized loss landscape in the paper is different from existing literature https://arxiv.org/pdf/2106.01548.pdf; where a sharpness aware optimizer is developed to flatten the loss landscape of ViTs.\n\n- I am not entirely convinced that the findings in the paper fully explain how ViTs work. I guess, the paper draws analogy between MSA and spatial smoothing, and then builds most of its argument around that. An appropriate title reflecting that will be better, in my opinion.\n\n- Some of the terminologies used in the paper confused me, e.g., multi-stage vs multi-scale. All deep architectures are multi-stage in my understanding. Most have multi-scale feature hierarchies. Vanilla ViTs have single-scale features throughout all stages.\n\n- Small patches improve performance (counter to what is shown in the paper). See https://arxiv.org/pdf/2106.09681.pdf", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "IGWADGFWREs", "reviewer": "Reviewer_DmHc", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "+ The paper is very well reasoned and supports all arguments with solid experimental evidence. The presented story and explanations are both interesting, refreshingly different, and solidly argued.\n+ The presented architecture is quite interesting, as it mixes many conv and MSA blocks, rather than a simple transition (i.e. in ViT-Hybrid or LeVIT, ...). Though the \"Building-up rule\" seems a bit arbitrary, and a more thorough reasoning would help the paper.\n- The organization of the paper could be improved a bit. There is currently a bit of repetition within section 1 when talking about points (1)-(3) multiple times.\n- Section 3 is a bit hand-wavy and relies on prior work for much of the argumentation. A similar line of though to section 2 would make the paper stronger.\n\nMinor: Stylistically the numbered circles don't work all that well. Simple text 1., or (1) might work much better.", "summary_of_the_paper": "This paper analyzes how vision transformers (and in contrast convolutional architectures) work from an optimization standpoints. The paper shows experimentally that contrary to popular belief ViTs do not directly benefit from their more expressive underlying representation, but instead smooth the loss landscape which aids training. The paper then presents a hybrid CNN-ViT architecture that combines some of the benefits of ViTs with advantages of CNNs.", "summary_of_the_review": "I like this paper, and the insights it provides in terms of optimization landscapes of ViT vs CNN architectures. It also provides a pretty clear route towards better ViT or CNN architectures by combining the strengths of both. I wish though the experimental evaluation would focus more in ImageNet (currently mainly in appendix), and not CIFAR.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents an empirical analysis of Vision Transformers - and in particular multi-headed self-attention - and ConvNets, with a focus on optimization-related properties (loss landscape, Hessian eigenvalues). The paper shows that both classes of models have their strengths and weaknesses and proposes a hybrid model that takes the best of both worlds and demonstrates good empirical performance.

Reviewers are mostly very positive about the paper. Main pro is that analysis is important and this paper does a thorough job at it and draws some useful insights. There are several smaller issues with the presentation and the details of the content, but the authors did a good job addressing these in their responses.

Overall, it's a good paper on an important topic and I recommend acceptance.
Decision: Accept (Spotlight)