Do Vision Transformers See Like Convolutional Neural Networks?
Authors: Maithra Raghu
Thomas Unterthiner
Simon Kornblith
Chiyuan Zhang
Alexey Dosovitskiy
Venue: NeurIPS 2021 Poster (2021)
Keywords: Vision Transformers
Representation Analysis
Representation Learning
Representation Similarity
Computer Vision
Convolutional Neural Networks

Abstract: Convolutional neural networks (CNNs) have so far been the de-facto model for visual data. Recent work has shown that (Vision) Transformer models (ViT) can achieve comparable or even superior performance on image classification tasks. This raises a central question: how are Vision Transformers solving these tasks? Are they acting like convolutional networks, or learning entirely different visual representations? Analyzing the internal representation structure of ViTs and CNNs on image classification benchmarks, we find striking differences between the two architectures, such as ViT having more uniform representations across all layers. We explore how these differences arise, finding crucial roles played by self-attention, which enables early aggregation of global information, and ViT residual connections, which strongly propagate features from lower to higher layers. We study the ramifications for spatial localization, demonstrating ViTs successfully preserve input spatial information, with noticeable effects from different classification methods. Finally, we study the effect of (pretraining) dataset scale on intermediate features and transfer learning, and conclude with a discussion on connections to new architectures such as the MLP-Mixer. 

Do Vision Transformers See Like Convolutional Neural Networks?
Maithra Raghu
Thomas Unterthiner Google Research, Brain Team unterthiner@google.cc com
Simon Kornblith Google Research, Brain Team kormblith@google.. com
Google Research, Brain Team mai taithar@@maal.. com
Chiyuan Zhang Google Research, Brain Team chi yuiiyuu@ggoglecomm
Alexey Dosovitskiy Google Research, Brain Team
adosovitakkiyggogle.. com
Abstract
Convolutional neural networks (CNNs) have so far been the de-facto model for visual data. Recent work has shown that (Vision) Transformer models (ViT) achieve comparable or even superior performance on image classification tasks. This raises central question: how Vision Transformers solving these tasks? Are they acting like convolutional networks, or learning entirely different visual representations? Analyzing the internal representation structure of ViTs and CNNs on image classification benchmarks, we find striking differences between the two ar- chitectures, such VIT having explore how these differences arise, finding crucial roles played by self-attention, which enables early aggregation of global information, and ViT residual connec- tions, which strongly propagate features from lower to higher layers. We study the ramifications for spatial localization, demonstrating ViTs successfully preserve input spatial information, with noticeable effects from different classification meth- ods. Finally, we study the effect of (pretraining) dataset scale on intermediate features and transfer learning, and conclude with discussion on connections to new architectures such as the M-.--Mixer.
uniform representations across all layers. We
Introduction
Over the past several years, the successes of deep learning on visual tasks has critically relied convolutional neural networks [20, 16). This is largely due to the powerful inductive bias of spatial equivariance encoded by convolutional layers, which have been key to learning general purpose visual representations for easy transfer and strong performance. Remarkably however, recent work has demonstrated that Transformer neural networks are capable of equal or superior performance on image classification tasks at large scale [14]. These Vision Transformers (ViT) operate almost identically Transformers used in language [13], using self-attention, rather than convolution, to aggregate information across locations. This is in contrast with large body of prior work, which has focused on more explicitly incorporating image- speciffc inductive biases [30. 4]
This breakthrough highlights a fundamental question: how are Vision Transformers solving these image based tasks? Do they act like convolutions, learning the same inductive biases from scratch? Or are they developing novel task representations? What the role of scale in learning these representations? And there ramifications for downstream tasks? In this paper, we study these questions, uncovering key representational differences between ViTs and CNNs, the ways in which these difference arise, and effects on classification and transfer learning. Specifically, our contributions are:
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

We investigate the internal representation structure of ViTs and CNNs. finding striking differences between the two models, such as ViT having more uniform representations, with greater similarity between lower and higher layers.
Analyzing how local/global spatial information is utilised, we find VIT incorporates more global information than ResNet at lower layers, leading to quantitatively different features.
Nevertheless, we find that incorporating local information at lower layers remains vital, with large-scale pre-training data helping early attention layers learn to do this
We study the uniform internal structure of VIT. finding that skip connections in VIT are even more influential than in ResNets, having strong effects on performance and representation similarity.
Motivated by potential future uses in object detection, we examine how well input spatial informa- tion is preserved, finding connections between spatial localization and methods of classification. We study the effects of dataset scale on transfer learning, with a linear probes study revealing its importance for high quality intermediate representations.
2 Related Work
Developing non-convolutional neural networks to tackle computer vision tasks, particularly Trans- former neural networks [44] has been an active area of research. Prior works have looked at local multiheaded self-attention, drawing from the structure of convolutional receptive fields [30, 36), directly combining CNNs with self-attention [4, 2, 46] or applying Transformers to smaller-size images [6, 9]. In comparison to these, the Vision Transformer [14] performs even less modification to the Transformer architecture, making it especially interesting to compare to CNNs. Since its development, there has also been very recent work analyzing aspects of ViT, particularly robustness [3, 31, 28] and effects of self-supervision [5, 7J. Other recent related work has looked at designing hybrid ViT-CNN models [49, 11], drawing on structural differences between the models. Comparison between Transformers and CNNs are also recently studied in the text domain [41].
Our work focuses on the representational structure of ViTs. To study VIT representations, draw on techniques from neural network representation similarity, which allow the quantitative comparisons of representations within and across neural networks [17, 34, 26, 19). These techniques have been very successful in providing insights on properties of different vision architectures [29, 22, 18), representation structure in language models [48, 25, 47, 21], dynamics of training methods [33, 24] and domain specific model behavior [27, 35, 38]. We also apply linear probes in our study, which has been shown be useful analyze the learned representations in both vision [1] and text [8, 32, 45]
models.
3 Background and Experimental Setup
Our goal is to understand whether there are differences in the way ViT' represent and solve image tasks compared to CNNs. Based the results of Dosovitskiy et al. [14], we take representative set of CNN and ViT models - ResNet5001, ResNet152x2, ViT-B/32, ViT-B/16, ViT-L/16 and ViT-H/14. Unless otherwise specified, models are trained on the JFT-300M dataset [40]. although we also investigate models trained on the ImageNet ILSVRC 2012 dataset [12, 37] and standard transfer learning benchmarks [50, 14]. We use a variety of analysis methods to study the layer representations of these models, gaining many insights into how these models function. We provide further details of the experimental setting in Appendix A.
Representation Similarity and CKA (Centered Kernel Alignment): Analyzing (hidden) layer representations of neural networks is challenging because their features are distributed across large number of neurons. This distributed aspect also makes it difficult to meaningfully compare represen- tations across neural networks. Centered kernel alignment (CKA) [17, 10] addresses these challenges, enabling quantitative comparisons of representations within and across networks. Specifically, CKA takes as input X IR xxx and Y E mmx which are representations (activation matrices), of two layers, with and P2 neurons respectively, evaluated on the same m examples. Letting K = X X and L YY denote the Gram matrices for the two layers (which measures the similarity of a pair of datapoints according to layer representations) CKA computes:
HSIC L) CKA(K, L) VHSIC(K, KICIK,KHHH L)
(1)
2

---

VIT-L/16
VIT-H/14
R50
R152
240
126
100
VIT-V14
150
500
125
Layers R50
20
R152
250
80
130
Layens
200
40
20
50
01
25
80
80
100
120
140
25
75 200
125
350
3.5
20 60 100 3.20
0
50
107
130
230 250
307
350
Layers VIF-L/16
Layers VIT-H/14
Layers
Layers R152
Figure I: Representation structure of VITs and convolutional networks show significant differences, with ViTs having highly similar representations throughout the model, while the ResNet models show much lower similarity between lower and higher layers. We plot CKA similarities between pairs of layers across different model architectures. The results are shown as healmap, with and axes indexing the layers from input to output. We observe that ViTs have relatively uniform layer similarity structure, with a clear grid-like pattern and large similarity between lower and higher layers. By contrast, the ResNet models show clear stages similarity structure, with smaller similarity scores between lower and higher layers.
VIT-L/16 VS R50
VIT-H/14 VS R50
Figure 2: Cross model CKA heatmap between ViT and ResNet illustrate that larger number lower layers ResNet are similar to a smaller set of the lowest VIT layers. We compute a CKA heatmap comparing all layers of ViT to all layers of ResNet. for two different ViT mod- els. We observe that the lower half of ResNet layers are similar to around the lowest quar- ter of ViT layers. The remaining half of the ResNet similar to approximately the next third of VIT layers, with the highest ViT layers dissimilar to lower and higher ResNet layers.
Layers VIT-L/16
Layers VIT-H/14
150
100
3.25
80
60
75
40
50
20
40
60
80
100
120
20
40
60
80 100
120
Layers R50
Layers R50
where HSTC is the Hilbert-Schmidt independence criterion [15]. Given the centering matrix H Ln 11 and the centered Gram matrices K' HKH H and L' HLH, HSIC(K, vec (') vevc(e))/ (11m-19) 1)2, the similarity between these centered Gram matrices. CKA invariant to orthogonal transformation of representations (including permutation of neurons). and the normal- izatiin term ensures invariance to isotropic scaling. These properties enable meaningful comparison and analysis of neural network hidden representations. To work at scale with our models and tasks, we approximate the unbiased estimator of HSTC [39] using minibatches, as suggested in [29].
4 Representation Structure of ViTs and Convolutional Networks
We begin our investigation by using CKA to study the internal representation structure of each model. How are representations propagated within the two architectures, and are there signs of functional differences? To answer these questions, we take every pair of layers X, Y within model and compute their CKA similarity. Note that we take representations not only from outputs of Vit/ResNet blocks, but also from intermediate layers, such normalization layers and the hidden activations inside a ViT MLP. Figure shows the results as a heatmap, for multiple ViTs and ResNets. We observe clear differences between the internal representation structure between the two model architectures: (I) ViTs show a much more uniform similarity structure, with a clear grid like structure (2) lower and higher layers in ViT show much greater similarity than in the ResNet, where similarity is divided into different (lower/higher) stages.
We also perform model comparisons, where we take all layers X from ViT and compare to all layers Y from ResNet. We observe (Figure 2) that the lower half of 60 ResNet layers are similar to approximately the lowest quarter of ViT layers. In particular, many more lower layers in the ResNet are needed to compute similar representations to the lower layers of ViT. The top half of the ResNet approximately similar to the next third of the ViT layers. The final third of VIT layers is less similar to
3

---

VIT-L/16 JFT to ImageNet
VIT-H/14 JFT to ImageNet
120
120
100
100
Mean Distance
Mean Distance
80
80
60
60
encoder_biin blockO
encoder, _blocko
encoder. block1 encoder: block22
encoder, block1 encoder block30 encoder block31
20
20
encoder block23
0
2
10
12
14
0
2
10 12 14
Sorted Attention Head
Sorted Attention Head
Figure 3: Plotting attention head mean distances shows lower ViT layers attend both locally and globally, while higher layers primarily incorporate global information. each attention head, we compute the pixel distance it attends to, weighted by the attention weights, and then average over 5000 datapoints to get an average attention head distance. We plot the heads sorted by their average attention distance for the two lowest and two highest layers in the ViT, observing that the lower layers attend both locally and glohally, while the higher layers attend entirely glohally.
all ResNet layers, likely because this set of layers mainly manipulates the CLS token representation, further studied in Section 6.
Taken together, these results suggest that (i) ViT lower layers compute representations in a different way to lower layers in the ResNet, (ii) VIT also more strongly propagates representations between lower and higher layers (iii) the highest layers of VIT have quite different representations to ResNet.
5 Local and Global Information in Layer Representations
In the previous section, we observed much greater similarity between lower and higher layers in VIT, and we also saw that ResNet required more lower layers to compute similar representations to smaller set of ViT lower layers. In this section, we explore one possible reason for this difference: the difference in the ability to incorporate global information between the two models. How much global information is aggregated by early self-attention layers in ViT? Are there noticeable resulting differences to the features of CNNs, which have fixed, local receptive fields in early layers? In studying these questions, we demonstrate the influence of global representations and a surprising connection between scale and self-attention distances.
start by analyzing VIT self-attention layers, which are mechanism for ViT to aggregate information from other spatial locations, and structurally very different to the fixed receptive field sizes of CNNs. Each self-attention layer comprises multiple self-altention heads, and for each head we compute the average distance between the query patch position and the locations attends to. This reveals how much local vs global information each self-attention layer is aggregating for the representation. Specifically, the attention weights for each attention head and average over 5000 datapoints, with results shown in Figure 3. In agreement with Dosovitskiy et al. [14], we observe that even in the lowest layers of ViT, self-attention layers have a mix of local heads (small distances) and global heads (large distances). This is in contrast to CNNs, which are hardcoded to attend only locally in the lower layers. At higher layers, all self-attention heads are global.
Analyzing Attention Distances:
weight the pixel distances by
clear effect of scale on attention. In Figure 4, we look at attention distances when training only on ImageNet (no large-scale pre-training), which leads to much lower performance in VIT-L/16 and ViT-H/14 [14]. Comparing Figure 3, we see that with not enough data, VIT does not learn to attend locally in earlier layers. Together, this suggests that using local information early on for image tasks (which is hardeoded into CNN architectures) is important for strong performance.
Interestingly,
Does access to global information result in different features? The results of Figure demonstrate that ViTs have access to more global information than CNNs in their lower layers. But does this result in different learned features? an interventional test, we take subsets of the ViT attention heads from the first encoder block, ranging from the subset corresponding the most local attention

---

VIT-L/16 on ImageNet
VIT-H/14 on ImageNet
120
120
100
100
Mean Distance
Mean Distance
80
80
60
60
encoder_ encoder_block0 blockO
encoder, Bllockk
encoder bleckt
encoder, blockt
20
encoder block22
20
enceder block30
encoder Block23
encoder blork31
10
12
14
12 Sorted Attention Head
10
14
Sorted Attention Head
Figure 4: With less training data, lower attention layers do not learn atfend locally. Comparing the results to Figure 3, we see that training only on ImageNet leads to lower layers not learning to attend more locally. These models also perform much worse when only trained on ImageNet, suggesting that incorporating local features (which is hardeoded into CNNs) may be important for strong performance. (See also Figure C.5.)
and Global VIT-L/16
Local Representations compared to R50
VIT-H/14
VIT-L/16 Representations compared to R15:
Representations compared to R50
0.55
0.50
6.55
OKA Similarity
0.50
CKA Similarity
0.50
GKA Similarity
0.45
0.40
0.40
blockdunitl
Hockiunitt
0.35
bocklun81
biockluntt2
0.35
bocckuuit2
biockluni2
biockluntts
blocklunit3
bocklunt3
0.30
block2uniti
Bockkuuii1
beck2unk1
block2uni22
buck2unk22
6.25
0.5
40
50
60
70
80
50
100
20
40
60
30
100
40
50
60
70
80
100
Mean Distance
Mean Distance
Mean Distance
Figure 5: Lower layer representations of ResNet are most similar to representations corresponding to local attention heads of VIT. We take subscts of ViT attention heads in the first encoder block, ranging from the most locally attending heads (smallest then compute CKA similarity between these subsets and lower layer representations in ResNet. We observe that lower ResNet layers are most similar to the features learned by local attention heads of ViT, and decrease monotonically in similarity as more global information is incorporated. demonstrating that the global heads learn quantitatively different features.
distance) the global heads (largest
distance). We
heads to a subset of the representation corresponding to the most global attention heads. We then compute CKA similarity between these subsets and the lower layer representations of ResNet.
The results, shown in Figure 5, which plot the mean distance for each subset against CKA similarity, clearly show monotonic decrease in similarity access to more global information also leads to quantitatively different features than computed by the local receptive fields in the lower layers of the ResNet.
attention distance grows, demonstrating that
Effective Receptive Fields: We conclude by computing effective receptive fields [23] for both ResNets and ViTs, with results in Figure 6 and Appendix C. We observe that lower layer effective receptive fields for VIT are indeed larger than in ResNets, and while ResNet effective receptive fields grow gradually, ViT receptive fields become much more global midway through the network. ViT receptive fields also show strong dependence on their center patch due to their strong residual connections, studied in the next section. As show in Appendix C, in attention sublayers, receptive fields taken before the residual connection show far less dependence on this central patch.
Representation Propagation through Skip Connections
The results of the previous section demonstrate that ViTs learn different representations ResNets in lower layers due access to global information, which explains some of the differences in represen-
5

---

VIT-B/32 Attention
Attention on
Attention 3
Attention 9
Attention 12
Figure 6: ResNet effective re- ceptive fields are highly local and grow gradually; VIT effec- tive receptive fields shift from local to global. We measure the effective receptive field of different layers the absolute value of the gradient of the cen- location of the feature map (taken after residual connections) with respect the input. Results averaged across all channels in each map for 32 randomly- sclected images.
ResNet-50 Block 7
Initial Conv
Block
Block 12
Block 16
Ratio of Norms (first few tokens)
Ratio of Norms (aggregated)
Average norn inpressetatioon
Token Index
Block Index
Figure Most information ViT passes through skip connections. Comparison representation norms between the skip-connection (identity) and the long branch for ViT-B/16 trained on ImageNet and a ResNet. For ViT, we show the CLS token separately from the rest of the representation. (left) shows the ratios separated for the first few tokens (token 0 is CLS), (right) shows averages all tokens.
tation structure observed in Section 4. However, the highly uniform nature VIT representations (Figure 1) also suggests lower representations are faithfully propagated to higher layers. But how does this happen? In this section, we explore the role of skip connections in representation propagation across ViTs and ResNets, discovering ViT skip connections are highly influential, with clear phase transition from preserving the CLS (class) token representation (in lower layers) to spatial token representations (in higher layers).
Like Transformers, ViTs contain skip (aka identity shortcu)) connections throughout, which added on after the (i) self-attention layer, and (ii) MLP layer. To study their effect, we plot the the hidden representation of the ith layer coming from the skip connection, and f(z)) is the transformation of z, from the long branch (i.c. MLP or self-attention.)
norm ratio [|=/ (llff((()) (2()l where
The results in Figure 7 (with additional cosine similarity analysis in Figure E.2.) The heatmap on the left shows | for different token representations. We observe a striking phase transition: in the first half of the network, the CLS token (token 0) representation is primarily propagated by the skip connection branch (high norm ratio), while the spatial token representations have a large contribution coming from the long branch (lower norm ratio). Strikingly, in the second half of the network, this is reversed.
The right pane, which has line plots of these norm ratios across ResNet50, the ViT CLS token and the ViT spatial tokens additionally demonstrates that skip connection much more influential ViT compared to ResNet: observe much higher norm ratios for ViT throughout, along with the phase transition from CLS to spatial token propagation (shown for the MLP and attentioo layers.)
VIT Representation Structure without Skip Connections:
The norm ratio results strongly suggest that skip connections play a key role the representational structure of VIT. test this intervention- ally, we train ViT models with skip connections removed in block for varying i, and plot the CKA representation heatmap. The results, in Figure 8, illustrate that removing the skip connections in block partitions the layer representations either side. (We note performance drop of 4% when

---

Remove Skpp Connection
Connection Block
Comnecionn Block
1.
10 26 9. 50 1. 29
Figure VIT models trained without any skip connections in block show very little representation similarity between layers before/after block i. We train several VIT models without any skip connections at block for varying i to interventionally test the effect on representation structure. For middle blocks without skip connections, observe performance drop of 4%. We also observe that removing skip connection block partitions similar representations before/after block this demonstrates the importance of skip connections in ViT's standard uniform representation structure.
ViT-B/32 final block
Token Location(O, 0) Token Location(2, Token at Location(4, 2) Token Location(4,
VIT-L/16 final block
TokenO Location(0, 0) Token31 Location(2,
3) Token35 Location(2, 7) Token61 Location(4, 5)
12
012345678910123 01234567880122
01234567852032
R50x1 final unit final block
Token at Location(o, 0) Taken at Location(2, Token at Location(4, 2) Token at Location(4, 4)
Figure 9: Higher layers of VIT maintain spatial location information more faithfully than ResNets. Each heatmap plot shows the CKA similarity between single token representation in final block of the model and the input images, which divided into non-overlapping patches. We observe that VIT tokens have strongest similarity their corresponding spatial location the image, but tokens corresponding to spatial locations the edge of the image (e.g. token 0) additionally show similarity to other edge positions. This demonstrates that spatial information from the input is preserved even the final layer of VIT. By contrast, ResNet "tokens" (features a specific spatial location) much less spatially discriminative, showing comparable similarity across broad of input spatial locations. See Appendix for additional layers and results.
removing skip connections from middle blocks.) This demonstrates the importance of representations being propagated by skip connections for the uniform similarity structure of ViT in Figure 1.
Spatial Information and Localization
The results so far, on the role of self-attention in aggregating spatial information in ViTs, and skip- connections faithfully propagating representations to higher layers, suggest an important followup question: how well can ViT's perform spatial localization? Specifically, is spatial information from the input preserved in the higher layers of VIT? And how does it compare in this aspect ResNet? An affirmative answer to this is crucial for uses of VIT beyond classification, such as object detection.
7

---

VIT-B/32 GAP final block
Token at Location(O, 0) Taken at Location(2, Token at Location(4, 2) Taken at Location[4, 4)
Figure 10: When trained with global average pooling (GAP) iGAPead instead of a CLS token, VTTs show less clear localization (compare Figure 9). We plot the same CKA heatmap between token and different input images patches as in Figure but for ViT model trained with global average pooling (like ResNet) instcad of CLS token. observe significantly less localization.
70
70
VIT-B/32 CLS
CLS CLS tolken
VIT-B/32 GAP
60
CLS; except taken
ResNet50
CLS) GAP
50
50
First
Telt accuracy
40
Test accuracy
40
GAP: GAP
30
20
20
10
10
1.0
0.6 Layer number
Normalized layer number
(a) Individual token evaluation
(b) CLS vs GAP models
Figure 11: Spatial localization experiments with linear probes. We train linear classifiers on 10-shot ImageNet classification from the representations extracted from different layers of ViT-B/32 models. We then plot accuracy the probe versus the (normalized) layer number. Left: We train classifier on each token separately and report the average accuracy over all tokens (excluding the CLS token for the ViT CLS model.) Right: Comparison of VIT models pre-trained with a classification token or with global average pooling (GAP) and then evaluated with different ways of aggregating the token representations.
We begin by comparing token representations in the higher layers of ViT and ResNet to those of input patches. Recall that ViT tokens have a corresponding input patch, and thus a corresponding input spatial location. For ResNet, we define token representation to be all the convolutional channels at particular spatial location. This also gives it a corresponding input spatial location. We can then take token representation and compute its CKA score with input image patches different locations. The results are illustrated for different tokens (with their spatial locations labelled) in Figure 9.
For VIT, we observe that tokens corresponding to locations at the edge of the image are similar to edge image patches, but tokens corresponding to interior locations are well localized, with their representations being most similar to the corresponding image patch. By contrast, for ResNet, significantly weaker localization (though Figure D.3 shows improvements for earlier layers.)
One factor influencing this clear difference between architectures is that ResNet is trained to classify with global average pooling step, while VIT has separate classification (CLS) token. To examine this further, we test a VIT architecture trained with global average pooling (GAP) for localization (see Appendix A for training details). The results, shown Figure 10, demonstrate that global average pooling does indeed reduce localization in the higher layers. More results in Appendix Section D.
Localization and Linear Probe Classification: The previous results have looked at localization through direct comparison of each token with input patches. To complete the picture, we look at using each token separately to perform classification with linear probes. We do this across different layers of the model, training linear probes to classify image label with closed-for fooed few-shot linear regression similar to Dosovitskiy al. [14] (details in Appendix A). Results are in Figure 11, with further results in Appendix F. The left pane shows average accuracy of classifiers trained on individual tokens, where we see that ResNet50 and ViT with GAP model tokens perform well at higher layers, while in the standard ViT trained with CLS token the spatial tokens do poorly likely because their representations remain spatially localized higher layers, which makes global classification challenging. Supporting this are results the right pane, which shows that single token from the
8

---

5mbrirt to 100% (CKG)
35 SV3 wTTTica
Block Index
Stock Index
Figure 12: Mcasuring similarity of representations Iearned with varying amounts of data shows the importance large datasets higher layers and larger model representations. of representations at each block for VIT models that have been trained on smaller subsets of the data to a model that has been trained on the full data on VIT-L/16 (left) and VIT-B/32 (right). We observe that while lower layer representations have high similarity significantly more data to learn similar representations.
compute the similarity
with 10% of the data, higher layers and larger models require
80
VIT-L/16
JFT-300M
80
VIT-B/32
70
JFT-30OM VIT-H/14
70
VIT-L/16 ImageNet
VIT-L/16
60
VIT:H/14 ImageNet
60
VITH/11
Test accuracy
50
50
ResNet50
ResNet152
40
40
ResNet152x2
30
30
20
10
1.0
Normallzedd layer number
Normalized layer number
(a) JFT-300M vs ImageNet pre- me--tranning
(b) ResNets
Figure 13: Experiments with linear probes. We train linear classifiers on 10-shot ImageNet classification from the aggregated representations of different layers of different models. We then plot the accuracy of the probe versus the (normalized) layer number. Left: Comparison of ViTs pre-trained JFT-300M ImageNet and evaluated with linear prohes on Imagenet. Right: Comparison of VIT ResNet models trained JFT-300m, evaluated with linear probes on ImageNet.
ViT-GAP model achieves comparable accuracy in the highest layer to all tokens pooled together. With the results of Figure 9, this suggests all higher layer tokens in GAP models learn similar (global) representations.
8 Effects of Scale on Transfer Learning
Motivated by the results of Dosovitskiy et al. [14] that demonstrate the importance of dataset scale for high performing ViT's, and our earlier result (Figure 4) on needing scale for local attention, we perform study of the effect of dataset scale on representations in transfer learning.
We begin by studying the effect on representations as the JFT-300M pretraining dataset size varied. Figure 12 illustrates the results on ViT-B/32 and ViT-L/16. Even with 3% of the entire dataset, lower layer representations are very similar to the model trained on the whole dataset, but higher layers require larger amounts of pretraining data to learn the same representations as large data scale, especially with the large model size. In Section G, we study how much representations change in finetuning, finding heterogeneity over datasets.
next look at dataset size effect on the larger ViT-L/16 and ViT-H/14 models. Specifically, the left pane of Figure 13, we train linear classifer probes on ImageNet classes for models pretrained on JFT-300M vs models only pretrained on ImageNet. We observe the JFT-300M pretained models achieve much higher accuracies even with middle layer representations, with 30% gap in absolute

---

accuracy to the models pretrained only on ImageNet. This suggests that for larger models, the larger dataset is especially helpful in learning high quality intermediate representations. This conclusion further supported by the results of the right pane of Figure 13, which shows linear probes on different ResNet and ViT' models, all pretrained on JFT-300M. We again see the larger ViT models learn much stronger intermediate representations than the ResNets. Additional linear probes experiments in Section F demonstrate this same conclusion for transfer to CIFAR-IO and CIFAR-100.
Discussion
Our study uses CKA [17], which summarizes measurements into a single scalar, to provide quantitative insights on representation similarity. While we have complemented this with interventional tests and other analyses (e.g. linear probes), more fine-grained methods may reveal additional insights and variations in the representations.
Limitations:
Conclusion: Given the central role of convolutional neural networks in computer vision break- throughs, it is remarkable that Transformer architectures (almost identical to those used in language) are capable of similar performance. This raises fundamental questions on whether these architec- tures work in the same way as CNNs. Drawing on representational similarity techniques, we find surprisingly clear differences in the features and internal structures of ViTs and CNNs. An analysis of self-attention and the strength of skip connections demonstrates the role of earlier global features and strong representation propagation ViTs for these differences, while also revealing that some CNN properties, e.g. local information aggregation at lower layers, are important to VITs, being learned from scratch at scale. We examine the potential for ViTs to be used beyond classification through study of spatial localization, discovering ViTs with CLS tokens show strong preservation of spatial information - promising for future uses in object detection. Finally, we investigate the effect of scale for transfer learning, finding larger ViT models develop significantly stronger intermediate representations through larger pretraining datasets. These results are also very pertinent to under- standing -ase- architectures for vision proposed by concurrent work [42, 43], further discussed in Section H, and together answer central questions on differences between ViTs and CNNs, and suggest new directions for future study. From the perspective of societal impact, these findings and future work may help identify potential failures as well as greater model interpretability.
References
[1] G. Alain and Y. Bengio. Understanding intermediate layers using linear classifier probes. arXiv preprint arXiv: 1610.01644, 2016.
[2] I. Bello, B. Zoph, A. Vaswani, J. Shlens, and Q. Le. Attention augmented convolutional networks. Proceedings of the IEEF/CVF International Conference on Computer Vision pages 3286-3295 2019.
[3] Bhojanapalli, A. Chakrabarti, D. Glasner, Li, Unterthiner, and A. Veit. Understanding robustness of transformers for image classification. arXiv preprint arXiv:2103. 14586, 2021.
[4] N. Carion, Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko. End-to-end object detection with transformers. In European Conference Computer Vision, pages 213-229. Springer, 2020.
[5] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin. Emerging properties in self-supervised vision transformers. arXiv preprint arXiv:2 /0044//294,.. 2021.
[6] M. Chen, A. Radford, R. Child, J. Wu, H. Jun, D. Luan, and I. Sutskever. Generative pretraining from pixels. International Conference on Machine Learning, pages 1691-1703. PMLR, 2020.
[7] X. Chen, S. Xie, and K. He. An empirical study of training self-supervised vision transformers. arXiv preprint arXiv:2104.020577 2021.
[8] A. Conneau, G. Kruszewski, G. Lample, I.. Barrault, and Baroni. What you can cram into a single vector: Probing sentence embeddings for linguistic properties. In ACL, 2018.
10

---

[9] J.-B. Cordonnier, A. Loukas, and M. Jaggi. On the relationship between self- self-attention and convolutional layers. arXiv preprint arXiv: 191100541 2019.
[10] C. Cortes, M. Mohri, and A. Rostamizadeh. Algorithms for learning kernels based on centered alignment. The Journal of Machine Learning Research, 13(1):795-822 2012.
[11] S. d' Ascooli H. Touvron, M. Leavitt, A. Morcos, G. Biroli, and Sagun. Convit: Improving vision transformers with soft convolutional inductive biases. arXiv preprint 10Xi:2203.10697, 2021.
[12] J. Deng, W. Dong, R. Socher, L.,J. K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pages 248-255, leee, 2009.
[13] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint XXi::1810.04805, 2018.
[14] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenbom, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelll, al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 1929, 2020.
[15] A. Gretton, K. Fukumizu, C. H. Teo, L. Song. B. Schölkopf, A. J. Smola, et al. A kernel statistical test of independence. In Nips, volume 20, pages 585-592 Citeseer, 2007.
[16] A. Kolesnikov, L. Beyer, X. Zhai, J. Puigcerver. J. Yung. S. Gelly, and N. Houlsby. Big transfer (bit): General visual representation learning, arXiv preprint arXiv:1912.11377 6(2):8, 2019.
[17] S. Kornblith, M. Norouzi, H. Lee, and G. Hinton. Similarity of neural network representations revisited. In ICML, 2019.
[18] S. Kornblith, H. Lee, T. Chen, and M. Norouzi. What's 's in a loss function for image classification? arXiv preprint arXiv:2010..6411 2020.
[19] N. Kriegeskorte, M. Mur, and P. A. Bandettini. Representational similarity analysis-comnecting the branches of systems neuroscience. Frontiers in systems neuroscience, 2:4, 2008
[20] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25:1097-1105, 2012.
[21] S. R. Kudugunta, A. Bapna. I. Caswell, N. Arivazhagan, and O. Firat. Investigating multilingual nmt representations at scale. arXiv preprint arXiv:1909.02197, 2019.
[22] G. W. Lindsay. Convolutional neural networks as model of the visual system: past, present, and future. Journal of cognitive neuroscience, pages 1-15, 2020.
[23] W. Luo, Y. Li, R. Urtasun, and R. Zemel. Understanding the effective receptive field in deep convolutional neural networks. arXiv preprint arXiv: 2017.
[24] N. Maheswaranathan, A. H. Williams, M. D. Golub, S. Ganguli, and D. Sussillo. Universality and individuality in neural dynamics across large populations of recurrent networks. Advances in neural information processing systems, 2019:15629, 2019.
[25] A. Merchant, E. Rahimtoroghi, E. Pavlick, and I. Tenney. What happens to bert embeddings during fine-tuning? arXiv preprint arXiv:2004. 14448, 2020.
[26] A. S. Morcos, M. Raghu, and S. Bengio. Insights on representational similarity in neural networks with canonical correlation. arXiv preprint arXiv:1806.0557 0578809 2018
[27] B. Mustafa, A. Loh, J. Freyberg, P. MacWilliams, M. Wilson, S. M. McKinney, M. Sieniek,
J. Winkens, Y. Liu, P. Bui, et al. Supervised transfer learning at scale for medical imaging. arXiv preprint arXiv:2101. 00.9119,,, 2021.
[28] M. Naseer, K. Ranasinghe, S. Khan, M. Hayat, Khan, and M.-H. Yang. Intriguing properties of vision transformers, 2021.

---

[29] T. Nguyen, M. Raghu, and S. Kornblith. Do wide and deep networks learn the same things? uncovering how neural network representations vary with width and depth. arXiv preprint arXiv:2010. 15327, 2020.
[30] N. Parmar, A. Vaswani, J. Uszkoreit, L. Kaiser, N. Shazeer, A. Ku, and D. Tran. Image transformer. In International Conference on Machine Learning, pages 4055 4064, PMLR, 2018.
[31] S. Paul and P.-Y. Chen. Vision transformers are robust learners. arXiv preprint arXiv:2005.07581, 2021.
[32] M. E. Peters, M. Neumann, I.. Zettlemoyer, and W.-t Yih. Dissecting contextual word embed- dings: Architecture and representation. In EMNLP, 2018.
[33] A. Raghu, M. Raghu, S. Bengio, and O. Vinyals. Rapid learning or feature reuse? towards understanding the effectiveness of maml. arXiv preprint arXiv: 1999009557 2019.
[34] M. Raghu, Gilmer, J. Yosinski, and Sohl-Dickstein. Sveea: Singular vector canonical corre- lation analysis for deep learning dynamics and interpretability. arXiv preprint arXiv:/706.05800 2017.
[35] M. Raghu, C. Zhang, J. Kleinberg. and Bengio. Transfusion: Understanding transfer learning for medical imaging. arXiv preprint arXiv: /902.0770,, 2019.
[36] P. Ramachandran, N. Parmar, A. Vaswani. I. Bello, A. Levskaya, and J. Shlens. Stand-alone 2019.
self-attention vision models. arXiv preprint arXiv: I
[37] O. Russakovsky, J. Deng, H. Su, Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211-252, 2015.
[38] Shi, E. Shea-Brown, and M. Buice. Comparison against task driven artificial neural networks reveals functional properties in mouse visual cortex. Advances in Neural Information Processing Systems, 32:5764-5774, 2019.
[39] I.. Song, A. Smola, A. Gretton, Bedo, and K. Borgwardt. Feature selection via dependence maximization. Journal of Machine Learning Research, 13(5), 2012.
[40] C. Sun, A. Shrivastava, S. Singh, and A. Gupta. Revisiting unreasonable effectiveness of data in deep learning era. In Proceedings of the IEEE international conference on computer vision, pages 843-852, 2017.
[41] Y. Tay, M. Dehghani, J. Gupta, D. Bahri, V. Aribandi, Z. Qin, and D. Metzler. Are pre-trained convolutions better than pre-trained transformers? arXiv preprint 2rXiv:2105.08::
2021. [42] I. Tolstikhin, N. Houlsby, A. Kolesnikov, L. Beyer, X. Zhai, T. Unterthiner, J. Yung. D. Keysers, J. Uszkoreit, M. Lucic, et al. Mlp-mixer: An all-mip architecture for vision. arXiv preprint arXiv:2105.01601, 2021.
[43] H. Touvron, Bojanowski, M. Caron, M. Cord, A. El-Nouby, E. Grave, A. Joulin, G. Synnaeve, J. Verbeek, and H. Jégou. Resmlp: Feedforward networks for image classification with data- efficient training. arXiv preprint arXiv:2105.00 033444, 2021.
[44] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. arXiv preprini arXiv: 170.03376, 2017.
[45] E. Voita, R. Sennrich, and I. Titov. The bottom-up evolution of representations in the transformer: study with machine translation and language modeling objectives. In EMNLP, 2019.
[46] B. Wu, C. Xu, X. Dai, A. Wan, P. Zhang, Z. Yan, M. Tomizuka, J. Gonzalez, K. Keutzer, and P. Vajda. Visual transformers: Token-basee image representation and processing for computer vision. arXiv preprint arXiv:2006. 03677, 2020.
[47] M. WU, Y. Belinkov, H. Sajjad, N. Durrani, F. Dalvi, and J. Glass. Similarity analysis of contextual word representation models. arXiv preprint 20Xiv=2005.01172. 2020.
12

---

[48] S. Wu, A. Conneau, H. Li, L. Zettlemoyer, and V. Stoyanov. Emerging cross-lingual structure pretrained language models. arXiv preprint arXiv:1911.01464 2019.
[49] L. Yuan, Y. Chen, T. Wang, Yu, Y. Shi, Z. Jiang, F. E. Tay, J. Feng, and S. Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. arXiv preprint arXiv:2101.11986, 2021.
[50] X. Zhai, J. Puigcerver, A. Kolesnikov, P. Ruyssen, C. Riquelme, M. Lucic, J. Djolonga, A. S. Pinto, M. Neumann, A. Dosovitskiy, et al. The visual task adaptation benchmark. 2019.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 9
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [No]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running ex- periments multiple times)? [No] Where possible, we show variation arising from different sampling aspects, that dataset samples or model runs. Note that for some of the pretrained models, we only have access to pretrained checkpoint, and show results across multiple model architectures to demonstrate the results hold more broadly.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [No]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "jQtq2m7dOhG", "reviewer": "Reviewer_SJG2", "summary": "The paper visualizes and analyzes the representation structure of ViT models and CNN. Results reveal intriguing findings, such as 1) ViT models yield more uniform representations, whereas CNNs yield stage-wise representations; 2) ViT models incorporate more global information in lower layers than CNNs; 3) ViT models, like CNNs, aggregate local information in lower layers; 4) Skip connections in ViT are essential.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The findings of the paper are interesting and valuable and should be shared with the broad research community. I do, however, have a few concerns or suggestions regarding the setups of a few experiments in the paper, as follows.\n\n-\tFigure 1 shows that ViT’s representation is more uniformly distributed compared to CNN’s representation by computing CKA on features. In L93 the authors wrote that these features were taken from normalization layers as well. ViT models, in contrast to CNNs, use layer normalization, instead of batch normalization. The layer norm of ViT applies normalization on \\emph{individual} tokens (each token in an HxW feature map is treated independently), whereas batch norm applies norm on HxW feature maps. The individual token normalization in ViT may have resulted in this uniformly distributed heat map in Figure 1. I encourage the authors to discuss, or provide evident about, the critical detail and its potential impact to the current observations.\n-\tIn the same figure the authors visualized the representation of ResNet. ResNet is designed in stages, a conv stem plus Res2, 3, 4 and 5. It seems that the heatmap shows such stage-wise distribution, i.e., intra-stage representations are more similar than inter-stage ones. Is this after all surprising?\n-\tIn Figure 3 the authors plotted attention distribution for the first two and the last two blocks in ViTs. What about intermediate blocks? Any interesting behavior?\n-\tIn Figure 4 the authors argued that “with less training data, lower attention layers do not learn to attend locally\". However, ViT-L/16 and ViT-H/14 are very big models, plus ViT models are notorious data thirsty. Does this observation still hold for ViT-B/16?\n-\tFigure 5 plotted the first encoder block of ViTs. This to me more of a cherry pick if only the first encoder block is visualized. Quantitative results are highly encouraged.\n-\tIn Figure 6 the authors visualized ViT-B/32 model versus a ResNet50. ViT-B/32 takes coarse input sequence. Have the authors tried the ViT-S/16 (in DeiT paper) model?\n-\tL168 the authors wrote it was “[striking that]” CLS token first relied on skip connection then main branch, whereas the reverse applies for spatial tokens. Why is this striking? CLS token would be just noise in early layers, and only when the representation gets better in deep layers CLS tokens would gather class information; spatial tokens are formed in early layers, thus in deep layers they are changed minimally.\n-\tIt is interesting that linear probing of intermediate ViT blocks yields much better accuracy than CNNs. However, since the model structure of ViT is uniform (simply stacking the same blocks for 12 or 24 times), whereas CNNs generally have stages, it is likely that the representation of ViT, because of the uniform design, is more uniform. Is this true or somewhat true?\n-\tThe authors sometimes use ViT-L or ViT-H, and sometimes ViT-B. What’s the general principle behind this? \n\nSome minor stuff\n-\tThe authors proclaimed, at the beginning of the paper, that “[the success of CNNs] is largely due to the powerful inductive bias of spatial equivariance encoded by convolutional layers …” First, do the authors really want to attribute all the huge success of CNNs to inductive bias? This seems to be a way too strong claim. Second, do convolution layers encode “equivariant” representations? Hinton seems to disagree with it, which is precisely the reason that he has been working on the capsule networks.\n\nOverall, I find many of the observations presented in the paper very interesting and valuable. On the other hand, I hope that the authors could adequately address my concerns and questions as stated above, and based on the authors’ feedback the final rating may be downgraded or upgraded.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "dYUnCHh1ztx", "reviewer": "Reviewer_qxEj", "summary": "The paper presents a comparison study between internal representations of recently Vision Transformers and ResNet networks for image classification, using various established tools: Centered kernel alignment, attention distance analysis, effective receptive fields, representation propagation analysis, spatial localization analysis and linear probes. The also train some new models for additional analysis. The findings show significant difference between the models.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Quality: the most intriguing findings to my mind are:\n- the observation that global average pooling reduces localization performance compared to CLS token. Localization performance of recently proposed Transformer-based detector might not correlate with the proposed localization study compared to Faster R-CNN-based detectors, since there is no requirement from features to correlate with positions in an input image. Would be interesting to confirm this by training object detectors on top of CLS token and GAP backbones.\n- the observation that ViT needs more data to learn \"local filters\" compared to CNN. This would indeed explain subpar performance of ViT in small data regime.\n- the observation that class label information emerges much earlier in ViT than in ResNet (linear probing). \n\nClarity: the paper is very well written and is easy to follow and understand.\n\nSignificance: the study is very timely. The fact that ViT which was designed for a very different task is able to match long-standing ResNet in image classification performance is intriguing, and demands such studies. The study might inspire new architectures for both CNNs and ViTs and their combinations, as well as their downstream applications.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors do not discuss potential negative societal impact of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "7vu420NUwtQ", "reviewer": "Reviewer_qGAU", "summary": "In this work, the authors explore how representations learned with vision transformers (ViT) compare with those of convolutional neural networks (resnet). Making use of centered kernel alignment (CKA), they compare the representations learned at different layers of the same mode, between models, and at different spatial positions. They find that ViT tends to learn similar representations  across all layers while resnets tend to show a clear distinction between lower-level and higher-level representations. They also provide experiments showing that residual connections play a crucial role in transformers, i.e. removing a residual connection causes a big change between preceding and following representations. Another interesting phenomenon in ViT is that residual connections tend to preserve information of the classification token during the first half of the model while it tends to change during the second half of the model. In addition, they compare how ViT and ResNets treat spatial information. They find that both process local information in the first layers but ViT uses global information earlier than ResNets. In fact, they show that while local processing is \"hardcoded\" in convolutions, ViT requires larger training sets to learn such inductive bias. Finally, they show that MLP Mixer is closer to ViT than to ResNet.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Overall Review\nThis work presents an interesting study that sheds some light on the differences between ViT and ResNets. I found the different experiments and conclusions very insightful and I think they are valuable to the research community. I missed some discussion on hybrid models such as [A, B, C] (which I encourage the authors to add, not necessarily citing these three papers), but overall the strengths clearly outweight the weaknesses and thus I recommend this work for acceptance.\n\n[A] *Srinivas, Aravind, et al. \"Bottleneck transformers for visual recognition.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.*\n\n[B] *d'Ascoli, Stéphane, et al. \"Convit: Improving vision transformers with soft convolutional inductive biases.\" arXiv preprint arXiv:2103.10697 (2021).*\n\n[C] *Wu, Haiping, et al. \"Cvt: Introducing convolutions to vision transformers.\" arXiv preprint arXiv:2103.15808 (2021).*\n\n## Strengths\n* This work is interesting and it provides valuable insights\n* The text is clear and easy to read\n* The experiments are technically sound\n\n## Weaknesses\n* Most comparisons are based on CKA but the authors already acknowledge it in the Limitations section.\n* The fact that the first layers of ViT require large datasets to learn the concept of spatial locality indicates that substituting the first layers of ViT for convolutions could greatly reduce the amount of data required to train vision transformers. It would be interesting if you could include some discussion or even an experiment to explore this.\n* You show how removing a residual connection affects the ViT. Is the effect similar for a ResNet?\n\n# Detailed comments\n## Originality\n* The experiments introduced in this work are original and the results are insightful.\n\n## Quality\n* Besides a couple of questions I asked in \"Weaknesses\", the overall quality is good.\n\n## Clarity\n* The text is well-written and easy to follow.\n\n## Significance\n* Given the current interest of the research community on vision transformers, this work is highly valuable in order to understand the behavior of these models compared to convolutional neural networks. \n\n## Reproducibility\nThe authors do not include the code and ViT is trained on JFT, which is a private dataset (not the fault of this work, but nonetheless anything based on JFT is very difficult to reproduce).\n\n## Other comments\nYou may find [D] interesting (in case you are not the authors of it). Since it is very recent, you do not need to cite it:\n\n[D] *Tuli, Shikhar, et al. \"Are Convolutional Neural Networks or Transformers more like human vision?.\" arXiv preprint arXiv:2105.07197 (2021).*\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors discuss limitations but not societal impact. A possible societal impact of better understanding image classification models could be to better understand their failures and limitations in real-life applications.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "1Ofsv7AxCnW", "reviewer": "Reviewer_sCgD", "summary": "Neural network architectures based on self-attention instead of convolutions (e.g., the Vision Transformer, ViT), have recently be shown to reach similar or better performances on ImageNet classification than CNNs. In this work, the authors investigate whether the \"internal representation structure\" learned by Vision Transformers is also similar to CNNs. Using Centered Kernel Alignment as main metric to compare internal representations, the authors perform several analyses comparing aspects of three Vision Transformer variants (ViT-B/32, ViT-B/16, ViT-L/16) and two ResNet variants (ResNet50x1, ResNet152x2). The analyses performed look at similarity of representations across different depths, local vs. global information aggregation, importance of skip connections, spatial localization of information and effects of the amount of training data. The analyses reveal several differences between the ViTs and ResNets, indicating that despite similar classification performance, the image processing strategies are quite different for the two model families.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Originality\n- The models discussed in the paper as well as the analysis methods in the paper are not new. Both are cited appropriately. To my knowledge, the analysis presented in the paper have not been performed before.\n\n### Quality\n- The main message of the paper is that ViTs and CNNs have a very different \"internal representation structure\" (L6). The authors perform a broad range of analyses targeting many aspects of the internal representation structures. Regarding ResNets, those analyses in its entirety support the main message well.\n\n- However the authors claim generalizability of their results to CNNs in general, by considering the two Resnet variants a \"representative set of CNN [...] models\" (L.65f). In my view, testing only two variants of a single model family is not sufficient to support this general claim.\n\n- A downside of the broad analysis strategy in my opinion, is that the individual analyses and the individual insights obtained into the inner workings of ViTs are rather shallow. Only few analyses are performed for every aspect. They suggest some underlying principles regarding the inner workings of ViTs, but empirical evidence for some of those principles should be strengthend by additional analyses (at the cost of handling fewer aspects in the paper) to go beyond the main message and answer \"how are Vision Transformers solving these tasks?\" (L.4) I will provide more details on the individual aspects below. In any case, I think the paper would profit from discussing how the trade-off between analysis depth and broadness has been chosen to answer the research questions at hand.\n\n- ViT and CNNs are very different architectures, so it is very plausible that their internal representation structures differ. To me it is not clear to which degree a ViT can actually mimic a CNN, and the paper does not discuss this point. This could be adressed by, e.g., a theoretical analysis or training a ViT to behave as similarly to a CNN as possible (e.g. regarding the similarity structure of the layers or the \"receptive field sizes\"). Showing first, that a ViT could behave very similar to a CNN in principle, and afterwards presenting the respective result showing that the ViT learns a very different strategy instead, would make the points much stronger.\n\n- Section 4: ViTs are shown to have highly similar representations throughout the model as measured by CKA. However in Section 8 it is found that the accuracy of linear classifiers rises when trained on the representation of later layers in the model. So both metrics seem to measure something different. It should be discussed in the paper, how those measures differ and how the seemingly contradictory results can be explained.\n\n- Section 5: It is concluded that using local information early is imoprtant for a strong performance (L.133). In my opinion, this should be analyzed in more depth. It could still be possible that a smaller range of attention is a by-product of the larger training set and not required for the performance. An additional experiment could be to train a ViT only on ImageNet while regularizing the self-attention modules to show local attention (and vice versa) and observe effects on the performance.\n\n- Section 6: Removing a skip connection is shown to have a very strong effect on the similarity structure. The decrease in the ovverall network performance (4%) is comparably mild. Similar as before, I think this deserves more discussion.\n\n- Section 7: The plots in Figure 9 visualizing the spatial localization for the ResNet show that it's \"tokens\" are much less spatially discriminative. However, models based on ResNets have been shown to work well for tasks requiring localized information. More details on this aspect would support the understanding of these observations. E.g., comparing to a ResNet backbone after being finetuned, does this \"correct\" the bad spatial localization? Do ViT models transfer to tasks requiring localization more easily?\n\n- The discussion of the MLP-Mixer is announced in the abstract, but not contained in the main paper and in my view not necessary to support the main message of the paper\n\n\n### Clarity\n- Overall, the experiments are described well and contain all the information necessary for reproducing the results, when given the trained models.\n\n- One exception is the analysis described in L134ff and Figure 5: How are the subsets chosen (i.e. how many heads)? The text only mentions the first encoder block, but the Figure also includes block2unit{1,2}. What is the \"Mean Distance\" on the x-axis? Does this correspond to the different subsets?\n\n- The broad range of analyses naturally makes this paper more complex. When skimming over the paper to get a first impression, it is therefore not easy to distill the main points. One idea to make this slightly easier could be to highlight the main research question and general approach more, compared to summarizing the individual results (e.g. in the abstract and the bullet list of contributions). E.g., something along the lines: 1. We want to study whether the representational structure is different in CNNs and ViTs. 2. We approach this from different angles, by comparing many aspects of the representations of CNNs of ViTs. 3. Throughout the analyses, we find clear differences between CNNs and ViTs. 4. Furthermore, some of the analyses hint at principles underlying the inner workgings of ViTs.\n\n### Significance\n- Object recognition is a core task for machine learning and computer vision. While dominated by CNNs during the last years, the ViTs have been proposed as a promising alternative. Works aimed at better understanding the similarities and differences of both approaches, are therefore highly interesting for researchers in the field.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "- Limitations are briefly discussed in L235-238. However I think the limitations of CKA should be addressed in greater detail. As described above, results according to the CKA can be very different compared to those obtained by linear probes or final model performance. So CKA is quantifying only some aspects of the representational similarity. In my opinion, this limitation should be discussed and argued why the aspects quantified by CKA are important for understanding the models.\n\n- Societal impacts are not discussed in the paper, the authors answered the respective question in the checklist with \"N/A\". However I think there are societal impacts that should be discussed. For example, better understanding the inner workings of DNNs could help to build trust in those models in the long run.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}]

Author rebuttal:

Meta-review: A good paper on a timely topic. All reviewers recommend acceptance. Could be a spotlight presentation.
Decision: Accept (Poster)