Do Vision Transformers See Like Convolutional Neural Networks?
Authors: Maithra Raghu
Thomas Unterthiner
Simon Kornblith
Chiyuan Zhang
Alexey Dosovitskiy
Venue: NeurIPS 2021 Poster (2021)
Keywords: Vision Transformers
Representation Analysis
Representation Learning
Representation Similarity
Computer Vision
Convolutional Neural Networks

Abstract: Convolutional neural networks (CNNs) have so far been the de-facto model for visual data. Recent work has shown that (Vision) Transformer models (ViT) can achieve comparable or even superior performance on image classification tasks. This raises a central question: how are Vision Transformers solving these tasks? Are they acting like convolutional networks, or learning entirely different visual representations? Analyzing the internal representation structure of ViTs and CNNs on image classification benchmarks, we find striking differences between the two architectures, such as ViT having more uniform representations across all layers. We explore how these differences arise, finding crucial roles played by self-attention, which enables early aggregation of global information, and ViT residual connections, which strongly propagate features from lower to higher layers. We study the ramifications for spatial localization, demonstrating ViTs successfully preserve input spatial information, with noticeable effects from different classification methods. Finally, we study the effect of (pretraining) dataset scale on intermediate features and transfer learning, and conclude with a discussion on connections to new architectures such as the MLP-Mixer. 

Do Vision Transformers See Like Convolutional Neural Networks?
Maithra Raghu
Thomas Unterthiner Google Research, Brain Team unterthiner@google.cc com
Simon Kornblith Google Research, Brain Team kormblith@google.. com
Google Research, Brain Team mai taithar@@maal.. com
Chiyuan Zhang Google Research, Brain Team chi yuiiyuu@ggoglecomm
Alexey Dosovitskiy Google Research, Brain Team
adosovitakkiyggogle.. com
Abstract
Convolutional neural networks (CNNs) have so far been the de-facto model for visual data. Recent work has shown that (Vision) Transformer models (ViT) achieve comparable or even superior performance on image classification tasks. This raises central question: how Vision Transformers solving these tasks? Are they acting like convolutional networks, or learning entirely different visual representations? Analyzing the internal representation structure of ViTs and CNNs on image classification benchmarks, we find striking differences between the two ar- chitectures, such VIT having explore how these differences arise, finding crucial roles played by self-attention, which enables early aggregation of global information, and ViT residual connec- tions, which strongly propagate features from lower to higher layers. We study the ramifications for spatial localization, demonstrating ViTs successfully preserve input spatial information, with noticeable effects from different classification meth- ods. Finally, we study the effect of (pretraining) dataset scale on intermediate features and transfer learning, and conclude with discussion on connections to new architectures such as the M-.--Mixer.
uniform representations across all layers. We
Introduction
Over the past several years, the successes of deep learning on visual tasks has critically relied convolutional neural networks [20, 16). This is largely due to the powerful inductive bias of spatial equivariance encoded by convolutional layers, which have been key to learning general purpose visual representations for easy transfer and strong performance. Remarkably however, recent work has demonstrated that Transformer neural networks are capable of equal or superior performance on image classification tasks at large scale [14]. These Vision Transformers (ViT) operate almost identically Transformers used in language [13], using self-attention, rather than convolution, to aggregate information across locations. This is in contrast with large body of prior work, which has focused on more explicitly incorporating image- speciffc inductive biases [30. 4]
This breakthrough highlights a fundamental question: how are Vision Transformers solving these image based tasks? Do they act like convolutions, learning the same inductive biases from scratch? Or are they developing novel task representations? What the role of scale in learning these representations? And there ramifications for downstream tasks? In this paper, we study these questions, uncovering key representational differences between ViTs and CNNs, the ways in which these difference arise, and effects on classification and transfer learning. Specifically, our contributions are:
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

We investigate the internal representation structure of ViTs and CNNs. finding striking differences between the two models, such as ViT having more uniform representations, with greater similarity between lower and higher layers.
Analyzing how local/global spatial information is utilised, we find VIT incorporates more global information than ResNet at lower layers, leading to quantitatively different features.
Nevertheless, we find that incorporating local information at lower layers remains vital, with large-scale pre-training data helping early attention layers learn to do this
We study the uniform internal structure of VIT. finding that skip connections in VIT are even more influential than in ResNets, having strong effects on performance and representation similarity.
Motivated by potential future uses in object detection, we examine how well input spatial informa- tion is preserved, finding connections between spatial localization and methods of classification. We study the effects of dataset scale on transfer learning, with a linear probes study revealing its importance for high quality intermediate representations.
2 Related Work
Developing non-convolutional neural networks to tackle computer vision tasks, particularly Trans- former neural networks [44] has been an active area of research. Prior works have looked at local multiheaded self-attention, drawing from the structure of convolutional receptive fields [30, 36), directly combining CNNs with self-attention [4, 2, 46] or applying Transformers to smaller-size images [6, 9]. In comparison to these, the Vision Transformer [14] performs even less modification to the Transformer architecture, making it especially interesting to compare to CNNs. Since its development, there has also been very recent work analyzing aspects of ViT, particularly robustness [3, 31, 28] and effects of self-supervision [5, 7J. Other recent related work has looked at designing hybrid ViT-CNN models [49, 11], drawing on structural differences between the models. Comparison between Transformers and CNNs are also recently studied in the text domain [41].
Our work focuses on the representational structure of ViTs. To study VIT representations, draw on techniques from neural network representation similarity, which allow the quantitative comparisons of representations within and across neural networks [17, 34, 26, 19). These techniques have been very successful in providing insights on properties of different vision architectures [29, 22, 18), representation structure in language models [48, 25, 47, 21], dynamics of training methods [33, 24] and domain specific model behavior [27, 35, 38]. We also apply linear probes in our study, which has been shown be useful analyze the learned representations in both vision [1] and text [8, 32, 45]
models.
3 Background and Experimental Setup
Our goal is to understand whether there are differences in the way ViT' represent and solve image tasks compared to CNNs. Based the results of Dosovitskiy et al. [14], we take representative set of CNN and ViT models - ResNet5001, ResNet152x2, ViT-B/32, ViT-B/16, ViT-L/16 and ViT-H/14. Unless otherwise specified, models are trained on the JFT-300M dataset [40]. although we also investigate models trained on the ImageNet ILSVRC 2012 dataset [12, 37] and standard transfer learning benchmarks [50, 14]. We use a variety of analysis methods to study the layer representations of these models, gaining many insights into how these models function. We provide further details of the experimental setting in Appendix A.
Representation Similarity and CKA (Centered Kernel Alignment): Analyzing (hidden) layer representations of neural networks is challenging because their features are distributed across large number of neurons. This distributed aspect also makes it difficult to meaningfully compare represen- tations across neural networks. Centered kernel alignment (CKA) [17, 10] addresses these challenges, enabling quantitative comparisons of representations within and across networks. Specifically, CKA takes as input X IR xxx and Y E mmx which are representations (activation matrices), of two layers, with and P2 neurons respectively, evaluated on the same m examples. Letting K = X X and L YY denote the Gram matrices for the two layers (which measures the similarity of a pair of datapoints according to layer representations) CKA computes:
HSIC L) CKA(K, L) VHSIC(K, KICIK,KHHH L)
(1)
2

---

VIT-L/16
VIT-H/14
R50
R152
240
126
100
VIT-V14
150
500
125
Layers R50
20
R152
250
80
130
Layens
200
40
20
50
01
25
80
80
100
120
140
25
75 200
125
350
3.5
20
60 100 3.20
0
50
107
130
230
250
307
350
Layers VIF-L/16
Layers VIT-H/14
Layers
Layers
Figure I: Representation structure of VITs and convolutional networks show significant differences, with ViTs having highly similar representations throughout the model, while the ResNet models show much lower similarity between lower and higher layers. We plot CKA similarities between pairs of layers across different model architectures. The results are shown as healmap, with and axes indexing the layers from input to output. We observe that ViTs have relatively uniform layer similarity structure, with a clear grid-like pattern and large similarity between lower and higher layers. By contrast, the ResNet models show clear stages similarity structure, with smaller similarity scores between lower and higher layers.
VIT-L/16 VS R50
VIT-H/14 VS R50
Figure 2: Cross model CKA heatmap between ViT and ResNet illustrate that larger number lower layers ResNet are similar to a smaller set of the lowest VIT layers. We compute a CKA heatmap comparing all layers of ViT to all layers of ResNet. for two different ViT mod- els. We observe that the lower half of ResNet layers are similar to around the lowest quar- ter of ViT layers. The remaining half of the ResNet similar to approximately the next third of VIT layers, with the highest ViT layers dissimilar to lower and higher ResNet layers.
Layers VIT-L/16
Layers VIT-H/14
150
100
3.25
80
60
75
40
50
20
40
60
80
100
120
20
40
60
80 100
120
Layers R50
Layers R50
where HSTC is the Hilbert-Schmidt independence criterion [15]. Given the centering matrix H Ln 11 and the centered Gram matrices K' HKH H and L' HLH, HSIC(K, vec (') vee((') 111m-19) 1)2, the similarity between these centered Gram matrices. CKA invariant to orthogonal transformation of representations (including permutation of neurons). and the normal- izatiin term ensures invariance to isotropic scaling. These properties enable meaningful comparison and analysis of neural network hidden representations. To work at scale with our models and tasks, we approximate the unbiased estimator of HSTC [39] using minibatches, as suggested in [29].
4 Representation Structure of ViTs and Convolutional Networks
We begin our investigation by using CKA to study the internal representation structure of each model. How are representations propagated within the two architectures, and are there signs of functional differences? To answer these questions, we take every pair of layers X, Y within model and compute their CKA similarity. Note that we take representations not only from outputs of Vit/ResNet blocks, but also from intermediate layers, such normalization layers and the hidden activations inside a ViT MLP. Figure shows the results as a heatmap, for multiple ViTs and ResNets. We observe clear differences between the internal representation structure between the two model architectures: (I) ViTs show a much more uniform similarity structure, with a clear grid like structure (2) lower and higher layers in ViT show much greater similarity than in the ResNet, where similarity is divided into different (lower/higher) stages.
We also perform model comparisons, where we take all layers X from ViT and compare to all layers Y from ResNet. We observe (Figure 2) that the lower half of 60 ResNet layers are similar to approximately the lowest quarter of ViT layers. In particular, many more lower layers in the ResNet are needed to compute similar representations to the lower layers of ViT. The top half of the ResNet approximately similar to the next third of the ViT layers. The final third of VIT layers is less similar to
3

---

VIT-L/16 JFT to ImageNet
VIT-H/14 JFT to ImageNet
120
120
100
100
Mean Distance
Mean Distance
80
80
60
60
encoder_biin blockO
encoder, _blocko
encoder. block1 encoder: block22
encoder, block1 encoder block30 encoder block31
20
20
encoder block23
0
2
10
12
14
0
2
10 12 14
Sorted Attention Head
Sorted Attention Head
Figure 3: Plotting attention head mean distances shows lower ViT layers attend both locally and globally, while higher layers primarily incorporate global information. each attention head, we compute the pixel distance it attends to, weighted by the attention weights, and then average over 5000 datapoints to get an average attention head distance. We plot the heads sorted by their average attention distance for the two lowest and two highest layers in the ViT, observing that the lower layers attend both locally and glohally, while the higher layers attend entirely glohally.
all ResNet layers, likely because this set of layers mainly manipulates the CLS token representation, further studied in Section 6.
Taken together, these results suggest that (i) ViT lower layers compute representations in a different way to lower layers in the ResNet, (ii) VIT also strongly propagates representations between lower and higher layers (iii) the highest layers of VIT have quite different representations to ResNet.
5 Local and Global Information in Layer Representations
In the previous section, we observed much greater similarity between lower and higher layers in VIT, and we also saw that ResNet required more lower layers to compute similar representations to smaller set of ViT lower layers. In this section, we explore one possible reason for this difference: the difference in the ability to incorporate global information between the two models. How much global information is aggregated by early self-attention layers in ViT? Are there noticeable resulting differences to the features of CNNs, which have fixed, local receptive fields in early layers? In studying these questions, we demonstrate the influence of global representations and a surprising connection between scale and self-attention distances.
start by analyzing VIT self-attention layers, which are mechanism for ViT to aggregate information from other spatial locations, and structurally very different to the fixed receptive field sizes of CNNs. Each self-attention layer comprises multiple self-altention heads, and for each head we compute the average distance between the query patch position and the locations attends to. This reveals how much local vs global information each self-attention layer is aggregating for the representation. Specifically, the attention weights for each attention head and average over 5000 datapoints, with results shown in Figure 3. In agreement with Dosovitskiy et al. [14], we observe that even in the lowest layers of ViT, self-attention layers have a mix of local heads (small distances) and global heads (large distances). This is in contrast to CNNs, which are hardcoded to attend only locally in the lower layers. At higher layers, all self-attention heads are global.
Analyzing Attention Distances:
weight the pixel distances by
clear effect of scale on attention. In Figure 4, we look at attention distances when training only on ImageNet (no large-scale pre-training), which leads to much lower performance in VIT-L/16 and ViT-H/14 [14]. Comparing Figure 3, we see that with not enough data, VIT does not learn to attend locally in earlier layers. Together, this suggests that using local information early on for image tasks (which is hardeoded into CNN architectures) is important for strong performance.
Interestingly,
Does access to global information result in different features? The results of Figure demonstrate that ViTs have access to more global information than CNNs in their lower layers. But does this result in different learned features? an interventional test, we take subsets of the ViT attention heads from the first encoder block, ranging from the subset corresponding the most local attention

---

VIT-L/16 on ImageNet
VIT-H/14 on ImageNet
120
120
100
100
Mean Distance
Mean Distance
80
80
60
60
encoder_ encoder_block0 blockO
encoder, Bllockk
encoder bleckt
encoder. blockt
20
encoder block22
20
enceder block30
encoder Block23
encoder blork31
10
12
14
12 Sorted Attention Head
10
14
Sorted Attention Head
Figure 4: With less training data, lower attention layers do not learn atfend locally. Comparing the results to Figure 3, we see that training only on ImageNet leads to lower layers not learning to attend more locally. These models also perform much worse when only trained on ImageNet, suggesting that incorporating local features (which is hardeoded into CNNs) may be important for strong performance. (See also Figure C.5.)
and Global VIT-L/16
Local Representations compared to R50
VIT-H/14
VIT-L/16 Representations compared to R15:
Representations compared to R50
0.55
0.50
6.55
OKA Similarity
0.50
CKA Similarity
0.50
GKA Similarity
0.45
0.40
0.40
blockdunitl
Hockiunitt
0.35
bocklun81
biockluntt2
0.35
bocckuuit2
biockluni2
biockluntts
blocklunit3
bocklunt3
0.30
block2uniti
Bockkuuii1
beck2unk1
block2uni22
buck2unk22
6.25
0.5
40
50
60
70
80
50
100
20
40
60
30
100
40
50
60
70
80
100
Mean Distance
Mean Distance
Mean Distance
Figure 5: Lower layer representations of ResNet are most similar to representations corresponding to local attention heads of VIT. We take subscts of ViT attention heads in the first encoder block, ranging from the most locally attending heads (smallest then compute CKA similarity between these subsets and lower layer representations in ResNet. We observe that lower ResNet layers are most similar to the features learned by local attention heads of ViT, and decrease monotonically in similarity as more global information is incorporated. demonstrating that the global heads learn quantitatively different features.
distance) the global heads (largest
distance). We
heads to a subset of the representation corresponding to the most global attention heads. We then compute CKA similarity between these subsets and the lower layer representations of ResNet.
The results, shown in Figure 5, which plot the mean distance for each subset against CKA similarity, clearly show monotonic decrease in similarity access to more global information also leads to quantitatively different features than computed by the local receptive fields in the lower layers of the ResNet.
attention distance grows, demonstrating that
Effective Receptive Fields: We conclude by computing effective receptive fields [23] for both ResNets and ViTs, with results in Figure 6 and Appendix C. We observe that lower layer effective receptive fields for VIT are indeed larger than in ResNets, and while ResNet effective receptive fields grow gradually, ViT receptive fields become much more global midway through the network. ViT receptive fields also show strong dependence on their center patch due to their strong residual connections, studied in the next section. As show in Appendix C, in attention sublayers, receptive fields taken before the residual connection show far less dependence on this central patch.
Representation Propagation through Skip Connections
The results of the previous section demonstrate that ViTs learn different representations ResNets in lower layers due access to global information, which explains some of the differences in represen-
5

---

VIT-B/32 Attention
Attention on
Attention 3
Attention 9
Attention 12
Figure 6: ResNet effective re- ceptive fields are highly local and grow gradually; VIT effec- tive receptive fields shift from local to global. We measure the effective receptive field of different layers the absolute value of the gradient of the cen- location of the feature map (taken after residual connections) with respect the input. Results averaged across all channels in each map for 32 randomly- sclected images.
ResNet-50 Block 7
Initial Conv
Block
Block 12
Block 16
Ratio of Norms (first few tokens)
Ratio of Norms (aggregated)
Average norn inpressetatioon
Token Index
Block Index
Figure Most information ViT passes through skip connections. Comparison representation norms between the skip-connection (identity) and the long branch for ViT-B/16 trained on ImageNet and a ResNet. For ViT, we show the CLS token separately from the rest of the representation. (left) shows the ratios separated for the first few tokens (token 0 is CLS), (right) shows averages all tokens.
tation structure observed in Section 4. However, the highly uniform nature VIT representations (Figure 1) also suggests lower representations are faithfully propagated to higher layers. But how does this happen? In this section, we explore the role of skip connections in representation propagation across ViTs and ResNets, discovering ViT skip connections are highly influential, with clear phase transition from preserving the CLS (class) token representation (in lower layers) to spatial token representations (in higher layers).
Like Transformers, ViTs contain skip (aka identity shortcu)) connections throughout, which added on after the (i) self-attention layer, and (ii) MLP layer. To study their effect, we plot the the hidden representation of the ith layer coming from the skip connection, and f(z)) is the transformation of z, from the long branch (i.c. MLP or self-attention.)
norm ratio [|=/ (llff((()) (2()l where
The results in Figure 7 (with additional cosine similarity analysis in Figure E.2.) The heatmap on the left shows | for different token representations. We observe a striking phase transition: in the first half of the network, the CLS token (token 0) representation is primarily propagated by the skip connection branch (high norm ratio), while the spatial token representations have a large contribution coming from the long branch (lower norm ratio). Strikingly, in the second half of the network, this is reversed.
The right pane, which has line plots of these norm ratios across ResNet50, the ViT CLS token and the ViT spatial tokens additionally demonstrates that skip connection much more influential ViT compared to ResNet: observe much higher norm ratios for ViT throughout, along with the phase transition from CLS to spatial token propagation (shown for the MLP and attentioo layers.)
VIT Representation Structure without Skip Connections:
The norm ratio results strongly suggest that skip connections play a key role the representational structure of VIT. test this intervention- ally, we train ViT models with skip connections removed in block for varying i, and plot the CKA representation heatmap. The results, in Figure 8, illustrate that removing the skip connections in block partitions the layer representations either side. (We note performance drop of 4% when

---

Remove Skpp Connection
Connection Block
Connectonn Block
1.
10 26 9. 50 1. 29
Figure VIT models trained without any skip connections in block show very little representation similarity between layers before/after block i. We train several VIT models without any skip connections at block for varying i to interventionally test the effect on representation structure. For middle blocks without skip connections, we observe performance drop of 4%. We also observe that removing skip connection block partitions similar representations before/after block this demonstrates the importance of skip connections in ViT's standard uniform representation structure.
ViT-B/32 final block
Token Location(O, 0) Token Location(2, Token at Location(4, 2) Token Location(4,
VIT-L/16 final block
TokenO Location(0, 0) Token31 Location(2,
3) Token35 Location(2, 7) Token61 Location(4, 5)
12
012345678910123 01234567880122
01234567852032
R50x1 final unit final block
Token at Location(o, 0) Taken at Location(2, Token at Location(4, 2) Token at Location(4, 4)
Figure 9: Higher layers of VIT maintain spatial location information more faithfully than ResNets. Each heatmap plot shows the CKA similarity between single token representation in final block of the model and the input images, which divided into non-overlapping patches. We observe that VIT tokens have strongest similarity their corresponding spatial location the image, but tokens corresponding to spatial locations the edge of the image (e.g. token 0) additionally show similarity to other edge positions. This demonstrates that spatial information from the input is preserved even the final layer of VIT. By contrast, ResNet "tokens" (features a specific spatial location) much less spatially discriminative, showing comparable similarity across broad of input spatial locations. See Appendix for additional layers and results.
removing skip connections from middle blocks.) This demonstrates the importance of representations being propagated by skip connections for the uniform similarity structure of ViT in Figure 1.
Spatial Information and Localization
The results so far, on the role of self-attention in aggregating spatial information in ViTs, and skip- connections faithfully propagating representations to higher layers, suggest an important followup question: how well can ViT's perform spatial localization? Specifically, is spatial information from the input preserved in the higher layers of VIT? And how does it compare in this aspect ResNet? An affirmative answer to this is crucial for uses of VIT beyond classification, such as object detection.
7

---

VIT-B/32 GAP final block
Token at Location(O, 0) Taken at Location(2, Token at Location(4, 2) Taken at Location[4, 4)
Figure 10: When trained with global average pooling (GAP) iGAPead instead of a CLS token, VTTs show less clear localization (compare Figure 9). We plot the same CKA heatmap between token and different input images patches as in Figure but for ViT model trained with global average pooling (like ResNet) instcad of CLS token. observe significantly less localization.
70
70
VIT-B/32 CLS
CLS CLS tolken
VIT-B/32 GAP
60
CLS; except taken
ResNet50
CLS) GAP
50
50
First
Telt accuracy
40
Test accuracy
40
GAP: GAP
30
20
20
10
10
1.0
0.6 Layer number
Normalized layer number
(a) Individual token evaluation
(b) CLS vs GAP models
Figure 11: Spatial localization experiments with linear probes. We train linear classifiers on 10-shot ImageNet classification from the representations extracted from different layers of ViT-B/32 models. We then plot accuracy the probe versus the (normalized) layer number. Left: We train classifier on each token separately and report the average accuracy over all tokens (excluding the CLS token for the ViT CLS model.) Right: Comparison of VIT models pre-trained with a classification token or with global average pooling (GAP) and then evaluated with different ways of aggregating the token representations.
We begin by comparing token representations in the higher layers of ViT and ResNet to those of input patches. Recall that ViT tokens have a corresponding input patch, and thus a corresponding input spatial location. For ResNet, we define token representation to be all the convolutional channels at particular spatial location. This also gives it a corresponding input spatial location. We can then take token representation and compute its CKA score with input image patches different locations. The results are illustrated for different tokens (with their spatial locations labelled) in Figure 9.
For VIT, we observe that tokens corresponding to locations at the edge of the image are similar to edge image patches, but tokens corresponding to interior locations are well localized, with their representations being most similar to the corresponding image patch. By contrast, for ResNet, significantly weaker localization (though Figure D.3 shows improvements for earlier layers.)
One factor influencing this clear difference between architectures is that ResNet is trained to classify with global average pooling step, while VIT has separate classification (CLS) token. To examine this further, we test a VIT architecture trained with global average pooling (GAP) for localization (see Appendix A for training details). The results, shown Figure 10, demonstrate that global average pooling does indeed reduce localization in the higher layers. More results in Appendix Section D.
Localization and Linear Probe Classification: The previous results have looked at localization through direct comparison of each token with input patches. To complete the picture, we look at using each token separately to perform classification with linear probes. We do this across different layers of the model, training linear probes to classify image label with closed-for fooed few-shot linear regression similar to Dosovitskiy al. [14] (details in Appendix A). Results are in Figure 11, with further results in Appendix F. The left pane shows average accuracy of classifiers trained on individual tokens, where we see that ResNet50 and ViT with GAP model tokens perform well at higher layers, while in the standard ViT trained with CLS token the spatial tokens do poorly likely because their representations remain spatially localized higher layers, which makes global classification challenging. Supporting this are results the right pane, which shows that single token from the
8

---

5mbrirt to 100% (CKG)
35 SV3 wTTTica
Block Index
Stock Index
Figure 12: Mcasuring similarity of representations Iearned with varying amounts of data shows the importance large datasets higher layers and larger model representations. of representations at each block for VIT models that have been trained on smaller subsets of the data to a model that has been trained on the full data on VIT-L/16 (left) and VIT-B/32 (right). We observe that while lower layer representations have high similarity significantly more data to learn similar representations.
compute the similarity
with 10% of the data, higher layers and larger models require
80
VIT-L/16
JFT-300M
80
VIT-B/32
70
JFT-30OM VIT-H/14
70
VIT-L/16 ImageNet
VIT-L/16
60
VIT:H/14 ImageNet
60
VITH/11
Test accuracy
50
50
ResNet50
ResNet152
40
40
ResNet152x2
30
30
20
10
1.0
Normallzedd layer number
Normalized layer number
(a) JFT-300M vs ImageNet pre- me--tranning
(b) ResNets
Figure 13: Experiments with linear probes. We train linear classifiers on 10-shot ImageNet classification from the aggregated representations of different layers of different models. We then plot the accuracy of the probe versus the (normalized) layer number. Left: Comparison of ViTs pre-trained JFT-300M ImageNet and evaluated with linear prohes on Imagenet. Right: Comparison of VIT ResNet models trained JFT-300m, evaluated with linear probes on ImageNet.
ViT-GAP model achieves comparable accuracy in the highest layer to all tokens pooled together. With the results of Figure 9, this suggests all higher layer tokens in GAP models learn similar (global) representations.
8 Effects of Scale on Transfer Learning
Motivated by the results of Dosovitskiy et al. [14] that demonstrate the importance of dataset scale for high performing ViT's, and our earlier result (Figure 4) on needing scale for local attention, we perform study of the effect of dataset scale on representations in transfer learning.
We begin by studying the effect on representations as the JFT-300M pretraining dataset size varied. Figure 12 illustrates the results on ViT-B/32 and ViT-L/16. Even with 3% of the entire dataset, lower layer representations are very similar to the model trained on the whole dataset, but higher layers require larger amounts of pretraining data to learn the same representations as large data scale, especially with the large model size. In Section G, we study how much representations change in finetuning, finding heterogeneity over datasets.
next look at dataset size effect on the larger ViT-L/16 and ViT-H/14 models. Specifically, the left pane of Figure 13, we train linear classifer probes on ImageNet classes for models pretrained on JFT-300M vs models only pretrained on ImageNet. We observe the JFT-300M pretained models achieve much higher accuracies even with middle layer representations, with 30% gap in absolute

---

accuracy to the models pretrained only on ImageNet. This suggests that for larger models, the larger dataset is especially helpful in learning high quality intermediate representations. This conclusion further supported by the results of the right pane of Figure 13, which shows linear probes on different ResNet and ViT' models, all pretrained on JFT-300M. We again see the larger ViT models learn much stronger intermediate representations than the ResNets. Additional linear probes experiments in Section F demonstrate this same conclusion for transfer to CIFAR-IO and CIFAR-100.
Discussion
Our study uses CKA [17], which summarizes measurements into a single scalar, to provide quantitative insights on representation similarity. While we have complemented this with interventional tests and other analyses (e.g. linear probes), more fine-grained methods may reveal additional insights and variations in the representations.
Limitations:
Conclusion: Given the central role of convolutional neural networks in computer vision break- throughs, it is remarkable that Transformer architectures (almost identical to those used in language) are capable of similar performance. This raises fundamental questions on whether these architec- tures work in the same way as CNNs. Drawing on representational similarity techniques, we find surprisingly clear differences in the features and internal structures of ViTs and CNNs. An analysis of self-attention and the strength of skip connections demonstrates the role of earlier global features and strong representation propagation ViTs for these differences, while also revealing that some CNN properties, e.g. local information aggregation at lower layers, are important to VITs, being learned from scratch at scale. We examine the potential for ViTs to be used beyond classification through study of spatial localization, discovering ViTs with CLS tokens show strong preservation of spatial information - promising for future uses in object detection. Finally, we investigate the effect of scale for transfer learning, finding larger ViT models develop significantly stronger intermediate representations through larger pretraining datasets. These results are also very pertinent to under- standing -lse- architectures for vision proposed by concurrent work [42, 43], further discussed in Section H, and together answer central questions on differences between ViTs and CNNs, and suggest new directions for future study. From the perspective of societal impact, these findings and future work may help identify potential failures as well as greater model interpretability.
References
[1] G. Alain and Y. Bengio. Understanding intermediate layers using linear classifier probes. arXiv preprint arXiv: 1610.01644, 2016.
[2] I. Bello, B. Zoph, A. Vaswani, J. Shlens, and Q. Le. Attention augmented convolutional networks. Proceedings of the IEEF/CVF International Conference on Computer Vision pages 3286-3295, 2019.
[3] Bhojanapalli, A. Chakrabarti, D. Glasner, Li, Unterthiner, and A. Veit. Understanding robustness of transformers for image classification. arXiv preprint arXiv:2103. 14586, 2021.
[4] N. Carion, Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko. End-to-end object detection with transformers. In European Conference Computer Vision, pages 213-229. Springer, 2020.
[5] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin. Emerging properties in self-supervised vision transformers. arXiv preprint arXiv:2 /0044//294,.. 2021.
[6] M. Chen, A. Radford, R. Child, J. Wu, H. Jun, D. Luan, and I. Sutskever. Generative pretraining from pixels. International Conference on Machine Learning, pages 1691-1703. PMLR, 2020.
[7] X. Chen, S. Xie, and K. He. An empirical study of training self-supervised vision transformers. arXiv preprint arXiv:2104.020577 2021.
[8] A. Conneau, G. Kruszewski, G. Lample, I.. Barrault, and Baroni. What you can cram into a single vector: Probing sentence embeddings for linguistic properties. In ACL, 2018.
10

---

[9] J.-B. Cordonnier, A. Loukas, and M. Jaggi. On the relationship between self- self-attention and convolutional layers. arXiv preprint arXiv: 191100541 2019.
[10] C. Cortes, M. Mohri, and A. Rostamizadeh. Algorithms for learning kernels based on centered alignment. The Journal of Machine Learning Research, 13(1):795-822 2012.
[11] S. d' Ascooli H. Touvron, M. Leavitt, A. Morcos, G. Biroli, and Sagun. Convit: Improving vision transformers with soft convolutional inductive biases. arXiv preprint 10Xi:2203.10697, 2021.
[12] J. Deng, W. Dong, R. Socher, L.,J. K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pages 248-255, leee, 2009.
[13] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint XXi::1810.04805, 2018.
[14] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenbom, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelll, al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 1929, 2020.
[15] A. Gretton, K. Fukumizu, C. H. Teo, L. Song. B. Schölkopf, A. J. Smola, et al. A kernel statistical test of independence. In Nips, volume 20, pages 585-592 Citeseer, 2007.
[16] A. Kolesnikov, L. Beyer, X. Zhai, J. Puigcerver. J. Yung. S. Gelly, and N. Houlsby. Big transfer (bit): General visual representation learning, arXiv preprint arXiv:1912.11377 6(2):8, 2019.
[17] S. Kornblith, M. Norouzi, H. Lee, and G. Hinton. Similarity of neural network representations revisited. In ICML, 2019.
[18] S. Kornblith, H. Lee, T. Chen, and M. Norouzi. What's 's in a loss function for image classification? arXiv preprint arXiv:2010..6411 2020.
[19] N. Kriegeskorte, M. Mur, and P. A. Bandettini. Representational similarity analysis-comnecting the branches of systems neuroscience. Frontiers in systems neuroscience, 2:4, 2008
[20] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25:1097-1105, 2012.
[21] S. R. Kudugunta, A. Bapna. I. Caswell, N. Arivazhagan, and O. Firat. Investigating multilingual nmt representations at scale. arXiv preprint arXiv:1909.02197, 2019.
[22] G. W. Lindsay. Convolutional neural networks as model of the visual system: past, present, and future. Journal of cognitive neuroscience, pages 1-15, 2020.
[23] W. Luo, Y. Li, R. Urtasun, and R. Zemel. Understanding the effective receptive field in deep convolutional neural networks. arXiv preprint arXiv: 2017.
[24] N. Maheswaranathan, A. H. Williams, M. D. Golub, S. Ganguli, and D. Sussillo. Universality and individuality in neural dynamics across large populations of recurrent networks. Advances in neural information processing systems, 2019:15629, 2019.
[25] A. Merchant, E. Rahimtoroghi, E. Pavlick, and I. Tenney. What happens to bert embeddings during fine-tuning? arXiv preprint arXiv:2004. 14448, 2020.
[26] A. S. Morcos, M. Raghu, and S. Bengio. Insights on representational similarity in neural networks with canonical correlation. arXiv preprint arXiv:1806.0557 0578809 2018
[27] B. Mustafa, A. Loh, J. Freyberg, P. MacWilliams, M. Wilson, S. M. McKinney, M. Sieniek,
J. Winkens, Y. Liu, P. Bui, et al. Supervised transfer learning at scale for medical imaging. arXiv preprint arXiv:2101. 00.9159,,, 2021.
[28] M. Naseer, K. Ranasinghe, S. Khan, M. Hayat, Khan, and M.-H. Yang. Intriguing properties of vision transformers, 2021.

---

[29] T. Nguyen, M. Raghu, and S. Kornblith. Do wide and deep networks learn the same things? uncovering how neural network representations vary with width and depth. arXiv preprint arXiv:2010. 15327, 2020.
[30] N. Parmar, A. Vaswani, J. Uszkoreit, L. Kaiser, N. Shazeer, A. Ku, and D. Tran. Image transformer. In International Conference on Machine Learning, pages 4055 4064, PMLR, 2018.
[31] S. Paul and P.-Y. Chen. Vision transformers are robust learners. arXiv preprint arXiv:2005.07581, 2021.
[32] M. E. Peters, M. Neumann, I.. Zettlemoyer, and W.-t Yih. Dissecting contextual word embed- dings: Architecture and representation. In EMNLP, 2018.
[33] A. Raghu, M. Raghu, S. Bengio, and O. Vinyals. Rapid learning or feature reuse? towards understanding the effectiveness of maml. arXiv preprint arXiv: 1999009557 2019.
[34] M. Raghu, Gilmer, J. Yosinski, and Sohl-Dickstein. Sveea: Singular vector canonical corre- lation analysis for deep learning dynamics and interpretability. arXiv preprint arXiv:/706.05800 2017.
[35] M. Raghu, C. Zhang, J. Kleinberg. and Bengio. Transfusion: Understanding transfer learning for medical imaging. arXiv preprint arXiv: /902.0770,, 2019.
[36] P. Ramachandran, N. Parmar, A. Vaswani. I. Bello, A. Levskaya, and J. Shlens. Stand-alone 2019.
self-attention vision models. arXiv preprint arXiv: I
[37] O. Russakovsky, J. Deng, H. Su, Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211-252, 2015.
[38] Shi, E. Shea-Brown, and M. Buice. Comparison against task driven artificial neural networks reveals functional properties in mouse visual cortex. Advances in Neural Information Processing Systems, 32:5764-5774, 2019.
[39] I.. Song, A. Smola, A. Gretton, Bedo, and K. Borgwardt. Feature selection via dependence maximization. Journal of Machine Learning Research, 13(5), 2012.
[40] C. Sun, A. Shrivastava, S. Singh, and A. Gupta. Revisiting unreasonable effectiveness of data in deep learning era. In Proceedings of the IEEE international conference on computer vision, pages 843-852, 2017.
[41] Y. Tay, M. Dehghani, J. Gupta, D. Bahri, V. Aribandi, Z. Qin, and D. Metzler. Are pre-trained convolutions better than pre-trained transformers? arXiv preprint 2rXiv:2105.08::
2021. [42] I. Tolstikhin, N. Houlsby, A. Kolesnikov, L. Beyer, X. Zhai, T. Unterthiner, J. Yung. D. Keysers, J. Uszkoreit, M. Lucic, et al. Mlp-mixer: An all-mip architecture for vision. arXiv preprint arXiv:2105.01601, 2021.
[43] H. Touvron, Bojanowski, M. Caron, M. Cord, A. El-Nouby, E. Grave, A. Joulin, G. Synnaeve, J. Verbeek, and H. Jégou. Resmlp: Feedforward networks for image classification with data- efficient training. arXiv preprint arXiv:2105.00 033444, 2021.
[44] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. arXiv preprini arXiv: 170.03376, 2017.
[45] E. Voita, R. Sennrich, and I. Titov. The bottom-up evolution of representations in the transformer: study with machine translation and language modeling objectives. In EMNLP, 2019.
[46] B. Wu, C. Xu, X. Dai, A. Wan, P. Zhang, Z. Yan, M. Tomizuka, J. Gonzalez, K. Keutzer, and P. Vajda. Visual transformers: Token-basee image representation and processing for computer vision. arXiv preprint arXiv:2006. 03677, 2020.
[47] M. WU, Y. Belinkov, H. Sajjad, N. Durrani, F. Dalvi, and J. Glass. Similarity analysis of contextual word representation models. arXiv preprint 20Xiv=2005.01172. 2020.
12

---

[48] S. Wu, A. Conneau, H. Li, L. Zettlemoyer, and V. Stoyanov. Emerging cross-lingual structure pretrained language models. arXiv preprint arXiv:1911.01464 2019.
[49] L. Yuan, Y. Chen, T. Wang, Yu, Y. Shi, Z. Jiang, F. E. Tay, J. Feng, and S. Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. arXiv preprint arXiv:2101.11986, 2021.
[50] X. Zhai, J. Puigcerver, A. Kolesnikov, P. Ruyssen, C. Riquelme, M. Lucic, J. Djolonga, A. S. Pinto, M. Neumann, A. Dosovitskiy, et al. The visual task adaptation benchmark. 2019.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 9
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [No]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running ex- periments multiple times)? [No] Where possible, we show variation arising from different sampling aspects, that dataset samples or model runs. Note that for some of the pretrained models, we only have access to pretrained checkpoint, and show results across multiple model architectures to demonstrate the results hold more broadly.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [No]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "rcsBrF1LH3", "reviewer": "Reviewer_fR12", "summary": "This paper provides a comprehensive analysis of visual features learned from vision transformers and CNNs. It applies central kernel alignment (CKA) to compare ViT and CNNs from several aspects mainly including a) local and global representations; b) roles of skip connections; and c) spatial localization.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1) ViT and its variants are demonstrating better performance than CNNs in many vision applications, thus understanding the differences between ViTs and CNNs is an important problem.\n2) The paper provides thorough analysis and comparisons of ViT and CNN features based on models trained on both small and large datasets.\n\nWeakness:\n1) This is basically an analysis paper. While some of its findings are helpful for understanding the differences between the two architectures, the primary claims about the local/global representations and skip connections are not extremely surprising. The paper also doesn't suggest much about what makes ViT superior to CNNs or how to improve the performance of ViT further.  \n\n2) I am not quite convinced by the analysis of spatial localization. Why is locality-preserving in the higher layers of ViT so relevant to object detection? How would this property be helpful for object detection if ViT is not learning contextual information? it would be better if the authors could add analysis on how this property would affect ViT in the task of object detection (either in a positive or negative way). \n\n3) Were all the models trained using the same dataset and augmentation in this paper?  It would be good to provide the classification accuracies of these models.  ResNet50 and ResNet152 are not the SOTA CNN models.  How would a better-performed model like EfficientNet affect the analysis?\n\n4) In Fig. 1, both CNN models, especially ResNet50, indicate two blocks along the diagonal. Is there any explanation for this pattern?\n \n5) Do different heads result in different mean attention distances? Some more analysis on this would be interesting.\n\n6) It’s not clear to me how the effective receptive field for VIT is defined and computed. \n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "no broader impact section is included in the paper", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "OvQJfZ9XDCq", "reviewer": "Reviewer_MKzA", "summary": "This paper deeply studies the working mechanism of vision transformers and explores the similarities and differences between vision transformers and CNNs.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- The originality of this paper is clear. Unlike most previous work that target designing novel network architectures or advanced attention mechanisms, this paper aims to deeply study what the vision transformers learn and help understand the differences between vision transformers and CNNs.\n\n- Through the representation analysis methods, the authors well visualize what the transformer layers learn at different network depth and show how vision transformers learn local and global information.\n\n- The presentation is good and the novelty is significant. I do think the explanations and conclusions can help in designing better vision transformers or hybrid networks.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "- Despite the comprehensive analysis, there is a lack of providing insights that could help design better network architectures.\n\n- The authors say that they provide ramifications for future applications in Section H but I cannot find exactly where the ramifications \n are.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "74AXGjzv6nL", "reviewer": "Reviewer_NaL7", "summary": "This work addresses some fundamental questions on Vision Transformers including 1. how are Vision Transformers solving vision tasks? 2. Do they act like convolutions, learning the same inductive biases from scratch? 3. And what is the role of scale in learning these representations? To answer these questions, they investigate the internal representation structure of ViTs and CNN by using representational similarity techniques like CKA (Centered Kernel Alignment). ", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper aims to answer some fundamental questions on Vision Transformers that why it can work as well as CNNs. By using CKA as a tool to calculate the representational similarity, it finds many interesting properties of ViT and the final conclusions and results of this work are helpful for the community. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "1. Some of the properties proposed in this work have been discussed by some previous works [1, 2, 3]. For example, \"ViT having more uniform representations, with greater similarity between lower and higher layers\" and \"ViT incorporates more global\n36 information than ResNet at lower layers\" have been visualized by the T2T-ViT (Figure1 in T2T-ViT) [2]; and \"incorporating local information at lower layers remains vital, with large-scale pre-training data helping early attention layers learn to do this\", which is exactly what T2T module [2] aims to solve. This work fails to discuss these.\n2. This work provides many interesting properties of ViT but does not solve any problems proposed by its findings. \n\n[1]. Touvron, Hugo, et al. \"Going deeper with image transformers.\" arXiv preprint arXiv:2103.17239 (2021). \n[2]. Yuan, Li, et al. \"Tokens-to-token vit: Training vision transformers from scratch on imagenet.\" arXiv:2101.11986 (2021).\n[3]. Zhou, Daquan, et al. \"Deepvit: Towards deeper vision transformer.\" arXiv preprint arXiv:2103.11886 (2021).", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "1sP1CWS3nFP", "reviewer": "Reviewer_16Eb", "summary": "The paper studies the representation of features learned by convnet and transformers. It evaluates the similarity of representations within and between architectures using Centered Kernel Alignment. It studies models trained on datasets of different sizes.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n- The study is interesting and quite new\n- The paper is well written and easy to follow\n\nWeakness:\n- The paper gives insights on the representation of features in convnets and transformers. Nevertheless from a practical point of view it does not allow to understand why this kind of representation emerges and which kind of representation should be privileged in function of the task we wish to solve.\n\n- The results in classification are only with linear probes which is a limited setup\n\n- For the transformers there are only results with ViT architecture and for the convnet there are only results with BiT architecture. \nThe BiT architecture is not necessarily the most representative of the convnet. It would be interesting to have results with other transformers architecture and more classical convnet architecture to see how the results changes.\n\"For CNNs, we look at ResNet50x1 which also shows saturating performance when pretraining on JFT-300M, and also ResNet152x2, which in contrast shows large performance gains with increased pretraining dataset size.\" L416 ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes the authors adequately addressed the limitations and potential negative societal impact of their work", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: After the discussion phase the reviewers scores were borderline with one recommending rejection, one leaning towards reject and two leaning towards accept. All reviewers found the paper to be well written and addresses an important problem of understanding the differences and similarities between ViT and CNN architectures. However, the reviewers all said the work fell short on providing analysis that was both sufficiently thorough (i.e., do these findings extend to different CNN architectures?) and sufficiently insightful, lacking explanations either as to why the observations hold true and/or what actions to take as a result of these findings. Without this, the study is somewhat incomplete. After considering all private comments, reviews, and discussion, the AC does not recommend acceptance and instead encourages the authors to expand their study to verify its validity on other CNN architectures and to further hypothesize and test reasons behind the observations made.  
Decision: Accept (Poster)