Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper provides a comprehensive analysis of visual features learned from vision transformers and CNNs. It applies central kernel alignment (CKA) to compare ViT and CNNs from several aspects mainly including a) local and global representations; b) roles of skip connections; and c) spatial localization.
main_review: Strengths:
1) ViT and its variants are demonstrating better performance than CNNs in many vision applications, thus understanding the differences between ViTs and CNNs is an important problem.
2) The paper provides thorough analysis and comparisons of ViT and CNN features based on models trained on both small and large datasets.

Weakness:
1) This is basically an analysis paper. While some of its findings are helpful for understanding the differences between the two architectures, the primary claims about the local/global representations and skip connections are not extremely surprising. The paper also doesn't suggest much about what makes ViT superior to CNNs or how to improve the performance of ViT further.  

2) I am not quite convinced by the analysis of spatial localization. Why is locality-preserving in the higher layers of ViT so relevant to object detection? How would this property be helpful for object detection if ViT is not learning contextual information? it would be better if the authors could add analysis on how this property would affect ViT in the task of object detection (either in a positive or negative way). 

3) Were all the models trained using the same dataset and augmentation in this paper?  It would be good to provide the classification accuracies of these models.  ResNet50 and ResNet152 are not the SOTA CNN models.  How would a better-performed model like EfficientNet affect the analysis?

4) In Fig. 1, both CNN models, especially ResNet50, indicate two blocks along the diagonal. Is there any explanation for this pattern?
 
5) Do different heads result in different mean attention distances? Some more analysis on this would be interesting.

6) It’s not clear to me how the effective receptive field for VIT is defined and computed. 


limitations_and_societal_impact: no broader impact section is included in the paper
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper deeply studies the working mechanism of vision transformers and explores the similarities and differences between vision transformers and CNNs.
main_review: - The originality of this paper is clear. Unlike most previous work that target designing novel network architectures or advanced attention mechanisms, this paper aims to deeply study what the vision transformers learn and help understand the differences between vision transformers and CNNs.

- Through the representation analysis methods, the authors well visualize what the transformer layers learn at different network depth and show how vision transformers learn local and global information.

- The presentation is good and the novelty is significant. I do think the explanations and conclusions can help in designing better vision transformers or hybrid networks.
limitations_and_societal_impact: - Despite the comprehensive analysis, there is a lack of providing insights that could help design better network architectures.

- The authors say that they provide ramifications for future applications in Section H but I cannot find exactly where the ramifications 
 are.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This work addresses some fundamental questions on Vision Transformers including 1. how are Vision Transformers solving vision tasks? 2. Do they act like convolutions, learning the same inductive biases from scratch? 3. And what is the role of scale in learning these representations? To answer these questions, they investigate the internal representation structure of ViTs and CNN by using representational similarity techniques like CKA (Centered Kernel Alignment). 
main_review: This paper aims to answer some fundamental questions on Vision Transformers that why it can work as well as CNNs. By using CKA as a tool to calculate the representational similarity, it finds many interesting properties of ViT and the final conclusions and results of this work are helpful for the community. 
limitations_and_societal_impact: 1. Some of the properties proposed in this work have been discussed by some previous works [1, 2, 3]. For example, "ViT having more uniform representations, with greater similarity between lower and higher layers" and "ViT incorporates more global
36 information than ResNet at lower layers" have been visualized by the T2T-ViT (Figure1 in T2T-ViT) [2]; and "incorporating local information at lower layers remains vital, with large-scale pre-training data helping early attention layers learn to do this", which is exactly what T2T module [2] aims to solve. This work fails to discuss these.
2. This work provides many interesting properties of ViT but does not solve any problems proposed by its findings. 

[1]. Touvron, Hugo, et al. "Going deeper with image transformers." arXiv preprint arXiv:2103.17239 (2021). 
[2]. Yuan, Li, et al. "Tokens-to-token vit: Training vision transformers from scratch on imagenet." arXiv:2101.11986 (2021).
[3]. Zhou, Daquan, et al. "Deepvit: Towards deeper vision transformer." arXiv preprint arXiv:2103.11886 (2021).
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: The paper studies the representation of features learned by convnet and transformers. It evaluates the similarity of representations within and between architectures using Centered Kernel Alignment. It studies models trained on datasets of different sizes.
main_review: Strength:
- The study is interesting and quite new
- The paper is well written and easy to follow

Weakness:
- The paper gives insights on the representation of features in convnets and transformers. Nevertheless from a practical point of view it does not allow to understand why this kind of representation emerges and which kind of representation should be privileged in function of the task we wish to solve.

- The results in classification are only with linear probes which is a limited setup

- For the transformers there are only results with ViT architecture and for the convnet there are only results with BiT architecture. 
The BiT architecture is not necessarily the most representative of the convnet. It would be interesting to have results with other transformers architecture and more classical convnet architecture to see how the results changes.
"For CNNs, we look at ResNet50x1 which also shows saturating performance when pretraining on JFT-300M, and also ResNet152x2, which in contrast shows large performance gains with increased pretraining dataset size." L416 
limitations_and_societal_impact: Yes the authors adequately addressed the limitations and potential negative societal impact of their work
review_title: null

Meta-review: After the discussion phase the reviewers scores were borderline with one recommending rejection, one leaning towards reject and two leaning towards accept. All reviewers found the paper to be well written and addresses an important problem of understanding the differences and similarities between ViT and CNN architectures. However, the reviewers all said the work fell short on providing analysis that was both sufficiently thorough (i.e., do these findings extend to different CNN architectures?) and sufficiently insightful, lacking explanations either as to why the observations hold true and/or what actions to take as a result of these findings. Without this, the study is somewhat incomplete. After considering all private comments, reviews, and discussion, the AC does not recommend acceptance and instead encourages the authors to expand their study to verify its validity on other CNN architectures and to further hypothesize and test reasons behind the observations made.  
Decision: Accept (Poster)