Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper introduces a Transformer-based image recognition model that is fully built on the Transformer layers (multi-head self-attention + point-wise MLP) without any standard convolution layers. Basically, it splits an image into patches and takes as input the set of linear embeddings of the patches and their positions. For classification, a learnable class token is added to the input and a classification head (MLP) is attached to the class token output of the final Transformer layer. Extensive experiments of transfer learning show that when pretrained on a sufficiently large dataset (100~300M), the proposed Vision Transformer can outperform state-of-the-art convolutional networks with less training cost as well as less number of parameters.

Pros
1. Clearly motivated and well written 
- The background of the research, the motivation of Vision Transformer, and the related work are all clearly stated and summarized. 

2. The first fully-Transformer-based image recognition model 
- The simple yet effective Vision Transformer is introduced by adapting the original Transformer with minimal modification; patch embedding, class token, and class head. 

3. Extensive evaluation and good analysis 
- This paper demonstrates the power of the Vision Transformer model by extensive large-scale experiments, outperforming SOTA CNN models. Comparative evaluation with important baselines, ResNet and hybrid models, are well designed and conducted with different scales of datasets and models. The results are impressive and interesting. Additional discussions in Appendix are also useful in understanding the model. 

Cons
1. No significant technical novelty
-  The proposed model is incremental modifications of the original Transformer and its existing variants. 

2. Lack of further analysis of the inductive bias 
- The authors contrast the Transformer with CNN, stating “Transformers lack some inductive biases inherent to CNNs, such as translation equivariance and locality, and therefore do not generalize well when trained on insufficient amounts of data.” and “the convolutional inductive bias is useful for smaller datasets, but for larger ones, learning the relevant patterns is sufficient, even beneficial.” However, it is not clear which inductive bias of CNN prevents better generalization; is it translation equivariance, locality, static weights of kernels, or static size of kernels? I don’t find any clear answers from the paper. Since there exist different inductive biases in standard convolution, it may be the case that some of them help generalization and some others do not; worse generalization of CNN with a larger dataset may be affected by all those factors in a tangle. In this regard, further analysis of this issue would improve this work. E.g., in order to check the benefit of locality bias, ViT with local self-attention can also be compared, etc. 

3. Some misleading statements
- The authors repeatedly call the input of patch embeddings an “input sequence”, which I guess is intended to remind of the NLP origin. But, this may confuse readers. The Transformer used in this work does not process the input as a sequence, and actually the output is equivariant to any permutation of the patch (+position) embeddings. 
- “Transformers lack some inductive biases inherent to CNNs, such as translation equivariance and locality”. This needs to be clarified. As I noted above, since the Transformer in this work is permutation equivariant, it can also be seen as translation equivariant. As for “locality”, the Transformer also performs point-wise (patch-wise) processing, thus leveraging locality. 
- “Unlike prior works using self-attention in computer vision, we do not introduce any image-specific inductive biases into the architecture.” This statement in conclusion is an overclaim, conflicting with the statement at the end of Sec. 2.1 (resolution adjustment and patch extraction). 
- The title of this paper “An image is worth 16x16 words”, what does it mean? I don’t find an answer from the paper. 

review_title: Good paper with interesting empirical results, but missing some analysis

final_score: rating: 7: Good paper, accept
review_content: This paper explores the Transformer architecture for image recognition. The authors mainly follow the settings of BERT in NLP where the input is a sequence of words. To adopt Transformer for image recognition, the authors split the image into grid patches as inputs for the vision Transformer (ViT). Other settings like adding a special token at the beginning for classification and adding position embeddings are the same as those used in BERT. While Transformer is not new, the way it is applied on raw input images directly for image classification has not been done before. I also like the analogy with the hybrid model where the early layers in ResNet serve the role similar to the patch embedding projection. More importantly, the authors have conducted extensive experiments to validate the effectiveness of the proposed approach. Some of the visualizations are also insightful. The main conclusion is that ViT can outperform BiT(ResNet) when pre-trained on large-scale training data with supervised training. 

As the Transformer model has become more popular in the vision community, it seems natural for the authors to come up with ViT for image recognition. I think the main contribution of this paper is not to propose this idea, but rather to make it work. The fact that ViT can outperform BiT with ResNet only when it scales-up to a huge model trained on a huge dataset makes it impossible for most researchers with limited resources to conduct such experiments. Although the results may not be that encouraging in my opinion, I think the experiments conducted in this paper are valuable for the research community. 

My major concern is that the pre-training conducted in this paper is fully supervised. This is different to BERT that can use unlabeled text data or Vision-Language Pre-training where weakly-labeled image-text pairs can be leveraged. The authors briefly mentioned that the masked patch prediction can still improve ImageNet classification accuracy compared to training from scratch, but it has a gap with supervised pre-training. It would be great to explore self-supervised pre-training in the future as also pointed out by the authors. 

Another concern is about the comparison with the hybrid model. The main drawback of applying ViT on the full image is that it cannot scale to large input resolution due to memory constraint. A hybrid approach seems to be a good work-around to handle this case. While the authors have mentioned the hybrid model, very limited comparisons are conducted in the experiments. Only Figure 5 shows some results of hybrid models. In some cases, the hybrid model outperforms ViT. Why not add more data points for hybrid models at higher pre-training compute cost? Why not compare the numbers in Table 2 and Figure 2-4? 

In the appendix, the authors mention that the absolute numbers of BiT are lower than those reported by Kolesnikov et al. (2020), since they pre-train only for 7 epochs, not 30. Is it possible that there is more room to improve for BiT (ResNet) in Figure 5 if we increase the pre-training cost? At least the trend does not show saturating performance of BiT. 

Overall, I think the authors have done a great job in conducting all these experiments. Although the results are not that encouraging as these experiments require too many resources, I think the experiments can provide some value for future research. I would like the authors to comment on the comparison with BiT and the hybrid model in Figure 5. 
review_title: It is a good idea. More importantly, it is great to make the idea work. 

final_score: rating: 7: Good paper, accept
review_content: ## Summary
This paper studies to adapt transformer model for image classification task. The new model performs comparably well on various popular benchmarks, and will out-perform when pre-trained on large dataset. 

## Pros 
1. The technical solution is surprisingly simple, yet achieves strong performance. All it needs is to cut input images into patches, and reshape it as an input sequence. The transformer architecture is kept almost the same as in NLP tasks. The simplicity of the model makes it easy to generalize many vision tasks potentially.
2. The experimental section has clearly demonstrated the pros and cons of the proposed model. The authors not only show the strong performance but also stay upfront about the limits. 
3. The additional analysis, visualizations, and self-supervised pre-training sections are very informative. 
4. The writing is mostly clear and easy to follow.

## Cons
1. The arguments about "inductive biases" are confusing and self-contradictory. On one hand,  the introduction section says that CNN generalize better due to the inductive biases such as translation equivariance and locality. On the other hand, the rest of the paper claims that avoid inserting inductive biases into the transformer is an advantage.  In my understanding, these inductive biases have been shown to be beneficial for most vision tasks in CNN. Why not introducing some of them into transformer? Please clarify.
2. It's not clear to me how the few shot accuracy is computed. Is the the same evaluation as in self-supervised representation learning? What kind of regularization is applied to the linear regression, and how to learn the mapping? 

## Minor
1. It might be better to give reference when mentioning ````100B parameters in the first paragraph of introduction.
2. $z_0^L$ on page 2 bottom seems to be a typo. Should it be $z_L^0$?
3. In Sec.1 ```Hybrid Architecture, the statement "In this hybrid model, the patch embedding projection E (Eq. 1) is replaced by the early stages of a ResNet" is misleading. As in Eq.(1) ${\bf E}$ is applied on different input patches $x_p^i$, whereas early stages of ResNet are applied on the same input (image $x$). 

## Questions
1.  If I understand correctly, a model like ViT-L/16 means that it takes 16x16 fixed-sized patches as input. Does this mean that the images get resized to fixed resolution before cutting into patches? How big are those patches? Does the size matter?
2.  As shown in prior work (Unsupervised Visual Representation Learning by Context Prediction, Doersch etal., ICCV 2015), enforcing the patches to be discontinued and with some randomness is beneficial to self-supervised representation learning. Have the authors tried similar processing method? An ablation on different ways of generating patches (discontinued, overlapped, mixed-scale...) might be useful as some simple techniques might greatly improve the performance.
review_title: Transformers for image classification. Strong performance on when pre-trained on large dataset.

final_score: rating: 7: Good paper, accept
review_content: Following the transformer successes in NLP, this paper explores the performance of a vanilla Transformer (with few simple modifications) for the task of image classification.  It has been experimentally validated that this Transformer (ViT) can attain superior performance for this task compared to SOTA CNN architectures if it is pre-trained on large amounts of data and then applied to mid-sized or small image classification benchmarks. Moreover, it has been shown that ViT requires considerably fewer computational resources to train compared to computationally demanding CNN backbones such as ResNet. Clearly, the paper has the potential to re-ignite another wave of excitement for exploring this great learning model on different computer vision tasks. To this end, I believe this paper has enough merit to be accepted.

The paper delivers a strong message " transformers can be a more powerful, yet efficient,  compared to the SOTA CNN backbones for image recognition tasks if there is a large-enough dataset available " and the authors prove this claim by performing comprehensive experiments on several large-scale image recognition benchmarks.  This level of experimental verifications is only possible if huge computation re-sources are only available, which is not accessible for most research teams, esp in academia. Otherwise, a similar message (replacing CNN with transformers/attention) has been attempted to be verified in a few earlier works (as acknowledged in this paper as well). 

Also, it is hard to imagine that ViT can yet compete with the CNN backbones for vision tasks beyond the considered tasks because:

1- it is not clear how the simple ViT can be extended for the vision tasks which require pixel-level predictions, e.g. image segmentation, depth prediction etc, or 3D vision tasks while being still computationally tractable.

2-  As argued in the paper, transformers are data-hungry to perform as well as ResNet for a task and the availability of such large-scale annotations (e.g. 300M samples) beyond image labelling may not be feasible yet. 

3-  Considering transferring a pre-trained model,  there is no evidence that a pre-trained ViT (e.g. on a large amount of data) can still learn a reliable representation for the other basic vision tasks beyond image recognition such as image segmentation or object detection, and still carry a superior performance compared to ResNet when it is fine-tuned... The superiority of DETR (Carion et al 2020) on the detection task is not solely due to the use of a transformer but rather formulating object detection as a set prediction problem to avoid heuristics such as NMS.

Other comments:

I couldn't find any ablation study on different choices of 2D patches sizes versus training time and accuracy except partial experiments on 14,16 and 32. I am not sure what would be best patch size given any image, eg a panoramic image with high resolution when considering both accuracy and training computation compared to ResNet. 



review_title: Transformers cannot yet be a competitor for CNN in different computer vision tasks 

Meta-review: This paper has generated a lot of great discussion and it presents a very different way of doing image recognition at scale compared to current state of the art practices.  All reviewers rated this paper as an accept.
This work is interesting enough that in my view it really deservers further exposure and discussion and an oral presentation at ICLR would be a good way to achieve that.
Decision: Accept (Oral)