An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Authors: Alexey Dosovitskiy
Lucas Beyer
Alexander Kolesnikov
Dirk Weissenborn
Xiaohua Zhai
Thomas Unterthiner
Mostafa Dehghani
Matthias Minderer
Georg Heigold
Sylvain Gelly
Jakob Uszkoreit
Neil Houlsby
Venue: ICLR 2021 Oral (2021)
Keywords: computer vision
image recognition
self-attention
transformer
large-scale training

Abstract: While the Transformer architecture has become the de-facto standard for natural language processing tasks, its applications to computer vision remain limited. In vision, attention is either applied in conjunction with convolutional networks, or used to replace certain components of convolutional networks while keeping their overall structure in place. We show that this reliance on CNNs is not necessary and a pure transformer applied directly to sequences of image patches can perform very well on image classification tasks. When pre-trained on large amounts of data and transferred to multiple mid-sized or small image recognition benchmarks (ImageNet, CIFAR-100, VTAB, etc.), Vision Transformer (ViT) attains excellent results compared to state-of-the-art convolutional networks while requiring substantially fewer computational resources to train.

Published as a conference paper at ICLR 2021
AN IMAGE IS WORTH 16x16 WORDS: TRANSFORMERS FOR IMAGE RECOGNITION AT SCALE
Alexey Dosovitskiy Lucas Beyer", Alexander Kolesnikov", Dirk Weissenborn',
Xiaohua Zhai', Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer,
Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, Neil Houlsby*: eqqul technical contribution, 'equal advising
Google Research, Brain Team
(adosovitskly, nei neilhoulsby @gg.hooolsyyyyooogglllll Aogoglll......ooooooooooo com
ABSTRACT
While the Transformer architecture has become the de-facto standard for natural language processing tasks, its applications to computer vision remain limited. In vision, attention is either applied in conjunction with convolutional networks, or used to replace certain components of convolutional networks while keeping their overall structure place. We show that this reliance on CNNs is not necessary and a pure transformer applied directly to sequences of image patches can perform very well on image classification tasks. When pre-trained on large amounts data and transferred to multiple mid-sized small image recognition benchmarks (ImageNet, CIFAR-100, VTAB, etc.), Vision Transformer (ViT) attains excellent results compared to state-of- of-the-art convolutional networks while requiring sub- stantially fewer computational resources to train
INTRODUCTION
Self-attention-base based architectures, in particular Transformers (Vaswani et al. 2017). have become the model of choice in natural language processing (NLP). The dominant approach a large text corpus and then fine-tune on a smaller task-specific dataset (Devlin et al. 2019). Thanks to Transformers' computational efficiency and scalability, it has become possible to train models of unprecedented size, with over 100B parameters (Brown et al. [2020] Lepikhin et al. 2020). With the models and datasets growing, there still no sign of saturating performance.
pre-train on
In computer vision, however, convolutional architectures remain dominant (LeCun et al. 1989 Krizhevsky et al. 2012, He et al. 2016). Inspired by NLP successes, multiple works try combining CNN-like architectures with self aelf-attention (Wang et al. [018)] Carion et al. (020),, some replacing the convolutions entirely (Ramachandran al. [219]] Wang et al.) (020a)) The latter models, while theoretically efficient, have not yet been scaled effectively on modern hardware accelerators due to the use of specialized attention patterns. Therefore, large-seale image recognition, classic ResNet-
like architectures are still state of the art (Mahajan et al. 2018 2020).
Xie et al. 2020 Kolesnikov et al.
Inspired by the Transformer scaling successes in NLP, we experiment with applying standard Transformer directly to images, with the fewest possible modifications. To do so, we split an image into patches and provide the sequence of linear embeddings of these patches as input to Trans- former. Image patches are treated the same way as tokens (words) in an NLP application. We train the model image classification in supervised fashion.
When trained on mid-sized datasets such as ImageNet without strong regularization, these mod- els yield modest accuracies of few percentage points below ResNets of comparable size. This seemingly discouraging outcome may be expected: Transformers lack some of the inductive biases
FFne-tuning code and pre-trained models are available
ht.pa://github.com
google e- www.ereeeeeevveeeeeee http://ssssoottttaafaaaoeoao

---

Published as a conference paper at ICLR 2021
inherent to CNNs, such translation equivariance and locality, and therefore do not generalize well when trained on insufficient amounts of data.
However, the picture changes if the models are trained larger datasets (14M-300M images). We find that large scale training trumps inductive bias. Our Vision Transformer (ViT) attains excellent results when pre-trained sufficient scale and transferred tasks with fewer datapoints. When pre-trained on the public ImageNet-21k dataset or the in-house JFT-300M dataset, ViT approaches or beats state of the on multiple image recognition benchmarks. In particular, the best model reaches the accuracy of 88.55% on ImageNet, 90.72% on ImageNet-Real, 94.55% on CIFAR- 100, and 77. 77.63% on the VTAB suite of 19 tasks.
RELATED WORK
Transformers were proposed by Vaswani et al. (2017) for machine translation, and have since be- come the state of the art method in many NLP tasks. Large Transformer-based models are often pre-trained on large corpora and then fine-tuned for the task at hand: BERT (Devlin et al. 2019) uses denoising self-supervised pre-training task, while the GPT line of work uses language mod-
cling as its pre-training task (Radford et al. 2018 2019; Brown et al. 2020).
Naive application of self attenttnnion to images would require that each pixel attends to every other pixel. With quadratic cost in the number of pixels, this does not scale to realistic input sizes. Thus, to apply Transformers in the context of image processing, several approximations have been tried in the past. Parmar et al. (2018) applied the self-attention only in local neighborhoods for each query pixel instead of globally. Such local multi-head dot-product self attention blocks can completely replace convolutions (Hu et al. 2019: Ramachandran et al.) 2019 Zhao et al.) (020). In different line of work, Sparse Transformers (Child et alll|2201) employ scalable approximations to global self- attention in order to be applicable to images. An alternative way to scale attention is to apply it in blocks of varying sizes (Weissenborn et al. al. 2019]. in the extreme case only along individual axes (Ho) et al., 2019; Wang et al.) (2020a). Many of these specialized attention architectures demonstrate promising results on computer vision tasks, but require complex engineering to be implemented efficiently on hardware accelerators.
Most related to ours is the model Cordonner et al. (2020), which extracts patches of size 2 x from the input image and applies full self-attention on top. This model is very similar to VIT, but our work goes further to demonstrate that large scale pre-training makes vanilla transformers competitive with (or even better than) state-of-the-arr CNNs. Moreover, [ordonnier al. (2020) use small patch size of 2 x 2 pixels, which makes the model applicable only to small-resolution images, while we handle medium-resolution images well.
There has also been a lot of interest in combining convolutional neural networks (CNNs) with forms
of self-attention, e.g. by augmenting feature maps for image classification (Bello al. (019 or by
further processing the output of a CNN using self-attention, e.g. for object detection (Hu et al. 2018 Carion et al. 2020). video processing (Wang al. 2018 Sun et 2019), image classification (Wu
et al. 2020). unsupervised object discovery (Locatello et al. 2020), or unified text-vision tasks CChn
ct al.
2020c
Lu al. 2019 Li et 2019).
Another recent related model is image GPT (iGPT) (Chen et al. 2020a), which applies Transformers to image pixels after reducing image resolution and color space. The model is trained in an unsu- pervised fashion as generative model, and the resulting representation can then be fine-tuned or probed linearly for classification performance, achieving a maximal accuracy of 72% on ImageNet.
Our work adds to the increasing collection of papers that explore image recognition at larger scales than the standard ImageNet dataset. The use of additional data sources allows to achieve state-of- the-art results standard benchmarks (Mahajan et al. 2018 Touvron et al. 2019 Xie et al.) 2020). Moreover, Sun et al. (2017) study how CNN performance scales with dataset size, and Kolesnikov et al. (2020); Djolonga al. (2020) perform an empirical exploration of CNN transfer learning from large scale datasets such as ImageNet-21k and JFT-300M. We focus on these two latter datasets as well, but train Transformers instead of ResNet-based models used in prior works.
2

---

Published as a conference paper at ICLR 2021
Vision Transformer (ViT)
Transformer Encoder
Class Bird Ball Car
Lx
MLP Head
MLP
Norm
Transformner Encoder
Position Embedding
+
00
Multi- Head Attention
"Der Inarsatie [cleas] crlboodnng
Projection Patches
Emmeeded Patches
Figure 1: Model overview. We split an image into fixed-size patches, linearly embed each of them, add position embeddings, and feed the resulting sequence of vectors to standard Transformer encoder. In order to perform classification, we use the standard approach of adding extra learnable "classification token" to the sequence. The illustration of the Transformer encoder was inspired by Vaswani et al. (2017).
3 METHOD
In model design we follow the original Transformer (Vaswani et al. 2017) as closely possible. advantage of this intentionally simple setup that scalable Nin'Transfoom architectures and their efficient implementations can be used almost out of the box.
3.1 VISION TRANSFORMER (VIT)
An overview of the model is depicted in Figure 1 The standard Transformer receives input 1D sequence token embeddings. To handle 2D images, we reshape the image x E H into sequence of flattened 2D patches Xp E RNxx(P where (H, W) is the resolution of the original image. C is the number of channels. (P, P) is the resolution of each image patch, and N = HW/ pa the resulting number of patches, which also serves the effective input sequence length for the Transformer. The Transformer uses constant latent vector size D through all of its layers, so we flatten the patches and map to dimensions with trainable linear projection (Eq. We refer to the output of this projection as the patch embeddings.
Similar to BERT's [class] token, we prepend a learnable embedding the sequence embed- ded patches (zf) Xctas), whose state at the output of the Transformer encoder (z%) serves as the image representation y (Eq. 4). Both during pre-training and fine-tuning, a classification head is at- tached to 24,- The classification head is implemented by a MLP with one hidden layer at pre-training time and by a single linear layer at fine-tuning time.
added the patch embeddings to retain positional information. We use standard learnable ID position embeddings, since we have not observed significant performance gains from using more advanced 2D-aware position embeddings (Appendix D.3). The resulting sequence of embedding vectors serves as input to the encoder.
Position embeddings
The Transformer encoder (Vaswani al. 2017) consists of altermating layers of multiheaded self- attention (MSA, see Appendix A) and MLP blocks (Eq. 2 3. Layernorm (LN) applied before every block, and residual connections after every block (Wang al. 2019 Baevski & Auli 2019
3

---

Published as a conference paper at ICLR 2021
The MLP contains two layers with GELU non-linearity.
20 Ielaass x E,E; x ...: X E] Eposs
E E R C)xD
Epos E
z'r (ISA(LN(Z--I)) 18-1,
l t 1=I...1
(2)
Z/ MLL(LL(ZZ) 1)) +2'
(=1...
(3)
y LN(2))
(4)
Inductive bias. We note that Vision Transformer has much less image-specifie inductive bias than CNNs. In CNNs, locality, two-dimensional neighborhood structure, and translation equivariance baked into each layer throughout the whole model. ViT, only MIP layers are local and transla- tionally equivariant, while the self-attention layers are global. The two-dimensional neighborhood structure is used very sparingly: in the beginning of the model by cutting the image into patches and fine-tuning tine-tuning time for adjusting the position embeddings for images of different resolution (as de- scribed below). Other than that, the position embeddings at initialization time carry no information about the 2D positions of the patches and all spatial relations between the patches have to be learned from scratch.
Hybrid Architecture. As an alternative to raw image patches, the input sequence can be formed from feature maps of CNN (LeCun et al. 1989). In this hybrid model, the patch embedding projection E (Eq. T is applied to patches extracted from a CNN feature map. As a special case, the patches can have spatial size 1x1, which means that the input sequence is obtained by simply flattening the spatial dimensions of the feature map and projecting to the Transformer dimension. The classification input embedding and position embeddings added as described above.
3.2 FINE-TUNING AND HIGHER RESOLUTION
Typically, we pre-train ViT on large datasets, and fine-tune to (smaller) downstream tasks. For this, we remove the pre-trained prediction head and attach a aero-initialized D K feedforward layer, where K is the number of downstream classes. It is often beneficial to fine-tune at higher resolution than pre-training [Touvron et al.) 2019 Kolesnikov et al., 2020). When feeding images of higher resolution, we keep the patch size the same, which results in larger effective sequence length. The Vision Transformer can handle arbitrary sequence lengths (up to memory constraints), however, the pre-trained position embeddings may longer be meaningful. We therefore perform 2D interpolation of the pre-trained position embeddings, according to their location in the original image. Note that this resolution adjustment and patch extraction are the only points at which an inductive bias about the 2D structure of the images is manually injected into the Vision Transformer.
4 EXPERIMENTS
We evaluate the representation learning capabilities of ResNet, Vision Transformer (ViT), and the hybrid. To understand the data requirements of each model, we pre- on datasets of varying size and evaluate many benchmark tasks. When considering the computational cost of pre-training the model, ViT' performs very favourably, attaining state of the art most recognition benchmarks a lower pre-training cost. Lastly, we perform a small experiment using self-supervision, and show that supersupe ViT holds promise for the future.
4.1 SETUP
Datasets. To explore model scalability, we use the ILSVRC-2012 ImageNet dataset with Ik classes and 1.3M images (we refer it as ImageNet in what follows), its superset ImageNet-21k with 21k classes and 14M images (Deng et al.) 2009), and JFT (Sun et al. (017)) with 18k classes and 303M high-resolution images. We de-duplicate the pre-training datasets w.r.t. the test sets of the downstream tasks following Kolesnikov et al.] (2020). We transfer the models trained on these dataset to several benchmark tasks: ImageNet on the original validation labels and the cleaned-up ReaL labels (Beyer et al.) 22200), CIFAR- 10/100000000011 (Krizhevsky) 2009), Oxford-III CiIT Pets (Parkhi et al.
2012). and Oxford Flowers- 102
NNlssbcc & Zisserman 2008). For these datasets, pre- pre-psrcessing
Tollows[Kolennike Kolesnikov al. (2020).
4

---

Published as a conference paper at ICLR 2021
Model
Layers
Hidden size
MLP size
Heads
Params
ViT-Base
12
768
3072
12
86M
ViT-Large
24
1024
4096
16
307M
ViT-Huge
32
1280
5120
16
632M
Table 1: Details of Vision Transformer model variants.
We also evaluate the 19-task VTAB classification suite Zhai et al., 2019b). VTAB evaluates low-data transfer to diverse tasks, using I 000 training examples per task. The tasks are divided into three groups: Natural tasks like the above, Pets, CIFAR, etc. Specialized medical and satellite imagery, and Structured tasks that require geometric understanding like localization.
Model Variants. We base VIT configurations those used for BERT (Devlin et al. 2019), as summarized in Table The "Base" and "Large" models are directly adopted from BERT and we add the larger "Huge" model. In what follows we use brief notation to indicate the model size and the input patch size: for instance. ViT-L/16 means the "Large" variant with 16 x 16 input patch size. Note that the Transformer's s sequence length is inversely proportional to the square of the patch size, thus models with smaller patch size are computationally more expensive.
For the baseline CNNs, we use ResNet (He et al. 2016), but replace the Batch Normalization lay- ers (loffe& & Szegedy (015)) with Group Normalization (Wu & He| 2018), and used standardized convolutions (Qiao al. 2019). These modifications improve transfer (Kolesnikov et al. 2020), and we denote the modified model "ResNet (BiT)". For the hybrids, we feed the intermediate fea- ture maps into ViT with patch size of one "pixel". To experiment with different sequence lengths, either (i) take the output of stage 4 of regular ResNet50 (īi) remove stage 4, place the same number of layers in stage 3 (kceping the total number of layers), and take the output of this extended stage 3. Option (ii) results in a 4x longer sequence length, and a more expensive VIT model.
Training & Fine-tuning. We train all models, including ResNets, using Adam (Kingma & Ba 2013) with 31 0.9, 32 = 0.999, batch size of 4096 and apply high weight decay of 0.1, which we found to be useful for transfer of all models (Appendix practices, Adam works slightly better than SGD for ResNets shows that, in contrast to common setting). We use linear learning rate warmup and decay, see Appendix B. for details. For fine-tuning we use SGD with momentum, batch size 512, for all models, see Appendix B. For ImageNet results in Table we fine-tuned higher resolution: 512 for ViT-L/16 and 518 for ViT-H/14, and also used Polyak & Juditsky (1992)
averaging with a factor of 0.9999 (Ramachandran al., 2019 Wang et al. (020b).
Metrics. We report results on downstream datasets either through few-shot fine-tuning accuracy. Fine-tuning accuracies capture the performance of each model after fine-tuning it on the respective dataset. Few-shot accuracies are obtained by solving regularized least-squares regression problem that maps the (frozen) representation of subset of training images to {-1, ] target vectors. This formulation allows to recover the exact solution in closed form. Though we mainly focus on fine-tuning performance, we sometimes use linear few-shot accuracies for fast on-the-fly evaluation where fine-tuning would be too costly.
4.2 COMPARISON TO STATE OF THE ART
We first compare our largest models ViT-H/14 and ViT-L/16 state-of-the-art CNNs from the literature. The first comparison point is Big Transfer (BiT) (Kolesnikov et al. 2020). which performs supervised transfer learning with large ResNets. The second is Noisy Student (Xie al. 2020), which is large EfficientNet trained using semi-supervised learning on ImageNet and JFT- 300M with the labels removed. Currently, Noisy Student the state of the art on ImageNet and BiT-L on the other datasets reported here. All models were trained on TPUv3 hardware, and report the number of TPUv3-core-days taken to pre-train each of them, that is, the number of TPU v3 cores (2 per chip) used for training multiplied by the training time in days.
Table Ssows the results. The smaller ViT-L/16 model pre-trained on JFT-300M outperforms BiT-L (which is pre-trained on the same dataset) all tasks, while requiring substantially less computa- tional resources to train. The larger model, ViT-H/14, further improves the performance, especially on the more challenging datasets ImageNet, CIFAR-100, and the VTAB suite. Interestingly, this
5

---

Published as a conference paper at ICLR 2021
Ours-JFT (ViT-I1/14)
Ours-JFT (ViT-L/16)
Ours-121k (VIT-L/16)
BiT-L (ResNec152x4)
Noisy Student (EfficientNet-L22)
ImageNet
58.55 0.04
87.76 a
85.30
87.54
88.4/88.5"
ImageNet ReaL
90.72 0.05
90.54 +: 0.03
$8.6240.00
90.54
90.55
CIFAR-10
99.50 0.06
99. = 0.03
99.15
99.37 = 0.06
CIFAR-100
94.55 0.04
93.90 +: 0.05
93.25
93.51 0.08
Oxford-IIIT Pets
97.56 0.03
97.32
94.67
96.62 = 99.63 = 0.03
Oxford Flowers-
99.68 +: 0.02
09.74. 0.00
99.61 0.00
VTAB (19 tasks)
77.63 0.23
76.28 + 0.46
72.72
76.29 = 1.70
TPUv3-core-days
2.5k
0.68k
0.23k
9.9k
12.3k
Comparison with state of the art on popular image classification benchmarks. We re- port mean and standard deviation of the accuracies, averaged over three fine-tuning runs. Vision Transformer models pre-trained on the JFT-300M dataset outperform ResNet-based baselines on all datasets, while taking substantially less computational resources to pre-train. ViT pre-trained on the smaller public ImageNet-21 kk dataset performs well too. "Slightly improved 88.5% result reported
Table 2:
in Touvron al. 2020).
VIT-H/14
BiT-L (R152x4)
VIVI-EX-I00S (R50x3)
S4L (R50x1)
Accuracy [%]
88
80
60
70
65
70
80
50 Structured (8 tasks)
VTAB (19 tasks)
Natural tasks)
Specialized (4 tasks)
Figure 2: Breakdown of VTAB performance in Natural, Specialized, and Structured task groups.
model still took substantially less compute pre-train than prior state of the art. However, we note that pre-training efficiency may be affected not only by the architecture choice, but also other pa- rameters, such training schedule, optimizer, weight decay, etc. We provide controlled study of performance vs. compute for different architectures in Section 4.4 Finally, the ViT-L/16 model pre-trained on the public ImageNet-2lk dataset performs well most datasets too, while taking fewer resources to pre-train: could be trained using a standard cloud TPUv3 with 8 cores in ap- proximately 30 days.
Figure 2 decomposes the VTAB tasks into their respective groups, and compares to previous SOTA methods this benchmark: BiT, VIVI- ResNet co-trained on ImageNet and Youtube (Tschannen et al. 2020), and S4L. - supervised plus semi-supervised learning on ImageNet (Zhai et al. (01 ViT- outperforms BiT-R152x4 and other methods, on the Natural and Structured tasks. On the Specialized the performance of the top two models is similar.
4.3 PRE-TRAINING DATA REQUIREMENTS
The Vision Transformer performs well when pre-trained on large JFT-300M dataset. With fewer inductive biases for vision than ResNets, how crucial is the dataset size? We perform two series of experiments.
First, we pre-train ViT models on datasets of increasing size: ImageNet, ImageNet-2Ik,, and JFT- 300M. To boost the performance on the smaller datasets, we optimize three basic regularization parameters weight decay, dropout, and label smoothing. Figure shows the results after fine- tuning to ImageNet (results on other datasets are shown in Table smallest dataset, ImageNet, ViT-Large models underperform compared ViT-Base models, despite (moderate) regularization. With ImageNet-21k pre-training, their performances are similar. Only with JFT-300M, do we see the full benefit of larger models. Figure also shows the performance
When pre-trained
the
Note that the ImageNet pre-trained models are also fine-tuned, but again ImageNet. This is because the resolution increase during fine-tuning improves the performance.

---

Published as a conference paper at ICLR 2021
Linear S-shot ImaggN et Top1 (%)
70
ImageNet Accurscy
85
60
80-
50-
75
BIT
VIT-L/32
VIT-B/32
VIT-L/16
VIT-L/16 - ViT-B/33
ReiNetXOX (BiT)
VII-B/16
VIT-H/I4
VIT-L/32 VIT--N32
Res (BiT)
70
30
ImageNet
ImageNet-21k
JFT-300
10
30
100 M
300
Pre-training dataset
Number JFT pre pre-training samples
Figure 3: Transfer to ImageNet. While large VIT models perform worse than BiT ResNets (shaded area) when pre-trained on small datasets, they shine when pre-trained on larger datasets. Similarly, larger ViT variants overtake smaller ones as the dataset grows.
Figure 4: Linear few-shot evaluation on Ima- geNet versus pre-training size. ResNets per- form better with smaller pre-training datasets but plateau sooner than ViT, which performs better with larger pre-training, ViT-b is ViT-B with all hidden dimensions halved.
Average-5
ImageNet
Trunsfer fer accuracy (%6
85
80
Transfomerr (VTT)
Transformer (VTT)
90
ResNet (BIT)
ResNet (BIT) Hybrid
Hybrid
75
10a
10
Total pre-training compute [exaFLOPPS]
Figure 5: Performance versus cost for different architectures: Vision Transformers, ResNets, and hybrids. Vision Transformers generally outperform ResNets with the same computational budget. Hybrids improve upon pure Transformers for smaller model sizes, but the gap vanishes for larger models.
region spanned by BiT models of different sizes. The BiT CNNs outperform ViT ImageNet, but with the larger datasets, ViT overtakes.
Second, we train our models random subsets of 9M, 30M, and 90M well the full JFT- 300M dataset. We do not perform additional regularization on the smaller subsets and use the same hyper-parameters for all settings. This way, assess the intrinsic model properties, and the effect of regularization. We do, however, use early-stopping, and report the best validation accuracy achieved during training. To save compute, we report few-shot linear accuracy instead of full fine- tuning accuracy. comparable computational cost on smaller datasets. For example, ViT-B/32 is slightly faster than ResNet50; it performs much worse on the 9M subset. but better 90M+ subsets. The same is true for ResNet152x2 and ViT-L/16. This result reinforces the intuition that the convolutional inductive bias is useful for smaller datasets, but for larger ones, learning the relevant patterns directly from data is sufficient, even beneficial.
the results. Vision Transformers overfit more than ResNets with
Overall, the few-shot results ImageNet (Figure 4) well the low-data results on VTAB (Table 2 seem promising for very low-data transfer. Further analysis of few-shot properties of ViT is exciting direction of future work.

---

Published as a conference paper at ICLR 2021
4.4 SCALING STUDY
We perform controlled scaling study of different models by evaluating transfer performance from JFT-300M. In this setting data size does not bottleneck the models' performances, and we assess performance versus pre-training cost of each model. The model set includes: 7 ResNets, R50x1, R50x2 R101x1, R152x1, R152x2, pre-trained for 7 epochs, plus R152x2 and R200x3 pre-trained for 14 epochs; 6 Vision Transformers, VIT-B/32, B/16, L/32, L/16, pre-trained for 7 epochs, plus L/16 and H/14 pre-trained for 14 epochs; and 5 hybrids, R50+ViT-B/32, B/16, L/32, L/16 pre- trained for 7 epochs, plus R50+ViT-L/16 pre- me-ttrinee for 14 epochs (for hybrids, the number the end of the model name stands not for the patch size, but for the total dowsampling ratio the ResNet backbone).
Figure 5 contains the transfer performance versus total pre-training compute (see Appendix D.4 for details computational costs). Detailed results per model provided in Table 6 in the Ap- pendix. A few patterns can be observed. First, Vision Transformers dominate ResNets on the performance/compute trade-off. ViT uses approximately performance (average over 5 datasets). Second, hybrids slightly outperform ViT at small compu- tational budgets, but the difference vanishes for larger models. This result is somewhat surprising, since one might expect convolutional local feature processing to assist ViT at any size. Third, Vision Transformers appear not to saturate within the range tried, motivating future scaling efforts.
4x less compute to attain the same
4.5 INSPECTING VISION PRANSFORMER
To begin to understand how the Vision Transformer processes im- age data, we analyze its internal representations. The first layer of the Vision Transformer linearly projects the flattened patches into a lower-dimensional space (Eq. [Tp. F (left) shows the top prin- cipal components of the the learned embedding filters. The com- ponents resemble plausible basis functions for a low-dimensional representation of the fine structure within each patch.
Input Attention
After the projection, a learned position embedding added to the patch representations. Figure 7 (center) shows that the model learns to encode distance within the image in the similarity of position em- beddings, i.e. closer patches tend to have more similar position em- beddings. Further, the row-column structure appears; patches in the same row/column have similar embeddings. Finally, a sinusoidal structure is sometimes apparent for larger grids (Appendix D). That the position embeddings learn represent 2D image topology ex- plains why hand-crafted 2D- aware embedding variants do not yield improvements (Appendix D.3).
Self-attention allows ViT to integrate information across the entire image even in the lowest layers. We investigate to what degree the network makes use of this capability. Specifically, we compute the average distance in image space across which information is integrated, based on the attention weights (Figure 7 right). This "attention distance" is analogous to receptive field size CNNs. We find that some heads attend to most of the image already in the lowest layers, showing that the ability to integrate information globally is indeed used by the model. Other attention heads have consistently small attention distances in the low layers. This highly localized attention is less pronounced in hybrid models that apply a ResNet before the Transformer (Figure 7 right), suggesting that it may serve similar function as early convolutional layers in CNNs. Further, the attention distance increases with network depth. Globally, we find that the model attends to image regions that are semantically relevant for classification (Figure 6
Figure 6: Representative amples of attention from the output token the input space. See Appendix D.6 for details.
4.6 SELF-SUPERVISION
Transformers show impressive performance on NLP tasks. However, much of their success stems not only from their excellent scalability but also from large scale self-supervised pre-training (Devlin)
8

---

Published as a conference paper at ICLR 2021
RGB embedding filters Ifirst 28 principal components)
Position em beddi ng simi larity
VIT-L/16
Maunn attertion distanc (okkesi
120
Cosine simlarty
Head
Head
10 15
patch column
Network deepth (layeri
Figure 7: Left: Filters of the initial linear embedding of RGB values of ViT-L/32. Center: Sim-
ilarity of position embeddings of ViT-L/32. Tiles show the cosine similarity between the position embedding of the patch with the indicated row and column and the position embeddings of all other patches. Right: Size of attended area by head and network depth. Each dot shows the mean attention distance across images for one of 16 heads at one layer. See Appendix D.6 for details.
al. 2019 Radford al. 2018). We also perform preliminary exploration masked patch prediction for self suf-supervisson, mimicking the masked language modeling task used in BERT. With self-supervised pre-training. our smaller ViT-B/16 model achieves 79.9% accuracy on ImageNet, significant improvement of 2% to training from scratch, but still 4% behind supervised pre-training. Appendix B. 1.1.2 contains further details. We leave exploration of contrastive pre-training (Chen)
et al. 2020b He et al. 2020 Bachman et al. 2019 Hénaff et al. 2020 to future work.
5 CONCLUSION
We have explored the direct application of Transformers to image recognition. Unlike prior works using self-altention in computer vision, do not introduce image-specific inductive biases into the architecture apart from the initial patch extraction step. Instead, we interpret an image as sequence patches and process it by standard Transformer encoder as used in NLP. This simple, yet scalable, strategy works surprisingly well when coupled with pre-training on large datasets. Thus, Vision Transformer matches or exceeds the state of the art on many image classification datasets, whilst being relatively cheap to pre-train.
While these initial results are encouraging, many challenges remain. One to apply VIT other computer vision tasks, such detection and segmentation. Our results, coupled with those in Carion jet al. 2020), indicate the promise of this approach. Another challenge to continue exploring self- supervised pre-training methods. Our initial experiments show improvement from self sulf-supe pre-training, but there is still large gap between self-supervised and large-scale supervised pre- training. Finally, further scaling of VIT would likely lead to improved performance.
ACKNOWLEDGEMENT
The work was performed in Berlin, Zürich, and Amsterdam. We thank many colleagues at Google for their help, in particular Andreas Steiner for crucial help with the infrastructure and the open- source release of the code; Joan Puigcerver and Maxim Neumann for help with the large-scale training infrastructure; Dmitry Lepikhin, Aravindh Mahendran, Daniel Keysers, Mario Lučić, Noam Shazeer, and Colin Raffel for useful discussions.
REFERENCES
Samira Abnar and Willem Zuidema. Quantifying attention flow in transformers. In ACL, 2020.
Philip Bachman, Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. In NewrlPS, 2019.

---

Published as a conference paper at ICLR 2021
Alexei Baevski and Michael Auli. Adaptive input representations for neural language modeling, In ICLR, 2019.
1. Bello, B. Zoph, Q. Le, A. Vaswani, and J. Shlens. Attention augmented convolutional networks. In ICCV, 2019.
Lucas Beyer, Olivier J. Hénaff, Alexander Kolesnikov, Xiaohua Zhai, and Aäron van den Oord. Are we done with imagenet? arXiv, 2020.
Tom В Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv, 2020.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers, In ECCV, 2020.
Mark Chen, Alec Radford, Rewon Child, Jeff Wu, and Heewoo Jun. Generative pretraining from pixels. In ICML, 2020a.
Ting Chen, Simon Korbllihh, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. In ICML, 2020b.
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. UNITER: UNiversal Image-TExx Representation Learning. In ECCV, 2020c.
Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv, 2019.
Jcan-Baptiste Cordonnier, Andreas Loukas, and Martin Jaggi. On the relationship between self- attention and convolutional layers. ICLR, 2020.
J. Deng. W. Dong, R. Socher, L. Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hicrarchical image database. In CVPR, 2009.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep
bidirectional transformers for language understanding. In NAACI, 2019.
Josip Djolonga, Jessica Yung, Michael Tschannen, Rob Romijnders, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Matthias Minderer, Alexander D' Amour, Dan Moldovan, Sylvan Gelly, Neil Houlsby, Xiaohua Zhai, and Mario Lucie. On robustness and transferability of convo- lutional neural networks. arXiv, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
Jonathan Ho, Nal Kalchbrenner, Dirk Weissenborn, and Tim Salimans. Axial attention in multidi- mensional transformers. arXiv, 2019.
Han Hu, Jiayuan Gu. Zheng Zhang, Jifeng Dai, and Yichen Wei. Relation networks for object detection. In CVPR, 2018.
Han Hu, Zheng Zhang, Zhenda Xie, and Stephen Lin. Local relation networks for image recognition. In ICCV, 2019.
Zilong Huang, Xinggang Wang, Yunchao Wei, Lichao Huang, Humphrey Shi, Wenyu Liu, and Thomas S. Huang. Cenet: Criss-cross attention for semantic segmentation. In ICCV, 2020.
Olivier J. Hénaff, Aravind Srinivas, Jeffrey De Fauw, Ali Razavi, Carl Doersch, S. M. Ali Eslami, and Aaron van den Oord. Data-efficient image recognition with contrastive predictive coding. In ICML., 2020.
10

---

Published as a conference paper at ICLR 2021
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. 2015.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big transfer (BiT): General visual representation learning. ECCV, 2020.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
Alex Krizhevsky, llya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convo- lutional neural networks. In NIPS, 2012.
Y. LeCun, B. Boser, J. Denker, D. Henderson, R. Howard, W. Hubbard, and L. Jackel. Backpropa- gation applied to handwritten zip code recognition. Neural Computation, 1:541-551, 1989.
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv, 2020.
Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. Simple and Performant Baseline for Vision and Language. Arxiv, 2019.
VisualBERT: A
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot atten- tion. arxiv, 2020.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. ViLBERT: Pretraining Task-Agnostic Visi- olinguistic Representations for -isonnaand-Language Tasks. In NeurIPS. 2019.
Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens van der Maaten. Exploring the limits of weakly supervised pretraining. In ECCV, 2018.
M. Nilsback and A. Zisserman. Automated flower classification over large number of classes. In ICVGIP, 2008.
Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In CVPR, 2012.
Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In JCML., 2018.
B. T. Polyak and A. B. Juditsky. Acceleration of stochastic approximation by averaging. SIAM Journal Control and Optimization, 30(4):838-855 855, 1992. doi: 10.1137/0330446.
URL
https://das /oi:://doo..ee 2tt..oogg/0.1117//3333333333
Siyuan Qiao, Huiyu Wang, Chenxi Liu, Wei Shen, and Alan Yuille. Weight standardization. arXiv
preprint arXiv: 10520, 2019.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under- standing with unsupervised learning. Technical Report, 2018.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. Technical Report, 2019.
Prajit Ramachandran, Niki Parmar, Ashish Vaswani, Irwan Bello, Anselm Levskaya, and Jon Shlens. Stand-alone self-attention in vision models. In NewrIPS, 2019.
Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable ef- fectiveness of data deep learning era. In ICCV, 2017.
Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. Videobert: A joint model for video and language representation learning. In ICCV, 2019.
11

---

Published as a conference paper at ICLR 2021
Hugo Touvron, Andrea Vedaldi, Matthijs Douze, and Herve Jegou. Fixing the train-test resolution discrepancy. In NeurlPS. 2019.
Hugo Touvron, Andrea Vedaldi, Matthijs Douze, and Herve Jegou. Fixing the train-test resolution discrepancy: Fixefficientnet. arXiv preprint arXiv:2003. 082377, 2020.
Michael Tschannen, Josip Djolonga, Marvin Ritter, Aravindh Mahendran, Neil Houlsby, Sylvain Gelly, and Mario Lucic. Self-supervised learning of video-induced visual invariances. In Pro- ceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention all you need. NIPS, 2017.
Huiyu Wang, Yukun Zhu, Bradley Green, Hartwig Adam, Alan Yuille, and Liang-Chiel Chen. Axial-dceplab: Stand-alone axial-attention for panoptic segmentation. In ECCV, 2020a.
Huiyu Wang, Yukun Zhu, Bradley Green, Hartwig Adam, Alan Yuille, and Liang-Chieh Chen. Axial-deeplab: Stand-alone axial-attention for panoptic segmentation. arXiv preprint arXiv:2003.07853, 2020b.
Qiang Wang. Bei Li, Tong Xiao, Jingbo Zhu, Changliang Li, Derek F. Wong, and Lidia S. Chao. Learning deep transformer models for machine translation. In ACL, 2019.
Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In CVPR, 2018.
Dirk Weissenborn, Oscar Täckström, and Jakob Uszkoreit. Scaling autoregressive video models. In ICLR, 2019.
Bichen Wu, Chenfeng Xu, Xiaoliang Dai, Alvin Wan, Peizhao Zhang, Masayoshi Tomizuka, Kurt Keutzer, and Peter Vajda. Visual transformers: Token-based image representation and processing for computer vision. arxiv, 2020.
Yuxin Wu and Kaiming He. Group normalization. In ECCV, 2018.
Qizhe Xie, Minh-Thang Thang Luong, Eduard Hovy, and Quoc V. Le. Self-training with noisy student improves imagenet classification. In CVPR, 2020.
Xiaohua Zhai, Avital Oliver, Alexander Kolesnikov, and Lucas Beyer. S4L: Self-Supervised Semi- Supervised Learning. In ICCV, 2019a.
Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, et al. A large-scale study of representation learning with the visual task adaptation benchmark. arXiv preprint arXiv:1910.04866 2019b.
Hengshuang Zhao, Jiaya Jia, and Vladlen Koltun. Exploring self-attention for image recognition. In
CVPR, 2020.
12

---

Published as a conference paper at ICLR 2021
Models
Dataset
Epochs
Basc LR
LR decay
Weight decay
Dropout
VIT-B/(16,32)
JFT-300M
10 10
linear
0.0
ViT-L/32
JFT-300M
6 10
linear
0.1
0.0
ViT-L/16
JFT-300M
7/14
4 1-10-
linear
0.1
0.0
VIT-H/14
JFT-300M
14
3 3-10-
linear
0.1
0.0
R50x(1.2)
JFT-300M
10
linear
0.1
0.0
R101x1
JFT-300M
8 8-104
linear
0.1
0.0
[1,2x
JFT-300M
7
6 6010-
linear
0.1
0.0
150+V
JFT-300M
8-10
linear
R50+VIT-L/32
JFT-300M
7
2 2-10
linear
0.1
0.0
R50+VIT-L/16
JFT-300M
7/14
4-10-4
linear
0.0
(66-2/(16.32)
ImageNet-22
90
10
linear
0.03
0.1
VIT-L/(16.32)
ImageNet-21k
30/90
10-3
linear
0.03
0.1
V
ImageNet
300
3 10
cosine
0.3
0.1
Table 3: Hyperparameters for training. All models are trained with a batch size of 4096 and learn- ing rate warmup of 10k steps. For ImageNet we found beneficial to additionally apply gradient clipping at global norm 1. Training resolution is 224.
APPENDIX
A MULTIHEAD SELF-ATTENTION
Standard qkv self-attention (SA. Vaswani et al. (2017)) is popular building block for neural archi- tectures. For each element in input sequence R. xx11, compute weighted sum over all values v in the sequence. The attention weights Aij are based on the pairwise similarity between two elements of the sequence and their respective query q' and key k representations.
q. k, v) = zU ykv
Ugkv E Dx3D,
(5)
= softinax (qk") /VDDa)
AERNNN
(6)
SA(z) Av.
Multihead self-attention (MSA) is an extension of SA in which we run self-attention operations, called "heads", in parallel, and project their concatenated outputs. To keep compute and number of parameters constant when changing k, Dn. (Eq. 5) is typically set to D/k:
MSA(2) (s), (A) : Umsa
Unso E & DD
(8)
B EXPERIMENT DETAILS
B. TRAINING
Table summarizes our training setups for our different models. We found strong regularization to be key when training models from seratch ImageNet. Dropout, when used, is applied after every dense layer except for the the qkv-projections and directly after adding positional- to patch embeddings. Hybrid models trained with the exact setup their ViT counterparts. Finally, all training is done on resolution 224.
B.I.1 FINE-TUNING
We fine-tune all ViT models using SGD with momentum of 0.9. We run a small grid search over learning rates, see learning rate ranges in Table To do so, use small sub-splits from the training set (10% for Pets and Flowers, 2% for CIFAR, 1% ImageNet) as development set and train on the remaining data. For final results train the entire training set and evaluate the respective test data. For fine-tuning ResNets and hybrid models we use the exact same setup. with the only exception of ImageNet where add another value 0.06 to the learning rate sweep. Additionally,
13

---

Published as a conference paper at ICLR 2021
Datasct
Steps
Base LR
ImageNet
20000
(0.003. 0.01. 0.03, 0.06)
CIFARIO0
10000
{0.001, 0.003, 0.01, 0.03
CIFARIO
10000
{0.001, 0.003, 0.01, 0.03
Oxford- IIIT Pets
500
{0.001, 0,003, 0.01, 0.03)
Oxford Flowers-102
500
(0.001, 0.003, 0.01, 0.03)
VTAB (19 tasks)
2,500
0.01
Table 4: Hyperparameters for fine-tuning. All models are fine-tuned with cosine learning rate decay. batch size of 512, no weight decay, and grad clipping at global norm 1. If not mentioned otherwise, fine-tuning resolution is 384.
for ResNets also run the setup of Kolesnikov al. (2020) and select the best results across this run and our sweep. Finally, if not mentioned otherwise, all fine-tuning experiments run at 384 resolution (running fine-tuning at different resolution than training is common practice (Kolesnikov et al. 2020)).
When transferring ViT models another dataset, remove the whole head (two linear layers) and replace it by single, zero-inan initialized linear layer outputting the number of classes required by the target dataset. We found this be a little more robust than simply re-initializing the very last layer.
For VTAB we follow the protocol in Kolesnikov et al. (2020), and use the same hyperparameter setting for all tasks. We use a learning rate of 0.01 and train for 2500 steps (Tab. 4 We chose this setting by running a small sweep over two learning rates and two schedules, and selecting the setting with the highest VTAB score on the 200-example validation sets. We follow the prepproesssing used in Kolesnikov et al. (2020), except that we do not use task-specific input resolutions. Instead we find that Vision Transformer benefits most from a high resolution (384 384) for all tasks.
B.1.2 SUPE-IIPERI
We employ the masked patch prediction objective for preliminary supervuupnii experiments. To do so we corrupt 50% of patch embeddings by either replacing their embeddings with a learnable [ma sk] embedding (80%), random other patch embedding (10%) or just keeping them is (10%). This setup is very similar to the one used for language by Devlin et al. (2019). Finally, we predict the 3-bit, mean color (i.e., 512 colors total) of every corrupted patch using their respective patch representations.
We trained our self-supervised model for 1M steps (ca. 14 epochs) with batch size 4096 JFT. We use Adam, with a base learning rate of 2- 10 10-4. 44, warmup of 10k steps and cosine learning rate decay. As prediction targets for pretraining we tried the following settings:
predicting only the mean, 3bit color (i.c., prediction of 512 colors), 2) predicting a 4 x 4 downsized version of the 16 x 16 patch with 3bit colors in parallel (i.e., 16 predictions of 512 colors), 3) regression on the full patch using I.2 (i.e., 256 regressions on the 3 RGB channels). Surprisingly, we found that all worked quite well, though L2 was slightly worse. We report final results only for option I) because it has shown best few-shot performance. We also experimented with 15% corruption rate as used by Devlin et al. 2019) but results were also slightly worse on our few-shot metrics.
Lastly, would like to remark that our instantiation of masked patch prediction doesn't 'tt require such an enormous amount of pretraining nor a large dataset such as JFT in order to lead to sim- ilar performance gains ImageNet classification. That is, observed diminishing returns downstream performance after 100k pretraining steps, and see similar gains when pretraining on ImageNet.
C ADDITIONAL RESULTS
We report detailed results corresponding to the figures presented in the paper. Table 5 corresponds to Figure 3 from the paper and shows transfer performance of different ViT models pre-trained on datasets of increasing size: ImageNet, ImageNet-21k, and JFT-300M. Table 6 corresponds to
14

---

Published as a conference paper at ICLR 2021
VIT-B/16
ViT-B/32
VIT-L/16
ViT-L/32
ViT-H/14
ImageNet
CIFAR-10
98.13
97.77
97.86
97.94
CIFAR-100
87.13
86.31
86.35
87.07
ImageNet
77.91
73.38
76.53
71.16
ImageNet ReaL
83.57
79.56
82.19
77.83
Oxford Flowers- 102
89.49
85.43
89.66
86.36
Oxford-IITT-Ptss
93.81
92.04
93.64
91.35
ImageNet-21k
CIFAR-10
98.95
98.79
99.16
99.13
99.27
CIFAR-100 100
91.67
91.97
93.44
93.04
93.82
ImageNet
83.97
81.28
85.15
80.99
85.13
ImageNet ReaL
88.35
86.63
88.40
85.65
88.70
Oxford Flowers- 102
99.38
99.11
99.61
99.19
99.51
Oxford-UIIT-Ptts
94.43
93.02
94.73
93.09
94.82
JFT-300M
CIFAR-IO
99.00
98.61
99.38
99.19
99.50
CIFAR- 100
91.87
90.49
94.04
92.52
94.55
ImageNet
84.15
80.73
87.12
84.37
88.04
ImageNet Real.
88.85
86.27
89.99
88.28
90.33
Oxford Flowers- 102
99.56
99.27
99.56
99.45
99.68
Oxford- IIT-Pess
95.80
93.40
97.11
95.83
97.56
Table 5: Top1 accuracy (in %) of Vision Transformer various datasets when pre-trained on Im- ageNet, ImageNet-21kk or JFT300M. These values correspond to Figure in the main text. Models fine-tuned at 384 resolution. Note that the ImageNet results computed without additional techniques (Polyak averaging and 512 resolution images) used to achieve results in Table
Model
Epochs
ImaggNet
ImageNet Real.
CIFAR-10
CIFAR-
Flowers
exaFLOPS
VIT-B/32
7
80.73
86.27
98.61
90.49
93.40
99.27
164
VIT-R/16
84.15
88.85
99.00
91.87
95.80
99.56
743
ViT-L/32
7
84.37
88.28
99,19
92.52
95.83
99.45
574
VIT-L/16
7
86,30
89.43
99.38
93.46
96.81
99.66
2586
VIT-L/16
14
87.12
89.99
99.38
94.04
97.11
99.56
5172
VIT-H/14
88.08
90.36
99.50
94.71
97.11
99.71
12826
ResNet50xl
7
77.54
84.56
97.67
86.07
91.11
94.26
150
ResNet50x2
82.12
87.94
98.29
89.20
93.43
97.02
592
ResNct101x1
80.67
87.07
98.48
89.17
94.08
95.95
285
152xl
7
81,88
87.96
98.82
90.22
94.17
96.94
427
ResNer152x2
7
84.97
89.69
99.06
92.05
95.37
98.62
1681
ResNet152x2
14
85.56
89.89
99.24
91.92
95.75
98.75
3362
ResNer200c3
87.22
90.15
99.34
93.53
96.32
99.04
10212
250x1+ViT- BB3V
7
84,90
89.15
99.01
92.24
95.75
99.46
315
R
85.58
89.65
99.14
92.63
96.65
99.40
VIT-L42
85.68
89.04
99.24
92.93
96.97
99.43
725
VIT-L-/
7
86.60
89.72
99.18
93.64
97.03
99.40
2704
R50x1+VTT-L116
14
87.12
89.76
99.31
93.89
97.36
99.11
5165
Table 6: Detailed results of model scaling experiments. These correspond to Figure 5 in the main paper.
Figure 5 from the paper and shows the transfer performance of ViT, ResNet, and hybrid models of varying size, as well as the estimated computational cost their pre-training.
D ADDITIONAL ANALYSES
D.1 SGD vs. ADAM FOR RESNETS
ResNets are typically trained with SGD and our use of Adam as optimizer quite unconventional. Here show the experiments that motivated this choice. Namely, compare the fine-tuning performance of two ResNets 50x1 and 152x2 pre-trained on JFT with SGD and Adam. For SGD, the hyperparameters recommended by Kolesnikov al. (020). Results presented
15

---

Published as a conference paper at ICLR 2021
ResNet50
ResNet152x3
Dataset
Adam
SGD
Adam
SGD
ImageNet
77.54
78.24
84.97
84.37
CIFARIO I0
97.67
97.46
99.06
99.07
CIFAR 100
86.07
85.17
92.05
91.06
Oxford-IIT Pets
91.11
91.00
95.37
94.79
Oxford Flowers-102
94.26
92.06
98.62
99.32
Average
89.33
88.79
94.01
93.72
Table 7: Fine-tuning ResNet models pre-trained with Adam and SGD.
0.8
0.6
ImageNet 5shot
0.5
Average 5shot
0.7
0.4
Models
Models
All
All
0.3
Depth
Depth
Patch size
0.5
Patch size
0.2
Width MLP
Width MLP
Width
Width
X
0.4
10"
10a
10"
10a
Relative Compute
Relative Compute
Figure 8: Scaling different model dimensions of the Vision Transformer.
in Table 7 Adam pre-training outperforms SGD pre-training on most datasets and on average. This justittes the choice of Adam as the optimizer used pre-train ResNets JFT. Note that the absolute numbers are lower than those reported by Kolesnikov et al. (2020). since we pre-train only for 7 epochs, not 30.
D.2 TRANSFORMER SHAPE
We ran ablations scaling different dimensions of the Transformer architecture to find out which are best suited for scaling to very large models. Figure [ shows 5-shot performance on ImageNet for different configurations. All configurations DMLP = 2048 and a patch size 32, the intersection of all lines. We can see that scaling the depth results in the biggest improvements which clearly visible up until 64 layers. However, diminishing returns are already visible after 16 layers. Interestingly, scaling the width of the net- work seems result in the smallest changes. Decreasing the patch size and thus increasing the effective sequence length shows surprisingly robust improvements without introducing parameters. These findings suggest that compute might be better predictor of performance than the number of parameters, and that scaling should emphasize depth over width if any. Overall, we find that scaling all dimensions proportionally results robust improvements.
based on VIT model with 8 layers, D = 1024,
D.3 POSITIONAL EMBEDDING
We ablations different ways of encoding spatial information using positional embedding. We tried the following cases:
Providing no positional information: Considering the inputs as a bag of patches.
1-dimensional positional embedding: Considering the inputs sequence of patches in the raster order (default across other experiments in this paper).
d-dmennsional positional embedding: Considering the inputs dimensions. In this case, two sets of embeddings are learned, each for one the axes, X Tembeddinng, and Yeembedding, each with size D/2. Then, based the coordinate
grid of patches in two
16

---

Published as a conference paper at ICLR 2021
Pos. Emb.
Default/Stem
Every Layer
Every Layer-Shared
No Pos. Emb.
0,61382
N/A
N/A
1-D Pos. Emb.
0.64206
0.63964
0.64292
2-D) Pos. Emb.
0.64001
0.64046
0.64022
Rel. Pos. Emb. 0.64032
N/A
N/A
Table 8: Results of the ablation study on positional embeddings with ViT- model evaluated on ImageNet 5-shot linear.
VIT
opochs, LR- 0.0002, WD-0.01
7 spochs, LR-0,0004, WD-0.1
14 cpochs, 0.000, WD-0.1
88
12
20
13
13
18
11402400 233mm
Prpu parch conume
rput columm
Figure 9: Position embeddings of models trained with different hyperparameters.
the path in the input, we concatenate the X and embedding to get the final positional embedding for that patch.
Relative positional embeddings: Considering the relative distance between patches to en- code the spatial information as instead of their absolute position. To do so, we 1- dimensional Relative Attention, in which we define the relative distance all possible pairs of patches. Thus, for every given pair (one as query, and the other as key/value in the tention mechanism), we have an offset P4 Ps. where each offset is associated with an embedding. Then, we simply run extra attention, where we use the original query (the content of query), but use relative positional embeddings as keys. We then use the log- its from the relative attention as bias term and add the logits of the main attention (content-based attention) before applying the softmax.
In addition to different ways of encoding spatial information, we also tried different ways of in- corporating this information in our model. For the 1-dimensional and 2-dimensional positional embeddings, we tried three different cases: (1) add positional embeddings to the inputs right after the stem of them model and before feeding the inputs the Transformer encoder (default across all other experiments in this paper); (2) learn and add positional embeddings to the inputs at the beginning of each layer; (3) add a learned positional embeddings the inputs at the beginning of each layer (shared between layers).
Table & summarizes the results from this ablation study on a ViT-B/I6 model. As we can see, while there IS large gap between the performances of the model with no positional embedding and mod- els with positional embedding, there little to no difference between different ways of encoding positional information. We speculate that since our Transformer encoder operates patch-level inputs, as opposed to pixel-level, the differences in how to encode spatial information is less impor- tant. More precisely, in patch-level inputs, the spatial dimensions pixel-level inputs, e.g., 14 x 14 as opposed to 224 x 224, and learning to represent the spatial re- lations in this resolution is equally easy for these different positional encoding strategies. Even so, the specific pattern of position embedding similarity learned by the network depends on the training hyperparameters (Figure
much smaller than the original
17

---

Published as a conference paper at ICLR 2021
VIT-L16
R50x1 VIT-L/16
Moan attention distance (pixate)
100
100
80
40
Head
Head
20
Head
Head
Head
Head
15
20
Network depth (layer)
Network depth (ayer)
Figure 10: Size of attended area by head and network depth. Attention distance was computed for 128 example images by averaging the distance between the query pixel and all other pixels, weighted by the attention weight. Each dot shows the mean attention distance across images for one of 16 heads at one layer. Image width is 224 pixels.
D.4 EMPIRICAL COMPUTATIONAL COSTS
We are also interested in real-world speed of the architectures on our hardware, which is not always well predicted by theoretical FLOPs due to details like lane widths and cache sizes. For this purpose, we perform timing of inference speed for the main models of interest, on TPUv3 accelerator; the difference between inference and backprop speed is constant model-independent factor.
Figure [I (left) shows how many images one core can handle per second, across various input sizes. Every single point refers to the peak performance measured across wide range of batch-sizes. As can be seen, the theoretical bi-quadratic scaling of ViT with image size only barely starts happening for the largest models at the largest resolutions.
Another quantity of interest is the largest batch-size each model can onto a core, larger being better for scaling large datasets. Figure 11 (right) shows this quantity for the same set of models. This shows that large ViT models have a clear advantage in terms memory-effiiiency over ResNet models.
D.5 AXIAL ATTENTION
Axial Attention (Haang et al. 2020: Ho et al. 2019) is simple, yet effective technique run self- attention on large inputs that are organized as multidimensional tensors. The general idea of axial attention is to perform multiple attention operations, each along single axis of the input tensor, instead of applying 1-dimensional attention to the flattened version of the input. In axial attention, each attention mixes information along particular axis, while keeping information along the other axes independent. Along this line, Wang et al. (2020b) proposed the AxialResNet model in which all the convolutions with kernel size x ResNet50 are replaced by axial self-attention, i.e. and column attention, augmented by relative positional encoding. We have implemented AxialResNet as a bascline model
Moreover, we have modified ViT to process inputs in the 2-dimensional shape, instead of 1- dimensional sequence patches, and incorporate Axial Transformer blocks, in which instead of
PyTorch implementation pattttp://// /ttp::://tttbt com/ rsrhdel am/ axial al- waalddeepplr deeplab) In our experiments, we reproduced the scores reported in (Wang et all 2020b) in terms of accuracy, however. our implementation. similar to the open-source implementation. is very slow on TPUs. Therefore, were not able to use it for extensive large-scale experiments. These may be unlocked by carefully optimized implementation.
1 "Our implementation based on
18

---

Published as a conference paper at ICLR 2021
R50x
VIT-B/32
VIT-B/16
VIT-R/14
RI52x4
Peak inference speed (Emg/scc/corr)
10a
10
Largest batch
10¹
10*
10
64
128
224
384
512
64
128
224
384
512
Input size [px]
Input size [px]
Figure 11: Left: Real wall-clock timings of various architectures across input sizes. VIT models have speed comparable similar ResNets. Right: Largest per-core batch-size fitting device with various architectures across input sizes. ViT models are clearly more memory-efficient.
self- attention followed by an MLP, we have row-self-attention plus an MLP followed by column-self-atteme plus an MLP.
AAGAVIT B:16
AzislVIT-B 4800
ViT-s-02
VTT-8.22
0.59
6:53
0.95
0.00
0.500
compote [ox.FLOPP)
speel [ingfiestoed
Figure 12: Performance of Axial-Attention based models, in terms of top-1 accuracy on ImageNet 5-shot linear, versus their speed in terms of number of FLOPs (left) and inference time (left).
Figure 2 present the performance of Axial ResNet, Axial- ViT-B/32 and Axial- VIN--VIT--IIII B/16 on Ima- geNet 5shot linear, when pretrained JFT dataset, verses the pretraining compute, both in terms of number FLOPs and inference time (example per seconds). As we can see, both Axial-ViT-B/32 and Axial-ViT-B/i do better than their VIT-B counterpart in terms of performance, but it comes the cost of more compute. This is because in Axial-ViT models, each Transformer block with global self-attention is replaced by two Axial Transformer blocks, one with and with column self- attention and although the sequence length that self attention operates on is smaller in axial case, there is a extra MLP per Axial-Viis block. For the AxialResNet, although it looks reasonable in terms of accuracy/compute trade-off (Figure 12 left), the naive implementation is extremely slow on TPUs (Figure 12 right).
D.6 ATTENTION DISTANCE
To understand how VIT uses self-attention to integrate information across the image, analyzed the average distance spanned by attention weights at different layers (Figure [0]. This "attention distance" is analogous receptive field size in CNNs. Average attention distance is highly variable
19

---

Published as a conference paper at ICLR 2021
across heads in lower layers, with some heads attending much of the image, while others attend to small regions at or near the query location. As depth increases, attention distance increases for all heads. In the second half of the network, most heads attend widely across tokens.
D.7 ATTENTION MAPS
To compute maps of the attention from the output token to the input space (Figures 6 and 13 we used Attention Rollout (Abnar& Zuidemal 2020). Briefly, we averaged attention weights of ViT- 1/16 across all heads and then recursively multiplied the weight matrices of all layers. This accounts for the mixing of attention across tokens through all layers.
D.8 VTAB BREAKDOWN
Table 9 shows the scores attained each of the VTAB-Ik tasks.
Table 9;
Breakdown of VTAB-IK performance across tasks.
Caltech111
CIARR-100 100
DTD
Flowers102
Pets
Sun
SVIN
Camelyon
EumSAT
Resise45
Retinopath
Clev Count
Clex-Dist
DMLab
dSpr-Loc
dSpr-Oni
KITT-Dis
SNORD- Azim
SNORB Elee
Mean
VIT-H/14 (JFT) 95.3
85.5
75.2
99.7
97.2
65.0
88.9
81.3
96.7
91.4
76.6
91.7
63.8
53.1
79.4
63.3
84.5
33.2
51.2
77.6
VIT-L/16 (JT))
95.4
81.9
74.3
99.7
96.7
63.5
87.4
83.6
96.5
89.7
77.1
86.4
63.1
49.7
74.5
60.5
82.2
36.2
51.1
76.3
VIT-L/16 (21k)
90.8
84.1
74.1
99.3
92.7
61.0
80.9
82.5
95.6
85.2
75.3
70.3
56.1
41.9
74.7
64.9
79.9
30.5
41.7
72.7
20

---

Published as a conference paper at ICLR 2021
12
16
25
27
32
17
55
CO
52
64
72
107
113
214
115
135
213
118
1:0
120
121
120
:25
126
:20
Figure 13: Further example attention maps as in Figure (random selection).
21


Reviews:
[{"review_id": "eXA6UlaiTPz", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Following the transformer successes in NLP, this paper explores the performance of a vanilla Transformer (with few simple modifications) for the task of image classification.  It has been experimentally validated that this Transformer (ViT) can attain superior performance for this task compared to SOTA CNN architectures if it is pre-trained on large amounts of data and then applied to mid-sized or small image classification benchmarks. Moreover, it has been shown that ViT requires considerably fewer computational resources to train compared to computationally demanding CNN backbones such as ResNet. Clearly, the paper has the potential to re-ignite another wave of excitement for exploring this great learning model on different computer vision tasks. To this end, I believe this paper has enough merit to be accepted.\n\nThe paper delivers a strong message \" transformers can be a more powerful, yet efficient,  compared to the SOTA CNN backbones for image recognition tasks if there is a large-enough dataset available \" and the authors prove this claim by performing comprehensive experiments on several large-scale image recognition benchmarks.  This level of experimental verifications is only possible if huge computation re-sources are only available, which is not accessible for most research teams, esp in academia. Otherwise, a similar message (replacing CNN with transformers/attention) has been attempted to be verified in a few earlier works (as acknowledged in this paper as well). \n\nAlso, it is hard to imagine that ViT can yet compete with the CNN backbones for vision tasks beyond the considered tasks because:\n\n1- it is not clear how the simple ViT can be extended for the vision tasks which require pixel-level predictions, e.g. image segmentation, depth prediction etc, or 3D vision tasks while being still computationally tractable.\n\n2-  As argued in the paper, transformers are data-hungry to perform as well as ResNet for a task and the availability of such large-scale annotations (e.g. 300M samples) beyond image labelling may not be feasible yet. \n\n3-  Considering transferring a pre-trained model,  there is no evidence that a pre-trained ViT (e.g. on a large amount of data) can still learn a reliable representation for the other basic vision tasks beyond image recognition such as image segmentation or object detection, and still carry a superior performance compared to ResNet when it is fine-tuned... The superiority of DETR (Carion et al 2020) on the detection task is not solely due to the use of a transformer but rather formulating object detection as a set prediction problem to avoid heuristics such as NMS.\n\nOther comments:\n\nI couldn't find any ablation study on different choices of 2D patches sizes versus training time and accuracy except partial experiments on 14,16 and 32. I am not sure what would be best patch size given any image, eg a panoramic image with high resolution when considering both accuracy and training computation compared to ResNet. \n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "d3cPgAwsikF", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces a Transformer-based image recognition model that is fully built on the Transformer layers (multi-head self-attention + point-wise MLP) without any standard convolution layers. Basically, it splits an image into patches and takes as input the set of linear embeddings of the patches and their positions. For classification, a learnable class token is added to the input and a classification head (MLP) is attached to the class token output of the final Transformer layer. Extensive experiments of transfer learning show that when pretrained on a sufficiently large dataset (100~300M), the proposed Vision Transformer can outperform state-of-the-art convolutional networks with less training cost as well as less number of parameters.\n\nPros\n1. Clearly motivated and well written \n- The background of the research, the motivation of Vision Transformer, and the related work are all clearly stated and summarized. \n\n2. The first fully-Transformer-based image recognition model \n- The simple yet effective Vision Transformer is introduced by adapting the original Transformer with minimal modification; patch embedding, class token, and class head. \n\n3. Extensive evaluation and good analysis \n- This paper demonstrates the power of the Vision Transformer model by extensive large-scale experiments, outperforming SOTA CNN models. Comparative evaluation with important baselines, ResNet and hybrid models, are well designed and conducted with different scales of datasets and models. The results are impressive and interesting. Additional discussions in Appendix are also useful in understanding the model. \n\nCons\n1. No significant technical novelty\n-  The proposed model is incremental modifications of the original Transformer and its existing variants. \n\n2. Lack of further analysis of the inductive bias \n- The authors contrast the Transformer with CNN, stating “Transformers lack some inductive biases inherent to CNNs, such as translation equivariance and locality, and therefore do not generalize well when trained on insufficient amounts of data.” and “the convolutional inductive bias is useful for smaller datasets, but for larger ones, learning the relevant patterns is sufficient, even beneficial.” However, it is not clear which inductive bias of CNN prevents better generalization; is it translation equivariance, locality, static weights of kernels, or static size of kernels? I don’t find any clear answers from the paper. Since there exist different inductive biases in standard convolution, it may be the case that some of them help generalization and some others do not; worse generalization of CNN with a larger dataset may be affected by all those factors in a tangle. In this regard, further analysis of this issue would improve this work. E.g., in order to check the benefit of locality bias, ViT with local self-attention can also be compared, etc. \n\n3. Some misleading statements\n- The authors repeatedly call the input of patch embeddings an “input sequence”, which I guess is intended to remind of the NLP origin. But, this may confuse readers. The Transformer used in this work does not process the input as a sequence, and actually the output is equivariant to any permutation of the patch (+position) embeddings. \n- “Transformers lack some inductive biases inherent to CNNs, such as translation equivariance and locality”. This needs to be clarified. As I noted above, since the Transformer in this work is permutation equivariant, it can also be seen as translation equivariant. As for “locality”, the Transformer also performs point-wise (patch-wise) processing, thus leveraging locality. \n- “Unlike prior works using self-attention in computer vision, we do not introduce any image-specific inductive biases into the architecture.” This statement in conclusion is an overclaim, conflicting with the statement at the end of Sec. 2.1 (resolution adjustment and patch extraction). \n- The title of this paper “An image is worth 16x16 words”, what does it mean? I don’t find an answer from the paper. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "MRG7Aw4c273", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Summary\nThis paper studies to adapt transformer model for image classification task. The new model performs comparably well on various popular benchmarks, and will out-perform when pre-trained on large dataset. \n\n## Pros \n1. The technical solution is surprisingly simple, yet achieves strong performance. All it needs is to cut input images into patches, and reshape it as an input sequence. The transformer architecture is kept almost the same as in NLP tasks. The simplicity of the model makes it easy to generalize many vision tasks potentially.\n2. The experimental section has clearly demonstrated the pros and cons of the proposed model. The authors not only show the strong performance but also stay upfront about the limits. \n3. The additional analysis, visualizations, and self-supervised pre-training sections are very informative. \n4. The writing is mostly clear and easy to follow.\n\n## Cons\n1. The arguments about \"inductive biases\" are confusing and self-contradictory. On one hand,  the introduction section says that CNN generalize better due to the inductive biases such as translation equivariance and locality. On the other hand, the rest of the paper claims that avoid inserting inductive biases into the transformer is an advantage.  In my understanding, these inductive biases have been shown to be beneficial for most vision tasks in CNN. Why not introducing some of them into transformer? Please clarify.\n2. It's not clear to me how the few shot accuracy is computed. Is the the same evaluation as in self-supervised representation learning? What kind of regularization is applied to the linear regression, and how to learn the mapping? \n\n## Minor\n1. It might be better to give reference when mentioning ````100B parameters in the first paragraph of introduction.\n2. $z_0^L$ on page 2 bottom seems to be a typo. Should it be $z_L^0$?\n3. In Sec.1 ```Hybrid Architecture, the statement \"In this hybrid model, the patch embedding projection E (Eq. 1) is replaced by the early stages of a ResNet\" is misleading. As in Eq.(1) ${\\bf E}$ is applied on different input patches $x_p^i$, whereas early stages of ResNet are applied on the same input (image $x$). \n\n## Questions\n1.  If I understand correctly, a model like ViT-L/16 means that it takes 16x16 fixed-sized patches as input. Does this mean that the images get resized to fixed resolution before cutting into patches? How big are those patches? Does the size matter?\n2.  As shown in prior work (Unsupervised Visual Representation Learning by Context Prediction, Doersch etal., ICCV 2015), enforcing the patches to be discontinued and with some randomness is beneficial to self-supervised representation learning. Have the authors tried similar processing method? An ablation on different ways of generating patches (discontinued, overlapped, mixed-scale...) might be useful as some simple techniques might greatly improve the performance.", "extra_scores": {}, "extra_text": {}}, {"review_id": "-MByllXmCk6", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper explores the Transformer architecture for image recognition. The authors mainly follow the settings of BERT in NLP where the input is a sequence of words. To adopt Transformer for image recognition, the authors split the image into grid patches as inputs for the vision Transformer (ViT). Other settings like adding a special token at the beginning for classification and adding position embeddings are the same as those used in BERT. While Transformer is not new, the way it is applied on raw input images directly for image classification has not been done before. I also like the analogy with the hybrid model where the early layers in ResNet serve the role similar to the patch embedding projection. More importantly, the authors have conducted extensive experiments to validate the effectiveness of the proposed approach. Some of the visualizations are also insightful. The main conclusion is that ViT can outperform BiT(ResNet) when pre-trained on large-scale training data with supervised training. \n\nAs the Transformer model has become more popular in the vision community, it seems natural for the authors to come up with ViT for image recognition. I think the main contribution of this paper is not to propose this idea, but rather to make it work. The fact that ViT can outperform BiT with ResNet only when it scales-up to a huge model trained on a huge dataset makes it impossible for most researchers with limited resources to conduct such experiments. Although the results may not be that encouraging in my opinion, I think the experiments conducted in this paper are valuable for the research community. \n\nMy major concern is that the pre-training conducted in this paper is fully supervised. This is different to BERT that can use unlabeled text data or Vision-Language Pre-training where weakly-labeled image-text pairs can be leveraged. The authors briefly mentioned that the masked patch prediction can still improve ImageNet classification accuracy compared to training from scratch, but it has a gap with supervised pre-training. It would be great to explore self-supervised pre-training in the future as also pointed out by the authors. \n\nAnother concern is about the comparison with the hybrid model. The main drawback of applying ViT on the full image is that it cannot scale to large input resolution due to memory constraint. A hybrid approach seems to be a good work-around to handle this case. While the authors have mentioned the hybrid model, very limited comparisons are conducted in the experiments. Only Figure 5 shows some results of hybrid models. In some cases, the hybrid model outperforms ViT. Why not add more data points for hybrid models at higher pre-training compute cost? Why not compare the numbers in Table 2 and Figure 2-4? \n\nIn the appendix, the authors mention that the absolute numbers of BiT are lower than those reported by Kolesnikov et al. (2020), since they pre-train only for 7 epochs, not 30. Is it possible that there is more room to improve for BiT (ResNet) in Figure 5 if we increase the pre-training cost? At least the trend does not show saturating performance of BiT. \n\nOverall, I think the authors have done a great job in conducting all these experiments. Although the results are not that encouraging as these experiments require too many resources, I think the experiments can provide some value for future research. I would like the authors to comment on the comparison with BiT and the hybrid model in Figure 5. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper has generated a lot of great discussion and it presents a very different way of doing image recognition at scale compared to current state of the art practices.  All reviewers rated this paper as an accept.
This work is interesting enough that in my view it really deservers further exposure and discussion and an oral presentation at ICLR would be a good way to achieve that.
Decision: Accept (Oral)