How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers
Authors: Andreas Peter Steiner
Alexander Kolesnikov
Xiaohua Zhai
Ross Wightman
Jakob Uszkoreit
Lucas Beyer
Venue: Accepted by TMLR ()

Abstract: Vision Transformers (ViT) have been shown to attain highly competitive performance for a wide range of vision applications, such as image classification, object detection and semantic image segmentation. In comparison to convolutional neural networks, the Vision Transformer's weaker inductive bias is generally found to cause an increased reliance on model regularization or data augmentation (``AugReg'' for short) when training on smaller training datasets. We conduct a systematic empirical study in order to better understand the interplay between the amount of training data, AugReg, model size and compute budget. As one result of this study we find that the combination of increased compute and AugReg can yield models with the same performance as models trained on an order of magnitude more training data: we train ViT models of various sizes on the public ImageNet-21k dataset which either match or outperform their counterparts trained on the larger, but not publicly available JFT-300M dataset.


Published in Transactions on Machine Learning Research (05/2022)
How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers
Andreas Steiner*
andstein@google.com
Alexander Kolesnikov
akolesmihonnggooll.ccom
Xiaohua Zhai"
wahai@google.com
Ross Wightman
ruightman&igmaill.cm
Jakob Uszkoreit
usz@google.com
Lucas Beyer"
lbeyer@google.com com
Google Research, Brain Team, Zürich
Equal technical contribution, independent researcher
Reviewed on OpenReview: https: // openreview. net/ forum? id= 4nPswr1kcP
Abstract
Vision Transformers (ViT) have been shown to attain highly competitive performance for wide range of vision applications, such image classification, object detection and semantic image segmentation. In comparison to convolutional neural networks, the Vision Transformer's weaker inductive bias is generally found to cause an increased reliance on model regularization or data augmentation ("AugReg" for short) when training on smaller training datasets. We conduct a systematic empirical study in order to better understand the interplay between the amount of training data, AugReg, model size and compute budget." As result of this study find that the combination of increased compute and AugReg can yield models with the same performance as models trained an order of magnitude more training data: we train VIT models of various sizes on the public ImageNet-21k dataset which hit match or outperfomm their counterparts trained on the larger, but not publicly available JFTT330M dataset.
1 Introduction
The Vision Transformer (ViT) (13) has recently emerged as competitive alternative to convolutional neural networks (CNNs) that ubiquitous across the field of computer vision. Without the translational equivariance of CNNs, VIT models are generally found to perform best in settings with large amounts of training data (13) to require strong AugReg schemes to avoid overfitting (39). However, so far there was no comprehensive study of the trade-offs between model regularization, data augmentation, training data size and compute budget in Vision Transformers.
In this work, we fill this knowledge gap by conducting thorough empirical study. We pre-train a large collection of ViT models (different sizes and hybrids with ResNets (18)) on datasets of different sizes, while
We release more than 50000 VIT models trained under diverse settings various datasets. We believe this be treasure trove for model analysis. Available at httpa://github com/ google-reeerrchvvviiio.... craassforrre and https: //githus //g/thub.ccmmmrigggie. wwtpr//igggtmt//ptooooooooooo //gi thub. con/ ham//ooggle-eeearee
maage-nodels. The code full reproduction of model training available https: vision.

---

Published in Transactions on Machine Learning Research (05/2022)
ImageNet top-1 accuracy after fine-tuning
90%
85%
80%
75%
70%
VIT-B/32
65%
VIT-R/16
VIT-L/16
60%
1.28M
1.25M-AngReg
13M
Aug+Ag
300M
Pro-training dataset size
Figure 1: Adding the right amount of regularization and image augmentation can lead to similar gains increasing the dataset size by an order of magnitude.
at the same time performing carefully designed comparisons across different amounts of regularization and data augmentation. We then proceed with extensive transfer learning experiments for the resulting models. We focus mainly on gaining insights useful for practitioner with limited compute and data budgets.
The homogencity of the performed study constitutes vast. majority of works involving Vision Transformers it is not practical to retrain all baselines and proposed methods on equal footing, in particular those trained on larger amounts of data. Furthermore, there are numerous subtle and implicit design choices that cannot be controlled for effectively, such as the precise implementation of complex augmentation schemes, hyper-parameters (e.g. learning rate schedule, weight decay), test-time preprocessing, dataset splits and so forth. Such inconsistencies can result in significant. amounts of noise added to the results, quite possibly affecting the ability draw any conclusions. Hence, all models which this work reports have been trained and evaluated in consistent setup.
of the key contributions of this paper. For the
The insights we draw from our study constitute another important contribution of this paper. In particular, we demonstrate that carefully selected regularization and augmentations roughly correspond (from the perspective of model accuracy) to a 10x increase in training data size. However, regardless of whether the models are trained with more data or better AugRegs, one has to spend roughly the same amount of compute to get models attaining similar performance. We further evaluate there difference between adding data better AugReg when fine-tuning the resulting models on dalasets of various categories. Other findings, such the overall beneficial effect of AugRegs for medium-sizze datasets, simply confirm commonly held beliefs. For those findings, the value of this study lies not in novelty, but rather in confirming these assumptions and quantifying their effect in strictly controlled setting.
In addition, we aim to shed light on other aspects of using Vision Transformers in practice such as comparing transfer learning and training from scratch for mid-sized datasets. Finally, we evaluate various compute versus performance trade-offs. We discuss all of the aforementioned insights and more in detail in Section 4.
2 Scope of the study
With the ubiquity of modern deep learning (25) in computer vision it has quickly become common practice to pre-train models on large datasets once and re-use their parameters as initialization or feature extraction part in models trained on a broad variety of other tasks (32; 45).
In this setup, there are multiple ways to characterize computational and sample efficiency. When simply considering the overall costs of pre-training and subsequent training or fine-tuning procedures togother, the cost of pre-training usually dominates, often by orders of magnitude. From the vantage point of researcher aiming to improve model architectures or pre-training schemes, the pre-training costs might therefore be most.
2

---

Published in Transactions on Machine Learning Research (05/2022)
relevant. Most practitioners, however, rarely, if ever perform pre-training today's largest datasets but instead use some of the many publicly available parameter sets. For them the costs of fine-tuning, adaptation or training a task-specific model from scratch would be of most interest.
Yet another valid perspective that all training costs are effectively negligible since they are amortized over the course of the deployment of a model in applications requiring a very large number of invocations of inference.
In this setup there are different viewpoints on computational and data efficiency aspects. One approach is to look at the overall computational and sample cost of both pre-training and fine-tuning. Normally, "pre-training cost" will dominate overall costs. This interpretation is valid in specific scenarios, especially purposes. However, in
when pre-training needs to be done repeatedly or reproduced for prudemic/indusshnn the majority of cases the pre-trained model can be downloaded or, in the worst case, trained once in a while. Contrary, these cascs, the budget required for adapting this model may become the main bottleneck.
Thus, we pay extra attention to the scenario, where the cost of obtaining a pre-trained model is free or effectively amortized by future adaptation runs. Instead, we concentrate on time and compute spent on finding good adaptation strategy (or on tuning from scratch training sctup), which we call "practitioner's cost".
A more extreme viewpoint is that the training cost is not crucial, and all that matters is eventual inference cost of the trained model, "deployment cost", which will amortize all other costs. This is especially true for large scale deployments, where a visual model is expected to be used a massive number of times. Overall, there are three major viewpoints on what considered to be the central cost of training a vision model. In this study we touch on all three of them, but mostly concentrate on "practitioner" and "deployment" costs.
3 Experimental setup
In this section we describe our unified experimental setup, which is used throughout the paper. We use a single JAX/Flax (19; 3) codebase for pre-training and transfer learning using TPUs. Inference speed measurements, however, were obtained on V100 GPUs (16G) using the timm PyTorch library (42). All datasets accessed through the TensorFlow Datasets library (15), which helps to ensure consistency and reproducibility. More details of our setup are provided below.
3.1 Datasets and metrics
For pre-training we use two large-scale image datasets: ILSVRC-2012 (ImageNet- 1k) and ImageNet-21k
ImageNet-21k datasct contains approximately 14 million images with about 21 000 distinct object categorics (11; 22; 30). ImageNet-1Ik is a subset of ImageNet-21k consisting of about 1.3 million training images and 1000 object categories. We make sure to de-duplicate images in ImageNet-2114 with respect to the test sets of the downstream tasks as described in (13; 22). Additionally, we used ImageNet V2 (29) for evaluation purposes.
For transfer learning evaluation we use 4 popular computer vision datasets from the VTAB benchmark (45): CIFAR-100 (24), Oxford IITT Pets (28) (or Pets37 for short), Resisc45 (6) and Kitti-distance (14). We selected these datasets to cover the standard setting of natural image classification (CIFAR-100 and Pets37), as well as classification of images captured by specialized equipment (Resisc45) and geometric tasks (Kitti-distance). In some cases we also use the full VTAB benchmark (19 datasets) to additionally ensure robustness of our findings.
For all datasets we report top-1 classification accuracy as our main metric. Saper-parameters for fine-tuning are selected by the result from the validation split, and final numbers are reported from the test split. Note that for ImageNet-11 1k we follow common practice of reporting our main results on the validation set. Thus, we set aside 1% of the training data into a minival split. that we use for model selection. Similarly, we use a minival split for CIFAR-100 (2% of training split) and Oxford HITT Pets (10% of training split). For Resisc45, we use only 60% of the training split for training, and another 20% for validation, and 20% for computing test metrics. Kitti-distance finally comes with an official validation and test split that we use for the intended purpose. See (45) for details about the VTAB dataset splits.
3

---

Published in Transactions on Machine Learning Research (05/2022)
Table 1: Configurations of VIT models.
Table 2: ResNet+ ViT hybrid models.
Model
Layers
Width
MLP
Heads
Params
Model
Resblocks
Patch-size
Params
ViT-Ti (39)
12
192
768
5.8M
R+
0
8
6.4M
VIT-S (39)
12
384
1536
22.2M
R26+
532
[2, 2,
2, 2
1
36.6M
ViT-B (13)
12
768
3072
86M
R504L/32
[3. 4,
6, 3
330.0M
VYT-L (13)
24
1024
4096
16
307M
3.2
Models
This study focuses mainly the Vision Transformer (ViT) (13). We use 4 different configurations from (13; 39): ViT-Ti, ViT-S, ViT-B and VIT-L, which span wide range of dilferent capacities. The details of each configuration are provided in Table 1. We use patch-size 16 for all models, and additionally patch-size 32 for the ViT-S and ViT-B variants. The only difference to the original VIT model (13) in our paper is that we drop the hidden layer the head, as empirically it does not lead to more accurate models and often results optimization instabilities: when pre-training on ImageNet-Tk we include both models with and without hidden layer, when pre-training on ImageNet-2lk we always drop the hidden layer.
In addition, we train hybrid models that first process images with ResNet (18) backbone and then feed the spatial output to ViT as the initial patch embeddings. We use a ResNet stem block (7 x 7 convolution + batch normalization + ReLU + max pooling) followed by a variable number of bottleneck blocks (18). We use the notation (Ti,S,L,/,) where n counts the number of convolutions, and denoles the patch-size in the input image for example R+Tii/1e reduces image dimensions by factor of two in the ResNet stem and then forms patches of size 8 as an input the ViT, which results in an effective patch-size of 16.
3.3 Regularization and data augmentations
To regularize our models we use robust regularization techniques widely adopted in the computer vision community. We apply dropout to intermediate activations of ViT as in (13). Morcover, we use the stochastic depth regularization technique (20) with linearly increasing probability of dropping layers.
For data augmentation, RandAugment (7). For Mixup, we vary its parameter a, where 0 corresponds to no Mixup. For RandAngment, we vary the magnitude parameter m, and the number of augmentation layers I. Note that we use the original Ranment implementation in TensorFlow, which differs from re-implementations found, for example, in timm (42).
rely the combination of two recent techniques, namely Mixup (47) and
We also try two values for weight decay (27) which we found to work well, since increasing AugReg may need decrease in weight decay (2).
Overall, our sweep contains 28 configurations, which is a cross-product of the following hyper-parameter choices:
Either no dropout and no stochastie depth (e-g. regularization) use dropout with probability 0.1 and stochastic depth with maximal layer dropping probability of 0.1, thus 2 configuration in total.
data augmentation setups for (I, m, a): none (0, 0), light1 (2, 0, light2 (2, 10, 0.2), mediuml (2, 15, 0.2), medium2 (2, 15, 0.5), strongl (2, 20, 0.5), strong2 (2, 22.)...88.
Weight decay: 0.1 or 0.03. The weight. decay is decoupled following (27), but multiplied by the learning-rate which peaks at 0.001.
3.4 Pre-training
We pre-trained the models with Adam (21), using 8 = 0.9 and 83 = 0.999, with batch size of 4096, and a cosine learning rate schedule with a lincar warmup (10k steps). To stabilize training, gradients were clipped global norm 1. The images pre-processed by Inception-style cropping (36) and random horizontal
4

---

Published in Transactions on Machine Learning Research (05/2022)
B/12, from scnech
B/32, tranittr Anglee
BF52, trunder
B/52, from scnach
BV52, trisfer
B/16, from scnach
B/16, AugRog
B016, trunsfer
8/16, from senrch
R/16, tnesfer
Pot37 (3312 images)
Resiso15 (31k imkgoss
Per5? (3312 imsaees)
Resise45 (31k images)
100%
305
90%
Accuracy
60%
60%
Bnn
10
40%
40%
10
0
20%
20%
00
10a
10a
10
10'
102
10"
10'
107
10"
Training time (TPUv2 core-hours]
Training time (TPUv2 core-hours]
Compute budget [TPUV2 coro-bours]
Computc budget core-horrs)
Figure 2: Left: When training small and mid-sized datasets from scratch it is very hard to achieve a test error that can trivially be attained by fine-tuning a model pre-trained on large dataset like ImageNet-21k. With our recommended models (Section 4.5), one can find a good solution with very few trials (bordered green dols, using recipe from B). Note that AugReg is not helpful when transferring pre-trained models (borderless green dots). Right: Same data as on the left side (ignoring the borderless green dots), but simulating the results of a random search. For a given compute budget (x-axis), choosing random configurations within that budget leads to varying final performance, depending on choice of hyper parameters (shaded area covers 90% from 1000 random samples, line corresponds to median).
flipping On the smaller ImageNNet-lk dataset we trained for 300 epochs, and for 30 and 300 epochs on the ImageNet-21k dataset. Since ImageNet--214 about 10x larger than maageNet-Ik, this allows us to examine the effects of the incrcased datasct size also with a roughly constant total compute used for pre-training.
3.5 Fine- tuneit
We fine-tune with SGD with a momentum of 0.9 (storing internal state as bfloat16), sweeping over 2-3 learning rates and 1-2 training durations per dataset as detailed in Table 4 in the appendix. We used fixed batch size of 512, gradient clipping at global norm and a cosine decay learning rate schedule with linear warmup. Fine-tuning was done both the original resolution (224), as well as at a higher resolution (384) described in (40).
4 Findings
4.1 Scaling datasets with AugReg and compute
One major finding of our study, which is depicted in Figure 1, is that by judicious use of image augmentations and model regularization, one can (pre-)train a model to similar accuracy as by increasing the dataset size by about an order of magnitude. More precisely, our best models trained on AugReg ImageNet-Ik, (31) perform about equal to the same models pre-trained on the 10x larger plain ImageNet-211, (11) dataset. Similarly, our best models trained on AugReg ImageNet-21k,, when compute is also increased (e.g. training run longer), match or outperform those from (13) which were trained on the plain JFT-300M (35) dataset with 25x more images. Thus, it is possible to match these privale results with a publicly available dalaset, and it is imaginable that training longer and with AugReg on JFT-300M might further increase performance.
Of course, these results cannot hold for arbitrarily small datasots. For instance, according to Table of (44), training ResNet50 on only 10% of ImageNet-lk with heavy data augmentation improves results, but does not recover training the full dataset.
4.2 Transfer is the better option
Here, we investigate whether, for reasonably-sized datasets a practitioner might encounter, it is advisable try training from scratch with AugReg, or whether time and money is better spent transferring pre-trained
5

---

Published in Transactions on Machine Learning Research (05/2022)
Natural datasets
Specialized datasets
Structured datasets
8835
R/32
8268
R500
8/16
1/16
832
R268
3/16
R300
1/16
S332
B/32
8268
$/15
R501
B/16
1/1
92%
80%
90%
90%
INe121k 300ep
30ep
88% INetlk 300ep
85%
10a
10"
10"
10"
Inference speed [img/sec]
Inference speed [img/scc]
Inference speed [img/sec]
Figure 3: Pretraining on more data yiclds more transfcrable models on average, tested on the VTAB suite (45) of 19 tasks across categories.
models that are freely available. The result is that, for most practical purposes, transferring pre-trained model is both more cost-efficient and leads to better results.
We perform a thorough search for a good training recipe? for both the small ViT-B/32 and the larger ViT-B/16 modols on two datasets practical size: Pet37 contains only about 3000 training images and is relatively similar the ImageNet-1k dataset. Resise45 contains about 30 000 training images and consists of very different modality of satellite images, which is not well covered by either ImageNet- 1k or ImageNet-21k: Figure 2 shows the result of this search.
The most striking finding is that, no matter how much training time is spent, for the tiny Pet37 dataset, it does not seem possible to train ViT models from scratch to reach accuracy anywhere near that of transferred models. Furthermore, since pre-trained models are freely available for download, the pre-training cost for a practitioner is effectively zero, only the compute spent on transfer matters, and thus transferring a pre-trained model is simullaneeossly significuntly chenper and gives beller resulls.
For the larger Resisc45 dataset, this result still holds, although spending two orders of magnitude more compute and performing heavy search may come close (but not reach) to the accuracy of pre-trained models.
Notably, this does not account for the "exploration cost", which is difficult to quantify. For the pre-trained models, we highlight those which performed best on the pre-training validation set and could be called recommended models (see Section 4.5). We can see that using a recommended model has a high likelihood of leading to good results just few attempts, while this is not the case for training from-scratch, as evidenced by the wide vertical spread of points.
4.3 More data yields more generic models
We investigate the impact of pre-training dataset size by transferring pre-trained models to unseen downstream tasks. We evaluate the pre-trained models on VTAB, including 19 diverse tasks (45).
Figure 3 shows the results three VTAB categories: natural, specialized and structured. The models are sorted by the inference time per step, thus the larger model the slower inference speed. We first compare two models using the same compute budget, with the only difference being the dataset size of ImageNet-1k (1.3M images) and ImageNet-214 (13M images). We pre-train for 300 epochs on ImageNet-lk, and 30 epochs on ImageNet-21k. Interestingly, the model pre-trained on ImageNet-21k is significantly better than the Image 1k one, across all the three VTAB calegories. This is in contrast with the validation performance on ImageNet-1k (Figure 6), where this difference does not appear so clearly.
As the compute budget keeps growing, obscrve consistent improvements ImageNet-211 dataset with 10x longer schedule. On a few almost solved lasks, e.g. flowers, the gain is small in absolute numbers. For
Not only da further increase available AugReg settings, but we also sweep over other generally important training hyperparameters: learning-rate, weight-deray, and training duration, as described Appendix

---

Published in Transactions on Machine Learning Research (05/2022)
ImageNet-Ik, 300ep
ImageNet-21K. Ik, 30ep
ImageNet-2Ik, 300ep
No regularizution
Regularization 0.1
No regularization
Regularization 0.1
No regularizaaion Regularizution 0.1
RT1-
as
83
26
23
TV/6
T2
89
12
27
26
23
23
37
26
36
S/32
71
76
14
14
10
12
72
21
11
64
35
35
36
37
34
93
52
51
90
29
3
42
41
40
41
40
50
56
36
36
S/16
71
77
83
82
80
80
76
19
90
79
99
77
79
40
30
42
40
39
58
57
35
35
2
45
45
45
45
45
43
45
42
40
B/52-1
48
74
15
15
89
14
77
76
77
77
43
40
42
40
40
38
36
35
36
-1
47
46
45
46
44
R265
72
74
78
79
50
90
90
75
76
11
92
92
81
8.
45
45
43
43
41
4
41
42
41
40
40
19
48
48
07
66
46
4
48
46
46
41
41
B/16-1
70
76
79
81
76
79
41
82
83
83
92
46
47
45
45
45
46
45
45
44
43
42
41
48
50
st
90
5o
30
47
49
41
I/16-1
76
77
78
76
75
74
78
78
76
79
77
77
45
48
47
4
48
48
40
45
45
49
40
St
51
51
46
48
31
31
51
51
51
RS0L-
TM
TM
78
79
48
48
57
nome
light
lightz
medi
med2.
heavy?
beary2-
1o00
light
lighe2
beary?
heavy2
19g- norog
nome
light
ligh12
med]
med2.
beavy]
beary2
toone
light
light2
mod]
med2.
heavy
beay2:
Teg noteg
Dome
light
laghi2.
medl
modd2
beavy
beavy2.
light
light2
medi
med22
beayy
beavy2
reg noteg
Figure 4: Validation accuracy (for ImageNet-1k: minival accuracy) when using varions amounts of augmenta- tion and regularization, highlighting differences to the unregularized, unaugmented setting. For relatively small amount of data, almost everything helps. However, when switching to imageNet-21k while keeping the training budget fixed, almost everything hurts; only when also increasing compute, does AugReg help again. The single column right of each plot show the difference between the best setting with regularization and the best setting without, highlighting that regularization typically hurts on ImageNett21k.
the rest of the tasks, the improvements are significant compared to the model pre-trained for short schedule. All the detailed results on VTAB could be found from supplementary section C.
Overall, we conclude that more dala yields more generic models, the trend holds across very diverse tasks. We recommend the design choice of using more data with a fixed compute budget.
4.4 Prefer augmentation to regularization
It not clear priori what the trade-offs are between data augmentation such as RandAugment and Mixup, and model regularization such as Dropout and StochasticDepth. In this section, we aim to discover general patterns for these that can be used as rules of thumb when applying Vision Transformers to new task. In Figure 4, we show the upstream validation score obtained for each individual setting, i.e. numbers are not comparable when changing dataset. The colour of a cell encodes its improvement or deterioration in score when compared to the unregularized, unaugmented setting, i.e. the leftmost column. Augmentation strength increases from left to right, and model "capacity" increases from top to bottom.
The first observation that bocomes visible, is that for the mid-sized ImageNet-1k 1k datasct, any kind of AugReg helps. However, when using the 10x larger TmageNet-21k dataset and keeping compute fixed, i.e. running for 30 epochs, any kind of AugReg hurts performance for all but the largest models. It is only when also incrcasing the computation budget to 300 epochs that AugReg helps more models, although even then, it continues hurting the smaller ones. Generally speaking, there are significantly more cases where adding augmentation helps, than where adding regularization helps. More specifically, the thin columns right of each map in Figure 4 shows, for any given model, its best regularized score minus its best unregularized score. This view, which is expanded in Figure 7 in the Appendix, tells us that when using ImageNet-211 regularization almost always hurts.
4.5 Choosing which pre-trained model to transfer
As show above, when pre-training ViT models, various regularization and data angmentation settings result in models with drastically different performance. Then, from the practitioner's 's point of view, a natural question emerges: how to select a model for further adaption for an end application? One way is to run downstream adaptation for all available pre-trained models and then select the best performing model, basod on the validation score on the downstream task of interest. This could be quite expensive in practice. Alternatively, one can select a single pre-trained model based on the upstream validation accuracy and then only use this model for adaptation, which is much cheaper.
7

---

Published in Transactions on Machine Learning Research (05/2022)
[N -k (V2
CIFAR 100
ImageNee-
Resise4
L/16 (best 100 validation score)
S/16 (hest 100 validation score)
Pot
Kini
R11
4.7
-04
-0.3
-13
+4.1
+0.0
TV/1
-02
-03
+4A
40.0
S/32
ImageNor-1k validatio
84%
84%
S/16
10.1
481
4.5
103
14.
113
82%
82%
-04
-00
+40
462
80%
80%
B/16
+0.3
40
+0.3
36
4.
156
78%
1/16
-33
-03
433
-.5
11.0
80%
85%
90%
95%
65%
68%
70%
72%
75%
78%
R50L
Minival accuracy
ImageNetV22 accuracy
Figure 5: Choosing best models. Left: Difference of fine-tuning test scores between models chosen by best. validation score pre-training data validation score fine-tuni tne-tuning data (negative values mean that sclecting models by pre-training validation deteriorates finc-tuning test metrics). Right: Correlation between "minival" validation score vs. ImageNetV2 validation score and official ImageNet- validation score (that serves as a test score in this study). Red circles highlight the best models by validation score, see Section 4.5 for an explanation.
In this section we analyze the trade-off between these two strategies. We compare them for a large collection of our pre-trained models on different datasets. Specifically, Figure 5 (left) we highlight the performance difference between the cheaper strategy of adapting only the best pre-trained model and the more expensive strategy of adapting all pre-trained models (and then selecting the best).
The results are mixed, but generally reflect that the cheaper strategy works equally well the expensive strategy in the majority of scenarios. Nevertheless, there are few notable outliers, when it is beneficial to adapt all models. Thus, we conclude that selecting a single pre-trained model based on the upstream score is a cost-offective practical strategy and also use it throughout our paper. However, we also stress that if extra compute resources are available, then certain cases one can further improve adaplation performance by fine-tuning additional pre-trained models.
A note on validation data for the ImageNet-1k dataset. While performing the above analysis, we observed subtle, but severe issue with models pre-trained ImageNet-21k and transferred to ImageNet-1k datasot. The validation score for these models (especially for large models) is not well correlated with observed test performance, see Figure 5 (right). This is due to the fact that ImageNet-21k data contains ImageNet- 1k training data and we use a "minival" split from the training data for evaluation (see Section 3.1). As a result, large models on long training schedules memorize the data from the training set, which biases the evaluation metric computed in the "minival" evaluation set. To address this issuc and enable fair hyper-parameter selection, we instead use the independently collected ImageNet V2 data (29) the validation split for transferring to ImageNet-lk: As shown in Figure 5 (right), this resolves the issue. We did not observe similar issues for the other datasets. We recommend that rescorchers transferring ImageNet-21k models to ImogeNet- 1k follow this strategy.
4.6 Prefer increasing patch- to shrinking model-size
One unxxpectd outcome of our study is that we trained several models that are roughly equal in terms of inference throughput, but vary widely terms of their quality. Specifically, Figure (right) shows that models containing the "Tiny" variants perform significantly worse than the similarly fast larger models with "/32" patch-size. For given resolution, the patch-size influences the amount of tokens which self-attention is
8

---

Published in Transactions on Machine Learning Research (05/2022)
90.0%
ilk
80
S/32
12 21k_3 30
Imagenel validation accuracy
TW16
85.0%
i21k
ImageNet top-1 accuracy [%]
75
S/32
R+TV16
TV16
70
80.0%
R+TV16
IntgeNet-211
Throughpul [img/sec]
75.0%
VIT-S,B.L. 624
ViT-S,8.L 384
D/32
hybrid @ 224
70.0%
hybrid 384
TW16
B/32
Ti/16 6224
Ti/16 @ 384
S/32
65.0%
TV16
3000
1000
500
300
100
50
30
R+TV/16
ImageNet-Ik
Images/sec/core
65
10000
3000
1000
300
Figure 6: ImageNet transfer. Left: For every architecture and upstream dataset, we selected the best model by upstream validation accuracy, Main ViT-S,B,L models are connected with a solid line to highlight the trend, with the exception of ViT-L models pre-trained ilk, where the trend breaks down. The same data is also shown in Table 3. Right: Focusing small models, it is evident that using larger patch-size (/32) significantly outperforms making the model thinner (Ti).
Table 3: ImageNet-Ik transfer. Column ilkup evaluates best checkpoint without adaptation, columns ilks00. i21k30 and i21kg00 (ImageNet-Ik 300 epochs and TmageNet-21k 30 and 300 epochs) report numbers after fine-tuning, which are shown in Figure the "recommended checkpoints" (see Section 4.5) were fine-tuned with two different learning rates (see Section B). For the column i21kv2 (ImageNet-211k; 300 epochs), the upstream checkpoint was instead chosen by ImageNet validation accuracy. The JFT-300M numbers are taken from (13) (bold numbers indicate our results that are on par or surpass the published JFT-300M results without AugReg for the same models). Inference speed measurements were computed on an NVIDIA V100 GPU using timm (42), sweeping the batch size for best throughput.
224px resolution
384px resolution
Model
img/sec
ilkup
ilka00
k21kau
i2 k20u img/sec
il K300
i21k30
121k300
121k
JFT300M
L/16
228
75.72
74.01
82.05
83.98
50
77.21
84.48
85.59
87.08
87.12
B/16
659
79.84
78.73
80.42
83.96
138
81.63
83.46
85.49
86.15
84.15
S/16
1508
79.00
77.51
76.04
80.46
300
80.70
80.22
83.73
83.15
R50+L/32
1047
76.84
74.17
80.26
82.74
327
76.71
83.19
85.99
86.21
R26+S/32
1814
79.61
78.20
77.42
80.81
560
81.55
81.11
83.85
83.80
Ti/16
3097
72.59
69.56
68.89
73.75
610
74.64
74.20
78.22
77.83
B/32
3597
74.42
71.38
72.24
79.13
955
76.60
78.65
83.59
83.59
80.73
8/32
8342
72.07
69.19
68.49
73.47
2154
75.65
75.74
79.58
80.01
R+Ti/16
9371
70.13
67.30
65.65
69.69
2426
73.48
71.97
75.40
75.33

---

Published in Transactions on Machine Learning Research (05/2022)
performed and, thus, is contributor to model capacity which is not reflected by parameter count. Parameter count is reflective neither of speed, nor of capacity (10).
5 Related work
The scope of this paper is limited to studying pre-training and transfer learning of Vision Transformer models and there already are number of studics considering similar questions for convolutional neural networks (23; 22). Here we hence focus on related work involving ViT models.
As first proposed in (13), VIT achieved competitive performance only when trained on comparatively large amounts of training data, with tf-to-of-tho-rt transfer results using the ImageNet-21k: and JFT-300M datasets, with roughly 13M and 300M images, respectively. In stark contrast, (39) focused on tackling overfitting of VIT when training from scratch ImageNet- 1k by designing strong regularization and augmentation schemes. Yet neither work analyzed the effects of stronger augmentation of regularization and augmentation in the presence of larger amounts of training data.
Ever since (22) first showed good results when pre-training BiT on ImageNet-21k, more architecture works have mentioned using for select few experiments (13; 38; 37; 8), with (30) arguing more directly for the use of ImageNet-211: However, none of these works thoroughly investigates the combined use of AugReg and ImageNet-21k and provides conclusions, we do here.
An orthogonal line of work introduces cleverly designed inductive biases in VIT variants retain some of the general architectural parameters of successful convolutional architectures while adding self-attention to them. (33) carefully combines a standard convolutional backbone with bottleneck blocks based on self-attention instead of convolutions. In (26; 17; 41; 43) the authors propose hierarchical versions ViT. (9) suggests a very elegant. idea. of initializing Vision Transformer, such that it behaves similarly convolutional neural network in the beginning of training.
Yet another way to address overfitting and improve transfer performance is to rely self-supervised learning objectives. (1) pre-trains VIT to reconstruct perturbed image patches. Alternatively, (4) devises self- supervised training procedure based on the idea from (16), achieving impressive results. We leave the systematic comparison of self-supervised and supervised pre-training to future work.
6 Discussion
Societal Impact. Our experimental study is relatively thorough and used a lot of compute. This could be taken as encouraging anyone who uses ViTs to perform such large studies. On the contrary, our aim is to provide good starting points and off-the-shelf checkpoints that remove the need for such extensive search in future work.
Limitations. In order to be thorough, we restrict the study to the default VIT architecture and neither include ResNets, which have been well studied over the course of the past years, nor more recent ViT variants. We anticipate though that many of our findings extend to other VrT-based architectures as well.
10

---

Published in Transactions on Machine Learning Research (05/2022)
7
Summary of recommendations
Below we summarize three main recommendations based on our study:
We recommend to use checkpoints that were pre-trained on more upstream data, and not relying only on ImageNet-lk as a proxy for model quality, since ImageNet-lk validation accuracy is inflated when pre-training on ImageNet-Ik, and more varied upstream data yields more widely applicable models (Figure and Section 4.3).
Judiciously applying data augmentation and model regularization makes it possible to train much better models on a dataset of a given size (Figure 1), and these improvements can be observed both with medium sized datasets like ImageNet-1k, and even with large datasets like ImageNet-211 there are no simple rules which AugReg settings to select. The best settings vary a lot depending on model capacity and training schedule, and one needs to be careful not apply AugReg to model that is too small, or when pre-training for too short otherwise the model quality may deteriorate (see Figure 4 for an exhaustive quantitative evaluation and Section 4.4 for further comments on regularization vs augmentations).
But
How to select the best upstream model for transfer on your own lask? Aside from always using ImageNet-21k checkpoints, we recommend to select the model with the best upstream validation performance (Section 4.5, table with paths in our Github repository"). As we show in Figure 5, this choice generally optimal for a wide range of tasks. If the user has additional computational resources available to fine-tune all checkpoints, they may get slightly better results in some scenarios, but also need be careful with respect to ImageNet-Ik and ImageNet-21k data overlap when it comes to model selection (Figure 5, right).
8 Conclusion
We conduct the first systematic, large scale study of the interplay between rogularization, data augmentation, model size, and training data size when pre-training Vision Transformers, including their respective effects on the compute budget needed achieve a certain level of performance. We also evaluate pre-trained models through the lens of transfer learning. As a result, we characterize a quite complex landscape of training settings for pre-training Vision Transformers across different model sizes. Our experiments yield number of surprising insights around the impact of various techniques and the situations when augmentation and regularization are beneficial and when not.
We also perform an in-depth analysis of the transfer learning setting for Vision Transformers. We conclude that across a wide range of datasets, even if the rownstream data of interest appears to only be weakly related to the data used for pre-training, transfer learning remains the best available option. Our analysis also suggests that among similarly performing pre-trained models, for transfer learning a model with more training data should likely be preferred over one with more data augmentation.
We hope that our study will help guide future research on Vision Transformers and will be a useful source of effective training settings for practitioners seeking to optimize their final model performance in the light of a given computational budget.
Acknowledgements
We thank Alexey Dosovitskiy, Neil Houlsby, and Ting Chen for insightful feedback; the Google Brain team large for providing a supportive research environment.
References
[1] Sara Atito, Muhammad Awais, and Josef Kittler. Sit: Self-supervised vision transformer. orXiv:2104.03602 2021.
10
22 Irwan Bello, William Fedus, Xianzhi Du, Ekin D. Cubuk, Aravind Srinivas, Tsung- Yi Lin, Jonathon Shlens, and
Barret Zoph. Revisiting resnels: Improved training and scaling strategies. arXiv:2103.0757% 2021. 4
Chttps://githii http:///gttuub.ooggggleeeeee wtwa://gggge-eeeree/eeieeeee http://viiiioorraaffoeeeee transformer
11

---

Published in Transactions on Machine Learning Research (05/2022)
[3] James Bradibury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wauderman-Millne, and Qiao Zhang. JAX: composable transforma- lions of Python+Nummly progrums, 2018. 3
[4] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. or 1rx:22104. 14294, 2021. 10
[5] Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12M: Pushing web-scale image-Lext
pre-training to recognize long-tail visual concepts. 2021. 16
[6] Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Benchmark and state of the art. Proceedings of the TEFE, (05(10):1865-188)
2017. 3
7 Ekin D. Cubuk, Barret Zoph, Jonathon Shiens, and Quoc V. Le. RandAugment: Practical automated data
augmentatiin with reduced search space. In CVPR Workshops, 2020. 4
8 Zihang Dai, Hanxiao Liu, Quoe V. Le, and Mingxing Tan. Coatnet: Marrying convolution and attention for all
data sizes, 2021. 10
9 Stéphane d' A'Acolli, Hugo Touvron, Matthew Leavitt, Ari Morcos, Giulio Biroli, and Levent Sagun. Convit: Improving vision transformers with soft convolutional inductive biases. arXiv:2103. 10667 2021. 10
[10] Mostafa. Dehghani, Anurag Arnab, Lucas Beyer, Ashish Vaswani, and Yi Tay. The efficiency misnomer. CoRR,
abs/2110. 12899,,,,,9, 2021. 10
[1]] J. Deng, W. Dong, R. Socher, L.. Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database.
In CVPR, 2009. 3, 5
[2] Jacob Devlin, Ming-Wa: Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional
transformers for language understanding. In NAACI-HLT,, 2019. 16
[13] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An
image worth 16x16 words: Transformers for image recognition at scale. arXiv:2010. 11929,, 2020. 1, 3, 4, 5, 9, 10 [14] A Geiger, P Lenz, C Stiller, and R Urtasun. Vision meets robotics: The kitti dataset. The International Journal
of Robotics Research, 12(111:12111237,, 2013. 3
[55] Google. TensorFlow Datasets, a collection of ready-to-use datasets. https://wwwt wwwsorroootoooooooooooooooooooor tow.eeoor......r/ttttasssss 3
[66] Bean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your latent: A new approach to self-supervised learning. ar Xiv:22006. 07783, 2020, 10
[17] Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, and Yunhe Wang. Transformer in transformer.
arXiv:2103.0013 2021. 10
[18] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016. 1, 4
[19] Jonathan Heek, Anselm Levskaya, Avital Oliver, Marvin Ritter, Bertrand Rondepierre, Andreas Steiner, and Marc van Zee. Flaz: A nearal nelwork library and ecosystem for JAX, 2020. 3
[20] Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth.
In ECCV, 2016. 4
21 Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic oplimization. In ICLR, 2015. 4
222] Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby.
Big transfer (BiT): General visual representation learning. In ECCV, 2020. 3, 10
[23)
Simon Kornblith, Jonathon Shlens, and Quoc V Le. Do better imagenet models transfer better? In CVPR, 2019.
224] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[25] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. In Peter L. Bartlett, Fernando C. Pereira, Christopher J. C. Burges, Léou Bottou, and Kilian Q. Weinberger, editors, Advances in Newral Information Processing Systems 25: 26th Annual Conference on Newral Information Processing Systems 2012. Proceedings of a meeting held December 3-6, 2012, Lake Tahoe, Nevada,
United States, pages 1106- 1114, 2012. 2
[26] Ze Låu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. arXim:2103. 100.:000,,,,00 2021. 10
(27] Tlya Loshchilov and Frank Hutter. Decoupled weight decay regularization. Tn International Conference
Learning Representations, 2019. 1, 14
28 Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In CVPR, 2012. 3
29 Beujamin Recht, Rebecca Roclofs, Ludwig Schmidtt, and Vaishaal Shankar. Do imagenet classifiers generalize to
imagenet? In JCML, 2019. 3,
330] Tal Ridnik, Emanuel Ben-Baruch, Asaf Noy, and Lihi Manor. Imagenet-21k pretraining for the masses.
Xrx:::200 200. 10972, 2021. 3, 10
[31] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Kranse, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. IIJCV,
2015. 5
12

---

Published in Transactions on Machine Learning Research (05/2022)
[32] Ali Sharif Razavian, Hossein Azizpour, Josephine Sullivan, and Stefan Carlsson. Cun features off-the-shelf: An ustounding baseline for recognition. In Proceedings of the IEEE Conference on Compuler Vision and Pallern Recogmition (CVPR) Workshops, June 2014.
[33] Aravind Srinivas, Tsung- Yi Lin, Niki Parmar, Jonathon Shlens, Pieter Abbeel, and Ashish Vaswani, Bottleneck transformers for visual recognition. arXi:: 201.1 1605, 2021. 10
[34] Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid. Segmenter: Transformer for semantic segmentation. 2115::2105.
2021. 16
[35] Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta, Revisiting unreasonable effectiveness of data in deep learning era. In Proceedings of the IEEE international conference on computer vision, pages 843-8% -922,, 2017. 5
366 C. Szegedy, W. Lin, Y. Jia, Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. In CVPR, 2015. Mingxing Tan and Quoe V. Le. Efficientnetv2: Smaller models and faster training. CoRR, abs 2hs/2000000298, 2021.
[37]
[88] Tlya O. Tolstikhin, Neil Houlshy, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, Mario Lucic, and Alexey Dosovitskiy. Mlp-mixer: An all-mip architecture for vision. CoRR, bbs/2205.0160) 2021. 10
399 Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. arXiv:2012. 12877, 2020. 1, 4, 10 [40] Hugo Touvron, Andrea Vedaldi, Matthijs Douze, and Herve Jegou. Fixing the train-test resolution discrepancy. NeurIPS, 2019.
41] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Ling-iing&& Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. 1rXiv:2102.11190
2021. 10
[42] Ross Wightman. Pytorch image models (timm): Vit training details.
pytorch-image-mon httla//ssseee///eeeeerooooo
2013. 3, 4, 9
[43] Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang, Cvt: Introducing convolutions vision transforuers. arXiv:2103. 15808, 2021. 10
[44] Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoc Le. Unsupervised data augmentation for consistency training. In H. Larochelle, M. Ranzalo, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neurul Information Processing Systems, 2020. 5
[45] Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, Lucas Beyer, Olivier Bachem, Michael Tschannen, Marcin Michalski, Olivier Bousquet, Sylvaín Gelly, and Neil Houlsby. A large-scale study of representation learning with the visual task adaptation benchmark, 2020. 2, 6,
[46] Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Tucas
Beyer. Lit: Zero-shot transfer with locked-image text tuning. CVPR, 2022. 16 [17] Hongyi Zhang, Moustapha Cisse, Yann N Danphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In ICLR, 2018. 4
13

---

Published in Transactions on Machine Learning Research (05/2022)
A From-scratch training details
We present from-scratch training details for B/32 and B/16 models, on both Resisc45 and Pets37 datascts. We perform a grid search over the following parameters:
B/32 on Pets37
Epochs: {Ik, 3k, 10k, 30k, 300k)
Learning rates: {1e-4, 2e le- 3e-33
Weight decays": {le-5, 3e 5, le- 3e-4)
B/16 on Pets37
Epochs: {1k, 3k, 10k]
Learning rales: (3e- 4, le-3)
Weight decays: (3e-5, le-4]
B/32 on Resisc45
Epochs:
275,250, 750, 2.5k, 7.5k, 25k}
Learning rates: 3e-4, 1, 3--3)
Weight decays: {le-5, 3e-5, 1e-4, 3e-44
B/16 Resisc45
Epochs: {75, 250, 750, 2.5k, 7.5k)
Learning rates: {le-3}
Weight decays: 1e-44,e-44)
All these {(0.0, 0.0), (0.1,0.1), (0.2, 0.2)}. and data augmentation (1, m, a) in range (0,0,0), (2, 0.2), (2, 15,0.2), (2, 15, 0.5), (2, 20, 0.5), (2, 20, 0.8), 15, 0.5), (4, 20, 0.8) }.
from-scratch runs sweep over dropout rate and stochastic depth in range
For the definition of (1, m, a) refer to Section 3.3
B Finetune details
In Table 4, we show the hyperparaneter sweep range for finetune jobs. We use the same finetune sweep for all the pre-trained models in this paper.
Table 4: Finetune details for the pre-trained models.
Dataset
Learning rate
Total, warmup steps
ImageNet-1k
(0.01, 0.03)
{[20k, 500))
Pets37
(1e-3, 0.01)
{(500, 100), (2.5k,
Kitti-distance
(1e-3, 3e-3, 0.01)
{5500, 100), (2.5k, 200)
CIFAR-100
(1e-3,3e-3, 0.01)
(2.5k, 200), (10k, 500)
Resisc45
1le-3,3e3,,0. 0.01)
([2.5k, (10k,
As opposed 3.3 where specify weight docay values typically defined in common frameworks, here the values "decoupled" following (27) that is better suited sweeps; multiplying weight decay by the base learning-rate the "coupled" value used elsewhere.
14

---

Published in Transactions on Machine Learning Research (05/2022)
Table 5: Detailed VTAB results, including the "Mean" accuracy shown in Figure 3. We show datasets under
NATURAL, SPECIALIZED, STRUCTURED groups, following (45).
Retinopathy
Clevv- Count
SNORR- Azzim
SYOBB Flev
Caltech101
CIFAR-
Flowersl0r
Sun397
Camelyon
EuroSAT
Resised5
Clevi- Dist
DMLab
dSpr-Loc
dSpr-Ori
KKI KITTIEDE
DTD
Pots
SSVSS
Mean
Mean
Mean
R+/17/16
91.6
81.9
68.0
94.0
91.9
70.6
95.6
84.8
85.2
4
94.8
80.4
89.7
96.1
89.8
67.4
99.9
86.9
81.9
25.1
46.3
74.2
ImageNet-11 (300ep)
S/32
92.7
86.4
70.7
93.6
91.2
72.9
95.8
86.2
83.6
98.6
95.5
79.6
89.3
94.2
88.4
65.8
99.9
86.1
80.7
24.9
68.2
76.0
B/32
92.6
87.6
72.7
92.2
73.8
95.8
87.0
82.7
94.9
79.8
89.0
94.0
89.6
66.1
99.8
84.7
80.3
24.7
62.4
75.2
11/16
92.7
84.0
68.9
93.8
92.5
72.0
96.1
85.7
83.7
98.7
95.6
81.6
89.9
98.0
91.9
68.5
99.7
83.2
82.0
26.5
65.9
77.0
R26 S/33
90.2
86.2
74.0
95.5
94.3
74.5
95.6
87.2
84.5
98.6
96.0
83.4
90.6
99.7
91.6
73.3
100
84.8
84.5
28.2
51.3
76.7
S/16
93.1
86.9
72.8
95.7
93.8
74.3
96.2
87.5
84.1
98.7
95.9
82.7
90.3
98.7
91.5
69.8
100
84.3
79.6
27.3
58.0
76.1
R50
L/32
90.7
88.1
73.7
95.4
93.5
75.6
95.9
87.6
85.8
98.4
95.4
83.1
90.7
99.8
90.4
71.1
100
87.5
82.4
23.5
53.0
76.0
B/16
93.0
87.8
72.4
96.0
94.5
75.3
96.1
87.9
85.1
98.9
95.7
82.5
90.5
98.1
91.8
69.5
99.9
84.5
84.0
25.9
53.9
76.0
L/16
91.0
86.2
69.5
91.4
93.0
75.3
94.9
85.9
81.0
98.7
93.8
81.6
88.8
94.3
88.3
63.9
98.5
85.1
81.3
25.3
51.2
73.5
R+T1/16
92.4
82.7
69.5
88.0
95.1
83.6
94.9
89.5
95.7
66.6
80.3
47.0
73.9
ImageNet-213 (30ep)
S/32
92.7
88.5
72.4
98.9
90.5
75.4
95.4
87.7
83.5
98.7
95.0
79.5
89.2
94.5
89.8
64.4
99.8
87.9
81.2
24.9
57.7
75.0
B/32
90.5
74.5
99.1
91.9
95.7
83.5
95.1
89.1
93.6
62.9
99.8
78.3
55.9
74.2
TI/16
93.3
85.5
72.6
99.0
90.0
74.3
95.1
87.1
85.5
98.8
95.5
81.6
90.4
97.7 91.7
67.4
99.9
83.8
81.2
26.3
55.1
75.4
R26+S/32
89.9
93.0
95.9
86.3
96.1
99.7 92.0
84.8
26.2
53.3
S/16
94.3
89.4
76.2
99.3
92.3
78.1
95.7
89.3
84.5
98.8
96.3
81.7
90.3
98.4 91.5
68.3
100
86.5
82.8
25.9
52.7
75.8
R50+L./32
94.3
81.7
96.0
85.9
95.9
99.9
72.9
82.6
25.4
57.4
76.9
B/16
95.1
91.6
77.9
99.6
94.2
80.9
96.3
90.8
84.8
99.0
96.1
82.4
90.6
98.9
90.9
72.1
100
88.3
83.5
26.6
69.6
78.7
L/16
95.7
93.4
79.5
94.6
96.7
96.5
81.8
99.3
72.1
83.7
25.0
62.9
77.9
TT/16
85.3
71.5
09.0
90.3
74.7
95.2
85.2
95.3
81.3
90.0
95.5
90.5
67.4
99.9
78.2
45.2
73.6
ImageNet-221 (300ep)
8/32
93.2
89.7
75.3
99.2
92.0
78.1
96.1
89.1
84.0
98.5
95.4
80.6
89.6
96.9
88.7
68.1
100
91.0
79.6
26.2
55.0
75.7
B/32
92.3
77.2
99.5
92.8
81.2
96.6
90.7
87.0
96.0
90.8
97.7
70.5
92.3
82.7
83.1
80.2
Ti/16
93.7
87.2
73.1
99.2
91.0
77.3
95.7
88.2
86.0
98.5
95.8
81.9
90.6
98.3
89.7
70.8
100
86.0
82.6
26.8
49.9
75.5
R26+8/32
90.9
96.7
87.5
96.4
84.2
91.7
99.9
92.4
77.0
83.4
28.6
S/16
95.2
90.8
77.8
99.6
93.2
80.6
96.6
90.5
86.7
98.8
96.4
82.9
91.2
99.1
89.8
73.9
100
87.6
85.1
26.8
61.1
77.9
R50+L/32
95.7
93.9
81.6
99.5
94.9
83.6
97.1
92.3
85.8
98.7
96.7
84.2
91.3
100
92.0
76.8
100
87.2
85.2
26.8
61.8
78.7
B/16
96.0
93.2
79.1
99.6
94.7
83.0
97.0
91.8
87.4
98.7
96.8
83.5
91.6
99.7
89.0
76.0
100
86.7
85.7
28.3
68.2
79.2
L/16
95.5
94.1
80.3
99.6
95.0
83.4
97.4
92.2
86.4
99.0
96.6
83.3
91.3
99.8
91.7
75.6
90.4
84.7
27.5
76.5
80.8
C VTAB results
In Table we show all the results in percentage for all the models on the full VTAB. We report VTAB score only for the best pre-trained models, selected by their upstream validation accuracy (*recommended checkpoints", see Section 4.5). For VTAB tasks, sweep 8 hyper parameters, include four learning rates (0.001, 0.003, 0.01, 0.03) and two schedules 2500) steps. The best run was selected on VTAB validation split.
15

---

Published in Transactions on Machine Learning Research (05/2022)
ImageNet Ik, 300ep
ImageNet 21k, 30ep
ImageNet-22 21k, 300ep
R+T1/16
TM/6
10
S/32
B/32
2
2
3
0
2
3
R264S/32-
B/16
2
-1
3
4
4
-1
2
L/16
R50+L/32
0
1
2
9
Done
light
light2
mediuml
medium2
heavyl
heavyy
none
light
light2
mediuml
mediam2
heavyl
heavy2
none
light
light2
mediuml
mediam2
heavy1
leavyy2
Figure The improvement or deterioration in validation accuracy when using using regularization
(e.g. dropout and stochastic depth) positive values when regularization improves accuracy for given model/augmentation For absolute values Figure 4.
D The benefit and harm of regularization
In Figure 7, we show the gain (green, positive numbers) or loss (red, negative numbers) in accuracy when adding regularization to the model by means dropout and stochastic depth. We did verify in earlier experiments that combining both with (peak) drop probability 0.1 indeed the best setting. What this shows, is that model regularization mainly helps larger models, and only when trained for long. Specifically, for ImageNet-21 pre-training, hurts all but the largest. of models across the board.
E
Using recommended checkpoints for other computer vision tasks
One limitation of our study is that it focuses mainly the classification task. However, computer vision is much broader field, and backboncs need to excel at many tasks. While expanding the full study to many more tasks such as detection, segmentation, tracking, and others would be prohibitive, here we take a peek at one further task: multi-modal image-text retrieval.
A detailed analysis this question beyond the scope of this study, but we evaluated our recommended contrastive training setup with a locked image tower (46). We initialize the text tower with BERT-Bas Base (12) checkpoint and train for 20 epochs on CC12M (5). The results in Table 6 indicate that the upstream validation accuracy is a good predictor for zero-shot classification. Moreover, the representations produced by such a model yield similarly better results for image-text retrieval, when compared to models that do not have the ideal amount of AugReg applied. We hope the community will adopt our backbones for other tasks, as already done by (34).
(see Section 4.5) B/32 checkpoint pre-trained on ImageNet-222
Table 6: Comparing our recommended (see Section 4.5) B/32 checkpoint with models that apply too little too much AugReg. The final validation accuracy from the ImageNet-21k pre-training is the same that is reported in Figure 4. The other columns are ImageNet-1K zero-shot accuracy, and image-text retrieval accuracy on different datasets, after contrastively training as described in (46).
AugReg
121k Val
T1k Oshot.
Coco 12T
Coco T2I
Flickr 12T
Flickr T2T
none/ none/0.0 /0.0
41.6
54.9
33.4
20.1
58.1
39.9
hcavy2/0.1
43.5
57.3
39.1
24.4
62.1
44.6
Recommended
47.7
60.6
41.1
25.5
65.9
46.9
16


Reviews:
[{"review_id": "H8bSvkWXBn", "reviewer": "Reviewer_Y8hX", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The authors provide a consistent setup and 50,000 ViT models with different training recipes, which could be a valuable model set for model analysis. \n- The paper is organized and well written. It’s not easy to organize and present so many experiments intuitively. The well-designed figures make it easy to comprehend the corresponding findings.\n- The ViTs have become more and more popular. Thus the findings in the paper could be useful to researchers and practitioners working on ViTs, and provide insights into future research.\n\nWeaknesses:\n- All the experiments are conducted in supervised image classification. However, the success and popularity of ViTs are beyond it. Do the findings still hold for other vision tasks, e.g., object detection, and other training paradigms, e.g., self-supervised learning.\n- I’m not surprised by the findings in this paper. Most of them are natural and straightforward, or commonly held beliefs. For example, either using more data or using appropriate image AugReg can improve accuracy, pre-training is better than training from scratch, and more pre-training data achieves better downstream results.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper conducts extensive experiments, e.g., training 50,000 models under different settings, to investigate the effects of different training recipes and their combinations for training Vision Transformers.  Specific aspects include regularization, augmentation, model size, and training data size. The authors draw a few conclusions from the aspects of (pre-)training and transfer learning in a strictly controlled setting.\nSome conclusions are:\n1) the gains from increasing dataset size can be alternatively obtained by carefully selecting image regularization and augmentations. \n2) transferring a pre-trained ViT is more efficient and effective compared to training from scratch.\n3) using more pre-training data leads to better results on unseen downstream tasks.\n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "The authors are suggested to add some discussion on other vision tasks or training paradigms as described above. Overall I recommend accepting this paper according to the policy of TMLR, i.e., the claims are convincing and clearly supported by experiments, and the practitioners working on ViTs could be interested in the findings.", "broader_impact_concerns": "The broader impact concerns are well addressed in Section 6.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ayBCSo9DdD", "reviewer": "Reviewer_Z7Wx", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- This paper presents some meaningful conclusions, which may inspire the community. These findings answer some important questions, whether transfer learning always helps, whether more data enhance the generalization ability, etc. These findings can help researchers/engineers implement a deep learning model to solve practical problems.\n\n- The authors conduct extensive experiments to obtain solid conclusions. For example, the authors investigate 28 configurations of training strategies with different weight decay, data augmentation, and regularization. All these models have been trained adequately with a large lot of computing resources, making the conclusions reliable.\n\n- The experimental results are impressive. With proper training strategy, the model trained on ImageNet-21k can similar performance to those trained on JFT-300M, though there is a large gap between the numbers of the training dataset.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper investigates the training strategies of vision transformers systematically, especially the impact of training data, augmentation, and regularization. There are extensive experiments to get solid findings on vision transformers.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "- The authors conduct most of the experiments on the standard ViT [1]. Could the findings in this paper generalize to other variants of vision transformers?\n\n- Obtaining the conclusions in these papers usually consumes extremely many computing resources, which is usually unaffordable in most of the research institutions. Could toy experiments with fewer training epochs obtain consistent conclusions?\n\n[1] An image is worth 16x16 words: Transformers for image recognition at scale. ICLR 2021.\n\nOverall, I recommend accepting with minor revision.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "XnMZRS6IRt", "reviewer": "Reviewer_bnNv", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The main strength of this work is that it provides a large set of apples-to-apples comparisons, from which people can either take the authors' conclusions or draw their own. The experimental setup appears to be clean and simple, and the results are presented clearly.\n\nA weakness here is that the paper does not use the conclusions to create something entirely new. For example, I might expect that the authors implementing and evaluating all this might walk away with a feeling that they can design a better transformer, a better stem, or a more general or safer strategy for transfer. There are helpful takeaways in the individual sections, but I think I would appreciate something more global. For example, even providing a summary of the optimal choices across the sweep, summarizing the analysis across the many experiments, would be useful. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents evaluations from a hyperparameter sweep on ViT and Resnet+ViT architectures, and evaluates the results to make some conclusions on what matters for downstream performance. \n\nThey consider training on ImageNet and ImageNet-21k data, with and without dropout and stochastic depth, with varying amounts of augmentation, and varying amounts of weight decay.\n\nThey arrive at a few interesting results. One that jumps out at me is: a model trained with data augmentation performs similar to a model trained without augmentations on 10x data. Another result I like is: there is clearly a huge benefit to transferring models learned on larger datasets (which is maybe obvious), but there is still substantial variance: many transferred models are worse than models trained from scratch. Another interesting result is that regularization hurts the smaller models, and improves the larger ones, whereas augmentation helps them all. \n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "I would like to see a new section, with a title like \"Summary of recommendations\". Even if these have some caveats, that's OK. I want to know what the authors learned. ", "broader_impact_concerns": "All good", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "WnEZiFf6JW", "reviewer": "Reviewer_H8zE", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\nStrengths:\n\n(1)\tThe experiments are pretty comprehensive and well-designed.\n\n(2)\tThe authors release more than 50 000 ViT models trained under diverse settings on various datasets for further application or research.\n\nWeaknesses:\n\n(1)\tThe experimental conclusions of this paper seem to be trivial and widely-known, e.g., strong AugRegs helps, large-scale pre-training helps, and fine-tuning is beneficial. Even though in the context of ViT, many recent papers have pointed these out, at least implicitly. Systemically studying these important issues is an acceptable contribution for me. However, I do believe that the authors should comment on this weakness.\n\n(2)\tThe study is limited to the recognition task and the plain ViT networks. Considering more downstream tasks (e.g., objection detection and semantic segmentation) or the recently proposed pyramid-style ViTs (e.g. swin Transformer and PVT) may make the experiments more strong.\n\n(3)\tIn Figure 5, if I do not misunderstand, adapting all pre-trained models should always achieve the best performance, since it traverses every pre-trained model. Why there are both positive and negative values in the left Figure?\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a comprehensive empirical study on the pre-training and fine-tuning of Vision Transformers (ViT). The aim is to understand how the size of training data, augmentation/regularization techniques, and fine-tuning policies affect the performance of ViT in practical image recognition tasks. The major conclusions include:\n\n(1)\tStrong data augmentation and regularization techniques are beneficial, and can alleviate the problem of insufficient pre-training data.\n\n(2)\tFor many downstream tasks, transferring existing pre-trained models leads to better performance and less computational cost (for training).\n\n(3)\tAs an efficient model selection policy, one can simply adopt the best pre-trained model on upstream datasets.\n\n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "The authors may consider revising this paper according to the listed weaknesses.", "broader_impact_concerns": "The authors have summarized the societal impacts and limitations of their work. I do not have some particular concerns on the broader impacts.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper investigates the size of data, augmentation, and regularization techniques in training vision transformers comprehensively. All the reviewers agree that this paper provide useful knowledge and contributions. The main concerns raised by the reviewers is that more experiments and discussions should be provided.  

During the rebuttal period, the authors address these concerns by performing new experiments on multi-modal image-text retrieval and adding the "Summary of recommendations" section to summarize the findings across the experiments. All the reviewers give positive recommendations after reading the authors' response.

Therefore, I recommend acceptance for this paper. 
Decision: Accept (poster)