Early Convolutions Help Transformers See Better
Authors: Tete Xiao
Mannat Singh
Eric Mintun
Trevor Darrell
Piotr Dollar
Ross Girshick
Venue: NeurIPS 2021 Poster (2021)
Keywords: Vision Transformer
CNN
Model Optimizability

Abstract: Vision transformer (ViT) models exhibit substandard optimizability. In particular, they are sensitive to the choice of optimizer (AdamW vs. SGD), optimizer hyperparameters, and training schedule length. In comparison, modern convolutional neural networks are easier to optimize. Why is this the case? In this work, we conjecture that the issue lies with the patchify stem of ViT models, which is implemented by a stride-p p×p convolution (p = 16 by default) applied to the input image. This large-kernel plus large-stride convolution runs counter to typical design choices of convolutional layers in neural networks. To test whether this atypical design choice causes an issue, we analyze the optimization behavior of ViT models with their original patchify stem versus a simple counterpart where we replace the ViT stem by a small number of stacked stride-two 3×3 convolutions. While the vast majority of computation in the two ViT designs is identical, we find that this small change in early visual processing results in markedly different training behavior in terms of the sensitivity to optimization settings as well as the final model accuracy. Using a convolutional stem in ViT dramatically increases optimization stability and also improves peak performance (by ∼1-2% top-1 accuracy on ImageNet-1k), while maintaining flops and runtime. The improvement can be observed across the wide spectrum of model complexities (from 1G to 36G flops) and dataset scales (from ImageNet-1k to ImageNet-21k). These findings lead us to recommend using a standard, lightweight convolutional stem for ViT models in this regime as a more robust architectural choice compared to the original ViT model design.

Early Convolutions Help Transformers See Better
Tete Xiao 1.2
Mannat Singh'
Eric Mintun
Trevor Darrell
Piotr Dollár
Ross Girshick"
Facebook Al Research (FAIR)
UC Berkeley
Abstract
Vision transformer (ViT) models exhibit substandard optimizability. In particular, they are sensitive the choice of optimizer (AdamW vs. SGD), optimizer hyperpa- rameters, and training schedule length. In comparison, modern convolutional neural networks are easier to optimize. Why is this the case? In this work, we conjecture that the issue lies with the patchify stem of ViT models, which is implemented by a stride-p pxp convolution (p 16 by default) applied the input image. This large-kernel plus large-stride convolution runs counter to typical design choices of convolutional layers neural networks. To test whether this atypical design choice causes an issue, we analyze the optimization behavior of ViT models with their original patchify stem versus simple counterpart where we replace the ViT stem by a small number of stacked stride-two 3x3 convolutions. While the vast majority of computation in the two ViT designs identical, we find that this small change in early visual processing results in markedly different training behavior in terms of the sensitivity to optimization settings as well as the final model accuracy. Using convolutional stem in VIT dramatically increases optimization stability and also improves peak performance (by ~ 11-2% top-1 1 accuracy on ImageNet- k), while maintaining flops and runtime. The improvement can be observed across the wide spectrum model complexities (from 1G to 36G flops) and dataset scales (from ImageNet-1k ImageNet-21k). These findings lead to recommend using a standard, lightweight convolutional stem for ViT models in this regime as a more robust architectural choice compared the original ViT model design.
Introduction
Vision transformer (ViT) models [13] offer an alternative design paradigm to convolutional neural networks (CNNs) [24]. ViTs replace the inductive bias towards local processing inherent in con- volutions with global processing performed by multi-headed self-attention [43]. The hope is that this design has the potential improve performance on vision tasks, akin the trends observed in natural language processing [11]. While investigating this conjecture, researchers face another unexpected difference between VITs and CNNs: ViT models exhibit substandard optimizability. ViT's are sensitive to the choice of optimizer [41] (AdamW [27] vs. SGD), to the selection of dataset specific learning hyperparameters [13, 41], to training schedule length, to network depth [42], etc. These issues render former training recipes and intuitions ineffective and impede research.
Convolutional neural networks, in contrast, are exceptionally easy and robust optimize. Simple training recipes based on SGD, basic data augmentation, and standard hyperparameter values have been widely used for years [19]. Why does this difference exist between ViT and CNN models? In this paper we hypothesize that the issues lies primarily in the early visual processing performed by ViT. ViT "patchifies" the input image into pxp overlaapppi patches form the transformer encoder's 's input set. This patchify stem is implemented as a stride-p pxp convolution, with p = 16 as default value. This large-kernel plus large-stride convolution runs counter the typical design
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

patchity (P) stots
fratsSommer block
Lx
Original VIT (baseline, termed VII,):
Sensitive to and wd choice Converges slowly
coff
Works with AdomW, but not SGD Underpprrfrms sola CNNs on ImageNet
convolutionl (C)
InasS block
(L=1)
Ours (termed ViT same runtime):
E PILL SPPP TV
XB
v Robust to and choice
mmng
V Converges quickly
Works with AdamW, and also SGD Ohtperforms sota CNNs on ImageNet
Nlow Buge de Ieansormer biucs
Figure 1: Early convolutions help transformers see better: We hypothesize that the substandard optimizability of ViT models compared to CNNs primarily arises from the early visual processing performed by its patchify stem, which is implemented by non-overlapping stride-p pxp convolution, with p 16 by default. We minimally replace the patchify stem in ViT with standard convolutional stem of only ~5 convolutions that has approximately the same complexity as a single transformer block. We reduce the number of transformer blocks by one (i.e., L vs. L) to maintain parity in flops, parameters, and runtime. We refer to the resulting model as ViT and the original ViT as ViTp. p. The vast majority of computation performed by these two models is identical, yet surprisingly we observe that ViTc (i) converges faster, (ii) enables, for the first time, the use of either AdamW or SGD without a significant accuracy drop, (iii) shows greater stability to learning rate and weight decay choice, and (iv) yields improvements in ImageNet top-1 error allowing ViTc to outperform stateoof-the-an CNNs, whereas V does not.
choices used in CNNs, where best-practices have converged to a small stack of stride-two 3 kernels as the network's 's stem (e.g.. [30, 36, 391).
To test this hypothesis, we minimally change the early visual processing of ViT by replacing its patchify stem with standard convolutional stem consisting of only ~5 convolutions, see Figure 1. To compensate for the small addition in flops, we remove one transformer block to maintain parity in flops and runtime. We observe that even though the vast majority of the computation in the two ViT designs is identical, this small change in early visual processing results in markedly different training behavior in terms of the sensitivity to optimization settings as well as the final model accuracy.
In extensive experiments we show that replacing the ViT patchify stem with a more standard convolutional stem (i) allows VIT converge faster ($5.1), (ii) enables, for the first time, the use of either AdamW or SGD without a significant drop in accuracy ($5.2), (iii) brings ViT's stability w.r.t. learning rate and weight decay closer that of modern CNNs ($5.3), and (iv) yields improvements in ImageNet [10] top-1 error of 111 percentage points ($6). We consistently observe these improvements across wide spectrum of model complexities (from 1G flops 36G flops) and dataset scales (ImageNet-Ii 1k to ImageNet-21k).
These results show that injecting some convolutional inductive bias into ViTs can be beneficial under commonly studied settings. We did not observe evidence that the hard locality constraint in early layers hampers the representational capacity of the network, might be feared [9]. In fact observed the opposite, as ImageNet results improve even with larger-scale models and acaee--ccale data when using convolution stem. Moreover, under carefully controlled comparisons, we find that ViT's are only able to surpass state-of-the-art CNNs when equipped with a convolutional stem ($6).
We conjecture that restricting convolutions in ViT to early visual processing may be a crucial design choice that strikes a balance between (hard) inductive biases and the representation lcarning ability of transformer blocks. Evidence comes by comparison to the "hybrid ViT" presented in [13], which uses 40 convolutional layers (most of a ResNet-50) and shows no improvement over the default VIT. This perspective resonates with the findings of [9]. who observe that early transformer blocks prefer to learn more local attention patterns than later blocks. Finally we note that exploring the design of hybrid CNN/VIT models is not goal of this work; rather demonstrate that simply using minimal convolutional stem with VIT is sufficient to dramatically change its optimization behavior.
In summary, the findings presented in this paper lead us to recommend using a standard, lightweight convolutional stem for VIT models the analyzed dataset scale and model complexity spectrum as a more robust and higher performing architectural choice compared to the original ViT model design.

---

2 Related Work
Convolutional neural networks (CNNs). The breakthrough performance of the AlexNet [23] CNN [15, 24] on ImageNet classification [10] transformed the field of recognition, leading to the development of higher performing architectures, <-g, [19, 36, 37, 48], and scalable training methods [16, 21J. These architectures are now core components in object detection (e.g.. [34]), instance segmentation (e.g., [18]), and semantic segmentation (e.g.. [26]). CNNs are typically trained with stochastic gradient descent (SGD) and are widely considered be easy to optimize.
Self-attention in vision models. Transformers [43] are revolutionizing natural language processing by enabling scalable training. Transformers use multi-headed self-attention, which performs global information processing and is strictly more general than convolution [6]. Wang et al. [46] show that single-headed) self- attention is a form of non-local means [2] and that integrating it into a ResNet [19] improves several tasks. Ramachandran et al. [32] explore this direction further with stand-alone self-attentio networks for vision. They report difficulties in designing an attention- network stem and present a bespoke solution that avoids convolutions, In contrast, we demonstrate the benefits of a convolutional stem. Zhao et al. [53] explore a broader set of self-attention operations with hard-coded locality constraints, more similar to standard CNNs.
Vision transformer (ViT). Dosovitskiy et al. [13] apply a transformer encoder image classification with minimal vision-specific modifications. As the counterpart of input token embeddings, they partition the input image into, e.g., 16x16 16 pixel, non-overlapping patches and linearly project them to the encoder's s input dimension. They report lackluster results when training on ImageNet- 1k, but demonstrate state-of-the-arr transfer learning when using large-scale pretraining data. ViTs are sensitive to many details of the training recipe, e-g., they benefit greatly from Adam [27] compared to SGD and require careful learning rate and weight decay selection. ViTs are generally considered to be difficult to optimize compared to CNNs (e.g., [13, 41, 42]). Further evidence of challenges comes from Chen et al. [4] who report ViT optimization instability in self-supervised learning (unlike with CNNs), and find that freezing the patchify stem at its random initialization improves stability.
VIT improvements. VITs are gaining rapid interest in part because they may offer a novel direction away from CNNs. Touvron et al. [41] show that with more regularization and stronger data aug- mentation ViT models achieve competitive accuracy on ImageNet- Ik alone (ef' [13]). Subsequently, works concurrent with own explore numerous other VIT improvements. Dominant themes include multi-scale networks [14, 17, 25, 45, 50], increasing depth [42], and locality priors [5, 9, 17, 47, 49]. In [9], d' Ascoli et al. modify multi- self-attention with a convolutional bias at initialization and show that this prior improves sample efficiency and ImageNet accuracy. Resonating with our work, [5. 17, 47, 49] present models with convolutional stems, but do not analyze optimizability (our focus).
Discussion. Unlike the concurrent work on locality priors in ViT, our focus is studying optimizability under minimal ViT modifications in order derive crisp conclusions. Our perspective brings several novel observations: by adding only ~5 convolutions to the stem, ViT can be optimized well with either AdamW or SGD (cf. all prior works use AdamW avoid large drops in accuracy [41]), it becomes less sensitive to the specific choice of learning rate and weight decay, and training converges faster. We also observe a consistent improvement ImageNet top-1 accuracy across wide spectrum of model complexities (1G flops to 36G flops) and dataset scales (ImageNet-II 1k to imageNett21kk) These results suggest that (hard) convolutional bias early in the network does not compromise representational capacity, as conjectured in [9], and is beneficial within the scope of this study.
3 Vision Transformer Architectures
Next, we review vision transformers [13] and describe the convolutional stems used in our work.
The vision transformer (ViT). ViT first partitions an input image into non-overlapping pxp patches and linearly projects each patch to d-dimensional feature vector using learned weight matrix. A patch size of p = 16 and an image size of 224x224 are typical. The resulting patch embeddings (plus positional embeddings and learned classification token embedding) are processed by standard transformer encoder [43, 44] followed by classification head. Using common network nomenclature, we refer to the portion of ViT before the transformer blocks the network's 's stem. ViT's stem is
3

---

model
ref
hidden
MLP
num
num
Hlps
params
acts
time
model
hidden
MLP num num blocks
flops
parums
acts time
model
size
mult
heads
blocks
(B)
(M)
(M)
(min)
size
mult
beads
(B)
(M)
(M)
(min)
-
-NTI-T
1.1
5.5
VGT
192
3
11
1.1
4.5
5.7
2.7
VIT p.GP
-VTTS
384
6
12
3.9
18.5
11,1
38
ViTe-4GF
384
3
6
11
4.0
17.8
11.3
3.9
VII,-1800
-WT-B
768
12
12
17.5
86.7
24.0
11.5
VITc-18GF
768
4
12
11
17.7
81.6
24.1
11.4
36T,
LWTL
1024
16
14
35.9
1784
37.3
18.8
VITc-36GF
1024
4
16
13
35.0
167.8
36.7
18.6
Table 1: Model definitions: Left: Our V models at various complexities, which use the original patckify stem and closely resemble the original ViT models [13]. To facilitate comparisons with CNNs. we modify the original ViT-Tiny, -Small, -Base, -Large models to obtain models at IGF, 4GF, 18GF, and 36GF, respectively. The modifications are indicated blue and include reducing the MLP multiplier from 4x to 3x for the 1GF and 4GF models, and reducing the number of transformer blocks from 24 to 14 for the 36GF model. Right: Our V models at various complexities that use the convolutional stem. The only additional modification relative to the corresponding p models is the removal of transformer block to compensate for the increased flops of the convolutional stem. We show complexity measures for all models (flops, parameters, activations, and epoch training time on ImageNet-1k;) the corresponding p and ViTc models match closely on all metrics.
specific case of convolution (stride-p, pxp kernel), but we will refer to it as the patchify stem and reserve the terminology of convolutional stem for stems with a more conventional CNN design with multiple layers of overlapping convolutions (i.e., with stride smaller than the kernel size).
VITp models. Prior work proposes VIT models of various sizes, such as ViT-Tiny, ViT-Small, ViT-Base, etc. [13, 41J. To facilitate comparisons with CNNs, which are typically standardized to I gigaflop (GF), 2GF, 4GF, 8GE, etc., we modify the original ViT models to obtain models at about these complexities. Details are given in Table 1 (left). For easier comparison with CNNs of similar flops, and to avoid subjective size names, refer the models by their flops, e.g., p-GF in place of Vmal- We use the P subscript to indicate that these models use the original patchify stem.
Convolutional stem design. We adopt typical minimalist convolutional stem design by stacking 3x3 convolutions [36], followed by a single 1 1x1 convolution at the end to match the d-dimensional input of the transformer encoder. These stems quickly downsample 224 x x24 input image using overlapping strided convolutions to 14x 14, matching the number of inputs created by the standard patchify stem. We follow simple design pattern: all 3x3 convolutions either have stride 2 and double the number of output channels or stride 1 and keep the number of output channels constant. We enforce that the stem accounts for approximately the computation of one transformer block of the corresponding model so that we can easily control for flops by removing one transformer block when using the convolutional stem instead of the patchify stem. Our stem design was chosen to be purposefully simple and we emphasize that it was not designed to maximize model accuracy.
VITe models. To form a ViT model with a convolutional stem, we simply replace the patchify stem with its counterpart convolutional stem and remove one transformer block to compensate for the convolutional stem's extra flops (see Figure 1). We refer the modified ViT with convolutional stem as ViTc. Configurations for ViTc at various complexities are given in Table I (right); corresponding V and ViTc models match closely on all complexity metrics including flops and runtime.
Convolutional stem details. Our convolutional stem designs use four, four, and six 3x3 convolutions for the IGF, 4GF, and 18GF models, respectively. The output channels are [24, 48, 96, 192], [48, 96, 192. 384]. and [64, 128, 128, 256, 256, 512], respectively. All 3x3 convolutions are followed by batch norm (BN) [21] and then ReLU [29], while the final 1x1 convolution not, be consistent with the original patchify stem. Eventually, matching stem flops to transformer block flops results in an unreasonably large stem, thus V3TG- uses the same stem as ViTc-18GF.
Convolutions in VIT. Dosovitskiy et al. [13] also introduced a "hybrid VIT architecture that blends modified ResNet [19] (BiT- [22]) with transformer encoder. In their hybrid model, the patchify stem is replaced by a partial BeT-ResNet-50 that terminates at the output of the conv4 stage or the output of an extended conv3 stage. These image embeddings replace the standard patchify stem embeddings. This partial BiT-ResNet-50 stem is deep, with 40 convolutional layers. In this work, we explore lightweight convolutional stems that consist of only to convolutions in total, instead of the 40 used by the hybrid VIT. Moreover, we emphasize that the goal of our work is not to explore the hybrid VIT design space, but rather to study the optimizability effects of simply replacing the patchify stem with minimal convolutional stem that follows standard CNN design practices.

---

4 Measuring Optimizability
It has been noted in the literature that VIT models are challenging to optimize, e.g., they may achieve only modest performance when trained on a mid-size dataset (ImageNet- 1k) [13], are sensitive to data augmentation [41] and optimizer choice [41], and may perform poorly when made deeper [42]. We empirically observed the general presence of such difficulties through the course of our experiments and informally refer to such optimization characteristics collectively as optimizability.
Models with poor optimizability can yield very different results when hyperparameters are varied, which can lead to seemingly bizarre observations, e.g., removing erasing data augmentation [54] causes catastrophic drop in ImageNet accuracy in [41]. Quantitative metrics to measure optimiz- ability are needed to allow for more robust comparisons. In this section, we establish the foundations of such comparisons; extensively test various models using these optimizability measures in $5.
Training length stability. Prior works train VIT models for lengthy schedules, e.g., 300 to 400 epochs ImageNet is typical (at the extreme, [17] trains models for 1000 epochs), since results at formerly common 100-epoch schedule are substantially worse (2-4% lower top-1 accuracy, see $5.1). In the context of ImageNet, we define top-1 accuracy at 400 epochs as approximate asymptotic result, i.e., training for longer will not meaningfully improve top-1 accuracy, and we compare it to the accuracy of models trained for only 50, 100, or 200 epochs. We define training length stability the gap to asymptotic accuracy. Intuitively, it's a measure of convergence speed. Models that converge faster offer obvious practical benefits, especially when training many model variants.
Optimizer stability. Prior works use A [27] to optimize ViT models from random initialization. Results of SGI) are not typically presented and we only aware of Touvron ef al. [41]'s report of dramatic -7% drop in ImageNet top-1 accuracy. In contrast, widely used CNNs, such as ResNets, can be optimized equally well with either SGD or AdamW (see $5.2) and SGD (always with momentum) is typically used in practice. SGD has the practical benefit of having fewer hyperparameters (e.g., tuning AdamW's 32 can be important [3]) and requiring 50% less optimizer state memory, which can ease scaling. We define optimizer stability as the accuracy gap between Adam W and SGD. Like training length stability, we use optimizer stability as a proxy for the ease of optimization of a model.
Hyperparameter (Ir, wd) stability. Learning rate (Ir) and weight decay (wd) are among the most important hyperparameters governing optimization with SGD and AdamW. New models and datasets often require a search for their optimal values as the choice can dramatically affect results. It is desirable have a model and optimizer that yield good results for a wide range of learning rate and weight decay values. We will explore this hyperparameter stability by comparing the error distribution functions (EDFs) [30] of models trained with various choices of Ir and wd. In this setting, to create an EDF for a model we randomly sample values of Ir and wd and train the model accordingly. Distributional estimates, like those provided by EDFs, give a more complete view of the characteristics of models that point estimates cannot reveal [30, 31J. We will review EDFs in $5.3.
Peak performance. The maximum possible performance of each model is the most commonly used metric in previous literature and it is often provided without carefully controlling training details such as data augmentations, regularization methods, number of epochs, and Ir. we tuning. To make more robust comparisons, we define peak performance the result of model at 400 epochs using its best-performing optimizer and parsimoniously tuned and wd values (details in $6), while fixing justifiably good values for all other variables that have known impact training. Peak performance results for ViTs and CNNs under these carefully controlled training settings are presented in $6.
5 Stability Experiments
In this section we test the stability of ViT models with the original patchify (P) stem the convolutional (C) stem defined in $3. For reference, we also train RegNetY [12, 31], a state-of-the-art CNN that is easy optimize and serves as a reference point for good stability.
We conduct experiments using ImageNet- [10]'s standard training and validation sets, and report top-1 error. Following [12], for all results, we carefully control training settings and we use minimal set of data augmentations that still yields strong results, for details see $5.4. this section, unless noted, for each model we use the optimal Ir and wd found under 50 epoch schedule (see Appendix).
5

---

1GF models
4GF models
18GF models
VIT,
ViTe
VITe
VITe
RegNetY
RegNetY
RegNecY
Atop-1
100
220
400
50
100
200
400
50
100
200
400
training epochs
training epochs
training epochs
Figure 2: Training length stability: We train 9 models for 50 to 400 epochs on ImageNet- Ik and plot the Atop- error the 400 epoch result for each. V demonstrates faster convergence than ViTp p across the model complexity spectrum, and helps close the gap to CNNs (represented by RegNetY).
4GF modilss
18GF modess
ViTp
VIT,
Wil,
VITz
RegNett
RegNetY
RegNety
50
150 200
50
100
203
training epochs
training epochs
training epochs
Figure 3: Optimizer stability: We train each model for 50 to 400 epochs with Adam (upward triangle A) and SGD (downward triangle )) For the baseline V.T,, SGD yields significantly worse results than AdamW. In contrast, ViTc and RegNetY models exhibit a much smaller gap between SGD and AdamW actoss all settings. Note that for long schedules, ViT often fails converge with SGD (i.e., loss goes to NaN), in such cases we copy the best results from a shorter schedule of the same model (and show the results via dashed line).
5.1 Training Length Stability
We first explore how rapidly networks converge to their asymptotic error ImageNet- Ikk i.e., the highest possible accuracy achievable by training for many epochs. We approximate asymptotic error as model's error using a 400 epoch schedule based on observing diminishing returns from 200 to 400. We consider a grid of 24 experiments for ViT: (P, C) stems x {1, 4, 18) GF model sizes x (50, 100, 200, 400) epochs. For reference also train RegNetY at { 4. 16) GF. We the best optimizer choice for each model (Adam for ViT models and SGD for RegNetY models).
Figure 2 shows the absolute error deltas (Atop-1) between 50, 100, and 200 epoch schedules and asymptotic performance (at 400 epochs). ViTc demonstrates faster convergence than ViTP across the model complexity spectrum, and closes much of the gap the rate of CNN convergence. The improvement most significant in the shortest training schedule (50 epoch), e.g., ViT 1-IGF has 10% error delta, while V reduces this to about 6%. This opens the door applications that execute a large number of short-scheduled experiments, such as neural architecture search.
Results.
5.2 Optimizer Stability
We next explore how well AdamW and SGD optimize ViT models with the two stem types. We consider the following grid of 48 ViT experiments: {P, C) stems (1, 4, 18) GF sizes x [50, 100, 200, 400) epochs [AdamW, W, SGD) optimizers. As reference, we also train 24 RegNetY baselines, one for each complexity regime, epoch length, and optimizer.
Results. Figure 3 shows the results. As a baseline, RegNet' models show virtually no gap when trained using either SGD or AdamW (the difference ~0.1 1-0.2% is within noise). On the other hand, VIT models suffer dramatic drop when trained with SGD across all settings (of up to 10% for larger models and longer training schedules). With convolutional stem, ViT models exhibit much smaller error gaps between SGD and AdamW across all training schedules and model complexities, including larger models and longer schedules, where the gap reduced to less than 0.2%. In other words, both RegNetY and VIT can be casily trained via either SGD AdamW, but VIT cannot.

---

Regilett
20
-12
-11
-10
-6
-7
-6
-4
-16
-14
-12
-10
log:(r)
logz(wd)
lag:(lr wd)
1.0
0.8
cumuittve
0.6
VTT-
VITC
VIT:
Regelett
RogNatt
RegNetr
Atop-1 error
Atop-1 error
Atop-1 error
Figure Hyperparameter stability for AdamW (Ir and wd): For each model, train 64 instances of the model for 50 epochs each with random Ir and wd (in a fixed width interval around the optimal value for each model). Top: Scatterplots of the Ir, wd, and wwd for three 4GF models. Vertical bars indicate optimal Ir, wel, and Ir wwd values for each model. Bottom: For each model, we generate an EDF of the errors by plotting the cumulative distribution of the Atop-1 errors (A to the optimal error for each model). A steeper EDF indicates better stability to Ir and wd variation. ViT significantly improves the stability over the baseline V across the model complexity spectrum, and matches or even outperforms the stability of the CNN model (RegNetY).
cumulative
9.6
2.4
IGF models
46F modes
J8GF model
VIT:
0.2
Reghieff
RegNety
Regheet
0.0
10
10
Atop-1 error
Atop-1 error
Mtop-1 1 error
Figure 5: Hyperparameter stability for SGD (Ir and wd): We repeat the setup from Figure 4 using SGD instead of AdamW. The stability improvement of ViTe over the baseline ViT is even larger than with AdamW. W. E.g., ~60% of ViTc-I8GF models are within 4% Atop- eτror of the best result, while less than 20% v-T p-&GF models are fact most V-Tp-18GF runs don't 't converge).
5.3 Learning Rate and Weight Decay Stability
Next, we characterize how sensitive different model families are to changes learning rate (Ir) and weight decay (wd) under both AdamW and SGD optimizers. To quantify this, we make use of emro distribution functions (EDFs) [30]. An EDF computed by sorting a set of results from low-to-high eπror and plotting the cumulative proportion of results error increases, particular, we generate EDFs model as a function of Ir and wd. The intuition is that a model is robust to these hyperparameter choices, the EDF will be steep (all models will perform similarly), while the model is sensitive, the EDF will be shallow (performance will be spread out).
[30] for details. In
We test 6 VIT models (IP, C) x (1,4, 18) GF) and RegNetY models (11, 4, 16) GF). For each model and each optimizer, we compute an EDF by randomly sampling 64 (Ir, we) pairs with learning rate and weight decay sampled in fixed width interval around their optimal values for that model and optimizer (see the Appendix for sampling details). Rather than plotting absolute error the EDF, we plot Atop-1 error between the best result (obtained with the optimal Ir and wd) and the observed result. Due to the large number of models, we train each for only epochs.
Results. Figure 4 shows scatterplots and EDFs for models trained by AdamW. Figure 5 shows SGD results. In all cases we see that ViT significantly improves the Ir and wd stability over ViTp for both optimizers. This indicates that the Ir and wd are easier to optimize for VIT than for VIT p.

---

5.4 Experimental Details
In all experiments we train with a single half-period cosine learning rate decay schedule with a 5 5-epoch linear learning rate warm-up [16]. We use a minibatch size of 2048. Crucially, weight decay not applied to the gain factors found in normalization layers nor to bias parameters anywhere the model; we found that decaying these parameters can dramatically reduce top-1 accuracy for small models and short schedules. For inference, we use an exponential moving average (EMA) of the model weights (e.g.. [8]. The Ir and wd used in this section are reported in the Appendix. Other hyperparameters use defaults: SGD momentum is 0.9 and Adam W's 81 = 0.9 and $2 = 0.999.
Regularization and data augmentation. We use simplified training recipe compared to recent work such as DeiT [41], which we found to be equally effective across a wide spectrum of model complexities and dataset scales. We use AutmAugment [7], mixup [52] (a 0.8), CutMix [5]] = (.0), and label smoothing [38] (E = 0.1). We prefer this setup because is similar to common settings for CNNs (e.g.. [12]) except for stronger mixup and the addition of CutMix (ViTs benefit from both, while CNNs are not harmed). We compare this recipe to the one used for DeiT models in the Appendix, and observe that our setup provides substantially faster training convergence likely because we remove repeating augmentation [1, 20], which is known to slow training [1].
6 Peak Performance
model's peak performance is the most commonly used metric network design. It represents what is possible with the hest-known-so-far settings and naturally evolves over time. Making fair comparisons between different models is desirable but fraught with difficulty. Simply citing results from prior work may be negatively biased against that work as it was unable to incorporate newer, yet applicable improvements. Here, we strive to provide a fairrrr comparison between state-of-the-art CNNs, ViTp, p. and ViTg. We identify a set of factors and then strike a pragmatic balance between which subset to optimize for each model vs. which subset share a constant value across all models.
In our comparison, all models share the same epochs (400), use of model weight EMA, and set of regularization and augmentation methods (as specified in $5.4). All CNNs are trained with SGD with Ir of 2.54 and we of 2.4e- -5; we found this single choice worked well across all models, as similarly observed in [12]. For all ViT models we found AdamW with Ir/wd of 1/0e330..22 was effective, except for the 36GF models. For these larger models we tested few settings and found Ir/wd of 6.0e-4/0.28 to be more effective for both P-GG and ViT models. For training and inference, ViTs use 224x 224 resolution (we do not fine-tune at higher resolutions), while the CNNs use (often larger) optimized resolutions specified in [12, 39]. Given this protocol, we compare ViTp, ViTc, and CNNs across spectrum of model complexities (IGF to 36GF) and dataset scales (directly training on ImageNet- 1k vs. pretraining on ImageNet-2lk 21 and then fine-tuning on ImageNet- 1k).
Results. Figure 6 shows progression of results. Each plot shows ImageNet-Ik Ik val top-1 error vs. ImageNet- Ik epoch training time.' The left plot compares several state-of-the-an CNNs. RegNetY and RegNetZ [12] achieve similar results across the training speed spectrum and outperform Effi- cientNets [39]. Surprisingly, ResNets [19] are highly competitive at fast runtimes, showing that under fairer comparison these years-old models perform substantially better than often reported (ef. [39]).
The middle plot compares two representative CNNs (ResNet and RegNetY) to ViTs, still using only ImageNet-Ik 1k training. The baseline ViTp underperforms RegNet' across the entire model complexity spectrum. To our surprise, V also underperforms ResNets in this regime. ViTe is more competitive and outperforms CNNs in the middle-complextty range.
The right plot compares the same models but with ImageNet-211 pretraining (details in Appendix). In this setting ViT models demonstrates a greater capacity to benefit from the larger-scale data: now ViTc strictly outperforms both ViTp and YegNetY. Interestingly. the original VIT p does not outperform a state-of-the-ar CNN even when trained on this much larger dataset. Numerical results are presented in Table for reference to exact values. This table also highlights that flop counts are not significantly correlated with runtime, but that activations are (see Appendix for more details), as also observed by [12]. E.g., EfficientNets are slow relative to their flops while ViTs are fast.
We time models in PyTorch on 8 32GB Volta GPUs We note that batch inference time is highly correlated with training time, but report epoch time it is easy interpret and does not depend on the use case.
8

---

ResNet RegNeet
VIT.
ImageNett 21k
VIT.
VIT,
etror
RegNeet2
ViTe
22
Efficienthiet
RegNatt
ResNet
RegNett
18
14
2.5
5.0 16 20
2.5
50
2.5
training speed (min.)
training speed (min.)
training speed (min.)
Figure 6: Peak performance (epoch training time vs. ImageNet-Ik val top-1 error): Results of a fair, controlled comparison of ViTp, ViTg, and CNNs. Each curve corresponds complexity sweep resulting in a training speed spectrum (minutes per ImageNet-II Ik epoch). Left: State-of-the-ar CNNs. Equipped with modern training recipe, ResNets are highly competitive in the faster regime, while RegNetY and Z perform similarly, and better than EfficientNets. Middle: Selected CNNs compared ViTs. With access to only ImageNet- 1k training data, RegNetY and ResNet outperform V across the board. ViTc is more competitive with CNNs. Right: Pretraining on ImageNet-211 improves the ViT models more than the CNNs, making v competitive. Here, the proposed C outperforms all other models across the full training speed spectrum.
model
model
flops
params
(M)
acts
(M)
(min) size
time
batch
opochs
200
IN
model
flops params
(M)
acts
(M)
(min) size
time batch
IN epochs
200
ResNet:50
4.1
25.6
11.3
3.4
2048
22.5
21.2
20.7
21.6
BITNet B2
1.0
9.1
13.8
5.9
2048
21.4
20.5
19.9
ResNet-101
7.8
44.5
16.4
5.5
2048
20.3
19.1
18.5
19.2
BAfNet-B4 B4
4,4
19.3
49.5
1944
512
18.5
17.8
17.5
ResNet-152
11.5
60.2
22.8
2048
19.5
18.4
17.7
18.2
EffNet-B5
10.3
30.4
98.9
41.7
256
17.3
17.0
17.0
ResNet-200
15.0
64.7
32.3
10.7
1024
19.5
18.3
17.6
17.7
Neg
1.0
9.6
6.2
2048
23.2
22.2
21.5
VTTP-IGF
1.1
5.5
2048
33.2
29.7
27.7
RegNetY-4GF
4.1
22.4
14.5
7.6
2048
19.4
18.3
17.9
18.4
WTT-4GF
3.9
18.5
11.1
3.8
2048
23.3
20.8
19.6
20.6
RegNetY-16GG 16GF
15.5
72.3
30.7
17.9
1024
17.1
16.4
16.3
15.6
VSG,-18GFF
17.5
86.6
24.0
11.5
1024
19.9
18.4
17.9
16.4
ReyNerY
31.1
128.6
46.2
35.1
512
16.2
15.9
15.9
15.0
36G
35.9
178.4
37.3
18.8
512
19.9
18.8
18.2
15.1
Reg Reg
1.0
11.0
888
4.2
2048
20.8
20.2
19.6
VTTe-GF
1.1
4.6
57
2.7
2048
28.6
26.1
24.7
RegNctZ-4GF
4.0
28.1
1024
16.6
VITe-GFF
4.0
2048
18.6
18.8
RegNetZ- 16GF
16.0
95.3
51.3
32.0
512
16.0
15.9
15.9
1SGF
17.7
81.6
24.1
11.4
1024
18.4
17.5
17.0
15.1
ReyNerZZ32GP
32.0
175.1
79.6
55.3
16.1
36G
35.0
167.8
36.7
18.6
14.2
Table 2: Peak performance (grouped by model family): Model complexity and validation top-1
error at 100, 200, and 400 epoch schedules on ImageNet-Ik, and the top-1 error after pretraining on ImageNet-21k 1 (IN 21k) and fine-tuning on ImageNet- 1k. This table serves as reference for the results shown in Figure 6. Blue numbers: best model trainable under 20 minutes per ImageNet- 1k epoch. Batch sizes and training times are reported normalized to 8 32GB Volta GPUs (see Appendix). Additional results on the ImageNet-V2 V2 [33] test set are presented in the Appendix.
These results verify that ViT convolutional stem improves not only optimization stability, as seen in the previous section, but also peak performance. Moreover, this benefit can be seen across the model complexity and dataset scale spectrum. Perhaps surprisingly, given the recent excitement over ViT, find that p struggles to compete with state-of-the-ard CNNs. We only observe improvements over CNNs when using both large-scale pretraining data and the proposed convolutional stem.
7 Conclusion
In this work we demonstrated that the optimization challenges of VIT models are linked to the large- stride, large-kemell convolution in ViT's patchify stem. The seemingly trivial change of replacing this patchify stem with a simple convolutional stem leads to remarkable change in optimization behavior. With the convolutional stem, ViT (termed ViTc) converges faster than the original ViT (termed ViT ($5.1). trains well with either Adam or SGD ($5.2), improves learning rate and weight decay stability ($5.3), and improves ImageNet top-1 error by ~ 1-22% ($6). These results are consistent across a wide spectrum of model complexities (IGF to 36GF) and dataset scales (ImageNet-Il 1k to ImageNet-21kk). Our results indicate that injecting small dose of convolutional inductive bias into the early stages of ViTs can be hugely beneficill. Looking forward, we are interested the theoretical foundation of why such a minimal architectural modification can have such large (positive) impact on optimizability. We are also interested in studying larger models. Our preliminary explorations into 72GF models reveal that the convolutional stem still improves top-1 error, however we also find that a new form of instability arises that causes training error to randomly spike, especially for ViTc.
Acknowledgements. We thank Hervé Jegou, Hugo Touvron, and Kaiming He for valuable feedback.


Reviews:
[{"review_id": "nYT5UZb5BZa", "reviewer": "Reviewer_VuQn", "summary": "The authors perform an extensive study investigating the stem of ViT models and replacing it with one that consists of a stack of downsampling 3x3 convolutions. The study reveals that this simple change makes ViT training more robust, faster to converge, but also scales better (cf. i21k pre-training).", "questions": "", "limitations": "", "rating": 9, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I have to admit, that I was initially set to reject the paper, because I have counter-evidence of the main claim that \"ViT cannot be optimized by sgdm\": I have been training original patch-stem ViT models with SGDM for a while, to same accuracy as AdamW (but unpublished), and am certain that the negative results reported in DeiT are due to confounders, such as not properly tuning sgdm.\n\nHowever, carefully reading the paper made me change my mind by 180°. This paper is really good and needs to be accepted. Many statements reflect my thoughts:\n- It is crucial to tune lr/wd, or one gets misleading results such as \"random erasing is necessary\", and actually tuning those reveals it (and other tricks) to be *unnecessary*.\n- Especially for ViT and EfficientNet models, flops and #params are not a good measurement to put on the X axis.\n- We need to compare architectures in ideal settings for each.\n\nI find the very last presented finding, Fig6(right), to be the most interesting: the ViT_C seems to scale better than both ViT_P and Re{s/g}Nets. This is very promising.\n\nNow some criticism on the paper:\n\n1) line 112: I would argue that the conv-stem is not \"strictly beneficial\" since:\n1) A) the patchify stem has led to some interesting recent ideas which are simply not possible with a conv-stem, such as using different random augmentations per patch.\n1) B) It re-introduces BatchNorm into the model, with all of its problems. I wish authors had tried using GroupNorm here too.\n2) line 94: the cited papers do not really support \"optimization difficulty\".\n3) Why do we need the final 1x1 conv in the stem? Couldn't we simply have the last 3x3 conv output the required number of channels? That would furhter simplify the design.\n4) line 139: \"we do not optimize the stem to maximize model accuracy\". This is not really true to spirit, given AppendixA. Even if the best stem ends up being the one that authors first thought of. What if a different one from AppendixA was found to perform substantially better? Surely, that one would have been proposed instead. Simply drop this sentence.\n5) Reusing optimal lr/wd from 50ep for up to 400ep. This is known to be suboptimal (cf AdamW paper itself) and generally for much longer training, wd needs to be decreased. It likely does not invalidate the results though.\n6) It is not explicitly stated what lr/wd are used in 5.1 and 5.2. Is it the best ones found in 5.3? If not, this might be an issue.\n7) Does the SGDM sweep also use \"decoupled\" WD as per AdamW? Do the authors actually use decoupled WD from AdamW? (That is not commonly used in practice, but better for such sweeps.) By \"decoupled\", I mean doing w_t+1 = w_t - lr * grad w_t - wd * w_t,  as opposed to w_t+1 = w_t - lr * grad w_t - wd * lr * w_t, see also the AdamW paper.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "None", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "RQW_XO8ctFP", "reviewer": "Reviewer_vj7C", "summary": "The paper studies the impact of stem in ViT. Specifically, the authors propose to replace the patch-based stem in ViT with a standard, lightweight convolutional stem, and show that this simple revision leads to substantially improved stability wrt the choices of optimizers and several other hyper-parameters. The resulting model achieves comparable performance with strong convnets baselines when trained on ImageNet-1K and has better accuracy-speed tradeoff than both ViTs and convents when trained over larger dataset (ImageNet-21K).", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n+ Overall, the paper is very well written and easy to follow.\n+ Extensive amount of ablation studies have been provided to back up the main claim that the conv stem can improve ViT's optimization stability/robustness.\n+ IMO the most interesting part of the manuscript is not the resulting model itself, but rather the methodology. For example, the notion of \"optimizability\" examined in the paper can be potentially used as an additional metric to evaluate new architectures and provide complementary insights other than accuracy or speed.\n\nWeaknesses:\n- The overall message is interesting but not strong enough to be exciting:\n  * In most ablation studies, ViT_c behaves like something in the middle of ViT and ConvNets (e.g., see Figure 2, Figure 3 and Figure 5). Those results are interesting, but on the other hand are more or less expected given the hybrid nature of the model.\n  * More importantly, although the authors provided extensive empirical evidence to back up their claim that a conv stem is helpful, the critical question that why it is effective is left unanswered. An in-depth analysis of the underlying reason for those empirical observations would make the paper a lot more informative.\n- The paper claims their model outperforms the state-of-the-art convnets, but I'm not sure whether it is true: while RegNet is a competitive baseline, there are more recent networks with stronger performance (e.g., NFNets).\n- There are missing details about the ImageNet-21K training setup that I cannot find in paper or appendix. E.g., what is the input resolution during fine-tuning (224 or 384)?\n\nOverall, I'm leaning positive about the paper because the methodology is interesting and I believe the data points would be useful for future studies. On the negative side, not much insight was provided to answer why conv stems are helpful, and I believe answering that technical question would substantially strengthen the work.\n\n========= post rebuttal ==========\n\nRaising my rating to 7 since some of my technical questions have been addressed in the author's response.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors mentioned \"understanding the theoretical foundation of why such a minimal architectural modification can have such large (positive) impact on optimizability\" as a future work. This is actually a major missing piece IMO, and I believe answering this question in future revisions could substantially strengthen the technical contribution of this work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "5EX43nP3-JJ", "reviewer": "Reviewer_G3R8", "summary": "This paper overall tackles a very important question and pin-points the important dissociation that when the computational complexity of two models is equalized: one with a first stage convolutional operator and another one without one -- both pre-seeded to Vision Transformers; it is the one with a convolutional stem/operator *that is only sufficiently necessary in the first layer!* that exploits the locality structure in image, has increase in accuracy, faster convergence and flexible optimizability. \n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nStrengths:\n\nThis paper presents all the correct ingredients for acceptance at any computer vision venue -- despite this being NeurIPS: The paper presents a strong set of experiments, baselines, and several simulations ran for different conditions (example GFlops, and hidden units in the ViT’s) that have been carefully tuned to show the stability of the results despite a lack of errorbars. Overall this paper should be accepted because it presents a simple idea that has been rigorously tested through various experimental manipulations and the novelty is presented quite clearly in Figure 1. The take-home (See above) is easy: if you want *better* results (accuracy/faster convergence/flexible optimizability) when training a Vision Transformer, add a convolutional stem/layer in the early block of visual processing (somewhat surprisingly similar -- to how the human visual system operates in early stages of processing).\n\nI have no follow-up questions to the authors, but I would like to add some limitations that prevent me from increasing my score (that is already very positive), as this paper could have had potential to have a more interdisciplinary scope beyond the Transformer movement.\n\nLimitations:\n\nThere is more to computer vision (and vision in general) than object recognition and ImageNet-based experiments. Several follow-up questions arise, such as: How would a convolutional-stem aid a perceptual system if the task was *not* object recognition (the title of ‘seeing’, suggests Object Recognition is the ultimate standard). But more critically, experiments with Robustness to either common corruptions (Hendrycks & Dietterich, ICLR 2019) or Adversarial attacks like any of the PGD-flavors should be explored in future work. Not to be nosey, but these are indeed important questions as several authors have shown that there is a trade-off between accuracy and robustness (Zhang et al., ICML 2019).\n\nI do wish that the paper adds more discussion on the ML theoretical aspects of the advantages of convolution when a locality constraint is induced by the image structure as explored in Poggio (PNAS 2020) and recently, Deza, Banburski, Liao & Poggio (ArXiv 2021). Similarly, it would be nice if the paper connects the importance of the convolutional operator as a biologically plausible inductive bias (eg. V1 gabor-like filters; Hubel & Wiesel in the 60’s) and how recent trends in computer vision have started to revisit the importance of inductive biases in 2-stage *neuroscience-inspired* perceptual systems as the work of Dapello, Marques et al. for the case of Adversarial Robustness (NeurIPS, 2020); Parthasarathy & Simoncelli (ArXiv, 2020) for texture recognition w/ a self-supervised objective and Deza & Konkle (ArXiv, 2020) for scene recognition w/ a foveated module. Perhaps adding a small section in the Relevant work about these and other works that have explored the importance of locality, convolutional operators in CNNs and 2-stage models should be addressed.\n\nOther works worth adding that are particularly relevant to the theory of locality and convolutional operators as a critical computation in machines:\n\n* Elsayed et al. “Revisiting Spatial Invariance with Low-Rank Local Connectivity”. ICML 2020.\n\n* Pogodin et al.: “Towards Biologically Plausible Convolutional Networks”. ArXiv 2021.\n\n* d’Ascoli et al: “Finding the Needle in the Haystack with Convolutions: on the benefits of architectural bias” NeurIPS 2019.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Not Applicable", "review_text": "", "extra_scores": {"ethical_concerns": "Not Applicable", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers agree after author's response that this paper is a (strong) accept (7,9,7). The paper doesn't have any major weak points. The message is simple and clear and the contribution thorough. I don't consider the implications of the result worthy of a spotlight though.
Decision: Accept (Poster)