Auto-scaling Vision Transformers without Training
Authors: Wuyang Chen
Wei Huang
Xianzhi Du
Xiaodan Song
Zhangyang Wang
Denny Zhou
Venue: ICLR 2022 Poster (2022)
Keywords: vision transformer
neural architecture search
training-free search
efficient training

Abstract: This work targets automated designing and scaling of Vision Transformers (ViTs). The motivation comes from two pain spots: 1) the lack of efficient and principled methods for designing and scaling ViTs; 2) the tremendous computational cost of training ViT that is much heavier than its convolution counterpart. To tackle these issues, we propose As-ViT, an auto-scaling framework for ViTs without training, which automatically discovers and scales up ViTs in an efficient and principled manner. Specifically, we first design a "seed" ViT topology by leveraging a training-free search process. This extremely fast search is fulfilled by a comprehensive study of ViT's network complexity, yielding a strong Kendall-tau correlation with ground-truth accuracies. Second, starting from the "seed" topology, we automate the scaling rule for ViTs by growing widths/depths to different ViT layers. This results in a series of architectures with different numbers of parameters in a single run. Finally, based on the observation that ViTs can tolerate coarse tokenization in early training stages, we propose a progressive tokenization strategy to train ViTs faster and cheaper. As a unified framework, As-ViT achieves strong performance on classification (83.5% top1 on ImageNet-1k) and detection (52.7% mAP on COCO) without any manual crafting nor scaling of ViT architectures: the end-to-end model design and scaling process costs only 12 hours on one V100 GPU. Our code is available at https://github.com/VITA-Group/AsViT.

Published as a conference paper at ICLR 2022
AUTO-SCALING VISION TRANSFORMERS WITHOUT TRAINING
Wuyang Chen', Wei Huang² Xianzhi Duu, Xiaodan Song². Zhangyang Wang', Denny Zhou?
I University of Texas, Austin University of Technology Sydney *Google wuyangg chen, at laswang) @utexas. edu weihuang. ut s com xianzhi, xiaodansong, dennyzhou) egoogle.com
ABSTRACT
This work targets automated designing and scaling of Vision Transformers (ViTs). The motivation comes from two pain spots: 1) the lack of efficient and principled methods for designing and scaling ViTs; 2) the tremendous computational cost of training VIT that is much heavier than its convolution counterpart. To tackle these issues, propose VIT, an auto-scaling framework for ViTs without training, which automatically discovers and scales up ViTs in an efficient and principled manner. Specifically, we first design "seed" ViT topology by leveraging a training- free search process. This extremely fast search is fulfilled by comprehensive study of ViT's network complexity, yielding a strong Kendall-tar correlation with ground-truth accuracies. Second, starting from the "seed" topology. we automate the scaling rule for ViT's by growing widths/depths to different ViT layers. This results in series of architectures with different numbers of parameters in single run. Finally, based on the observation that ViTs can tolerate coarse tokenization early training stages, propose a progressive tokenization strategy to train ViTs faster and cheaper. As a unified framework, VIT achieves strong performance classification (83.5% topl on ImageNet- 1k) and detection (52.7% mAP on COCO) without any manual crafting nor scaling of ViT architectures: the end-to- end model design and scaling process costs only 12 hours one V100 GPU. Our code is available at https:::///.. /tte://gttttt com/ VITA Grtrr/GrrupuAASSVV.
INTRODUCTION
Transformer (Vaswani et al. 2017). a family of architectures based on the self-attention mechanism. notable for modeling rangerange dependencies in the data. The success of transformers has evolved from natural language processing computer vision. Recently, Vision Transformer (ViT) (Doso- vitskiy et al. 2020). transformer architecture consisting of self-attention encoder blocks, has been proposed to achieve competitive performance to convolution neural networks (CNNs) (Simonyan & Zisserman 2014 He et al. 2016) on ImageNet (Deng et al. 2009).
However, it remains clusive on how to effectively design, scale-up, and train ViTs, with three important gaps awaiting. First, Dosovitskiy et al. (2020) directly hard-split the 2D image into a series of local patches, and learn the representation with pre-defined number of attention heads and channel expansion ratios. These ad-hoc "tokenization" and embedding mainly inherit from language
tasks (Vaswani al., (017)) but are customized for vision, which calls for more flexible and principled designs. Second, the learning behaviors of ViT, including (loss of) feature diversity (Zhou
al. 202T). receptive fields (Raghu al. (202]) and augmentations (Touvron et al.) 2020] Jiang et al. 2021), differ vastly from CNNs. Benefiting from self-attention, ViT can capture global information even with shallow layers, yet its performance quickly plateaued as going deeper. Strong augmentations are also vital to avoid ViT's from overfitting. These observations indicate that ViT architectures may require uniquely customized scaling-up laws to learn a more meaningful representation hierarchy. Third, training ViTs is both data and computation-consumiii state-of-the of-he-of-the-art performance, ViT requires up to 300 million images and thousands of TPU-days. Although recent works attempt enhance ViT's 's data and resource efficiency (Touvron et al. (0200 Hassani et al.] [22TT] Pan et al. 2021] Chen et al., 2021d), the heavy computation cost (e.g., quadratic with respect to the number of tokens) is still overwhelming, compared with training CNNs.
To achieve

---

Published as a conference paper at ICLR 2022
We point out that the above gaps are inherently connected by the core architecture problem: how to design and scale-up ViTs? Different from the convolutional layer that directly digests raw pixels, VITs embed coarse-level local patches input tokens. Shall we divide image into non-overlapping tokens of smaller size, or larger but overlapped tokens? The former could embed more visual details in each token but ignores spatial coherency, while the latter sacrifices the local details but may benefit more spatial correlations among tokens. A further question is on ViT's depth/width trade-off: shall we prefer a wider and shallower ViT. or a narrower but deeper one? A similar dilemma also persists for ViT training: reducing the number of tokens would effectively speed up the ViT training. but meanwhile might sacrifice the training performance if sticking to coarse tokens from end to end.
In this work, we aim to reform the discovery of novel ViT architectures. Our framework, called As- ViT (Auto-scaling ViT), allows for extremely fast, efficient, and principled ViT design and scaling. In short, Vi- first finds promising "seed" topology for ViT of small depths and widths, then progressively "grow" it into different sizes (number of parameters) to meet different needs. Specifically, our "seed" ViT topology is discovered from search space relaxed from recent manual ViT designs. To compare different topologies, we automate this process by a training-free architecture search approach and the measurement of VIT's complexity, which are extremely fast and efficient. This training-free free search is supported by our comprehensive study of various network complexity metrics, where find the expected length distortion has the best trade-off between time costs and Kendall-tau correlations. Our "seed" ViT topology is then progressively scaled up from a small network to a large one, generating a series of ViT variants in a single run. Each step, the increases of depth and width are automatically and efficiently balanced by comparing network complexities. Furthermore, to address the data-hungry and heavy computation costs of ViTs, we make our ViT tokens elastic, and propose progressive re-tokenization method for efficient ViT training. We summarize our contributions as below:
1. We for the first time automate both the backbone design and scaling of ViTs. A "seed" ViT topology is first discovered (in only seven V100 GPU-hours), and then its depths and widths grown with a principled scaling rule in a single run (five more V100 GPU-hours).
2. To estimate ViT's performance at initialization without any training, we conduct the first compre- hensive study of ViT's network complexity measurements. We empirically find the expected length distortion has the best ooff between the computation costs and its Kendall-tau correlations with ViT's ground-truth accuracy.
During training, we propose a progressive re-tokenization scheme via the change of dilation and stride, which demonstrates to be highly efficient ViT training strategy that saves up to 56.2% training FLOPs and 41.1% % training time, while preserving a competitive accuracy.
4. Our ViT achieves strong performance on classification (83.5% top-1 on ImageNet- 1k) and detection (52.7% mAP on COCO).
2 WHY WE NEED AUTOMATED DESIGN AND SCALING PRINCIPLE FOR VIT?
Background and recent development of ViT To transform a 2D image into a sequence, ViT (Dosovitskiy et al.] 2020) splits each image into 14 x 14 16 16 patches and embeds them into a fixed number of tokens; then following practice of the transformer for language model- ing, ViT applies self-attention leverages FFN (Feed-Forward Network) layers to learn feature embeddings. To better facilitate the visual representation learning, recently works try to train deeper ViTs (Touvron al., 2021 Zhou et al., (02T), incorporate convolutions (Wu et al. 2021 d' Ascoli et al. [2021] Yuan et al. 202 la). and design multi-scale feature extractions (Chen et al. 202 lb| /hang et al. 2021 Wang et al. 2021
learn reweighting masks as relationship modeling for tokens, and
Why manual design and scaling may be suboptimal? As the ViT architecture is still in its infant stage, there is no principle in its design and scaling. Barly designs incorporate large token sizes, constant sequence length, and hidden size (Dosovitskiy et al.] 2020: Touvron et al.) 2020). and recent trends include small patches, spatial reduction, and channel doubling (Zhou et al.) 202 Liu et al. 2021). They all achieve comparably good performance, leaving the optimal choices unclear. Moreover, different learning behaviors of transformers from CNNs make the scaling law of ViTs
We generally use the term "ViT" to indicate deep networks of self-atiention blocks for vision problems. We always include clear citation when we specifically discuss the ViT's proposed by Dosovitskiy et al. (2020)
2

---

Published as a conference paper at ICLR 2022
highly unclear. Recent works (Zhou et al. 2021 demonstrated that attention maps of ViTs gradually become similar in deeper layers, leading to identical feature maps and saturated performance. ViT also generates more uniform representations across layers, enabling early aggregation of global context (Raghu et al., 2021). This is contradictory to CNNs as deeper layers help the learning of visual global information (Chen et al. 2018). These observations all indicate that previously studied scaling laws (depth/width allocations) for CNNs (Tan & Le. 2019) may not be appropriate to ViTs.
What principle do we want? We aim to automatically design and scale-up ViTs, being principled and avoiding manual efforts and potential biases. We also want to answer two questions: 1) Does ViT have any preference in its topology (patch sizes, expansion ratios, number of attention heads, etc.)? 2) Does ViT necessarily follow the same scaling rule of CNNs?
AUTO-DESIGN & SCALING OF VITS WITH NETWORK COMPLEXITY
To accelerate ViT designing and avoid tedious man- ual efforts, we target efficient, automated, and prin-
Image: 3xixxW
topology search scaling-up
cipled search and scaling of ViTs. Specifically, we have two problems to solve: 1) with zero training cost
Patch Embedding: Kernel K,, Stride
(Section 3.2). how to efficiently find the optimal ViT
4 L, leyers
"
Attention: asplits FFN: expansion ratio E,
architecture topology (Section ( ? 2) how to scale-up depths and widths of the ViT topology to meet different needs of model sizes (Section 3.4
Patch le-embedding Kernel Kg, Stride
H W 8 x layers
Attention:: asplits
FFN: expansion ratio E.
3.1 EXPANDED TOPOLOGY SPACE FOR VITS
Pstch ombodding Kornel Ky, Stride
Before designing and scaling, we first briefly introduce our expanded topology search space for our As-ViT (blue italics in Figure [].. We first embed the input image into patches of scalee resolution, and adopt a stage-wise spatial reduction and channel doubling strategy. This is for the convenience of dense prediction tasks like detection that require multi-scale features. Table|[ details of our topology space, and will be explained below.
H W Ly layers
Attention: Asplits
FFN: expansion ratio P3
Patch embedding Kernel Nr, Stride
BEX 32
Attention: Aeplits =1 expansion ratio Loulput
by layers
Figure 1: Overall architecture of our As VIT. Blue italics indicates topology configurations be searched [T. Red indicates depth/width Flastic kernels. Instead of generating non- to be scaled-up.
overlapped image patches, we propose to search for the kernel size. This will enable patches to be overlapped with their neighbors, introducing more spatial correlations among tokens. Moreover, cach time we downsample the spatial resolution, also introduce overlaps when re-embedding local tokens (implemented by either a linear or a convolutional layer).
Table Topology Search Space for our ViT.
Elastic attention splits. Splitting the attention into local windows is an important design to reduce the computation cost of self-attention without sacrificing much performance (Zaheer et al. [202) Liu et al. (021).
Stage
Sub-space
Choices
Kernel Ki
4,5, 6,
#1
Atlention Splits S
2, 4,
Instead of using a fixed number of splits, we propose to search for elastic attention splits for each stage" Note
FFN Expansion Ej
2, 3, 4, 6,
that we try to make our design general and do not use shifted windows (Liu et al. 2021
Kernel K₂
2,
#2
Attention Splits Sa
1, 1,2,
FFN Expansion Fg
2, 3, 4, 6
More search dimensions. VIT (Dosovisskiy et al., 2020) by default leveraged an FFN layer with 4x ex- panded hidden dimension for each attention block. To enable more flexible design ViT architectures, for
Kernel K₃
2,
#3
Attention Splits Sa
1.2 2. 5,
FFN Expansion Ess
Kernel K₁ FFN Expansion Fa
2, 3,
#4
each stage we further search over the FFN expansion ratio. We also search for the final number of heads for the self-attention module.
3, 4, 5, 6
Num. Heads
16. 32, 64
Due to spatial reduction, the th stage may already reach resolution at 7 x 7 on ImageNet, and we set its splitting as 1.
3

---

Published as a conference paper at ICLR 2022
3.2 ASSESSING VIT COMPLEXITY AT INITIALIZATION VIA MANIFOLD PROPAGATION
Training ViTs is slow: hence architecture search guided by evaluating trained models' accu- racies will be dauntingly expensive. We note a recent surge of training- free neural architecture search methods for Rel_e-based CNNs, leveraging local linear maps (Mellor et al. 2020), gradient sensitivity (Abdelfattah et al. 2021), number of linear regions (Chen et al.) 202 I. or network topology (Bhardwaj et al. 2021), However, ViTs are equipped with more complex non-linear func- tions: self-attention, softmax, and GeLU. Therefore, we need to measure their learning capacity in more general way. In our work, we consider measuring the complexity of manifold propagation through ViT, to estimate how complex functions can be approximated by ViTs.
Intuitively, a complex network can propagate a simple imput into a complex manifold at its output layer, thus likely to possess a strong learning capacity. In our work, we study the manifold complexity of mapping a simple circle input through the ViT: h()) /NN [uo cos(() + u sin( Here, N is the dimension of ViT's input (e.g- N = 3 x 224 x 224 for ImageNet images), u and u² form an orthonormal basis for a 2-dimensional subspace of IR- which the circle lives. We further define the ViT network as N,, its input-output Jacobian v0) (g.((((P)) at the input 0, and a()) Qgv( We will calculate expected complexities over a certain number of Os uniformly sampled from [0, 2T). In our work, we study three diffeernnt types of manifold complexities:
1. Curvature can be defined as the reciprocal of the radius of the osculating circle on the ViT's output manifold. Intuitively, a larger curvature indicates that (0)) changes fast at certain 0. According to Riemannian geometry (Lee) 2006; Poole et al.] 2016), the curvature can be explicitly calculated as K = [(((0) v0)) 3/2 (v(() (a)) (0)
(w(0) (0))
2. Length Distortion in Euclidean space is defined LE length(A((0) ʃ V [) It length(P)
measures when the network takes a unit-length curve as input. what the length of the output curve. Since the ground-truth function we want to estimate (using A) is usually very complex, one may also expect that networks with better performance should also generate longer outputs.
3. The problem of L is that, stretched outputs not necessarily translate to complex outputs. A simple example: even an appropriately initialized linear network could grow a straight line into a long output (i.c. a large norm of input-output Jacobian). Therefore, one could instead use Length Distortion taking curvature into consideration to measure how quickly the normalized Jacobian
(0) = v(0) /v(0) v(0) changes with respect to 8, defined as C = / v()) 12d9.
Kontai las 6.43
Rendal 0.72
50
Accuracy (5)
%
0.14
50
02
046
00
1.0 1.4
0.8022
0.0022
1,0000
3.0004
0.0255
11.5
52.8
525
33.8
51.5
Expectto Curvature
Expectes Lengtt Distort
Lougth Disiattonn Qunvature
Figure 2: Correlations between L. C and trained accuracies of VIT topologies from our search space.
In study, aim to compare the potential of using these three complexity metrics to guide the ViT architecture selection. As the core of neural architecture search is to rank the perfor- mance of different architectures, we measure the Kendall-tau correlations (r) between these metries and models" ground-trute truth accuracies. We randomly sampled 87 ViT topologies from Ta- (with I,, T.2 L.g L4 = 1, C = 32), fully train them on ImageNet-1k for 300 epochs (following the same training
Table 2: Complexity Study. Kendall-tau correlation. Time: per VIT topology on average on 1 V100 GPU.
Complexity
Time
K
-0.49
38.3s
C
0.49
12.8s
C
-0.01
48.2s
recipe of DeiT (Touvron et al., ) and also measure their
k. _. LL at initialization. As shown in Figure 2 Kendall-tau correlations. x has a negative correlation, which may indicate that changes of output manifold on the tangent direction direction. Mcanwhile, x costs too much computation time due to second derivatives. We decide to choose L as our complexity measure for highly fast VIT topology search and scaling.
we can clearly see that both x and L exhibit high
more important to ViT training, instead of on the perpendicular
4

---

Published as a conference paper at ICLR 2022
3.3 L AS REWARD FOR SEARCHING VIT TOPOLOGIES
We now propose our training-free search based on _ (Algorithm Most NAS (neural architecture search) methods evaluate the accuracies loss values of single-path super networks proxy inference. This taining-bassdd search will suffer from more computation costs when applied to ViTs. Instead of training ViTs, for each architecture we sample, we calculate _ and treat it as the reward to
guide the search process. In addition to L", we also include the NTK condition number Ke
Amaz
Amip
to indicate the trainability of ViTs (Chen al. 202le Xiao et al. 2019 Yang 2020 Hron et al.
2020).
Amax and Amin are the largest and smallest eigenvalue of NTK matrix +
Algorithm 1: Training- ViT Topology Search.
Input: RI policy т, step 0, total steps T.
while <, T do
Sample topology a, from π.
Calculate C. and Ke,t for dç-
Normalization: E
max, E.y. mine C.y. Ro,t max,r "o,e! mine 1, 1,...
C. 28-Ci c.p
Ne. 1
Update policy x using reward rs = Le Re, by policy gradient Williams 1992). f = +++ 1.
return Topology a" of highest probability x.
We use reinforcement learning (RL) for search. The RL policy is formulated as a joint categorical distribution the choices in Table D and is updated by policy gradient (Williams) 1992). We update policy for 500 steps, which is observed enough for the policy to converge (entropy drops from 15.3 to 5.7). The search process extremely fast: only seven GPU-hours (V100) on ImageNet-Ik, k, thanks to the fast calculation of _ that bypasses the ViT training. To address the different magnitude of _ and Ke, we normalize them by their relative value ranges (line 5 in Algorithm Tb. We summarize the VIT topology statistics from our search in T We can see that _ and Ke highly prefer: (1) tokens with overlaps (K1 ~ K₁ are all larger than strides), and (2) larger FFN expansion ratios in deeper layers (E₁ < E2 < Eg < E1). No clear preference of C and Ke are found on attention splits and number of heads.
Table Statistics of topology search. *Standard deviation is nor- malized by mean due to different value ranges.
Scarch Space
Mean
Std
Ki
7.3
0.1
K₃
0
Kg
0
K4
Ei
3.3
0.4
E2
0.4
Es
4.2
0.3
E4
5.2
0,2
81
0.6
82
2.7
0.5
Ss
Head
42.7
0.5
3.4
AUTOMATIC AND PRINCIPLED SCALING OF VITS
After obtaining an optimal topology, another question is: how to balance the network depth and width? Currently, there is no such rule of thumb for ViT scaling. Recent works try to scale-up or grow convolutional networks of different sizes to meet various resource constraints (Liu al. 2019a
Tan & Le 2019). However, to automatically find a principled scaling rule, training ViTs will cost enormous computation costs. It is also possible search different ViT variants (as in Section 3.3) but that requires multiple runs. Instead, "scaling-up" is a more natural way to generate multiple model variants one experiment. We are therefore motivated scale-up our searched basic "seed" ViT to a larger model in an efficient training-free and principled manner.
We depict our auto-scaling method in Algorithm 2 The starting-point architecture has one attention block for each stage, and an initial hidden dimension C' = 32. In each iteration, we greedily find the optimal depth and width scale-up next. For depth, we try to find out which stage to deepen (i.e.. add one attention block to which stage); for width, we try to discover the best expansion ratio (i.c., widen the channel number to what extent). The rule to choose how to scale-up is by comparing the propagation complexity among set of scaling choices. For example, in the case of four backbone stages (Table [ and four expansion ratio choices (00.05x, x, 0.1x, 0.15x, 0.2x]), we have scaling choices in total for each step. We calculate _ and Ke after applying each choice, and the one with the best C / Ke trade-off (minimal sum of rankings by L and ke) will be selected to scale-up up with. The scaling stops when certain limit of parameter number is reached. In work, we stop the scaling process once the number of parameters reaches 100 million, and the scaling only takes five GPU hours (V100) on ImageNet-Ik.
4 = 16
5

---

Published as a conference paper at ICLR 2022
12000
Swin-B4
+10
Expected Length Distorion (1)
1003
V11-8/116
Total Widths
SwirSS
47
Recla-131
0000
VIT-S/18
4000
Sein-1
Itaddn staling
(our)
07
Sads [Ii it 20711
40
(Cha 2021)
+5
Rarten (He 2015)
16
18
20
22
21
26 28 30
1230 1422
1000
1000
Total Depths
Alp. NTK condition number (L)
Figure 3: Left: Comparing scaling rules from VTT, random scaling. Swin (Luu et al. 2021). VIT Zhai et al. 202T), and ResNet (He et (016). "Total Depths": number of blocks ("bottleneck" of ResNer, "attention- block" of ViTs). "Total Widths": sum of output channel numbers from all blocks. Grey areas indicate standard deviations from runs with different random seeds. Right: During auto-scaling, both the network's 's complexity and trainability improve (numbers indicate scaling-up up steps, L higher the better, Ke lower the better).
The scaling trajectory is visualized in Figure By comparing our automated scaling against random scaling, we find our scaling principle prefers to sacrifice the depths to win more widths, keeping shallower but wider network. Our scaling more similar the rule developed by Zhai et al. (202T). In contrast, ResNet and Swin Transformer (Liu et al. 2021) choose to be narrower and deeper.
Algorithm 2: Training- Auto-Scaling VīTs.
Input: seed As-VIT VIT topology stop criterion (#parameters) P,t O,
channel expansion ratio choices C = {1.05x, 1.1x, 1.15x, L.2x) } (to increase the width by 5%, 10%, 15%, or 20%), depth choices D {[++1111]++ 1, 0, 0)0), , (0, +1,0, 0), (0, 0, +1, (0, 0, +1)3 (to add one more layer to one the four stages in Table
while P > number of parameters of (2 do
for each scaling choice 9i E C
do
Scale-up: dr,; a, +- 9t.
Grow both the channel width and depth.
Calculate L and Ke,i for at,s-
Get ranking of each scaling choice rz, by descendingly sort i 1, |C |Cx D). Get ranking of each scaling choice r by ascendingly sort Ke. i ,,, D). Ascendingly sort each scaling choice gi by Tg + Tre
Select the scaling choice Of with the top (smallest) ranking.
10
a.+1 a. t- g
11
L=
12 return Growed VIT architectures as, ....... Cr.
EFFICIENT VIT TRAINING VIA PROGRESSIVE ELASTIC RE-TOKENIZATION
Recent works (Jia et al. 2018 Zhou
0.8
ct al. 2019 Fu et al. 2020) show that one mixed or progressive preci- sion to achieve an efficient training pur- pose. The rationale behind this strategy is that, there exist some "short-cuts" on the network's loss landscape that can be manually created to bypass perhaps less important gradient descent steps, espe- cially during early training phases. As in ViT, both self-attention and FFN have quadratic computation costs to the num- ber of tokens. It is therefore natural to ask: do need full-resolution tokens during the whole training process?
0.6
strday
dission
Accuracy
R: -
FLOR: 2855%
0.4
strider 16
diaatto,,
FLOP-
0.2
Small #Toleens
Large 4Tokens
Coamee Sampling
Fine line-grained Sampling
150
250
Epoch
Figure 4: By progressively changing sampling granularity (stride and dilation) of the first linear project layer, we can re- duce the spatial resolutions of tokens and save training FLOPs (37. 4% here), while still maintain competitive final perfor- mance (ImageNet- 1K 224 224). See Table more studies.
We provide affirming answer by proposing a progressive elastic re-tokenization training strategy. To update the number of tokens during training without affecting the shape of weights in linear projections, adopt different sampling granularities in the first linear projection layer. Taking the

---

Published as a conference paper at ICLR 2022
first projection kernel K1 with stride 4 an example: during training gradually change the (stride, dilation) pair of the first projection kernel to (16, 5) (8, 2), and (4, 1), keeping the shape of weights and the architecture unchanged.
This re-tokenization strategy emulates curricu- lum learning for ViTs: when the training be- gins, we introduce coarse sampling to signif- icantly reduce the number of tokens. In other words, our As-Vi ViT quickly learns coarse infor- mation from images in early training stages at extremely low computation cost (only 13.2% FLOPss of full-resolution training). Towards the late phase of training, we progressively switch to fine-grained sampling, restore the full token resolution, and maintain the com- petitive accuracy. As shown in Figure 4 when the ViT trained with coarse sampling early training phases, it can still obtain high accuracy while requiring extremely low computation cost. The transition between different sampling granularity introduces jump in performance, and eventually the network restores its competitive final performance.
Table 4: ViT topology and scaling rule.
Design
Stage
S
E
Head
#1
2
3
Soed Topology
2
(Blire italies in Fig
32
Stago-wisc Depth
Scaling (Red Fig.
Width (C)
L44
As-ViT-Small
As- ---Vee-Base
2
116
Larg
180
5
EXPERIMENTS
Table 5: Image Classification ImageNet- Ik (224 224).
Method
Params.
FLOPs
Top-1
5.1 As-VIT: AUTO- SCALING VIT
RegNetY-EGGF
(Raousowic et al
2020
21.0 M
4.0 B
80.0%
We show searched As-ViT topology in T This architecture facilitates strong overlaps among tokens during both the first projection ("tokenization") step and three re-embedding steps. FFN expansion ratios are first narrow then become wider in deeper layers. A small number of attention splits are leveraged for better aggregation of global information.
VIT-S
Dosavisskly et al.
2020
22.1 M
9.2
81.2%
DeiT-S
Jouyron
2020
22.0 M
4.6
79.8%
T2T-ViT-1
14
Yoar et 202
21.5 M
6.1
81.7%
TNT-S
Han 2121
23.8
5.2
81.5%
PVT-Smal
Wing el 2021
24.5 M
3.8
79.8%
Cai' XS-24
Touvron et
2021
26.6 M
5.4
81.8%
Veep
Zhou et al.
2021
6.2
82.3%
ConViT-S
Asonli
5.4 R
81.3%
CvT-13
We al.
2021
4.5 B 81.6%
CvT-21
Wa
X021
7.1 82.5%
Swin-T
Liu
2121
29.0 M
4.5 B 81.3%
The seed topology is automatically scaled- up, and three As- variants of compara- ble sizes with previous works will be bench- marked. Our scaling rule prefers shallower and wider networks, and layers are more bal- anced among different resolution stages.
BossNet-TU
2012
3.4 R 80.8%
AutoFormer-s
Chen
2021e
22.9 M
5.1 B 81.7%
GLi'--mall
hen al.
2022
24.6 M
4.4 B 80.5%
As- Small (ours)
29.0 M
5.3 B
81.2%
RegNerY-BGF
Radosmoviic at.
2020
39.0
8.0
81.7%
T2T-VTT-P
Yuan et al
2021bi
9.8
82.2%
CaiT S-24
46.9
9.4
82.7%
ConViT-S+
A'soolli
2021
48 M
82.2% 10
VIT-S/16 Desovisslly et al.
48.6 M
20.2
78.1%
5.2 IMAGE CLASSIFICATION
Swin-S
Liu 2021
50.0
8.7 83.0%
ViT-V
Zhou al 2021
55 M
12.5 B 82.2%
Settings. We benchmark our Vi- on ImageNet-Ik kk (Deng el al. 2009). We use Tensorflow and Keras for training implemen- tations and conduct all training on TPUs. We set the default image size as 224 224, and use Adam W (Loshchilov & Hutter, 2017) as the optimizer with cosine learning rate decay (Loshchilov & Hutter 2016). A batch size 1024, an initial learning rate of 0.001, and weight decay of 0.05 are adopted.
PVT-Medium
Wang
202
44.2 M
6.7 B 81.2%
PVT-Large
Wang
2021
61.4 M
9.8 81.7%
T2T-ViT-24
Yuan et all 202
64.1 M
15.0 B 82.6%
TNT-B
Hon al. 2021
65.6 M.
14.1 B 82.8%
BossNet-TT
Li 2021
7.9
82.2%
AutoFormer-6
hen et al
202
54M
11 B
82.4%
ViT-ResNAS...
Lia et
3021
41 M
1.8
80.8%
ViT-ResNAS-s
Lino et
2021
2.8
81.4%
As-VIT Base (ours)
52.6 M
8.9 B
82.5%
RegNetY-16GF Radossvovic et
2020
84.0 M
16.0 B8 82.9%
ViT-B/16
Dowoviskki al.
2020
86.0 M
55.4 18 77.9%
DeñT-B
Tousson al. 320
86.0 M
17.5 B 81.8%
Table demonstrates comparisons of our As- Vit to other models. Compared to the previous both Transformer-based and CNN- based architectures, As-Vit ViT achieves state- of-the-art performance with a comparable number parameters and FLOPs.
ConViT-B
d A'Asoll et
202
86 M
17
82.4%
Swin-B
Liu
2021
88.0 M
15.4 83.3%
GLiT-Buse
Chen al., 202
96.1 M
17.0 82.3%
m Liso et
2021
97 M
4.5 B 82.4%
CaiT S-48
Touvroo et al. 2021
89.5 M
18.6 B 83.5%
As-VIT Large (ours)
88.1 M
22.6 B 83.5%
Under 384 384 resolution.
'dilation (ourdd(st 1) K2/(K1 1)) + S1 is the stride the full token resolution.
7

---

Published as a conference paper at ICLR 2022
More importantly. our As- ViT framework achieves competitive stronger performance than con- current NAS works for ViTs with much more search efficiency. As- ViTs are designed with highly reduced human NAS efforts. All our three As- ViT variants are generated in only 12 GPU hours (on a single V100 GPU). In contrast, BoneNAS (Li et al. 2021) requires 10 GPU days to search a single architecture. For each variant of ViT-ResNAS [iao et al. 2021), the super- super-network training takes 16.7~21 hours, followed by another 6 hours of evolutionary search.
Efficient Training. We lever- age the progressive elastic re- tokenization strategy proposed in
Table 6: Efficient training on ImageNet- 1k (224 x 224) via pro- gressive elastic re-tokenization strategy (Section I 4x (resp. 2x) indicates reduce the number of tokens by 4 (resp. 2) times, and "N/A" indicates no token reduction.
Section 4 reduce both FLOPs
and training time for large VIT models. As illustrated Figure 4 we progressively apply 4x and 2x reductions on the number of tokens
Token Reduction (Epochs)
FLOPs
Training Time (TPU days)
Topl Acc.
Saving
4x
2x
N/A
1~440
41~-70
71~~300
18.7%
36.9
83.1%
1~880
81.~140
141~300
37.4%
31.0
82.9%
during training by changing both the dilation and the stride first linear projection layer. We tune the epochs allocated to each token reduction stage and show the results in Table Standard training takes 42.8 TPU days, whereas our efficient training could save up to 56.2% training FLOPs and 41.1% training TPU days, still achieving strong accuracy.
the
1~~100
121-2100
211~3300
56.2%
25.2
82.5%
Baseline
100%
42.8
83.5
Table 7: Decoupling the contributions from the seed topology and the scaling, on ImageNet-IK. K.
Disentangled Contributions from Topology and Scaling, To better verify the contribution from our searched topology and scaling rule, we conduct more ablation studies (Table 7). First, we directly train the searched topology before scaling. Our searched seed topology is better than the best from 87 random
Model
Params.
FLOPs
Top-1
ViT Topology
2.4 M
0.5 B 61.7%
Random Topology
2.2 M
0.4 B
61.4%
As-ViT Small
29.0
5.3 B
81.2%
topologies Figure 2 Second, we compare our complexity-basee scaling rule with "random scaling + As- ViT topology". At different scales, our auto- mated scaling is also better than random scaling.
Random Scaling
24.2
8.7
80.5%
As-VIT Base
52.6 M
8,9
82.5%
Random Scaling
42.4 M
15.5 B
82.2%
As- Large
88.1 M
22.6 B
83.5%
Random Scaling
81.1 M
28.7 B
83.2%
5.3 OBJECT DETECTION ON COCO
Settings Beyond image classification, we further evaluate designed As-ViT on the detection task. Object detection is conducted on COCO 2017 that contains 1 118,000 training and 5000 validation images. We adopt the popular Cascade Mask R-CNN the object detection framework for our As-ViT. We use an input size of 1024 x 1024, Adam optimizer (initial learning rate of 0.001), weight decay of 0.0001, and batch size of 256. Efficiently pretrained ImageNet- checkpoint (82.9% in Table 6 is leveraged as the initialization.
We compare our As-ViT standard CNN (ResNet) and previous Transformer network (Swin (Liu et al. )).T). The comparisons are conducted by changing only the backbones with other settings unchanged. Table g we can see that our As- ViT can also capture multi-scale features and achieve state-of-the-art detection performance, although being designed on ImageNet and its complexity is measured for classification.
Table 8: Twa-stage object detection and instance segmentation results. We compare employing different.
backbones with Cascade Mask R-CNN on single model without test-time augmentation.
Backbone
Resolution
FLOPs
Params.
AP
Apm
ResNet- 152
480~~800 1333
527.7 B
96.7 M
49.1
42.1
Swin-B (Liu et al. 2021
480~800x 1333
982 B
145 M
51.9
45
As-Vit Vi Large (ours)
1024 1024
1094.2
138.8 M
52.7
45.2
8

---

Published as a conference paper at ICLR 2022
6
RELATED WORKS
6.1 VISION TRANSFORMER
Transformers Vaswwan al., 2017) leverage the self-attention to extract global correlation, and
become the dominant models for natural language processing (NLP) (Devlin et al. 20188 Radford
ett al.
2018 [Brown et al., 2020
Liu et al. 2019b). Recent works explored transformers to vision
problems: image classification Dosovitskiy et al. 2020), object detection (Carion et al., 2020 Zhu
cct al. 2020:
Zheng et al. 2020; Dai et al. 2020; Sun et al. (020),, segmentation (Chen et al. 2020
Wang et al. 2020), etc. The Vision Transformer (ViT) (Dosovitskiy et al., 2020) designed a pure
transformer architecture and achieved SOTA performance on image classification. However, VIT
heavily relies on large-scale datasets (ImageNet-211 (Deng et al.) 2009), JFT-300M (Sun al. 2017)) for pretraining, requiring huge computation resources. DeiT (Touvron et al. (0200) proposed Knowledge Distillation (KD) (Hinton et al.] 2015 Yuan et al. 2020) via special KD token to improve both performance and training efficiency. In contrast, our proposed As-ViT introduces more flexible tokenization, attention splitting, and FFN expansion strategies, with automated discovery.
6.2 NEURAL ARCHITECTURE DESIGN AND SCALE
Manual design of network architectures heavily relies on human prior, which is difficult to scale-up. Recent works leverage AutoML to find optimal combinations of operators/opology in a given search space (Zoph & Le 2016 Real et al. 2019: Liu et al.) 2018: Dong & Yang 2019). However, the searched model are small due to the fixed and hand-crafted search space, far from being scaled-up to modern networks. For example, models from NASNet space (Zoph et al. 2018) only have ~MM parameters, much smaller than real-world ones (20 to over 100M). One main reason for not being scalable is because NAS is computation-consumiig task, typically costing I~2 GPU days to search even small architectures. Meanwhile, many works try to grow a "seed" architecture to different variants. EfficientNet (Tan & Le, 2019) manually designed scaling rule for width and depth. Give a template backbone with fixed depth. Liu et al. (2019a) grow the width by gradient descent. For ViT, we for the first time bring both architecture design and scaling together in one framework. To overcome the complexity of manifold propagation as a surrogate measure towards a training- free search and scale.
problem in the training of transformers, we directly use the
6.3 EFFICIENT TRAINING
A number of methods have been developed to accelerate the training of deep neural networks, including mixed precision (Jia et al., 2018). distributed optimization (Cho et al.) (017). large-batch training (Goyal et al., 2017] Akiba et al. 2017, You et al.) 2018). etc. Jia et al. (2018 8) combined distributed training with a mixed-precistee
framework. Wang al. (2019) proposed save deep CNN training energy cost via stochastic mini-batch dropping and selective layer update. In our work, customized progressive tokenization via the changing of stride/dilation can effectively reduce the number of tokens during ViT training, thus largely saving the training cost.
7 CONCLUSIONS
To automate the principled design of vision transformers without tedious human efforts, we propose As- unified framework that searches and scales VITs without any training. Compared with hand-crafted ViT architecture, our As- ViT leverages more token overlaps, increased FFN cxpansion ratios, and is wider and shallower. Our VIT achieves state-of-the-at accuracies on both ImageNet- 1K classification and COCO detection, which verifies the strong performance of our framework. Moreover, with progressive tokenization, can train heavy VIT models with largely reduced training FLOPs and time. We hope our methodology could encourage the efficient design and training of ViT's for both the transformer and the NAS communities.
ACKNOWLEDGEEMENT
Z. Z.W. is in part supported by the NSF AI Institute for Foundations of Machine Learning (IFML) and a Google TensorFlow Model Garden Award.

---

Published as a conference paper at ICLR 2022
REFERENCES
Mohamed S Abdelfattah, Abhinav Mehrotra, łukasz Dudziak, and Nicholas D Lane. Zero-cost proxies for lightweight nas. arXiv preprint arXiv:2101.08/33 2021.
Takuya Akiba, Shuji Suzuki, and Keisuke Fukuda. Extremely large minibatch sgd: Training resnet-50 on imagenet 15 minutes. CoRR, abs/17111.433 1.4325, 2017.
Kartikeya Bhardwaj. Guihong Li, and Radu Marculescu. How does topology influence gradient propagation and model performance of deep networks with densenet-type skip connections? In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 13498-13500 2021.
Navaneeth Bodla, Bharat Singh, Rama Chellappa, and Larry S Davis. Soft-nms oms-improving object detection with one line of code. In Proceedings of the IEEE international conference on computer vision, pp. 5561-5569, 2017.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. arXiv preprint arXiv:2005. 14165, 2020.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. arXiv preprint arXi::2005.1281 2020.
Boyu Chen, Peixia Li, Chuming Li, Baopu Li, Lei Bai, Chen Lin, Ming Sun, Wanli Ouyang, et Glit: Neural architecture search for global and local image transformer. arXiv preprini arXiv:2107.02960, 2021a
Chun-Fu Chen, Quanfu Fan, and Rameswar Panda. Crossvit: Cross-attention multi-scale vision transformer for image classification. arXiv preprint arXiv:2103. 2021b.
Hanting Chen, Yunhe Wang. Tianyu Guo, Chang Xu, Yiping Deng, Zhenhua Liu, Siwei Ma. Chun jing Xu, Chao Xu, and Wen Gao. Pre-trained image processing transformer. arXiv preprint arXiv:2012.00364. 2020.
Kai Chen, Jiangmiao Pang, Jiaqi Wang, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jianping Shi, Wanli Ouyang, et al. Hybrid task cascade for instance segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp.
4974-4983, 2019.
Liang-Chich Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder- decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European conference on computer vision (ECCV), pp. 801 211-818. 2018.
Minghao Chen, Houwen Peng, Jianlong Fu, and Haibin Ling, Autoformer: Searching transformers for visual recognition. arXiv preprint arXiv:2107.00 00651, 2021c.
Tianlong Chen, Yu Cheng, Zhe Gan, Lu Yuan, Lei Zhang, and Zhangyang Wang. Chasing sparsity in vision transformers: An end-to-end exploration. Advances Neural Information Processing Systems, 34, 2021d.
Wuyang Chen, Xinyu Gong, and Zhangyang Wang. Neural architecture search on imagenet in four gpu hours: A theoretically inspired perspective. International Conference Learning Representations (ICLR), 2021e.
Wuyang Chen, Xinyu Gong, Yunchao Wei, Humphrey Shi, Zhicheng Yan, Yi Yang, and Zhangyang Wang. Understanding and accelerating neural architecture search with training- free and theory- grounded metrics. arXiv preprint arXiv:2 /939, 2021f.
Minsik Cho, Ulrih Finkler, Sameer Kumar, David Kung, Vaibhav Saxena, and Dheeraj Sreedhar. Powerai ddl. arXiv preprint arXiv: 1708.02 188, 2017.
10

---

Published as a conference paper at ICLR 2022
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with reduced search space. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition Workshops, pp. 702-703, 2020.
Zhigang Dai, Bolun Cai, Yugeng Lin, and Junying Chen. Up-detr: Unsupervised pre-training for object detection with transformers. arXiv preprint arXiv:201 2020.
Stéphane d' Ascoli, Hugo Touvron, Matthew Leavitt, Ari Morcos, Giulio Biroli, and Levent Sagun. Convit: Improving vision transformers with soft convolutional inductive biases. arXiv preprint arXiv:2103. 10697, 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. leee, 2009.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.044 04805, 2018.
Xuanyi Dong and Yi Yang. Searching for a robust neural architecture four gpu hours. In Proceedings of the IEEE Conference on computer vision and pattern recognition, pp. 1761-1770, 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 11929, 2020.
Yonggan Fu, Haoran You, Yang Zhao, Yue Wang, Chaojian Li, Kailash Gopalakrishnan, Zhangyang Wang, and Yingyan Lin. Fractrain: Fractionally squeezing bit savings both temporally and spatially for efficient dnn training. Advances in Neural Information Processing Systems, 33:12127-12133 2020.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in hour. arXiv preprint 170.007706.0677 2017.
Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, and Yunhe Wang. Transformer in transformer. arXiv preprint arXiv:2103.00112, 2021.
Ali Hassani, Steven Walton, Nikhil Shah, Abulikemu Abuduweili, Jiachen Li, and Humphrey Shi. Escaping the big data paradigm with compact transformers. arXiv preprint arXiv:2104.055 0500000 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint arXiv:1503.0253: 2015.
Jiri Hron, Yasaman Bahri, Jascha Sohl-Dickstein, and Roman Novak. Infinite attention: Nogp and ntk for deep attention networks. In International Conference on Machine Learning, 4376-4386. PMLR, 2020.
Gao Huang. Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In European conference computer vision, pp. 646-661 Springer, 2016.
Xianyan Jia, Shutao Song. Wei He. Yangzihao Wang. Haidong Rong, Feihu Zhou, Liqiang Xie, Zhenyu Guo, Yuanzhou Yang, Liwei Yu, et al. Highly scalable deep learning training system with mixed-precision: Training imagenet in four minutes. arXiv preprint arXiv: /205, 2018.
Yifan Jiang, Shiyu Chang, and Zhangyang Wang. Transgan: Two pure transformers can make one strong gan, and that can scale up. Advances in Neural Information Processing Systems, 34, 2021.

---

Published as a conference paper at ICLR 2022
John M Lee. Riemannian manifolds: an introduction curvature, volume 176. Springer Science & Business Media, 2006.
Changlin Li, Tao Tang, Guangrun Wang, Jiefeng Peng, Bing Wang. Xiaodan Liang, and Xiaojun Exploring hybrid cnn-transformers with block-wisely self-supervised neural architecture search. arXiv preprint arXiv:2103.1242: 12424, 2021.
Chang. Bossnas:
Yi-Lun Liao, Sertac Karaman, and Vivienne Sze. Searching for efficient multi-stage vision transform- ers. arXiv preprint MxXiv:2/09.0044: 2021.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search, arXiv preprint arXiv:1806.09000 2018.
Qiang Liu, Lemeng Wu, and Dilin Wang. Splitting steepest descent for growing neural architectures. 2019a.
arXiv preprint
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1907. 1699, 2019b.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hicrarchical vision transformer using shifted windows. arXiv preprint 1:Xi::2/03.14030 2021.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint orXiv:1608.03990 2016.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 2017.
Joseph Mellor. Jack Tumer, Amos Storkey, and Elliot J Crowley. Neural architecture search without training. arXiv preprint urXiv:2006.0464/ 2020.
Bowen Pan, Rameswar Panda, Yifan Jiang, Zhangyang Wang, Rogerio Feris, and Aude Oliva. Ia- red?: Interpretability-awwe redundancy reduction for vision transformers. Advances in Neural Information Processing Systems, 34, 2021.
Ben Poole, Subhaneil Lahiri, Maithra Raghu, Jascha Sohl-Dickstein, and Surya Ganguli. Exponential expressivity in deep neural networks through transient chaos. arXiv preprint 1rXiv:1606.05..0 2016.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under- standing by generative pre-training, 2018.
Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick. Kaiming He, and Piotr Dollár. Designing network design spaces. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10428-10436, 2020.
Maithra Raghu, Thomas Unterthiner, Simon Komblith, Chiyuan Zhang, and Alexey Dosovitskiy. Do vision transformers see like convolutional neural networks? arXiv preprint arXiv:2108.06. 08810, 2021.
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoe V Le. Regularized evolution for image classifier architecture search. In Proceedings of the aaai conference on artificial intelligence, volume 33, pp. 4780-4789, 2019.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint 1409. 2014.
Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable effectiveness of data in deep learning era. In Proceedings of the IEEE international conference on computer vision, pp. 843-852, 2017.
Zhiqing Sun, Shengcao Cao, Yiming Yang, and Kris Kitani. Rethinking transformer-based set prediction for object detection. arXiv preprint arXiv:2011. 10881, 2020.
12

---

Published as a conference paper at ICLR 2022
Mingxing Tan and Quoc V Le. Efficientnet: Rethinking model scaling for convolutional neural networks. arXiv preprint 1XX::1905.119400 2019.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé
Jégou. Training data-efficient image transformers distillation through attention. arXiv preprint arXiv:2012.12877 2020.
Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. arXiv preprint v:Xiv22103172255 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30:5998-600 6008, 2017.
Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. arXiv preprint arXiv:2102. /2/22, 2021.
Yue Wang, Ziyu Jiang, Xiaohan Chen, Pengfei Xu, Yang Zhao, Yingyan Lin, and Zhangyang Wang. E2-train: Training state-of-the-ar cnns with over 80% energy savings. arXiv preprint a9Xiv: 13349, 2019.
Yuqing Wang, Zhaoliang Xu, Xinlong Wang, Chunhua Shen, Baoshan Cheng, Hao Shen, and Huaxia Xia. End-to-end video instance segmentation with transformers. arXiv preprint urXiv:2011.14503, 2020.
Ronald J Williams. Simple statistical folliin algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229-256, 1992.
Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. Cvt:
Introducing convolutions to vision transformers. arXiv preprint arXiv:2103.15800 2021.
Lechao Xiao, Jeffrey Pennington, and Samuel S Schoenholz. Disentangling trainability and general- ization in deep learning. arXiv preprint 1912. 2019.
Greg Yang. Tensor programs i:: Neural tangent kernel for any architecture. arXiv preprint arXiv:2006.14542 2020.
Yang You, Zhao Zhang, Cho-Jui Hsieh, James Demmel, and Kurt Keutzer. Imagenet training in minutes. Proceedings of the 47th International Conference on Parallel Processing ICPP 2018, 2018. doi: 10. |11144/32225558322555666 URL http://dx doi, wwo.or///// 0tgg/00.114/3325505 3225069
Kun Yuan, Shaopeng Guo, Ziwei Liu, Aojun Zhou, Fengwei Yu, and Wei Wu. Incorporating convolution designs into visual transformers. arXiv preprint arXiv:2103.11816, 2021a.
Li Yuan, Francis EH Tay, Guilin Li, Tao Wang, and Jiashi Feng. Revisiting knowledge distillation via label smoothing regularization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattem Recognition, pp. 1903-3911, 2020.
Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Francis EH Tay, Jiashi Feng, and Shuicheng Yan. Tokenn-to-token vit: Training vision transformers from scratch on imagenet. arXiv preprint arXiv:2101. 2021b.
Sangdoo Yun, Dongyoon Han, Scong Joon Oh. Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6023 6032, 2019.
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang. Li Yang, et al. Big bird: Transformers for longer sequences. In NeurIPS, 2020.
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. arXiv preprint arXiv: 106.0006000 2021.
13

---

Published as a conference paper at ICLR 2022
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412, 2017.
Pengchuan Zhang, Xiyang Dai, Jianwei Yang, Bin Xiao, Lu Yuan, Lei Zhang, and Jianfeng Gao. Multi-scale vision longformer: A new vision transformer for high-resolution image encoding. arXiv preprint arXiv:2103.1 15358, 2021.
Minghang Zheng, Peng Gao, Xiaogang Wang, Hongsheng Li, and Hao Dong. End-to-end object detection with adaptive clustering transformer. arXiv preprint arXiv:201 1.09315, 2020.
Daquan Zhou, Bingyi Kang, Xiaojie Jin, Linjie Yang, Xiaochen Lian, Zihang Jiang, Qibin Hou, and Jiashi Feng. Deepvit: Towards deeper vision transformer. arXiv preprint arXiv:2103./1885 2021.
Zhengguang Zhou, Wengang Zhou, Xutao Lv, Xuan Huang, Xiaoyu Wang, and Houqiang Li.
Progressive learning of low-precision networks. arXiv preprint arXiv: 11781,
Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers end-to-end object detection. arXiv preprint urXiv:2010.04159, 2020.
Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning, arXiv preprint 1rXiv:1611.0157. 2016.
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoe V Le. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8697-8710, 2018.
A IMPLEMENTATIONS
Training-free Topology Search and Scaling, We calculate _ by uniformly sampling 10 0s in [0, 2m). For one architecture, the calculation of _ is repeated five times with different (random) network initializations, and L is set as their mean.
Image Classification. We use epochs linear warm-up, a batch size of 1,024, an initial learning rate of 0.001, and weight decay of 0.05. Augmentations including stochastic depth (Huang et al. 2016), Mixup (Zhang et al.] 2017), Cutmix (Yun et al., 2019). RandAug (Cubuk et al.) 2020). Exponential Moving Average (EMA) also applied.
Object Detection. Our training adopts batch size of 256 for 36 epochs, with also stochastic depth. We do not use any stronger techniques like HTC (Chen et 2019), multi-scale testing, sotf-NMS (Bodla et al. 2017). etc.
B CONVERGENCE OF TRAINING-FREE SEARCH
To demonstrate the convergence of the pol- icy learned by our RL search, we show the entropy during learning the policy in Fig- ure 5 We can see that training of 500 steps is enough for the policy converge to low entropy (high confidence).
14
12
10
Step
Figure 5: Entropy of policy during search (Section 3.3
14


Reviews:
[{"review_id": "tS7024kiIFf", "reviewer": "Reviewer_Bp2F", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n\n- A two-stage approach is followed for architecture search. First, there is a search for seed architecture using a training-free strategy. The seed ViT is found from existing designs. Second, the seed is scaled upon along width and depth.\n\n- the paper also proposes elastic-tokens: progressive re-tokenization for fast ViT training.\n\n- While the design search is efficient: requires only 7+5 GPU hours for seed+scaling stages. Compared with existing NAS based approaches, the proposed method requires less time to search for the optimal architecture. However, the final searched architecture by the proposed method has more FLOPs than existing methods e.g., ViT-ResNAS-t with 1.8B FLOPs compared with 8.9B of the proposed architecture, with slightly better performance.", "summary_of_the_paper": "Initial variants of ViTs borrow their architecture from the seminal ‘attention is all you need’ work. Some recent methods amend the default transformer architecture to incorporate convolutions in the design (e.g., CCT: Compact Convolutional Transformers), or a hierarchical architecture with feature pyramids (e.g., Swin, PyramidViT etc) to make ViTs suitable for dense prediction tasks (detection and segmentation). \nInstead of relying on hand-designed architectures, the paper proposes to automatically search for a ViT architecture, that is both efficient & accurate. ", "summary_of_the_review": "\n- It is unclear if the proposed strategy works to search a ViT architecture suitable for small-scale datasets?\n\n- The advantages of automatically searched ViT architectures are not immediately clear. Their major limitation for ViTs is the time they require for training, and the number of FLOPs introduced by quadratic complexity in self-attention. Compared with existing “hand-designed” ViT architectures, the proposed automatically searched ViT architecture does not significantly reduce compute training time. \n\n- I believe an automatically searched ViT architecture, that is efficient to train, will help democratize ViTs research, and make it accessible to broader resource-constrained academic labs.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "5yfk-oVVsw5", "reviewer": "Reviewer_rFuj", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros: \n-1- As-ViT is a novel all-in-one efficient framework: it first finds a promising “seed” topology for ViT of minimal depths and widths, then progressively “grow” it into different s capacities to meet different needs, and finally “train” the desired architecture. All three steps are coherently designed to be very efficient, which is important for designing and verifying ViTs.\n-2- For searching ViTs, the seed topology space is relaxed from recent vanilla ViT designs. To compare different topologies, the authors automate this process by a training-free architecture search approach via measuring the network complexity, which is extremely fast and efficient. \nWhile several previous works already explored NAS for CNN with gradient features or linear regions, transformer has more nonlinear structures and different components. The authors introduced a new general complexity metric of manifold propagation that can be applicable to transformers. The proposed training-free search is experimentally supported by a comprehensive study involving measure time costs and Kendall-tau correlations.\n-3- The “seed” ViT topology is then progressively scaled up from a small to a large network, generating a series of ViT variants in a single run. At each step, how to increase the depth and width is automatically determined by comparing network complexities of different scaling choices, being both principled and efficient. The authors claimed they discovered ViT scaling principles that echo the empirical observation of (Zhai 2021).\n-4- Further, to address the heavy training costs of ViTs, the authors proposed to make ViT tokens elastic, an idea similar to dynamically downsampling image resolution for efficient CNN training. They then propose a progressive re-tokenization method for efficient ViT training. The idea is not particularly novel but is the first time of this type in ViTs.\n-5- The authors made comprehensive comparisons with various SOTA ViTs and their NAS methods, on not only ImageNet-1K classification but also COCO detection. As-ViT performs better than or competitively to most of them, meanwhile largely saving training FLOPs and time so that is also impressive. However, I have some reservations regarding their discussion of results, as to be laid out below.\n\nThe paper writing is overall good. The results and findings suggest potentially broad impact of this work.\n\nCons:\n-1- Comparison to Prior Arts of NAS for ViTs\nMy main issue is that, from table 5, I cannot conclude the same as the authors stated “As-ViT framework significantly outperforms concurrent NAS works for ViTs in both performances and search efficiency”. For example, BossNet has marginal inferior accuracies but also smaller models; the more recent AutoFormer has also the same accuracies for Small/Base, only costing more FLOPs at Base level. \nI understand that their search space comparison is not apple-to-apple (e.g.,  BossNet used conv layers too); and As-ViT has other selling points such as efficient search and training. However, the authors are urged to put their accuracy comparison in the fairer and more detailed context, and the current conclusions look a bit over-rush and misleading to me.\n-2- Comparison to Swin Transformers\nFor image classification, the reported performance of As-Vit is slight less competitive than Swin at comparable costs. Again, I understand the two are not fully apple-to-apple comparable due to vastly different building blocks, and I am not asking As-ViT to outperform Swin since the former’s current search space is more vanilla.  Yet, as Swin follows a different scaling philosophy to be narrower and deeper, how can you then be convinced that your discovered “shallower but wider” principle is optimal?\nAlso, in object detection, As-ViT Large seems to outperform Swin-B, but the comparison is questionable to me: why the image resolutions are not aligned between the two? If Swin on averages sees smaller image inputs, it can possibly explain why its FLOPs and AP are both lower than As-ViT, and a fairer comparison should be appended.\nFurther, the current As-ViT space searches for attention splits, which seems to be a simplified sliding window without overlapping. Then, could you expand your search space to include Swin-type architecture into your search space? To be fair, I am not asking the authors to perform so during the rebuttal timeframe. I’m instead asking to address the above fairness issues.\n", "summary_of_the_paper": "The paper proposed As-ViT to automate the principled design of vision transformers without tedious human efforts. To my best knowledge, it is the first framework that unifies efficient search, scaling and training in ViTs. The empirical performance is in general satisfactory. ", "summary_of_the_review": "The biggest merit of this paper is to unify efficient search, scaling and training as one -- that is unique and strong. The results prove their concepts and can be considered as promising since the adopted search space is vanilla. However, the authors are expected to clarify a number of experiment and comparison issues as aforementioned. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "5J7nM7XxPW", "reviewer": "Reviewer_eYy9", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n+The problem is significant\n\nViT training is time-consuming and the input resolution is fixed for a specific ViT model, which makes it important to have a proper scaling rule for choosing the size of ViT. The search-based auto-scaling rule proposed in this paper is a good way to solve this issue.\n\n+NAS on ViT\n\nThe author uses neural architecture search on ViT-like framework on the number of kernels, attention splits, expansion ratio, depth and width jointly. The searched architecture are stage-wise and can be used for down-stream tasks like detection, which is another technical contribution of this paper.\n\nWeaknesses:\n\n-Comparison with other methods\n\nCompared with original single-scale ViT or a multi-scale vision transformer like Swin, the proposed AS-ViT does not outperform them (ViT-S 81.2%, Swin-T 81.3%, AS-S 81.2%; Swin-S 83.0%, AS-B 82.5%; Swin-B 83.3%, AS-L 83.5%). Swin here can be viewed as an augmented multi-scale vision transformer. The author probably needs to propose a Multi-scale ViT baseline for comparison here and show the effectiveness of the searched AS-ViT. ", "summary_of_the_paper": "In this paper, the author proposes auto-scaling ViT, which is to search seed ViT topology based on number of kernels, attention splits, expansion ratio, depth and width jointly. The searched ViT is trained with a progressive re-tokenization scheme that saves ~40% training time and preserves the accuracy with less parameters. Experiments on ImageNet and MSCOCO show that AS-ViT can reach strong performances on classification and detection.", "summary_of_the_review": "The paper AS-ViT is focusing on solving the scaling issue of ViT with searched topology and auto-scaling rules. However, it does not show significant improvement compared with other multi-scale vision transformers and my concern is that a multi-scale ViT baseline for fair comparison and evaluation is needed here. I will increase my rating if the concerns are well addressed here.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces As-ViT, an interesting framework for searching and scaling ViTs without training. Overall, the paper received positive reviews. On the other hand, R1 rated the paper as marginally below the threshold, raising concerns about search on small datasets and issues regarding the comparison in terms of FLOPS/accuracy with other methods. The authors adequately addressed these concerns in the rebuttal, and helped clarify other questions by R2 and R3. R1 did not participate in the discussion after the author response nor updated his/her review. The AC agrees with R2 and R3 that the paper passes the acceptance bar of ICLR, as the unified approach for efficient search/scaling/training is novel and should be interesting to the ICLR audience.
Decision: Accept (Poster)