Searching the Search Space of Vision Transformer
Authors: Minghao Chen
Kan Wu
Bolin Ni
Houwen Peng
Bei Liu
Jianlong Fu
Hongyang Chao
Haibin Ling
Venue: NeurIPS 2021 Poster (2021)
Keywords: Vision Transformer
Neural Architecture Search
Design Space

Abstract: Vision Transformer has shown great visual representation power in substantial vision tasks such as recognition and detection, and thus been attracting fast-growing efforts on manually designing more effective architectures. In this paper, we propose to use neural architecture search to automate this process, by searching not only the architecture but also the search space. The central idea is to gradually evolve different search dimensions guided by their E-T Error computed using a weight-sharing supernet. Moreover, we provide design guidelines of general vision transformers with extensive analysis according to the space searching process, which could promote the understanding of vision transformer. Remarkably, the searched models, named S3 (short for Searching the Search Space), from the searched space achieve superior performance to recently proposed models, such as Swin, DeiT and ViT, when evaluated on ImageNet. The effectiveness of S3 is also illustrated on object detection, semantic segmentation and visual question answering, demonstrating its generality to downstream vision and vision-language tasks. Code and models will be available at https://github.com/microsoft/Cream.

Searching the Search Space of Vision Transformer
Minghao Chen Kan Wu?*", Bolin Ni Houwen Peng Bei Liu', Jianlong Fu", Hongyang Chao?, Haibin Ling' Sttony Brook University, "Sun Yat-sen University, Institute of Automation, CAS, "Microsoft Research
Abstract
Vision Transformer has shown great visual representation power in substantial vision tasks such as recognition and detection, and thus been attracting fast-growing efforts on manually designing more effective architectures. In this paper, we propose to use neural architecture search to automate this process, by searching not only the architecture but also the search space. The central idea to gradually evolve different search dimensions guided by their E-T Error computed using a weight-sharing supernet. Moreover, we provide design guidelines of general vision transformers with extensive analysis according to the space searching process, which could promote the understanding of vision transformer. Remarkably, the searched models, named S3 (short for Searching the Search Space), from the searched space achieve superior performance to recently proposed models, such Swin, DeiT and ViT, when evaluated on ImageNet. The effectiveness of S3 is also illustrated on object detection, semantic segmentation and visual question answering, demonstrating its generality downstream vision and vision- tasks. Code and models will be available at here.
Introduction
Vision transformer recently has drawn great attention in computer vision because of its high model capability and superior potentials in capturing long-range dependencies. Building on top of trans- formers [41], modern state-of-the-art models, such ViT [10] and DeiT [40], are able achieve competitive performance on image classification and downstream vision tasks [13, 20] compared to effective architectures
CNN models. There fast-growing efforts manually designing
further unveil the power of vision transformer [27,55 53, 15].
Neural architecture search (NAS) as powerful technique for automating the design of networks has shown its superiority to manual design [58, 39]. For NAS methods, search space crucial because determines the performance bounds of architectures during the search. It has been observed that the improvement of search space induced a favorable impact on the performance of many state-of-the-an models [2, 16, 50]. Researchers have devoted ample efforts the design of CNN space [45, 12, 42]. However, limited attention paid to Transformer counterparts, leading to the dilemma of search space design when finding more effective and efficient transformer architectures.
In this paper, we propose to Search the Search Space (S3) by automatically refining the main changeable dimensions of transformer architectures. In particular, we try answer the following two key questions: 1) How to effectively and efficiently measure a specific search space? 2) How to automatically change defective search space to good one without human prior knowledge?
For the first question, we propose a new measurement called E-T Error for search space quality estimation and utilize a onc supernet for effective and efficient computing. Specifically, the "E" part the E-T error similar to the error empirical distribution function used [35] focusing on
"Equal contributions. Work done when Minghao, Kan, Bolin are interns MSRA. 'Corresponding author.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Searrch Dim 1
Step 2: Compute tendencles
Head
Don secapling
FFN
Search
Search Dim 2
Stage
Block d,
North
Step I:
D Decocupose by search dimensions
1
Step Evolve by each search dimension
Stage
Block 2
WSS
Searth Space
Strp Compose the evolivd spare
Search
Fablh Embeddiigg
Evolved Soarch Space
Search Dim 3
Search Dim
Infinite Search Space
(e) Network
( Stagei
(c Blockj
Figure 1: Left: The pipeline of searching the search space. Right: The general vision transformer architecture explored in this paper.
the overall quality, while the "T" part considers the quality of top-tier architectures in search space. In contrast to [35] that trains each subnet for only a few epochs, we train a once-for-all supernet with AutoFormer [6] to obtain a large amount of well-trained subnets, providing more reliable proxy for performance estimation.
For the second question, as shown in Fig. I we decompose the search space by multiple dimensions, including depth, embedding dimension, MLP ratio, window size, number of heads and Q-K-V dimension, and progressively evolve each dimension to compose a better space. Particularly, we use linear parameterization model the tendencies of different dimensions as guide the search.
In addition, we analyze the search process of search space and provide some interesting observations and design guidelines: 1) The third stage is the most important one and increasing its number of blocks leads to performance improvement. 2) Shallow layers should use a small window size while deep layers should use large window size. 3) MLP ratio is better to be progressively increased along with network depth. 4) Q-K-V dimension could be smaller than the embedding dimension without performance drop. We hope these observations will help both the manual design of vision transformer and the space design for automatic search.
In summary, make the following contributions:
We propose a new scheme, i.e. S3, to automate the design of search space for vision transformer. We also present a new pipeline for vision transformer search with minimal human interactions. Moreover, we provide analysis and guidelines on vision transformer architecture design, which might be helpful for future architecture search and design.
The experiments on ImageNet verify the proposed automatic search space design method can improve the effectiveness of design space, thus boosting the performance of scarched architectures. The discovered models achieve superior performance compared to the recent VIT [10] and Swin [27] transformer families under aligned settings. Moreover, the experi- ments on downstream vision and laniuae tasks, such as object detection, semantic segmentation and visual question answering, demonstrate the generality of the method.
2 Approach
2.1 Problem Formulation
Most of existing NAS methods can be formulated constrained optimization problem as:
a'A arg min L(W"; Deal) s.t. W arg min L(W.; Dimin, g(ax) < c, aEA
(1)
W.
where W,, is the weights of the network architecture a, A is predefined search space, Z(-) is the loss function, Dimin and D.al represent train and validation datasets respectively, 9(-) denotes a function calculating resource consumption of model while c denotes given resource constraint.
Ideally, the search space A is an infinite space that contains all possible architectures. practice, A is usually small subspace of Ω due to limited computation resource. In this paper, we break the

---

convention of searching in fixed space by considering the architecture search together with the search space design. More concretely, we decouple the constrained optimization problem of Eq. (1) into three separate steps:
Step I. Searching for an optimal search space under specific constraint:
A min ((A;Dval s.t. |A|| M,
Acn
where Q measurement of search space to be elaborated in Sec. 2.3, and M represents the maximum search space size.
Step 2. Following one-shot NAS [12, 34J, we encode the search space into supernet and optimize its weights. This is formulated as solving the following problem:
W arg min B [<(Wa: Pinin))s
where W is the weights of the supemeet, Wa is the weights in W specified by the architecture a.
Step 3. After obtaining the well-trained supemet,, we search for the optimal architecture via ranking the performance using the learned weights, which formulated as:
2' arg min : s.t. g(a) < <G
(4)
"E.A*
where W is the weights of architecture a inherited from W
2.2 Basic Search Space
We set up the general architectures of vision transformer by following the settings in VIT [10] and Swin-T [27] as shown Fig. (right). To be specific, given an input image, we uniformly split into non-overlap patches. Those patches are linearly projected to vectors named patch embeddings. The embeddings are then fed into the transformer encoder, which performs most of the computation. At last, a fully connected layer is adopted for classification.
The transformer encoder consists of four sequential stages with progressively reduced input resolu- tions. Each stage contains blocks with the same embedding dimension. Therefore, stage i has two search dimensions: the number of blocks d, and embedding dimension h;. Each block contains window-basee multi-head self-attention (WSA) module and a feed-forward network (FFN) module. We do not force the blocks in one stage to identical. Instead, block j in the 7th stage have several search dimensions including window size wy. number of heads n/.. MLPratio m,, Q-K-V embedding dimension of.
2.3 Searching the Search Space
Space Quality Evaluation. For a given space A, we propose to use E-T Error, denoted as Q(AA), for evaluating its quality. It is the average of two parts: expected error rate (x and top-tier error (A).. For a given space A, the definition of (AA) is given as:
(A) RocAola)cc((a)
where e,, is the top-1 error rate of architecture a evaluated on ImageNet validation set, g(-) is the computation cost function and e is the computation resource constraint, It measures the overall quality of a search space. In practice, we use the average error rate of N randomly sampled architectures to approximate the expectation term. The top-tier error (A) is the average eгтor rate of top 50 candidates under resource constraints, representing the performance upper bound of the search space.
Once-for-all Supernet. In constrast to RegNet [35] that trains hundreds of models with only a few epochs (~100) and uses their errors as a performance proxy of the fully trained networks, we encode the search space into supernet and optimize it similar to AutoFormer [6]. Under such training protocol, we are able to train an once-for-all transformer supernet that serves as a reliable and efficient performance indicator.
Searching the Search Space. The search of search space mainly contains two iterative steps: 1) supernet optimization and 2) space evolution.
3

---

Algorithm Scarching the Search Space
Input: Infinite search space Ω, max iteration T, threshold 7 for subspace evolved, the subspaces $ .... D evolve steps for subspaces "1y""" +TD Output: The most promising search space AA'
1: Initialize a search space AA from with minimal prior human knowledge
2: while / [0, 1,... ,T] do
3: Optimize the weights W of supernet corresponding to the space AA
4: Randomly sample N architectures from the well converged supernet
5: for space 5 in s(t) ,..., S do
6: Get the subspace {ala A na, u for each choice of E ((1)
where a; is the
7t dimension value of the architecture
7:
Calculate the E-T Error of each subspace according to Eq. (5)
8: Evolve the subspace and get s(1+1) according to Eq. (8)
9:
end for
10: A" = A s((+1) 52 c((+1) ... Sp
gl1+1)
11: end while
Step I. For the search space ( thh iteration, we encode it into a supernet. We adopt sandwich training [52] to optimize the weights of supernet, i.e. sampling the largest, the smallest, and two middle models for each iteration and fusing their gradients to update the supernet. Notably, we do not use inplace-distill since it leads to collapse for transformer training.
Step 2. We first decompose the search space A(t) by the search dimensions, defined in Sec .2.2, and four stages. Denote their corresponding subspaces as S(4), " ........ S where D is the product of the number of stage and the number of search dimensions. The search space A(t) now can be expressed as the Cartesian product of all search spaces:
A s(1) 8
5(1)
(6)
Then the search space ( could be evolved to better one 1(111), with the update of S Particu-
larly, For certain subspace S(t),, we first get the subspace {ala A no; = ofth) 3 for each choice (()) s(0,, where a; is the ith dimension value of the architecture a. Then compute the E-T Errors the subspaces. We hypothesize there is strong correlation between the continuous choices and the E-T Error in most cases, we fit linear function estimate its tendency for simplification:
y WX
where x E R is the choices in scalar parameters. Then the new subspace S (((++ is given as:
((t) and y E IR is the corresponding E-T Error. and b are both
s((+1)
1 (t)
(1)
SES! s(t)
where is the threshold for search space evolving, " is the steps for search dimension S. Note that we remove the choices if they become less than zero. The detailed algorithm is also shown in Alg. 1.
2.4 Searching the Searched Space
Once the space search is done, perform neural architecture search over the searched space. The search pipeline contains two sequential steps: 1) supernet training without resource constraints, and 2) evolution search under resource constraints.
The supernet training follows the same train recipe in space search. Similar to AutoFormer [6] and BigNas [52], the maximal, minimal subnets and several random architectures are sampled randomly from the evolved search space each iteration and their corresponding weights are updated. We provide more details in Sec. 4.1. The evolution search process follows the similar setting in SPOS [12). Our objective in this paper is maximize the classification accuracy while minimizing the model size and FLOPs. Notably, vision transformer does not include any Batch Normalization [19],

---

Stage
Stage
Stage
Blocks Nu.
1
Embed Dim
640
640
640
640
384
3.4
384
384
120
120
128
1
Q.V-V Dm
640
640
640
640
304
304
384
384
128
128
128
128
MLP Ra00
Window Sze
14
14
14
7
7
Search Iteration
Search Iteration
Search Iteration
Search Reertion
Figure 2: Visualization of space search process. Columns and rows represent different stages and search dimensions. The dots in each sub-figure means the choices of that search dimension. We plot addition lines for better viewing the change tendency of dimensions.
10-
00.04 AM
800 Punder
2004 Namder
Mumber Biooia
Figure 3: The left 4 subplots present the empirical distribution function of different stages of the searched space focusing on the number of blocks. The rightest the top-tier error of different stages.
hence the evolution process is much faster than CNN. We provide more details in the supplementary materials.
3 Analysis and Discussion
In this section we provide analysis and discussion according to the space searching process. We hope that our observations can shed lights both manual design of vision transformer architecture and the search space design for NAS methods. To minimize the prior knowledge used in search space design, initialize the search space by setting same search space for different stages. We provide the detailed search space and its evolution process in Fig. 2. Fig. shows a direct illustration of space quality of the searched space focusing the number of blocks.
The third stage is most important and increasing blocks the third stage leads to better performance. As shown in the first row of Fig. 2, the blocks of stage 3 is constantly increasing from step 0 to step 2. We also observe that there are more blocks in stage in some famous CNN networks, such as ResNet [14], EfficientNet [39], which share the same architecture characteristics with our finding in transformers, As a result, it seems that the third stage is the most important one. On the other hand, contrary stage 3, the number of blocks decrease in stage 1,2,4 showing that these stages should include fewer layers.
Shallow layers should use small window size while deep layers should use a larger one. As shown the last row of Fig. 2, with the search steps increasing, the window size space decreases stage while increases in stage 3. It shows that shallow layers tend to have smaller window size while deep layers tend to have larger one, We assume that transformers increase focusing region with deeper layers. To verify this, we visualize the attention maps (top row in Fig, 4 (b,c) ) in different layers and
5

---

10
10
10
10
5
10
5
10
0
10
(a) Input Image
(b) 1-1
(c) 2-1
(d)
(c) 3-7
3-5
3-5
3-3
3-3
2
2:3
2:3
9 2-1
8 2-1
8
1-1
0
10
15
10
15
average relative distance
average relative distance
(f) Top-5
g Top-10
Figure 4: Illustration of attention maps of some selected layers ((b)-e)), and the average relative distance a query and its top-k keys with highest attention ((f), (g)). For attention maps ("3-7" represents the "stage 3 block 7", while (c)-(e) follow the rule), the reference position is left top, marked with + The numbers 1-5 on attention map subpatches represent the top-5 highest attention. Besides, bottom shows top-5/10 average relative distances of some selected layers. The orange lines represent the median and the green triangles
represent the mean.
draw a box plot (bottom row in Fig. 4 (f.g)) to show the relative distances from the target position to its top-5/10 focusing positions. The attention maps show that the highest attentions (noted by numbers) expand with deeper layer, which is illustrated from (b) to (c). As for the average relative distance, deeper layers have larger medians and means, indicating larger focusing regions. The above observations are consistent with our assumption and provide solid evidence for it.
Deeper layer should use larger MLP ratio. Traditionally, all layers use the same MLP ratio. However, the third row Fig. illlsstrates that shallow layers should use small MI. ratio to result in better performance while deeper ones should use larger MLP ratio.
Q-K-V dimension should be smaller than the embedding dimension. In the original transformer block [41, 10]. the Q-K-V dimension is the same as the embedding dimension. However, comparing the second row with the third row of Fig. we find that the searched space has slightly smaller Q-K-V dimension than embedding dimension, shown from stage 1 to stage 4. The gap between them are more obvious in the deeper layers. We conjecture the reason is that there are many heads with similar features in deeper layers. Therefore, the a relative small Q-K-V dimension is able to have great visual representation ability. We present more details in the supplementary materials.
4 Experiments
4.1 Implementation Details
Space searching. The initial search space is presented in Fig. 2, the number of blocks and embedding dimension are set to {2.3, 3, 4} and (224, 256, 288} respectively for all stages, while the window size, number of heads, MLP ratio, Q-K-V dimensions are set {7, 14]. (3,3,5, 4]. 87,8,91 {224, 256, 288} respectively for all blocks. We set the total iteration of space searching to 3, while setting the steps in Eq. (8) for block number, embed dim, MLP ratio, Q-K-V dim, windows size to 1, 64, 0.5, 64, 7, respectively. We set the threshold T of space evolution in Eq. (8) as 0.4.
Superner training. Similar to DeiT [40], we train the supernet with the following settings: AdamW [28] optimizer with weight decay 0.05, initial learning rate 10-3 and minimal learning rate 1 x 10 with cosine scheduler, 20 epochs warmup, batch size of 1024, 0. label smoothing, and stochastic depth with drop rate 0.2. We also adopt sandwich strategy as describe Sec. 2.3. The models trained for 300 epochs with 16 Nvidia Tesla 32G V100 GPUs.

---

Table I: S3 performance ImageNet in comparison with state-of-the-arts. We group the models according to their parameter sizes.
Models
Top-1 Acc.
Top-5 Acc.
#Params.
FLOPs
Res.
Model Type
ResNet50 [14]
76.2%
92.9%
25.5M
4.1G
224
CNN
RegNerY-4GF [35]
80.0%
21.4M
4.0G
224
CNN
EfficietNet-B4 [39]
82.9%
95.7%
19.3M
4.2G
380
CNN
T2T-ViT-14 [53]
80.7%
21.5M
6.1G
224²
Transformer
DeiT-S [40]
79.9%
95.0%
22.1M
4.7G
224
Transformer
VIT-S/16 [10]
78.8%
22.1M
4.7G
224
Transformer
Swin-T [27]
81.3%
28.3M
4.5G
224
Transformer
S3-T (Ours)
82.1
95.8%
28.1M
4.7G
224*
Transformer
ResNet152 [14]
78.3%
94.1%
60M
1IG
2242
CNN
EfficietNet-B7 [39]
84.3%
97.0%
66M
37G
600
CNN
BoTNet-S1-110 [38]
82.8%
96.4%
55M
10.9G
224
CNN +
T2T-ViT-24 [53]
82.2%
64M
15G
224
Transformer
Swin-S [27]
83.0%
50M
8.7G
224
Transformer
53-S (Ours)
83.7
96.4 %
50M
9.5G
224
Transformer
S3-S-384 (Ours)
84.5%
96.8% %
50M
33G
384
Transformer
RegNetY-16GFF [35]
80.4%
83M
16G
224
CNN
VIT-B/16 [10]
79.7%
86M
18G
224
Transformer
DeiT-B [40]
81.8%
95.6%
86M
18G
224
Transformer
Swin-B [27]
83.3%
88M
15.4G
2242
Transformer
Swin-B-384 [27]
84.2%
88M
47G
384²
Transformer
S3-B (Ours)
84.0%
96.6 %
71M
13.6G
224
Transformer
S3-B-384 (Ours)
84.7%
96.9 %
71M
46G
384
Transformer
61.2
41.2
Retrain Pefermance
01.3
21.0
00.3
20.0
Oumulettve prob.
90.5
20.6
066
00.4
20.4
01.2
50.2
84
Space
70.3
76.8
Space
Space
50.5 196 mi 002 912 93.4
81.2
135 /35 s00 s02 50. 60.8 83.2
Inherited Performance
Inherited Performance
Error
Figure 5: Comparison of subnets with inherited weights, fine-tuned and trained from scratch.
Figure 6: Error empirical distribution function of different search spaces.
4.2 Ablation Study
Effectiveness of once-for-all supernet sampling. To verify the effectiveness, we randomly sample 10 architectures from the well-trained supernet with the inherited weights, then compare the ImageNet classification performance of (1) models with only inherited weights, (2) models with finetuning, and (3) models retrained from scratch. As shown in Fig. the architecture with inherited weights has comparable even better performance when compared with the same architecture with retraining or finetuning. This phenomenon shows that our sampled architectures provide a precise and reliable performance proxy for space quality measurement.
Effectiveness of search space evolution. We conduct experiments. In the first experiment, randomly sample 1, (000 architectures with inherited weights from search space, then compute and plot error empirical distribution, as [35], of them. We denote the initial space, the one-step evolved space and the two-step evolved space by Space A, Space B and Space C respectively. As shown Fig. 6, the space has an obviously better overall quality compared to A and In second experiment, we use evolution algorithm to search for top architectures in each space. Tab. 5 illustrates that the space C has much better top architectures, being 16.6% 1 better than that in the spaces A and B. These results verify the effectiveness of search spaces evolution.
7

---

Table 2: Results COCO object detection using Cascaded Mask R-CNN. We provide settings
with FPN (1x: 12 epochs; 3 xx: 36 epochs). Results are taken from [44] and its official repository.
iParams
#FLOPs
Cas. Mask R A-CNN w/
FPN 1x
Cas. Mask R-CNN
w/
FPN
Backbone
(M)
(G)
AP
AP
AP
APm
AP
AP
AP
AP
AP
APmh
AP
AP%
ResNet-50 [14]
82
739
41.2
59.4
45.0
35.9
56.6
38.4
46,3
64.3
50.5
40.1
61.7
43.4
DeiT-S [44]
80
889
48.0
67.2
51.7
41.4
64,2
44.3
Swin-T [27]
86
745
48.1
67.1
52.2
41.7
64,4
45.0
50.4
69.2
54.7
43.7
66,6
47.3
83-T(Ours)
86
748
48.4
67.8
52.3
42.0
64.8
45.2
50.5
69.4
54.9
43.8
66.8
47.5
ResNeXt-101-322 [48]
101
819
44.3
62.7
48.4
38.3
59.7
41.2
48.1
66.5
52.4
41.6
63.9
45.2
Swin-S [27]
107
838
50.3
69.6
54.8
43.4
66.7
47.0
51.8
70.4
56.3
44.7
67.9
48.5
S3-S(Ours)
107
855
50.8
70.1
55.0
43,9
67.5
47.2
52.6
71.5
56.9
45.5
68.8
49,4
ResNeXt-101 [48]
140
972
45.3
63.9
49,6
39.2
61.1
42.2
48.3
66.4
52.3
41.7
64.0
45.1
Swin-B [27]
145
982
50.5
69.5
55.0
43.5
66.9
46.9
51.9
70.7
56.3
45.0
68.2
48.8
S3-B(Ours)
128
947
50.8
69.7
55.2
43.7
67.1
47.0
52.5
71.2
57.1
45.4
68.8
49.1
imsogeeet Top 1 Accurscy t%)
13.
02.5
83
12
85-
12
20
80
13
80
79
SmirT Taptery
Smas7
Seev Reptey
Tastel
12
16
60
80
100
600
10000
1200
FLOPs MMlliooss
Number Parameters (Millions)
Throughput (Imagels)
Figure 7: Comparison with state-of-the the-tre-rrtt models under different constraints.
4.3 Image Classification
We compare our searched models with state-of-tte -oee--oftthhrrree on ImageNet [9]. As described in Sec. 2.1 and Sec. 4.1, we first evolve the search space, then optimize the weights of supernet and finally search on the supernet. For the experiments of high resolution images, the models are finetuned with 3842 inputs for 10 epochs, using an AdamW [28] optimizer with constant learning rate of 10-5 and weight decay of 10-8
We compare our S3 with state-of-the-ar CNN models, vision transformers and hybrid CNN- transformer models. The results are reported in Tab. 1. Our proposed models S3-tiny/small/base (T/S/B for short) achieve 82.1%//.. 00..3..%%/.400000 .%% top-1 accuracy on ImageNet [9] using 224 input images. Compared with vision transformer and hybrid models, our method obtains superior perfor- mance using similar FLOPs and parameters. For example, when considering parameters as the main constraint, we find architecture that has 0.6% higher top-1 accuracy than Swin Transformer [27] while using 42% fewer parameters as shown in Fig. 7 Compared to CNN models, our S3 models are also very competitive, especially small parameter size. The proposed S3-S-384 (84.5%) performs slightly better than EfficientNet-B7 (84.3%) [39] while using 24% fewer parameters and 11 % fewer FLOPSs For base models with fewer computational cost, S3-B also achieves superior performance to state-of-the-ar methods, such as RegNet [35] and ViT [10].
4.4 Vision and Vision-Languaa Downstream Tasks
To further evaluate the generalizability the discovered architectures, we transfer them to vision and vision-language downstream tasks. We conduct experiments on COCO [24] for object detection, ADE20K [56] for semantic segmentation and VQA 2.0 [54] for visual question answering, respec- tively. The training and inference settings for object detection and semantic segmentation the same as those in Swin transformer [27], while for visual language answering task, the settings are aligned with SOHO [18]. More experimental details elaborated in supplementary materials.
Object Detection. As shown in Tab. 2, S3-T outperforms ResNet-50 [14] by +4.2 box AP and +3.7 mask AP under 3x learning schedule. Compared to DeiT-S [40]. S3-T achieves +2.5 higher box AP and +2.4 higher mask AP. Increasing the model size and FLOPs, S3-S and S3-B surpass ResNeXt-100 101 [48] by +4.5 box AP, +3.9 mask AP and +4.2 box AP, +3.7 mask AP respectively. Compared to Swin-S and Swin-B, S3-S and S3-B outperform them by +0.8 box AP, +0.8 mask AP and +0.6 box AP, +0.4 mask AP respectively. All comparisons are conducted under the similar model

---

Table 3: Results on ADE20K Semantic Segmenta- tion using UperNet. t: reproduced by the official code. *: without adding deconvolution in heads.
Table 4: Results on VQA v2.0 dataset.
FLoPs
VQA
VQA
Backbone
(M)
(G)
(est-dev(%%
test-std(
Ner-e [14]
4.1
26
64.65
65.12
#param mLoU
mloU
ResNet-101 [14]
7.8
65.44
65.71
Method
Backbone
(M)
(G)
(%)
(ms+flip %)
Swin- [27]
4.5
29
67.13
67.40
UperNet ResNet-50 [4]
67
951
42.05
42.78
S3-T (Ours)
4.6
29
67.84
68.00
UperNet DeiT-S* [40]
52
1099
43.15
43.85
UperNet Swin-T [27]
60
945
44.51
45.81
Table 5: Comparisons of searched architec-
UperNet S3-T(Ours)
60
954
44.87
46.27
tures in different spaces.
UperNet ResNet-101 [14]
1029
43.82
44.85
Space
#Param(()) FLOPP(() Top-1(SF)
UperNet DeiT-B" [40]
121
2772
44.09
45,68
A 28M
4.7
65.5
UperNet Swin-Si [27]
81
1038
47.60
48.87
B
29M
4.7
78.0
UperNet S3-S(Ours)
81
1071
48.04
49.31
C
28M
4.7
82.1
size and FLOPs. Under the 1x training schedule, our S3 model is also superior to other backbones, demonstrating its robustness and good transferability.
Semantic Segmentation. We report the mloU score on ADE20K [56] validation set in Tab. 3. It can be seen that our S3 models consistently achieve better performance than ResNet [14], DeiT [40] and Swin [27]. Specifically, S3-T is +3.5 mIoU superior to ResNet-50. Compared to other transformer- based models, S3-T is +2.42 mloU higher than DeïT-S, and +0.46 mIoU higher than Swin-T. S3-S outperforms ResNet-101 with a large margin (+4.46). It also surpasses DeiT-B and Swin-S by +3.63 and +0.44 mIoU, respectively. Note that tried to add deconvolution layers build hierarchical structure for DeiT, but did not get performance improvements.
Visual Question Answering. As shown in Tab. 4, we report the accuracy on the test-dev and test-std set of VQA v2.0 dataset [54] for different vision backbones. Our model S3-T brings more than +2% accuracy gains over ResNet-101 [14], and surpassing Swin-T [27] by around 0.6% accuracy under aligned model complexity. All the above experiments demonstrate the good generality of the searched models as well as proving the effectiveness of the searched design space of vision transformer.
5 Related Work
Vision Transformer. Originally designed for natural language processing. Transformer has been recently applied to vision tasks [4, 55, 5, 46). ViT [10] is the first one that introduces pure transformer architecture for visual recognition. Alleviating the data hungry and computational issues, DeiT [40] introduces series of training strategies to remedy the limitations. Some further works resort to feature pyramids to purify multi-scale features and model global information [44, 15, 31]. Swin Transformer [27] introduces non-overlapping window partitions and restricts the calculation of self-attention within each window, leading to a linear runtime complexity w.r.t. token numbers.
Search Space. There is a wide consensus that a good search space is crucial for NAS. Regarding CNN, the current search space, such as cell-based space [26, 7, 49] and MBConv-based space [3, 32, 51], is pre-defined and fixed during search. Several recent works [17. 29, 30, 33, 25, 23] perform search space shrinking to find a better compact one. Moreover, NSENet [8] proposes an evolving search space algorithm for MBCоnv space optimization. RegNet [35] presents new paradigm for understanding design space and discovering design principles. However, the exploration of vision transformer space is still limited, and our work is the first along the direction.
Search Algorithm. There has been an increasing interest in NAS for automating network design [11, 20]. Early approaches use either reinforcement learning [57, 58] or evolution algorithms [67, These approaches require training thousands of architecture candidates from scratch. Most recent works resort to the one-shot weight sharing strategy to amortize the searching cost [22, 34, 1. 12]. The key idea is to train an over- paee-paraeteriieed supernet where all subnets share weights. For transformer, there are few works employing NAS to improve architectures. Evolved Transformer [37] searches the inner architecture transformer block. HAT [43] proposes to design aerdware-wware transformers with NAS to enable low-latency inference on coouure--oonstrinnee hardware platforms. BossNAS [21] explores hybrid CNN-Transformers with block-wisely self- self-supervised. The most related work is AutoFormer [6] which provides a simple yet effective framework of vision transformer search. It could achieve once-for-all training with newly proposed weight entanglement. this work, we focus on both the search space design and architectures of vision transformer.

---

6 Conclusion
In this work, we propose to search the search space of vision transformer. The central idea is to gradually evolve different search dimensions guided by their E-T Error computed using weight- sharing supernet. We also provide analysis on vision transformer, which might be helpful for understanding and designing transformer architectures. The search models, S3, achieve superior performance to recent prevalent ViT and Swin transformer model families under aligned settings. We further show their robustness and generality on downstream tasks. In further work, we hope to investigate the usage of S3 in CNV search space design.
Acknowledgements and Disclosure of Funding
Thank Hongwei Xue for providing helps and discussions on vision-language tasks. No external funding was received for this work.
References
[1] Andrew Brock, Theodore Lim, James M Ritchic, and Nick Weston. Smash: one-shot model architecture search through hypernetworks. In ICLR, 2018.
[2] Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, and Song Han. Once for all: Train one network and specialize it for efficient deployment. In ICLR, 2020.
[3] Han Cai, Ligeng Zhu, and Song Han. Proxylessnas: Direct neural architecture search on target task and hardware. ICLR, 2018.
[4] Nicolas Carion, Francisco Massa, Gabriel Synnaeve. Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020. 9
[5] Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In International Conference on Machine Learning, pages 1691-1703. PMLR, 2020.
[6] Minghao Chen, Houwen Peng, Jianlong Fu, and Haibin Ling. Autoforner: Scarching transformers for visual recognition. 2021. 2, 3,4,
[7] Xin Chen, Lingxi Xie, Jun Wu, and Qi Tian. Progressive differentiable architecture search: Bridging the depth gap between search and evaluation. In ICCV, 2019.
[8] Yuanzheng Ci, Chen Lin, Ming Sun, Boyu Chen, Hongwen Zhang, and Wanli Ouyang. Evolving search space for neural architecture search. arXiv preprint arXiv:201 I. 10904, 2020. 9
[9] Jia Deng. Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
[10] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn. Xiaohua Zhai. Thomas Unterthiner. Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. ICLR, 2021. 1, 2, 3, 6, 8,
[11] Thomas Elsken, Jan Hendrik Metzen, Frank Hutter, et al. Neural architecture search: A survey, J. Mach. Learn. Res., 20(55):1-21, 2019.
[12] Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng, Zechun Liu, Yiehen Wei, and Jian Sun. Single path one-shot neural architecture search with uniform sampling. ECCV, 2020. 1,3,4,9
[13] Kai Han, Yunhe Wang, Hanting Chen, Xinghao Chen, Jianyuan Guo, Zhenhua Liu, Yehui Tang, An Xiao, Chunjing Xu, Yixing Xu, al. A survey visual transformer. arXiv preprint arXi::2012.12556 2020. 1
[14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016. 7, 8.9
[15] Byeongho Heo. Sangdoo Yun, Dongyoon Han. Sanghyuk Chun, Junsuk Choe, and Seong Joon Oh. Rethinking spatial dimensions of vision transformers. arXiv preprint trXiv:2103.163002 2021. 1,9
[16] Andrew Howard, Mark Sandler, Grace Chu, Liang-Chiel Chen, Bo Chen, Mingxing Tan, Weijun Wang. Yukun Zhu, Ruoming Pang. Vijay Vasudevan, et al. Searching for mobilenetv3. In ICCV, 2019.
10

---

[17] Yiming Hu, Yuding Liang, Zichao Guo. Ruosi Wan, Xiangyu Zhang, Yichen Wei, Qingyi Gu, and Jian Sun. Angle-based search space shrinking for neural architecture search. ECCV, 2020. 9
[18] Zhicheng Huang, Zhaoyang Zeng, Yupan Huang, Bei Liu, Dongmei Fu, and Jianlong Fu, Seeing out of the box: End-to-end pre-training for vision-language representation learning. In CVPR, 2021. 8
[19] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, 2015.
[20] Salman Khan, Muzammal Naseer, Munawar Hayat, Syed Waqas Zamir, Fahad Shahbaz Khan, and Mubarak Shah. Transformers in vision: survey, arXiv preprint arXiv:2101.011000 2021. 1.
[21] Changlin Li, Tao Tang, Guangrun Wang, Jiefeng Peng, Bing Wang. Xiaodan Liang, and Xiaojun Chang- Bossnas: Exploring hybrid con-transformers with block-wisely self-supervised neural architecture search. arXiv preprint arXiv:2 103. 12424, 2021. 9
[22] Liam Li and Ameet Talwalkar. Random search and reproducibility for neural architecture search. UAI, 2019. 9
[23] Xiang Li, Chen Lin, Chuming Li, Ming Sun, Wei Wu, Junjie Yan, and Wanli Ouyang. Improving one-shot nas by suppressing the posterior fading. In CVPR, 2020. 9
[24] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014. 8
[25] Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Jonathonn Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy. Progressive neural architecture search. In ECCV, 2018. 9
[26] Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable architecture search. In ICLR, 2019. 9
[27] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. arXiv preprint 110300. 2021. 1, 2, 7,
[28] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019. 6, 8
[29] Niv Nayman, Asaf Noy, Tal Ridnik, Itamar Friedman, Rong Jin, and Lihi Zelnik. Xnas: Neural architecture search with expert advice. In NeurIPS, 2019. 9
[30] Asaf Noy, Niv Nayman, Tal Ridnik, Nadav Zamir, Sivan Doveh, Itamar Friedman, Raja Giryes, and Lihi Zelnik. Asap: Architecture search, anneal and prune. AISTATS, pages 193-500 PMLR, 2020. 9
[31] Zizheng Pan, Bohan Zhuang, Jing Liu, Haoyu He, and Jianfei Cai. Scalable visual transformers with hierarchical pooling. arXiv preprint arXiv:2103. 10669, 2021. 9
[32] Houwen Peng, Hao Du, Hongyuan Yu, Qi Li, Jing Liao, and Jianlong Fu. Cream of the crop: Distilling prioritized paths for one-shot neural architecture search. NeurlPS, 2020.
[33] Juan-Manuel Férez-Rúa, Moez Baccouche, and Stephane Pateux. Efficient progressive neural architecture search. In BMVC, 2018. 9
[34] Hieu Pham, Melody Guan, Barret Zoph, Quoe Le, and Jeff Dean. Efficient neural architecture search via parameters sharing. In JCML, 2018. 3,9
[35] Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Designing network design spaces. In CVPR, 2020. 1, 8,
[36] Esteban Real, Alok Aggarwal, Yanping Huang, and Quoe V Le. Regularized evolution for image classifier architecture search. In AAAI, 2019. 9
[37] David So, Quoc Le, and Chen Liang. The evolved transformer. In International Conference on Machine Learning. PMLR, 2019. 9
[38] Aravind Srinivas, Tsung- Yi Lin, Niki Parmar, Jonathon Shlens, Picter Abbeel, and Ashish Vaswani. Bottleneck transformers for visual recognition. arXiv preprini arXiv:2101. 2021. 7
[39] Mingxing Tan and Quoe V. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In ICML, 2019. 1, 5, 7,
11

---

[40] Hugo Touvron, Matthieu Cord, Mathhij Douze, Francisco Massa, Alexandre Sablayrolles. and Herv'e J'egou. Training data-efficient image transformers & distillation through attention. arXiv preprint 1rXiv:2012.12877. 2020. 1. 6. 8,
[41] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. NeurIPS, 2017. 1,6
[42] Dilin Wang, Meng Li, Chengyue Gong, and Vikas Chandra, Attentivenas: Improving neural architecture search via attentive sampling. CVPR, 2021.
[43] Hanrui Wang, Zhanghao Wu, Zhijian Liu, Han Cai. Ligeng Zhu, Chuang Gan, and Song Han. Hat: Hardware-aware transformers for efficient natural language processing, In ACL, 2020.
[44] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang. Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: versatile backbone for dense prediction without convolutions. arXiv preprint arXiv:2102. 12122, 2021. 8,9 9
[45] Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang, Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer. Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search. In CVPR, 2019.
[46] Kan Wu, Houwen Peng, Minghao Chen, Jianlong Fu, and Hongyang Chao. Rethinking and improving relative position encoding for vision transformer. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pages 10033- 10041, 2021. 9
[47] Lingxi Xie and Alan Yuille. Genetic cnn. In /CCV, 2017. 9
[48] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transforma- tions for deep neural networks. In CVPR, 2017. 8
[49] Yuhui Xu, Lingxi Xie, Xiaopeng Zhang, Xin Chen, Guo-Jun Qi, Qi Tian, and Hongkai Xiong. PC-DARTS: Partial channel connections for memory-efficient architecture search. In ICLR, 2020. 9
[50] Antoine Yang, Pedro M Esperança, and Fabio M Carlucci. Nas evaluation is frustratingly hard. In ICLR, 2020. 1
[51]
Shan You, Tao Huang, Mingmin Yang, Fei Wang, Chen Qian, and Changshui Zhang. Greedynas: Towards fast one-shot nas with greedy supernet. In CVPR, 2020. 9
[52] Jiahui Yu, Pengchong Jin, Hanxiao Liu, Gabriel Bender, Pieter-Jan Kindermans, Mingxing Tan, Thomas Huang, Xiaodan Song, Ruoming Pang, and Quoe Le. Bignas: Scaling up neural architecture search with big single-stage models. NeurIPS. 2020. 4
[53] Li Yuan, Yumpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Francis EH Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch imagenet. arXiv preprint arXiv:2101. 1986, 2021. 1.7
[54] Peng Zhang, Yash Goyal, Douglas Summers-Stay Dhruv Batra, and Devi Parikh. Yin and Yang: Balancing and answering binary visual questions. In CVPR, 2016. 8, 9
[55] Sixiao Zheng, Jiachen La, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang. Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip HS Torr, et al. Rethinking semantic segmentation from a sequence-to-seqqeeee perspective with transformers. arXiv preprint arXi::2012.15840 2020. 9
[56] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In CVPR, 2017. 8, 9
[57] Barret Zoph and Quoe V Lc.. Neural architecture search with reinforcement learning. ICI.R, 2016.
[58] Barret Zoph, Vijay Vasudevan, Jonathon Shiens, and Quae V Le. Learning transferable architectures for scalable image recognition. In CVPR, 2018. 1.9
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] In the supplementary materials
(c) Did you discuss any potential negative societal impacts of your work? [Yes] In the supplementary materials
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [No]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No]
(c) Did you include any assets either in the supplemental material URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you's 're using/curating? [No]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[No]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "ytiroAq_kzP", "reviewer": "Reviewer_uQyW", "summary": "This work applies NAS to the vision transformer architecture. Besides searching the architectures, it also automatically adjusts the search space guided by the E-T error that combines the error distribution and top model errors to evaluate the quality of different choices in a search dimension. By fitting a linear function to approximate the E-T error, the new search space is updated through a simple formula. They used the supernet to efficiently compute the E-T errors, and use evolutionary search to search for the best algorithm in a given search space. The discovered architecture is shown to be competitive to the state-of-the-art models, and also generalize to other vision tasks.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n\n(1) The idea of automatically iterating the search space and using super-net to make the calculation of the E-T error are interesting. It combines existing ideas in designing the search space and weight sharing NAS and proposes a new method to update the search space by fitting a linear function to the E-T error. \n\n(2) The evaluation result is competitive and additional experiments are conducted on several vision tasks to verify the discovered architectures' generality.\n\n(3) Analysis are performed on how the search space changes in different iterations, however I had some questions regarding the reliability of the results. \n\n(4) The paper is well written and easy to follow. (A minor issue in formatting, the caption of Figure 2 is covered a bit by Figure 3.)\n\nWeakness:\n\n(1) This paper combines several existing ideas and applies them to a new domain (Vision Transformer), and the novel contribution in the method is the way to update the search space using the linear function fitting of the search choices and the E-T error. However, there should be more analysis or supporting evidence for this simplifying assumption in the experiments. For example, it would help to add some plots and calculate the R-squared to verify how good this linear approximation assumption is. \n\n(2) Some of the hyperparameter choices need more discussion. For example, how is the value of tau (0.4) picked? Was it tuned over some range? How sensitive is the result with respect to this value? Same for the number of search space iterations. Why does it stop at 3, is it due to compute limits or maybe 3 is the sweet spot? It would help to add some analysis regarding how the accuracy changes with the number of search space iterations.\n\n(3) The analysis of the trends in Section 3 is quite interesting. Since the conclusions are draw from the results in Figure 2, I wonder how much variance is in the result. For example, if you repeat the experiments multiple times, how much would the search space evolution differ between different repeats. This would help evaluate the reliability of the analysis.  \n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I listed a few additional analysis in the \"Weakness\" section in the Main Review that would help shed more light on the limitations of the proposed methods. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "K-yREySyZXB", "reviewer": "Reviewer_3mCn", "summary": "This work propose to first search for the search space for transformers and then applies once-for-all NAS methods to search for architectures within the searched search space. In particular, the work proposes an E-T error metric to evaluate the quality of the search spaces. From the results of the whole search process, this work also summarizes some design guidelines for vision transformers.\nThe experiments are extensive and they show the superiority of the proposed search method. ", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n- The paper is easy to follow.\n- The idea of search the search space first is intuitive.\n- The experiments on different downstream tasks are extensive.\n- The design guidelines for transformer are good for the research community.\n\nCons:\n- The improvements over Swin transformers on object detection and semantic segmentation are not so obvious, comparing to the gap between Swin and the other methods.\n- What is the search cost (GPU-days) of each step?\n- The ablation study seems not enough. Are there any ablations on the E-T error, such as changing the weighting of E and T, or varying the number of sampled architectures for computing E and for computing T.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, the limitations are addressed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5"}, "extra_text": {}}, {"review_id": "95G9NHEIh9k", "reviewer": "Reviewer_YJYN", "summary": "This paper proposes a method for searching the search space of vision. The authors propose decomposing the search space by the search dimensions and evaluating them with the proposed E-T error to explore the optimal search space. Then the new subspace is formulated as Equation (8). The results reported in this paper have achieved state-of-the-art results.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "(+) This paper is easy to follow.\t\t\t\n\n(+) State-of-the-art performance. This paper conducts extensive experiments and achieves impressive performance improvements.\n\n(+) Searching the search space is a good problem in NAS methods.\n\n(+) This paper provides lots of interesting and useful observations in section 3, which may help the development of the research of vision transformer.\n\n(-) The proposed method seems very time-consuming in searching search space. Simply calculating each search space's expected error for all the search space is computationally inefficient (as in the 5 line in Algorithm 1). I wonder how many search spaces are considered in the searching process.\n\n(-) I am not satisfied with the Experimental part. The authors did not provide the final searched search space nor the searched architecture. If the authors can provide detailed evidence about this issue, I will support accepting this paper. I may change my rate according to the rebuttal.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "0.75"}, "extra_text": {}}, {"review_id": "-B8MSiAYWKm", "reviewer": "Reviewer_W7fM", "summary": "This paper introduces a neural architecture search method called S3 that first searches the search space before identifying good architectures from the discovered search space.  The authors focus on applying S3 to vision transformers and design a search space modeled after ViT.  They evaluate S3 discovered architectures on various computer vision tasks and show good performance on image classification, object detection, and semantic segmentation.\n", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Originality \nThe main points of novelty introduced in this paper are:\n- Applying one-shot NAS approaches to vision transformers. The authors note that in contrast to Once-for-all and BigNAS style approaches for CNNs, inplace distillation did not help when training the vision transformer supernetwork.\n- Defining the search space for vision transformers.\n- Searching the search space using the E-T Error heuristic along with a linear parameterization to guide the search.\nThe novelty though is fairly small in scope so I lean towards a neutral rating here.  Additionally, there is a point to be made that the search space should be as large as possible and include all possible sub-spaces of interest.  In this case, the search algorithm would be responsible for efficiently identifying the most suitable architectures from the space. \n\n## Quality\nThe E-T error and linear parameterization are not well motivated and I want to see further justification for choosing these heuristics. \n\nI liked the discussion provided in Section 3 of the search phase results but the writing con be improved significantly.  \n\nThe empirical results in Section 4 are promising and demonstrate S3 to yield highly performant architectures.  \n\nHowever, I have quite a few questions that I hope the authors can address:\n- What do you mean by compose the evolved search space in Figure 1?\n- It appears that the relationships between E-T error and a search dimension is not always linear.  Have you observed failure cases where using a linear parameterization didn't learn a good search space? \n- How sensitive is the search process to the parameter \\tau?  Have you tried other values?  \n- What is the search cost for 3 iterations?  Have you conducted search space refinement to convergence? \n- Have you tried comparing S3 to a one-shot model trained on a larger search space that includes the resulting search spaces from all iterations of the search process?\n- Have you tried extending the search space refinement process to other NAS benchmarks like the popular MobileNetV3 search space?  Does S3 transfer to these other search spaces?\n\n## Clarity\nThe paper is poorly written and errors are littered across the entire text.  I have enumerated a subset of issues I noticed but there are certainly more.  \n\n_Need clarification_\n- L45: \"the third stage is the most important...\" what is the third stage? this is not introduced until much later\n- There are 4 steps in Figure 1 but there is nothing corresponding to these four steps in the actual text.  I see steps 1 and 2 in section 2.3 but not 3 and 4.  \n\n_Typos & Grammar_\n- L37-38: \"Different from [34] that trains...\" -> In contrast to [34], which trains...\n- L40-41: \"We decouple the search space by multiple dimensions...\" -> We characterize search spaces by multiple dimensions...\n- L42-43: \"Particularly, we use linear parameterization to model the tendencies of different dimensions so as to guide the search.\" -> In particular, we parameterize the model along different search dimensions...\n- L44: \"we analyze the search process of search space and provide some interesting observations\" -> needs to be rephrased\n- L49: \"we hope these observations could help...\" -> we hope these observations will help\n- L:104: \"Different form RegNet [34] that trains...\" -> needs to be rephrased\n- Figure 2 caption cut off\n- L145: \"Fig. 3 shows an direct...\" -> Fig. 3 shows a direct...\n- L147: \"The third stage is the most important and increasing blocks of it contribute to better performance\" -> needs to be rephrased\n- L151-153: needs to be rephrased\n- L153: \"should include less layers\" -> should include fewer layers\n- L154: use a different verb from prefer perhaps \"more compatible\" or \"more suitable\"\n- L156: \"deep layers attend to\" -> deep layers tend to\n- L157: \"We assume that transformers increase focusing region with deeper layers\" -> needs to be rephrased\n- L161: \"which illustrated from\" -> which is illustrated from\n- \"focusing regions\" -> regions of focus\n- L164: \"Deeper layer, larger MLP ratio\" -> phrase as a sentence to mirror other points.\n- L166: \"It gives our insight to design different MLP ratio\" -> needs to be rephrased\n- L167: \"Q-K-V dimension could smaller than the embedding dimension\" -> Q-K-V dimension can be smaller than the embedding dimension\n- L171-173: needs to be rephrased\n- L172: \"less heads\" -> fewer heads\n- L215: \"we could get model...\" -> we found a model...\n\n## Significance\nDespite the impressive empirical results, the significance of the paper is greatly limited by the numerous grammatical errors that litter the text. I would evaluate its potential for impact to be low in its current state.  \n\n===============Post Author Response===============\n\nAfter reading the authors' response, I have decided to change my score to a 5.  I am hesitant to recommend acceptance given the unpolished state of the paper at submission.  I do see the merit of the method proposed and believe it would benefit tremendously from improved  writing and additional experiments evaluating S3 on different search spaces in a future submission.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors do not adequately evaluate their proposed S3 method for sensitivity to internal parameters and robustness/transferability to other search spaces.  ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Four experts reviewed this paper and gave ratings 6, 6, 6, and 5, respectively --- Reviewer 3mCn decided to change 7 to 6 in a private discussion. The reviewers expressed concerns about novelty and writing. Particularly, some reviewers commented that the E-T error heuristic and linear approximation were not well-motivated, and the novelty was small. Reviewer W7fM felt strongly about the “poor state of the writing.” However, the reviewers generally appreciated the study of NAS for Transformers, and a reviewer considered the design of the Transformer-tailored search space original. AC agreed that such a study could benefit future research on Transformers and felt that the concerns can be addressed in a reasonable revision of the paper. Hence, the decision is to recommend the paper for acceptance. The authors are encouraged to make necessary changes in the camera-ready to address the reviewers' questions to the best of their ability. Especially, the authors may consider finding a native English speaker to help polish the writing. We congratulate the authors on the acceptance of their paper!
Decision: Accept (Poster)