IT-NAS: Integrating Lite-Transformer into NAS for Architecture Seletion
Authors: Zihao Sun
Yu Hu
Longxing Yang
Shun Lu
Jilin Mei
Yinhe Han
Venue: Submitted to ICLR 2023 (2023)
Keywords: Neural Architecture Search
Transformer
Self-Attention

Abstract: Neural Architecture Search (NAS) aims to search for the best network in the pre-defined search space. However, much work focuses on the search strategy but little on the architecture selection process. Despite the fact that the weight-sharing based NAS has promoted the search efficiency, we notice that the architecture selection is quite unstable or circuitous. For instance, the differentiable NAS may derive the suboptimal architecture due to the performance collapse caused by bi-level optimization, or the One-shot NAS requires sampling and evaluating a large number of candidate structures. Recently, the self-attention mechanism achieves better performance in terms of the long-range modeling capabilities. Considering that different operations are widely distributed in the search space, we suggest leveraging the self-attention mechanism to extract the relationship among them and to determine which operation is superior to others. Therefore, we integrate Lite-Transformer into NAS for architecture selection. Specifically, we regard the feature map of each candidate operation as distinct patches and feed them into the Lite-Transformer module along with an additional Indicator Token (called IT). The cross attention among various operations can be extracted by the self-attention mechanism, and the importance of each candidate operation is then shown by the softmax result between the query of indicator token (IT) and other values of operational tokens. We experimentally demonstrate that our framework can select the truly representative architecture in different search spaces and achieves 2.39% test error on CIFAR-10 in DARTS search space, and 24.1% test error on ImageNet in the ProxylessNAS search space, as well as the stable and better performance in NAS-Bench-201 search space and S1-S4 search spaces, outperforming state-of-the-art NAS methods.

Under review as a conference paper at ICLR 2023
IT-NAS: INTEGRATING LITE-TRANSFORMER INTO NAS FOR ARCHITECTURE SELETION
Anonymous authors
Paper under double-blind review
ABSTRACT
Neural Architecture Search (NAS) aims search for the best network in the pre- defined search space. However, much work focuses on the search strategy but little on the architecture selection process. Despite the fact that the whirht-sharing based NAS has promoted the search efficiency, we notice that the architecture selection is quite unstable or circuitous. For instance, the differentiable NAS may derive the suboptimal architecture due to the performance collapse caused by bi- level optimization, or the One-shot NAS requires sampling and evaluating a large number of candidate structures. Recently, the self-attention mechanism achieves better performance in terms of the long-range modeling capabilities. Considering that different operations are widely distributed in the search space, suggest leveraging the self-attention mechanism to extract the relationship among them and to determine which operation superior others. Therefore, we integrate Lite-Transformee into NAS for architecture selection. Specifically, we regard the feature map of each candidate operation as distinct patches and feed them into the Lite- Transfrrnsorm module along with an additional Indicator Token (called ITT). The cross attention among various operations can be extracted by the self attentttenion mechanism, and the importance of each candidate operation is then shown by the softmax result between the query of indicator token (TT) and other values operational tokens. We experimentally demonstrate that framework the truly representative architecture different search spaces and achieves 2.39% test error CIFAR-10 in DARTS search space, and 24. 1 % test error ImageNet in the ProxylessNAS (w/o SE module) search space, as well as the stable and comparable performance in WAS-Bench-201 search space, S1-S4 search spaces and NAS-Bench- Shotl search space.
select
INTRODUCTION
Neural Architecture Search (NAS) is emerging
paradigm for designing network structures. has been demonstrated to outperform manually designed networks in many tasks, including image
classification (Zoph & Le 2016 Zoph al. 2018 Guo et al.) 2020), object detection (Chen al.) 2019 Ghiasi et al. 2019), semantic segmentation (Chen et al. 2018 Liu et al. (019) and so on. The fundamental disad vantage of earlier NAS methods, which primarily relied on heuristic algorithms like reinforcement learning (Baker et al. 2016 Bello et al. 2017 Zoph et al. (2018) or evolutionary algorithms (Real et al. 2017; Liu al.
2019), the necessity to train each architecture from scratch for validation, thereby impeding further advancement of NAS.
2018a| Real
Fortunately, ENAS (Pham et al.) 2018) proposes weight-sharing mechanism, which greatly im- proves search efficiency. More recently, the differentiable NAS methods (Liu et al.) [018b] Xic al. 2018) and the One-shot NAS methods (Guo et al., 2020 Chu al. [021b] You et al.) (0200)
have been more popular. They both firstly train super-network, and then derive the final architec- ture based on different strategies. However, the differentiable approach selects the target network with the largest architecture parameters, which cannot fully reflect the true operation strength (Wang) et al., 2021a Xie et al., 20211). Nevertheless, DARTS-PT requires fine-tuning after discretizing each edge, resulting in additional selection time. The One-shot method selects the optimal one by sampling and evaluating a large number of candidate structures, yielding the come-consuming issues (Chen al. 2021a). Whereas, BN-NAS is not suitable for search space without batch normaliza-

---

Under review as a conference paper at ICLR 2023
tion layer. We endeavoring to explore whether there is more appropriate way for both popular search spaces to robustly and quickly select the architecture, as has rarely been studied before.
The attention mechanism can be used to emphasize the important components of the input while ig- noring other trivial ones. The Transformer architecture (Vaswani et al. 2017) has reignited boom
in the field of Computer Vision (CV) (Touvron al.) 2021 Liu al. 2021 since VIT (Dosovit- skiy et al. 2021> achieves competitive performance compared to Convolutional Neural Networks
(CNNs). It simply slices the image into small patches and then model the cross-attention between long sequences to locate key information, resulting in better performance. Meanwhile, Batchformer (Hou et al.
each 2022) introduces a batch transformer module that is applied to the batch dimension of of data to implicitly explore the sample relationships. Inspired by this, we can intu- itively analogize the candidate operations in the NAS search space to patches, and then leverage the self-attention mechanism to describe the interactions among different operations. As result, the optimal architecture can be selected according to the self-attention weights.
In summary, we suggest integrating Lite-Transformet into NAS for architecture selection. Specif- ically, we insert the Trae-rranffrmer module on each edge and determine the optimal operation associated with that edge in the cell-based search space. While in the chain-style search space, where the network is defined by a sequence of layers containing various choice blocks, the Lite- Transformer module is inserted broad perspective that treats candidate operations as patches. The patches are then linearly mapped and further packed into three matrices, namely Q K, and V. The softmax result between Q and K is a square matrix termed the attention map, which can be regarded as the attention weight between different candidate operations.
each layer to select the appropriate block. We adopt more
Furthermore, to address the issue of asymmetric attention matrix, i.e., the mutual attention value between any two operations cannot determine which is more favorable. We introduce an additional indicator token (called IT) to calculate the cross- coss-attennionn between IT and the other operational tokens, which is inspired by the truth elaimed by EViT (Liang et al.) (022) that the class token can be used to determine the importance of other tokens. In this way, the importance corresponding to each operation can be represented by the row of the indicator token (IT) in the attention matrix. After the super-network along with the Lite- Tite-Tranffomme module is trained to convergence, we simply need to forward propagate once on the validation dataset in order to determine the optimal architecture by computing the self-attention weights based on the indicator token (IT).
In general, our main contributions can be summarized as follows:
We revisit the architecture selection process of neural architecture search (NAS) in a fresh perspective and, to our knowledge, are the first integrate Site-Transfomer into NAS for architecture selection, utilizing the self-attention mechanism to explore the interaction among different candidate operations by regarding each one as operational token.
We introduce an additional indicator token (called IT) to compute the cross-altention be- tween IT and the other operational tokens. In this case, the row of IT in self-attention weights matrix can be used to establish the priority of each candidate operation.
Experimental results show that IT-NAS achieves better performance in DARTS and Prox- ylessNAS search space, as well as stable and comparable performance in MAS-Benches, including S1-S4, NAS- MAS--eench-201, and -es---Bnch--hhoee
More comprehensive experiments demonstrate the robustness and effectiveness of IT-NAS in selecting architectures. We also theoretically and empirically analyze why the self- attention mechanism can effectively select optimal architectures, proving the priority of our proposed method.
2 RELATED WORKS
Neural Architecture Search. Neural Architecture Search (NAS) aims to select the optimal architec- ture in the pre- pre-defined search space. Earlier NAS approaches (Baker et al.) 2016 Zoph et al.) 2018 Real al. 2017 2019) incurred substantial search overhead due to the requirement to train each candidate architecture from scratch. ENAS (Pham et al. (018)) firstly proposed the weight-sharing mechanism such that weights be shared among different sub-structures in the super-network,
2

---

Under review as a conference paper at ICLR 2023
greatly reducing the validation time. Based on the mechanism, differentiable NAS and One-shot NAS became the two most mainstream paradigms. Thereafter, a series of subsequent works focused on the design of search strategies and the improvement of super-network training.
The differentiable NAS (Liu et al. 2018b) introduced architecture parameters which are mainly trained alternately with the network weights. However, the search process is very unstable and suffers from the performance collapse issue. To improve the robustness of the search process, Ro- bustDARTS (Zela et al. (2200) and SDARTS (Chen & Hsieh| 2020) analyzed and fixed optimization errors from mathematical point of view. To reduce the discretization gap of architecture selection, SNAS (Xie al. 2018). GDAS (Dong& & 2019, DATA (Chang et al., 2019) leveraged repa- rameterization techniques, and FairDARTS (Chu et al., 2020) introduced auxiliary loss, all with the aim of optimizing architecture parameters to approximate discrete forms. These prior methods select target network based on architecture parameters, but recent work Wang et al. (2021a) pointed out that the operation associated with the largest magnitude of architecture parameters does not neces- sarily result in the highest validation accuracy after discretization. It proposed perturbation-basee architecture selection strategy, which, however, requires fine-tuning after discretizing each edge, leading to substantial computation costs.
2019),
The One-shot NAS methods (Guo et al., (2200) focused on the super-network training. SPOS (Guo et al. 2020) proposed to sample and train single-pate sub- nub-network uniformly: FairNAS (Chu al. 2021b) guaranteed all paths are sampled each time for faimess;; RLNAS (Zhang et al. (021) focused training with random labels. These methods then select the optimal substructure by sampling many sub-networks based on the evolutionary algorithm to evaluate their performance, which is very time-consuming. BN-NAS (Chen al.) 2021a) proposed to train only BN layers and then select the architecture according to the parameters of BN layer. But it is obvious that this method may not be effective for some networks lacking BN layer in the search space, which limits its application.
Vision Transformer. Transformer (Vaswani et al. 2017) has drawn much attention to computer vision recently due to its strong capability of modeling long-range relations. ViT (Dosovitskiy et al. 2021) firstly introduced the pure Transformer Encoder without any convolutional layers and achieved comparable performance to CNNs. From then on, many works attempted to modify the VIT architecture to image classification (Yuan et al. 2021; Zhou et al. 2021). object detection (Carion et al., 2020 Zhu et al.) 2020 Liu et al., 2021 and semantic segmentation (Wang et al., 2021b Xie et al. 202 Chu al. (02
To further improve the performance of Transformer-like network, some works combine with NAS to automatically discover better architecture. For example, AutoFormer (Chen et al. 2021c) firstly aimed to search ViT. GLiT (Chen et al. 20216) introduced locality modules into the search space and searched VIT from both global and local levels. ViT-ResNAS (Liao al. 2021) proposed to search for multi-stage ViT architecture. Besides, other approaches make ViT more efficient by reducing the number of tokens. Dynamic (Rao et al. (0211) introduced an additional learnable
neural network to reduce the tokens of a fully trained VIT. TokenLearner (Ryoo el al.] 2021) aggre-
gated the entire feature map weighted by dynamic attention map. and EViT (Liang et 2022)
focused on the progressive selection of informative tokens during training.
Unlike searching the ViT network, we leverage Transformer to boost NAS. Inspired by the fact that the number of tokens can be reduced with little impact on performance, we analogize the candidate operations to tokens, and then use self-attention mechanism to select the truly important operations.
METHODOLOGY
3.1 NAS OVERVIEW
Differentiable NAS. In the traditional cell-based search space, each cell is defined as directed acyclic graph (DAG) with N nodes, and each edge (i.j) between every node is associated with mixed operation ((hd) that is parameterized as architecture parameters ((d)) by using softmax re- laxation. The differentiable architecture search can be formulated bi-level optimization problem:
min (a)( s.t. w" w²(a) arg mirain (w, o:)
3

---

Under review as a conference paper at ICLR 2023
++1
B0002 4
Pood Forwar
02
Life- Traredereas
0
On
A Su (yla)
Scare of Operation
Peelieedd Eneading
C-1
(a) Cell-Based Search Space
(b) Chain-Style Search Space
Figure I: The main framework of IT-NAS. Specifically, the candidate operations on each searchable layer regarded operational tokens, added position encoding, and fed into the Lite-Transformer module. The Indicator Token (IT) is then introduced to extract the relationship between different operations and assign attention weights to them for selecting the optimal architecture.
The optimal cell architecture is selected according to the architecture parameters by discretizing each edge. In this way. the differentiable NAS methods argue that the optimized architecture param- eters can represent the importance of the candidate operation, which is unfortunately not the case (Wang et al. 2 Xie et al. (0216).
One-shot NAS. In the chain-style search space, One-shot NAS methods mainly focus on super- network training, which is expressed as:
w" (a) arg min, Be- (w, a)
Compared to Eq.(T. the continuous architecture parameters discarded, and only network weights are optimized. So the architecture selection process needs to sample a large number of sub-networks by evolution algorithm and evaluate each of them by sharing the super-network weights, resulting in the time-consuming issues (Guo et al. 22200] Chen et al. 2021a).
No matter in the cell-based search space or the chain-style search space, any network layers are composed of all candidate operations, with the goal of selecting out the optimal one from them. We take new perspective for architecture selection, that is, neither using architecture parameters nor sampling sub-networks. but leveraging the self-attention mechanism to evaluate which operation is superior, and this has never been studied NAS before.
3.2 SELF-AFTENTION MECHANISM
To model the relationship between individual different candidate operations, we first regard them as patches and then package all candidate operations of the layer into a sequence and feed them to the Lite-Tranffrmer Encoder.
Let's denote the feature maps sequence of n candidate operations on the searchable layer as O% E Rbxxxxxx(F = 2, ..., n). We first flatten each feature O E 1 so that each one can be analogized feature maps in the second dimension as Og Bhxnx (oxke(( denotes the embedding dimension.
patch similar VIT (Dosovitskiy et al. 2021b. And then concatenate For shorthand, let d = c* h
Self-Attention A transformation layer projects each operational sequence X E R to three different sequential vectors (namely, query Q, key K, and value V/, where and d the length and dimension of the input sequences, respectively. The two sequences of query and key input scheme are referred to as cross-attention mechanism. Specifically, it explicitly aggregates the query with the corresponding key, assigns them to the value, and updates the output vector:
SelfAttention(Q, K, (Q, V) = Softmax QKT
V
(3)
Vag
where d, is the length of query vector. The result of Softmax(.) is the self-attention weight square matrix, where each entry represents the cross-attention between any query and key. However,
4

---

where qie is the query of indicator token, K and V are operational key matrix and value matrix. So, Zu denotes the linear combination of the value vectors V [V1, U2..... Un with the attention weights vector a 00,,0,.... an). Because V; is the linear transformation from i-th operation, the Under review as a conference paper at ICLR 2023
this matrix is not symmetric shown in Figure That is, the attention weight between query O and key Oj is not equal to the attention weight between query Oj and key O,, so it is not possible to directly measure whether O or Oj is more important.
Indicator Token. We introduce an additional indicator token (called IT), which aims to explicitly indicate the importance of each candidate operation. The IT is concatenated in series in front of the operational tokens before feeding them into the Tran-Transfrmmer Encoder. The interaction between IT and other operational tokens computed by the self-attention mentioned before:
RitKT V =
Zu Softmax
(4)
attention weight therefore can be seen how importance of this operation over other operations.
3.3 ARCHITECTURE SELECTION
Supernet Training. The Lite-Transfore Transformer Encoder is simplified to repeat only once and consists of single-head Self-Attention (SA) and a Feed=ForwarddNetworkk (FFN). Residual connections along- side layer normalization (LN) (Ba et al. 2016) are employed after each layer. We integrate Lite- Transformer into super-network N, and then train them following the settings same as DARTS (Liu et al. 801 for differentiable NAS or SPOS (Guo et al., 2020) for One-shot NAS. Therefore,
Eq.01 and Eq. can unified as follows:
WN arg min Etrain (N(Ww), T(Wr; Wx
where Wy denotes the super-network weights and Wr is the Transformer wights which is optimized together with the super- network by minimizing losses on the training dataset. As result, we unify the differentiable NAS and One-shot NAS into a mono-level optimization. The indicator token (IT) can access the knowledge about the effect of different operations to the network during this period by exploring the interaction among candidate operations.
Architecture Selection. The architecture selection process is performed after the training is com- pleted. We propagate all images of the validation datasets to obtain the self-attention weights. Specifically, the first row of the result of the Softmax(() matrix, i.e., the cross-attention between the query of the indicator token (IT) and the keys of other operational tokens, denotes the importance of each candidate operation. In this case, can select the optimal operation each searchable layer according to this importance indicator:
A argmar(a :])
argmar((aa ([a1: a2....,a aml)
Because ag indicates the self-attention of the query of indicator token (TT) and the value of itself, we only need to index the last n cross-attention weights alai, a2,,.., an). where a; represent the importance of the i-th operations. Finally, the operation associated with the largest attention weight for each layer is selected to construct the target network.
3.4 IN-DEPTH ANALYSIS
The self-attention layer actually acts like low-pass filter with the aim of reducing high-frequency signals. Given the indicator token (IT) query vector qa. sequence of operational token key vectors K = (k1, k2,,,, kn). and the corresponding value vectors V = {vi, U2,... Cn).
Fourier Analysis.
The indicator token (IT) output of self-attention is:
o(ga) = h(ga) j=1
exp(qa kz)0
5

---

Under review as a conference paper at ICLR 2023
where h(qie) = Ei=1 exp(9ie kz). Further, rewrite Eq. as follows:
o(qu) = h'(aa)
Σ
exp(
l1qie - kT)11)os
2
j=1
1
1
exp(-
11gsc kkll)
8(k: kj)o,)dk
(8)
h((ga)
j=1
h'(ga) G(gne; 1) S(yee K,
where hi (qa)) convolution, G(-) is the Gaussian kernel, and S(-) is high-dimensional sparse signal. Because Gaus- sian filter is low-pass, the indicator token output (gse) contains redundant information based on Shannon sampling theorem (Shannon 1949). In other words, not all operational tokens equally important, but the one with the largest attention weight contributes most to network performance.
-i==1 exp(-$llax- is the normalized factor, denotes the high-dimensional
Complexity Analysis. Compared with the com- plexity of (nn²d) in the classical VīT, our method can be approximated to O(d) under the condition of n² < d. Because n generally does not exceed ten, and d can be hundreds or thousands in NAS. Therefore, the search efficiency is not significantly impacted by the calculation of attention. As shown in Table [ GPU-memon overhead is not much in- creased than that of the bascline method.
Table 1: Comparison of the computation com- plexity and GPU-Memory overhead with the baseline. The hyperparameters are all the same for fair comparison.
Complexity
GPU-Memory
IT-NS
ITNAS
DARTS
IT-NAS
SPOS
O(4)
O(
9UGB
8.5GB
9.4GB
7,9B
EXPERIMENTS
We firstly search on two popular search spaces, including DARTS search space on CIFAR- dataset Krizhevsky et al. 2009), and ProxylessNAS search space on ImageNet dataset (Krizhevsky et al. 2017). Moreover, we also evaluate the robustness and effectiveness of IT-NAS on three benchmark of
$1-S4 and NAS-Bench- Shotl. Details are in Appendix A.1 A.2 A.5
4.1 RESULTS IN DARTS SEARCH SPACE
Unlike DARTS (Lu et al. 2018b) that performs bi-level optimization by alternately updating chitecture parameters and network weights, IT-NAS only need to train the super-network along with the Lite-Transformer module half of the CIFAR-10 training dataset. The search process merely elapses hours in total on Tesla V100 GPU. After the training convergence, we propagate the other half of the CIFAR-IO training dataset validation dataset to compute the self-attention weights. Then we derive the optimal operation on each edge of the normal cell and reduction cell according Eq. To evaluate the performance, the target network consisting of 20 cells with initial channel size of 36 is trained on the whole training dataset from scratch. The details of nuper-network training and target network retraining settings are in Appendix A.3
As shown in Table 2 we can see that our IT-NAS achieves state-of-the-art performance compared with other methods in the DARTS search space. We report the average results of independent runs with different random seeds to test the effectiveness and stability of our method. Our approach achieves the average test error 2.41% with the standard deviation of 0.02 on CIFAT-10 dataset, demonstrating IT-NAS is very stable. Moreover, the best result reaches the error rate of 2.39%. outperforming all the other methods. We also transfer the cells searched on CIFAR 10 to other datasets for evaluation. As result, IT-NAS achieves the accuracy of 83.30%=0..0 on CIFAR-100 by validating the three searched cells in Figure 15) Moreover, IT-NAA achieves the Top-1 Accuracy of 75.5% ImageNet by validating the searched Cell 2 in Figure
4.2 RESULTS IN PROXYLESSNAS SEARCH SPACE
We directly search on ImageNet dataset to evaluate our method in the ProxylessNAS (w/o SE mod- ule) search space. Following PC-DARTS Xu al. 2019), randomly sample subsets from

---

Under review as a conference paper at ICLR 2023
Table 2: Search results DARTS search space and comparison with other state-of-the-ard methods. We report the average results for three independent runs with different initial random seeds. 'C10', 'C100', 'INN' denotes CIFAR-10, CIFAR- 100 and ImageNet, respectively.
Eroo(())
Params(M)
C10
C100
IMN
C10
C100
IMN
(GPU-days)
Algorithm
N
(Zopber 2015)
2.63
1500
RL
AmoehaNet-A Iee Srale
2019
3.34--0006
NA
25.5
NA
5.1
3150
EA
AmoebaNet- B a0
4U1
2.55::0.05
NA
26.0
2.8
NA
5.3
31.0
EA
PNAS
Liu
3.41+:0099
225
SMBO
ENAS
N/A
RL
DARTS
order)
LJu etal
20180
3.00200114 14
17.76
NIA
3.3
3.3
NIA
04
Gradieut
DARTS,
2nd mnlerr
20136
2.761:0.09
17.54
26.7
3.3
3,
4.7
10
Gralient
Xie
2011
2.85::0002
1.5
Gradient
GDAS
DinE& Tinz
2019
2.93
18.38
26.0
5.3
0.11
Grudient
RayesNAS
A311
2.81::0044
N/A
26.5
34
N/A
3.
0.2
Gradient
RobRst-DARES
2120
2.95:20.21
18 2630
N/A
NIA
NSA
N/A
166
Gradieut
PC-DARTS
2.57:10.07
Gradient
DATA Chanz
Goulient
FaiDARTS Cil ef 20100
2.54::000
N/A
24.9
9.320046
N/A
5.0
04
Gradient
ADARTS-ADY CClnn
Turch
2020
2.11:0002
N/A
25.6
3.3
6.1
13
Gralient
DARTS4PT
2.110.00
Gradient
BaLeNAS Zhingg Ai12
2.50+0007
NIA
Gradieut
JENAS (org.)
2.41:0.02
16.00--0000
N7A
3,07
40.400440
NIA
025
Gradient
IANAS (best)
2.39
16.63
5.5
0.2
Gradient
Table Search results on ProxylessNAS (w/o SE module) search space and comparison with other state-of-the-a art methods. denotes the search cost includes the additional subnet searching with the evolutionary algorithm.
Test Err. (%)
Params FLOPs
Search Cost
Search
Methods
Top-1
Top-5
(M)
(M)
(GPU-days)
Algorithm
MnasNet-92 Tan al.
2019)
25.2
8.0
4.4
388
2000
RI
NASNet-A Zoph 2018
26.0
8.4
5.3
564
1800
RL
AmoebaNet-C [Rell el
2019).
7.6
3150
EA
PNAS (Liu et al. 2018a
25.8
8.1
5.1
588
225
SMBO
F (Wu
2019)
25.1
7.9
4.4
375
9
Gradient
Proxyles&NASSGPU)
(Car
2018
24.9
7.5
7.1
465
8.3
Gradient
SPOS (uo ct al.
2020)
25.2
N/A
5.4
472
11
Evolution
FairNAS-A (Chu et
2021b
24.7
7.8
4.6
388
16
Evolution
RLNAS (Zhang al.
20211
24.4
7.4
5.3
473
N/A
Evolution
IT-NA
24.1
7.3
5.2
591
4
Gradient
1.3M training dataset of ImageNet, with 10% and 2.5% images as training and validation dataset, respectively. In particular, we train the super-network by uniformly sampling single paths follow- ing SPOS (Guo et al., 2020) on 8 Tesla V100 GPUs with total batch size of 512 except for 240 epochs. The training process elapses 12 hours totally. After that, the optimal sub-network is de- rived according the self-attention weights each searchable layer by propagating the validation dataset once.
We restrict the mobile setting to under 600M FLOPs for fair comparison with other methods. The target network is retrained from scratch on the whole ImageNet training dataset for 240 epochs with the batch size of 1024 on 8 Tesla V100 GPUs. From Table 3] we can see that IT-NAS achieves the best performance on ImageNet. Besides, the search cost is also the lowest compared with other evolutionary algorithm- baaortth--baaedddd methods, indicating that our approach is effective and efficient. The retrained architecture with SE module in ProxylessNASS space is summarized in Appendix A.6
4.3 RESULTS IN BENCHMARK SEARCH SPACE
S1-S4. We also conduct experiments the reduced search spaces S1-$4 introduced by Robust- DARTS (Zela et al. 2020). As shown in Table compared with other methods to regularize architecture parameters, IT-NAS is able consistently select the optimal architectures with the im- pressive performance on all three datasets of different search spaces. This once again confirms the effectiveness of self-attention weights in indicating the importance of candidate operations.

---

Under review as a conference paper at ICLR 2023
Table 4: Comparison in the reduced search spaces S1-S4 and 3 datasets. The results follow the setting of RobustDARTS (Zela et al. (020) where CIFAR-I0 models have 20 layers and 36 initial channels except that S2 and S4 have 16 initial channels, CIFAR-I00 and SVHN models have layers and 16 initial channels. The best is underlined and in bold face, the second best is in bold.
R-DARTS
DARTS
Benchmark
DARTS
SDARTS-RS
DARTS+PT
IT-NAS
DP
L2
ES
ADA
ST
3.84
3.11
2.78
3.01
3.10
2.78
3.50
2.57
S2
4.85
3,48
3.31
3.26
3.35
3.33
N/A
3.11
C10
S3
3.34
2.93
2.51
2.74
2.59
2.53
2.49
2.43
S4
7.20
3,58
3.56
3.71
4.84
4.84
N/A
3.36
S1
29.46
25.93
24.25
28.37
24.03
23.51
24.48
23.31
S2
26.05
22.30
22.24
23.25
23.52
22.28
23.16
20.58
C100
S3
28.90
22.36
23.99
23.73
23.37
21.09
22.03
20.76
S4
22.85
22.18
21.94
21.26
23.20
21.46
20.80
20.69
S1
4.58
2.55
4.79
2.72
2.53
2.35
2.62
2.44
$2
3.53
2.52
2.51
2.60
2.54
2.39
2.53
SVHN
S3
3.41
2.49
2.48
2.50
2.50
2.36
2.42
2.38
S4
3.05
2.61
2.50
2.51
2.46
2.46
2.42
2.34
12
26
Epoth
Epech
Epeicn
(a) CIFAR-10
(b) CIFAR-100
(c) ImageNet-16-122 16-100
Figure 2: Trajectory of accuracy in WAS-Bench-201 on (a) cifar10, (b) 100a (c) Imagenet- 16- 120, respectively. The shaded area represents the standard deviation of six trials of experiments.
Robustness. To evaluate the robustness of IT-NAS, track the performance of IT-NAS and DARTS over the search epochs on NAS-Bench-200 As plotted in Figure 2 ble and state-of-the-ard results three datasets compared to DARTS. Whereas, the performance of DARTS is always inferior to that of our method and the standard deviation fluctuates widely as the search epochs through different trials. Because the bi-level optimization of DARTS suffers from performance collapse issues (Zela et al. 2020 Chen & Hsieh] 2020), the architecture selection pro- cess is easily disturbed by the magnitude of architecture parameters. More seriously, DARTS will downgrade the performance due to the domination of skip-connect operation at the end of the search phase (Chu et al., 2020). On the contrary, the architecture selection of IT-NAS dominated by the self-attention mechanism is more stable than architecture parameters. The results in Table
IT-NAS achieves sta-
Effectiveness. The effectiveness of the search results can be expressed by the relationship between the architecture selection indicator and its derived architecture performance. We conduct the periments MAS-Bench-201 search space. We rank the selection indicator corresponding to all candidate operations each edge, and also rank the discretization accuracy obtained by retaining the corresponding operation on this edge. The results are plotted in Figure 3 the higher the indica- rank, the more important its corresponding operation is, and is the accuracy ranking. We also regress the relationship between the two rankings, from which we can see that the kendall-tar tau of IT-NAS better than DARTS on all edges, indicating that the architecture selection criteria based on self-attention mechanism is more accurate and effective than architecture parameters.
4.4 VISUALIZATION ANALYSIS
Here we intuitively demonstrate whether the optimal operation selected by the self-attention truly beneficial to the network. Let's take the super-network with six different operations the last layer
8

---

Under review as a conference paper at ICLR 2023
ped's Aarca -6.2
ende Rancing
Indraea Rardon Rarding (b) Edge_1
Misater Ranvrra
(a) Edge_0
(c) Edge
PASS broabi 53
Indcamte foverro
Indaanor Rarking
ediattec Famdrr
(d) Edge.3
(c) Edge_4
(
Edge_5
Figure 3: The relationship between indicator ranking and accuracy ranking of IT-NAS and DARTS NAS-Bench-201 CIFAR-IO dataset.
Input Image
K3
Figure 4: Visualization of the Grad-CAM (Selvaraju et al. 2017) of different candidate operations on ImageNet images. indicates the operation selected by IT-NAS in this layer.
of the ProxylessNAS search space as an example. We map the gradient information corresponding each operation to the input image by using Grad-CAM technique (Selvaraju al. 2017). The area with higher class activation mapping represents greater responses. As shown Figure the operation MBE3 K3 locates the object most accurately, and the operation also selected by IT-NAS on the last layer as shown in Figure 16 This consistency phenomenon demonstrates that the self- attention mechanism can select the most suitable operation for object perception, that in turn proves the effectiveness of our method from the perspective of gradient information.
5 CONCLUSION
In this paper, we propose for the first time to integrate Site-Transformer into NAS for architecture selection. The candidate operations are regarded tokens, and then introduce indicator token (IT) to explore the relationship between other operations and assign attention weights to which for se- lecting the optimal one. Comprehensive experiments demonstrate that IT-NAS more effective compared with other architecture parameter-based evolutionary-based architecture selection pro- cesses, and we also achieve stable results in different benchmark search spaces. This work may inspire researchers to explore more appropriate architecture selection criterion, such borrowing from pruning or other related research, to promote the progress of the NAS field.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint
arXiv: 1607.v/607.000 (607006550, 2016.
Bowen Baker, Otkrist Gupta, Nikhil Naik, and Ramesh Raskar. Designing neural network architec- tures using reinforcement learning. arXiv preprint arXiv: 1.02 2016.
Irwan Bello, Barret Zoph. Vijay Vasudevan, and Quoe Le. Neural optimizer search with rein-
forcement learning. In ICML, 2017.
Han Cai, Ligeng Zhu, and Song Han. Proxylessnas: Direct neural architecture search target task and hardware. In ICLR, 2018.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020.
Jianlong Chang, Xinbang Zhang, Yiwen Guo, Gaofeng Meng. Shiming Xiang, and Chunhong Pan. Data: Differentiable architecture approximation. NeurlPS, 2019.
Boyu Chen, Peixia Li, Baopu Li, Chen Lin, Chuming Li, Ming Sun, Junjie Yan, and Wanli Ouyang. Bn-nas: Neural architecture search with batch normalization. In ICCV, 2021a.
Boyu Chen, Peixia Li, Chuming Li, Baopu Li, Lei Bai, Chen Lin, Ming Sun, Junjie Yan, and Wanli Ouyang. Glit: Neural architecture search for global and local image transformer. In /CCV, 2021b.
Liang-Chieh Chen, Maxwell D Collins, Yukun Zhu, George Papandreou, Barret Zoph, Florian Schroff, Hartwig Adam, and Jonathon Shlens. Searching for efficient multi-scale architectures for dense image prediction. In NeurIPS, 2018.
Minghao Chen, Houwen Peng, Jianlong Fu, and Haibin Ling. Autoformer: Searching transformers for visual recognition. In ICCV, 2021c.
Xiangning Chen and Cho-Jui Hsieh. Stabilizing differentiable architecture search via perturbation- based regularization. In ICML,, 2020.
Yukang Chen, Tong Yang. Xiangyu Zhang, Gaofeng Meng, Xinyu Xiao, and Jian Sun. Detnas: Backbone search for object detection. In NeurlPS, 2019.
Xiangxiang Chu, Tianbao Zhou, Bo Zhang, and Jixiang Li. Fair darts: Eliminating unfair advantages in differentiable architecture search. ECCV, 2020.
Xiangxiang Chu, Zhi Tian, Yuqing Wang, Bo Zhang, Haibing Ren, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Twins: Revisiting the design of spatial attention in vision transformers. In NeurlPS, 2021a.
Xiangxiang Chu, Bo Zhang, and Ruijun Xu. Fairnss: Rethinking evaluation faimess of weight sharing neural architecture search. ICCV, 2021b.
Xuanyi Dong and Yi Yang. Searching for a robust neural architecture in four gpu hours. In CVPR, 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x 16 words: Transformers for image recognition at scale. In ICLR, 2021.
Golnaz Ghiasi, Tsung- Yi Lin, and Quoc V Le. Nas-fpn: Learning scalable feature pyramid archi- tecture for object detection. CVPR, 2019.
Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng. Zechun Liu, Yichen Wei, and Jian Sun. Single path one-shot neural architecture search with uniform sampling. ECCV, 2020.
Zhi Hou, Baosheng Yu, and Dacheng Tao. Batchformer: Learning to explore sample relationships for robust representation learning. In CVPR, 2022.
10

---

Under review as a conference paper at ICLR 2023
Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for mobilenetv3. In ICCV, 2019.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks. Communications of the ACM, 60(6):84-90, 2017.
Changlin Li, Jiefeng Peng. Liuchun Yuan, Guangrun Wang, Xiaodan Liang, Liang Lin, and Xiaojun Chang. Block-wisely supervised neural architecture search with knowledge distillation. In CVPR, 2020.
Changlin Li, Tao Tang. Guangrun Wang, Jiefeng Peng, Bing Wang. Xiaodan Liang, and Xiaojun Chang. Bossnas: Exploring hybrid enn cnn-transe unn-transformers with block-wisely self-supervised neural architecture search. In ICCV, 2021.
Youwei Liang, Chongjian Ge, Zhan Tong, Yibing Song, Jue Wang, and Pengtao Xie. Not all patches are what you need: Expediting vision transformers via token reorganizations. In ICLR, 2022.
Yi-Lun Liao, Sertac Karaman, and Vivienne Sze. Searching for efficient multi-stage vision trans- formers. In NeurIPS, 2021.
Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy. Progressive neural architecture search. In ECCV, 2018a.
Chenxi Liu, Liang-Chien Chich Chen, Florian Schroff, Hartwig Adam, Wei Hua, Alan Yuille, and Li Fei- Fei. Auto-dceplab: Hierarchical neural architecture search for semantic image segmentation. In CVPR, 2019.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. In ICLR, 2018b.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
Hieu Pham, Melody Guan, Barret Zoph, Quoc Le,, and Jeff Dean. Efficient neural architecture search via parameters sharing. In ICML, 2018.
Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsich. Dynamicvit: Efficient vision transformers with dynamic token sparsification. NeurIPS, 2021.
Esteban Real, Sherry Moore, Andrew Selle, Saurabh Saxena, Yutaka Leon Suematsu, Jie Tan, Quoc Le, and Alexey Kurakin. Large-scale evolution of image classifiers. In ICML, 2017.
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoe V Le. Regularized evolution for image classifier architecture search. In AAAI, 2019.
Michael Ryoo, AJ Piergiovanni, Anurag Arnab, Mostafa Dehghani, and Anelia Angelova. Token- learner: Adaptive space-time tokenization for videos. NeurIPS, 2021.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In ICCV, 2017.
Claude Shannon. Communication in the presence noise. Proceedings of the IRE, 3701):00-21, 1949.
Mingxing Tan and Quoe Le. Efficientnet: Rethinking model scaling for convolutional neural net- works. In ICML, 2019.
Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoe V Le. Mnasnet: Platform-aware neural architecture search for mobile. In CVPR, 2019.

---

Under review as a conference paper at ICLR 2023
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In ICML, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention all you need. NeurIPS, 2017.
Ruochen Wang, Minhao Cheng, Xiangning Chen, Xiaocheng Tang, and Cho-Jui Hsich. Rethinking architecture selection in differentiable nas. In ICLR, 2021a.
Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In ICCV, 2021b.
Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang, Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer. Fbnet: Hardware- efficient convnet design via differentiable neural architecture search. In CVPR, 2019.
Lichuan Xiang, Lukasz Dudziak, Mohamed S Abdelfattah, Thomas Chau, Nicholas Lane, and Hongkai Wen. Zero-cost proxies meet differentiable architecture search. arXiv preprint orXiv:2106.06799 2021.
Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Seg- former: Simple and efficient design for semantic segmentation with transformers. In NeurIPS, 2021a.
Sirui Xie, Hehui Zheng, Chunxiao Liu, and Liang Lin. Snas: stochastic neural architecture search. In ICLR, 2018
Xukai Xie, Yuan Zhou, and Sun-Yuan Kung. Exploiting operation importance for diffcrentiable neural architecture search. In TNNLS, 2021b.
Yuhui Xu, Lingxi Xie, Xiaopeng Zhang. Xin Chen, Guo-Jun Qi. Qi Tian, and Hongkai Xiong. Pc-darts: Partial channel connections for memory-efficient architecture search. ICLR, 2019.
Shan You, Tao Huang, Mingmin Yang, Fei Wang, Chen Qian, and Changshui Zhang, Greedynas: Towards fast one-shot nas with greedy supernet. In CVPR, 2020.
Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zi-Hang Jiang, Francis EH Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. In ICCV, 2021.
Arber Zela, Thomas Elsken, Tonmoy Saikia, Yassine Marrakehi, Thomas Brox, and Frank Hutter. Understanding and robustifying differentiable architecture search. In ICLR, 2020.
Miao Zhang, Shirui Pan, Xiaojun Chang, Steven Su, Jilin Hu, Gholamreza Reza Haffari, and Bin Yang. Balenas: Differentiable architecture search via the bayesian learning rule. In CVPR, 2022.
Xuanyang Zhang, Pengfei Hou, Xiangyu Zhang, and Jian Sun. Neural architecture search with random labels. In CVPR, 2021.
Daquan Zhou, Bingyi Kang, Xiaojie Jin, Linjie Yang, Xiaochen Lian, Zihang Jiang, Qibin Hou, and Jiashi Feng. Deepvit: Towards deeper vision transformer. arXiv preprint arXiv:2103. 1886, 2021.
Hongpeng Zhou, Minghao Yang, Jun Wang, and Wei Pan. Bayesnas: A bayesian approach for neural architecture search. In ICML, 2019.
Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint arXiv:0010004159 2020.
Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning, arXiv preprint arXiv:1611.01575 2016.
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoe V Le. Learning transferable architectures for scalable image recognition. In CVPR, 2018.
12

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A.1 DATASETS
We conduct the experiments different image classification datasets for various search spaces.
R-FA The dataset contains 50K training images and 10K testing images with a fixed resolution of 32x32. The training dataset is divided into two parts, one half is used for training network weights and Lite- Transformer modules, and the other half is used as the validation dataset to forward propagation once obtain the self-attention weights of each candidate operation for selecting the final network architecture.
The dataset has the same number of images as CIFAR-10 but more categorized into 100 fine-grained classes. There are 500 training images and 100 testing images per class.
CIFAR-100
ImageNet. The dataset has 1.28M training images and 50K validation images with 1000 object categories. We sample 10% of the training datasets to train the super-network weights as well as the Tran-frasfrrmer module weights, and another 2.5% training datasets to select the final architecture based on the self-attention weights.
ImageNet-16-- 120. The dataset down-samples the original ILSVRC2012 ImageNet to 16x16 reso- lution and only selects the first 120 categories.
SVHN. The dataset is digit classification benehmark dataset that contains around 600,000 32x32 RGB images of printed digits (from 0 to 9) cropped from pictures of house number plates. SVHN has three sets: 73257 digits for training, 26032 digits for testing and an extra with 531,131 images that are less difficult and can be used for helping with the training process.
A.2 SEARCH SPACES
We experiment on four popular search spaces, including DARTS search space, ProxylessNAS search space, NAS- MAS--Bnch-200 search space, and S1-S4.
DARTS Search Space. The cell-based search space aims to search for the normal cell and reduction cell. Each cell is defined as a directed acyclic graph (DAG) consisting of an ordered sequence of N nodes. The edge between two nodes is mixed up by the searchable candidate operations, including {sep-conv-3x3, sep-conv-5x5, dil-conv-3x3 dil-comv-5x5, gug-pool-3xx3 max-pool-3 x3,, identity and none].
ProxylessNAS Search Space. The chain-style search space defines the network macro architecture and directly searches for appropriate operations each layer. There a total of 21 searchable layers, and each layer includes 6 candidate bottleneck blocks with different kernel size {3, 5, 7} and expansion ratio (3, 6). Besides, the network depth can also be scaled depending on whether skip-connect operation is selected for each layer.
NAS-Bench-201 Search Space. It is also the cell-based search space, but it only needs to search for the normal cell and maintain the reduction cell as residual block with stride of two. There are 4 nodes the normal cell, resulting in searchable edges, each of which contains candidate operations, including (nor-comv-1 I, nor-comv-3 x3, 3rg-pool-3x33, identity and none).
S1-S4. The search space is the reduced original DARTS search space. Specifically, optimized space with two different candidate operations on each edge. S2 has the candidate op- crations of (sep-com--x33, identity) per edge. S3 has the candidate operations of (sep-conv-3x3, identity, none) per edge. S4 has the candidate operations of (sep-comv-3 noise) per edge.
pre-
NAS MAS-Bench-1Shol.. The Benchmark splits NAS- Bench- into search space 1, search space and search space 3. The three search spaces contain 6240, 29160, 363648 architectures respectively. Each architecture is consisted of three stacked blocks with max- max-iooolig in between. Each block contains three searchable cells, and each of cell includes nine nodes.
13

---

Under review as a conference paper at ICLR 2023
Table 5: Search results NAS-bench-201. We report the average performance for six independent runs of searching. "Optimal" indicates the highest accuracy for each dataset on NAS-Bench-
CIFAR-10
CIFAR-100
ImageNet- 120
Methods
validation
test
validation
test
validation
test
Optimal
91.61
94.37
73,49
73.51
46.77
47.31
RSPS
80.4213.55
84.07=3.61
52.12=53.55
52.31=5.77
27.22:3.24
26.28:13.09
DARTS
39.77++0.00
54.30==0.00
15.03++0.00
15.61+00.00
16.43++0.00
16.32++0.00
GDAS
89.89+-0.08
93.61+0.009
71.34+00.4
70.70+0.30
41.59++1.33
41.71++0.98
SETN
84.0410.28
87.6410.00
58.8610.06
59.05=0.22
33.06:10.02
32.52::.211
ENAS
37.51
53.89=0.58
13.37+2.35
13.9612.33
15.06
14.8442.10
SNAS
90.10#1.04
92.77=#0.33
69.69+2.39
69.34+1.98
42.84
43.16++2.64
PC-DARTS
89.16=0.115
93.41=0.30
67.12+0..39
67.48+0.89
40.83=0008
41.31
DrNAS
91.55.10.00
94.0610.00
7..49
73.51
46.37:10.00
4..34
IT-NAS
90.15=0.42
93.65=0.04
69.96+1.00
70.31+0.66
43.04
43.88
A.3 IMPLEMENTATION DETAILS
To train the super-network together with Trensfraafor module on CIFAR-I0 in DARTS search space, we use the SGD optimizer with initial learning rate 0,025, momentum 0.9 and weight decay 3 x 10 The super-networf is trained for 50 epochs with the batch size of 64. After deriving the final architecture, the target network consisted of 20 cells with initial channel size of 36 is trained on the whole training dataset from scratch. Specifically, we employ the SGD optimizer with initial learning rate 0.025, momentum 0.9 and weight decay 3 x 10-1 The target network is trained for 600 epochs with the batch size of 96.
To train the super-network together with fite-Tranffomerr module on ImageNet in ProxylessNAS search space, we uniform sample single path from the super-network per step. Specifically, we use Tesla V100 GPUs and train it for 240 epochs with total batch size of 512. The SGD optimizer with initial learning rate 0.25, momentum 0.9 and weight decay 4 x 10-5, and the minimum learning rate 10-4 The derived final network is retrained from scratch the whole ImageNet training dataset for 240 epochs with the batch size of 1024 on 8 Tesla V100 GPUs. The SGD optimizer with initial learning rate 0.5, momentum 0.9 and weight decay 4 10-5
A.4 RESULTS NAS-BENCH-201
NAS-Bench-201 is benchmark for almost up-to-date NAS algorithms, and the diagnostic infor- mation about accuracy, loss, and parameters is accessible on three datasets including CIFAR-10. 10, CIFAR-100, and ImageNet-16-120 respectively. We experiment with the search process CIFAR- 10 and then index the accuracy on three different datasets. We keep the hyper-parameters the same DARTS and repeat the experiments six times with different random seeds. The results in Table S show that DrNAS has indeed achieved state-of-ther -ortte--rttttttt results that are almost close to the global optimum. Our approach achieves comparable performance with the second best almost datasets except obtaining the third performance on CIFAR-100.
A.5 RESULTS IN PAS-BENCH- 1SHOT1
We implement IT-NAS separately in three search spaces of NAS-Bench-IShott. The hyper- parameters are kept the same as DARTS, All methods, including DARTS, GDAS, PC-DARTS, ENAS, as well as IT-NAS are independently searched three times with different seeds for fair com- parison. As shown in Figure IT-NAS achieves almost the lowest test regret in different spaces with a smaller variance after convergence, indicating the effectiveness and robustness IT-NAS.
14

---

Under review as a conference paper at ICLR 2023
COVET RECENTT E'len
Spaces
(a) Space
(b) Space2
(c) Space3
Figure 5: Comparison of IT-NAS with other One-Shot NAS methods on three different search spaces in See--Bnnh--sshll The solid lines show the anytime test regret (mean = std), while the dashed blurred lines represent validation error (Best viewed color).
Table 6: Comparison of the search results ProxylessNAS (with SE module) search space on ImageNet.
Methods
Top-1(%)
Top-5(%) (%)
MobileNetV3 Howard et al.
2019
75.2
N/A
MnasNet-A3 lan et al. 2019)
76.7
93.3
EfficientNet-BU) Tan & Te 2019)
76.3
93.2
DNA-b (Li et al. 2020)
77.5
93.3
BossNet-M2 (Li
2021
77.4
93.6
IT-NAS
78.2
94.0
A.6 RESULTS IN PROXYLESSNAS (WITH SE MODULE) SEARCH SPACE
We compare the searched results on ProxylesSNAS search space that includes
(SE) module. The experimental settings of retraining the search architecture are the same as DNA Li et al. (2020). Comparing the results in Table 3] the SE module can indeed promote achieving impressive results under the mobile setting. As shown in Table O IT-NAS achieves the best per- formance when compared with other methods with SE module, demonstrating the superiority of IT-NAS.
A.7 RANKING CONSISTENCY ANALYSIS
Instead of using the discretization accuracy convergence that proposed in DARTS-PT [Wan al. (201aa) to represent the accuracy ranking of an operation, here we leverage the best. acc definition proposed in Zero-Cost-PT (Xiang al. 2021.
accttocc(Ass
argmaa max ("(AEE)
OEO, AleleAr,e..
where V* denotes validation accuracy of network after full training, A denotes all possible fully-discretized subnetworks, & is the discretization iteration of an edge with operation
We plot the relationship between best accuracy ranking and indicator ranking of IT-NAS and CIFAR-10 dataset. The indicator can be self-attention weights of IT-
DARTS in NAS or architecture parameters of DARTS. As shown in Figure 0 we can see that IT-NAS achieves almost positive ranking consistency except on the Edge_2( Even so, IT-NAS also obtains better ranking than DARTS on Edge.2.) Besides, the Kendall tau ranking higher than DARTS on all edges, demonstrating the effectiveness of self-attention weight in evaluating the importance of can- didate operations.
To compare the ranking results with Zero-Cost-PT that leverage Zero-Cor proxy to predict the performance of candidation operation, we choose to combine Zero-Cost-PTT with NASWOT and SynFlow proxy to predict the score of each operation per edge. The results plotted in Figure
15

---

Under review as a conference paper at ICLR 2023
7 and Figure The results show that SynFlow obtains generally better accuracy ranking than NASWOT when combining with Kero-Cost-PP. but IT-NAS still achieves better ranking on most edges than Zcro-PTTNASSOO and Zero-PT-Syn Though zero-cost proxies are fast in assessing the importance of operations, its accuracy is inferior to the self-attention mechanism.
A.8
MORE EMPIRICAL STUDIES ON ACCURACY RANKING
Based on A.7 we analyze the self-attention weights architecture parameters in indicating the best accuracy ranking. We plot the accuracy and indicator of all six edges on NAS- 20--ennch-211 in Figureg [4 Taking the first edge as an example, the maximum self-attention weight and the highest accuracy of IT-NAS are both nor_conv _3x3 operation, demonstrating the effectiveness of self-attention weight in evaluating the importance of candidate operations. Whereas, the largest architecture parameter of DARTS is skip_cconnee which does not contribute to the highest accuracy, so that the final derived architecture according to alpha would not obtain the satisfactory accuracy. On the other hand, the skip connect weights always the highest across all edges, showing that the bi-level optimization of DARTS is prone to performance collapse. The search results of IT-NAS stable due the mono-level optimization.
A.9 VISUALIZATION
Here we visualize the searched normal cells and reduction cells from three different experiments with different initial random seeds DARTS search space. As shown in Figure [3] the test error and parameters of network based (a)(b) are 2.39% and 3.54M respectively, the test error and pa- rameters network based on (c) are 2.39% and 4.21M respectively; the test error and parameters of network based on (f) are 2.44% and 3.27M respectively.
Moreover, as shown in Figure 16 we also visualize the searched chain-style architecture in the ProxylessNAS search space mageNet directly.
[ separatlly shows the searched cells on CIFAR-10. CIFAR-100, and SVHN in the reduced search spaces $1-$4, respectively. The corresponding accuracy of each cell can be seen in the T
Figure 17 Figure 18
Pinnumy forunoo
IRA Ervial
Indicator Ranking (a) Edge_0
Indicator Ranking (b) Edge_1
Indicator Ranking (c) Edge.2
Murguur Kre noni
mads, Eendalnd 60
DARTS. Kandali- O
Indicator Ranking
Indicator Ranking
Ranking
(d) Edge_3
(e) Edge_4
(I) Edge_
Figure 6: The relationship between indicator ranking and accuracy ranking of IT-NAS and DARTS in NAS Be---Bench222 201 CIFAR-10 10 dataset.
16

---

Under review as a conference paper at ICLR 2023
Accuraty Ranking
Zers NANAIT (endub-
Jeiv NANAATT, Erodeine
Jon SANAAT, Eredniydas
Indicator Ranking
Indicator Ranking (b) Edge_1
Indicator Ranking
(a) Edge.0 O
(c) Edge .2
Accuraicy hanking
Acouraccy Ranking
7.445 112
TH4S Canciaio.
24 NARWOT
Zereli NASWOl Keedalne22
Ho0 Keedaind
Indicator Ranking
Indicator Ranking (e) Edge.4
Indicator Ranking
(d) Edge_
(I) Edge_5
Figure 7: The relationship between indicator ranking and accuracy ranking of IT-NAS and Zero- Cost-PT with NASWOT in JAS-Bench-201 201 on CIFAR-10 dataset.
Accuracy Ranking
odyare 2remmo
E-M5, Cantel -335
T5AM, fandal -3
TAMA, fandal -6.11
Indicator Ranking
Indicator Ranking (b) Edge_1
Indicator Ranking (c) Edge. 2
(a) Edge.0 O
Bupury bouu
Bupury focuncie
Bupury focuncii
ITAG,, Kanda
Indicator Barking
Indicator Barking
Indicator Barking
(d) Edge_3
(c) Edge_4 4
(f) Edge_
Figure 8: The relationship between indicator ranking and accuracy ranking of IT-NAA and Zero- Cost-PT with SynFlow in NAS- AAS-Bench-201 on CIFAR-10 dataset.
17

---

Under review as a conference paper at ICLR 2023
We gttts
Accuracy
Weights
Accuracy
Self Weights
94.50
Alpha Weights
0.4
94.50
94.25
94.25
Test
0.2
93.75
93.50
93.50
0.1
93.25
93.25
convest
mer.omm Jel
mone
commess
evg.pool
(a) IT-NAS
(b) DARTS
Discretization Accuracy
Attention Weights
Alpha Weights
Diseretization Accuracy
Operation
94.37
0.1728
avg_pool__ 3x3
0.1487
94.37
94.37
0.1728
nor nor_conv __33
0.171
94.30
94.30
0,1662
nor conv 1x1
0.1345
94.01
94,01
0.1659
skip_conipe connect
0.4195
94.01
93.92
0.1627
none
0.1261
93.92
IT-NAS
DARTS Kendall 0.32
Kendall 0.95
(c) The ranking of attention weights or alpha weights against discretization accuracy.
Figure 9: Comparison of self-attention weights and architecture parameters on measuring the im- portance of candidate operations both on the 1st edge of the cell in AAS-Bench-201.
We gita
Accuracy
0.5
Weights
Accuracy
94.75
attention Weight
94.50
Test Accuracaa Accuraiy
Alpha Wrights
0.6
94.50
94.25
$4.25
0.3
93.50
93.50
0.1
93.25
93 03.25
connest
mi.cim Ast
vercom, 3x2
comnec
(a) IT-NAS
(b) DARTS
Discretization Accuracy
Attention Weights
Alpha Weights
Diseretization Accuracy
Operation
94.37
0.1575
avg_pool_
0.1411
94.37
94,37
0.1996
nor con_ __x3
0.1580
94.34
94.34
0.1706
nor hor_conny 1x1
0.1024
94.18
94. 18
0.1571
skip_compee connect
0.4729
94.02
94.02
0.1610
none
0.1256
94.02
IT-NAS
DARTS
Kendall 0.53
Kendall -0032
(c) The ranking of attention weights or alpha weights against discretization accuracy.
Figure 10: Comparison of self-attention weights and architecture parameters on measuring the im- portance of candidate operations both on the 2nd edge of the cell in BAS-Bench-201.
18

---

Under review as a conference paper at ICLR 2023
Weights
Accuracy
Weights
Accuracy
ii-aattrtiion Weights
94.50
Alpha Weights
0.4
94.50
94.25
94.25
Test
0.2
93.75
93.50
93.50
0.162
0.1
93.25
93.25
connect
commess
evg.pool
(a) IT-NAS
(b) DARTS
Discretization Accuracy
Attention Weights
Alpha Weights
Diseretization Accuracy
Operation
94.37
0.1666
avg_pool__ 3x3
0,1690
94.37
94.29
0.1665
nor nor_conv
___333333333
0.1571
94.29
94.29
0.1670
nor conv 1x1
0. 0.1244
94,24
94. 24
0.1668
skip_ connect
0.3193
94.18
18
0.1665
none
94.18
IT-NAS Kendall -0.11
DARTS Kendall -0.53
(c) The ranking of attention weights or alpha weights against discretization accuracy.
Figure 11: Comparison of self-attention weights and architecture parameters on measuring the im- portance of candidate operations both on the 3rd edge of the cell in MAS-Bench-200
0.20
We Nts
Accuracy
0.5
Weights
Accuracy
94.75
Weight
0.19
94.50
94.25
Test Accuras
Alpha Wrights
0.6
94.50
$4.25
0.17
0.3
0.36
0.14
93.50
0.1
93.50
93.25
93 19.25
connect
crrcomn.. Ast
oercom, 3x2
comnec
paoi
(a) IT-NAS
(b) DARTS
Discretization Accuracy
Attention Weights
Alpha
Diseretization Accuracy
Operation
Weights
94.37
0.1849
avg_pool_33
0.1264
94.37
94.37
0.1904
nor con_ __x3
0.1393
94.37
94.02
0.1530
nor hor_conny 1x1
0. 0.1277
94,02
93.94 94
0,2078
skip_compig connect
0.5197
93,94
93.89
0.1259
none
0.0869
93.89
IT-NAS
DARTS Kendall 0.11
Kendall 0.32
(c) The ranking of attention weights or alpha weights against discretization accuracy.
Figure 12: Comparison of self-attention weights and architecture parameters on measuring the im- portance of candidate operations both on the 4th edge of the cell in NAS-Bench-201.
19

---

Under review as a conference paper at ICLR 2023
Weights
Accunacy
Weights
Accuracy
ci-atttrtiinn Weights
94.50
Alpha Weights
0.4-
94.50
94.25
94.25
Test
0.2
93.75
93.50
93.50
0.162
0.1
93.25
93.25
connect
mine
comness
evg.pool
(a) IT-NAS
(b) DARTS
Discretization Accuracy
Attention Weights
Alpha
Diseretization Accuracy
Operation
Weights
94.37
0,1669
avg_ evg_pool__t 3x3
0. 0.1422
94.37
94.37
0.1667
nor mor_conv___ __3
0.1814
94.37
94,37
0.1671
nor conv 1x1
0.1773
94.13
94. 13
0. 1664
skip_connee connect
0,3964
94.02
94,02
0.1668
none
0.1067
94.02
IT-NAS
DARTS Kendall 0
Kendall 0.60
(c) The ranking of attention weights or alpha weights against discretization accuracy.
Figure 13: Comparison of self-attention weights and architecture parameters on measuring the im-
portance of candidate operations both on the 5th edge of the cell in
Weights
Accuracy
0.5-
Weightt
Accuracy
94.75
attttennion Weights
-4.50
Test Accuras Accuraiy
Alpha Wrights
0.6
94.50
94.25
$4.25
0.3
93.50
93.50
0.162
0.1
93.25
93 03.25
connec
sony
poel i
comnec
(a) IT-NAS
(b) DARTS
Discretization Accuracy
Attention Weights
Alpha
Diseretization Accuracy
Operation
Weights
94.37
0.1672
avg_pool_
0.1502
94.37
94.37
0.1669
nor cor__cov___ __x3
0.1447
94.37
94.37
0.1667
nor conv 1x1
0. 1227
94.02
94.02
0.1663
skip_connect
0.4064
94.02
93.78
0.1662
none
0.1760
93.78
IT-NAS
DARTS Kendall -0167
Kendall 0.84
(c) The ranking of attention weights or alpha weights against discretization accuracy.
Figure 14: Comparison of self-attention weights and architecture parameters on measuring the im-
portance of candidate operations both on the 6th edge of the cell in NAS-Bench-201.
20

---

Under review as a conference paper at ICLR 2023
161
der
(a) Normal.Cell 1
(b) Redlction.CelI.1
skip connect
scp_conv 3x3
(k-2)
sep_conv _o0
0
_wg_pooo____ wtw.ppooopaaa. _x3
3
(_k)
jep_conv_3te _3x3
2
c [k-1]
dil cuny _33
sep CODy 5x5
sep_com conv__oc
Normal Cell 2
(d) Reduction Cell
coe
(e) Normal.Cell 3Cell_
(l) Reduction Cell_3
Figure 15: The searched normal cells and reduction cells on CIFAR-10 in DARTS search space.
MIBE3 K7
Identiry
NB56 Ks
MBB3 K3
MBB6 KS
MIBES K7
MBES K7
MIRR3 KS
MBK6 K3
MBB3 K7
Identity
MBE6 K3
Identity
Ideatity
MBB0
MBB3
MIBE
MBB6
Figure 16: The searched chain-style architecture on ImageNet in ProxylessNAS search space.
21

---

Under review as a conference paper at ICLR 2023
Bed
(a) S1
(b) S2
CIKK
(c) S3
(d) S4
Figure 17: IT-NAS best cells (paired in normal and reduction) on CIFAR-10 in reduced search spaces of RobustDARTS.
HE
(a) S1
(b) S2
IKI
(c) S3
(d) S4
Figure 18: IT-NAS best cells (paired in normal and reduction) CIFAR-100 in reduced search spaces of RobustDARTS.
22

---

Under review as a conference paper at ICLR 2023
(a) ST
(b) S2
(c) S3
(d) S4
Figure 19: IT-NAS best cells (paired in normal and reduction) on SVHN in reduced search spaces of RobustDARTS.
23


Reviews:
[{"review_id": "ghDUs6dS0K", "reviewer": "Reviewer_3FDx", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Existing training-based NAS algorithms typically select the final architecture that achieves the largest architecture parameters. This however can be unstable and unreliable in practice because of the limited representation ability of these architecture parameters. To this end, this paper introduces the self-attention mechanism (lite-transformer) to better characterize the importance of different operations in NAS, which has been supported by the empirical results and ablation studies in this paper.", "summary_of_the_review": "In general, this is an interesting paper and I hope the authors can address my concerns during the rebuttal period.", "strength_and_weaknesses": "strengths:\n1. This paper is generally well-written and easy to follow.\n2. To the best of my knowledge, this paper is the first to introduce a lite transformer for selecting the final architecture in NAS, which is simple yet works well.\n2. Empirical results in this paper are attractive and have achieved SOTA.\n\nweaknesses/questions:\n1. Zit in eq.4 and W in eq.5 are not well-explained. Specifically, how is Zit used during the model training of supernet and what's W in eq.5?\n2. There lacks a theoretical justification for why a lite transformer is a better alternative to the architecture parameters in DARTS.\n3. How many additional parameters have been introduced to NAS compared with the architecture parameters? Will it affect the bi-level optimization (e.g., make the optimization harder) due to an increasing number of parameters?\n4. In addition to figure 3, more empirical studies on why a lite transformer can achieve a better characterization of accuracy ranking perhaps can make this paper more attractive and inspiring.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "X7MkVkSLnVY", "reviewer": "Reviewer_CZtJ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "\nThis paper integrates Transformer into NAS for architecture selection by regarding each candidate operation as a patch and also introducing an additional importance indicator token (IT) to calculate cross-attention among the other operational tokens. \n", "summary_of_the_review": "\nThis method seems to apply an existing well-known approach (transformer architecture) into architecture search to address the architecture selection problem. However, compared to earlier works, like DARTS-PT, Zero-cost-PT and BossNAS, they did not provide sufficient evidence that their method can indeed generate better architecture ranking results. The only proof is they show is via the final searched architecture results, which is interesting but definitely inadequate to prove the claim is valid.\n\n---\n\nUpdate after reading the response\n\nI would like to thank the authors to provide such a detailed and comprehensive responses to address my concerns. Before the revision, this paper suffers from two major weaknesses, marginally technical novelty that directly apply the transformer self-attention mechanism without much support of their claim, using self-attention can boost the architecture selection problem; insufficient comparison with the recent literature. After the rebuttal, several of my previous concerns are addressed, especially regarding the empirical one. IT-NAS surpasses many state-of-the-art methods in the differentiable architecture search domain. \n\nHowever, as there is no borderline score, I would still maintain my score to reject this work for the following reasons. \n\n1. Major revision needed to truly address the weakness but is not reflected in the current revision. \nMany interesting results to support their claims (such as Table 5, A.4, A.6, A.7) are still in the appendix, even in their revised version. With respect, if I read this paper again without the supplementary material, I would still have the same concern. This is because reading supplemenary materials should not be mandatory to the reviewers or general readers. In the revised version, for example, Figure 2, 3, only shows the comparison between IT-NAS with the original DARTS, giving the readers a feeling that they only compared to the weakest baseline that was introduced almost four years ago. I am aware these can be fixed, but it is essentially equivalent to re-write the entire experiment section. Given such substantial changes, I would recommend this paper to go over another review cycle. \n\n2. Incremental technical novelty to use a well-known technique, self-attention to another domain.\nThis work seems novel in the first screening, however, it does not change the fact that this is simple attempt to use a well known successful technique, self-attention, into another domain, differentiable architecture search. This is not wrong, but leads to incremental technical novelty. \n\nCombining these two, I cannot recommend acceptance at this time.\n\n", "strength_and_weaknesses": "# Strength:\n\nThe paper proposes to explore self-attention to select the superior operations of NAS.\n\n# Weaknesses:\n\nAs claimed in the introduction, the motivation to incorporate transformer into differentiable architecture search pipeline is to improve the architecture selection process in differentiable NAS algorithms. So ideally there should be two aspect that this IT-NAS is truly superior:\n\nShowing the search results is superior on commonly benchmarked dataset, such as in DARTS+PT original paper. Recently, there is a zero-cost PT showing a best architecture of 2.43 top1 error on CIFAR-10, while only using 0.018 hours search cost. Compared to this transformer based one, I do not see much reason why involving a transformer can improve the result. And the figure 3 is showing their architecture ranking is superior than the original DARTS in 2022 is too weak to be accepted to the top-tier conference.\nResults on ProxylessNAS search space are not state-of-the-art, e.g., in BossNAS, the architecture search results on proxylessNAS space achieve 77.4% in 2021 under the 600M FLOPs constraint. I hope the authors can consider doing their experiment in recent literature rather than comparing to those out-dated methods. \n\nIn one word, this paper claim to discover a better architecture ranking in differentiable architecture search domain, I suggest the authors take a closer look at these papers and provide a detailed comparison to show IT-NAS truly surpasses these baselines. \n\n[a] Zero-Cost Proxies Meet Differentiable Architecture Search, Xiang et al. \n\n[b] BossNAS: Exploring Hybrid CNN-transformers with Block-wisely Self-supervised NAS, Li et al.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "JzlRIlvktn", "reviewer": "Reviewer_o9WQ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to apply NAS to architecture selection in Transformers. Each candidate operation is measured by the attention mechanism that can be learned and optimized in the network training. The proposed method is insightful and interesting, and achieves competitive performance in efficiency and accuracy, as well as the search quality.\n", "summary_of_the_review": "I would lean to accept the paper in terms of its improvements over the last version. Some critical points still need to be further clarified. ", "strength_and_weaknesses": "[ Strength ]\n+ The proposed method seems new and interesting. Search space selection/optimization is a classical topic in NAS, and the authors take advantage of popular transformers (more precisely, the attention mechanism) to measure the importance of each candidate operation. This is intuitive and straightforward.\n+ In-depth analysis in Section 3.4 is important and informative. It helps to better understand the proposed method, especially the motivation of introducing the indicator token. The complexity analysis is also very important because it would be confusing that the complexity is quadratic to the length of input tokens (i.e., the number of candidate operations) without the analysis.\n+ Experimental results can well support the effectiveness of the proposed method. It achieves good performance in accuracy, efficiency, and search quality.\n\n[ For further improvements ]\nIn essence, the method applies the attention mechanism into NAS. It looks like that this intuitive operation is just engineering-level rather than being important to the scientific research of NAS or transformers. Also, using attention for selection is not the unique property of transformers, i.e., some other network structures may also use attention for selection. It seems that the role of transformers here is a little bit over-claimed (It is also confusing that the authors use the term \"lite-transformer\" as it is just the very common transformer module).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "3shRyt40Nef", "reviewer": "Reviewer_2Hwu", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Motivated by the fact that architecture parameters in one-shot differentiable NAS are not indicative of the true operator importance, the authors integrate the Lite-Transformer into the one-shot model, and train it jointly with the model, to predict the ranking of operators at each edge / layer. \n\nThe operations are passed into the Lite-Transformer as a sequence, together with an extra indicator token whose attention to other operational tokens determines the operator importance. \n\nComparison with other search strategies is performed on DARTS, ProxylessNAS, S1-S4 and NAS-Bench-201 search spaces, on most of which superior performance is achieved.", "summary_of_the_review": "The idea is very interesting and worth exploring, and the results seem promising. However, some claims are disputable, and the presentation can be slightly improved. I would therefore rate the paper as marginally above the acceptance threshold. ", "strength_and_weaknesses": "Using the self-attention mechanism to rank operations is interesting, especially that the Lite-Transformer is trained jointly with the one-shot model, making the search a mono-level optimization.  \n\nHowever, the experiments on the NAS-Bench-201 space do not include comparisons with some well-known methods such as DARTS-PT (already included in Table.2 for comparisons on the DARTS search space) and DrNAS. Based on the results in Table.5, the method does not seem to achieve state-of-the-art performance on NAS-Bench-201 as claimed in the summary of contributions (third bullet point). \n\nIn order have a better assessment of the method and perhaps strengthen the results it would be useful to extend the experiments to other search spaces, compatible with one-shot optimization. NAS-Bench-101 which is also a tabular benchmark, would be a natural choice.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Decision: Reject