A Surgery of the Neural Architecture Evaluators
Authors: Xuefei Ning
Wenshuo Li
Zixuan Zhou
Tianchen Zhao
Shuang Liang
Yin Zheng
Huazhong Yang
Yu Wang
Venue:  (2021)
Keywords: Neural architecture search (NAS)
Parameter Sharing NAS
Predictor-based NAS

Abstract:  Neural architecture search (NAS) has recently received extensive attention due to its effectiveness in automatically designing effective neural architectures. A major challenge in NAS is to conduct a fast and accurate evaluation of neural architectures. Commonly used fast architecture evaluators include parameter-sharing ones and predictor-based ones. Despite their high evaluation efficiency, the evaluation correlation (especially of the well-performing architectures) is still questionable. In this paper, we conduct an extensive assessment of both the parameter-sharing and predictor-based evaluators on the NAS-Bench-201 search space, and break up how and why different configurations and strategies influence the fitness of the evaluators. Specifically, we carefully develop a set of NAS-oriented criteria to understand the behavior of fast architecture evaluators in different training stages. And based on the findings of our experiments, we give pieces of knowledge and suggestions to guide NAS application and motivate further research.

Under review as a conference paper at ICLR 2021
A SURGERY OF THE NEURAL ARCHITECTURE EVALU- ATORS
A Anonymout authors
Paper under double-blind review
ABSTRACT
Neural architecture search (NAS) has recently received extensive attention due to its effectiveness in automatically designing effective neural architectures. A major challenge in NAS is to conduct a fast and accurate evaluation (i.c., performance estimation) of neural architectures. Commonly used fast architecture evaluators include parameter-sharing ones and predictor-base based ones. Despite their high eval- uation efficiency, the evaluation correlation (especially of the perforrmng ar- chitectures) is still questionable. In this paper, we conduct an extensive assess- ment of both the parameter-sharing and redicctor-based evaluators on the NAS- Bench-201 search space, and break up how and why different configurations and strategics influence the fitness of the evaluators. Specifically, we develop a set of NAS-oriented criteria to understand the behavior of fast architecture evaluators in different training stages. And based on the findings of our experiments, we give pieces of knowledge and suggestions to guide NAS application and motivate further research.
INTRODUCTION
Studies have shown that the automatically discovered architectures by NAS can outperform the hand-crafted architectures for various applications, such as classification (Nayman et al., Zoph
& Le 2017), detection (Ghiasi et al.] [2019] [hen et al. (019b), video understanding (Ryoo al.] 2019 text modeling (Zoph & Le 2017), etc. Harly NAS algorithms (Zoph & Le 2017) suffer from the extremely heavy computational burden, since the evaluation of neural architectures is slow. Thus, how to estimate the performance of neural architecture in fast and accurate way is vital for addressing the computational challenge NAS.
A neural architecture evaluator outputs the evaluated score of an architecture that indicates its qual- ity. The straightforward solution is to train an architecture from scratch to convergence and then test it on the validation dataset, which is extremely time- consuminn..ing. Instead of exactly evaluating architectures on the target task, researchers usually construct a proxy model with fewer layers or fewer channels (Pham et al.) 2018 Real et al.) 2019 Wu al. 2019), and train this model solve a proxy task of smaller scales (Cai et al.) 2018a] Elsken et al. 2018 Klein et al.] (017]] Wu et al., 2019) e.g., smaller dataset or subsets of dataset, training or finetuning for fewer epochs.
Traditional evaluators conduct separate training phases to acquire the weights that are suitable for each architecture. In contrast, one-shot evaluation amortized the training cost of different architec- tures through parameter sharing or a global hypernetwok,, thus significantly reduce the evaluation cost. Pham et al. (2018) constructs an over- wver-parametrieed super network (supernet) such that all ar- chitectures in the search space are sub-architecture of the supernet. Throughout the search process, the shared parameters the supernet are updated on the training dataset split, and each architecture is evaluated by directly using the corresponding subset of the weights. Afterwards, the parameter sharing technique is widely used for architecture search different search spaces (Wu et al.) 2019 Cai et al. 2020), incorporated with different search strategies (Liu et al.) 220 Nayman et al. 2019 Xie et al. 2018 Yang et al., 2019 Cai et al. 2020 Hypernetwork (Brock et al. 2018 Zhang al. 2018) based evaluation another type of shot evaluation strategy, in which hypernet- work 18 trained to generate proper weights for each architecture. Since hypernetwork solutions are not generic currently, this paper concentrates the evaluation of parameter sharing evaluators.

---

Under review as a conference paper at ICLR 2021
One-Shot (Shared weights)
Training
Gradient
Inference
Update
Train Loss
SuperNet
One-Shot (Hypernetwork)
Hio HyperNet
Gradient
Update
Valid Accuracy
(Evaluator Output)
Architecture (Evaluator Input)
Generate weights
Architecture (with weights)
[Arch, 78.3%) (Arch, 80.4%]
Predictor-based
Ground-Truth Performance
Performance Loss
Gradient
Provided by another
Update
"oracle" evaluator
Encoder
MLP
Predicted Score
(Evaluator Output)
Architecture (Evaluator input)
Predictor
Figure 1: An overview of fast neural architecture evaluators (i.e., performance estimators).
Whether or not one-shot strategies can provide highly-correlated architecture evaluation results is essential for the efficacy of the NAS process. Many recent studies have been focusing on assessing the evaluation correlation of one-shot architecture evaluators (Bender 2018 Sciuto al. 2019
Zela et al. 2020).
Besides one-shot evaluation strategies, predictor-based evaluation strategies (Luo al. 2018 Liu
et al. 201 2a Deng et al.) 2017, Sun et al. 2019; Wang et al.) 2018 Xu et al.) 2019] Ning et al.)
2020 use a performance predictor that takes the architecture description as inputs and outputs predicted performance score. The performance predictor should be trained using "ground-truth" architecture performances. This paper utilizes the same set of criteria to evaluate and compare different performance predictors.
The fast neural architecture evaluators (i.e., performance estimators) are summarized in Fig. D in- cluding parameter sharing, hypernetworks, and predictor-basee based ones. And this paper aims reveal- ing the status of current architecture evaluation strategies systematically. Specifically, we develop of NAS-oriented criteria to understand the behavior of fast architecture evaluators in different training stages. And based on the findings of our experiments, we give pieces of knowledge and suggestions guide NAS application and motivate further research.
The knowledge revealed by this paper includes: 1) Layer proxy brings a larger evaluation gap than using channel proxy, thus channel proxy be utilized reduce the computational cost, while proxy-less search w.r.t the layer number is worth studying. 2) The convergence rate of different cri- teria varies during the one-shot supernet training, which shows that the good architectures are distin- guished from bad architectures in the early stage. 3) As training goes on, the one-shot performances of isomorphic sub-architectures become closer. 4) De-isomorphic sampling or post de-isomorphism handling can help avoid the over-estimation of simple architectures. 5) Parameter sharing evaluator tends to over-estimate smaller architectures, and better at comparing smaller models than larger models. 6) One should use ranking losses rather than regression losses to train predictors, since they are more stable. 7) Different predictors under- or over estimate different architectures, and cur- rently, the best predictor might still have trouble in comparing large architectures. 8) As expected, architecture predictors can distinguish good architectures better after multiple stages of training, as the training data are more and more concentrated on the good architectures.
2

---

Under review as a conference paper at ICLR 2021
2 RELATED WORK
2.1
ONE-SHOT EVALUATORS
One-shot evaluation mainly consists of two types of strategies: 1) parameter sharing (Pham et al.
2018 Wu et al. 2019 Liu et al. 2018b Nayman et al. 2019 Xie et 2018: Yang al. 2019 Cai
et al. 2020p. 2) hypernetworks (Brock et al. 2018 Zhang et al. 2018). These two strategies both
amortize the training cost of different architectures via the sharing of the network or hypernetwork parameters.
The ranking correlation gaps of existing shared weights evaluators are brought by two factors: 1) proxy model and task: due to the memory constraint, a proxy supernet (supernet) (L.iu et al. 2018b) Pham et al. 2018) with fewer channels or layers is usually used: 2) parameter sharing. To alleviate the first factor, there are some studies (Cai al.) [018b] Chen et al., 2019a) that aim making one- shot evaluation more memory efficient, thus the one-shot search could be conducted without using proxy supernet. As for the second factor, there are few studies that carried out correlation eval- uation for one-shot evaluators. Zhang et al. (2018) conducted a correlation comparison between the GHN hypernetwork evaluator, shared weights evaluator, and several small proxy tasks. However, the correlation is evaluated using 100 architectures randomly sampled from a large search space, which is not convincing and consistent benchmark metric. [uo al. (2019) did preliminary investigation into why parameter sharing evaluation fails to provide correlated evaluations, and pro- posed to increase the sample probabilities of the large models. Their evaluation is also conducted on dozens of architectures sampled from the search space. Zela et al. (2020) compare the evaluation correlation of different search strategies on NAS- LAS--Bnnh- (01.|Sciuto sharing NAS in toy RNN search space with only 32 architectures in total, and discover that the improve the
al. (2019) conduct parameter
parameter sharing rankings do not correlate with the true rankings of architectures. evaluation correlation, Chu et al. (219)) proposed a sampling strategy in a layer-wise search space.
In this paper, we analyze the ranking correlation gaps brought by the model proxy (difference in the number of channels and layers) and the parameter sharing technique, well as the behavior of one-shot evaluators during the training process.
2.2 PREDICTOR-BASER EVALUATORS
An architecture performance predictor takes the architecture descriptions as inputs, and outputs the predicted performance scores without training the architectures. Two factors are crucial to the fitness of the predictors: 1) embedding space: 2) training technique. On the one hand, to embed neural architectures into continuous space and get meaningful embedding space, there are studies that
propose different architecture encoders, e.g., sequence-base (Luo et al.) 2018: Liu et al.] 2018 Sa Deng et al. 2017 Sun et al.) 2019: Wang et al. (2088), graph-based (Shi et al. 2019 Ning et al., 2020 As for nonparametric predictors, Kandasamy et al. (20T8) design a kernel function in the architecture space and exploits gaussian process to get the posterior of the architecture performances. Shi et al. (2019) combined graph-based encoder and nonparametric gaussian process to construct the performance predictor. On the other hand, from the aspect of training techniques, [uo et al. (2018) employed an encoder-decoder structure and used an auxiliary reconstruction loss term. Xu al. (0119): Ning et al. (2020) employed learning-to-rank techniques train the predictors.
Actually, in the overall NAS framework, the predictor-based evaluator plays a different role from the traditional one-shot evaluators, since the predictor should be trained using "ground-truth" architecture performances. Usually, expensive traditional evaluators that can provide relatively ac- curate architecture performances are chosen the "oracle" evaluators to output the "ground-truthh
scores (Kandasamy et al. 2018 Liu et al. 2018a Luo et al. 2018).
EVALUATION CRITERIA
In this section, introduce the evaluation criteria used in this paper. We denote the search space size as M, the true performances and approximated evaluated scores of architectures {a.)=.,..... .M as (us hi .M and (sx is,.,......MM ,M, respectively. And we denote the ranking of the true performance
3

---

Under review as a conference paper at ICLR 2021
Y/ and the evaluated score s; r3 (1,... M and n, {1, MM (r; = 1 indicates that a, is the best architecture the search space). The correlation criteria adopted in our paper are
Linear correlation: The pearson correlation coefficient corr(y, s)/ corr(y- c))rrr((s (s, s). Kendall's Tau ranking correlation: The relative difference of concordant pairs and discor- dant pairs Ei<ssgn(t sgn(yi - yj (sgn(s; -,)/(2))
Spearman's 's ranking correlation: The pearson correlation coefficient between the rank vari- ables corr(r, n/) /cor((r, c))
Besides these correlation criteria, criteria that emphasize more the relative order of architectures with good performances are desired. Denoting Aς = {az in; < KMM as the set of architectures whose evaluated scores is among the top K portion of the search space,
criteira
Precision K (P@K) E (0, 1] = # m)<KM), The proportion of true top-K proportion architectures in the top-K architectures according to the scores.
KM
BestRanking @K K (BR@K) E (0. arg min The best normalized ranking among the top proportion of architectures according to the scores.
The criteria similar those used in Ning al. (020), except that rankings and architecture numbers are all normalized with respect to the search space size M.
The above criteria are used to compare the fitness of various architecture evaluators with different configurations and in different stages. Besides that, we'd also like to interpret their evaluation results. To identify which architectures are under- over-estimated by various evaluators, and analyze the reasons accordingly, we investigate the relationship of the true-predicted ranking differences (ra n4)=1,.... .M and the architecture properties such as the FLOPs: (a)
PARAMETER-SHANE SHARING EVALUATORS
In this section, assess the behavior of one-shot evaluators and evaluate the influence of several sampling strategies and training techniques.
4.1 EXPERIMENTAL SETUP
During the supernet training process, candidate architectures are randomly sampled, and their cor- responding weights are updated in every iteration.
We conduct our experiments on CIFAR-10 using a recent NAS benchmarking search space, NAS- Bench-201 (Dong & Yang 2020). TAS-Bench-201 is NAS benchmark that provides the perfor- mances of all the 15625 architectures in a cell-based search space. Actually, there are architectures with different matrix representations that isomorphic (i.e., represent the same architecture) in this search space. As reported by their original paper, there are 6466 unique topology structures after de-isomorphism.
The hyper-paramente parameters used to train all the parameter-sharing supernets are summarized in Tab. 1 We train parameter sharing evaluators via momentum SGD with momentum 0.9 and weight decay 0.0005. The batch size is set to 512. The learning rate is set to 0.05 initially and decayed by 0.5 each time the supernet accuracy stops increase for 30 epochs. During training, the dropout rate before the fully-connected classifier is set to 0.1, and the gradient norm is clipped to be less than 5.0.
4.2 TREND OF DIFFERENT CRITERIA
We inspect how BR@K, P@K, and the correlation criteria converge during the training process. We train a parameter sharing model with 17 layers and 16 initial channels on the de-isomorphic NAS- Bench-201 search space. Shown in Fig. 2 a). the speed convergence highly different. BR@K K converges very fast. P@K K converges in around 250 epochs and then Meanwhile, linear correlation, Kendall's Tau, and Spearman correlation are still growing till 500 epochs, while the parameter sharing accuracy grows during the whole 1000 epochs. This indicates
gradually decreases.
4

---

Under review as a conference paper at ICLR 2021
Table 1: Supernet training hyper-parameterss
optimizer
SGD
initial LR
0.05
momentum
0.9
LR schedule
ReduceL.ROnPlateer
weight decay
0.0005
LR decay
0.5
batch size
512
LR patience
30
dropout rate
0.1
grad norm clip
5.0
t:
Learning rate is decayed by 0.5 when the loss stops decreasing for 30
epochs, see htttp::/pytorree
www//orgg..o.oooo.ooo htt.ddoosssaalleeeeeoeee stable http:/stabeeeepteeeeeee// htm.t//ttt..mmtt...........
opt.im. we__srsssuullerre RRO
0.6-
PigSs
Weight Sharing Accurscy
0.6
Pg20%
7g5%
Kendal's Thu
Unner Cor cettintioon
Speorman Corsittiio
Bhess
0.2
02
(a)
(b)
Figure Dynamics of different criteria during the supernet training process.
that the models with different rankings change at different speeds as the training progresses, and the top-ranked models stand out faster. Another evidence is shown in Fig. b that P@5% converges much faster than P@ 50%. This means that as the training goes on, the supernet is learning to compare architectures with medium or poor performances instead of good ones. Another fact note in Fig- bb) (also see Tab. 3 is that P@5% shows slow decreasing trend from 200 epochs on. This might be due that, while the some best architectures stand out very fast one-shot training, their one-shot performances will be caught up with by others as the training goes on.
4.3 SAMPLING STRATEGY
The 201 search space includes many isomorphic architectures. We expect that one-shot evaluators could handle isomorphic architectures correctly, which means that their evaluated rewards should be close. We calculate the average variances of test accuracy and ranking isomorphic groups during the training process, shown in Tab. 2 As the training progresses, the variance
8.10-
Kerca Tea
LCVomdaal's TRUNCIINE
2.02
AV Aooraa ogggge Ramic DIT
6.2
Kondail's Teka
6.00
Brugls
LCVCondal's 6.6
Leaar Camrittee Poo%s
Laear Cora aton 9g5%
e45-
Layers
Initial Channels
Ranking
K2%
(a)
(b)
(c)
Figure 3: (a) Influence layer proxy. (b) Influence channel proxy. (c) Kendall's Tau and average rank difference of architectures in different FLOPs groups
5

---

Under review as a conference paper at ICLR 2021
within the group gradually shrinks, which indicates that more sufficient training makes one-shot evaluator handle isomorphic architectures better.
Table 2:
Average standard deviation of accuracies and rankings in architecture isomorphic groups.
"GT" stands for the average of the Ground-Truth performance deviations in all isomorphic groups
epochs
GT
200
400
600
800
1000
Accuracy std (%)
0.0392
0.193
0.161
0.0901
0.0977
0.0911
Global Accuracy std (%)
12.9
11.4
16.5
13.3
21.1
21.1
Ranking std
65.1
162
153
123
116
117
We compare the results of sampling with/without isomorphic architectures during training in Tab. The results show that if de-isomorphism sampling is not used, the supernet performs much worse on good architectures in that BR %, BR 55% and P@5% are significantly worse (2.515% V.S. 0.015%, 2.221% % V.S. 0.015%, 9.22% V.S. 28.48%). In this case, we find that the top-ranked cell architectures are simple architectures (e.g., single convolution). That is to say, parameter sharing training without de-isomorphism training might over-estimate simple architectures. This might be due to that the equivalent sampling probability is larger for these simple architectures with many isomorphic counterparts. We also experiment with the post de-isomorphism technique, in which the performances of architectures in each isomorphic group changes are incorporated in the training process. We find that post de-isomorphism can achieve im- provements over "No De-isomorphism" This might owe to the decreased variance of the estimates.
averaged during testing, while
Table 3: Comparison of (no) de-isomorphism sampling in supernet training. "BR @ 1" indicates the relative ground- truth ranking of the architecture with the highest one-shot score
Epochs
criterion
200
400
600
800
1000
BR @
3.372%
4,493%
21.555%
13.139%
11.379%
BR 1%
0.012%
0.019%
2.854%
2.739%
2.515%
BR @ 5%
0.006%
0.013%
2.336%
2.310%
2.221%
No De-isomorphism
P@ 5%
39.69%
12.03%
6.53%
7.68%
9.22%
P@ 50%
79.27%
81.59%
88.04%
89.20%
89.72%
0.6033
0.6500
0.6951
0.7032
0.7127
BR @ I
1.840%
2.737%
5.212%
5.707%
2.273%
BR @ 1% 0.046%
0.031%
0.062%
0.139%
0.015%
BR @ 5%
0.015%
0.015%
0.045%
0.090%
0.015%
De-isomorphism
P@ 5%
41.48%
39.31%
30.65%
28.48%
28.48%
P@ 50%
73.03%
83.76%
87.44%
86.39%
86.45%
0.4967
0.6735
0.7087
0.6989
0.7005
BR
3.866%
4.964%
20.260%
12.465%
10.950%
BR @ 1 %
0.015%
0.015%
0.479%
0.634%
0.247%
BR @ 5%
0.015%
0.015%
0.340%
0.217%
0.093%
Post De-isomorphism
P@5%%
46.13%
37.77%
23.52%
21.36%
24.77%
P@ 50%
76.89%
83.14%
87.44%
87.44%
87.81%
0.5511
0.6558
0.7123
0.7109
0.7226
Tab. shows the comparison of using different architecture sample numbers in supernet training. We can see that using multiple architecture MC samples is not beneficial, and MC sample=1 is good choice. We also adapt Fair-NAS (Chu al. (019)) sampling strategy the NAS-Bench-201 search space (a special case of MC sample 5), and find does not bring improvements.
4.4 PROXY MODEL.
Due to memory and time constraints, it is common to use a shallower or thinner proxy model the search process. The common practice is to search using small proxy models with fewer channels and layers, and then "model augment" the discovered architecture to large neural networks. From the experimental results shown in Fig. ab)b),
that channel proxy has little influence while

---

Under review as a conference paper at ICLR 2021
Table 4: Comparison of using different numbers of architecture Monte-Carlo samples in every super- net training step. Upper: The training epochs of models with 1/3/5 MC samples and Fair-NAS are 1000 /3/33333/22 10//200/22 /200. Lower: The training epochs of models with 1/3/5 MC samples and Fair-NAS are all 1000. All these results are tested with post de- isomorphism
Equivalent 1000 epochs
3
Fair-NAS (Chu et al.
2019
BR 55%
0.093%
0.139%
0,495%
0.139% 11.76%
P@5% 7
24.77%
9.60%
20.74%
0.7226
0.7128
0.6714
0.7137
1000 epochs
1
3
5
Fair-NAS (Chu et al. 2019
BR @5%
0.093%
0.015%
0.124%
0.031 %
P@5%
24.77%
14.24%
17.03%
15.17% 0.6965
TT
0.7226
0.7025
0.7018
layer proxy reduces the reliability of search results. Thus, for cell-based search spaces, proxy-less search w.r.t the layer number worth studying.
4.5 OVER- AND UNDER-ESTIMATION OF ARCHITECTURES
For one-shot evaluators, we expect that the training process is fair and balance for all architectures. However, sub-architectures have different amounts of calculation, and they might converge with different speed. To understand which architectures are under- or over-estimated by the one-shot evaluators, we inspect the Ranking Diff' of the ground truth performance and the one-shot evaluation of an architecture a,: T; Thg, We divide the architectures into ten groups according to the amount of calculation (FLOPs), and show Kendall's Tau and Average Rank Diff of each group in Fig, B(.).
Note that a positive Ranking Diff indicates that this architecture is evtr-estimated, otherwise it is underestimated. The x-axis is organized such that the architecture group with the least FLOPs is at the leftmost. The average Rank Diff shows a decreasing trend, which means that the larger the model, the easier is to be underestimated. Also, the decreasing intra-group Kendall's Tau indicates that harder for the one-shot evaluator to compare larger models (which usually have better performances) than comparing smaller models.
PREDICTOR- BASSDD EVALUATORS
In this section, we employ the same criteria (i.e., Kendall's Tau, Precision K, BestRanking @K K) to assess the architecture predictors (i.e., predictor-based evaluators).
5.1 EXPERIMENTAL SETUP
We experiment with four different architecture predictors: MLP, LSTM, GATES (Ning al.. 2020 and random forest regressor (RF). For MIP, LSTM, and RF, we serialize each architecture matrix using the six elements of its lower triangular portion. We follow (Ning et al. 2020) to construct MIP, LSTM, and GATES. The RF predictor applies a random forest regressor on the 6-dim sequence.
For optimizing MLP, LSTM, and GATES, an ADAM optimizer with learning rate of 1e-3 is used, the batch size is 512, and the training lasts for 200 epochs. Following (Ning al.) 2020). hinge pairwise ranking loss with margin 0.1 is used for training these predictors. For RF, random forest with 100 CARTs to predict architecture performances.
5.2 EVALUATION RESULTS
We train these predictors on training sets of different sizes: 39 (0.25%), 78 (0.5%), 390 (2.5%), 781 (5%). Specifically, for each training set size, randomly sample three different training sets, and train each predictor on each training set with three different random seeds (20, 2020, 202020), After training, we evaluate each model on the whole NAS-Bench-200 search space using Kendall's Tau,

---

Under review as a conference paper at ICLR 2021
LSTM
MuP
13802
04
0.3
LSTN
0.2
(a) Kendall's 's Tau
(b) P@5%
(c) BR@ 1%
Figure 4: Comparison predictor performances. x axis: Different training set size. Each line contains the results of 3 differently randomly sampled training set. The results of using different training seeds are averaged.
0.3
3.8
0.8
0.7
0.6
0.4
Pg0.15
GATES Pg0.1%
02
PS15%
0.3
P010%
LSTH Pgo 1%
LSTH Pi2
LSTM Pg1%
02
LSTN Pg13%
LITH PS10%.
LSTN PO3
61
(a) P@K (N=78)
(b) P@K (N=781)
Figure 5: Comparison of predictors after training of 5 stages. In each stage, N architecture is chosen, evaluated, and used to train the predictor along with previous architecture data.
Precision@E @K K, and BestRanking @ K. Different from (Ning et al. 2020), the evaluation is carried all the architectures, instead of separate validation split. As shown in Fig. GATES outperforms other predictors in all settings.
As can be seen, training with different seeds on different training sets leads similar results. In contrast, we found that training predictors with regression loss is not stable and sensitive to the choice of the training set. For example, the Kendall's Taus of GATES models trained using the regression loss on different training sets of size 78 are 0.7135, 0.7240, and 0.2067, respectively, while with ranking loss, the results are 0.7597, 0.7750, and 0.7645, respectively. These additional results are listed in the Appendix.
As illustrated Fig. Al the Appendix, there are usually multiple stages in a predictor-based NAS search process. In each stage the current predictor is used to choose N architectures to be evaluated by the oracle evaluator, then the results will be used to update the predictor. In our experiment, we select the top-N architectures predicted by the current predictor in the whole search space. As shown in Fig. P@0.1 % increases a lot after multiple stages of training, which indicates that the predictors perform better and better distinguishing the good architectures. This is expected since the training data are more concentrated on the good architectures. Also, the performance gap between different predictors (LSTM/GATES) shrinks more training data become available. The Kendall's Tau, BR @ K, and performance distribution results of multiple stages are listed in the Appendix.
CONCLUSION
This paper assesses parameter-sharie evaluators and architecture predictors on the search space, with a set carefully developed criteria. Hopefully, knowledge revealed by this paper guide future NAS application and motivate further research.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Gabriel Bender, Pieter-Jan Kindermans, Barret Zoph, Vijay Vasudevan, and Quoe Le. Understand- ing and simplifying one-shot architecture search. International Conference on Machine Learn- ing. pp. 550-559, 2018.
Andrew Brock, Theodore Lim, James Millar Ritchie, and Nicholas J Weston. Smash: One-shot model architecture search through hypernetworks. In 6th International Conference on Learning Representations, 2018.
Han Cai, Tianyao Chen, Weinan Zhang, Yong Yu, and Jun Wang. Efficient architecture search by network transformation. In Thirty-Second AAAI conference on artificial intelligence, 2018a.
Han Cai, Ligeng Zhu, and Song Han. Proxylessnas: Direct neural architecture search on target task 2.00/22.000...... 2018b.
and hardware. arXiv preprint
Han Cai, Chuang Gan, Tianzhe Wang. Zhekai Zhang, and Song Han. Once for all: Train one
network and specialize for efficient deployment. In International Conference on Learning Representations, 2020. URL http://pp../aaaav.oo//////////// www.:.....999...... pddf
Xin Chen, Lingxi Xie, Jun Wu, and Qi Tian. Progressive differentiable architecture search: Bridging the depth gap between search and evaluation. In Proceedings of the IEEE International Confer- ence on Computer Vision, pp. 1294-1303, 2019a.
Yukang Chen, Tong Yang, Xiangyu Zhang, Gaofeng Meng, Xinyu Xiao, and Jian Sun. Detnas: Backbone search for object detection. Advances Neural Information Processing Systems, pp. 6638-6648, 2019b.
Xiangxiang Chu, Bo Zhang, Ruijun Xu, and Jixiang Li. Fairmas: Rethinking evaluation faimess of weight sharing neural architecture search. arXiv preprint arXiv:1907.01844 2019.
Boyang Deng. Junjie Yan, and Dahua Lin. Peephole: Predicting network performance before train- ing. arXiv preprint arXiv: 171.0337,, 2017.
Xuanyi Dong and Yi Yang. Nas-bench-201: Extending the scope of reproducible neural architec-
ture search. In International Conference on Learning Representations, 2020. URL https:
//openeviiww /ww//eorumeeeee wtw.mmmiidd-iiiiiii HJxy2xBKDr
Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Efficient multi-objective neural architecture search via lamarckian evolution. International Conference on Learning Representations, 2018. 8.
Golnaz Ghiasi, Tsung- Yi Lin, and Quoe Le. Nas-fpn: Learning scalable feature pyramid architec- ture for object detection. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 7036-7045, 2019.
Kirthevasan Kandasamy, Willie Neiswanger, Jeff Schneider, Barnabas Poczos, and Eric P Xing. Neural architecture search with bayesian optimisation and optimal transport. In Advances in Neural Information Processing Systems, pp. 2016-2025, 2018
Aaron Klein, Stefan Falkner, Simon Bartels, Philipp Hennig. Frank Hutter, et al. Fast bayesian hyperparameter optimization on large datasets. Electronic Journal of Statistics, 11(2):945549968 2017.
Yanxi Li, Minjing Dong, Yunhe Wang, and Chang Xu. Neural architecture search in a proxy valida- tion loss landscape. In International Conference Machine Learning, 2020.
Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy. Progressive neural architecture search. In Proceed- ings of the European Conference Computer Vision (ECCV), pp. 19-34, 2018a.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. arXiv 2018b.
preprint 1
Renqian Luo, Fei Tian, Tao Qin, Enhong Chen, and Tie- Yan Liu. Neural architecture optimization. In Advances in Neural Information Processing Systems 31, pp. 7816-7827. 2018.

---

Under review as a conference paper at ICLR 2021
Renqian Luo, Tao Qin, and Enhong Chen. Understanding and improving one-shot neural architec- 2019.
ture optimization. arXiv preprint
Niv Nayman, Asaf Noy, Tal Ridnik, Itamar Friedman, Rong Jin, and Lihi Zelnik. Xnas: Neural architecture search with expert advice. In Advances in Neural Information Processing Systems, pp. 1975-1985, 2019.
Xuefei Ning, Yin Zheng. Tianchen Zhao, Yu Wang, and Huazhong Yang. A generic graph-based neural architecture encoding scheme for predictor- nas. In The European Conference on Computer Vision (ECCV), 2020.
Hieu Pham, Melody Y Guan, Barret Zoph, Quoe V Le, and Jeff Dean. Efficient neural architecture search via parameter sharing. arXiv preprint arXiv: / 880.03368, 2018.
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoc V Le. Regularized evolution for image classifier architecture search. In Proceedings of the aaai conference on artificial intelligence, volume 33, pp. 4780-4789, 2019.
Michael S Ryoo, AJ Piergiovanni, Mingxing Tan, and Anelia Angelova. Assemblenet: Searching for multi-stream neural connectivity video architectures. arXiv preprint arXiv: /905. 132009, 2019.
Christian Sciuto, Kaicheng Yu, Martin Jaggi, Claudiu Musat, and Mathieu Salzmann. Evaluating the search phase of neural architecture search. arXiv preprint arXiv:7902.08144 2019.
Han Shi, Renjie Pi, Hang Xu, Zhenguo Li, James Kwok, and Tong Zhang. Multi-objective neural architecture search via predictive network performance optimization. arXiv preprint arXiv:1911.09336. 2019.
Yanan Sun, Handing Wang, Bing Xue, Yaochu Jin, Gary Yen, and Mengjie Zhang. Surrogate- assisted evolutionary deep learning using end-to-end random forest-based performance predic- tor. IEEE Transactions on Evolutionary Computation, 2019.
Linnan Wang, Yiyang Zhao, Yuu Jinnai, and Rodrigo Fonseca. Alphax: exploring neural architec- tures with deep neural networks and monte carlo tree search. arXiv preprint 10XXv:/805507400 2018.
Tianzhe Wang, K. Wang, H. Cai, J. Lin, Zhijian Liu, and Song Han. Apq: Joint search for network architecture, pruning and quantization policy. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2075-2084 2020.
Bichen Wu, Xiaoliang Dai. Peizhao Zhang, Yanghan Wang. Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer. Fbnet: Hardware-awarr efficient convnet design via differentiable neural architecture search. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 10742- 2019.
Sirui Xie, Hehui Zheng, Chunxiao Liu, and Liang Lin. Snas: stochastic neural architecture search. arXiv preprint 1xX1v:1812.09926. 2018
Yixing Xu, Yunhe Wang, Kai Han, Shangling Jui, Chunjing Xu, Qi Tian, and Chang Xu. Re- nas:relativistic evaluation of neural architecture search, 2019.
Zhaohui Yang, Yunhe Wang, Xinghao Chen, Boxin Shi, Chao Xu, Chunjing Xu, Qi Tian, and Chang Xu. Cars: Continuous evolution for efficient neural architecture search. arXiv preprint arXiv: (tto::99004477, 2019.
Arber Zela, Julien Siems, and Frank Hutter. Nas-bench-Ishotl. Benchmarking and dissecting one- shot neural architecture search. arXiv preprint arXiv:2001.10422, 2020.
Chris Zhang, Mengye Ren, and Raquel Urtasun. Graph hypernetworks for neural architecture search. arXiv preprint arXiv:1810.05744 2018.
Barret Zoph and Quoc V. Le. Neural architccture search with reinforcement learning. In ICLR, 2017. URL https: /ttp//aavvvvv// wwwpiii.oogg/bbaaa 1611.11111111
10

---

Under review as a conference paper at ICLR 2021
A
ADDITIONAL RESULTS ABOUT PREDICTOR-BASED EVALUATORS
A.1
PREDICTOR PERFORMANCES RESULTS: SINGLE- AND MULTIPLE-STAGE
Searcher
Evelutionary Sample d. Genemtor-aaten
Exhaustive Test
?
ressonable to use one-shot evaluator Instesd traditional evaluator the oracle
Architecture
(Arch Arch.)
Sample
Predictor
Architecture
Oracle Evaluator
Parametric/ / Nonparametric
(Archh. 73.9%) (Arch,, 43.0%)
Update
Predicted
Ground-truth
Performance
Figure Al: The overview of predictor-based neural architecture search (NAS). The underlined de- scriptions between the parenthesis denote different methods.
Table Al: The Kendall's Tau of different predictors on 3 different randomly sampled training dataset of size 78
Training Loss
Ranking
Regression
Dataset
MLP
0.1330-.00074
D. 0.0077
0.2481 10,0069
0.0011:0.00000
0.0548:00002
1 30
LSTM
0.563110.0060
10.0454
1.0087
0.60241000009
0.578410000100
0.465610.0116
GATES Ning 2020
0.759720.0079
0.7750=0.0106
0.7645=0.0054
0.2067=0.0000
0.7240=0.0077
0.713520.0055
RF (Sun et al 2019
0..29900.0077
0.4123:0.0104
0.421820.0119
Table A2: The performance distribution, BR@K, (K, Kendall's Tau of 5 training stages. In each stage, N = 78 architectures are chosen, evaluated, and used to train the predictor along with previous architecture data. Note that in this table, K in BR @K is the absolute architecture number without normalization
Stage
2
Perf, Range Perf. Std
[0.560, 0.938]
[0.921, 0.944]
[0.935, 0.944]
[0.933,
0.944]
[0.933, 0.944]
GATES
6.43e-2
4,59e-3
2.166-3
2.18
2.30e-3
BR 11/BR @ 7
1/2/306
1/1/3
1/1/2
1/1/3
1/1/3
Kendall's Tau
0.769
0.759
0.752
0.742
0.725
Stage
1
2
3
Perf. Range
[0.560, 0.938]
[0.922, 0.944]
[0.922, 0.944]
[0.932, 0.944]
[0.934, 0.944]
LSTM
Perf. Std
6.43e-2
4,52c-3
2,63e-3
2.42c-3
1.98e-3
BR@11/BR @ @
99/268/393
2/2/9
1/1/6
1/2/5
1/1/3
Kendall's & Tau
0.562
0.556
0.571
0.739
0.724
A.2 OVER- AND UINER-ESTIMATION
ARCHITECTURES
Fig. ()) illustrates the relationship between the FLOPs of architectures and how it is likely to be over-estimated. It seems that MLP and RF are more likely to overestimate the smaller archi- tectures and underestimate the larger ones, while LSTM and GATES show obvious preference the architectures' FLOPs. Fig. A2 (b) shows that GATES can give more accurate rankings on smaller architectures than larger architectures, which indicates that GATES might still have trouble in comparing larger architectures that usually have good performances.
A.3
ONE-SHOT ORACLE EVALUATOR
Luo et al. 2018) made an attempt to use a parameter-sharing evaluator as the oracle evaluator in Fig. A1
That say, they use the noisy signals provided by the parameter sharing evaluator train
11

---

Under review as a conference paper at ICLR 2021
LTM
LTM
04.
02
02
10
60
10
10
(8)
(b)
(c)
LSTW
NLP
215
31
08
2.0
-120
-115
16
(d)
(c)
(f)
Figure A2: (a)(b)(c Kendall-tas -tau in different FLOPs groups. the training set size is 39, 78 and 390, respectively. (f)(e)(f) Average rank difference different FLOPs groups, the training set size 39, 78 and 390, respectively.
the predictor. This will significantly accelerate the NAS process, compared with using an expensive traditional evaluator. However, it is found to cause the NAS algorithm to fail to discover good architectures. Also, predictors have been used accelerate parameter-sharing et al. 2020] Wang et al., (020), since one predictor forward pass is faster than testing on the whole validation queue, even no separate training phase needed, In this section, we explore whether predictor can recover from the noisy training signals provided by the parameter-sharing evaluator. Since GATES achieves consistently better results than other predictors, used in the following experiments. Specifically, want answer two questions:
NAS methods (Li
1. Can sampling only subset during supernet training help achieve better one-shot Kendall's Tau on these architectures?
2. Can predictor training help recover from the noisy training signals provided by the one-shot evaluator?
We randomly sample 78 architectures from the search space. Two differently trained parameter- sharing evaluators are used to provide the one-shot instruction signal of these 78 architectures: 1) Uniformly sampling from the whole search space, 2) Uniformly sampling from the 78 architectures. We find that strategy (sampling from the whole search space) can get a higher evaluation Kendall's Tau, no matter whether the evaluation is on the 78 architectures (0.657 V.S. 0.628) or the whole search space (0.701 V.S. 0.670). Thus the answer to Question is "No".
Then, to answer the second question, we utilize the one-shot instruction signal provided by the supernet trained with 15625 architectures train the predicto The Kendall's Tau between the architecture scores given by the resulting predictor and the ground-truth performances is 0.718 on all the 15625 architectures. which slightly worse than the one-shot instruction signals (0.719). More importantly, BR @ 1% degrades from 2.5% to 12.1%, 1,, thus the answer to Question 2 is "No'.
Thus, we conclude that although training predictor using one-shot signals can bring acceleration, since no extra inference is needed during the search, it is not beneficial in the sense of evaluation quality (especially of good architectures). Perhaps, incorporating more manual prior knowledge and regularizations can increase the denoising effect, which might be worth future rescarch.
'The average of scores provided by 3 supernets trained with different seeds is used.
12

---

Under review as a conference paper at ICLR 2021
B ADDITIONAL DISCUSSION AND RESULTS ABOUT ONE-SHOT EVALUATORS
B.1 DISCUSSION ON DE-ISOMORPHIC SAMPLING
To conduct isomorphic sampling, we first design simple encoding method, which can canonical- ize computationallyissoooppii architectures to the same string and non-isomorphic architectures to different strings. In our paper, we use the encoding method find out all isomorphic groups in the search space and make them table. Then, during the supernet training process, we sam- ple architecture groups from this table uniformly. This method feasible since the benchmark search space is not large. In practice, duct de-isomorphic sampling, by only accepting new or representative architecture samples. More specifically, one first encodes each sampled architecture into a canonical string. If this canonical string has not appeared before, this string stands for a new isomorphic group, and the architecture is recorded the representative architecture for this isomorphic group. This architecture sample is also accepted. If this canonical string has been recorded before, we only accept the architecture sample if it is the representative architecture for its canonical string.
use lazy mable-making and rejection sampling
The simple encoding method is similar to the one in NASBench-201 To be more specific, the encoding method goes follows. Denoting the expression the i-th node as S,, and the operation in the directed edge (j.i) as Ay,. the expression S; can be written as
S: = Concat(Sort(( ("""t(((""""" + +5 + ")" + "%" + AstssEP()))
where P(i) denotes the set of predecessor nodes of i, and Sort sorts the strings in dictionary order. We calculate Si 1.... 4) topological order, and the expression Sy at the final output node is used as the encoding string of the architecture.
B.2
RESULTS ON DE-ISOMOPPHIC SAMPLING
2.3
Ne De Dessomaationn PEYTS
No te me-iiceeepppee deai
De corank PatS
3.5
Dr docreepiion thal
0.52
Delomerpam Egoh Pett Deacnerpaan 1300%
Do-aemarp
Fost www.smmmaaaa.. par B1g11
Re be comerahi or Krould
Delaomaptan Kandalitaa
260
460
560
353
3006
262
453
530
830
1000
300
460
500
800
1000
(a) Kendall's s Tau
(b) P@ (%. P@ 50%
(c) BR@ @ 1., BR@5%
Figure A3: How the evaluation quality evolves along the supernet training process with different
sampling strategy. The data is also listed in Tab. 3 in the main text.
Fig. A3 a) shows that the ranking correlation keeps increasing throughout the training process. Fig. A3(b) (b) indicates that P@50% increases with more sufficient training, while P@5% slightly shrinks, especially the supernet trained without de-isomorphism sampling. As also been analyzed in the main text, this might arise from the fact that simple architectures have more isomorphic counter- parts. Fig. (3(c) shows that without de-isomorphic sampling, BR @ I % and BR@ 5% deteriorate in the late stages of the training process.
13


Reviews:
[{"review_id": "ofSDBiWPIIO", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# Summary\n\nThe paper assesses two different approaches to speed up the evaluations of neural network architectures for neural architecture search (NAS). The first one is weight sharing, which trains a supernetwork that contains all possible architecture of the search space. The performance of single architectures can be then approximated by simply using the shared parameters of the supernetwork. The second approach is to use different kind of predictors that are trained on offline evaluated architectures. Several methods following these two approaches from the literature are evaluated on the NASBench201 benchmark based on different rank-based evaluation scores.\n\n\nIn general I think the paper addresses an interesting problem in neural architecture search. However, compared to existing work, the new insights that the paper presents are rather limited. Particularly, since the results are only based on a single benchmark. I therefore don't think that the paper reaches the bar of acceptance.\n\n\n# pros\n\n- The main contribution of the paper to provide a detailed analysis and comparison of different evaluation strategies for neural architecture search.\n\n- The paper is well written and easy to follow. Also the usage of the different metrics is well motivated and appears reasonable.\n\n\n# cons\n\n\n - The paper only evaluates on the NASBench201 benchmark, which consists of the same set of architectures evaluated on three different datasets. However, it is unclear how reliable conclusions are and if they also transfer to other search spaces. At least other tabular benchmarks, e.g NASBench101 and NASBench101-shot could be added to the evaluation. Even better would be to also include a larger non-tabular search space (e.g DARTS), which, due to the computational requirements, could be then less exhaustive.\n\n\n- The novelty of the paper seems to be rather small.  Ning et. al. also compared different predictor (MLP, LSTM, GATES) on  Nasbench201 benchmarks. They even report the same metrics (e.g Precision@K). It seems that the only new contribution of this paper is to extend this comparison by including a random forest predictor, which is arguably a rather simple baseline.\n\n-Also, the correlation between shared weights of the one-shot model and the architectures from the search space has been investigated several times before (for example Dong et. al., Yang et. al.). Unfortunately, the paper  does not state  which new insights it provides  that extends the contributions of previous papers. \n\n\n# Questions for rebuttal\n\n- How does the de-isomorphic sampling works in general?\n\n- Could be that the ranking of top performing architectures is low because the difference between architecture are almost the same?\n\n- How exactly is the parameter sharing model in Section 4.2 trained on the de-isomorphic NAS-Bench201 search space? How does it differ from training on the full search space?\n\n\n\n# minor comments / typos\n\n- Figure 3: I think caption and plot a) and b) are mixed up.\n\n- Section 4.2 pool -> poor\n\n\n# Post Rebuttal\n\nI thank the authors for replying to my question. Unfortunately, the authors did not present any new results on other benchmarks (e.g Nasbench101-shot).  I will therefore keep my score.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "fW_AOTpQQjz", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors assess both the parameter-sharing evaluators and predictor-based evaluators on the NAS-Bench-201 search space, with a set of NAS-oriented criteria to understand the behavior of fast architecture evaluators in different training stages. Experimental results on the CIFAR-10 dataset using the NAS-Bench-201 search space are reported. \n\nStrengths:\n\nThe authors conduct an extensive assessment of fast architecture evaluators on the NAS-Bench-201 search space, and provide insights on how different configurations and strategies could influence the fitness of the evaluators. \n\nWeaknesses:\n\n(1) This paper is focused on the assessment of existing fast architecture evaluators, rather than the proposal of new methods or architecture evaluators. It aims to share experiences on neural architecture evaluations. The overall novelty of this work may not be very strong. \n\n(2) The experiments are conducted on a single dataset CIFAR-10. It is not clear if the provided suggestions can be generalized to a broad set of machine learning tasks.\n\nAdditional comments:\n\nThe parameter-sharing evaluators and predictor-based evaluators are evaluated separately under different experiment setups. I think it would be interesting to compare the performance of all the evaluators on the same task. This may provide further insights as well as concrete examples on the pros and cons of these two types of evaluators.", "extra_scores": {}, "extra_text": {}}, {"review_id": "_Cx-EtAH_N", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Post-Rebuttal Update\nSince many of the concerns from my original review would require a major revision to address, I've tentatively left my original review score (3) unchanged. The biggest reason is that I'm still concerned that insufficient hyper-parameter tuning could lead to wrong conclusions; this would need more work to address. I've included some additional notes below.\n\nIn their response to my review and those of the other reviewers, the authors promise to address a number of issues with the current draft. While I hope these updates will ultimately improve the paper, the authors' current responses don't provide me with enough information merit increasing my ICLR review score. For example:\n* \"We will try to auto-tune the hyper-parameters of different predictors in the future.\"\n* \"We think we can improve Figure 3c\" by measuring mean/variance across multiple runs.\n\nOn a positive note: I'd like to thank the authors for clearing up my confusion about ranking diff under \"additional notes\". I also think the authors' explanation of their isomorphic sampling procedure in the response to AnonReviewer2 addresses one of my questions.\n\n## Recommendation\nI like the submission's high-level goal of trying to do careful, controlled studies of different factors which can affect a NAS algorithm's ability to ability to rank different candidate architectures within a search space. And the paper seems reasonably well-organized and well-written. However, due to some concerns about the paper's methodology and conclusions (discussed below), I do not feel comfortable recommending the paper for acceptance in its current form.\n\n## Background\nThe goal of Neural Architecture Search (NAS) is to automatically identify network architectures from a human-defined search space which have good accuracy/speed tradeoffs. However, ranking the accuracies of different candidate architectures within a search space can be extremely compute-intensive if done naively, by training each candidate network from scratch and then evaluating it.\n\nThe submission conducts empirical studies on the NASBench-201 benchmark dataset for two previously proposed classes of techniques which can be used to efficiently rank different network architectures within a search space without training all of them from scratch:\n1. train a one-shot model -- a single shared set of weights which can be used to evaluate many different candidate architectures.\n2. train a predictor by (a) sampling a limited number of candidate architectures from the search space, (b) training/evaluating each one, and then (c) using the resulting <architecture, accuracy> pairs to train a regression model which can predict the accuracy of *any* architecture within the search space.\n\n## Paper Contributions\nThe submission conducts experiments to estimate how changes in training/evaluation setups will affect the ability of these two methods to rank different candidate architectures within a search space. \n\nFor one-shot models:\n* Evaluating how the one-shot model's ability to rank the most accurate architectures in the search space vs. ranking random architectures evolves over the course of a search.\n* Evaluating two tricks for reducing the cost of training a one-shot model: (a) reducing the number of layers in the one-shot model, or (b) reducing the number of filters per layer.\n* Changing the architecture sampling strategy that's used when training the one-shot model.\n* \"Post de-isomorphism:\" The NASBench-201 search space has multiple ways to encode the same architecture which produce different accuracies according to a one-shot model; here, the authors propose taking the average accuracy over all possible encodings of a given architecture.\n* Some experiments to quantify the predictor's ability to rank large vs. small models.\n\nFor predictors/regressors:\n* Comparing the relative ranking ability of predictor models trained using random forests regressors, MLPs, LSTMs, and GATES.\n\n## Concerns about methodology\nMany of the paper's experiments -- especially in earlier sections -- seemed reasonable. In particular: \"the influence of channel vs layer proxy\" (Figure 3) and the effects of post-de-isomorphism (Table 3) and sampling (Table 4) on one-shot model / supernet rankings. However, I have concerns that about a number of experiments, especially in later sections of the paper:\n\n**Insufficiently tuned hyper-parameters?**\nIn Section 5.1, when describing their experiments comparing different predictors/regressors, the authors write that \"For optimizing MLP, LSTM, and GATES, an ADAM optimizer with a learning rate of 1e-3 is used, the batch size is 512, and the training lasts for 200 epochs.\" This makes me concerned that (i) the authors used a default set of hyper-parameters to optimize their models, and (ii) few attempts were made to tune the hyper-parameters. Because the quality of models trained using gradient descent can be quite sensitive to the training hyper-parameters, this makes me wonder whether the authors would've drawn different conclusions about the performance of these predictors if they'd spent more time on hyper-parameter tuning. For example: experimenting with learning rate schedules, tuning the learning rate and weight decay used during training, tuning the MLP's hidden layer sizes, etc.\n\nThe authors also claim that \"training predictors with regression loss is not stable and sensitive to the choice of training set.\" If the authors' hyper-parameters were not sufficiently well-tuned, that could explain the instabilities the authors saw. Other standard tricks (e.g., normalizing the labels to have mean 0 and variance 1 before performing a regression) could also drastically affect this conclusion.\n\n**Seemingly trivial results presented as non-trivial?**\nIn Section 4.5, the submission argues that \"\"the average rank diff shows a decreasing trend, which means that the larger the model, the easier it is to be underestimated.\" And the intro states that \"parameter sharing evaluator tends to over-estimate smaller architectures.\" These findings are presented in a way which suggests that there's bias in the one-shot model itself, but the observation itself seems trivial, and would likely hold even if the one-shot model was unbiased. To take an extreme example: the one-shot model can underestimate but cannot overestimate the rank of the most accurate architecture in the search space. So the expected rank of this top architecture would almost certainly be less than the true rank unless the one-shot model provided a perfect ranking of architectures within the search space.\n\n**Unsupported claims related to multi-stage training?**\nIn Section 5.2: the authors claim that \"As shown in Fig. 5, P@0.1% increases a lot after multiple stages of training, which indicates that the predictors perform better and better on distinguishing good architectures. This is expected since the training data are more and more concentrated on good architectures.\"\n\nIf I understand this figure correctly, the authors are using progressively more training data at each stage. In this case, the improvements could simply be explained by the fact that they're using a larger training dataset, and have nothing to do with the fact that the training data is concentrated on good architectures. Additional baselines would be needed to test the authors' original hypothesis.\n\n## Additional notes\n\n**Missing description of de-isomorphic sampling?**\nI had trouble finding details about how de-isomorphic sampling was performed during one-shot model training. (Please let me know if there's a description I overlooked.)\n\n**End of Section 4.2:** \"with medium or pool performance:\" I think this is a typo. Did you mean \"poor\" rather than \"pool\"\n\n**Section 4.5:** \"We inspect the Ranking Diff of the ground truth performance and the one-shot evaluation of an architecture [...] r_i - n_i. [...] Note that a positive Ranking Diff indicates that this architecture is over-estimated, otherwise it is underestimated.\" I interpreted \"diff of the ground truth performance and the one-shot performance\" to mean \"[ground truth performance] minus [one-shot performance]\". But under this interpretation, I think a *negative* ranking diff would mean that the architecture is over-estimated by the one-shot model. Did you mean to write \"ground truth diff of the one-shot evaluation and the ground truth evaluation\" rather than the other way around?", "extra_scores": {}, "extra_text": {}}, {"review_id": "EXKWh8xs5HV", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Contributions ###\n* The paper conducts an extensive study of efficient neural architecture \"evaluators\", in particular of one-shot evaluators with shared weights and of predictor-based evaluators.\n* Several findings are extracted from the empirical evaluation on the NasBench-201 benchmark. To summarize some key findings: (a) proxy models need not be wide but should be deep (channel proxy works better than layer proxy), (b) one-shot evaluators with parameter-sharing tend to overestimate smaller architectures and are also better at comparing these, (c) predictors benefit from being trained with a ranking loss rather than regression loss.\n\n### Significance ###\nScaling neural architecture search (NAS) to large-scale tasks requires efficient ways of estimating architecture performance. Several different, more-or-less efficient ways of estimating architecture performance have been proposed; it thus becomes increasingly important to understand errors and biases of these different estimators. This work provides very valuable insights regarding these points for the most promising classes of architecture performance estimators. It thus can guide and motivate future directions for efficient performance estimation for NAS, and is thus a significant contribution.\nSignificance would be strengthened by releasing code for the conducted experiments. This would allow future methods be  easily compared to the results obtained in this paper.\n\n### Originality ###\nThere are some prior works also studying quality of efficient architecture performance estimators (summarized by the authors in Section 2.1). However, the proposed work goes considerably beyond the scope of these works. It heavily builds upon NasBench-201. However, I consider the paper still sufficiently original for a publication at ICLR.\n\n### Clarity ###\nGenerally, the summary of the conducted studies and the presentation of the results is clear and easy to follow, with two exceptions: (1) Figure 2a shows too many different things in one plot and is hard to grasp; I think splitting this plot into several independent ones would be helpful. (2) Table 3 would best be represented graphically rather than as a table.\nIn terms of wording, it would make sense in my opinion to use some established terms and concepts from statistics. For instance, the models studied in this paper dot not really \"evaluate\" performance but estimate it; thus calling them \"estimator\" would be more appropriate than \"evaluator\". Estimators might have a bias (\"performance of small models is typically overestimated\"), and their variance might vary for different parts of the search space (\"ranking smaller models is easier than ranking larger ones).\n\nI would ask the authors to clarify which architecture is used for GT column in Table 2. A randomly sampled one?\nMoreover, does dropout in Table 1 actually refer to dropout or rather to the more common Droppath?\n\n### Quality ###\nThere are some evaluations for which I am not totally convinced that the drawn conclusions are trustworthy: \n* Stating that the ranking loss is more stable than regression loss at the end of Section 5.2 seems too far stretched given that there is only 1 outlier in 3 runs where regression loss failed. For drawing this conclusion, far more repetitions and a statistical significance test would be required.\n* In general, having error bars on the reported results (from several repetitions of the same setting) and checking for statistical significance would increase quality of the evaluation considerably. For example, effects in Figure 3c are rather small and noisy and the trend might in the worst case be due to chance.\n* Table 2 indicates a decrease of variance for estimated accuracy of isomorphic architectures for longer training. However, couldn't this be an indication that all architectures converge to high accuracy and thus total variance shrinks (and not only those of isomorphic ones?) I would ask the authors to also report global accuracy standard deviation (or normalize by it)\n* Generally, the quality of a ranking depends not only on the strength of an estimator but also on the scale of differences in performances. In the extreme case, two architectures have the same performance; how should these be ranked? A discussion of this and how it is addressed would be helpful.\n* The fact that post de-isomorphism works surprisingly well is stated but not explained. An attempt at explaining this observation would be highly appreciated. For instance: post de-isomorphism averages performance estimates of isomorphic architectures. By this, it should also decrease variance of the estimates. Could this explain the improved ranking quality?\n* How specific are the obtained results for the hyperparameters used (Table 1) and the benchmark? For drawing general conclusions, it would be required testing also other hyperparameters and on other datasets/benchmarks.\n\n### Recommendation ###\nTo summarize, I think this paper has the clear potential for getting accepted at ICLR. However, in the current form, there are several issues that would need to be fixed. For the time being, I lean towards rejection but I would be willing to increase my score if these points would be addressed.\n\n### Final Recommendation after Author Response ###\nI have read the author response and appreciate their feedback. As the authors could not address some of the issues (error bars/statistical significance testing, other hyperparameters and other datasets/benchmarks) in the restricted time of the rebuttal period, I will keep my rating and recommend rejecting this submission for ICLR. However, I also encourage the authors to resubmit a revised version of the paper taking all feedback into account since I see clear potential.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents an empirical study of different efficient ways to estimate the performance of architectures in NAS, focussing on weight sharing and performance prediction methods.
Most reviewers appreciated the paper's goal of performing a careful, controlled study of different factors that can affect the ranking of architectures. However, all reviewers also had substantial concerns. Many of these could in principle be fixed by additional experiments, but the short time window of the author response period did not allow for this. 
As a result, all reviewers voted for rejection, and I will follow that recommendation.
Nevertheless, I would like to encourage the authors to continue this work, as I believe that the NAS community needs more careful controlled studies of this type. For their next version, I encourage the authors to take into account the many points mentioned by the reviews.
Decision: Reject