Stronger NAS with Weaker Predictors
Authors: Junru Wu
Xiyang Dai
Dongdong Chen
Yinpeng Chen
Mengchen Liu
Ye Yu
Zhangyang Wang
Zicheng Liu
Mei Chen
Lu Yuan
Venue: NeurIPS 2021 Poster (2021)
Keywords: Neural Architecture Search
Predictor
Bayesian Optimization

Abstract: Neural Architecture Search (NAS) often trains and evaluates a large number of architectures. Recent predictor-based NAS approaches attempt to alleviate such heavy computation costs with two key steps: sampling some architecture-performance pairs and fitting a proxy accuracy predictor. Given limited samples, these predictors, however, are far from accurate to locate top architectures due to the difficulty of fitting the huge search space. This paper reflects on a simple yet crucial question: if our final goal is to find the best architecture, do we really need to model the whole space well?. We propose a paradigm shift from fitting the whole architecture space using one strong predictor, to progressively fitting a search path towards the high-performance sub-space through a set of weaker predictors. As a key property of the weak predictors, their probabilities of sampling better architectures keep increasing. Hence we only sample a few well-performed architectures guided by the previously learned predictor and estimate a new better weak predictor. This embarrassingly easy framework, dubbed WeakNAS, produces coarse-to-fine iteration to gradually refine the ranking of sampling space. Extensive experiments demonstrate that WeakNAS costs fewer samples to find top-performance architectures on NAS-Bench-101 and NAS-Bench-201. Compared to state-of-the-art (SOTA) predictor-based NAS methods, WeakNAS outperforms all with notable margins, e.g., requiring at least 7.5x less samples to find global optimal on NAS-Bench-101. WeakNAS can also absorb their ideas to boost performance more. Further, WeakNAS strikes the new SOTA result of 81.3% in the ImageNet MobileNet Search Space. The code is available at: https://github.com/VITA-Group/WeakNAS.

Stronger NAS with Weaker Predictors
Junru Wu', Xiyang Dai², Dongdong Chen², Yinpeng Chen', Mengchen Liu', Ye Yu?, Zhangyang Wang'. Zicheng Liu², Mei Chen?, Lu Yuan²
Texas A&M University, Microsoft Corporation, UUniversity of Texas at Austin sandbozmaster@ttmm.c {xidai, dochen, dochen,yilche yilche, wwwgmmlinmmmiieeeee.e...ee crosoft. com, atlaswangdutexas.cmmm edu, {yeyu1, zliu, mei. chen, luyuan emicrosot.. com
Abstract
Neural Architecture Search (NAS) often trains and evaluates a large number of ar- chitectures. Recent predictor-based NAS approaches attempt to alleviate such heavy computation costs with two key steps: sampling some cetittecture-peffommmat pairs and fitting a proxy accuracy predictor. Given limited samples, these predictors, however, are far from accurate to locate top architectures due to difficulty fitting the huge search space. This paper reflects on a simple yet crucial question: if our sinal goal is to find the best architecture, do we really need model the whole space well?. We propose paradigm shift from fitting the whole architecture space using one strong predictor, to progressively fitting a search path towards the nigh-performance sub-space through set of weaker predictors. As key property the weak predictors, their probabilities of sampling better architectures keep increasing. Hence only sample few well-performed architectures guided by the previously learned predictor and estimate a new better weak predictor. This em- barrassingly easy framework, dubbed WcakNAS, produces coarse-to-fine iteration to gradually refine the ranking of sampling space. Extensive experiments demon- strate that WeakNAS costs fewer samples to find top-performance architectures on NAS-Bench- 101 and NAS-Bench-201. Compared to state-of-the-an (SOTA) predictor- based NAS methods, WeakNAS outperforms all with notable margins, e.g., requiring at least 7.5x less samples to find global optimal on NAS-Bench-101 WeakNAS can also absorb their ideas boost performance more. Further, Weak- NAS strikes the new SOTA result of 81.3% in the ImageNet MobileNet Search Space. The code is available at: https:: /tt///gtttt/.///////// com Coan/VITA-Gra GraTTAA-rroo/Weee /eakup/
Introduction
47.0
oost-iillttoon accurscy
Neural Architecture Search (NAS) [H22] methods aim
to find the best network architecture by exploring the
46.0
cechitecuuretto-pefffrmmee
manifold, using reinforced-
beanenng-based [13,, evolution-baat based [4 [5] or gradient-
based [6] approaches. In order to cover the entire search space, they often train and evaluate large number of archi- tectures, leading to tremendous computation cost.
Weel Freckthor
grade
16.5
Figure 1: Comparison hetween method using set of weak predictors (it- erative sampling). and single strong pre- dictor (random sampling) on MAS-Bench- 201. For fair comparison, the NAS predic- tor in both methods adtops the same type MLP described in 22.4] Solid lines and shadows denote the mean and standard deviation (std), respectively.
Recently, predictor-based NAS methods alleviate this prob- lem with two key steps: one sampling step to sample some caniieecture-eeffommnncc pairs, and another performance modeling step to fit the performance distribution by training proxy accuracy predictor. An in-depth analysis of existing methods (2 found that most of those methods [5, 6. I7 7 [1] consider these two steps independently and attempt model the performance distribution over the whole architec-
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Previous:
Proposed:
Top Semples
Sample Spare 5
Sample Space S
Search Space X
Search Space X
Sort
Sampling
Learning
Sampling
Learning
Pradicted Performance on X/5*
Weak Prodictors fx
Strong Predictor
Iteration
Iteration 2
Iteration
Ground-Trcft Soundd-ruath Accuracy
Gronn I Accuracy
Architocture
Architecture
Optimal &ampled
Sample Space
Excluded
Figure 2: An illustration of WeakNAS's progressive approximation. Previous rrdictor-based NAS uniformly sample in the whole search space to fit a strong predictor. Instead, our method progressively shrinks the sample space based on predictions from previous weak predictors, and update new weak predictors towards subspace of better architectures, hence focusing on fitting the search path.
ture space using strong predictor. However, since the architecture space is often exponentially large and highly non-convex, even very strong predictor model has difficulty fitting the whole space given limited samples. Meanwhile, different types of predictors often demand handcraft design of the architecture representations to improve their performance.
This paper reflects on a fundamental question for predictor-based NAS: best architecture, do really need to model the whole space well?". We investigate the alternative of utilizing few weak- predictors to fit small local spaces, and to progressively move the search space towards the subspace where good architecture resides. Intuitively, we assume the whole space could be divided into different sub-spaces, some of which are relatively good while some are relatively bad. We tend choose the good ones while discarding the bad ones. which makes sure more samples will be focused on modeling only the good subspaces and then find the best architecture. It greatly simplifies the learning task of each predictor. Eventually, a line of progressively evolving weak predictors can connect a path to the best architecture.
"if our final goal is to find the
We present novel, general framework that requires only to estimate series of weak predictors progressively along the search path, we denoted it as WeakNAS in the rest of the paper. To ensure moving towards the best architecture along the path, at each iteration, the sampling probability of better architectures keep increasing through the guidance of the previous weak predictor. Then, the consecutive weak predictors with better samples will be trained the next iteration. We iterate until we arrive at an embedding subspace where the best architectures reside and can be accurately assessed by the final weak predictor.
Compared to the existing predictor-based NAS, our proposal represents a new line of attack and has several merits. First, since only weak predictors required, it yields better sample efficiency. As shown in Figure it costs significantly fewer samples to find the top-performance architecture than using one strong predictor, and yields much lower variance in performance over multiple runs. Second, it is flexible to the choices of architecture representation (e.g., different architecture embeddings) and predictor formulation (e.g.. multilayer perceptron (MLP), gradient boosting regression tree, or random forest). Experiments show our framework performs well in all their combinations. Third, it highly generalizable to other open search spaces, e.g. given a limited sample budget, we achieve the state-of-the -hht---rtt ImageNet performance on the NASNet and MobileNet search spaces. Detailed comparison with tfate-of-the-an predictor-ban based NAS 19 21
is presented in Section 4.
'"Strong" vs "Weak" predictors: name a "weak" predictor if it only predicts a local subspace of the search space thus can be associated with our iterative sampling scheme; such predictors therefore usually do demand very heavily parameterized models. the contrary, "strong" predictors predict the global search space and are often associated with uniform sampling. The terminology of strong versus weak predictors does not represent their number of parameters or the type of NAS predictor used. An overparameterized NAS predictor with our iterative sampling scheme may still be considered a "weak" predictor.

---

2 Our Framework
2.1
Reformulating Predictor-based NAS as Bi-Level Optimization
Given a search space of network architectures X and an architecture-tor tonprctrr--t---rffmmaaccc mapping function f X + P from the architecture set X to the performance set P, the objective is to find the best neural architecture x² with the highest performance f (x)) in the search space X:
x² arg max (xx
xēX
A naive solution is to estimate the performance mapping fax through the full search space. However, this is prohibitively expensive since all architectures have to be exhaustively trained from scratch. To address this problem, predictor-based NAS learns proxy predictor f(xx to approximate f(x) by using some
pairs, which significantly reduces the training cost. In general, predictor-based NAS can be re-cast as a bi-level optimization problem:
arg max f (x][S), s.L. f arg min S.jEF
Σ [(((s),
=EX
where L is the loss function for the predictor f. F is a set of all possible approximation to f. S === {S C X |S| S C} all architectures satisfying the sampling budget C. C' is directly related to the total training cost, e.g.. the total number of queries. Our objective is to minimize the loss L based on some sampled architectures S.
Previous predictor-basee NAS methods attempt to solve Equation 2 with two sequential steps: (I) sampling some architecture- -erfformrreeeeooomance pairs and (2) learning a proxy accuracy predictor. For the first step, a common practice is to sample training pairs S uniformly from the search space X to fit the predictor. Such sampling however inefficient considering that the goal of NAS only to find well architectures without caring for the bad ones.
2.2 Progressive Weak Predictors Emerge Naturally as A Solution to the Optimization
Optimization Insight: Instead of first (uniformly) sampling the whole space and then fitting the
predictor, we propose to jointly evolve the sampling S and fit the predictor f. which helps achieve better sample efficiency by focusing on only relevant sample subspaces. That could be mathematically formulated as solving Equation in a new coordinate descent way, that iterates between optimizing the architecture sampling and predictor fitting subproblems:
(Sampling)
ph (fa(s)]s e X\ \ SM C Topw(PP), s
Sas U.
(3)
where Top (Pt) denote the set of top N architectures in Pbk
(Predictor Fitting)
x arg max f(x]s(t1), s.t. fu+1 = arg min =EX
Σ c(f(s),
(4)
f2eF sESt+1
In comparison, existing predictor-based NAS methods could be viewed as running the above coordi- nate descent for just one iteration special case of our general framework.
As well known in optimization, many iterative algorithms only need to solve (subsets of) their subproblems inexactly 24 for properly ensuring convergence either theoretically or empirically, Here, using a strong predictor to the whole space could be treated as solving the predictor fitting subproblem relatively precisely, while adopting weak predictor just imprecisely solves that. Previous methods solving Equation truncate their solutions to "one shot" and hinge on solving subproblems with higher precision. Since
take a joint optimization view and allow for multiple iterations, we can afford to only use weaker predictors for the fitting subproblem per iteration.
Implementation Outline: The above coordinate descent solution has clear interpretations and is straightforward to implement. Suppose our iterative methods has K iterations. We initialize S by randomly sampling a few samples from X, and train an initial predictor f. Then at iterations = 2.... K, we jointly optimize the sampling set S and predictor fa in an alternative manner.
Subproblem 1: Architecture Sampling. At iteration i ++1 1, we first sort all architectures in the search space X (excluding all the samples already in 5*) according to its predicted performance pok at
"One only exception is the Section 3.2 domain experiments: we will sub-sample all architectures in the search space before sorting. More details can be found in Appendix Section H

---

Search Dynemics
0.00
50
0.02
0.4
0.44
Heral
Eamtion
Beraton Reratie 2
0--
0022
Iten on
taniion
Rematiee
nanore
Optimal Weddmted
Itrot in
Zerioon
Recotiie
Earstion
Ranation
602
60
C0
70
72
73
27
29
Accruacy
Hole
(a)
(h)
(c)
Figure 3: Visualization of the search dynamics in NAS-Bench-201 Search Space. (best viewed in
color) (a) The trajectory of the predicted best architecture and global optimal through out 5 iterations;
(b) Error empirical distribution function (EDF) of the predicted top-200 architectures throughout iterations (c) Triangle marker: probability of sampling top-50 architectures throughout 5 iterations; Star marker: Kendall's Tau ranking of NAS predictor Top 50 architectures through out iterations.
Sampling Space (Top,, (P*)
4000
mi--h Ramking
5000
8000
Optanal
Sprimal
Ootimal
14000
35625
(8) 1 iteration
(b) 2dd Iteration
(c) 3 iteration
(d) Last iteration
Sampled Architectures (5*))
2000
4000
m00h Rankknn
8000
10000
12000
Oostinal Prediited kest
Optiral Preckced
Optinal
Optiral
Reddcted
Prediced Bext
Iteration
(f) 2n Ittration
Iteration
Last Iteration
Figure 4: Visualization of search dynamics in NAS MAS--Bnnh-225 201 Search Space via t-SNE. At i-th iteration, we randomly sample M = 40 new architectures from the top N 400 ranked architectures in Pk The top row from (a)-(d) show the sampling space I T'op'N (P²),, and the bottom row from (c)-(h) show the sampled architectures 5* The performance ranking of architectures is encoded by color, and those not-sampled architectures colored in grey.
every iteration We then randomly sample M new architectures from the top N ranked architectures in ph Note this step both reduces the sample budget, and controls the exploitation-exploriie trade-off (see Section 3.1p. The newly sampled architectures together with S become S
Subproblem 2: (Weak) Predictor Fitting. We learn a predictor fh+1, by minimizing the loss C of the predictor f++11 based on sampled architectures gk++1, We then evaluate architectures using the learned predictor fa++1 to get the predicted performance p4++1
As illustrated in Figure 2 through alternating iterations, we progressively evolve weak predictors to focus on sampling along the search path, thus simplifying the learning workload each predictor. With these coarse-to-Ne fone iterations, the predictor f* would guide the sampling process to gradually zoom into the promising architecture samples. In addition, the promising samples S*+11 would in turn improve the performance of the updated predictor fk+1 among the well-performed architectures, hence the ranking of sampling space also refined gradually. other words, the solution quality to the subproblem 2 will gradually increase as natural consequence of the guided zoom-in. For

---

derivation, we simply choose the best architecture predicted by the final weak predictor. This idea is related to the classical ensembling (25]. yet a new regime NAS.
Experiment. Figure (a) shows the progressive procedure of finding the optimal architecture x² and learning the predicted best architecture म over 5 iterations. As we can see from
Figure 3 (a), the optimal architecture and the predicted best one are moving towards each other
closer and closer, which indicates that the performance of predictor over the optimal architecture(s) is growing better. In Figure we use the error empirical distribution function (EDF) [26] to visualize the performance distribution of architectures in the subspace. We plot the EDF of the top-200 models based on the predicted performance over iterations. As is shown, the subspace of top-performed architectures is consistently evolving towards more promising architecture samples over iterations. Then in (3 we validate that the probabilities of sampling better architectures within the top N predictions keep increasing. Based this property, we can just sample few well-performing architectures guided by the predictive model to estimate another better weak predictor. The same plot also suggests that the NAS predictor's 's ranking among the top-performed models gradually refined, since more and more architectures in the top region are sampled.
In Figure 4 we also show the t-SNE visualization of the search dynamic in space. We can observe that: (1) NAS- Bench-201 search space is highly structured: (2) the sam- pling space T Topx (Pt) and sampled architectures 5 are both consistently evolving towards more promising regions, can be noticed by the increasingly warmer color trend.
search
2.3 Relationship to Bayesian Optimization: A Simplification and Why It Works
Our method can be alternatively regarded vastly simplified variant of Bayesian Optimization (BO). It does not refer to any explicit uncertainty- based modeling such as Gaussian Process (which often difficult to scale up); instead it adopts a very simple step function as our acquisition function. For a sample x in the search space X, our special "acquisition function" can be written as:
acq(x) = u(x - 0)
(5)
where the step function u(x) is i 2 0, and 0 otherwise; e a random variable from the uniform distribution (0, 1); and 0 is the threshold to split Top. from the rest, according to their predicted performance pk (z)) We then choose the samples with the M largest acquisition values:
SM arg max (z)
(6)
Top
Why such "oversimplified BO" can be effectively for our framework? We consider the reason to be the inherently structured NAS search space. Specifically, existing NAS spaces are created either by varying operators from a pre-defined operator set (DARTS/NAS-Beil) 1002110 Search Space) or by varying kernel size, width or depth (MobileNet Search Space). Therefore, as shown in Figure 4 the search spaces are often highly-structured, and the best performers gather close to each other.
Here comes our underlying prior assumption: we can progressively connect a piecewise search path from the initialization, to the finest subspace where the best architecture resides. At the beginning, since the weak predictor only roughly fits the whole space, the sampling operation will be "noisier", inducing more exploration. When it comes to the later stage, the weak predictors fit better the current well-performing clusters, thus performing more exploitation locally. Therefore our progressive weak predictor framework provides a natural evolution between exploration and exploitation, without explicit uncertainty modeling, thanks to the prior of special NAS space structure.
Another exploration-expllottiii trade-off is implicitly built in the adaptive sampling step of our subproblem 1 solution. To recall, at each itcration, instead of choosing all Top N models by the latest predictor, randomly sample M models from Top N models to explore new architectures in a stochastic manner. By varying the ratio = M/N and the sampling strategy (e.g., uniform, linear-decay or exponential-decay), we can balance the sampling exploitation and exploration per step, in a similar flavor to the e-greedy (27 approach in reinforcement learning.
2.4 Our Framework is General to Predictor Models and Architecture Representations
Our framework is designed to be generalizable to various predictors and features. In predictor-
NAS, the objective of fitting the predictor í is often cast as a regression 7 or ranking 5 problem.
The choice of predictors is diverse, and usually critical to final performance 5 6 7-9. To
illustrate our framework is generalizable and robust to the specific choice of predictors, we compare the following predictor variants.
5

---

01.0
47.
86.0
00.0
00.6
GEIT
Random Search
Random Search
GENT
Randomfores
43.0
GORT
Ransem Search
Fondamt
MPP
RLP
MLF
onci
diacle
4419
orace
230
300
330
200
200
200
502
120
200
200
402
Number of Sarppine
Kumber of Samplee
Namber of Sarrples
CIFARIO
(b) CIFARIO0
(c) ImageNet166
Figure 5: Evaluations of robustness across different predictors shadow regions denote the mean and std, respectively.
Solid lines and
Multilayer perceptron (MLP): MLP is the common baseline in predictor-based NAS [5] due to its simplicity. For our weak predictor, we use a 4-layer MLP with hidden layer dimension of (1000, 1000, 1000, 1000).
Regression Tree: tree-based methods are also popular 9
28 since they suitable for
categorical architecture representations. As our weak predictor, we use the Gradient Boosting Regression Tree (GBRT) based XGBoost 29 consisting of 1000 Trees.
Random Forest: random forests differ from GBRT in that they combines decisions only the end rather than along the hierarchy, and are often more robust to noise. For each weak predictor, we use random forest consisting of 1000 Forests.
The features representations to encode the architectures are also instrumental. Previous methods hand-craft various features for the best performance, e-g., raw architecture encoding (6], supernet statistics [30]. and graph convolutional network encoding 7 5 g 19 Our framework is also agnostic
various architecture representations, and compare the following:
One-hot vector: NAS-Bench-201 31. its DARTS-style search space has fixed graph connectivity, hence the one-hot vector commonly used to encode the choice of operator.
Adjacency matrix: In NAS-Bench-101,
used the same encoding scheme in [32 6).
where a x adjacency matrix represents the graph connectivity and a 7-dimensional vector represents the choice of operator on every node.
As shown Figure all predictor models perform similarly across different datasets. Comparing per- formance JAS-Bench-IDe 101 and NAS-Bench-201 although they different architecture encoding methods, our method still performs similarly well among different predictors. This demonstrates that our framework robust various predictor and feature choices.
3 Experiments
Setup: For all experiments, use Intel Xeon E5-2650v4 CPU and single Tesla P100 GPU, and use the Multilayer perceptron (MLP) as our default NAS predictor, unless otherwise specified.
NAS-Bench-101 332 provides Directed Acyelic Graph (DAG) based cell structure. The connectiv- ity of DAG can be arbitrary with a maximum number of nodes and 9 edges. Each nodes on the DAG can choose from operator of 1 x convolution, 3x3 convolution or 3x3 max-pooling. After removing duplicates, the dataset consists of 423,624 diverse architectures trained on CIFAR 10
NAS-Bench-201 31 is more recent benchmark with reduced DARTS-like search space. The DAG of each cell is fixed, and one can choose from different operations (1x1 convolution, 3x3 convolution, 3x3 avg-pooling, skip, connection), on each of the 6 edges, totaling 15,625 architectures. It is trained on different datasets: CIFARIO, CIFARIO8 and ImageNet16-122 [34].
For experiments on both NAS- NAS-Benches, followed the same setting (8).
Open Domain Search: We follow the same NASNet search space used
and MobileNet
Search Space used in (36] to directly search for the best architectures on ImageNet 37 Due to
the huge computational cost to evaluate sampled architectures on ImageNet, we leverage weight- sharing supernet approach. On NASNet search space, we use Single- One-shot [38 approach to train our SuperNet, while on MobileNet Search Space we reused the pre-trained supernet from [F.] We then use the supernet accuracy as the performance proxy to train weak predictors. We clarify that despite using supernet, our method more accurate than existing differentiable weight-sharing methods, meanwhile requiring less samples than evolution based weight-sharing

---

methods, as manifested in Table and 7 We adopt Py and image models library (timm) [39 to implement our models and conduct all ImageNet experiments using 8 Tesla V100 GPUs. For derived architecture, we follow similar training from scratch strategies used in LaNAS 21
3.1 Ablation Studies
We conduct series of ablation studies the effectiveness of proposed method NAS-Bench- 101. To validate the effectiveness of our iterative scheme, Table we initialize the initial Weak Predictor f1 with 100 random samples, and set M 10, after progressively adding more weak predictors (from 1 to 191), we find the performance keeps growing. This demonstrates the key property of our method that probability of sampling better architectures keeps increasing more iteration goes. It's worth noting that the quality of random initial samples Mo may also impact on the performance of WeakNAS, but if MMl sufficiently large, the chance of hitting good samples (or its neighborhood) is high, and empirically we found J.Mo==10f to already ensure highly stable performance NAS-Bencha 101: a more detailed ablation can be found the Appendix.
Sampling
#Predictor
#Queries
Test Acc.(%)
SD(%)
Test Regret %))
Avg. Rank
Uniform
Strong Predictor
2000
93.92
0.08
0.40
135.0
Weak Predictor
100
93,42
0.37
0.90
6652.1
11 Weak Predictors
200
94.18
0.14
0.14
5.6
Iterative
91 Weak Predictors
1000
94.25
0.04
0.07
1.7
191 Weak Predictors
2000
94.26
0.04
0.06
1.6
Optimal
94.32
0.00
1
Table 1: Ablation the effectiveness of our iterative scheme NAS-Bench-101
Sampling from TopN)
M
TopN
#Queries
Test Acc.(%) (%)
SD(%)
Test Regret(%)
Avg. Rank
Exponential-decaa
10
100
1000
93.96
0.10
0.36
85.0
Linear-decay
10
100
1000
94.06
0.08
0.26
26.1
Uniform
10
100
1000
94.25
0.04
0.07
1.7
Uniform
10
1000
1000
94.10
0.19
0.22
14.1
Uniform
10
100
1000
94.25
0.04
0.07
1.7
Uniform
10
10
1000
94.24
0.04
0.08
1.9
Table 2: Ablation on exploiaation-exploaaiio trade-off on NAS- NAS-Bench-100
Method
#Queries
Test Acc.(%)
SD(%))
Test Regret(S)
Avg. Rank
WcakNAS
1000
94.25
0.04
0,07
WeakNAS (BO Variant)
1000
94.12
0.15
0.20
Optimal
94.32
0.00
1.0
Table 3: Comparing to the BO variant of WeakNAS on MAS-Bench-
We then study the exploitation-exploaaiie trade- off in Table NAS-Bench-10d 101 (a similar ab- lation in Mobilenet Search space on ImageNet also included in Appendix Table 6 by inves- tigating two settings:
94.2
Best Test Accuracy
(a) We gradually increase N allow for more exploration, similar to con- trolling in the opsilon-greedy
BONAS
(27) approach in the RL context; (b) We vary the sampling strat- egy from Uniform, Linear-decay to Exponential- decay (top models get higher probabilities by fol- lowing either linear-decay or exponential-dess distribution). We empirically observed that: (a) The performance drops more (Test Regret 0.22% vs 0.08%) when more exploration (TopN=1000 vs TopN=10) is used. This indicates that exten- sive exploration is not optimal for NAS-Bench- 101; (b) Uniform sampling method yields better performance than sampling method that biased
93.8
BANANAS
NAO
93.6
NASBOT
WeakNAS
oracle
200
400
600
800
1000
Number of Samples
Figure 6: Comparison with SoTA methods NAS-Bench-101. Solid lines and shadow regions denote the mean and std, respectively.

---

towards top performing model (e.g. linear- decar, aeponential-decay This indicates good architec- tures are evenly distributed within the Top 100 predictions of Weak NAS, therefore simple uniform sampling strategy for exploration is more optimal in NAS-Bench-IDe 101. To conclude, our Weak NAS Predictor strikes a good balance between exploration and exploration.
Apart from the above exploitation-explraaiiie trade-off of WeakNAS, we also explore the possibility of integrating other setal-ampling methods. We found that the local search algorithm could achieve comparable performance, while using Semi-NAS 20] as meta sampling method could further boost the performance of WeakNAS: more details are Appendix Section
3.2 Comparison to State-of-the-ae (SOTA) Methods
NAS-Bench-001: On NAS BencB-Bnhheeee 101 benchmark, we compare our method with several popular 41 44 Table that our method significantly outperforms baselines terms of sample efficiency. Specifically, our method costs 964 x, 447x, 378x, 245 x, 58x, and 7.5x less samples to reach the optimal architecture, compared to Random Search, Reg- ularized Evolution 14. MCTS [40], Semi-NAS
methods [14 40 21 7 20 19
LaNAS BONAS [9], respectively. We then plot the best accuracy against number of samples in Table 4 and Figure to show the sample efficiency on the NAS-Bench- B--B-Bennn... from which we can see that our method consistently costs fewer sample to reach higher accuracy.
Method
# #uueries
Test Acc.(% (%)
SD(%)
Test Regret (%)
Avg. Rank
Random Search
2000
93.64
0.25
0.68
1750.0
NAO [2]
2000
93.90
0.03
0.42
168.1
Reg Evolution
14
2000
93.96
0.05
0,36
85.0
Semi-NAS 20
2000
94.02
0,05
0.30
42.1
Neural Predictor
2000
94.04
0.05
0.28
33.5
WeakNAS
2000
94.26
0.04
0.06
1.6
Semi-Assessor
42
1000
94.01
0.31
47.1
LaNAS 21
1000
94.10
0,22
14.1
BONAS 19
1000
94.22
0.10
3.0
WeakNAS
1000
94.25
0.04
0.07
1.7
Arch2vec 41
400
94.10
0.22
14.1
WeakNAS
400
94.24
0.04
0.08
1.9
LaNAS 21
200
93.90
0.42
168.1
BONAS [19
94.09
0.23
18.0
WeakNAS
200
94.18
0.14
0.14
5.6
NASBOWI.r 45
94.09
0.23
18.0
CATE (cate-DNGO-LS) 43
150
94.10
0.22
12.3
WeakNAS
150
94.10
0.19
0.22
12.3
Optimal
94.32
0.00
1.0
Table Comparing searching efficiency by limiting the total query amounts on NAS-Bench-100
Method
NAS-Bench-
NAS-Bench-20
Dataset
CIFARIO
CIFARIO
CIFARI00
I
Random Scarch
188139,8
7782.1
7621.2
7726.1
Reg Evolution 11
87402.7
563.2
438.2
715.1
MCTS 40
73977.2
1528.3
t405.4
f578.2
Semi-NAS
20
147932.3
LaNAS 21
11390.7
1247.1
118.5
1292.4
BONAS 19
1465.4
WeakNAS
195.2
182.1
78.4
268.4
Table 5: Comparison on the number samples required to find the global optimal on NAS-Bench-1ue and BAS-Bench-201. denote reproduced results using adapted code.
JAS-Bench-201 We further evaluate on JAS-Bench-201 and compare with random search, Regu- larized Evolution [4], Semi-NAS (0), LaNAS BONAS(T) As shown in Table we conduct searches all three subsets (CIFARIO, CIFAR 100, ImageNet16-1200) and report the average number

---

of samples needed to reach global optimal on the testing set over 100 runs. It shows that our method has the smallest sample cost among all settings.
we further apply our method to open domain search without ground-truth 48 21 As shown in Tables 6 and using the fewest samples (and only fraction of GPU hours) among all, our method can achieve state-of-the-ard ImageNet top-1 accuracies with comparable parameters and FLOPs. Our searched architecture is also competitive to expert-design networks. On the NASNet Search Space, compared with the SoTA predictor-basee based NAS method LaNAS (Oneshot) 211, our method reduces 0.6% top-1 error while using less GPU hours. On the MobileNet Search Space, we improve the previous SoTA LaNAS [1 81.3% top-1 accuracy on ImageNet while costing less FLOPs.
Open Domain Search:
and compare with several popular methods [35] 14 46
3.3 Discussion: Further Comparison with SOTA Predictor-based NAS Methods
BO-based NAS methods [19 45 BO-based methods in general treat NAS as a black-box optimiza-
tion problem, for example, BONAS [9 customizes the classical BO framework in NAS with GCN embedding extractor and Bayesian Sigmoid Regression to acquire and select candidate architectures. The latest BO-based NAS approach, NASBOWL [45], combines the Weisfeiler-Lehman graph kernel in BO capture the topological structures of the candidate architectures.
Compare with those BO-based method, our WeakNAS is an "oversimplified" version of BO as explained in Section 2.3 Interestingly, results in 4 that WeakNAS is able to outperform BONAS [9]. and is comparable to NASBOWLr [45] on MAS--ench--01. showcasing that the simplification does not compromise NAS performance. We hypothesize that the following factors might be relevant: (1) the posterior modeling and uncertainty estimation in BO might be noisy; (2) the inherently structured NAS search space (shown in Figure could enable "shorteut" simplification to explore and exploit. In addition, the conventional uncertainty modeling in BO, such as the Gaussian Process used by (45), is not as scalable when the number of queries is large. In comparison, the complexity of WeakNAS scales almost linearly, as can be verified in Appendix Table [ In our experiments, we observe WeakNAS to perform empirically more competitively than current BO-based NAS methods at larger query numbers, besides being way more efficient.
To further convince that WeakNAS is indeed effective simplification compared to the explicit posterior modeling in BO, we report an apple-to-apple comparison, by use the same weak predictor from WeakNAS, plus obtaining its uncertainty estimation by calculating its variance using deep ensemble of five model [4 we then use the classic Expected Improvement (EI) [50] acquisition function. Table 3 confirms that such BO variant of WeakNAS is inferior our proposed formulation.
Advanced Architecture Encoding 41 43
We also compare WeakNAS with NAS using custom architecture representation either in unsupervised way such as arch2vee [41, or a supervised way such as CATE 43). We show our WeakNAS could achieve comparable performance to both methods. Further, those architecture embedding are essentially complementary our method to further boost the performance of WcakNAS, as shown in Appendix Section C
LaNAS (21]: LaNAS and our framework both follow the divide-and-conquee idea, yet with two methodological differences: (a) How to split the search space: LaNAS learns a classifier to do binary "hard" partition on the search space (no ranking information utilized) and split it into two equally sized subspaces. Ours uses a regressor to regress the performance of sampled architectures, and utilizes the ranking information to sample percentage of the top samples ("soft" partition), with the sample size N being controllable. (b) How to do exploration: LaNAS uses Upper Confidence Bound (UCB) to explore the search space by not always choosing the best subspace (left-most node) for sampling, while ours always chooses the best subspace and explore new architectures by adaptive sampling within it, via adjusting the ratio M/N to randomly sample M models from Top N. Tables 4 and 5 shows the superior sample efficiency of WeakNAS over LaNAS on BAA--Benc--Be 101/211
Semi-NAS 20 and Semi-Assessor 42]:
Both our method and Demi-NAS/Semi-AAe
use an
iterative algorithm containing prediction and sampling. The main difference lies in the use of pseudo labels: Semi-NAS and Semi-Assessor use pseudo labels as noisy labels to augment the training set, therefore being able to leverage "unlabeled samples" (e.g., architectures without true accuracies, but with only accuracies generated by the predictors) to update their predictors. Our method explores an orthogonal innovative direction, where the "pseudo labels" " generated by the current predictor guide sampling procedure, but are never used for training the next predictor.

---

That said, we point out that our method can be complementary to those semi- cumi-supervised methods (20 42], thus they can further be integrated as one, For example, Semi-NAS can be used meta sampling method, where each iteration we further train a Semi-NAS predictor with pseudo labeling strategy to augment the training set of our weak predictors. We show Appendix Table that that the combination of our method with Semi-NAS can further boost the performance of WeakNAS.
BRP-NAS 8 BRP-NAS uses stronger GCN-based binary relation predictor which utilize extra lopological prior, and leveraged a different scheme to control exploitation and exploration trade-off compare to our WeakNAS. Further, BRP-NAS also use somehow unique setting, i.e. evaluating Top-40 predictions by the NAS predictor instead of the more common setting of Top- [2 19 21 20 Therefore, we include our comparison BRP-NAS and more details Appendix Section
Model
Queries(ff) Top-1 (ir.
'lop-5 (mr.
Params(M)
FLOPs(M)
GPU Days
MobileNetV2
25.3
6.9
585
ShuffletNetV2
25.1
5.0
591
SNAS
27.3
9.2
4,3
522
1.5
DARTS
26.9
9.0
4.9
595
4.0
P-DARTS 52
24.4
7.4
4.9
557
0.3
PC-DARTS
24.2
7.3
597
3.8
DS-NAS(53
24.2
7.3
10.4
NASNet-A [35
20000
26.0
8.4
5,3
564
2000
AmoebaNet-A
14
10000
25.5
8.0
555
3150
PNAS 46
1160
25.8
8.1
NAO
1000
24.5
7.8
LaNAS
21
(Oneshot)
800
24.1
5,4
567
3
LaNAS
21
23.5
150
WeakNAS
800
23.5
6.8
5,5
591
2.5
Table 6: Comparison to SOTA results on ImageNet using NASNet search space.
Model
Queries(4)
Top-1 Acc.(%)
Top-5 Acc.(%)
FLOPP(M)
GPU Days'
Proxyless NAS 54
75.1
92.9
Semi-NAS 20
300
76.5
93.2
599
BigNAS
76.5
586
FBNetv3
20000
80.5
95.1
557
OFA 36
16000
80.0
595
1.6
LaNAS[21
800
80.8
598
0.3
1000
81.3
95.1
WeakNAS
800
81.2
95.2
593
0.13
Table 7: Comparison to SOTA results on ImageNet using MobileNet search space. *Does not include
supernet training cost.
Conclusions and Discussions of Broad Impact
In this paper, present novel predictor-based NAS framework named WeakNAS that progressively shrinks the sampling space, by learning a series of weak predictors that can connect towards the best architectures. By co-evolving the sampling stage and learning stage, our weak predictors progressively evolve to sample towards the subspace of best architectures, thus greatly simplifying popular NAS benchmarks prove that
the learning task of each predictor. Extensive experiments the proposed method is both sample-efficient and robust to various combinations of predictors and architecture encoding means. However, WeakNAS is still limited by the human-designed encoding of neural architectures, and our future work plans to investigate how to jointly learn the predictor and encoding in framework.
For broader impact, the excellent sample-efficinnyy of WeakNAS reduces the resource and energy consumption needed to search for efficient models, while still maintaining SoTA performance. That can effectively serve the goal GreenAI, from model search model deployment. It might meanwhile be subject to the potential abuse of searching for models serving malicious purposes.
Acknowledgment
Z.W W. is in part supported by NSF CCRI project (42016727).
10

---

References
[1] Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. arXiv preprint arXiv: 1806. 09050,005550 2018.
[2] Renqian Luo, Fei Tian, Tao Qin, Enhong Chen, and Tie- Liu. Neural architecture optimiza- tion. In Advances neural information processing systems, pages 7816-7827, 2018.
[3] Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang, Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer, Fbnet: Hardware-awar aware efficient convnet design via differentiable neural architecture search. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 10734-10742, 2019.
[4] Andrew Howard, Mark Sandler, Grace Chu, Liang- Ching Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for mobilenetv3. In Proceedings of the IEEE International Conference Computer Vision, 2019.
[5] Xuefei Ning, Yin Zheng, Tianchen Zhao, Yu Wang, and Huazhong Yang. A generic graph-based neural architecture encoding scheme for predictor-based nas. arXiv preprint .xXv922004001899, 2020.
[6] Chen Wei, Chuang Niu, Yiping Tang, and Jimin Liang. Npenas: Neural predictor guided evolution for neural architecture search. arXiv preprint (:Xiv:2003.12850 2020.
[7] Wei Wen, Hanxiao Liu, Hai Li, Yiran Chen, Gabriel Bender, and Pieter-Jan Kindermans. Neural predictor for neural architecture search. arXiv preprint
2019.
[8] Lukasz Dudziak, Thomas Chau, Mohamed Abdelfattah, Royson Lee, Hyeji Kim, and Nicholas Lane. Brp-nas: iiciicttoo-bbasee nas using gens. Advances in Neural Information Processing Systems, 33, 2020.
[9] Renqian Luo, Xu Tan, Rui Wang, Tao Qin, Enhong Chen, and Tic-Yan Liu. Neural architecture search with gbdt. arXiv preprint 2rXiv:2007.04785, 2020.
[10] Yunhe Wang, Yixing Xu, and Dacheng Tao. De-nas: Divide-and-conquer neural architecture search. arXiv preprint arXiv:2005. 11:0005.144550 /4456, 2020.
[11] Xiyang Dai, Dongdong Chen, Mengehen Liu, Yinpeng Chen, and Lu Yuan. Da-nas: Data adapted pruning for efficient neural architecture search. ECCV 2020, 2020.
[12] Zhaohui Yang. Yunhe Wang, Xinghao Chen, Jianyuan Guo, Wei Zhang, Chao Xu, Chunjing Xu, Dacheng Tao, and Chang Xu. Hournas: Extremely fast neural architecture search through an hourglass lens. arXiv preprint arXiv:2005. 14044446,.4 2020.
[13] Barret Zoph and Quoe V Le. Neural architecture search with reinforcement learning. arXiv preprint arXiv: 161110178,, 2016
[14] Esteban Real, Alok Aggarwal, Yanping Huang, and Quoe V Le. Regularized evolution for image classifier architecture search. In Proceedings of the aaai conference on artificial intelligence, volume 33, pages 4780 4789, 2019.
[15] Zhaohui Yang, Yunhe Wang, Xinghao Chen, Boxin Shi, Chao Xu, Chunjing Xu, Qi Tian, and Chang Xu. Cars: Continuous evolution for efficient neural architecture search. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 1829-1838, 2020.
[16] Weijun Hong, Guilin Li, Weinan Zhang, Ruiming Tang, Yunhe Wang. Zhenguo Li, and Yong Yu. Dropnas: Grouped operation dropout for differentiable architecture search. International Joint Conference on Artificial Intelligence, 2020.
[17] Yixing Xu, Yunhe Wang. Kai Han, Shangling Jui, Chunjing Xu, Qi Tian, and Chang Xu. Renas: Relativistic evaluation of neural architecture search. arXiv preprint
2019.

---

[18] Yanxi Li, Minjing Dong, Yunhe Wang, and Chang Xu. Neural architecture search in proxy validation loss landscape. In International Conference on Machine Learning, pages 5853-5862. PMLR, 2020.
[19] Han Shi, Renjie Pi, Hang Xu, Zhenguo Li, James Kwok, and Tong Zhang. Bridging the gap between sumpl--baedd and one-shot neural architecture search with bonas. Advances in Neural Information Processing Systems, 33, 2020.
[20] Renqian Luo, Xu Tan, Rui Wang, Tao Qin, Enhong Chen, and Tie- Yan Liu, Semi-supervised neural architecture search. arXiv preprint 200X::2000..1389,, 2020.
[21] Linnan Wang, Saining Xie, Teng Li, Rodrigo Fonseca, and Yuandong Tian. Sample-efficient neural architecture search by learning actions for monte carlo tree search. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
[22] Rachael Tappenden, Peter Richtárik, and Jacek Gondzio. Inexact coordinate descent: complexity and preconditioning. Journal of Optimization Theory and Applications,
2016.
[23] Mark Schmidt, Nicolas Le Roux, and Francis Bach. Convergence rates of inexact proximal- gradient methods for convex optimization. arXiv preprint arXiv: 109.2415, 2011.
[24] William W Hager and Hongchho Zhang. Convergence rates for an inexact admm applicd to separable convex optimization. Computational Optimization and Applications, 2020.
[25] Zhi-Hua Zhou. Ensemble methods: foundations and algorithms. CRC press, 2012.
[26] Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Design- ing network design spaces. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10428-10436, 2020.
[27] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
[28] Julien Siems, Lucas Zimmer, Arber Zela, Jovita Lukasik, Margret Keuper, and Frank Hutter. Nas-bench-301 and the case for surrogate benchmarks for neural architecture search. arXiv preprint arXiv:2008. 09777, 2020.
[29] Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference knowledge discovery and data mining, pages 785-794, 2016.
[30] Yiming Hu, Yuding Liang, Zichao Guo, Ruosi Wan, Xiangyu Zhang, Yichen Wei, Qingyi Gu, and Jian Sun. Angle-based search space shrinking for neural architecture search. arXiv preprint arXiv:2004.13331, 2020.
[31] Xuanyi Dong and Yi Yang. Nas-bench-201: Extending the scope of reproducible neural architecture search. In International Conference on Learning Representations, 2020.
[32] Chris Ying, Aaron Klein, Eric Christiansen, Esteban Real, Kevin Murphy, and Frank Hutter. Nas-bench-101: Towards reproducible neural architecture search. In International Conference on Machine Learning, pages 7105-7114, 2019.
[33] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[34] Patryk Chrabaszez, Ilya Loshchilov, and Frank Hutter. A downsampled variant of imagenet as an alternative to the cifar datasets. arXiv preprint 10X7.07777.88/199 2017.
[35] Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8697-8710, 2018
[36] Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, and Song Han. Once-for-all: Train one network and specialize it for efficient deployment. arXiv preprint 1xXiv:1908.0999),
12

---

[37] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255 leee, 2009.
[38] Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng, Zechun Liu, Yichen Wei, and Jian Sun. Single path one-shot neural architecture search with uniform sampling. arXiv preprint 1rXiv:1904.04420 2019.
[39] Ross Wightman. Pytorch image models. pytorch-Image-model. 2019.
https://github. hwwpthuub..om//wwggttaat.
[40] Linnan Wang. Yiyang Zhao, Yuu Jinnai, Yuandong Tian, and Rodrigo Fonseca. Alphax: exploring neural architectures with deep neural networks and monte carlo tree search. arXiv preprint arXiv: 2019.
[41] Shen Yan, Yu Zheng, Wei Ao, Xiao Zeng, and Mi Zhang. Does unsupervised architecture representation learning help neural architecture search? Advances in Neural Information Processing Systems, 33, 2020.
[42] Yehui Tang, Yunhe Wang, Yixing Xu, Hanting Chen, Boxin Shi, Chao Xu, Chunjing Xu, Qi Tian, and Chang Xu. A semi-supervised assessor of neural architectures. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1810-1819, 2020.
[43] Shen Yan, Kaiqiang Song, Fei Liu, and Mi Zhang. Cate: Computation-aware neural architecture encoding with transformers. arXiv preprint arXiv:2102.077 108, 2021.
[44] Colin White, Willie Neiswanger, and Yash Savani. Bananas: Bayesian optimization with neural architectures for neural architecture search. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 10293-10301, 2021.
[45] Binxin Ru, Xingchen Wan, Xiaowen Dong, and Michael Osborne. Interpretable neural ar- chitecture search via bayesian optimisation with weisfeiler-lehman kernels. In International Conference on Learning Representations, 2021.
[46] Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy. Progressive neural architecture search. In Proceedings of the European Conference on Computer Vision (ECCV), pages 19-34, 2018.
[47] Jiahui Yu, Pengchong Jin, Hanxiao Liu, Gabriel Bender, Pieter-Jan Kindermans, Mingxing Tan, Thomas Huang, Xiaodan Song, Ruoming Pang, and Quoc Le. Bignas: Scaling up neural architecture search with big single-stage models. In European Conference on Computer Vision, pages 702-717. Springer, 2020.
[48] Xiaoliang Dai, Alvin Wan, Peizhao Zhang, Bichen Wu, Zijian He, Zhen Wei, Kan Chen, Yuandong Tian, Matthew Yu, Peter Vajda, et al. Fbnetv3: Joint rechttecture-reippe search using neural acquisition function. arXiv preprint arXiv:2006.0000 02049, 2020.
[49] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. arXiv preprint arXiv:1612.01477 2016.
[50] Donald R Jones, Matthias Schonlau, and William Welch. Efficient global optimization of expensive black-box functions. Journal of Global optimization, 13(4):455-492, 1998.
[51] Sirui Xie, Hehui Zheng, Chunxiao Liu, and Liang Lin. Snas: stochastic neural architecture search. arXiv preprini arXiv:1812.000 09926. 2018
[52] Xin Chen, Lingxi Xie, Jun Wu, and Qi Tian. Progressive differentiable architecture search: Bridging the depth gap between search and evaluation. In Proceedings of the IEEE International Conference Computer Vision, pages 1294-1303. 2019.
[53] Yuhui Xu, Lingxi Xie, Xiaopeng Zhang, Xin Chen, Guo-Jun Qi, Qi Tian, and Hongkai Xiong. Pe-darts: Partial channel connections for memory-efficient differentiable architecture search. 2019.
arXiv preprint
13

---

[54] Han Cai, Ligeng Zhu, and Song Han. Proxylessnas: Direct neural architecture search on target task and hardware. arXiv preprint arXi:18112.03332. 2018.
14

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 4
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See
Section 4
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or URL)? [Yes] Code and instructions are released in htttps//ggituub com Swtp//VIAA--ee VTTA-GGrop/WeeaWAA...
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See main paper Section 2.4 for predictor setup, and Section 3 for detailed setup in each dataset, and the suppiemental material for ImageNet training details.
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes] Experiments times, see confidence interval in Figure 6 and Standard Deviation in Table [2 ImageNet experiments in Table
MAS--ench-100 101/201 are all ran multiple
ran only once due to the high computational cost (#=600 GPU hours) re-training each searched model, However, we will continue running more ImageNet experiments and report the confidence intervals in future versions.
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Section
4. If you using existing assets (e.g., code, data, models) curating/releasing
assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We used pub- licly available data and cited the corresponding papers, including CIFARI0[3) CIFAR 130 ImageNet16 1-166-10200 (2((4)) ImageNet (37. NAS-Bench-101 32 NAS-
Bench-201 31]. (F) TIMM Library (39]
(b) Did you mention the license of the assets? [Yes] We double check and make sure that the publication for the data and code we use include the corresponding licenses, See
Appendix for more details.
(c) Did you include any new assets either in the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent obtained from people whose data you's re using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "t04eIxXMcs", "reviewer": "Reviewer_eYgg", "summary": "Most Neural Architecture Search (NAS) techniques rely on being able to predict the performance of a network on a task without the cost of a full training run. This paper breaks down the methods of prediction into two categories: strong predictors and weak predictors. The authors note that this assumes that we want to evaluate all samples from the search space uniformly, even if we know certain subspaces are less interesting. Overall I think this is an interesting insight and in itself quite novel. \n\nThey therefore propose a method that attempts to improve the sample efficiency of NAS by using a series of “weaker” predictors to identify interesting subspaces, and then sample from those subspaces.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall I think this paper has fallen foul of unfortunate timing, since it seems like a nice idea which has potentially become less relevant in the last year with the advent of zero-cost proxies. For me, at least a comparison to [1] or discussion of how the methods compare is essential for this paper to be accepted. \n\nOtherwise, the paper is well written and the narrative is easy to follow but I found it quite hard to glean the specific implementation details. The idea that we can use weak predictors to select subspaces is appealing, but the mechanics of actually what that predictor looks like, what it’s predicting, and how that fits into the framework was quite hard for me to figure out.\n\nMy understanding is as follows: the typical setup here is that we have an MLP (or many MLPs?) which is going to predict a performance value for each network. We iterate between: sampling the top-N scoring networks from our predictions, and fitting the predictor\n\nI have a few questions:\n1. Do we have to start by first predicting a value for all of the networks in the space? \n2. We sample the top-N scoring networks from the predictions and fit the predictor on them. Then we “evaluate” the architectures using the predictor. What does this mean? Haven’t you just trained the networks in order to fit the predictor? Doesn’t it make more sense to just use the accuracies of the trained networks directly?\n3. On the next iteration, do we sample from the whole search space again or do we just sample from the top-N architectures we ranked last iteration?\n4. If we have multiple predictors, are they sampling different parts of the space? Or are they both being trained on exactly the same data? \n\nThe authors write out predictor-based NAS as a bi-level optimization problem (sampling + predictor fitting), which feels obvious at first but actually makes for quite a nice general framework: for example, we see that predictor-based NAS methods are really doing co-ordinate descent, which is something I had never really thought of before. As a consequence of viewing it this way, the idea of having weak predictors starts to make a lot more sense, which is really nice for the reader to follow along with.\n\n## Positive points\n- The use of figures is nice and illustrative, and makes the narrative of the paper clear and easy to read\n- The results are good and don’t seem cherry picked — all of the baselines/architectures look very sensible (e.g. using 1000 forests, 1000 trees, 1000 hidden units). The comparison to random search is very welcome, and the authors include error bars over 50 runs which is good to see\n- Using average rank + test regret as evaluation measures makes for a compelling case \n\n## Points to address\n5. I think the authors should be a lot more upfront about what a strong/weak predictor is (i.e. within the first page we should have some idea of what is a strong vs a weak predictor)\n6. I’m a little confused by the proof-of-concept experiment in section 2.2. What is the predictor being used? In figure 3(a) I can’t wrap my head around why the predicted accuracy of the global optimal point moves. Can the authors clarify this? \n7. I know that the papers were released at very similar times, but the paper feels incomplete without a reference/comparison to [1], which proposes a range of extremely cheap predictors. From the results in tables 1 and 3 it seems like this method will be stronger (and indeed it seems like it might be possible to use the two approaches in conjunction). Some discussion on this or a convincing comparison would go a long way to improving the paper.\n\n**Originality** Addressing the expense of neural architecture search is not particularly \"new\" or \"novel\", but I do think the idea to break the problem down into subsampling tasks is nice and not something I have seen before.   Differentiation between this work and previous contributions is clear. Related work is mostly adequately cited, though the paper is missing comparisons to the various \"zero-cost\" estimators listed in [1]\n\n**Quality:** The submission seems technically sound, with a nice logical flow to the experiments. I enjoyed reading the paper and seeing the figures/tables to support the claims being made in the paper. Each claim seems well-justified.\n\n**Clarity:** The submissions is relatively clear but I did get a little confused by the details of the sampling/resampling process (see my above questions). I'm not 100% sure I could reproduce the method from its current description.\n\n**Significance:** The importance of the results may be slightly diminished by the development of work on cheap estimators for neural architecture search (though I do also believe the authors could address this). Otherwise, the results are nice.\n\n## Minor typographical notes: \n- Capitalization on subsection 2.2 is strange (why is A capitalized?) \n- Figure 3 (a) accuracy is misspelt \n- “Why such “oversimplified BO\" can be effectively for our framework?“ -> Why can such “oversimplified” BO be effective for our framework?\n\n[1] Mohamed S Abdelfattah, Abhinav Mehrotra, Łukasz Dudziak, and Nicholas Donald Lane. Zero-cost proxies for lightweight nas. In Proceedings of the International Conference on Learning Representations (ICLR), 2021. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have adequately potential negative societal impact. Please see the main review above for my questions.", "review_text": "", "extra_scores": {"ethical_concerns": "I have no ethical concerns with this paper.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "UeTg6w8n59d", "reviewer": "Reviewer_nK9v", "summary": "The authors propose a progressively sampling approach to sample architecture-performance pairs for training performance predictors. The sampling subspace shrinks guided by the previous weak predictor. Experiments are conducted on NAS-Bench-101, NAS-Bench-201, and Open Domain Search (i.e. NASNet Search Space and MobileNet Search Space)", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Concerns:\n\n1. The proposed method is not well-justified. The progressively sampling approach seems to be biased to sample good-performing architectures. It is not clear why this could yield better results since this biased sampling could cause poorly predicted performance on under-sampled regions. Does this imply a biased sampled architecture set is good for training predictors? An ablation needs to be carried out to verify this. A simple ablated experiment could be training a strong predictor on a biased sampled architecture based on oracle and comparing it with uniform sampling.\n\n2. The authors spend quite a few sentences to motive exploitation-exploration trade-off by sampling M architectures from TopN. However, in Table 2, sampling with M=10 and TopN=100 is only 0.01 better than sampling with M=10 and TopN=10. When searching in an open domain, it incurs $\\frac{TopN-M}{M}$ times computation. it is not clear how much performance gain can bring by using this exploitation-exploration strategy. And this is not evaluated in the open domain search experiments.\n\n3. In Table 3, the proposed method is about 0.1 better than LaNAS and BONAS. For 200 queries, the standard derivation is 0.14. It is not significant enough to conclude the method is truly better.\n\n--------- post-rebuttal 1 --------- \n\nI raised my score from 4 to 5. I am willing to further raise my score if the Q2 is well-addressed during the rolling discussion.\n\n--------- post-rebuttal 2 --------- \n\nAfter reading the response to Q2, I decided to my score from 5 to 6. However, reproducibility is still my main concern since the code is not provided. If the paper is accepted, I hope all the details and reproducible GitHub repo should be released publicly.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No.", "review_text": "", "extra_scores": {"ethical_concerns": "No.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5 hours"}, "extra_text": {}}, {"review_id": "T1lw2msNTk_", "reviewer": "Reviewer_YANc", "summary": "This paper proposes a simple approach to predictor-based neural architecture search. It is motivated by the observation that in NAS, we are really just interested in precisely estimating the performance of well performing architectures while less accurate estimates are tolerable for low performing architectures (as long as the predictor facilitates to discriminate between the two). Therefore the paper proposes to learn a series of performance predictors that subsequently focus on better performing architectures, where each new predictor relies of the previous one, instead of learning a single predictor in one step.  \nIn each step, a new sample of predicted to be well performing architectures is drawn and evaluated. The experiments show that this leads to very efficient sampling of increasingly good architectures on several search spaces inclusing NasBench 101 and NasBench201, NatNet and MobileNet Search spaces from the literature such that well performing architectures can be found with few samples.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The idea proposed in this paper is very simple. Yet, it seems to be efficient on the considered search spaces. I would expect an additional discussion on the search spaces in this respect: What assumptions do we need to make on the search space? Under which conditions would well performing architectures be discarded early on? In my understanding, this approach should work on convex search spaces and it is likely  to discard narrow optima in non-convex ones. Is that correct?\n\nBesides the simplicity and practical effectiveness of of the idea, further strengths of the paper are:\n\n- The paper/idea is well motivated. \n- The mathematical formalization makes sense.\n- The approach is evaluated for several predictors and datasets.\n- An ablation study in the number of query samples is provided.\n- An ablation study on the new hyperparameters M and N is provided.\n- The evaluation on the NasBench101 and 201 search spaces is thorough.\n\nWeaknesses and questions:\n\n- The method introduces new hyperparameters that need ti be optimzed per search space.\n- In the ablation study on the number of samples, the chosen #of predictors is a bit unusual (6, 11, 91 predictors, etc).\n- What is the training time per predictor?\n- The Improvement over BONAS with respect to Test acc. for a give query budget is very small on NasBench101 (Table.  3)\n- It would be interesting to see the same experiment as in Table 3 for NasBench 201.\n- All number in tables 1 - 5 are already given as average values over 25 runs. The standard deviation should be reported as well.\n- In Figure 5, the comparison to BONAS would be very helpful. \n\n\nThe paper is overall well written and well structured. Yet, there are several minor language errors that should be addressed, for example:\nline 56: \"denoted\" should be \"denote\"\nline 105: \"just imprecisely solves that\" please reword\nline 105: \"previous methods\" - please specify which methods you refer to\nline 147: \"can be effectively for our framework\" should be \"can be effective\"\nline 202: \"choose from operator of\" please reword\nline 226: \", In\"\nline 231: \"its\" should be \"their\"\nThese are minor issues since the understanding is not impaired, but they should be fixed.\n\nSimilarly, in Equation (3), please check the \\setminus. This does not seem to be correct. I think is should be \\subseteq\n\n\n\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "With respect to the limitations, it would be good to also discuss the assumptions implicitly made on the regularity of the search space (see above). Otherwise, limitations and broader impact are appropriately addressed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4h"}, "extra_text": {}}, {"review_id": "9CHu56TocoW", "reviewer": "Reviewer_onF1", "summary": "\nThis paper proposes a new NAS method, which is a crucial step to rescue the NAS community. This paper achieves state-of-the-art results on several benchmarks, especially Open Domain Search on ImageNet. Sufficient experiments have been conducted to prove the effectiveness of the proposed method. Proper principled analyses have been provided to show the validness of the proposed method.\n\n", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n\n(Positive) I am really convinced by the motivation of this paper, especially by the impressive curve in Figure 1.\n\n(Weakness) The motivation section needs to be improved. In the first paragraph, the authors say, \"In order to cover the entire search space, they often train and evaluate a large number of architectures, leading to tremendous computation cost.\" This is not true. Gradient-based approaches and SPOS families do not have this limitation. Please revise this description.\n\n(Positive) This paper studies the architecture-performance prediction, which is a crucial step in NAS. As we know, inaccurate architecture-performance prediction is the cause of ineffective NAS in almost all existing NAS methods. Given that most current NAS methods are based on weight-sharing NAS that also use shared weights to perform architecture-performance prediction, and given that many NAS methods are not better than the random architecture selection (suggested by two ICLR 2020 papers and many ICLR 2021 submissions), analyzing the architecture-performance prediction is an important step to rescue the NAS community. This paper takes a deep insight into the architecture-performance prediction, which provides a timely metric for evaluating NAS's effectiveness. \n\nOverall, this paper provides a timely analysis of the current NAS's ineffectiveness.  I recommend accepting this paper to rescue the NAS community.\n\nI believe this paper deserves acceptance. As we know, variants of efforts have been made to improve NAS's effectiveness since 2016, and a great process has been reached. Despite the high expectation and solemn devotion, NAS's effectiveness is believed to be still low. This is inconsistent with many pioneer researchers' expectations four years ago, in which NAS is expected to be another revolutionary technique similar to 2012's deep learning. Currently, there are many NAS papers published every year. But their effectivenesses are unclear. Differently, this paper comprehensively analyzes the architecture-performance prediction, which provides a timely analysis of the current NAS's ineffectiveness. I think this paper can attract the community's attention, encouraging the community to pay attention to the architecture-performance prediction in NAS, especially when reviewing a NAS paper. Therefore, I recommend an acceptance for this paper.\n\n(Positive) The claimed improvement over its ICLR/ICML submission version is clear and significant. The improved version deserves acceptance by NeurIPS. Specifically, the authors have added discussion on the relationship with Bayesian Optimization. The paper also adds a comparison with Semi-NAS and ImageNet results in MobileNet Search Space. The improvement is indeed significant.\n\n(Positive) This paper should also inspire Gradient-based approaches and SPOS families because weight-sharing NAS can be regarded as using shared weights to perform architecture-performance prediction. I think this paper is a timely solution to the NAS community.\n\n(Positive) This paper achieves state-of-the-art results on several benchmarks, especially Open Domain Search on ImageNet.\n\n(Positive) Sufficient experiments have been conducted to prove the effectiveness of the proposed method.\n\n(Positive) Proper principled analyses have been provided to show the validness of the proposed method.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "\n\nPlease refer to and respond to the \"negative\" items in the above main review.\n\n", "review_text": "", "extra_scores": {"ethical_concerns": "\nNo.\n\n", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}, {"review_id": "78qsu-OaUBT", "reviewer": "Reviewer_93Rt", "summary": "This work proposes a query-based NAS approach, WeakNAS, which selects the best predicted architectures at each iteration, and then updates weak predictors iteratively. By using epsilon-greedy criteria to select new query data, WeakNAS bias the training data for the predictors to good subregion of the search space, leading to better modelling of the performance of good architectures at the expense of that of bad architectures. Empirically, the authors show that WeakNAS achieves SOTA performance on a variety of search spaces. However, the novelty of the work is rather weak. As the proposed framework is behind almost every query-based NAS and can be see as a simplified BO as the authors acknowledged and the design components (sampling strategy, predictor model) themselves are not novel as well.", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**: \n1.\tThe proposed method, WeakNAS, achieves very impressive empirical performance i.t.o query efficiency or final test accuracy on a variety of popular search spaces and datasets. The extensive experiment efforts are highly appreciated. This gives strong evidence supporting query-based NAS.\n2.\tThe paper is well written and the method is well explained. \n\n\n**Weakness**:\nThe main weakness of the approach is the lack of novelty. \n1.\tThe key contribution of the paper is to propose a framework which gradually fits the high-performing sub-space in the NAS search space using a set of weak predictors rather than fitting the whole space using one strong predictor. However, this high-level idea, though not explicitly highlighted, has been adopted in almost all query-based NAS approaches where the promising architectures are predicted and selected at each iteration and used to update the predictor model for next iteration. As the authors acknowledged in Section 2.3, their approach is exactly a simplified version of BO which has been extensively used for NAS [1,2,3,4]. However, unlike BO, the predictor doesn’t output uncertainty and thus the authors use a heuristic to trade-off exploitation and exploration rather than using more principled acquisition functions.  \n2.\tIf we look at the specific components of the approach, they are not novel as well. The weak predictor used are MLP, Regression Tree or Random Forest, all of which have been used for NAS performance prediction before [2,3,7]. The sampling strategy is similar to epsilon-greedy and exactly the same as that in BRP-NAS[5]. In fact the results of the proposed WeakNAS is almost the same as BRP-NAS as shown in Table 2 in Appendix C. \n3.\tGiven the strong empirical results of the proposed method, a potentially more novel and interesting contribution would be to find out through theorical analyses or extensive experiments the reasons why simple greedy selection approach outperforms more principled acquisition functions (if that’s true) on NAS and why deterministic MLP predictors, which is often overconfident when extrapolate, outperform more robust probabilistic predictors like GPs, deep ensemble or Bayesian neural networks. However, such rigorous analyses are missing in the paper.\n\n\n**Detailed Comments**:\n1.\tThe authors conduct some ablation studies in Section 3.2. However, a more important ablation would be to modify the proposed predictor model to get some uncertainty (by deep-ensemble or add a BLR final output layer) and then use BO acquisition functions (e.g. EI) to do the sampling. The proposed greedy sampling strategy works because the search space for NAS-Bench-201 and 101 are relatively small and as demonstrated in [6], local search even gives the SOTA performance on these benchmark search spaces. For a more realistic search space like NAS-Bench-301[7], the greedy sampling strategy which lacks a principled exploitation-exploration trade-off might not work well. \n2.\tFollowing the above comment, I’ll suggest the authors to evaluate their methods on NAS-Bench-301 and compare with more recent BO methods like BANANAS[2] and NAS-BOWL[4] or predictor-based method like BRP-NAS [5] which is almost the same as the proposed approach. I’m aware that the authors have compared to BONAS and shows better performance. However, BONAS uses a different surrogate which might be worse than the options proposed in this paper. More importantly, BONAS use weight-sharing to evaluate architectures queried which may significantly underestimate the true architecture performance. This trades off its performance for time efficiency.\n3.\tFor results on open-domain search, the authors perform search based on *a pre-trained super-net*. Thus, the good final performance of WeakNAS on MobileNet space and NASNet space might be due to the use of a good/well-trained supernet; as shown in Table 6, OFA with evalutinary algorithm can give near top performance already. More importantly, if a super-net has been well-trained and is good, the cost of finding the good subnetwork from it is rather low as each query via weight-sharing is super cheap. Thus, the cost gain in query efficiency by WeakNAS on these open-domain experiments is rather insignificant. The query efficiency improvement is likely due to the use of a predictor to guide the subnetwork selection in contrast to the naïve model-free selection methods like evolutionary algorithm or random search. A more convincing result would be to perform the proposed method on DARTS space (I acknowledge that doing it on ImageNet would be too expensive) without using the supernet (i.e. evaluate the sampled architectures from scratch) and compare its performance with BANANAS[2] or NAS-BOWL[4].\n4.\tIf the advantage of the proposed method is query-efficiency, I’d love to see Table 2, 3 (at least the BO baselines) in plots like Fig. 4 and 5, which help better visualise the faster convergence of the proposed method. \n5.\tSome intuitions are provided in the paper on what I commented in Point 3 in Weakness above. However, more thorough experiments or theoretical justifications are needed to convince potential users to use the proposed heuristic (a simplified version of BO) rather than the original BO for NAS.\n6.\tI might misunderstand something here but the results in Table 3 seem to contradicts with the results in Table 4. As in Table 4, WeakNAS takes 195 queries on average to find the best architecture on NAS-Bench-101 but in Table 3, WeakNAS cannot reach the best architecture after even 2000 queries.  \n7.\tThe results in Table 2 which show linear-/exponential-decay sampling clearly underperforms uniform sampling confuse me a bit. If the predictor is accurate on the good subregion, as argued by the authors, increasing the sampling probability for top-performing predicted architectures should lead to better performance than uniform sampling, especially when the performance of architectures in the good subregion are rather close.\n8.\tIn Table 1, what does the number of predictors mean? To me, they are simply the number of search iterations. Do the authors reuse the weak predictors from previous iterations in later iterations like an ensemble?\n\n\nI understand that given the time constraint, the authors are unlikely to respond to my comments. Hope those comments can help the authors for future improvement of the paper.\n\n\n**References**:\n[1] Kandasamy, Kirthevasan, et al. \"Neural architecture search with Bayesian optimisation and optimal transport.\" NeurIPS. 2018.\n[2] White, Colin, et al. \"BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Search.\" AAAI. 2021.\n[3] Shi, Han, et al. \"Bridging the Gap between Sample-based and One-shot Neural Architecture Search with BONAS.\" NeurIPS. 2020.\n[4] Ru, Binxin, et al. \"Interpretable Neural Architecture Search via Bayesian Optimisation with Weisfeiler-Lehman Kernels.\" ICLR. 2020.\n[5] Dudziak, Lukasz, et al. \"BRP-NAS: Prediction-based NAS using GCNs.\" NeurIPS. 2020.\n[6] White, Colin, et al. \"Local search is state of the art for nas benchmarks.\" arXiv. 2020.\n[7] Siems, Julien, et al. \"NAS-Bench-301 and the case for surrogate benchmarks for neural architecture search.\" arXiv. 2020.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitation and social impacts are briefly discussed in the conclusion. ", "review_text": "", "extra_scores": {"ethical_concerns": "No applicable.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a query-based NAS method that is a simplified version of Bayesian optimization and is also very similar to BRP-NAS. Novelty is therefore low. On the upside, surprisingly, the method performs very well compared to many baselines that were requested during the rebuttal period. After the initial set of reviews, the paper had borderline scores, and the internal reviewer discussion was not moving further. I therefore recruited an emergency reviewer, who first gave a rejecting score (citing lack of novelty, and not expecting major performance improvements over the many similar previous methods). However, the authors provided strong results. If the code is made available for fully reproducing the work I am in favour of accepting the paper as this would allow studying *why* the proposed method performs better than Bayesian optimization etc, and since it would also provide a solid available baselines for future advances.
Specifically, I asked the authors to make available the following:
- Code for their NAS method, including hyperparameters and information on how the hyperparameters were tuned.
- Code for their training pipelines (i.e., including the supernet) and final evaluation pipelines for the open domain search benchmarks.
- Code/scripts used to run all baseline methods
- Code/scripts for reproducing all tables and figures
- Final architectures found by the methods
- Anything else they can think of that a reviewer/reader would need to reproduce your results.

Decision: Accept (Poster)