How Predictors Affect Search Strategies in Neural Architecture Search?
Authors: TianJin Deng
Jia Wu
Venue: Submitted to ICLR 2023 (2023)
Keywords: Neural Architecture Search
Predictor-based Neural Architecture Search
Reinforcement Learning

Abstract: Predictor-based Neural Architecture Search (NAS) is an important topic since it can efficiently reduce the computational cost of evaluating candidate architectures. Most existing predictor-based NAS algorithms aim to design different predictors to improve prediction performance. Unfortunately, even a promising performance predictor may suffer from the accuracy decline due to long-term and
continuous usage, thus leading to the degraded performance of the search strategy. That naturally gives rise to the following problems: how predictors affect search strategies and how to appropriately use the predictor? In this paper, we take reinforcement learning (RL) based search strategy to study theoretically and empirically the impact of predictors on search strategies. We first formulate a
predictor-RL-based NAS algorithm as model-based RL and analyze it with a guarantee of monotonic improvement at each trail. Then, based on this analysis, we propose a simple procedure of predictor usage, named mixed batch, which contains ground-truth data and prediction data. The proposed procedure can efficiently reduce the impact of predictor errors on search strategies with maintaining performance growth. Our algorithm, Predictor-based Neural Architecture Search with Mixed batch (PNASM), outperforms traditional NAS algorithms and
prior state-of-the-art predictor-based NAS algorithms on three NAS-Bench-201 tasks.

Under review as a conference paper at ICLR 2023
How PREDICTORS AFFECT SEARCH STRATEGIES IN NEURAL ARCHITECTURE SEARCH?
Anonymous authors Paper under double-blind review
ABSTRACT
Predictor-based Neural Architecture Search is important topic since effi- ciently reduce the computational cost of evaluating candidate architectures. Most existing predictor-bat based NAS algorithms aim design different predictors im- prove the prediction performance. Unfortunately, even promising performance predictor may suffer from the accuracy decline due to long-term and continuous usage, thus leading to the degraded performance of the search strategy. That natu- rally gives rise to the following problems: how predictors affect search strategies and how to appropriately use the predictor? In this paper, take reinforcement learning (RL) based search strategy to study theoretically and empirically the im- pact of predictors NAS algorithm as model-based RL and analyze with guarantee of monotonic improvement at each trail. Then, based this analysis, we propose simple pro- cedure of predictor usage, named mized batch, which contains ground ground-truth data and prediction data. The proposed procedure can efficiently reduce the impact of predictor eπrоrs on search strategies with maintaining performance growth. Our al- gorithm, redictor-based Neural Architecture Search with Mixed batch (PNASM), outperforms traditional NAS algorithms and prior state-of -fe-----therrttt predictor- mrdictor-based NAS algorithms on three NAS-Bench-200 tasks and one MAS-Bench-ASS task.
search strategies. We first formulate predictor-RL. RL-boase
INTRODUCTION
Neural Architecture Search (NAS) aims automati- cally find out effective architectures in pre-defined search space for a given dataset (Baker al.) 2016 Zoph & Le 2016), which has shown to generate ar- chitectures that achieve promising results in many do- mains (Zoph et al.] 20188) Tan & Le. 2019: Howard jet al.] 2019: Chen et al.j 2020). However, due to the high computational cost for evaluating the generated architecture performance, traditional NAS methods are prohibitively costly in real-world deployment.
7000
Camenn
1000
Recently, many approaches have been proposed to re- duce the evaluation cost, which be categorized into training-free predictors (Pham et al. 200 Mellor 2021) and training-based predictors (Wei al.
REASH
Samplec ench
al.
2022
Springenberg et al. 2016: Shi et al. 2020; White
Figure 1: Cumulative error between true val
al.
2021a Lu al.] 2021 Wen al., 2020;; Tang
and predicted val over sampled architectures. REINFORCE+Prredicoo means long and con- tinuous usage of predictor without updating it.
et al.
2020; Luo et al.
2018 Training-base based meth-
ods, which require training a performance predictor to predict the final validation accuracy based on the fea- ture of architecture, have received much more atten- tion due to their better generalization ability. Recent efforts training-based methods focus on improving the prediction performance by designing models to precisely capture features of network architectures, e.g., GCN and Transformer. Several works demonstrate their robust predictions and combine them with the traditional search strategy such as Bayesian Optimization (BO) (Springen- berg et al. 2016 Shi et al. 2020 White et al. 202 and Evolutionary Algorithms (EA) (Wei al.

---

Under review as a conference paper at ICLR 2023
2022 Lu et al. 2021; Wei et al. 2022). Unfortunately,
promising performance predictor may suffer from the accuracy decline due to long-term and continuous usage (Fig. T. thus lead- ing to performance collapse. Most existing works barely consider the impact of predictor usage the search strategy. The inappropriate usage of predictor may perform worse asymptotically than their predictor-fies counterparts. That leads strategies and how to appropriately use the predictor to improve search efficiency?
natural questions: how predictors affect search
Long-term usage predictor
usage of predictor
D
True update
better than A
Updated by samples from the predictor
Updated samples
Figure 2: Parameters of policy updated by different ways. Left. Parameters of policy deviate far from the optimal one due to compounding error of long-term usage of predictor. Right. Limited mix usage of predictor balance performance and computational cost.
In this paper, we take RL-based search strategy to study the impact of predictors on search strate- gies both theoretically and empirically. We first formulate predictor-Rl-bbsed NAS algorithm as model-based RL and analyze class of predictor-based NAS algorithms with improvement guar- antees. Formula derivation results indicate that if the predictor is used for long time, enlarged predictor error compounding policy error will lead to performance collapse. Then, based on the analysis, we propose simple procedure of predictor usage, named mized batch, update the search strategy, which contains ground-truth data and prediction data. The prediction data, on the one side, can greatly improve sample efficiency, and on the other side encourages policy explo- ration. The ground-truth data allows the updated parameters close in parameter space and prevents a bad-update accidentally cause performance collapse (Fig. [ We empirically demonstrate that the proposed procedure can achieve pronounced improvements in performance compared to other predictor- based NAS approaches.
To summarize, contribution in this work are following:
We conduct the first study of the impact of predictors NAS search strategies both theo- retically and empirically.
We formulate and analyze a category of predictor-RL-baa NAS algorithms with improve- guarantees based predictor and policy emror. Theoretical analysis indicates that the long-term use of predictor declines the performance of search strategy.
We propose a novel predictor-based NAS framework, namely PNASM (Predictor-based Neural Architecture Search with Mixed batch), make limited usage of the performance predictor and improve the search performance.
Our proposed method outperforms both traditional and predictor-based NAS methods and achieves Nate-of-the-art results on CIFAR-I0, CIFAR-100, and maagNNet-16-12) of NAS- Bench-201, and TIMIT of NAS-Bench-ASR.
2 RELATED WORK
2.1 NEURAL ARCHITECTURE SEARCH
Traditional NAS methods, such as reinforcement learning (Zoph & Le 2016 Baker et al. 2016). evolutionary search (Real al. 2019), and gradient-based based search (Liu et al. 2019). have shown to generate networks that outperform quire enormous search costs due to the high evaluation cost for generated architectures. To reduce
Hanually-designee networks. However, these algorithms re-
2

---

Under review as a conference paper at ICLR 2023
the search costs, researchers have proposed predictor-basee based methods to quickly estimate the perfor-
mance of architectures instead of training from scratch (Wei et al. [022 Springenberg et al. 2016
Shi et al. 2020; White et al., 2021: la Lu et al. 2021; Wen et al.) 2020 Tang et al. 2020 Luo et al.
2018). There are two classes of predictor-based NAS methods:
Training-based predictors. Training-bassd predictors follow supervised learning paradigm to learn the correlation between network architectures and their corresponding performance. These predictors are usually used within BO frameworks (Springenberg al., 2016, [Shi et al. [2020] [hite
et al. 2021a), evolutionary frameworks (Wei et al. [2022] Lu et al. 2021 Wei et al. 2022). or with-
out any search strategy (Wen et al., 2020), conduct NAS. BONAS (Shi et al. 2020) adopts
GCN-based accuracy predictor as surrogate function of BO to search for architectures. Similarly,
BANANAS (White et al. 202 also uses BO perform NAS and provides a thorough analysis of the relation between BO and the neural predictor. NPENAS (Wei et al. 2022) develops two kinds of neural predictors to guide evolutionary strategy to boost the exploration ability. NPNAS (Wen) 2020) directly uses regression model to predict the validation accuracy of large number of random architectures and chooses the top-k architectures to obtain the best one. SemiNAS (Tang et al. unlabeled architectures. TNASP (Lu et al.] 2021) uses a Transformer-batt -baed predictor and evolu- tionary algorithms to perform NAS. Similar to these algorithms, our model PNASM also adopts the training-based predictor. Differently, we propose a novel update scheme by combining ground- truth data and prediction data to optimize the reinforcement learning strategy, which can reduce the impact of predictor error on the optimization strategy and improve search efficiency.
(et al.
2020) proposes semi-supervised predictor capture the intrinsic similarities of labeled and
Training- frree predictors. Recently, several works have proposed compute statistics from single minibatch data by a single forward and backward propagation. NASWOT (Mellor et al. (02T) evaluates sundomly-iittialinn
architectures based on binary activation codes of ReLU units. TE- NAS CChen et al., (021 analyzes the spectrum of the neural tangent kernel and the number of linear regions in the input space to rank architectures. Zero-Cost NAS (Abdelfattah et al. 202T) compares six conventional reduced-training proxies to compute model's score. Although these training- free predictors achieve satisfying results on some datasets, their performance cannot be guaranteed in practice due to the limited generalization ability (Lu al. 2021
White et al. 2021bb.
2.2 MODEL-BASED REINFORCEMENT LEARNING
Model-based Reinforcement Learning (MBRL) methods have shown great success on real-world sequential decision problems due to their sample efficiency ability (Kaclbling et al., 1996). MBRL learns a model of the environment, which predicts state transitions and rewards. Thus, they are widely used to solve problems where the data are hardly collected in real-world physical sys- tems. The dynamics of environment are usually modeled by Gaussian processes (Deisenroth & Rasmussen 20 local linear models (Levine & Koltun| 2013 Kumar et al. (016)) and neural net- work function approximators (Draeger et al. 1995: Gal et al. 2016)] Nagabandi et al. 20188] Janner al. 2019 Yu et al. 2020, [hhen al. 2020). If we consider the evaluating of candidate architec- tures as an RL-environment, we can formulate predictor-Re -L-ettrr-----aeedd NAS as MBRL. The difference between our formulation and the traditional MBRL is that our predictor is trained predict rewards not the state transitions.
3 METHOD
3.1 PRELIMINARY
NAS problem. Given dataset DD and a search space O of neural architectures, the RL-based optimization strategy is to search the best architecture A E that maximizes the expected accuracy on the validation set Dvatid, which is defined by:
A argmax E (Dem Paau DND [R (Aw-, Dvali))
AEO
(1)
s.t. w = argminC (Aw, Dusin)
w
where R(Aw-, Dvalid) measures the accuracy of an architecture A with parameters w" on the val- idation data D Puaiid- w is the parameters of architectures. represents the loss of architectures on the training data D Drrain-
3

---

Under review as a conference paper at ICLR 2023
Predictor-based NAS. Since evaluating architecture in Eq. typically takes hours, many NAS methods use performance predictors to speed up this process. A performance predictor fa generally consists of an encoder fæ and regressor fn. which encodes the information of discrete architectures into continuous feature representations and learns the correlation from the network features and the network performance, respectively. Most training-basee predictors
trained by the supervised learning from a database containing neural architectures A and their corresponding performances
R(A Dealid) (Luo al., 2018 Wen et al. 2020; Chen et al., 202 Lu al. (021). That is, predictor fø is trained to minimize the MSE loss between the predicted accuracy and the true accuracy of the architecture sampled from the database:
e arg min Σ (R. (R( ; - (a()))²
where (A( denotes the predicted performance After the above process, the performance predictor can quickly predict the final accuracy ranking of unseen architectures.
3.2 MBRL-BASED OPTIMIZATION STRATEGY
Predictor-RL-basse NAS can be formulated as MBRL problem with a tuple (S, A, R) where S, A. T, and y denote the state space, the action space, the state transition dynamics, the reward function and the discount factor (Please see Section A of the Suppl. for more details). Normally, MBRL corresponds to recovering the state transition dynamics T and the reward function R. Fol- lowing the RL-based NAS framework (Zoph & Le 2016)] Baker et al. [2166] Zoph et al. 2018). we only need to recover the reward function R, which corresponds to the predictor fg.
In this framework, (1,...,.... 8T; (T) at each episode, which corresponds to the description
agent, also called the controller, samples T-step trajectory
neural architecture A = 01.7. Then, evaluate the performance of the generated architecture A either by training from scratch or by the predictor. The evaluation result R(T) is used as a reward signal to update the parameters of the policy After several iterations, the agent will learn to generate, with high probability, an architecture with high reward (accuracy). The goal of the agent is to maximize the expected reward:
((TT) = Eynur [R(T) = Σ R(T)p(TT()
(3)
where R(T) denotes the evaluated performance of the generated architecture т. p(r () denotes the probability of trajectory
As the reward signal R is non-differentiable, one common approach is to use REINFORCE
Williams 1992) update the parameters 0 of the policy:
0x+1 = 0x + (o)olo)
(4)
where (TT) is given by:
N R R( Vo (T"10)
VOJ( = ETano log
N
(5)
T
V lo no Rc("s)) R(T")
n=It=l
where N is the number of neural architectures that the agent generates at each iteration (equivalent the batch size). T is the number of candidate operations (actions) of neural architecture.
3.3 MONOTONIC PREDICTOR BASED IMPROVEMENT
In this section, we will give monotonic improvement based on general predictor-RL-based NAS as described in Algorithm|] where the policy is optimized based on the data provided by the predic- tor. The performance of policy is affected by the predictor usage since errors in the predictor may be exploited by the policy optimization, thus leading to a large gap between the true performance of the policy and that under the predictor.
4

---

Under review as a conference paper at ICLR 2023
Algorithm Monotonic Predictor-Based Policy Optimization.
Input: of initial samples S; Batch-size N
1: Collect S architectures by running policy
2: Evaluate S architectures by training from scratch and store the samples into Dr
3: Initialize predictor fe and policy by the collected samples
4: while time limit not exceeded do
5: Collect N architectures by running policy me
6: Evaluate N architectures by the predictor fo and store the samples into
7: Update the policy Th via Eq. with B3;
8: end while
Our goal is to build a performance guarantee for the predictor-RL-basee NAS. Motivated by MBPO Janner et al. 2019), wish construct lower bound of the following form:
7> 2 -
(6)
where (TT) represents the expected true performance of policy which are updated by the reward signal training architectures from scratch, i.e., in the true dynamics; whereas 17( denotes the expected performance of policy that updated based dictor. Such statement guarantees that, as long as we improve by at least C' under the performance guarantee improvement
the reward signal provided by the pre-
predictor,
the true performance 1).
The difference C between the true performance and that under the predictor comes from two er- quantities of the performance predictor: generalization error due the prediction ability, and policy error (distribution shift) due to the updated policy receiving the reward signal provided by the predictor. Since the performance predictor is trained using supervised learning, we define this generalization eтror Cm by:
max TNED
R1 (T) R2(T)) cm
where (T) denotes the true performance of an architecture described by and (2(T) denotes prediction performance under the performance predictor, i.e., Rī == Sai n denotes the data- collecting policy. This eπrо cm can be estimated in practice by measuring the difference between the true reward and the prediction reward on the same trajectory т, which obtained under the data-collecting policy TD. We define policy error by the maximum total-variation distance of the policy between iterations:
max Drv (a( S
In practice, we can measure the KL divergence between policies. Based on these sources of eπтотs (generalization error em and policy eтror en). we now give our bound:
Theorem Let the generation error between the true reward and the prediction reward be bounded at each trajectory by and the policy divergence be bounded by c- Then the expected true reward and expected prediction reward of the policy are bounded as:
N
n(T)) 2 nf+) - E 2Rman En +
Σ r=1
Cmp(T(0)
Proof. See Appendix Theorem B.1
This bound implies that as long as we improve the expected reward 17( under the predictor by more than C'(m, <x), we can guarantee improvement under the expected true reward.
3.4 MIXING REAL-BASED AND PREDICTOR-BASER UPDATES
Theorem provides a useful relationship between true rewards and prediction rewards. However, noted that if the predictor eπrо cm is too high. there may not exist policy that can guarantee the improvement. Besides, the analysis of Theorem relies on using the prediction reward to update the policy continuously, i.e., equivalent to increasing N, which allows model error compound with
5

---

Under review as a conference paper at ICLR 2023
policy error and results in large gap value C. Thus, improve the algorithm by relying less on the performance predictor when the performance predictor inaccurate and instead by training neural architectures to rely more real data.
For the above issues, we introduce a simple procedure mixed batch to reduce the influence of two emrors the policy. A policy with mixed batch, denoted as Tmiz. means batch of N samples collected by the following two steps: first, run policy т to generate first architectures which are evaluated by training from scratch (under the true environment); then, generate N k architectures under the learned performance predictor fe (Algorithm 2. Under this scheme, the expected reward can be bounded as follows:
Theorem Given the expected reward Tiie) from the k-steps mixed batch method, we have
N
N
N
TT(x) 2 17( (Tizz)
Σ
Rmaa +
Σ
Rmar
+
Σ T=k+1
CmP(T
(10)
T=k+1
Proof. See Appendix Theorem
B.2
This bound implies that as long as we mix the true data and prediction data into one batch, we can reduce the error caused by long-term use of the performance predictor.
Algorithm Predictor-basee Neural Architecture Search with Mixed batch.
Input: # of initial samples S; batch size N; # of true samples k;; Collect S architectures by running policy my
2: Evaluate S architectures by training from scratch and store the samples into Dz
3: Initialize predictor fa and policy Ty by the collected samples
4: while time limit not exceeded do
5: Collect & architectures by running policy Te
6: Evaluate k architectures by training from scratch and store the samples into D3
7: Collect N - architectures by running policy To architectures by the predictor fa and store the samples into D,:
8:
Evaluate N
9:
Select k and N k pairs from D, and Dp respectively to form a mini-batch B; Update the policy ITe via Eq. 4 with B;
11: Retrain the predictor fo via Eq. with Dj;
end while
4 EXPERIMENTS
We employ our model on four datasets, specifically NAS-Bench-201 CIFAR-10, CIFAR-100, and ImageNet-
120, 120, MAS-Bench-AASR TIMIT. We split up our experiments into three categories: select- ing the best predictor for search spaces, evaluating the performance of our model and other popular algorithms on two NAS benchmarks, and performing ablation experiments. Moreover, we put sev- eral experimental results; the implementation details; information of baselines in Appendix D
4.1 CHOOSE PREDICTOR FOR NAS
To obtain high-performance prdictoorRL-basse NAS algorithm, we first choose a high-performance predictor among the currently most popular performance predictors, including MLP, GCN, BA- NANAS, BONAS, NAO, SemiNAS, and Transformer. We randomly run 20 times for each predictor on CIFAR-10 and report the mean and variance of the Kendall's Tau correlation coefficient on test samples. Kendall's s Tau is common indicator measuring the correlation between the ranking of prediction values and the true labels, and higher value indicates more accurate prediction. The pairs.
training and testing samples are sampled randomly from the collected architectur- Table presents the comparison results, from which we can make the following observations: 1) As the number of training samples increases, the performance of predictor improves. Therefore, it's important for all predictors to have enough initial training samples. 2) SemiNAS and Transformer perform well even with small number of training samples. For example, their Kendall's Tau can achieve around 0.550 when the number of training samples is 100 or 200. According to the ex- perimental results, choose SemiNAS as our performance predictor on NAS NAS-Bench-201 since it

---

Under review as a conference paper at ICLR 2023
adopts semi-supervised learning (Tang et al. 2020) to train the predictor, which could make full use of the unlabeled architecture information as the size of training samples increases, thus allowing to outperform Transformer.
Table 1: Performance Comparisons of Predictors on CIFAR-10.
Training Samples
100
200
400
600
800
1000
Test Samples
200
200
200
200
200
200
MLP
0.350 + 0.06
0.417
0.05
0.500 + 0.05
0.573
0.03
0.636 + 0.04
0.671
0.03
GCN
0.509 0.09
0.570
0.04
0.609 = 0.07
0.604 0.06
0.02
0.613 0.09
BANANAS
0.318 0.06
0.404
0.06
0.506 = 0.03
0.577 0.03
0.664 0.02
0.02
BOHAMIANN
0.435 0.08
0.525 0.04
0.529 0.04
0.00 0.54
BONAS
0.329 0.10
0.422
0.08
144001000
0.466 0.06
0.491
0.519
NAO
0,486 +: 0.05
0.512 0.06
0.05
0.556
1.55=0000 0.02
0.575
0.587 0.03
SemiNAS
0.546 + 0.05
0.588 0.02
0.635
0.02
0.669 0.02
0.687
+ 0.02
0.704 0.02
Transformer
0.574 +: 0.05
0.620 + 0.03
0.646 + 0.02
0.664 + 0.02
0.680 + 0.02
0.686 + 0.03
4.2 EMPIRICAL RESULTS
Performance on NAS-Bench-201. Table shows the comparison between the proposed method and strong baselines. According to the time budget, randomly run 100 and 20 times for tradi- tional and predictor-based methods, respectively. "search" means the total search time (time bud- get) including the time for initializing the predictor. "+B" and "+E" denote that combine the predictors with optimization strategies of Bayesian Optimization and Evolutionary algorithm, re- spectively. The best result on each dataset is in boldface. We can easily see from the experimental results that: 1) Our method (PNASM) outperforms both traditional and predictor-based methods and achieves the performance of 94.33%, 72.89%, and 46.44% on the three datasets, respectively, close to the optimal performance. 2) Compared with REINFORCE, our method improves the test accuracy on CIFAR-IO, CIFAR-100, and ImageNet- 16-120 by 0.31% %. 0.54%, and 0.7%, respec- tively, which demonstrates that the appropriate usage of predictor can help the optimization strategy to explore the more promising space. 3) Compared with the advanced SemiNAS+E, our PNASM (SemiNAS+RL) improves the validation accuracy CIFAR-10. 10, CIFAR- 100, and ImageNet- -ereeNett-1--11 16-120 by 0.06%, 0.24%, and 0.39%, respectively. The experiment results suggest that compared with EA, the advantage achieved by RL-based optimization strategy lies in the proper usage of predictor, i.e., mized batch which contains true architecture-accuuaay pairs and predicted ones, thus leading to the improvement in performance. 4) The performance of all predictors combined with BO is worse than those combined with EA and BOHB, which is consistent with the experiment result in White et al. 2021bb.
Table 2: Performance Comparisons of NAS methods on CIFAR-10 10, CIFAR- 100, and ImageNet- 120. "Optimal value" indicates the highest accuracy achieved on NAS-Bench-2011
CUAR
CHEAR
ImageNer
sctechral
salidtion
scacchs)
valadation
scarchts
validation
Traditionl
Methed
REA
20000
91.29 à 0.22
94.17
= 0.24
75000
72.75 = 0.73
72.50 d: 0.63
200000
45.03 4: 0.57
46.01 0.65
BS
40000
91.17
0.25
93.96
= 0.25
75800
71.81 - 0.87
71.89
+ 0:55
200000
45.18 + 0.81
45.41
0.91
REINFORCE
20000
91.27
0.21
94.02
= 0.22
75800
72.58 - 0.63
72.35
+ 0.55
200000
45.65 + 0.58
45.74-0.77
BOFIB
51000
91.34
0:2.1
94,10
: 0.23
96000
72.62
= 0.81
72,42
1: 0.65
290000
45.98 :: 0.50
46.11 0.63
Peehicoo--seed Melheel
MIP-B
3000
91.22 + 0.23
94.04
+ 0.22
71000
72.16 - 0.88
72.32
0.91
210000
45.47 + 0.61
45.63 - 0.79
MLP+E
39000
91.37 1 0.25
94,19
0.20
75000
72.67
0.34
72.58
0.35
230000
46.08 1 0.36
45,87 0.60
GCN+B
74000
U.54
210000
OCN+F
0.23
72.75
0.48
230000
0.000
BANANAS-B
0:20
70000
71,93
71,96
BONAS-B
71.67
0.31
210000
NAD4B
37000
91.27
94.08
72000
72,14
D/42
72.07
0.63
210000
NAD+E
40000
01.40
0.26
94.23
0.27
78000
72.90
0.38
72.62
0.40
230000
45.78
0.60
45.73
SemiNAS+B
37000
91.17
1 0.22
93.88
1 0.24
72000
72.17
0.43
72.10
1: 0.66
210000
45.13 11 0.74
45.24
0.71
SumINAS+
40000
91.45-4
0.19
94.26
0.23
76000
72,84
0.55
72.77
+ 0.53
250000
45.05
+ 0.98
45.67
0.54
PNASMOOUS)
2000
91.11
1 0.05
94.33
- 0.06
75000
73.08
0.22
7239
1: 0.36
200000
46.5
11 0.08
46.44
0.07
PNASM-AUUSS)
20000
91.48
4 0.10
94.26
- 0.12
65800
72,85
= 0.42
72.68
J: 0.49
200000
46.09
2: 0.30
45.94
0.69
Optinal value
91.61
94:37
73.49
73.51
46.75
47.31
Performance on TAS-Bench-ASR. Table 3 shows the comparison between the proposed method and traditional NAS algorithms. Since there is no information about the time cost of each architec- ture on NAS- VAS-Bench-ASR, we terminate the search process the number of sampled true architec- tures reaches 300. We randomly run 20 times for each method. The best result is in boldface. We can easily see from the experimental results that: 1) Our model (PNASM) achieves the best results.
7

---

Under review as a conference paper at ICLR 2023
2) Compared with REINFORCE, method improves the validation PER by 0.12%. 3) REA per- forms well compared to other traditional NAS algorithms. As we can see from the above results, our model still performs well on other NAS benchmarks, which indicates that mixed batch does help the RL-bbased search strategy make full use of the predictor.
Table 3: Performance Comparisons of NAS methods NAS-Bench-ASE
Method
Sampled True Archs
Val. PER(%)
Test PER(%)
REA
300
19.13 0.12
21.76 1 0.36
RS
300
19.17 + 0.08
21.91 + 0.29
REINFORCE
300
19.21 0.13
21.82 + 0.29
PNASM(OUrS)
300
19.09 0.10
21.73 + 0.21
PNASM-A(Ours)
300
19.10 0.09
21.74 + 0.17
4.3 ABLATION STUDY
Impact of Number of True Samples k. The value of k determines the ratio of true samples to prediction ones in a batch, which balances the performance and the computational costs. To study the impact of value & on the final performance, datasets with different values of k. Each experiment samples 1000 architectures. "search" means the total time costs including the time of predictor initialization.
conduct series of experiments on the three
Table 4 shows the results of PNASM with different k values over 20 runs with different seeds, from which we can see that: 1) The model without using the predicted data (k=all) performs well on the three datasets, but incurs a large computational cost. Conversely, if we use the predictor all the time (k=0), the model performs poorly on the three datasets but with least computational cost, which demonstrate that a long term usage of predictor will amplify both the predictor error em and the policy error éx, thus leading to policy collapse. 2) A specific value of & can achieve comparable performance to that k=all. For example, with the setting of k=5, the model achieves the performance 91.50% on the validation dataset on CIFAR-10. Similarly, k=2 on CIFAR-100 and k=2 on ImageNet- 16--1111111 120, which demonstrates that mixed batch is effective way of using predictor, which allows the search strategy to maintain excellent performance with less computational cost. Compared to the model with k=all, the model with k=5 achieves around 2 speedups CIFAR- 10, and the model with k= brings around 4x speedups on CIFAR- and ImageNet-16-1220, 3) Models with k=5, 8, and 15, outperform that with k=15 on CIFAR-100. We speculate the reason of this phenomenon is that the usage of predictor injects noise into the parameter space of policy. The parameter noise limited in reasonable range allows the policy to better explore the search space, as indicated by (Fortunato et al. 2018 Plappert et al. 2018).
To further study how affect the search strategy during the search process, Fig. 3 presents the current best architecture's s validation accuracy, from which we can make the following observations: 1) The performance of k: = (long-term and continuous predictor usage) inferior to others in most cases, which demonstrates that the long-term usage of unreliable predictor further exacerbates policy error. 2) As the number of ground-truth data increases (k: from 2 to 15), the performance curve gets close to that of k=all (blue line). In particular, there is specific value & can achieve comparable validation accuracy to that of k=all, e.g., h=5,, with low computational cost. Therefore, k=5 a good value that balances the performance and the computational costs. We recommend the true samples in the mixed batch is 5 for other datasets.
Adaptive Method. Although i can achieve good trade-off between the performance and the time cost, but the proper setting of ls requires precisely fine-tuning. To simplify the setting, we propose an adaptive method (PNASM-A), which can dynamically adjust k by measuring the differences between policies in two consecutive iterations:
==axN
(1I)
where a is given by:
= (x TT),
0 < DKI (T-(T:-1,55) <
(12)
1,
1 < DRL (T;-1,
8

---

Under review as a conference paper at ICLR 2023
Table 4: Comparison PNASM with different values on NAS-Bench-2D)
Thue samples(A))
k=0
k=2
k=5
A=8
k=15
A=all
CIFAR-I0
validation
91.23 = 0.20
91.37 = 0.22
91.50 = 0.06
91.50 + 0.05
91.40 = 0.14
91.50 + 0.04
test
93.98 + 0.22
94.06 = 0.26
94.28 +: 0.16
94.26 0.16
94.19 + 0.19 110000
94.31 + 0.05 130000
search(s)
17000
31000
49000
69000
CIFAR-100
validation
72.18 + 0.68
72.99 = 0.36
73.09 = 0.12
73.14 + 0.16
73.02 = 0.13
72.99 + 0.10
test
72.14 + 0.52
72.84 = 0.41
72.79 = 0.42
72.77 + 0.40 130000
72.44 = 0.32 220000
72.36 + 0.24 260000
search(s)
34000
59000
96000
1mageNett166-11
validation
45.85 + 0.82
46.30 + 0.23
46.21 + 0.29
46.21 + 0.16
46.32 + 0.00
46.31 1 0.02
test
45.84 +: 0.92
46.31 + 0.35
46.41 + 0.36
45.93 +: 0.66
46.47 + 0.00
46.47 + 1.35 780000
search(s)
100000
170000
280000
390000
640000
0.30
(a) CIFAR-10
(b) CIFAR-100
(c) ImageNet16-
Figure 3: Validation accuracy vs. number of architectures on different settings of k.
where T represents the policy after thh iteration: m, denotes the policy after 7th iteration. For the i + th iteration, a.N. As Table 2 shown, the adaptive method (PNASM-A) still outperforms other baselines all three datasets. Besides, the performance of PNASM-A is close that PNASM, which demonstrate the effective of our adaptive variant.
CONCLUSION AND FUTURE WORK
In this paper, we investigate the role of predictor usage in neural architecture search procedures both theoretically and empirically. We first formulate predictor-RL-basee NAS as model-based RI. problem, and provide with monotonic improvement guarantees, which suggests that the long-term and continuous usage of predictor will degrade the performance due to the model error exploited by the search policy. Motivated by this analysis, we then propose a novel framework PNASM that uses special procedure, mèzed batch, justify predictor usage, which can mitigate the impact of predictor errors on search strategies and reduce the computational cost. Extensive experiments on NAS-Bench-201 have shown the effectiveness of the proposed method. In the future, plan to investigate how to appropriately use the predictor with other search strategies.
REFERENCES
Mohamed S. Abdelfattah, Abhinav Mehrotra, Lukasz Dudziak, and Nicholas Lane. Zero-Cost Proxies for Lightweight NAS. In International Conference on Learning Representations (ICLR), 2021.
Bowen Baker, Otkrist Gupta, Nikhil Naik, and Ramesh Raskar. Designing neural network architec- tures using reinforcement learning, arXiv preprint urXiv:1611.02100 2016.
Irwan Bello, Barret Zoph, Vijay Vasudevan, and Quoc Le. Neural optimizer search with rein- forcement learning. In International Conference on Machine Learning, pp. 459-468. PMLR, 2017.
James Bergstra and Yoshua Bengio. Random search for machine learning research, 13(2), 2012.
optimization. Journal of

---

Under review as a conference paper at ICLR 2023
Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pp. 785-794, 2016.
Wuyang Chen, Xinyu Gong, Xianming Liu, Qian Zhang, Yuan Li, and Zhangyang Wang. Fasterseg: Searching for faster real-time semantic segmentation. In International Conference on Learning Representations, 2020.
Wuyang Chen, Xinyu Gong, and Zhangyang Wang. Neural architecture search on imagenet in four gpu hours: A theoretically inspired perspective. In International Conference on Learning Representations, 2021a.
Yaofo Chen, Yong Guo, Qi Chen, Minli Li, Wei Zeng, Yaowei Wang, and Mingkui Tan. Contrastive neural architecture search with neural architecture comparators. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattem Recognition, pp. 9502-9511, 2021b.
Patryk Chrabaszez, Ilya Loshchilov, and Frank Hutter. downsampled variant of imagenet as an alternative to the cifar datasets. arXiv preprint arXiv::707.088/1 2017.
Mare Deisenroth and Carl E Rasmussen. Pilco: A model-basee and data-efficient approach policy search. In Proceedings of the 28th International Conference on machine learning (ICML-I1), pp- 465-472. Citeseer, 2011.
Xuanyi Dong and Yi Yang. Nas-bench- 201: Extending the scope reproducible neural architecture search. In International Conference on Learning Representations (ICLR), 2020. URL https:
//openreview
Xuanyi Dong, Lu Liu, Katarzyna Musial, and Bogdan Gabrys. Nats-bench: Benchmarking nas
algorithmss for architecture topology and size. IEEE transactions on pattern analysis and machine intelligence, 2021.
Andreas Draeger, Sebastian Engell, and Horst Ranke. Model predictive control using neural net- works. IEEE Control Systems Magazine, 15(5):61-66, 1995.
Stefan Falkner, Aaron Klein, and Frank Hutter. Bohb: Robust and efficient hyperparameter opti- mization at scale. In International Conference on Machine Learning, pp. 1437-1446. PMLR, 2018.
Meire Fortunato, Mohammad Gheshlaghi Azar, Bilal Piot, Jacob Menick, Matteo Hessel, Ian Os- band, Alex Graves, Volodymyr Mnih, Remi Munos, Demis Hassabis, Olivier Pietquin, Charles Blundell, and Shane Legg. Noisy networks for exploration. In International Conference Learn- ing Representations, 2018. URL http::ttt////tttm / .et.
Yarin Gal, Rowan McAllister, and Carl Edward Rasmussen. Improving pilco with bayesian neural network dynamics models. In Data- Eifa-ifficient Machine Learning workshop, ICML, volume 4, pp. 25, 2016.
Andrew Howard, Mark Sandler, Grace Chu, Liang- Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for mobilenetv3. In Pro- ceedings of the IEEE/CVF international conference on computer vision, pp. 1314-1324, 2019.
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model- based policy optimization. Advances in Neural Information Processing Systems, 32, 2019.
Leslie Pack Kaelbling, Michael Littman, and Andrew W Moore. Reinforcement learning: A survey. Journal of artificial intelligence research, 4:237-285, 1996.
Vikash Kumar, Emanuel Todorov, and Sergey Levine. Optimal control with learned local models: Application to dexterous manipulation. In 2016 IEEE International Conference on Robotics and Automation (ICRA), pp. 378-383. IEEE, 2016.
Sergey Levine and Vladlen Koltun. Guided policy search. International conference on machine learning, pp. 1-9. PMLR, 2013.
10

---

Under review as a conference paper at ICLR 2023
Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable architecture search. In
International Conference on Learning Representations, 2019. URL htttpss://ppeeeeiiiiss net/ feet//orunndddsleeeeeee
Shun Lu, Jixiang Li, Jianchao Tan, Sen Yang, and Ji Liu. Tnasp: A transformer-basee nas predictor with svoluucn framework. Advances in Neural Information Processing Systems, 34:15125- 15137, 2021.
Rengian Luo, Fei Tian, Tao Qin, Enhong Chen, and Tie- Yan Liu. Neural architecture optimization. Advances in neural information processing systems, 31, 2018.
Abhinav Mehrotra, Alberto Gil CP Ramos, Sourav Bhattacharya, Lukasz Dudziak, Ravichander Vipperla, Thomas Chau, Mohamed Abdelfattah, Samin Ishtiaq, and Nicholas Donald Lane. Nas-bench-asee Reproducible neural architecture search for speech recognition. In International Conference on Learning Representations, 2020.
Yash Mchta, Colin White, Arber Zela, Arjun Krishnakumar, Guri Zabergja, Shakiba Moradian, Mahmoud Safari, Kaicheng Yu, and Frank Hutter. Nas-hench-suite: Nas evaluation (now) surprisingly easy. In International Conference Learning Representations, 2022.
Joe Mellor, Jack Turner, Amos Storkey, and Elliot J Crowley. Neural architecture search without training. In International Conference on Machine Learning, pp. 7588-7598. PMLR, 2021.
Anusha Nagabandi, Gregory Kahn, Ronald S Fearing, and Sergey Levine. Neural network dynamics for model- deep reinforcement learning with model-free fine-tuning, In 2018 IEEE Interna- tional Conference on Robotics and Automation (ICRA), pp. 7559-7566. IEEE, 2018.
Hicu Pham, Melody Guan, Barret Zoph, Quoe Le, and Jeff Dean. Efficient neural architecture search via parameters sharing. In International conference on machine learning, pp. 4095-4104. PMLR, 2018.
Matthias Plappert, Rein Houthooft, Prafulla Dhariwal, Szymon Sidor, Richard Y. Chen, Xi Chen, Tamim Asfour, Pieter Abbeel, and Marcin Andrychowicz. Parameter space noise for ploration. International Conference on Learning Representations, 2018. URL https: //openzeview. httt://oruump..eeaaaa.....
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoe V Le. Regularized evolution for image classifier architecture search. In Proceedings of the aaai conference artificial intelligence, volume 33, pp. 4780-4789, 2019.
Jian Shen, Han Zhao, Weinan Zhang, and Yong Yu. Model-based policy optimization with unsu- pervised model adaptation. Advances in Neural Information Processing Systems, 33:2823-2834 2020.
Han Shi, Renjie Pi, Hang Xu, Zhenguo Li, James Kwok, and Tong Zhang. Bridging the gap between sample-based and one-shot neural architecture search with bonas. Advances in Neural Information Processing Systems, 33: 1811-1899 2020.
Jost Tobias Springenberg, Aaron Klein, Stefan Falkner, and Frank Hutter. Bayesian optimization with robust bayesian neural networks. Advances in neural information processing systems, 29, 2016.
Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural net- works. In International conference on machine learning, pp. 6105-6114 14. PMLR, 2019.
Yehui Tang, Yunhe Wang, Yixing Xu, Hanting Chen, Boxin Shi, Chao Xu, Chunjing Xu, Qi Tian, and Chang Xu. A semi-supervised assessor of neural architectures. In proceedings of the JEFF/CVF conference on computer vision and patlern recognition, pp. 1810-1819, 2020.
Chen Wei, Chuang Niu, Yiping Tang, Yue Wang, Haihong Hu, and Jimin Liang. Npenas: Neural predictor guided evolution for neural architecture search. IEEE Transactions on Neural Nenworks and Learning Systems, 2022.
11

---

Under review as a conference paper at ICLR 2023
Wei Wen, Hanxiao Liu, Yiran Chen, Hai Li, Gabriel Bender, and Pieter-Jan Kindermans. Neural predictor for neural architecture search. In European Conference on Computer Vision, pp. 660- 676. Springer, 2020.
Colin White, Willie Neiswanger, and Yash Savani. Bananas: Bayesian optimization with neural architectures for neural architecture search. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 10293-10301, 2021a.
Colin White, Arber Zela, Robin Ru, Yang Liu, and Frank Hutter. How powerful are performance predictors in neural architecture search? Advances in Neural Information Processing Systems, 14:28454-28469, 2021b.
Ronald J Williams.
Simple statistical gradient-folloing algorithms for connectionist reinforcement learning. Machine learning, 8(3):229-22 256, 1992.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. Advances in Neural Information Processing Systems, 33:14129-14442 2020.
Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning. arXiv preprint arXiv:1611.01578, 2016.
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc Le. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE conference on computer vision and paltern recognition, pp. 8697-8710, 2018.
12

---

Under review as a conference paper at ICLR 2023
A
WORKFLOW OF GENERATING ARCHITECTURE BY RL-AGENT
Y1
Architecture description
MLP
Sanaple
sample
Y2
Yr
LSTM
ert
unroll
Controller
Contraller
Controller
MLP
1
Sr
S7-1
== 1
t=
Structure of Controller
Workflow of Controller
Figure 4: Left. Internal structure of the controller. Right. Workflow of the controller. It unrolls T steps to output a trajectory T {s1, a1............................ , ST, ar), which describes an architecture.
Figure 4 presents the main structure of the controller. The controller consists of two multilayer perceptron (MI.) layers which serve as the input-embedding layer and the output-emhedding layer, and LSTM network which is the core of the controller for remembering previous decisions. At each episode, the controller unrolls time-steps to sample trajectory т {51, a1...., ST, ar}. describes the representation of an architecture. At each step t, it works follows:
The input s, fed into the input-embedding layer, which converts s, into a high-dimensional em- bedding e,, thus making the agent better observe the state:
C Win St + bin
(13)
where W.n and bin are embedding parameters of the input layer.
Then, is fed to the core network consisting of LSTM layer, which helps the agent explore the correlation between decisions:
Ot, h1 = (s,
(14)
Next, the output of LSTM o, is fed to the output layer to convert the output of the LSTM into a low- dimensional representation yt, which denotes the distribution of the candidate operations, where = [Pt, oz).
W out O2 bout
(15)
where Wont and bout embedding parameters of the output layer.
At last, given the distribution for candidate operations, the agent samples one by the same sampling technique:
a, Sample(N(MH. o1))
(16)
Wt is fed to the state at the the next time step +
S = yt
(17)
The initial state 81 is a zero embedding vector.
Under the special workflow of the controller, the probability of trajectory is equal to:
13

---

Under review as a conference paper at ICLR 2023
p(T - = ne (01[s1) (s2)s11,a1) a1) no (azlsz) p (sals2, a2) To (arlsr)
T
(18)
= (s1) II no (ad)s) (s441)5(,
Since the next state 8 1 is equal to yt, which is converted by the previous action a,, both p(s1) and P(8++1 4x) are equal to one. Thus, p(r (0) is simplified as:
T
0( = II (0g)s:)
(19)
1=1
Therefore, the RL-based NAS can be formulated MDP with fixed transition probability but un- known reward function.
B
THEOREMS
Theorem B.1 Monotonic predictor-based improvement:
N
N
TTTT) 2 n(()
Σ
2mmax
EmP(T(0)
(20)
Proof. Let aD denote the data collecting policy (old policy under the true environment), Since the performance predictor relies on the training data collected by the policy WD, we need to introduce by adding and subtracting T)(TD), to get:
T) (+) ( (TTD) + [((T)) (1(
We can bound L1 and L2 both using Lemma C.2
For L,, there is no predictor error (generation error) since a and TD run under the true environ- ment:
Rmaz CT
For Lz, policy â rums under the predictor model which incurs predictor error and policy error. Thus, have:
L 2 Σ Rmast
Σ <mP(T
The desired result is obtained by adding the two bounds together.
Theorem B.2 Mixed batch bound:
N
πT()) 2 m)(mmiz))
Σ Rmarkz
Rmaa
+
Σ +=R+1
pmp(T
(21)
Proof. Let Amiz := TD. ā denote the policy with the mixed batch which runs the old policy TD under the true dynamics until samples, then executes the new policy under the predictor in the last N samples. As the proof for Theorem we add and subtract the correct reference quantity D. which can be also denoted TD RD,
14

---

Under review as a conference paper at ICLR 2023
(TT) TT(Tmiz) (T,,, πT) - ((TD, R)
n(T, πT) M(TD, TD) + (TDD TD) TT(T 4) (T, - (I(TDTD))) (TD TD) M(TD; 1
After k samples, L3 differ in both model and policy, which incorporates both predictor error N-k and policy error N in the last N k samples. This can be bound by Lemma C.2 with setting NN Cm and N Cr. which results in:
N Σ T=k+1
L3
C
Gx
Σ r=k+1
CmP(T10)
Adding two bounds L1 and Ly together yields the result.
C
LEMMAS
Lemma C.1 Policy error:
T
(agls)) - II n2 (2 (8))
(22)
Proof. Considering that p(s1) and p(sc+1lss. az) in Eq.
18 are equal to one, we make the following
approximation:
Drv ()v(m((bb))
Σ
nT (als) [2(a|))
2
s,a
T II (a(a))))) Drv((((10))ll((100))
T. (a,
Thus,
max (a)) xs max (T(( (p(T(()))ll(TTT 02))
T
T
max
II
n (a(alss))
x2 (ar(se) S
Lemma C.2 Expected reward bound:
In(T1) - ((T2) <
Rmarex
Cmp(( (())
(23)
Proof. Here, T7(11) denotes the expected true reward of with the reward function R1, and (TT2))
denotes the expected reward of my with the reward function R2. maxT~m: [R (T) - R2(T) S 6m and max, Drv (m
< 8x. According Eq. we have:
(m(m) (((22 Σ (R((())(7)))) R2(T))(((02)))
According Lemma
C.I.
have:
15

---

Under review as a conference paper at ICLR 2023
a) n(TT)) 2 (T2). Since max [R)(7) -R2(T)] S 8m we can ger: -8m Ri((T) = R2(T) S
6m + (T). Then.
(n(m1) - (T(221 = (RI(T)P(T(0)) - Re(T))((T(22)
(RI(T)((T1))) Re(()min)(7(82))
((R((T)TT)))) - (R((T) 5m)
Σ R((T))((1)))) Σ R1(T)p(T(02) + Σ 6mp(T(022
$ Σ R1(T))p(T(8)) - p(r(02)) Σ 8mp(F((22
T
Σ R1(T) II (acl())
II To (a)ls)))) + Σ 6mp(T 02)
Rmazl II
I (oels))
II (2(a)(s)) + Σ pmp(T(02)
1=1
1=1
$ Σ Rmazon + Σ 0mp(T(02)
b) nT(T)) < 11(T2). Since max [R((T) - -R((T) S 6m. we obtain: -5m + R2(T) S Ri(T) s
8m (T) Then.
(m(m1) - (7((5=))) = (1((T2) (7(m1
Σ (Ra(T)p(T(B2) - R1(T)p(T(81))
(Ba(T))(7(02) -
Σ ((2(F))((((2) - (Re(T) 8m)p((((1))
Σ Ra(T)p(T(02) Σ R2(T)p(T(01)
E8mp(T(01))
$ Σ Re(T))P(T(0)) p(T(0))]+ + Σ 8mp(T102)
Σ Re(7))p(T(81) - p(T(02)) + Σ 8mp(r(01)
T
T II (a((ss)) + Σ 0mp(T 01)
Σ R2(7)) (ac()))
t=1
t=1
Rmaz
(oolse)
II m2((a + Σ 8mp((((11) 1=1
t=1
Rmazd + Σ 8mp((T(0,)
In summary, we have (n{[T) nTTT2) S Er Rmazo 6n + Ex 6mp(TT0), where δ = 02 if n(T))) 2
n(T2)); otherwise, 0=00 0-
16

---

Under review as a conference paper at ICLR 2023
D EXPERIMENTAL DETAILS AND RESULTS
D.I EXPERIMENTAL DETAILS
NAS Benchmarks. NAS-Bench-200 (Dong& & Yang, 2020 is a benchmark dataset for NAS algo- rithms built on image classification tasks, including CIFAR-10, CIFAR-100, and TrageNet-16-113
[Chrabaszzz al., 2017). CIFAR-I0 consists of 60,000 (50,000 training images and 10,000 test images) 32 x 32 color images in 10 classes and each class contains 6,000 images. CIFAR-100 has 100 classes, and each class contains 600 images (500 training images and 100 testing images). NAS-Bench-22 provides a cell-based search space, where a cell is represented by a directed acyclic graph (DAG). DAG contains 4 nodes and edges, and each edge has representative operation candidates, which results in 15,625 neural cell architectures in total. Each architecture contains full training logs, validation accuracy, and test accuracy on CIFAR-10, 10, CIFAR- 100, and ImageNet-Ie 16- 120. In summary, MAS--Bnnc--20 allows researchers to easily compare different approaches by providing all architecture evaluation results.
NAS-Bench-ASR (Mehrotra et al. 2020 is tabular NAS benchmark for automatic speech recog- nition. The search space consists of 8242 unique models trained on TIMIT dataset. Each model includes all kinds of runtime measurements, such the per epoch validation and final test metries, Phoneme Error Rate (PER), and CTC loss. The search space consists of four nodes, with three main edges that can take or off.
of six operations, and six skip connection edges, which can be set to on
Baselines. We compare our method with two types of state-of-the-art methods: traditional NAS algorithms and predictor-based NAS algorithms:
1. Traditional NAS algorithms include: random search (RS) (Bergstra & Bengio 2012). REA (Real et al. 2019), REINFORCE (Williams) 1992), and BOHB (Falkner et al. 2018)) We use the code provided by Dong et al. (2021) to implement these algorithms. ensure fair comparison, we also require the search strategy of the baselines sample unique architectures as our method does.
2. Predictor- based NAS algorithms include: MLP (White et al., 2021a GCN (Wen et al.
2020 BANANAS (White et al., 2021a), BONAS (Shi et al. 22020. NAO (Luo et al.
2018 SemiNAS (Tang al. 2020). Transformer (La al. (021) and XGBoost (Chen & Guestrin 2016). We compare the most representative performance predictors to select the best performance predictor our predictor. In addition, we combine these performance predictors except Transformer with two widely used search strategies, BO and EA, as the predictor-based NAS algorithms. We use the code from (White et al. 2021b) and NAS-
BENCH-SUITE Meehta et al. 2022) to implement these algorithms.
Implementation details. Our method consists of two modules, an RL agent as the search strategy and performance predictor. The agent consists of input and output layers and LSTM model. The input-layer an embedding layer, and the size of each embedding vector is 32. The LSTM model two-layer LSTM with 35 hidden units each layer. The output-layer is linear layer with 32 hidden units. The agent is trained with the Adam optimizer with the learning rate 0.001. Weights of the agent initialized uniformly between -0.1 and 0. We use tanh of 2.5 and temperature of 5.0 for the sampling logits [Bello et al.) 2017) to prevent premature convergence and add the controller's sample entropy to the reward. weighted by 0.0001. Especially, the batch size N is 20, which is different from the setting of Zoph & Le (2016). We set k to 2, 5, 2, and 2 for CIFAR-10, CIFAR- 100, ImageNet- 620, and TIMIT, respectively. The initial sechitctture-acurraae PNASM and PNASM-A is 100 across all experiments. (Note that following Dong et al. (002)), we train candidate architectures 12 epochs and retrieve the best architecture by training in full epochs on NAS 20nh-
pairs for
D.2 EFFECTIVENESS STUDY ON COMPONENTS OF PNASM.
To figure out which part of PNASM boost the improvement, we conduct a series of ablation exper- iment PNASM, which mainly consists of LSTM-based agent, predictor and the batch strategy. 5Joows comparison results of different PNASM variants on duct the same experiment NAS-Bench-ASR, see Table 7. The four variants are:
(We also con-
17

---

Under review as a conference paper at ICLR 2023
03.5
Cumre Brae waddetton Artumay
22.0
00.5
$0.0
20.0
LOTM
05.0
LSTH- PIASH
LSTM-2
LSTM--Buldr+P
23000
10000
LOKC
Search 1ime
Search time
Figure 5: Performance of different variants of PNASM CIFAR-I0. Left figure presents the com- parison results among "LSTM", ""LTM--rAA" and PNASM. Right figure shows the comparison results between "LSTM+P" and "LSTM+Batch+P", because the search time of the two variants is inconsistent with others.
1. "LSTM" denotes that we run the search just using LSTMMbaasdd agent, without predictor. The agent (policy) updated after every candidate.
2. "LSTM+Batch" denotes that we update LTSM-based agent (without the predictor) after a batch of candidates is found.
3. "LSTM+P" denotes that we run the search with LSTM-base agent combined with pre- dictor. The agent is updated after every architecture is found.
4. "LSTM+Batch+P" denotes that we use LSTM-based agent and predictor to search, but perform batch-wise update of the agent, without evaluating any additional architectures from scratch.
5. PNASM denotes that LSTM-based agent and predictor but perform mixed batch update of the agent.
Predictors in all variants are initialized by 100 sampled architectures. We randomly run 20 times for each variant and the search time the time budget. Since "LSTM+P" and "LSTM+Batch+P" use the predictor for each candidate, the time cost of evaluating architecture is low. To ensure fair experiments, we allow "LSTM+P" and "LSTM-Batch+P" to sample 2000 architectures, which is greater than that by other variants. From Table 5 we can make the following observation: 1) Batch strategy does not improve LSSTM-based agent's performance on CIFAR-10, which is oppo- site on TIMIT. 2) Although the predictor can reduce the search time, but the long and continu- ous usage brings large error to the search strategy, as indicated by the results of "LSTM+P" and "LSTM+Batch+P" 3) Mixed batch helps LSTM-based agent to make better use of the predictor, since ground-truie truth data corrects the error caused by long-term usage. Additionally, to better demon- strate the search process of all variants, Figure 5 plots the current best validation accuracy of all variants over the search time on CIFAR-10.
Table 5: Comparisons of different Variants of PNASM on CIFAR-10.
Method
Search(s)
Validation
Test
LSTM
40000
91.29 0.19
94.03 + 0.24
LSTM+Batch
40000
91.21 0.23
94.01 + 0.24
LSTM+P
15000
91.16 0.24
93.96 + 0.24
LSTM+Batch+P
15000
91.14 0.30
93.94 1 0.27
PNASM
40000
91.51 0.05
94.33 +: 0.06
18

---

Under review as a conference paper at ICLR 2023
D.3 ADDITIONAL EXPERIMENTAL ON
D.3.1 CHOOSE PREDICTOR FOR NAS-BENCH-ASR
Since the predictor usually has poor generalization ability across the search space, we choose bigh-performance predictor for TIMIT on NAS-Bench. correlation of PER of four predictors: MLP, NAO, SemiNAS, and XGBoost. We randomly run 20 times for each predictor. We can easily make the following observation from Table 1) Predictors show poor generalization ability across the search spaces. For example, SemiNAS performs well on NAS-Bench.
again. compares the spearman
in Table [ but fails on NAS-Ba JAS-Bench-ASRE 2) MLP performs well if it has suffi- cient initial training data. 3) XGBoost performs good across the different settings of initialization. According to the experimental results, we choose XGBoost as the predictor on NAS-Bench-ASS
Table 6: Spearman Correlation of Predictors on TIMIT.
Training Samples
100
200
400
600
800
Test Samples
200
200
200
200
MLP
0.464
0.08
0.512 0.05
0.614 + 0.06
0.635 + 0.07
0.646 + 0.04
NAO
0.417 + 0.05
0.446 + 0.03
0.475 + 0.08
0.482 + 0.06
0.477 + 0.05
SemiNAS
0.418 0.04
0.435
0.08
+ 0.08
0.466 0.05
0.502 + 0.06
0.495 + 0.07
XGBoost
0.572 0.05
0.590 1 0.07
0.628 0.07
0.611 0.04
0.640 = 0.04
D.3.2 VARIANTS PÑASM ON NAS-BENCH-ASR,
Table shows comparisons different variants PNASM on NAS-Bench-ASR. Predictors take 100 true architecture-ve vall pairs to initialize. We make the similar observation Table except for the batch strategy. On NAS-Bench-ASR. Batch strategy can help LSTM-based agent achieve lower validation PER. We speculate that batch strategy is highly related to statisties of dataset on NASBench.
Table 7: Comparisons of different components of PNASM on NAS-Bench-ASR.
Method
Sampled True Archs Total Sampled Archs
Val. PER(%)
Test (ER(%)
LSTM
300
300
19.21 0.13
21.82 : 0.29
LSTM+Batch
300
300
19.18 0.10
21.85 0.29
LSTM+P
100
2000
19,25 + 0.07
21.87 + 0.23
LSTM+Batch+P
100
2000
19.21 0.14
21.91 0.21
PNASM
300
1500
19.09 + 0.10
21.73 + 0.21
19


Reviews:
[{"review_id": "bw3MDQE8Up", "reviewer": "Reviewer_k5Nf", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper evaluates model based performance predictors in NAS inside a model-based reinforcement learning (MBRL) framework. The authors show theoretically that as long as the expected predicted reward by the performance predictor improves by a certain factor during the optimization, one can guarantee improvements under the expected true reward. The efficacy of the method is demonstrated on the NAS-Bench-201 benchmark, where it outperforms most of the other black-box and predictor based NAS algorithms.", "summary_of_the_review": "Despite the theoretical guarantees and the motivation, the paper is lacking more empirical evidence  that the method works on a wide range of benchmarks, therefore it needs more work before it is ready for acceptance.", "strength_and_weaknesses": "The motivation of the paper to investigate the usage of model-based performance predictors for NAS inside a MBRL framework seems plausible. The paper is in general easy to follow and well-structured. There are some interesting theoretical guarantees adopted from the MBRL literature, which the authors built their algorithm, which seems simple and effective. Nevertheless, I have the following criticism:\n\n- **Not enough empirical evaluations**. While it is useful to evaluate a new proposed NAS method on tabular benchmarks such as NAS-Bench-201 used by the authors, I also find it necessary to evaluate on other tabular benchmarks, such as the ones in NAS-Bench-Suite [1], and on a real NAS benchmark. \n\n- **Novelty and limitations**. The proposed algorithm is based on a simple modification of model-based RL, derived from theoretical guarantees adopted from the RL literature. The theoretical justification is interesting but the novelty in the method itself is slightly incremental. Moreover, RL nowadays is not the off-the-shelf choice for NAS, as there are much more efficient methods existing.\n\n-- Questions --\n\n- What is the motivation behind using the specific proposed adaptive method in Section 4.4?\n\n- Would the same Mixed batch method be used for other black-box methods such as evolutionary strategies, or does it work only for MBRL because of the theoretical guarantees?\n\n-- Minor --\n\nLine 3 in Algorithm 1 seems unnecessary.\n\n-- References --\n\n[1] NAS-Bench-Suite: NAS Evaluation is (Now) Surprisingly Easy. Mehta et al. ICLR 2022", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "7beb2SfmsSW", "reviewer": "Reviewer_qrWv", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a predictor-based neural architecture search with N-sized mixed batch, in which the performances of k architectures are evaluated by training from scratch and the performances of the rest N-k ones are predicted by the predictor. Besides, it studies the impact of predictors on NAS theoretically and empirically. ", "summary_of_the_review": "To summarize, this paper is good in clarity, novelty and reproducibility. However, I am concerned about the weaknesses mentioned above.", "strength_and_weaknesses": "Strengths:\n1.\tThe paper is well written and easy to understand.\n2.\tThe theoretical and empirical study of predictors on NAS is good.\nWeaknesses:\n1.\tThe first question is that the evidence of the motivation is not direct. Since the problem to be solved is that “a predictor suffers from the accuracy decline due to long-term and continuous usage”, the authors need to plot a figure about the decline in accuracy of a predictor over time (search steps) in different settings to support their claim. \n2.\tAnother question is why choose k = 2, 5, 2 in cifar-10, cifar-100, imagenet-16-120 in Table 1, while the result in Table 3 shows that the best k should be 5, 8, 2 ? The best results of the two tables do not seem to match.\n3.\tIs there any related work about the mixed-batch method? \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "6b7tg7p3Iw", "reviewer": "Reviewer_tJna", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "During the search, the neural architecture search (NAS) algorithm uses the validation accuracy of the trained candidate model for feedback. As training a candidate every time is expensive, performance predictors, which take the architecture as input and output the validation accuracy, are used. This paper proposes a trick to improve performance predictor based RL search algorithm. \n\nThey use a batched search where the policy samples N architectures, the validation accuracy of all the architectures are obtained to   The RL policy first samples S architectures and the predictor is trained on them. Rather than using this trained predictor to estimate the accuracy of all the sampled architectures at every iteration to perform batch-wise updates of the policy,  the predictor is used to obtain the accuracy of only N-K architectures. The rest of the architectures are trained from scratch. They claim that this alleviates the error contributed in the search owing to the predictor component. They were able to  empirically demonstrate that their technique works.\n\n", "summary_of_the_review": " We need further justification to bolster the claim that this algorithm reduces the error better than just training the predictor with a lot more data. Also, they need to provide the results of the ablation study that I requested.", "strength_and_weaknesses": "Strength:\n1. They are able to find well performing architectures in the neural architecture search\n\nWeakness:\n1. Is your boost actually owing to your LSTM based RL agent, your batch-wise update policy of the agent or the entire PNASM algorithm? Can you do an ablation study on that? (1) Run the search without a predictor with just the LSTM agent, but update the policy after every candidate. (2) Update the policy after a batch of candidates are found, still without the predictor. (3) Use the agent and the predictor but update after every architecture is found. (4) Use the agent and the predictor but perform batchwise update of the policy but without evaluating any additional architectures from scratch. Rely completely on the predictor performance.\n\n2.  In the PNASM algorithm, the performance predictor is trained on fewer architectures to begin with and is trained as new architectures are encountered. So the contribution of error due to performance predictor is high initially. Including the accuracies of architectures trained from scratch would alleviate it. But if one trained the same performance predictor with more architectures initially, it would lead to a better Kendall Tau score and hence a lower predictor error component. Given that we know the accuracies of the architectures in the NASBench 201 search space, is there a way to analyze these two paths?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "5ul3A8Rs0k", "reviewer": "Reviewer_9HrM", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The proposed approach corresponds to an RL based acquisition strategy in a NAS search space, that is basing its trajectory on both the observed and predicted (surrogate) evaluations. This is called a mixed batch, whose share in either is a relevant hyperparameter to the success of the model by balancing the prediction error from a surrogate with the computational cost of a full evaluation. This trade-off is identified by a theoretic decomposition of the RL-based acquisition error, comprising of the generalization error of the predictor - to which an unmixed-ground-truth policy may overfit - and the policy error induced by a reward signal of the predictor on the other hand.", "summary_of_the_review": "Considering the limited novelty and the insufficiently precise formulation of the problem & problem setting I would recommend not accepting the paper in its current form.", "strength_and_weaknesses": "### Strengths:\n\n1. The theoretic analysis on the sources of error is nice and basing the method off-of that is a good idea.\n1. I presume, that formulating the archtiecture as a policy (albeit I’d strongly encourage to elaborate on that) is a beneficial way to express the architecture, as an incremental buildup on the architecture and attributing reward to model components and their sequential composition is akin to how humans would go about finding new architectures and may be helpful to reduce overal cost. The upside of this is that this heuristic is performance driven and the compositional strategy is being optimized.\n1. Having an ablation on the introduced hyperparameter.\n\n### Weaknesses:\n\n1. Considering https://arxiv.org/pdf/1904.02642.pdf, the proposed approach corresponds to an RL based acquisition strategy in a NAS search space, that (just like the linked approach) is allowed to update the surrogate model in its sequential acquisition process. The main difference is that for the policy update, both ground turth architecture evaluations and predictor based evaluations are mixed. The novelty is limited to the construction of the batch and the theoretic analysis. I do not see a clear benefit of having a mixed batch (which is also very costly due to the truth evaluation) over said paper other than speeding up the training of the acquisition function (RL agent) by making use of predicted reward signals as cheap proxy. Just like in the linked paper, the surrogate becomes a moving target by virtue of newly collected trajectories, that takes into account all what can be known in terms of ground truth and is guaranteed to be informative eventually by the sequential process.\n1. Fitting a supervised model to the observed rewards as a surrogate model is basically nothing new (as surrogates are doing that). Also the method is rather simplistic in the sense that it merely introduces a hyperparameters for the composition of “true” & “predictor” datapoints which is arbitrarily set albeit having some Ablation on that hyperparameter for their specific problem instances.\n1. There are still quite a few language issues (regarding declination etc.) that disturb the flow of reading.\n1. At least in the BO literature, one would rather prefer “surrogate” over predictor. In RL one would consider the predictor to be a “model”. This is just a rephrasing but makes it clearer. In the same sense, “predictor-RL-based NAS” should rather be reduced to model-based NAS using an RL controller or something similar. To facilitate readers from other fields the entry to your paper, this might be helpful footnote.\n\n#### Specifically:\n\n1. (Abstract) “Unfortunately, even a promising performance predictor may suffer from the accuracy decline due to long-term and  continuous  usage,  thus  leading  to  the  degraded  performance  of  the  search strategy” seems to be a somewhat wrong or at least insufficient description of what you seek to improve upon. Reading this the first time, it was absolutely unclear to me what exactly you were set out to do (what your issue is). Instead - from my understanding, you could rephrase it like this: \nUsing the fixed target predictor to an RL controller necessarily will cause performance degradation on new datapoints, as it does not allow for observations updating the predictor and is limited by its precision and mislead by its generalization error. This can be observed in k=0 (4.4). On the other hand k=N is the other extreme case where we only ever collect new datapoints - making it prohibitive to train brittle and expensive RL methods on it.\n1. (2.1.) The distinction between training-based & training-free predictors is poorly disambiguated. In particular, any of the methods described here can be used as feature space to a predictor, which then learns how to relate the feature space to a performance measure. After all both categories aim at linking their “statistics” of the architecture - be it the graph or cell representation or the e.g. zero-shot statistics to a performance score.\n1. I do not see the difference between a predictor-based NAS (having an encoder & regressor) to a surrogate model such as a GP, that defines a specific kernel (encoder).\n1. (3.2) Stating an MBRL problem is not sufficient on its own, but a specification of all its components in your setup-interpreation would bring value in understanding how the model is constructed. \n1. (3.2) I would very much appreciate a more detailed outline of how a trajectory corresponds to a description of a neural architecture. \n1. (4.4) k=all is informal. Make it k=N, Figure 2. The variation over seeds would be more indicative of the actual performance.\n1. At least to me, the depiction of Algorithm 1 is not required.\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject