What to expect of hardware metric predictors in NAS
Authors: Kevin Alexander Laube
Maximus Mutschler
Andreas Zell
Venue: ICLR 2022 Submitted (2022)
Keywords: Neural Architecture Search
Hardware-Aware
Predictors
Deep Learning

Abstract: Modern Neural Architecture Search (NAS) focuses on finding the best performing architectures in hardware-aware settings; e.g., those with an optimal tradeoff of accuracy and latency. Due to many advantages of prediction models over live measurements, the search process is often guided by estimates of how well each considered network architecture performs on the desired metrics. Typical predic-
tion models range from operation-wise lookup tables over gradient-boosted trees and neural networks, with little known information on how they compare. We evaluate 18 different performance predictors on ten combinations of metrics, devices, network types, and training tasks, and find that MLP models are the most promising. We then simulate and evaluate how the guidance of such prediction models affects the subsequent architecture selection. Due to inaccurate predictions, the selected architectures are generally suboptimal, which we quantify as
an expected reduction in accuracy and hypervolume. We show that simply verifying the predictions of just the selected architectures can lead to substantially improved results. Under a time budget, we find it preferable to use a fast and inaccurate prediction model over accurate but slow live measurements.

Under review as a conference paper at ICLR 2022
WHAT TO EXPECT OF HARDWARE METRIC PREDIC- TORS IN NEURAL ARCHITECTURE SEARCH
Anonymous authors Paper under double-blind review
ABSTRACT
Modern Neural Architecture Search (NAS) focuses on finding the best perform- ing architectures in hardware- settings; e-g., those with an optimal tradeoff of accuracy and latency. Due to many advantages of prediction models over live measurements, the search process is often guided by estimates of how well each considered network architecture performs on the desired metrics. Typical predic- tion models range from operation-wing lookup tables over gradient-boostdd trees and neural networks, with little known information evaluate 18 different performance predictors on ten combinations of metries, de- vices, network types, and training tasks, and find that MLP models are the most promising. We then simulate and evaluate how the guidance such prediction models affects the subsequent architecture selection. Due to inaccurate predie- tions, the selected architectures are generally suboptimal, which we quantify as an expected reduction in accuracy and hypervolume. We show that simply ver- ifying the predictions of just the selected architectures can lead to substantially improved results. Under a time budget, we find preferable to use fast and inaccurate prediction model over accurate but slow live measurements.
how they compare. We
INTRODUCTION
Modern neural network architectures are designed not only considering their primary objective, such as accuracy. While existing architectures can be scaled down to work with the limited available memory and computational power of, e.g., mobile phones, they are significantly outperformed by specifically designed architectures (Howard et al. 2017, Sandler al., 2018 Zhang et al. 2018 Ma et al., 2018). Standard hardware metrics include memory usage, number of model parameters, Multiply-Accumula operations, energy consumption, latency, and more: each of which may be limited by the hardware platform or network task. As the range of tasks and target platforms grows, specialized architectures and the methods to find them efficiently are gaining importance.
The automated design and discovery of specialized architectures is the main intent of Neural Archi- tecture Search (NAS). This recent field of study repeatedly broke state of the art records (Zoph et al.)
2018 Real al. 2018 Cai al., 2019 Tan & Le. 2019: Chu al., 2019a Hu et al. 2020) while aiming to reduce the researchers' involvement with this tedious and cone-consuming process to a minimum. As the performance of each considered architecture needs to be evaluated, the hardware metrics need to be either measured live or guessed by a trained prediction model. While measur- ing live has the advantage of not suffering from inaccurate predictions, the corresponding hardware needs to be available during the search process. Measuring on-demand may also significantly slow down the search process and necessitates further measurements for each new architecture search. On the other hand, a prediction model abstracts the hardware from the search code and simplifies changes to the optimization targets, such as metries or devices. The data set to train the predictor also has to be collected only once that a trained predictor then works in the absence of the hard- ware it is predicting for, e.g., in a cloud environment. Furthermore, a differentiable predictor can be used for gradient- based architecture optimization of typically non-differentiable metrics (Cai al.
2019 Xu et al. 2020 Nayman et al.] (021).
While the many advantages make predictors popular choice of hardware-aware NAS (e.g. Xu jet al. (2020); /Wu et al.] (2019); WWan et al.] (2020); Dai et al.] (2020): Nayman et al. (20TT), there guidelines which predictors perform best, how many training samples required, or

---

Under review as a conference paper at ICLR 2022
what happens when a predictor is inaccurate. This work investigates the above points. As first contribution, we conduct large-scale experiments on ten hardware- NAS-Bench (Li et al.] 2021a) and TransNAS-Bennn Ber---BBnc-111 (Duan et al. 202I). We explore how powerful the different predictors are when using different amounts of training data and whether these results generalize across different network architecture types. As second contribution, extensively simulate the subsequent architecture selection to investigate the impact of inaccurate predictors. Our results demonstrate the effectiveness of network-based prediction models; provide insights into predictor mistakes and what to expect from them. To facilitate reproducibility and further research, our experimental results and code are made available in Appendix A
martrare-metric metric datasets chosen from HW-
2 RELATED WORK
As the search spaces of NAS methods often differ from one another and lack extensive studies, the difficulty of fair comparisons and reproducibility have become a major concern Yang et al.] 2019 Li & Talwalkar 2020). To alleviate this problem, researchers have exhaustively evaluated search spaces of several thousand architectures to create benchmarks (Ying et al.) 2019 Dong & Yangl 2020 Dong et al., 2020; (iiems al.) 2020),, containing detailed statistics for each architecture. TransNAS-Benchh 101 (Duan et al. (021) evaluates several thousand architectures across seven diverse tasks and finds that the best task-specific architectures may vary significantly.
NAS Benchmarks:
The popular NAS-BENCH 201 benchmark (Dong & Yang. 2020) has been further extended with ten different hardware metrics for all 15625 architectures each of the three data sets CIFARIO, CI- FARION (Krizhevsky et al. (2009) and (mageNet16-1222 (Chrabaszez et al. 2017). Major findings of this HW-NAS Bench (Li et 2021a) include that FLOPs and the number of parameters are a poor approximation for other metrics such as latency. Many existing NAS methods use such inadequate substitutes for their simplicity and would benefit from their replacement with better prediction mod- els. Li et al. also find that hardware-specins costs do not correlate well across hardware platforms. While accounting for each device's 's characteristics improves the NAS results, it also expensive. Predictors reduce costs by requiring fewer measurements and shorter query times. |
Predictors in NAS: Aside from real-time measurements (Tan al. 2019 Yang al., 2018
hardware metric estimation in NAS is commonly performed via Lookup Table (Wu et al. Analytical Estimation or Prediction Model (Dai et al. 2020; Xu et al. 2020). While an operation- and layer-wise Lookup Table can accurately estimate bardware-agnostic metrics, such as FLOPs or the number of parameters (Cai et al. 2019 Guo et al.) 2020, Chu et al., (2099)), they may be subop- timal for device-dependent metrics. Latency and energy consumption have non-obvious factors that depend on hardware specifics such as memory, cache usage, the ability to parallelize each operation, and an interplay between different network operations. Such details can be captured with neural
2019
networks (Dai et al., 2020 Mendoza & Wang 20200] Ponomarev et al. 2020 Xu et al. 20200 or
other specialized models (aoo et al.) 2018 Wess et al. 2021).
Of particular interest the correct prediction of the model loss or accuracy, possibly reducing the architecture search time by orders of magnitude (Mellor et al.) (2220) Wang et al. [222] Li et al. 2021b). In addition to common predictors such as Linear Regression, Random Forests (Liaw al, 2002] or Gaussian Processes (Rasmussen) 2003); specialized techniques may exploit training curve extrapolation, network weight sharing or gradient information. Our experiments follow the recent large-scale study of White et al. (2021), who compare 31 diverse accuracy prediction methods based on initialization and query time, using three NAS benchmarks.
3 PREDICTING HaRDWARE METRICS
Our methods follow the large-scale study of White et al. (2021). who compared total of 31 curacy prediction methods. The differences between accuracy and hardware-metric prediction, our selection of predictors, and the general training pipeline are described in this section. In our ex- periments on HW-NAS BennhB and TransNAS-Pa Bench-101 described in Section we then compare these predictors across different training set sizes.
For further reading, we recommend recent survey on hardware-aware NAS Benmeziane al.
2021
2

---

Under review as a conference paper at ICLR 2022
Differences to accuracy predictors: There fundamental differences when predicting hard- ware metrics and the accuracy of network topologies. The most essential the cost to obtain a helpful predictor. which may vary widely for accuracy prediction methods. While determining the test accuracy requires the costly and lengthy training of networks, measuring hardware metrics does not necessitate any network training. Consequentially, specialized sccrracy-stimmttion methods that rely on trained networks, loss history, learning curve extrapolation, or early stopping do not apply to hardware metrics. Furthermore, so-called zero-cost proxies that predict metrics from the gradients of a single batch are dependant on the network topology but not on the hardware the network is placed on. Therefore, the dominant hardware-mette predictor family is madel-based.
Since all relevant predictors are model-based, they can be compared by their training set size. This simplifies the initialization time of a predictor as the number of prior measured architectures on which they are trained. In stark contrast, some accuracy predictors do not need any training data, while others require several partially or fully trained networks. Since an untrained network and few batches suffice to measure a hardware-metric the collection of such training set is comparably inexpensive.
Additionally, hardware predictors are generally used supplementary to one-shot network optimized for loss or accuracy. Depending on the NAS method, a fully differentiable predictor is required in order to guide the gradient- based architecture selection. Typical choices are Lookup Tables (Cai et al. 2019 Nayman et al. 2021) and neural networks el. 2020).
Model-based predictors: The goal of a predictor fa(a) is to accurately approximate the function fa), which may be, e-g., the latency of architecture from the search space A. A model-based predictor is trained via supervised learning on set Dtrain of datapoints (a. f()), after which it can be inexpensively queried for estimates on further architectures. The collection of the dataset and the duration of the training are referred to as inirialization time and training time respectively.
The quality of such a trained predictor generally determined by the (ranking) correlation between measurements {f(a)]a E Ateet) } and predictions {fp(a)]a la E A on the unseen architectures Atest C A. Common correlation metric choices are Pearson (PCC), Spearman (SCC) and Kendall's
Tau (KT) (Chu et al. 2019b Yu et al. 2020 Siems et al.) 2020)
Our experiments include 18 model-based predictors from different families: Linear Regression,
Ridge Regression (Saunders et al. 1998). Bayesian Linear Regression (Bishop) 2007). Support
Vector Machines (Cortes & Vapnik (995). Gaussian Process (Rasmussen] (003), D. Sparse Gaus-
sian Process (Candela & Rasmussen 2005). Random Forests (Liaw et 2002). XGBoost (Chen &
Guestrin 2016). NGBoost (Duan et al. 2020). LGBoost (Ke et al.
2017). BOHAMIANN (Sprin-
genberg et al. 2016), BANANAS (White et al. 2019). BONAS (Shi et al. 2020), GCN (Wen
jtt al. 2020), small and large Multi-Layer-Perr Percee-Perestsooss (MLP), NAO (Luo et al. 2018 and a layer-
operation Lookup Table model. We provide further descriptions and implementation details in
Appendix
B
Hyper-parameter tuning: The default hyperparameters of the used predictors vary significantly in their levels of hyper-parameter tuning, especially in the context of NAS. Additionally, some pre- dictors may internally make use of cross-validation, while others do not. Following White et al.) (2021), we attempt level the playing field by running cross-vee cross-validttion random- seaochh over hyper- parameters each time predictor is fit to data. Fach search is limited to 5000 iterations and a total run time of 15 minutes and naturally excludes any test data. The predictor-specin parameter details are given in Appendix( C
Training pipeline (020). see Appendix (A). We fit each predictor on each dataset and training size 50 times, using seeds (0, (0....,4)).
To make reliable comparison,
the NASLib library (Ruchte et al.
Some predictors internally normalize the training values (subtract mean, divide by standard devia- tion). We choose explicitly do this for all predictors and datasets, which reduces the dependency of
(e.g. learning rate) on the dataset and allows us to analyze and compare the prediction errors across datasets more effectively.
3

---

Under review as a conference paper at ICLR 2022
4 PREDICTOR EXPERIMENTS
We compare the different predictor models based on two NAS benchmarks, SW-NAAS-Bench LLi al. 202 and SannsNAS--Bncc--. (Duan et al.] [222T) They differ considerably by their network tasks, hardware devices, and architecture designs.
HW-NAS- architecture design and datasets In SW--NAS-Bench each architecture is solely defined by the topology of a building block ("cell"), which is stacked multiple times to create a complete network. Each cell is completely defined by choosing six candidate operations. Since they select from five different candidates each time, there are 5 15625 unique cell topologies. These cells are not fully sequential but contain paths of different lengths, which is visualized in Appendix D
JW-NAS-Bendh provides ten hardware statistics on CIFARIO, CIFAR 100 Krizhevsky et al. (2009) and ImageNet16- 1101666-12000 Chrabbaszz al. (2017), of which exclude the incomplete EdgeTTPU met- ric. Thus there are 27 data sets of varying difficulty. As detailed in Appendix D 12 of them can be accurately fit with Linear Regression and only 25 training samples. Many are also very similar since their measured networks differ only by the number of image classes. We therefore select five datasets that (I.) are not trivial to learn they are non-linear and (2.) not redundant:
ImageNet166 6-020, raspi4, latency
CIFAR100, edgegpu, energy consumption ImageNet16- eyeriss, arithmetic intensity
CIFAR100, pixel3, latency
CIFAR10, edgegpu, latency
BaansNAS-Bencc--ae architecture design and datasets CransNAS-Benchh 101 contains infor- mation for 7,352 different network architectures, used as backbones in seven diverse vision tasks. Since 4,096 are also a subset of HW-NAS-Benae with macro-level search space. Unlike a micro-level search space, where a cell is stacked multiple times create network, each network layer and block is considered individually. In particular, the networks consist of four to six pairs of ResNet blocks (He et al. 2016), which may modify the image size and channels in four ways: not at all, double the channel count, halve the spatial size, and both. Every network has to double the channel count to 3 times, resulting in 3 3,256 unique architectures. The networks may consequentially differ their number of layers (depth), the number of channels (width), and image size at any layer.
we focus the remaining 3,256 architectures
As done for HW-NAS- -enehh we select five of the seven available datasets for their latency measure- ments. Aside from the self-supervised Jigsaw task, there is little difference between the cross-task latency measurements (see Appendix E. We evaluate the possibly redundant datasets nonetheless, since latency predictions in macro-level search spaces classification and object detection tasks:
important domain for NAS on image
Object classification
Jigsaw
Scene classification
Semantic segmentation
Room layout
Fitting results and comparison The results, averaged over all selected HW-NAS-Bench and TransNAS --N----eech-111 datasets, are presented in Figures Ia and [b] respectively. The left plots present the absolute predictor performance, the right ones make relative comparisons easier.
Unsurprisingly, more training samples (i.e., evaluated architectures) generally lead to better predic- tion results, even until the entire search space is known (aside from the test set). This true for most of the predictors, although e.g. Gaussian Processes and BOHAMIANN saturate early. The simple Linear Regression and Ridge Regression models also fail to make proper use of hundreds of data points but perform decently when only a few training samples are available. Interestingly, the same is true for the graph-encoding network-based predictors BONAS and GCN. While know- ing how the different paths within each cell connect (see Appendix B) is especially useful given less than fifty training samples, the advantage disappears afterward. In contrast, the saapheencoiin decoder approach of NAO performs decently all times.
4

---

Under review as a conference paper at ICLR 2022
02
Lin. Reg.
Bayes. Lin. Reg.
Ridge Reg.
Kendail's average)
XGBoost
0.1
NGBcost
Kendalts Tau (absolute)
0.6
Random Forests
Sparse GP
BOHAMIANN
0.4
SVM Reg.
GCN
-0.1
BONAS
0.2
MLP (large)
MLP (small)
Lookup
0.0
10a
10a
training size
training
(a) Results Sei-NAS-Bencch NAO performs decently at all times, and none of the prediction models requires more than training samples improve over a Lookup Table model.
0.10
0.8
Lin. Reg.
Bayes. Lin Reg.
0.7
0.05
Ridge Reg.
Kendally Tau (absooutte)
XGBoost
0.6
NGBcost
Kondall's Tau (centered
Random Forests
0.5
Sparse GP
BOHAMIANN
0.4
SVM Reg.
(large)
0.3
MLP (small)
Lookup Table
10a
training set size
training set size
(b) Results on TransNAS Bench-101. Since all network architectures are purely sequential by design, we do not evaluate predictors that specifically encode the architecture connectivity (BANANAS, BONAS, GCN, NAO). After 20 training samples, MLP models outclass all other predictors.
Figure 1: How well the different predictors rank the test architectures, depending the training set size averaged over the five selected datasets. Left plots: absolute Kendall's Tau ranking correlation, higher is better. Right plots: same as left, but centered on the predictor-average.
Due their powerful rule-based approach. tree-based models perform much better given many training samples. Under such circumstances, I.GBoost is candidate for the best predictor model. Similarly, the predictions of Support Vector Machines also benefit strongly from more samples.
The model we find to perform best for most training set sizes are MLPs. They are among the top predictors almost all times the HW-NAS-Bench although tree-based models are competitive given enough data. After around 3,000 training samples, thinner and deeper MLPs improve over the wider and smaller ones. The path-encoding BANANAS model behaves similarly to regular large MLP but requires more samples reach the same performance. This is interesting since, aside from the data encoding, BANANAS is an ensemble of three large MLP models. Even though only the first network layer is affected by the data encoding, the more complicated path-encoding proves harmful
5

---

Under review as a conference paper at ICLR 2022
NAS Bench
TransNAS Bench 101
Kacpi4
HPGA
kyeris
Pixets
EdgeGIUU
Tesia V100
latency
0.45 (0.75)
0.99 (0.97)
0.99 (0.
0.49 (0.78)
8:21 (0.79)
(0.70)
energy
0.99 (0.97)
1.00 (0.99)
0.23 (0.79)
erilmmtiiciniemmity
0.84 (0.81)
Table 1: The Kendall's Tau correlation of Lookup Tables and Linear Regression (in brackets, using only 124 training samples) across metries and devices. Lookup Tables perform only marginally better on the FPGA and Eyeriss devices, but considerably worse in all other cases. More detailed statistics are available in Appendix E
when the connectivity of the architectures in the search space is fixed. On TransNAS-Bent 101, MLP perform exceptionally well. They are much better than any other tested predictor once more than just 20 training samples are available. The small MLP model can achieve KT correlation of 80% with just 200 training samples, which takes the best non-network-ban Vector Machine) four times as many. They are also the only models that achieve KT correlation of over 90%, about 5% higher than the next best model (LGBoost).
predictor (Support
Finally, the Lookup Table models (black horizontal lines) perform poorly in comparison any other predictor. Even though building such a model for HW-NAS- -e--ehh-Benchh datasets requires only 25 neigh- bored architectures, NAO and GCN perform better after just ten random samples. More than half of the predictor models require less than 25 random samples, while the worst need at most 60. On TransNAS- SaA-A---Bech--O-. Lookup Tables perform comparably better. Building one requires only neighbored architectures, and it takes most models between 50 and 100 random training samples to achieve better performance. When measured on a per dataset basis, we find that the Lookup Table models display a severe performance difference ranging from about 20% KT correlation (cifar 10- edgegpu.latency and Jigsaw) to over 70% (ImageNet16- 120-eyeriss www.hhettttcc.c.. crithmetic.intensity and Se- mantic Segmentation, see Appendix E). Other models prove to be much more stable.
Devices and Metrics NAS-Bench and TransNAS-Bench-Il shown in [ that is not always the case. The FPGA and Eyeriss hardware devices are very suit- able for Lookup Tables, achieving an almost perfect ranking correlation is possible. Nonetheless, Linear Regression requires only 124 training samples to compete even there and is significantly bet- ter in every other case. We finally observe that the difficulty of fitting predictors primarily depends on the hardware device, much more than the measured metric.
The previously described results are based on a specific selection of HW-
datasets that are hard to fit for Lookup Table models. As
5 EVALUATING THE PREDICTOR-GUIDER ARCHITECTURE SELECTION
Although the experiments in Section 4 greatly assist in selecting a predictor, it is not clear what specific Kendall's Tau correlation implies for the subsequent architecture selection. Given a perfect hardware metric predictor (Kendall's Tau = 1.0), we can expect that an ideal architecture search process will select the architectures with the best tradeoff of accuracy and the hardware metric, i.e., the true Pareto front. On the other hand, imperfect predictions result in the selection of supposedly- best architectures that are wrongly believed to be better.
To study how hardware predictors affect NAS results, extensively evaluate the selection of such supposedly- architectures in simulation. This approach can evaluate any combination of predic- tor quality, test set size, and dataset, without the technical difficulties of obtaining actual predictor models that precisely match such requirements. Since the hardware and accuracy prediction models are usually independent and can be studied in isolation, we use ground -tutth-thhhhh accuracy values in all cases.
Simulating predictors The main challenge of the simulation to quickly and accurately model predictor outputs. We base our simulation on how predictor-generated values deviate from their ground-trute targets on the test set, which explained in Figure 2 and further detailed in Ap- pendix G Since the simulated deviations similar to those of actual predictors, simulated pre- dictions are obtained by drawing random values from this deviation distribution and adding them to the ground-truth hardware measurements.

---

Under review as a conference paper at ICLR 2022
Predictions and targets
Predictor deviations
Simulated predictor deviations
KT=00, SCCw0, PCC=0.05
std=0.477
500
1.4
generated
predisted values
2-
1.0
1.2
0.8
0.6
0.4
02-
0.2
-3
-2
-1
5
-2
-3
true Values
deviation of the predictions
deviation the simulated predictions
Figure 2: A trained XGBoost prediction model on normalized ImageNet16-122 raspi4-latency test data. Left: The latency prediction (y-axis) for every architecture (blue dot) is approximately correct (red line). Center: The same data as on the left, the distribution of deviations made by the predictor (blue) and a normal distribution fit to them (orange). Right: A mixed distribution can simulate typical deviation distributions as that in the center plot.
0.45
0.40
0.40
0.35
bus parcta fort predeted partts hort allachksarr e5 selected edvtacttres
-2.0
5.0
0.5
normalzed 2nd 1ww.hhet:011120mmmmma. sitercy
normaaeee www. atancy
The simulated pre- dictor makes an inaccurate latency prediction for each architecture (blue), resulting in the selection of the supposedly-bes, architectures (orange dots). Even though the predicted Pareto front (orange line) may differ significantly from the ground-truth Pareto front (red line), most selected architec- tures are close to optimal. Right: Same data as left. The true Pareto front (red) and that of the selected architectures (orange). Simply accepting all selected architectures results a Mean Re- duction of Accuracy (MRA) of 1.06%. while verifying the predictions and discarding inferior results improves that to 0.43%. The hypervolume (HV, area under the Pareto- fronts) is reduced by 0.07.
Figure 3: An example of predictor-guiee guided architecture selection, std=0.5. Left:
single example of a simulation can be seen Figure 3 Although most selected architectures (orange) are close to the true optimum (red Pareto front), there almost always exists an architec- ture that has superior accuracy and, at most, the same latency. Simply accepting the 13 selected architectures in this particular example results in a mean reduction of accuracy (MRA..t of 1.06%. In other words, the average selected architecture has 1.06% lower accuracy than a comparable one on the true Paret front. However, simply verifying the hardware metric predictions through actual measurements reveals that some selected architectures are suboptimal. By choosing only the Pareto subset of the selection, the opportunity loss can be reduced 0.43% pMRApareto).
An important property of this approach is that it is independent of any particular optimization method. The supposedly-best architectures are always correctly identified, which an upper bound on how well Bayesian Optimization, Evolutionary Algorithms, and other approaches can perform. The exemplary MRAAparste opportunity loss of 0.43% is therefore unavoidable and depends solely on the hardware metric predictor, the dataset, and the number of considered architectures.
Results We simulate 1,000 architecture selections for each of the five chosen WV-NASS-Bench datasets, six different test set sizes, and eleven distribution standard deviations between 0.0 and 1.0. As exemplarily shown in Figure 3 each such simulation allows us to compute the mean reduction in accuracy (MRA) and the hypervolume (HV) under the Pareto fronts. The most important insights are visualized in Figure 4 and summarized below.
7

---

Under review as a conference paper at ICLR 2022
mapahatie 220-geria arthenatic Intersty
20-
dav11 povil jewey
mean
aiseectst vndescus
135
-6-
1392
5056
parstcoud the HESCIES WoRassun
630
2302
15425
4,0100 0.215.4 6.G8.74 2.00LS 2.80.22
1.00
0.00000 020.00 040.28
05:0.5
0.60.:22 15:037
2.0000
0.29 60
0.40 70
0.500 29
200.52
200.7
Suc. prucktioon dedercea Kencalatae
5id
predictoo deviation fareals
Figure 4: Simulation results, with the standard deviation of the predictor deviations and the resulting KT correlation on the x-axis. Left: Verifying the hardware predictions can significantly improve the results, even more so for better predictors. Center: The drops in average accuracy are dependant on the dataset and hardware metric. Right: Considering more candidate architectures and using better prediction models improves the results; larger values are better.
Verifying the predicted results matters (Figure 4 left). The best prediction models achieve KT correlation of almost 0.9, which translates to a mean reduction in accuracy of MRA 11.5%. That means, for each selected architecture, there exists an architecture of equal or lower latency in the true Pareto set (if latency is the hardware metric) that improves the average accuracy by 1.5%. Even though all selected architectures are believed to form a Pareto set, that is not the case. Their optimal subset has reduction of only MRApaveto R 0.5%, significant improvement. However, finding this optimal subset requires actually measuring the hardware metrics of the architectures selected by the used NAS method.
Furthermore, the left of Figure Jaids in anticipating the MRA given a specific predictor. If one used e.g. BOHAMIANN (KT=008, see Figure Ta instead of MLPs LGBoost (KT==09)), MRApareto increases from around 0.5% to roughly 1.2%. The average accuracy of the selected architectures is thus reduced by another 0.7%, just by using unsuitable hardware metric predictor. Lookup Tables (KT=00.45) are not even visualized anymore, they have an MRAparato of over 2.5%.
Another interesting observation is that the gap between MRAaN and MRAparcto is wider for better predictors. This is a shortcoming of the MRA metric that we elaborate on in Appendix
The dataset and metric matter (Figure 4 center). While generally present the results averaged over datasets, there exists some discrepancy among them. Most interestingly, predicting hardware metrics on harder classification problems (ImageNet16- 120 is harder than CIFARIO) also results in higher MRA. This is especially important since MRA is an absolute accuracy reduction. Even though the CIFARIO networks achieve twice the accuracy of ImageNet16- 120 networks, they lose less absolute accuracy to imperfect predictions. The order of MRA/dataset is primarily stable for any predictor KT correlation. Finally, as visualized by the shaded areas, the standard deviation of the MRA is generally huge. Consequentially, predictor-guided NAS is very likely to produce results of varying quality for each different predictor or search attempt, especially with less accurate predictors.
The number of considered architectures matters (Figure 4 right). We measure the hypervolume of the discovered Pareto front (i.e., the area beneath it, see Appendix H). which, unlike MRA, also considers the hardware metric. Quite obviously, if the architectures from the true Pareto set are not considered, they can not be selected. To achieve the highest possible hypervolume of around 4.2 (i.e. find the true Pareto set), every architecture in the search space must be evaluated with a perfect predictor. This impossible in most real-world cases, where only tiny fraction of all possible architectures can ever be considered.
For HW-NAS- -W--NAS--Bench, considering 5000 architectures with perfect live measurements and predicting the metrics for all 15625 with ranking correlation KT=00.33 results in selecting equivalent sets of architectures. As seen in Figurç Ridge Regression can achieve this performance with fewer than 100 training samples. Thus, worse predictor leads to better results if it enables considering more architectures. This insight is especially crucial for live measurements, which are accurate but slow. Similarly, estimating the network accuracy with super-networks takes much more time than predicting their performance with a neural predictor (Wen et al., (020). the measurement of any metrics is the limiting factor, a guided selection of cheap predictor is likely to do better.
8

---

Under review as a conference paper at ICLR 2022
6 DISCUSSION
Chosen prediction methods Given the nature of hardware-metric prediction, only the subset of model-based predictors evaluated by White et al. (202T) is suitable. We extended this subset with four models, including the popular Lookup Table. We abstained from evaluating layer-wise pre- dictors (e.g. WWess et al. (202T)) since such data is not available, and meta-learning predictors (Lee) et al. 2 due to the vast possibilities to configure them. A separate and specialized comparison between classic and meta-learning predictors seems favorable to us.
Simulation limitations In contrast to evaluating real predictors, the simulation allows us to quickly make statements for any test set sizes and predictor-inaccuraiies. However, naturally, the results are only approximations. While they match actual values, they are generally slightly pes- simistic (see Appendix D We also limit the simulation to HW-NAS Benchh since the changes to classification results are more accessible interpretation than changes loss values across dif- ferent problem types. Finally, the current simulation approach can not investigate methods that absolutely require trained one-shot network, such as gradient-based approaches. Including such methods is interesting direction for future research.
Transferability of the results Our evaluation includes five challenging and diverse datasets based on the micro-level search space of HW-NAS-Bench) and five latency-based datasets of var- ious macro-level search space architectures in MansNASS-Benche 101. Nonetheless, we find shared trends: All tested prediction models improve over Lookup Tables with little amounts of training data. Furthermore, most predictors benefit from more training data, even until the entire search space (aside from the test set) known. We also find that network- -baed predictors are generally best but may be challenged by tree-based predictors if enough training data is available. Given only a few samples, Ridge Regression performs better than most other models.
Recommendations While Lookup Tables are cheap, simple, and popular model in gradient- based architecture selection, we find a significant variance in performance across tasks and devices (see Table and Appendix ED. We recommend replacing such models with either MLPs or Ridge Regression, which are more stable, fully differentiable, and often take less than 100 training samples to achieve better results.
For most realistic scenarios where more than 100 training samples are available, MLP models are the most promising- They are among the top predictors on HW-NAS-Bene B-Bnhch and demonstrate out- standing performance on the TransNAS-Bencht 101 datasets. We found that specialized architecture encodings are primarily beneficial for little training data but suspect that they enjoy an additional advantage when network topologies
more complex and diverse (White et al. 2021).
While the query time for all predictors is less than 0.05s and thus negligible, there is a notable difference in training time (see Appendix F), primarily due to the recommend Ridge Regression for very little amounts of training data and LGBoost otherwise if that an important factor.
optimization. We
a NAS method selects architectures based on hardware metric predictions, we strongly suggest verifying the results by measuring the true metric value afterward. Doing so may eliminate inferior candidates and improve the average result substantially. Finally, the limiting factor to a NAS method is the slow measurement of hardware metrics, using much faster predictor may lead an improvement, even if the prediction model is less accurate.
7 CONCLUSIONS
This work evaluated various hardware-metric prediction models on ten problems of different met- rics, devices, and network architecture types. We then simulated the selection process for different test set sizes and predictor inaccuracies to improve our understanding of predictor-based architec- ture selection. We find that even imperfect predictors may improve NAS if their low query time enables considering more candidate architectures. Finally, verifying the predictions for the selected candidates can lead to a drastic improvement of their average performance. The code and results are made available, thus acting both for recommendation and baseline for future works.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Hadjer Benmeziane, Kaoutar El Maghraoui, Hamza Ouarnoughi, Smail Niar, Martin Wistuba, and Naigang Wang. A Comprehensive Survey on Haadware-Aware Neural Architecture Search. CoRR, abs/2101.000 001.6936, 2021.
http:/bb///2110 09336
Christopher M. Bishop. Pattern recogmition and machine learning, 5th Edition. Information science and statistics. Springer, 2007. ISBN 9780387310732. URL https: /www. www.wwow...oooooo
oclc/7100811
Han Cai, Ligeng Zhu, and Song Han. ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware. In 7th International Conference Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL https:
//apenzeview.net//EEee.ee iidiiid--in Ly
Joaquin Quiñonneo Candela and Carl Edward Rasmussen. A Unifying View of Sparse Approximate Gaussian Process Regression. J. Mach. Learn. Res., 6:199--1955 2005. URL URL http:t
-aneeerrr--ccceeeeeeeeeeee la05a html
Tianqi Chen and Carlos Guestrin. XGBoost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '16, pp. 785 785-794, New York, NY, USA, 2016. ACM ISBN 978- 4203---2222222 doi: 10.1145/ 2939672.2939785. URL http://///do acm. org 10. 1ttp://0.1115//99922299999
Patryk Chrabaszez, Ilya Loshchilov, and Frank Hutter. A downsampled variant of imagenet as an alternative to the cifar datasets. arXiv preprint
08819, 2017.
Xiangxiang Chu, Bo Zhang, Jixiang Li, Qingyuan Li, and Ruijun Xu. ScarletNAS: Bridging the Gap Between Scalability and Faimess in Neural Architecture Search. CoRR, abs/ abss/1908.nn 00211198.00000 908.0602,, 2019a. UtL:tttp:///rat www.iivorrg/bbae abs////b///1 908. 06022.
Xiangxiang Chu, Bo Zhang, Ruijun Xu, and Jixiang Li. FairNAS: Rethinking Evaluation Fairness of Weight Sharing Neural Architecture Search. CoRR, abs/1907.00 845, 2019b. URL http:
// w/aaxii.org//aaa 1907 .0..900..0000 01845
Corinna Cortes and Vladimir Vapnik. Support-vector networks. Machine learning, 20(3):273-297, 1995.
Xiaoliang Dai, Alvin Wan, Peizhao Zhang, Bichen Wu, Zijian He, Zhen Wei, Kan Chen, Yuandong Tian, Matthew Yu, Peter Vajda, and Joseph E. Gonzalez. FBNetV3: Joint Architecture-Recipp Search using Neural Acquisition Function. CoRR, abs/2006.02044 2020. URL |http::/ arxiv. wwwporgg/g.b..oooooeoeeeee /2006. 02049
Xuanyi Dong and Yi Yang. NAS-Bench-201: Extending the Scope of Reproducible Neural Archi- tecture Search. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL
net met//ooumm 1eiidd-oot HJ=HJxxkKKKDr
Xuanyi Dong, Lu Liu, Katarzyna Musial, and Bogdan Gabrys. NATS-Bench: Benchmarking NAS Algorithms for Architecture Topology and Size. arXiv preprint arXiv:2009.00437 2020.
Tony Duan, Avati Anand, Daisy Yi Ding, Khanh K. Thai, Sanjay Basu, Andrew Y. Ng, and Alejan- dro Schuler. Ngboost: Natural gradient boosting for probabilistic prediction. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 2690- PMLR, 2020. URL http::ttt:::/phhhhhhhhhh
m.r. wwwpppeesvv.v.... v duan2 html
Yawen Duan, Xin Chen, Hang Xu, Zewei Chen, Xiaodan Liang, Tong Zhang, and Zhenguo Li. MaisNNA--Bench--l0 Improving Transferability and Generalizability of Cross-Task Neural Ar- chitecture Search. CoRR, abs/2105.11871 2021. URL htt/::tts://aaae /ttpa/////xx..///// orgp.r//00...55.
11871
10

---

Under review as a conference paper at ICLR 2022
Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng, Zechum Liu, Yichen Wei, and Jian Sun. Sin- gle Path One-Shot Neural Architecture Search with Uniform Sampling. In European Conference Computer Vision, pp. 544-560. Springer, 2020. URL http:// httpvaxiivv/.. org/ abs/ 1904.
00420
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016. URL htttt:::pppiiiii
abs/ 1atpp//5512..33 03385)
Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang. Tobias Weyand, Marco Andreetto, and Hartwig Adam. MobileNets: Efficient Convolutional Neural Networks for Mobile Vision applications. CoRR, abs/1 0e5/1704.04550 04861, 2017.
URL http:://axxv..org/////// /bs/
1704. 04861
Shoukang Hu, Sirui Xie, Hehui Zheng, Chunxiao Liu, Jianping Shi, Xunying Liu, and Dahua Lin. DSNAS: Direct Neural Architecture Search without Parameter Retraining. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12084-12092, 2020.
URL http:ttttxxxi..oooo org/ abs/ 09128
Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang. Wei Chen, Weidong Ma, Qiwei Ye, and Tie- Yan Liu. Lightgbm: A highly efficient gradient boosting decision tree. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 3146-3154, 2017. URL http:::/pproccceiigg.ccoo neurips cc/ htcppappre/211//eeeeeee
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. CIFAR-10 (Canadian Institute for Advanced Research). 2009. URL httt:://twww cs toronto................. wttpl///kkii//ffffaaaa htmm
Hayeon Lee, Sewoong Lee, Song Chong, and Sung Ju Hwang. HELP: Hardware-Adaptive Efficient Latency Predictor for NAS via Meta-Leamnigg CoRR, abs/2106.08630, 2021. URL https:
/ //arxiv.or//aat abs/ aalaaaaa/2111
Chaojian Li, Zhongzhi Yu, Yonggan Fu, Yongan Zhang, Yang Zhao, Haoran You, Qixuan Yu, Yue Wang, and Yingyan Lin. TW-NAS-Bende Hardware-Aware Neural Architecture Search Bench- mark. CoRR, abb/21133.0058 2021a. URL
10584
Guihong Li, Sumit K. Mandal, Umit Y. Ogras, and Radu Marculescu. FLASH: Fast Neural Ar- chitecture Search with Hardware Optimization. CoRR, abs/2108.00568, 2021b. URL https:
// //arxiv. org/ www///sss.. 00568
Liam Li and Ameet Talwalkar. Random Search and Reproducibility for Neural Architecture Search. In Uncertainty in Artificial Intelligence, pp. 367-377. PMLR, 2020.
Andy Liaw, Matthew Wiener, et al. Classification and Regression by randomForest. R news, 2(3): 18 18-22, 2002.
Marius Lindauer and Frank Hutter. Best Practices for Scientific Research on Neural Architecture Search. Journal of Machine Learning Research, 21 (1443)) 2020.
Renqian Luo, Fei Tian, Tao Qin, Enhong Chen, and Tie- Yan Liu. Neural Architecture Optimization. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett (eds.), Advances in Neural Information Processing Systems
31: Annual Con- ference on Neural Information Processing Systems 2018, NeurlPS 2018, December 3-8, 2018, Montréal, Canada, pp. 7827-7838, 2018. URL httph:tttt//ttt /ptppeeee//pocceeeiieeeee ngg. nwwrips....... cc/ paper 22w12/hash///e//e httphashh/3336111 htwpahhh//......oooccccoooo61111
Abst:/5--bbbtte Abst html
Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design. In Vittorio Ferrari, Martial Hebert, Cristian Sminchis- escu, and Yair Weiss (eds.), Computer Vision - ECCV 2018 15th European Conference, Mu- nich, Germany, September 8-14, 2018, Proceedings, Part XIV, volume 11218 of Lecture Notes in Computer Science, pp. 122-132 138. Springer, 2018 doi: 10. 003077888--300-2--00 8. URL https: .eriilorgggooe 10000000777-0-00000202-0-0000000

---

Under review as a conference paper at ICLR 2022
Joseph Mellor, Jack Turner, Amos Storkey, and Elliot J. Crowley. Neural Architecture Search with- out Training, 2020. URL www.i.oorgg..a...aaa.o.......oat 20t./2006..o 04647
Daniel M. Mendoza and Sijin Wang. Predicting Latency of Neural Network Inference, 2020. URL
stanford. www.oooddddduupoooeeeeeetee fall 20202200000rrtree
5579 55793069.
Niv Nayman, Yonathan Aflalo, Asaf Noy, and Lihi Zelnik. HardCoRe-NAS: Hard Constrained difieRentiable Neural Architecture Search. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML, 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 7979-7990. PMLR, 2021. m.l.
URL http://pccceeeeeooonoooeeeses///
html
Evgeny Ponomarev, Sergey A. Matveev, and Ivan V. Oscledets. LETI:
Latency Estimation Tool and Investigation of Neural Networks inference Mobile GPU. CoRR, abs/2010.02871, 2020. URL
https://aaxii: //wpiv./rxvvvvv/mm ottp//as////0/// 0020/
Carl Edward Rasmussen. Gaussian Processes in Machine Learning. In Olivier Bousquet, Ulrike von Luxburg, and Gunnar Rätsch (eds.), Advanced Lectures on Machine Learning, ML Sum- mer Schools 2003, Canberra, Australia, February 2-14, 2003, Tübingen, Germany, August 4- 16, 2003, Revised Lectures, volume 3176 of Lecture Notes in Computer Science, pp. 63-71. doi: 978-3- 578-3-540-28650-11
Springer, 2003.
-, URL sttps://aci.ore
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoe V Le. Regularized Evolution for Image Classifier Architecture Search, 2018. URL httpp://arxi...rg/abbs...
00222.15444 01548
Michael Ruchte, Arber Zela, Julien Siems, Josif Graboeka, and Frank Hutter. Naslib: modular and flexible neural architecture search library. https://gitan: com oautt/sii/ 2020.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4510-4520, 2018
Craig Saunders, Alexander Gammerman, and Volodya Vovk. Ridge Regression Learning Algorithm in Dual Variables. In Proceedings of the Fifteenth International Conference on Machine Learning, ICML '98. pp. 515521, San Francisco, CA, USA, 1998. Morgan Kaufmann Publishers Inc. ISBN 1558605568.
Han Shi, Renjie Pi, Hang Xu, Zhenguo Li, James T. Kwok, and Tong Zhang Bridging the Gap between Sample-based and One-shot Neural Architecture Search with BONAS. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Con- ference Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL/https://prooeeding... wttppari.s.cccppppeeeee http://ssc.cppppr///////sss/
Julien Siems, Lucas Zimmer, Arber Zela, Jovita Lukasik, Margret Keuper, and Frank Hutter. NAS- Bench-301 and the Case for Surrogate Benchmarks for Neural Architecture Search, 2020.
Jost Tobias Springenberg, Aaron Klein, Stefan Falkner, and Frank Hutter. Bayesian Optimization with Robust Bayesian Neural Networks. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 29: Annual Conference on Neural Infor- mation Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pp. 4134- http://wsss.c.ppppp///////////aa 4142, 2016. URL httpsss/ppxoceddngggg httpp//oooeedd.....eeiii........ neurips......p....o..m..
Mingxing Tan and Quoc V. Le. EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks. CoRR, abss1905.. 1946, 2019. hillhhttp///arxx/ wwwp:/i.vwrrg/aaaaa.oee/ae/ htgg//b//// 1905 11946
12

---

Under review as a conference paper at ICLR 2022
Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoc V. Le. MnasNet: Platform-Aware Neural Architecture Search for Mobile. In IEEE Conference Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp. 2820-2828. Computer Vision Foundation IEEF, 2019. doi: URL htttp://oeenacoe
10.1109/CVPR.22 200.CVV9...01900292.
thecvf. com content.
CVPR 2019/html/Iaa MnasNet- Platform Aware Search for_io_lee_M CePR 2019 paper.
- Neural Architecture.
Alvin Wan, Xiaoliang Dai, Peizhao Zhang, Zijian He, Yuandong Tian, Saining Xie, Bichen Wu, Matthew Yu, Tao Xu, Kan Chen, Peter Vajda, and Joseph Gonzalez. FBNetV2: Differentiable Neural Architecture Search for Spatial and Channel Dimensions. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seartle, WA, USA, June 13-19, 2020, URL https:
pp. 1ww.:1109/ccPPP. 3tc/10..100//CPR426000 2000 01298
IEEE, 2020. doi: 10.1009/CVPR22200000000000 2020
Ruochen Wang, Xiangning Chen, Minhao Cheng, Xiaocheng Tang, and Cho-Jui Hsieh. RANK- NOSH: Efficient Predictor-Based Architecture Search via Non-Uniform Successive Halving.
CoRR, abs/2108.080 08019, 2021. httphttsss///xxiv..oo www.oo/////b./ooo.oaoo../...ee/ 2108. 08019
Wei Wen, Hanxiao Liu, Yiran Chen, Hai Helen Li, Gabriel Bender, and Pieter-Jan Kindermans. Neural Predictor for Neural Architecture Search. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (eds.), Computer Vision ECCV 2020 16th European Conference, Glasgow, UK. August 23-28, 2020, Proceedings. Part XXIX, volume 12374 of Lecture Notes in Computer Science, pp. 660-676. Springer, 2020. doi: https:
.3., URL
1 3-000- 39
Matthias Wess, Matvey Ivanov, Christoph Unger, Anvesh Nookala, Alexander Wendt, and Axel Jantsch. ANNETTE: Accurate Neural Network Execution Time Estimation With Stacked Models. IEEE Access, 9:35453556, 2021. ISSN 2169-3536. doi: 10. (009//ccess.222.34472222 URL
http://ddtt doi. org/ 101110//CCC ACCCSS.. 2020. 3047259
Colin White, Willie Neiswanger, and Yash Savani. BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Search. arXiv preprint arXiv: 1858, 2019.
Colin White, Arber Zela, Binxin Ru, Yang Liu, and Frank Hutter. How Powerful are Performance Predictors in Neural Architecture Search? CoRR, abs/2104.01177, 2021. URL https://
011777774400
Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang, Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer. FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search. In IEEE Conference on Computer Vision
and Pattern Recognition, CVPR 2019, Long Beach, CA, USA. June 20, 2019, pp. 10734-
10742. Computer Vision Foundation IEEE, 2019. doi: 10.1109/CVPR.AA 00T./PP..211..000000 URL
htt.t///peraccssstteeeeee Ha
twwcss..sthccetc.cc com/ comt/ntnntt...mm CVPR_ 21FR__119/htaat Wu_ _ FeNet_
Conettt__oveee__eeee Design Laaooveee_Desiig__ia_____. www.Diifffeeetibbll....
Cis.ralArchitettrree Serrhteeeccueeeeerchheeeee Sewrch_aarhae CV____CPP____ 2019 paper paper.html
Yuhui Xu, Lingxi Xie, Xiaopeng Zhang, Xin Chen, Bowen Shi, Qi Tian, and Hongkai Xiong-
Latency-Aware Differentiable Neural Architecture Search. CoRR, abs/2001.06399 2020. URL
https://arxivy wrg/ /twpg//bb///00... 201.200000..000
Antoine Yang, Pedro M Esperança, and Fabio M Carlucci. Nas evaluation is frustratingly hard.
arXiv preprint 1912. 12522, 2019.
Tien-Ju Yang, Andrew G. Howard, Bo Chen, Xiao Zhang, Alec Go, Mark Sandler, Vivienne Sze, and Hartwig Adam. NetAdapt: Platform-Awan Neural Network Adaptation for Mobile Ap- plications. In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss (eds.), Computer Vision ECCV 2018 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part X, volume 11214 of Lecture Notes in Computer Science, pp. 289-304 304. Springer, 2018. doi: 10. 10.11007978833330--1144.066 978-3-030-01249-66
18. URL battps://dae org
13

---

Under review as a conference paper at ICLR 2022
Shuochao Yao, Yiran Zhao, Huajie Shao, ShengZhong Liu, Dongxin Liu, Lu Su, and Tarek Ab- delzaher. FastDeeploT: Towards Understanding and Optimizing Neural Network Execution Time Mobile and Embedded Devices. In Proceedings of the 16th ACM Conference Embedded Networked Sensor Systems, pp. 278-291, 2018.
Chris Ying, Aaron Klein, Eric Christiansen, Esteban Real, Kevin Murphy, and Frank Hutter. NAS- Bench- 101: Towards Reproducible Neural Architecture Search. In Kamalika Chaudhuri and Rus- lan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learn- ing, ICMI. 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 7105-7114, PMLR, 2019. html
mlr. mLr.ppressv99/yynngg
Kaicheng Yu, René Ranftl, and Mathieu Salzmann. How to Train Your Super-Net: An Analysis of Training Heuristics in Weight-Sharing NAS. CoRR, abs/2003.04276, 2020. URL https:
//aaxiv.org/ab///2200 04276
Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. ShuffleNet: An Extremely 2018 IEEE Conference
Efficient Convolutional Neural Network for Mobile Devices. Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pp. 6848 6856. IEEE Computer Society, 2018. doi: 10.1 2018.00716. URL HtRLhttp:// opwnaccesppcccsss... twectheeecccc com/ content_nnntteccccc cvpr Chang_Shure Shuff An Extremely Setpreemel__yeRReeeee CVPR 2018 htmo
2018/htm1/
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoe Le. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 8697-8710, 2018. URL http://aax....rr///////10.....00
14

---

Under review as a conference paper at ICLR 2022
A BEST PRACTICES FOR NAS, CODE AND DATA
To improve the reproducibility and facilitate fair experimental comparisons, we follow the best- practices checklist (Lindauer & Hutter 2020):
Release Code for the Training Pipeline(s) you use. Our experiments are based on White et al.J (202I), who use NASLib (Ruchte et al. 2020) to compare 31 methods for accuracy prediction. Our NASLib fork, extending the framework for WW-NAS--Benche TransNAS- Bench, some performance predictors and the hypervolume simulations, is provided in the supplementary materials. We intend either make our fork available on GitHub or submit the changes upstream once this paper is accepted/published..
Use the Same Evaluation Protocol for the Methods Being Compared. Aside from the implementation of each predictor, all experiments use the same pipeline.
Validate The Results Several Times. We each predictor 50 times, with seeds {o. }9). The reductions in hypervolume are simulated 1000 times using different different subset of the data set, for each combination of (iteration, HW-NAS data set, noise on HW metric).
Control Confounding Factors. While all experiments used the same software libraries and hardware resources, they were run different machines speed up the evaluation. We found hardly any benefit in using GPU even for the network-based predictors, which is why every method only used two CPU cores. The OS is Ubuntu 18.04, notable software packages are Py PyTochh 1.9.0, numpy 1.19.5, scikit-learn 0.24.2, pybnn 0.0.5, ngboost 0.3.11, and xgboost 1.4.2
Report the Use of Hyperparameter Optimization. See Appendix C
In addition to the code in the supplementary materials, we also provide the experimental results as csv files. Running the predictors and hypervolume simulations takes some time, but the casy access the data of the finished experiments may prove useful for future research. Please see readme. md in the accompanying code zip file for instructions.
B ENCODINGS AND PREDICTORS
B. DATA ENCODINGS
Every architecture a A requires a unique representation, which depends on the used predictor. The common encoding types are:
Adjacency one-hot: Each architecture a is uniquely defined by the chosen candidate operation on every path. For example, each architecture in NAS- BAS-BENCH--01 consists of a repeated cell structure, which has five candidate operations on each the six paths. Therefore there are 56 = 15625 unique architectures, which each be referenced by sequence of operation-indices such [0 2 3 0)- Many predictors perform better if the sequence is presented as a one-hot encoding, which is this case [10000 01000 00100 00010 00001 10000).
Similarly, the path-encoding (used by BANANAS) is a one-hot representation over the used can- didate operation all possible paths. Since the connectivity within cells for HW-NAS-Bench and TransNAS --N----enhhh11- is fixed, it provides no more information than the adjacency one-hot encod- ing. If the connectivity can be adjusted more freely, as in the NAS-Bench-101 search space, the additional information may improve the fit.
The encodings for BONAS, GCN, and NAO each provide further information in addition to the Adjacency one-hot vector, most notably the adjacency-matrix. This {0, 1) (N+)) matrix lists describes which of the architecture paths (rows) serves as inputs for each other path (column), and also includes input/output.
15

---

Under review as a conference paper at ICLR 2022
B.2 PREDICTORS
We briefly describe the 18 predictor methods in our experiments. We adopt their implementations from the NASLib library (see Appendix [A], which we extend with Linear Regression, Ridge Re- gression, and Support Vector Machines from the scikit-learn package; and a simple Lookup Table implementation. Unless specified otherwise, the methods use the adjacency one-hot encoding.
BANANAS An ensemble of three MLP models with five to 20 layers, each using the path- encoding (White al. 2019).
Bayesian Linear Regression bayesian model that assumes (1) a linear dependency be- tween inputs and ouiputs, and (2) that the samples are normally distributed (Bishop) 2007).
BOHAMIANN A bayesian inference predictor using stochastic gradient Hamiltonian Monte Carlo (SGHMC) to sample from a bayesian neural network (Springenberg et al,
2016).
BONAS Bayesian Optimization for NAS (Shi et al. 2020) uses a GCN predictor within an outer loop of bayesian optimization, as a meta- task. The GCN requires encoding the adjacency matrix of each architecture.
Gaussian Process A simple model that assumes a joint Gaussian distribution underlying the training data (Rasmussen
2003).
GCN A Graph Convolutional Network that makes use of adjacency-matrix encoding of each architecture (Wen et al.
2020
Linear Regression A simple model that assumes an independent value/cost for each oper- ation/layer, which only need to be summed up. Unlike the Lookup Table model, uses a least-square fit on the training data.
Lookup Table The most simple and perhaps widely used model for differentiable archi- tecture selection. generally assumes a single baseline architecture (e.g. [001 001] in adjacency one-hot encoding), and a lookup matrix @u (ayers)) candidates) that contains the increases/reductions in the metric for each layer and candidate operation. The metric value of new architecture can be predicted with simple sum over the respective matrix entries and the baseline value. The model is obtained from measuring either each candidate operation isolation, or by computing the differences between the baseline architecture and specific variations (e-g- [010 001] or [100 001), to measure the first candidates). This model always requires 1++nnm layers) (num candidates neighbored architectures to fit. We detail the resulting correlation values for each used dataset in Appendix
LGBoost Light Gradient Boosting Machine (LightGBM or LGBoost, Ke et al. 2017 is lightweight brosted decision tree model.
MLP We use fully-connected Multi Layer Perceptrons in two size-categories.
NAO NAO (Luo et al.) 2018) uses encoder-decoder topology, which en-
codes/compresses an architecture to a continuous representation, and decodes again. This representation is further used make architecture predictions.
NGBoost Natural Gradient Boosting (NGBoost, Duan al. (2020) is gradient-boosted decision tree model that uses natural gradients to estimate uncertainty.
Ridge Regression Ridge Regression (Saunders et al. 1998) extends the Linear Regression lqaat-squares fit with a regularization term that serves as bias-variance tradeoff
Random Forests An ensemble of decision trees (Liaw et al. 2002).
Sparse Gaussian Process an approximation of Gaussian Processes that summarizes train- ing data (Candela & Rasmussen 2005).
Support Vector Machine model that maps inputs a high-dimensionall space, where training samples are used as support-vectors for bouudarress (Cortes & Vapnik 1995).
XGBoust eXtreme Gradient Boosting (XGBoost, Chen & Guestrin (2016) is a gradient- boosted decision tree model.
16

---

Under review as a conference paper at ICLR 2022
C HYPERPARAMETERS
We list our default and hyper-parameter sample ranges in Table 2 For comparability with White al. (2021), we only change the values of newly introduced parameterized predictors: Ridge Re- gression, Support Vector Machines, and small MLPs.
Model
Range/Choice
Log -eg-tranffrmm
Default
Num. Layers
[,, 25]
false
20
BANANAS
Layer width
[, 25]
fälse
20
Learning rate
[0.0001, 0.1]
true
0.001
Num. Layers
[16, 128)
true
64
BONAS
Batch size
32, 256]
true
128
Learning rate
[0.00001, 0.1]
true
0.0001
Num. Layers
200]
true 144
Batch size
[5,32]
true 7
GCN
Learning rate
0.000011, 0.1
true
0.0001
Weight decay
[0.00001,00 0.1]
true 0.0003
Num. leaves
[10, 100]
false 31
LGBoost
Learning rate
0..00. 0.1]
true 0.05
Feature fraction
00.1
fälse 0.9
Num. layers
[,5
false
3
Layer width
16, 128]
true 32
MI.P (small)
Learning rate
00.0001, 0.1]
true relu
0.001
Activation function
(relu, tanh, hardswish]
Num. layers
5,
false 20
MLP (huge)
Layer width
5, 25
false 20
Learning rate
[0.0001,
true 0.001
Num. layers
[16, 128)
true 64
NAO
Batch size
[32, 256]
true 100
Learning rate
[0.00001, 0.1]
true 0.001
Num. estimators
[128,
true 64
Learning
0.001. 0.1]
0.081
NGBoost
Max depth
[1,25]
false
Max features
[0.1,
false
0.79
Ridge Regression
Regularization a
[0.25, 2.5)
false
1.0
Num. estimators
[16, 128]
true
116
Random Forests
Max fcatures
0.11, 0.9
true
0.17
Min samples (leaf)
[0
false
2
Min samples (split)
[2,20]
true
2
Regularization C'
0.5, 1.5) (linear, poly, rbf, sigmoid)
false
1.0
Support Vector Machine
Kernel
rbf
Max depth
[1, 15)
false
6
Min child weight
L, 10]
fälse
1
XGBoost
Col sample (tree)
0,
false
Learning rate
00.001.055]
true
0.3
Col sample (level)
[.1]
false
Table 2: Hyper- parame-e ranges and default values of the configurable predictors
17

---

Under review as a conference paper at ICLR 2022
D NAS-BENCH-200 HW-NAS-BENCE CELL DESIGN
Figure 5: Basic with exactly one of five candidate operations (Zero, Skip, Convolution 1 Convolution 3x3, Average Pooling 3x3}.
HW-NAS cell design. Each of the six orange paths is finalized
E SELECTION OF DATASETS
Linear Regression
XGBoost
LUT
11
25
55
124
276
614
1356
3056
6748
15000
150000
ImageNet16- 120 raspié latency
0.324
0.205
0.206
0.676
0.705
0.715
0.715
0.723
0.728
0.729
0.757
0,443
100-i100 Intency
0.392
0.292
0.732
0.780
0.797
0.805
0.806
0,809
0.812
0812
0.877
0.484
difurddeek cfful0-edgeegu.b edgegpuJntnneyy
0.570
0.806
0.819
0.330
0.175
edgegpu energy
0882
0331
0.90
ImageNart 6-1 20ceyeriss mith, int.
0.369
0.293
0.748
0.805
0.817
0.827
0.825
0,832
0.843
0.846
0.970
0.861
cffa10--picet Jatency
0.780
0.797
0.813
0.878
cffee1--raspi Intency
0.393
0.315
0.740
0.787
0.799
0.805
0.807
0.810
0.813
0811
0.890
0.462
cife 100r-00--sspp4.. Jitency
0,393
0.308
0.744
0.786
0.801
0.807
0.800
0.800
0.814
0.814
0.888
0.445
ImageNet
Inteccy
00llllggge Intency
0.375
0.723
0810
0.821
820
0.531
0.81
0.94
0.191
cifer titett@egguuuoooeee
0.375
0.284
0.728
0.792
0.10
0.81
0.824
0.531
1300
0.922
0.183
ImageNet 6-120s 120-olgeguu
0.377
0.281
0.733
0.707
0.14
0.825
0.K25
0.526
0.834
0.833
0.926
0,280
ImageNNet Di-allgggppulttnnn
0.277
10-eyriis arith. int.
0.384
0826
0.835
0.854
0.54
0.69
0.826
cifee100-eyeriss wrthh int.
0.384
0.297
0.757
0.811
0.266
0.835
0.835
0.844
0.855
0356
0.971
0.830
ImageNNriin (6- 200ffpgn
0.443
0.494
0.504
0.966
0.947
0.951
D.948
0.951
0.952
0.952
0.983
0.965
ImageNet16- fppouuuunggyy
ImageNat 66 20-eyeriss hetenyy
0.457
0.937
0.944
0.954
0.954
0.955
0.954
0.94
0.952
0.989
affe 10eyer--eerris Jatency
0.461
0.943
0.959
0.959
0.960
0.960
0.959
0.960
0.900
0.060
0.958
0.995
cifar100 eeere-eerllleeeeeeee Jatency
0,462
0.946
0.963
0.963
0.963
0.963
0.963
0.963
0.961
0.963
0.962
0.998
sfferl0-gyerienmme unergy
ImageNatt 6-1 10eeyeris,emorgy
0.967
0.885
0.966
0.985
0.885
0.966
0.972
0.998
rifee000 eyerissu
0.457
0.967
0.965
0.985
0.985
0.985
0.985
0.996
0.986
0.986
0.976
0.998
difar1- offr10-fppp.mmee upproeerryy
0.458
0.973
0.987
0.987
0.987
0.987
0.9 0.987
0.987
0.987
0.987
0.986
0.999
fpga.cnergy
sibr1000 (00 (agx lntency
0.457
0.973
0.967
0.987
0.987
0.987
0.987
0.987
0.987
0.987
0.986
0.999
cifeel0-fppo_leee litenyy
0.457
0.973
0.967
0.987
0.987
0.987
0.967
0.967
0.987
0.987
0.986
0.999
Table 3: Kendall's Tau correlation for Linear Regression, XGBoost, and Lookup Table (LUT) on all HW-NAS-Bench datasets (rows), for different amounts of available training data (columns), tested on the remaining 625 samples. The Lookup Table model is tested on all 15625 architectures. We selected the five data sets at the top.
Repression
XGRooot
65
1585
0.410
(1.35
0.586
0.605
0.624
0.631
0.661
classuobject
0.518
0.780
0828
r00m layuu
0.275
0.271
0.527
0.653
0.721
0.753
0.768
0.780
0.789
0.789
0.896
0.685
class.scene
0.275
0.268
0.527
0.63
0.221
0.755
0.768
0.782
0.789
0.790
0.907
0.710
segmenb-manticc
0.282
0.259
0.545
0.684
0.346
0.780
0.798
0.809
0.816
0.871
0.726
Table 4: Kendall's Tau test correlation for Linear Regression and XGBoost on the five used TransNAS datasets (rows), for different amounts of available training data (columns), tested the remaining 256 samples. The Lookup Table model (LUT) is tested on all 3256 architectures.
HW-NAS-Bench: To select five datasets that are (1) non-linear and (2) different from one another, we first fit Linear Regression to every available dataset, with the results listed Table The bottom 12 datasets can be accurately fit with only 25 training samples, so they not very interesting
18

---

Under review as a conference paper at ICLR 2022
challenge. On these datasets, the Lookup Table model achieves exceptional performance. Since the networks for CIFARIO, CIFARI00 and ImageNet16- 120 only differ slightly, their measurements on the same device and metric (e.g. raspi4 latency) is very similar. To improve the generalizability of our results, we thus select datasets on different devices and metrics, which are listed at the top of As displayed in Figure 6 their data distributions are generally different.
Table 3
Since the latency measurements of the architectures is generally very sim- ilarly distributed (see Figure 77.. is not necessary to train the predictors on all of them. We select all data sets that provide the Test loss and inference time attributes for all architectures, resulting in exactly the five datasets listed in Section (the other two datasets contain more specific test losses).
TransNAA-BBnchh-il
maagetente 120 rusple, Sterny
cherio- stancy
1550
1200
50
180
12A
554
2000
1.50
400
750
400
500
Figure 6: How the data of each selected NEW-NAS-Bende dataset is distributed (not normalized).
1200
1200
1.05
100
400
200-
rar Jyyke
1.00
800
400
080
0.50
0.00
015
0.38
6.15
0.0
0.5
Figure 7: How the data of each selected ized). Since all architectures are measured for latency on the same hardware, the resulting datasets are much less diverse than the TI-NAA-Beend ones.
dataset is distributed (not normal-
19

---

Under review as a conference paper at ICLR 2022
F PREDICTOR FIT TIME
Average over TransNAS datasets
Average over HW-NAS datasets
3000
20000
Reg.
17500
Bayes, Lin. Reg
2500
Ridge Reg.
Time fit the predictor (aboalut))
Time fit the predictor (abosllur)
15000
NGBoost
2000
LGBoost
12500
Random Forests
Sparse GP
1500
10000
BOHAMIANN
SVM
7500
1000
GCN
BANANAS
MLP (lagg)
(small)
10a
10"
10
10a
10"
20
10
training
training
Figure 8: Fit time (in seconds) of predictors to data, depending on the training set size. By far the most expensive methods are setwork-based However, significant portion of this time spent on the optimization prior to the actual fitting.
G APPROXIMATING PREDICTOR MISTAKES
Predictar deviations
Predictar deviations
Predictar deviations
2.00
nomaa rt. std-0.309
1.75
normal fit sto-0 348 devistions
nernol ftt, std-0 oevattons
oevations
1.50
1.50
1.25
1.00
1.00
0.75
0.75
0.50
0.50
0.50
0.25
0.25
0.00
0.00
0.00
-2
2
-1
deviation of the predictions
deviation of the presictions
devintion the precictions
Figure 9: Further examples of predictor deviation distributions, as visualized in the center of Fig- ure 2 Left: Linear Regression on CIFAR1OM edgegpu, energy consumption. Center: Support Vector Machine Jigsaw. Right: small MLP ImageNet16-120, raspi4, latency.
Intuitively, the predictor deviation distributions (see Figures and 9 generally resemble normal distribution. However, most predictors:
(1) Have a notable peak, sometimes off-center (e.g. at x=0.2)
Have less density than normal distribution almost everywhere else
Have some outliers (e.g. 1.5) that extremely unlikely for normal distribution
We measured the p-vulue for different distributions on the first 100 test samples using T-Test, every time we evaluated a predictor. The average statistics can be found Table 5 Since a large number of empirical observations generally pushes the p-value to 0, this only serves compare them each other. We find that the outliers (3) appear often enough and are so unlikely to happen for a normal distribution, that even uniform distribution has higher statistical support. Consequentially, approximate the common predictor deviations by sampling from a mixed distribution that adresses (1) to (3).
20

---

Under review as a conference paper at ICLR 2022
p-value
normal
0.028
cauchy
0.030
lognorm
0.028
0.028
uniform 0.037
Table 5: P-values of different distributions, trying to fit the distribution of all predictor mistakes according to a t-test. Larger values are better, but comparing many empirically sampled points with true density function tends to push the p-values to 0.
This mixed distribution consists of two Normal distributions (N,, I, N22 and one Uniform distribution (U), from which we sample with 72.5%, 26.5% and 1% respectively. For some constant e:
We uniformly sample shift e from [0, 2 v). that is used to push the centers of N₁ and N₂ to x > 0 and x < O respectively.
We sample each value from [i v), N2( (2(-c, v), and (-5115 15.v) randomly, with the weighting given above.
We normalize (subtract mean, divide by standard deviation) sampled distribution and then scale to the desired standard deviation.
The predictors produce non-smooth distributions. We simulate that by sampling 15 times fewer values needed, and repeat them as often.
The code for the simulation is also provided (see Appendix A As seen in Figure 10 the resulting simulated deviation distributions generally resemble a common predictor pattern. We do not account for differences in predictors, training set sizes or more, since that may become too specific and over- engineered.
Appendix y visualizes simulation sanity checks. We find that the simulation is slightly pessimistic and simplified, but resembles the results of actual predictors.
Simulated predictor deviations
Simulated predictor deviations
Simulated predictor deviations
nermal ft. 550=0.500
nermal ft., 2d-0.500 mikes diz. gendratos with std-0.5
nermal fir, atde 050000
mixed dst. generated with 505-0.5
1.2
maxes dist. ganerated with sto-0.5
dendtty
0.8
0.6
0.4
0.2
0.2
0.0
devistion of the simulatted predictions
deviation of the simmlaltd predictions
deviallnt of the situllsted predictions
Figure 10: The sampled values of gaussian+uniform fit the measured predictor mistakes better than a single distribution, as they are roughly normally distributed, but include outliers.
21

---

Under review as a conference paper at ICLR 2022
H
MEASURING SIMULATED MISTAKES
0.35
0.5
0.33
Dru pardo hart preticeed pareoo forrt Wlaretheeiree selocio andhoddrrr
0.30
025
-15
-10
-2.5
NOTTAAROO mmuoo Mo1.-220002200000000000 lalonic
Similar to Figure 3 When the discovered Pareto set is considerably worse than the true Pareto set, it is possible for the Mean Reduction of Accuracy of the Parcto subset (MRApareta to be worse than the average over all architectures (MRA,u). This naturally happens more frequently for predictors with high sampling std. and low KT correlation. Consequentially. the difference between aRA and MRApareto is wider for better predictors (see Figure 4 Additionally, all of the selected non-Pareto-front members are clustered other. This emphasizes the limitations of just considering drops in accuracy, as the hardware metric aspect ignored. In this case, the rredictor-guide selection failed find low-latency solution. In this regard, hypervolume is better but less intuitive metric.
Figure 11:
high-latency area and redundant with each
true actually selected architacture
pareto hypenolame rederence point
60
10%
A4
54-
52-
accurccy diffrence
1WV metric diference
46
reforance point
64
18
22
24
26
30
18
28
30
32
handware metric
hardware mebric
Figure 12: Examples explain measurement methods.
Left: The distance of each selected candidate architecture C1 to the true Pareto front is measured, for accuracy and the hardware metric. C1 dominated by A2, A3, and A4 of the true Pareto set. A2 has slightly higher accuracy than Cl while being much better the hardware metric, e.g. latency. A4 has a slightly better hardware metric value, but much higher accuracy. Given several candidate architectures, their differences averaged.
Right: We compute the reference point for the hypervolume (for two objectives: area under a Pareto front) by multiplying the highest hardware metric value from the true Pareto front with 1.1, and accuracy O. While we are consistent throughout all experiments, this choice is arbitrary, as there obviously correct choice for the reference point. If the hypervolume of supposed Pareto front is computed, the reference point of the true Pareto front is reused. Thus, choosing inferior architectures will always reduce the hypervolume. We arbitrarily chose the multiplier of m 1.1 as middle ground between making the rightmost point of the Pareto front irrelevant (m = 1.0) and overemphasizing it (m >> 1.0).
22

---

Under review as a conference paper at ICLR 2022
1
SIMULATION SANITY CHECK
1.-
/mageNel16-122--aa aritteeeissssssittiiiiiiiii
www.taameeicciieeeeee intersity
ImageNet16-120-ae Jatencyy
dffrt0-edgeopuulleeee
cifar100-piieei8 lalency
mean
0.2
<lm-0.75, S0C+ 3.Cm-0.80 4.85, PCC+0.77 77
1.0 0.0/1.00 0.4/0.78 0.8/0.62 1.0/0.5;
0.0
0.2
Std. of prodiction desiattion
0.4
Sta. of prediction deviations Kendall's Tau
Figure 13: Standard deviation over the predictor deviations (x axis) and Kendall's Tau correlation (y axis), for the trained predictors on HW-NAS-Bend (left) and in simulation (right). The simulated predictor inaccuracies are slightly pessimistic (low KT), but still match the true values.
23

---

Under review as a conference paper at ICLR 2022
remalte, 0,44
1.5
0.5
030
015
0.00
All candidates
der as the preditiona
candidate occurrences in the architecture
candidate
not at all
exactly once
exactly twice
2.5
2.00
1.0
ceder
Souationn
destriine
3.53
2.00
6:75
2.0
02
8:25-
Zero
-15
-2.0
ador
0O
65
1.0
1.
1.5
-1.0
deviadin pradctions
-05
00
35
10
15
-1
deviaton prdd
144
comPaTTe, SEOIO 502
2.75
norna T: ande2
normal Ft. 012-0.612 dediciin
12-
Besket
3.53
2.0
10-
3.25
3.5
2.00
2.75
0:50
5:23
0.00
Skip
olar the protition
desidiin
Jevidton
4.00
2.0
ceidet
3.35
Stvandion
de-tatione
3.00
2:73
055
0:50
0.35
0.00
Conv 1x1
-20
-05
adar predittion
05
1.
dexiabion prodetions
-.0
-0.
devidtonn
200
100
2.5
Fonralt, 240
2.00
normaal R. 1010--442
1.0
tormell ai0-0.505
Reddet
2.73
1.2
Sestvtane
5.25
2.00
0.75
3.6
6:35
12-
0.00
3.0
Conv3x3
Jovatoo
devidian
2.00
4.00
130
125-
1.00
2.00
2.75
030
250
025
Pool
prodiciot
-2.0
-0.3
C11 Us prodicoTT
2.0
25
10
15
-33
-1.
doriation prod
-0.3
500
0.
20
35
Table 6: How a trained XGB predictor deviates from the ground-truth values for different architec- ture subsets, akin to Figure While they not exactly the same, they still resemble the distribution over the entire test set (top plot, 625 samples). One noteworthy exception is when no Conv3x3 op- crations used at all, in which case the standard deviation is considerably smaller.
24


Reviews:
[{"review_id": "fbbGi7jpeXW", "reviewer": "Reviewer_GUHu", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n- This is an important problem. Although there have been experimental studies on regular performance predictors, not much was known about hardware metric predictors until this paper, which is a more realistic setting.\n- The analysis is good. For example, Fig 1 (right) and Fig 2 (right) give easily visualizable data by centering on predictor-average. Also, they focused on analyzing only the most challenging prediction tasks, which seems beneficial.\n- The authors have thorough appendices where they release all of their code, raw results, and methodologies. So reproducibility and ease of follow-up is high.\n- It is interesting that in Section 6, the authors simulate the predictor (now there is a prediction of a prediction) to speed up results. However, I have some follow-up questions about this in “weaknesses”.\n\nWeaknesses\n- The authors state a few interesting insights at the end of Section 4, 5, 6. For example, MLP works great on HW-NAS-Bench and TransNAS-Bench-101, and the widely used lookup table does not perform nearly as well. Although experimental survey papers are very useful, the best ones follow up on something that is surprising, interesting, or insight-provoking. For example, the authors could go a step further and look into why MLP performed so well, or, find some other type of insight in their study that leads to new directions.\n\nI also have a few smaller nitpicks and questions about the experiments:\n - Section 6 is nice, but don’t a lot of NAS methods use a predictor-guided search such as BO, where the predictor is constantly being updated (even for hardware aware NAS algorithms)? So it would also be interesting to see how the predictors compare within NAS.\n - Section 6 is hard to follow. I don’t fully understand how the predictors are simulated even when reading the figures and Appendix G. It seems like the authors are focusing on approximating the distribution of mistakes. But then do you draw from the distribution on each prediction? What if the mistakes are one-sided based on properties of the architecture? Maybe I am misunderstanding.\n - In Section 6, it sounds like the authors estimated the hardware metrics but “All accuracy values remain accurate.” In the context of real NAS, isn’t this unrealistic because hardware metrics are much easier to get than the accuracies? Why not also use a predictor for the accuracy, and then compute the error of the Pareto-front after that? Or maybe I am misunderstanding?\n - When the authors explained in Section 4 that they only selected five out of 27 datasets, I thought this was totally arbitrary, especially when looking that they are imbalanced (e.g., CIFAR 10 only used once, latency used three times while energy and intensity are used once). But then eventually I got to Appendix E and realized that it is a principled selection, and some of the 27 are trivial and not worth analyzing. So I suggest that the authors stress this methodology more in Section 4. Also, it could still be useful to add some analysis on all 27, in addition to keeping the experiments that are already there. Or, some aggregate analysis about which metrics are hardest and easiest, which edge devices, which image datasets (e.g., like a summary of Table 2).\n- Also on page 6, there are a few statements that the authors should have citations for: \"Latency is generally the most crucial hardware metric\" and \"the macro-level search space is more applicable to most real-world problems than a micro-level search space.\"", "summary_of_the_paper": "In neural architecture search (NAS), performance predictors are an important tool, because predicting attributes such as accuracy can save costly measurements. Hardware metric predictors predict metrics such as latency in addition to accuracy. This paper gives an empirical study of 18 different performance predictors on NAS-Bench-201 and TransNAS-Bench-101, evaluating the rank correlation of the predictors on different datasets and predicted metrics. Then they evaluate how the inaccuracy of the predictors affect the predicted Pareto-front of architectures, compared to the true Pareto front. The authors run this experiment by simulating the errors in predictors. The authors find that MLP models perform the best on average across all settings they tried, and leads to an acceptable predicted Pareto-front.", "summary_of_the_review": "Overall, this paper is important and useful for the community, and has pretty good and thorough analysis. However, it could have more interesting insights, and there are some questions about Section 6. Overall, I currently will give a weak reject and look forward to the authors’ response.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "K6j2Uangp39", "reviewer": "Reviewer_bHy6", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strength:**\n1. The paper is well-written with clear logical flow and easy to follow.\n2. The evaluation across different predictors and tasks is solid and the codes are provided.\n3. The simple but effective insights can benefit the NAS community.\n\n**Weakness:**\n1. The major concern of this paper is the technical novelty since no new technique is proposed and the provided insights are relatively intuitive. In particular, a less accurate but fast predictor can intuitively lead to better results if it can see more architectures, where the measurement cost is assumed to be the bottleneck. More novel insights are expected.\n2. The averaged correlation in Fig. 1 across different devices (and tasks) may not be the only metric for evaluating the predictors, i.e., the device-specific correlation is also important. For example, as demonstrated in HW-NAS-Bench, for some search spaces (i.e., a sequential space like FBNet space) and many devices, the operation-wise hardware performances are additive, indicating a lookup table will be accurate enough as a predictor. For most deployment scenarios with determined types of devices, such device-specific discussions could strength the contributions of this paper and provide more detailed insights for the community.\n3. The discussions about the influences of the predictors to the architecture selection should be NAS-algorithm-specific while such discussions are missing in the paper. For example, for reinforcement learning based NAS methods, the hardware measurement is independent of the subnetwork training, while the hardware performance will participate in the training process of FBNet as a regularization term, where an inaccurate hardware performance predictor will more directly affect the results after convergence. Such NAS-algorithm-specific discussions may also guide the NAS algorithm selection given affordable hardware performance predictors in addition to predictor selections.\n\n\n\n", "summary_of_the_paper": "This paper systematically evaluates a wide range of hardware performance predictors across different networks/devices/tasks and analyzes the influence of such predictors to the architecture selection process. It provides insights for the community about how to select a proper hardware performance predictor in different scenarios.", "summary_of_the_review": "Given the limited novelty and lacked discussions elaborated in the weakness part, I tend to deem this paper marginally below the acceptance threshold. I'm willing to adjust my scores if the concerns are properly addressed.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "FvwvxtYgcC", "reviewer": "Reviewer_bBS4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n- This work did a very comprehensive survey across different kinds of the performance predictors and across two existing hardware performance dataset. The network architectures considered are also well rounded.\n- This work gives reasonable explanations/insights for most of the phenomena in the conducted experiments.\n\nWeakness:\n- Novelty is limited\n- The insights are shallow and obvious\n- Unclear on the aspects of how to counter the imperfect predictions on guiding the network architecture selection\n\n", "summary_of_the_paper": "This work provides a compressive analysis across multiple (18) different hardware performance predictors by \n- (1) collecting their performance under different amounts of training data and different input network structures and showing each prediction method’s advantageous/disadvantageous scenarios\n- (2) analyzing the  prediction accuracy's influence on selecting subsequent hardware architecture and giving the insights on how to pick/design hardware performance predictors for the NAS.\n\nBased on the observation drawn, MLP ones are found to be the most promising predictors in terms of the accuracy with limited training samples, while the Lookup Table model serves as a very cheap and straightforward guidance. In terms of the architecture selection guided by the model-based predictor, the work conjectures by verifying some of the selected network  structures’  hardware performance and increase the number of explored network structures can lead to better pareto front of the selected network architecture", "summary_of_the_review": "- **Novelty is limited**: Most of this work is simply using the existing hardware performance predictors and running them to collect results on the existing hardware dataset. Too much focus on introducing the setup and configurations of these hardware performance predictors and dataset, and simply numerating the results from the conducted experiment, while the proposed insights are overshadowed. \n\n- **The insights are shallow and obvious**: Even though the paper presents multiple insights from the experiments, most of them are rather shallow and obvious. For instance, the paper stressed that the model-based can benefit more from more training data as compared to the lookup table based predictors, which is rather obvious as model based is data driven while lookup table is deterministic. However, some specific necessary and useful  insights are missing. For instance for page 5, most of the texts are devoted to reporting which methods perform better and which methods scale better  with more data. The underlying reasons or hypotheses are not provided.\n\n- **Unclear on the aspects of how to counter the imperfect predictions on guiding the network architecture selection**: One of the most interesting aspects about this work is that it proposes imperfect predictions from the hardware predictors can also lead to decent selected architectures. However, it is rather unclear how to achieve this and why. For instance, in page 7, the authors mention “verifying the hardware metric predictions of the 13 architectures”; which 13 architectures? It was never mentioned elsewhere. I really hope the author can stress this part more.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Butg48DVIti", "reviewer": "Reviewer_7VPE", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "While I think that the motivation to have a large-scale study of hardware metric predictors similar as White et al. do for performance predictors is a reasonable one, however I think that this paper needs to improve its presentation and clarity. There is a lot of text (e.g. in Sections 3, 4 and 5) which might be compressed or moved to the appendix, adding space for additional experiments to be conducted. Considering that the novelty and the results presented in the paper are not surprising, I think a more thorough empirical evaluation is necessary.\n\nThese are some of my questions and recommendations:\n\n- I am quite confused by the experiments and results in Section 6. In Figure 4, how do you obtain the simulated pareto front? Do you evaluate all the architectures in the space using the predictor (for both accuracy and hardware metric)? \n- What are the \"selected architectures\"? Are they the architectures in the \"simulated pareto front\" evaluated on the true HW-NAS-Bench?\n- The main purpose of tabular NAS benchmarks, including HW-NAS-Bench that is for hardware metrics, is to provide a fast way to prototype and evaluate NAS algorithms, which will ultimately be evaluated on real world benchmarks. Therefore, I think, for the case at hand, it would be useful to evaluate the trained predictors on a setting where they are used in a multi-objective NAS algorithm together with the performance predictors.\n- I found many of the plots to be redundant, e.g. the right-hand side plots in Fig 1 and 2. I think the authors can gain some space by moving them to the appendix and adding additional experimental results in the main paper.", "summary_of_the_paper": "This paper provides a large-scale study of hardware metric predictors on 2 recent tabular benchmarks: Trans-NAS-Bench-101 and HW-NAS-Bench. The authors compare the predictors in terms of rank correlation and their ability to simulate the true pareto front and demonstrate that model-based predictors outperform Lookup Tables with very few training data points and that simple MLPs can outperform the more specialized models such as GNNs.", "summary_of_the_review": "I think that this paper is useful since it evaluates model-based predictors on the same open-source framework, enabling a fair comparison of different predictors. However, I think that the experimental evaluation conducted by the authors is not enough in order for me to give an acceptance score. There is also quite some work to be done in order to improve the structuring and clarity of the text in my opinion.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper comprehensively evaluated 18 different performance predictors on ten combinations of metrics, devices, network types, and training tasks for NAS. While evaluating and comparing different prediction models is not itself novel, the authors provided many insights that are potentially interesting to future NAS developments. 

Reviewer reactions to this paper are rather mediocre and lukewarm. It is in general consensus that this work gives a good empirical analysis on hardware metric predictors for NAS, but the novelty is low and it is perhaps a bit incremental (e.g., nothing "shockingly new" was revealed, and observations are mostly "as expected"). Despite the authors improving the paper during rebuttal with new plots/tables, there remain to be unaddressed comments, e.g., adding experiments that run BO / evolution / etc with different hardware predictors and comparing the quality of the Pareto front. Those missed points were also raised in the private discussion. 

After personally reading this paper, AC sides with most reviewers that this paper lacks true novelty nor technical excitement. While the empirical study is valuable, it perhaps suits venues other than ICLR, e.g., the NeurIPS benchmark track.
Decision: Reject