What to expect of hardware metric predictors in NAS

1 INTRODUCTION . Modern neural network architectures are designed not only considering their primary objective , such as accuracy . While existing architectures can be scaled down to work with the limited available memory and computational power of , e.g. , mobile phones , they are significantly outperformed by specifically designed architectures ( Howard et al. , 2017 ; Sandler et al. , 2018 ; Zhang et al. , 2018 ; Ma et al. , 2018 ) . Standard hardware metrics include memory usage , number of model parameters , Multiply-Accumulate operations , energy consumption , latency , and more ; each of which may be limited by the hardware platform or network task . As the range of tasks and target platforms grows , specialized architectures and the methods to find them efficiently are gaining importance . The automated design and discovery of specialized architectures is the main intent of Neural Architecture Search ( NAS ) . This recent field of study repeatedly broke state of the art records ( Zoph et al. , 2018 ; Real et al. , 2018 ; Cai et al. , 2019 ; Tan & Le , 2019 ; Chu et al. , 2019a ; Hu et al. , 2020 ) while aiming to reduce the researchers ’ involvement with this tedious and time-consuming process to a minimum . As the performance of each considered architecture needs to be evaluated , the hardware metrics need to be either measured live or guessed by a trained prediction model . While measuring live has the advantage of not suffering from inaccurate predictions , the corresponding hardware needs to be available during the search process . Measuring on-demand may also significantly slow down the search process and necessitates further measurements for each new architecture search . On the other hand , a prediction model abstracts the hardware from the search code and simplifies changes to the optimization targets , such as metrics or devices . The data set to train the predictor also has to be collected only once so that a trained predictor then works in the absence of the hardware it is predicting for , e.g. , in a cloud environment . Furthermore , a differentiable predictor can be used for gradient-based architecture optimization of typically non-differentiable metrics ( Cai et al. , 2019 ; Xu et al. , 2020 ; Nayman et al. , 2021 ) . While the many advantages make predictors a popular choice of hardware-aware NAS ( e.g . Xu et al . ( 2020 ) ; Wu et al . ( 2019 ) ; Wan et al . ( 2020 ) ; Dai et al . ( 2020 ) ; Nayman et al . ( 2021 ) ) , there are no guidelines on which predictors perform best , how many training samples are required , or what happens when a predictor is inaccurate . This work investigates the above points . As a first contribution , we conduct large-scale experiments on ten hardware-metric datasets chosen from HWNAS-Bench ( Li et al. , 2021a ) and TransNAS-Bench-101 ( Duan et al. , 2021 ) . We explore how powerful the different predictors are when using different amounts of training data and whether these results generalize across different network architecture types . As a second contribution , we extensively simulate the subsequent architecture selection to investigate the impact of inaccurate predictors . Our results demonstrate the effectiveness of network-based prediction models ; provide insights into predictor mistakes and what to expect from them . To facilitate reproducibility and further research , our experimental results and code are made available in Appendix A . 2 RELATED WORK . NAS Benchmarks : As the search spaces of NAS methods often differ from one another and lack extensive studies , the difficulty of fair comparisons and reproducibility have become a major concern ( Yang et al. , 2019 ; Li & Talwalkar , 2020 ) . To alleviate this problem , researchers have exhaustively evaluated search spaces of several thousand architectures to create benchmarks ( Ying et al. , 2019 ; Dong & Yang , 2020 ; Dong et al. , 2020 ; Siems et al. , 2020 ) , containing detailed statistics for each architecture . TransNAS-Bench-101 ( Duan et al. , 2021 ) evaluates several thousand architectures across seven diverse tasks and finds that the best task-specific architectures may vary significantly . The popular NAS-BENCH 201 benchmark ( Dong & Yang , 2020 ) has been further extended with ten different hardware metrics for all 15625 architectures on each of the three data sets CIFAR10 , CIFAR100 ( Krizhevsky et al. , 2009 ) and ImageNet16-120 ( Chrabaszcz et al. , 2017 ) . Major findings of this HW-NAS Bench ( Li et al. , 2021a ) include that FLOPs and the number of parameters are a poor approximation for other metrics such as latency . Many existing NAS methods use such inadequate substitutes for their simplicity and would benefit from their replacement with better prediction models . Li et al . also find that hardware-specific costs do not correlate well across hardware platforms . While accounting for each device ’ s characteristics improves the NAS results , it is also expensive . Predictors can reduce costs by requiring fewer measurements and shorter query times . 1 . Predictors in NAS : Aside from real-time measurements ( Tan et al. , 2019 ; Yang et al. , 2018 ) , hardware metric estimation in NAS is commonly performed via Lookup Table ( Wu et al. , 2019 ) , Analytical Estimation or a Prediction Model ( Dai et al. , 2020 ; Xu et al. , 2020 ) . While an operationand layer-wise Lookup Table can accurately estimate hardware-agnostic metrics , such as FLOPs or the number of parameters ( Cai et al. , 2019 ; Guo et al. , 2020 ; Chu et al. , 2019a ) , they may be suboptimal for device-dependent metrics . Latency and energy consumption have non-obvious factors that depend on hardware specifics such as memory , cache usage , the ability to parallelize each operation , and an interplay between different network operations . Such details can be captured with neural networks ( Dai et al. , 2020 ; Mendoza & Wang , 2020 ; Ponomarev et al. , 2020 ; Xu et al. , 2020 ) or other specialized models ( Yao et al. , 2018 ; Wess et al. , 2021 ) . Of particular interest is the correct prediction of the model loss or accuracy , possibly reducing the architecture search time by orders of magnitude ( Mellor et al. , 2020 ; Wang et al. , 2021 ; Li et al. , 2021b ) . In addition to common predictors such as Linear Regression , Random Forests ( Liaw et al. , 2002 ) or Gaussian Processes ( Rasmussen , 2003 ) ; specialized techniques may exploit training curve extrapolation , network weight sharing or gradient information . Our experiments follow the recent large-scale study of White et al . ( 2021 ) , who compare 31 diverse accuracy prediction methods based on initialization and query time , using three NAS benchmarks . 3 PREDICTING HARDWARE METRICS . Our methods follow the large-scale study of White et al . ( 2021 ) , who compared a total of 31 accuracy prediction methods . The differences between accuracy and hardware-metric prediction , our selection of predictors , and the general training pipeline are described in this section . In our experiments on HW-NAS-Bench and TransNAS-Bench-101 , described in Section 4 , we then compare these predictors across different training set sizes . 1For further reading , we recommend a recent survey on hardware-aware NAS ( Benmeziane et al. , 2021 ) Differences to accuracy predictors : There are fundamental differences when predicting hardware metrics and the accuracy of network topologies . The most essential is the cost to obtain a helpful predictor , which may vary widely for accuracy prediction methods . While determining the test accuracy requires the costly and lengthy training of networks , measuring hardware metrics does not necessitate any network training . Consequentially , specialized accuracy-estimation methods that rely on trained networks , loss history , learning curve extrapolation , or early stopping do not apply to hardware metrics . Furthermore , so-called zero-cost proxies that predict metrics from the gradients of a single batch are dependant on the network topology but not on the hardware the network is placed on . Therefore , the dominant hardware-metric predictor family is model-based . Since all relevant predictors are model-based , they can be compared by their training set size . This simplifies the initialization time of a predictor as the number of prior measured architectures on which they are trained . In stark contrast , some accuracy predictors do not need any training data , while others require several partially or fully trained networks . Since an untrained network and a few batches suffice to measure a hardware-metric , the collection of such a training set is comparably inexpensive . Additionally , hardware predictors are generally used supplementary to a one-shot network optimized for loss or accuracy . Depending on the NAS method , a fully differentiable predictor is required in order to guide the gradient-based architecture selection . Typical choices are Lookup Tables ( Cai et al. , 2019 ; Nayman et al. , 2021 ) and neural networks ( Xu et al. , 2020 ) . Model-based predictors : The goal of a predictor fp ( a ) is to accurately approximate the function f ( a ) , which may be , e.g. , the latency of an architecture a from the search space A . A model-based predictor is trained via supervised learning on a setDtrain of datapoints ( a , f ( a ) ) , after which it can be inexpensively queried for estimates on further architectures . The collection of the dataset and the duration of the training are referred to as initialization time and training time respectively . The quality of such a trained predictor is generally determined by the ( ranking ) correlation between measurements { f ( a ) |a ∈ Atest } and predictions { fp ( a ) |a ∈ Atest } on the unseen architectures Atest ⊂ A . Common correlation metric choices are Pearson ( PCC ) , Spearman ( SCC ) and Kendall ’ s Tau ( KT ) ( Chu et al. , 2019b ; Yu et al. , 2020 ; Siems et al. , 2020 ) . Our experiments include 18 model-based predictors from different families : Linear Regression , Ridge Regression ( Saunders et al. , 1998 ) , Bayesian Linear Regression ( Bishop , 2007 ) , Support Vector Machines ( Cortes & Vapnik , 1995 ) , Gaussian Process ( Rasmussen , 2003 ) , Sparse Gaussian Process ( Candela & Rasmussen , 2005 ) , Random Forests ( Liaw et al. , 2002 ) , XGBoost ( Chen & Guestrin , 2016 ) , NGBoost ( Duan et al. , 2020 ) , LGBoost ( Ke et al. , 2017 ) , BOHAMIANN ( Springenberg et al. , 2016 ) , BANANAS ( White et al. , 2019 ) , BONAS ( Shi et al. , 2020 ) , GCN ( Wen et al. , 2020 ) , small and large Multi-Layer-Perceptrons ( MLP ) , NAO ( Luo et al. , 2018 ) , and a layeroperation-wise Lookup Table model . We provide further descriptions and implementation details in Appendix B. Hyper-parameter tuning : The default hyperparameters of the used predictors vary significantly in their levels of hyper-parameter tuning , especially in the context of NAS . Additionally , some predictors may internally make use of cross-validation , while others do not . Following White et al . ( 2021 ) , we attempt to level the playing field by running a cross-validation random-search over hyperparameters each time a predictor is fit to data . Each search is limited to 5000 iterations and a total run time of 15 minutes and naturally excludes any test data . The predictor-specific parameter details are given in Appendix C. Training pipeline To make a reliable comparison , we use the NASLib library ( Ruchte et al . ( 2020 ) , see Appendix A ) . We fit each predictor on each dataset and training size 50 times , using seeds { 0 , ... , 49 } . Some predictors internally normalize the training values ( subtract mean , divide by standard deviation ) . We choose to explicitly do this for all predictors and datasets , which reduces the dependency of hyper-parameters ( e.g . learning rate ) on the dataset and allows us to analyze and compare the prediction errors across datasets more effectively .

Summary: In neural architecture search (NAS), performance predictors are an important tool, because predicting attributes such as accuracy can save costly measurements. Hardware metric predictors predict metrics such as latency in addition to accuracy. This paper gives an empirical study of 18 different performance predictors on NAS-Bench-201 and TransNAS-Bench-101, evaluating the rank correlation of the predictors on different datasets and predicted metrics. Then they evaluate how the inaccuracy of the predictors affect the predicted Pareto-front of architectures, compared to the true Pareto front. The authors run this experiment by simulating the errors in predictors. The authors find that MLP models perform the best on average across all settings they tried, and leads to an acceptable predicted Pareto-front.