HW-NAS-Bench: Hardware-Aware Neural Architecture Search Benchmark

1 INTRODUCTION . The recent performance breakthroughs of deep neural networks ( DNNs ) have attracted an explosion of research in designing efficient DNNs , aiming to bring powerful yet power-hungry DNNs into more resource-constrained daily life devices for enabling various DNN-powered intelligent functions ( Ross , 2020 ; Liu et al. , 2018b ; Shen et al. , 2020 ; You et al. , 2020a ) . Among them , HardWareaware Neural Architecture Search ( HW-NAS ) has emerged as one of the most promising techniques as it can automate the process of designing optimal DNN structures for the target applications , each of which often adopts a different hardware device and requires a different hardware-cost metric ( e.g. , prioritizes latency or energy ) . For example , HW-NAS in ( Wu et al. , 2019 ) develops a differentiable neural architecture search ( DNAS ) framework and discovers state-of-the-art ( SOTA ) DNNs balancing both accuracy and hardware efficiency , by incorporating a loss consisting of both the cross-entropy loss that leads to better accuracy and the latency loss that penalizes the network ’ s latency on a target device . Despite the promising performance achieved by SOTA HW-NAS , there exist paramount challenges that limit the development of HW-NAS innovations . First , HW-NAS requires the collection of hardware efficiency data corresponding to ( all ) the networks in the search space . To do so , current practice either pre-collects these data to construct a hardware-cost look-up table or adopts device-specific hardware-cost estimators/models , both of which can be time-consuming to obtain and impose a barrier-to-entry to non-hardware experts . This is because it requires knowledge about device-specific compilation and properly setting up the hardware measurement pipeline to collect hardware-cost data . Second , similar to generic NAS , it can be notoriously difficult to benchmark HW-NAS algorithms due to the required significant computational resources and the differences in their ( 1 ) hardware devices , which are specific for HW-NAS , ( 2 ) adopted search spaces , and ( 3 ) hyperparameters . Such a difficulty is even higher for HW-NAS considering the numerous choices of hardware devices , each of which can favor very different network structures even under the same target hardware efficiency , as discussed in ( Chu et al. , 2020 ) . While the number of floating-point operations ( FLOPs ) has been commonly used to estimate the hardware-cost , many works have pointed out that DNNs with fewer FLOPs are not necessarily faster or more efficient ( Wu et al. , 2019 ; 2018 ; Wang et al. , 2019b ) . For example , NasNet-A ( Zoph et al. , 2018 ) has a comparable complexity in terms of FLOPs as MobileNetV1 ( Howard et al. , 2017 ) , yet can have a larger latency than the latter due to NasNet-A ( Zoph et al. , 2018 ) ’ s adopted hardware-unfriendly structure . It is thus imperative to address the aforementioned challenges in order to make HW-NAS more accessible and reproducible to unfold HW-NAS ’ s full potential . Note that although pioneering NAS benchmark datasets ( Ying et al. , 2019 ; Dong & Yang , 2020 ; Klyuchnikov et al. , 2020 ; Siems et al. , 2020 ; Dong et al. , 2020 ) have made a significant step towards providing a unified benchmark dataset for generic NAS works , all of them either merely provide the latency on server-level GPUs ( e.g. , GTX 1080Ti ) or do not provide any hardware-cost data on real hardware , limiting their applicability to HW-NAS ( Wu et al. , 2019 ; Wan et al. , 2020 ; Cai et al. , 2018 ) which primarily targets commercial edge devices , FPGA , and ASIC . To this end , as shown in Figure 1 , we develop HW-NAS-Bench and make the following contributions in this paper : • We have developed HW-NAS-Bench , the first public dataset for HW-NAS research aiming to ( 1 ) democratize HW-NAS research to non-hardware experts and ( 2 ) facilitate a unified benchmark for HW-NAS to make HW-NAS research more reproducible and accessible , covering two SOTA NAS search spaces including NAS-Bench-201 and FBNet , with the former being one of the most popular NAS search spaces and the latter having been shown to be one of the most hardware friendly NAS search spaces . • We provide hardware-cost data collection pipelines for six commonly used hardware devices that fall into three categories ( i.e. , commercial edge devices , FPGA , and ASIC ) , in addition to the measured/estimated hardware-cost ( e.g. , energy cost and latency ) on these devices for all the networks in the search spaces of both NAS-Bench-201 and FBNet . • We conduct comprehensive analysis of the collected data in HW-NAS-Bench , such as studying the correlation between the collected hardware-cost and accuracy-cost data of all the networks on the six hardware devices , which provides insights to not only HW-NAS researchers but also DNN accelerator designers . Other researchers can extract useful insights from HW-NAS-Bench that have not been discussed in this work . • We demonstrate exemplary user cases to show : ( 1 ) how HW-NAS-Bench can be easily used by non-hardware experts to develop HW-NAS solutions by simply querying the collected data in our HW-NAS-Bench and ( 2 ) dedicated device-specific HW-NAS can indeed lead to optimal accuracy-cost trade-offs , demonstrating the great necessity of HW-NAS benchmarks like our proposed HW-NAS-Bench . 2 RELATED WORKS . 2.1 HARDWARE-AWARE NEURAL ARCHITECTURE SEARCH . Driven by the growing demand for efficient DNN solutions , HW-NAS has been proposed to automate the search for efficient DNN structures under the target efficiency constraints ( Fu et al. , 2020b ; a ; Zhang et al. , 2020 ) . For example , ( Tan et al. , 2019 ; Howard et al. , 2019 ; Tan & Le , 2019 ) adopt reinforcement learning based NAS with a multi-objective reward consisting of both the task performance and efficiency , achieving promising results yet suffering from prohibitive search time/cost . In parallel , ( Wu et al. , 2019 ; Wan et al. , 2020 ; Cai et al. , 2018 ; Stamoulis et al. , 2019 ) explore the design space in a differentiable manner following ( Liu et al. , 2018a ) and significantly improve the search efficiency . The promising performance of HW-NAS has motivated a tremendous interest in applying it to more diverse applications ( Fu et al. , 2020a ; Wang et al. , 2020a ; Marchisio et al. , 2020 ) paired with target hardware devices , e.g. , Edge TPU ( Xiong et al. , 2020 ) and NPU ( Lee et al. , 2020 ) , in addition to the widely explored mobile phones . As discussed in ( Chu et al. , 2020 ) , different hardware devices can favor very different network structures under the same hardware-cost metric , and the optimal network structure can differ significantly when considering different application-driven hardware-cost metrics on the same hardware device . As such , it would ideally lead to the optimal accuracy-cost trade-offs if the HW-NAS design is dedicated for the target device and hardware-cost metrics . However , this requires a good understanding of both device-specific compilation and hardware-cost characterization , imposing a barrier-to-entry to non-hardware experts , such as many NAS researchers , and thus limits the development of optimal HW-NAS results for numerous applications , each of which often prioritizes a different application-driven hardware-cost metric and adopts a different type of hardware devices . As such , our proposed HW-NAS-Bench will make HW-NAS more friendly to NAS researchers , who are often non-hardware experts , as it consists of comprehensive hardware-cost data in a wide range of hardware devices for all the networks in two commonly used SOTA NAS search spaces , expediting the development of HW-NAS innovations . 2.2 NEURAL ARCHITECTURE SEARCH BENCHMARKS . The importance and difficulty of NAS reproducibility and benchmarking has recently gained increasing attention . Pioneering efforts include ( Ying et al. , 2019 ; Dong & Yang , 2020 ; Klyuchnikov et al. , 2020 ; Siems et al. , 2020 ; Dong et al. , 2020 ) . Specifically , NAS-Bench-101 ( Ying et al. , 2019 ) presents the first large-scale and open-source architecture dataset for NAS , in which the ground truth test accuracy of all the architectures ( i.e. , 423k ) in its search space on CIFAR-10 ( Krizhevsky et al. , 2009 ) are provided . Later , NAS-Bench-201 ( Dong & Yang , 2020 ) further extends NAS-Bench-101 to support more NAS algorithm categories ( e.g. , differentiable algorithms ) and more datasets ( e.g. , CIFAR-100 ( Krizhevsky et al. , 2009 ) and ImageNet16-120 ( Chrabaszcz et al. , 2017 ) ) . Most recently , NAS-Bench-301 ( Siems et al. , 2020 ) and NATS-Bench ( Dong et al. , 2020 ) are developed to support benchmarking NAS algorithms on larger search spaces . However , all of these works either merely provide latency on the server-level GPU ( e.g. , GTX 1080Ti ) or do not consider any hardware-cost data on real hardware at all , limiting their applicability to HW-NAS ( Wu et al. , 2019 ; Wan et al. , 2020 ; Cai et al. , 2018 ) that primarily targets commercial edge devices , FPGA ( Wang et al. , 2020b ) , and ASIC ( Chen et al. , 2016 ; Lin et al. , 2017 ; 2016 ; Zhao et al. , 2020a ) . This has motivated us to develop the proposed HW-NAS-Bench , which aims to make HW-NAS more accessible especially for non-hardware experts and reproducible . A concurrent work ( published after our submission ) is BRP-NAS ( Chau et al. , 2020 ) , which presents a benchmark for the latency of all the networks in NAS-Bench-201 ( Dong & Yang , 2020 ) search space . In comparison , our proposed HW-NAS-Bench includes ( 1 ) more device categories ( i.e. , not only commercial devices , but also FPGA ( Wang et al. , 2020b ) and ASIC ( Chen et al. , 2016 ) ) , ( 2 ) more hardware-cost metrics ( i.e. , not only latency , but also energy ) , and ( 3 ) more search spaces ( i.e. , not only NAS-Bench-201 ( Dong & Yang , 2020 ) but also FBNet ( Wu et al. , 2019 ) ) . Additionally , we ( 4 ) add a detailed description of the pipeline to collect the hardware-cost of various devices and ( 5 ) analyze the necessity of device-specific HW-NAS solutions based on our collected data . 3 THE PROPOSED HW-NAS-BENCH FRAMEWORK . 3.1 HW-NAS-BENCH ’ S CONSIDERED SEARCH SPACES . To ensure a wide applicability , our HW-NAS-Bench considers two representative NAS search spaces : ( 1 ) NAS-Bench-201 ’ s cell-based search space and ( 2 ) FBNet search space . Both contribute valuable aspects to ensure our goal of constructing a comprehensive HW-NAS benchmark . Specifically , the former enables HW-NAS-Bench to naturally integrate the ground truth accuracy data of all NAS-Bench-201 ’ s considered network architectures , while the latter ensures that HW-NASBench includes the most commonly recognized hardware friendly search space . NAS-Bench-201 Search Space . Inspired from the search space used in the most popular cell-based NAS , NAS-Bench-201 adopts a fixed cell search space , where each architecture consists of a predefined skeleton with a stack of the searched cell that is represented as a densely-connected directed acyclic graph ( DAG ) . Specifically , it considers 4 nodes and 5 representative operation candidates for the operation set , and varies the feature map sizes and the dimensions of the final fully-connected layer to handle its considered three datasets ( i.e. , CIFAR-10 , CIFAR-100 ( Krizhevsky et al. , 2009 ) , and ImageNet16-120 ( Chrabaszcz et al. , 2017 ) ) , leading to a total of 3× 56 = 46875 architectures . Training log and accuracy are provided for each architecture . However , NAS-Bench-201 can not be directly used for HW-NAS as it only includes theoretical cost metrics ( i.e. , FLOPs and the number of parameters ( # Params ) ) and the latency on a server-level GPU ( i.e. , GTX 1080Ti ) . HW-NASBench enhances NAS-Bench-201 by providing all the 46875 architectures ’ measured/estimated hardware-cost on six devices , which are primarily targeted by SOTA HW-NAS works . FBNet Search Space . FBNet ( Wu et al. , 2019 ) constructs a layer-wise search space with a fixed macro-architecture , which defines the number of layers and the input/output dimensions of each layer and fixes the first and last three layers with the remaining layers to be searched . In this way , the network architectures in the FBNet ( Wu et al. , 2019 ) search space have more regular structures than those in NAS-Bench-201 , and have been shown to be more hardware friendly ( Fu et al. , 2020a ; Ma et al. , 2018 ) . The 9 considered pre-defined cell candidates and 22 unique positions lead to a total of 922 ≈ 1021 unique architectures . While HW-NAS researchers can develop their search algorithms on top of the FBNet ( Wu et al. , 2019 ) search space , tedious efforts are required to build the hardware-cost look-up tables or models for each target device . HW-NAS-Bench provides the measured/estimated hardware-cost on six hardware devices for all the 1021 architectures in the FBNet search space , aiming to make HW-NAS research more friendly to non-hardware experts and easier to be benchmarked .

Summary: The paper presents a benchmark / dataset, HW-NAS-Bench, for evaluating various neural architecture search algorithms. The benchmark is based on extensive measurements on real hardware. An important goal with the proposal is to support neural architecture searches for non-hardware experts. Further, the paper provides a good overview of related work in the domain.