HW-NAS-Bench: Hardware-Aware Neural Architecture Search Benchmark
Authors: Chaojian Li
Zhongzhi Yu
Yonggan Fu
Yongan Zhang
Yang Zhao
Haoran You
Qixuan Yu
Yue Wang
Cong Hao
Yingyan Lin
Venue: ICLR 2021 Spotlight (2021)
Keywords: Hardware-Aware Neural Architecture Search
AutoML
Benchmark

Abstract: HardWare-aware Neural Architecture Search (HW-NAS) has recently gained tremendous attention by automating the design of deep neural networks deployed in more resource-constrained daily life devices. Despite its promising performance, developing optimal HW-NAS solutions can be prohibitively challenging as it requires cross-disciplinary knowledge in the algorithm, micro-architecture, and device-specific compilation. First, to determine the hardware-cost to be incorporated into the NAS process, existing works mostly adopt either pre-collected hardware-cost look-up tables or device-specific hardware-cost models. The former can be time-consuming due to the required knowledge of the device’s compilation method and how to set up the measurement pipeline, while building the latter is often a barrier for non-hardware experts like NAS researchers. Both of them limit the development of HW-NAS innovations and impose a barrier-to-entry to non-hardware experts. Second, similar to generic NAS, it can be notoriously difficult to benchmark HW-NAS algorithms due to their significant required computational resources and the differences in adopted search spaces, hyperparameters, and hardware devices. To this end, we develop HW-NAS-Bench, the first public dataset for HW-NAS research which aims to democratize HW-NAS research to non-hardware experts and make HW-NAS research more reproducible and accessible. To design HW-NAS-Bench, we carefully collected the measured/estimated hardware performance (e.g., energy cost and latency) of all the networks in the search spaces of both NAS-Bench-201 and FBNet, on six hardware devices that fall into three categories (i.e., commercial edge devices, FPGA, and ASIC). Furthermore, we provide a comprehensive analysis of the collected measurements in HW-NAS-Bench to provide insights for HW-NAS research. Finally, we demonstrate exemplary user cases to (1) show that HW-NAS-Bench allows non-hardware experts to perform HW-NAS by simply querying our pre-measured dataset and (2) verify that dedicated device-specific HW-NAS can indeed lead to optimal accuracy-cost trade-offs. The codes and all collected data are available at https://github.com/RICE-EIC/HW-NAS-Bench.

Published as a conference paper at ICLR 2021
HW-NAS-BENCH HARDWARE-AWARE NEURAL AR- CHITECTURE SEARCH BENCHMARK
Chaojian Li, Zhongzhi Yu, Yonggan Fu, Yongan Zhang, Yang Zhao, Haoran You, Qixuan Yu, Yue Wang & Yingyan Lin
Department of Electrical and Computer Engineering
Rice University
{c1114, zy42, yf22, yz87, zy34, hy34, qyl2, yw68, yingyan. lineerice.
ABSTRACT
HardWare-aware Neural Architecture Search (HW-NAS) has recently gained tremendous attention automating the design of deep neural networks deployed in more resource-constrainee daily life devices. Despite its promising perfor- mance, developing optimal HW-NAS solutions can be prohibitively challenging as it requires dissidiicyllynary knowledge in the algorithm, micro-ane architecture, and device-specific compilation. First, to determine the hardware-cost to in- corporated into the NAS process, existing works mostly adopt either pre-collected hardware-cost look-up tables or device-specific hardware-cost models. The for- mer can be time-consuming due to the required knowledge of the device's com- pilation method and how to set up the measurement pipeline, while building the latter is often barrier for non-hardware experts like NAS researchers. Both of them limit the development of HW-NAS innovations and impose a barrier-to-entry non-hardware experts. Second, similar generic NAS, it be notoriously difficult to benchmark HW-NAS algorithms due to their significant required com- putational resources and the differences in adopted search spaces, hyperparame- ters, and hardware devices. To this end, we develop HW-NAS-Bench. the first public dataset for HW-NAS research which aims to democratize HW-NAS re- search to non-hardware experts and make HW-NAS research more reproducible and accessible. To design HW-NAS-Bent
we carefully collected the mea- sured/estimated hardware performance (e.g.. energy cost and latency) of all the networks in the search spaces of both NAS-Bench-201 and FBNet, on six hard- ware devices that fall into three categories (i.e., commercial edge devices, FPGA, and ASIC). Furthermore, we provide a comprchensive analysis of the collected measurements in HW-NAS-Bench provide insights for HW-NAS research. Fi- nally, we demonstrate exemplary user cases to (1) show that HW-NAS-Beneh allows non-hardware experts to perform HW-NAS by simply querying our pre- measured dataset and (2) verify that dedicated device-specific HW-NAS can in- deed lead to optimal accuracy-cost trade-offs. The codes and all collected data are available at https:: / www/eICoi.....ee.. RICE-EICCCCCCIIICC HW NAN--NNS--- MAS-Bench
INTRODUCTION
The recent performance breakthroughs of deep neural networks (DNNs) have attracted explosion of research in designing efficient DNNs, aiming to bring powerful yet power-hungry DNNs into more constraine daily life devices for enabling various DoN-powered intelligent func- tions (Ross) 2020: Liu et al.) 2018b] [Shen et al., 2020, You et al. 2020a). Among them, HardWare- aware Neural Architecture Search (HW-NAS) has emerged as one of the most promising techniques as it can automate the process of designing optimal DNN structures for the target applications, each which often adopts a different hardware device and requires a different hardware-cost met- ric (e.g., prioritizes latency or energy). For example, HW-NAS in (Wu et al. 2019) develops differentiable neural architecture search (DNAS) framework and discovers state-of-the-art (SOTA) DNNs balancing both accuracy and hardware efficiency, by incorporating loss consisting of both the cross-entropy loss that leads to better accuracy and the latency loss that penalizes the network's latency target device.

---

Published as a conference paper at ICLR 2021
Possible Architectures
Efficiency Performance
HW-NAS w/o Our Proposed HW-NAS-Bench
Quarying
Hardware Expertise/
Energy
Knowledge Needed
Target Hardware Devices
Feedback
HW-NAS Algorithms
Acc
Hardware-cost ModelingiCollection Pipeline
Erriciency
Searched Architecture
Funeback
Latency
Energy
HW-NAS Researchers Can Focus On This Part With HW-NAS-Bend
Hardware Expertise! Knowledge Needed
HW-NAS-Bench
Our Proposed
HW-NAS Our Proposed HW-NAS-Bench
Figure I: An illustration of our proposed HW-NAS-Bendh
Despite the promising performance achieved by SOTA HW-NAS, there exist paramount challenges that limit the development of HW-NAS innovations. First, HW-NAS requires the collection of hardware efficiency data corresponding to (all) the networks the search space. To do so, cur- rent practice either pre-collects these data to construct hardware-cost look-up table adopts device-specific hardware-cost estimators/models, both of which can be time-consuming to obtain and impose barrier-to-enee to non-hardware experts. This is because requires knowledge about device-specific compilation and properly setting up the hardware measurement pipeline to collect hardware-cost data. Second, similar to generic NAS, it be notoriously difficult to benchmark HW-NAS algorithms due to the required significant computational resources and the differences in their (1) hardware devices, which are specific for HW-NAS, (2) adopted search spaces, and (3) hy- perparameters. Such a difficulty is even higher for HW-NAS considering the numerous choices of hardware devices, each of which can favor very different network structures even under the same target hardware efficiency, as discussed in (Chu et al. 2020). While the number of floating-poit point op- erations (FLOPs) has been commonly used estimate the hardware-coos many works have pointed out that DNNs with fewer FL.OPs are not necessarily faster or more efficient (Wu et al. 2019 2018 Wang et al., 2019b). For example, NasNet-A (Zoph et al., 2018) has a comparable complexity in terms of FLOPs as MobileNetVI (Howard al. (017). yet can have larger latency than the latter due to A (Zoph et al. 2018)'s adopted hardware-unfriendly structure.
It is thus imperative to address the aforementioned challenges in order to make HW-NAS more accessible and reproducible to unfold HW-NAS's full potential. Note that although pioneering NAS benchmark datasets (Ying et al.) 2019 Dong & Yang 2020; Klyuchnikov al. 2020; Siems al., 2020 Dong et al.) 2020) have made a significant step towards providing a unified benchmark dataset for generic NAS works, all of them either merely provide the latency server-level GPUs (e.g., GTX 1080Ti) or do not provide any hardware-cost data on real hardware, limiting their applicability HW-NAS (Wu al. 2019 Wan et al. 2020) Cai et al. (0188) which primarily targets commercial edge devices, FPGA, and ASIC. To this end, as shown in Figure [ we develop HW-NAS-Bench and make the following contributions this paper:
We have developed EW-NAS-Bench, the first public dataset for HW-NAS research aiming to (1) democratize HW-NAS research to non-hardware experts and (2) facilitate a unified benchmark for HW-NAS to make HW-NAS research more reproducible and accessible, covering two SOTA NAS search spaces including NAS-Bench-201 and FBNet, with the former being one of the most popular NAS search spaces and the latter having been shown to be one of the most hardware friendly NAS search spaces.
We provide bardware-cost data collection pipelines for six commonly used hardware de- vices that fall into three categories (i.e.. commercial edge devices, FPGA, and ASIC), in addition to the measured/estimated hardware-cost (e.g., energy cost and latency) on these devices for all the networks the search spaces of both NAS-Bench-200 and FBNet.
We conduct comprehensive analysis of the collected data in WW--NAS-Bench such as studying the correlation between the collected hardware-cost and accuracy-cost data of all
2

---

Published as a conference paper at ICLR 2021
the networks on the six hardware devices, which provides insights to not only HW-NAS re- searchers but also DNN accelerator designers. Other researchers can extract useful insights from HW-NAS-Bente that have not been discussed in this work.
We demonstrate exemplary user cases to show: (1) how TW--NS--Bende can be easily used by non-hardware experts to develop HW-NAS solutions by simply querying the col- lected data in our HW-NAS-Bent lead to optimal accuracy-cost trade-offs, demonstrating the great necessity of HW-NAS benchmarks like our proposed NWW-NAS-Bench
and (2) dedicated device-specific HW-NAS can indeed
2 RELATED WORKS
2.1
HARDWARF-AWARE NEURAL ARCHITECTURE SEARCH
Driven by the growing demand for efficient DNN solutions, HW-NAS has been proposed to au- tomate the search for efficient DNN structures under the target efficiency constraints (Fu et al,,
2020b Zhang et al. 2020). For example, (Tan et al. 2019 Howard et al., 2019 Tan & Le, 2019) adopt reinforcement learning based NAS with a multi-objective reward consisting of both the task performance and efficiency, achieving promising results yet suffering from prohibitive search time/cost. In parallel, (Wu et al. [219:] Wan et al. [020)] Cai et al.] [0188] Stamoulis et al., 2019)) explore the design space in a differentiable manner following (Liu al.) 2018a) and significantly
improve the search efficiency. The promising performance of HW-NAS has motivated a tremendous
interest in applying to more diverse applications (Fu et al. 2020a; Wang et al. 2020a Marchisio
et al., 2020) paired with target hardware devices, e.g., Edge TPU (Xiong et al. 2020) and NPU (Lee
ct al. 2020). in addition to the widely explored mobile phones.
As discussed in (Chu et al. 2020), different hardware devices can favor very different network structures under the same hardware-cost metric, and the optimal network structure can differ signif- icantly when considering different application-driven hardware-cost metrics the same hardware device. As such, it would ideally lead to the optimal accuracy-cost tradc-offs if the HW-NAS de- sign is dedicated for the target device and hardware-cost metries. However, this requires good understanding of both device-specific compilation and hardware-cost characterization, imposing a barrier-to-entry to non-hardware experts, such as many NAS researchers, and thus limits the devel- opment of optimal HW-NAS results for numerous applications, each of which often prioritizes a different application-driven hardware-cost metric and adopts a different type of hardware devices. As such, our proposed HW-NAS Benchhhhhhhhh will make HW-NAS more friendly to NAS researchers, who are often non-hardware experts, as it consists of comprehensive hardware-cost data wide range of hardware devices for all the networks in two commonly used SOTA NAS search spaces, expediting the development of HW-NAS innovations.
2.2 NEURAL ARCHITECTURE SEARCH BENCHMARKS
The importance and difficulty of NAS reproducibility and benchmarking has recently gained in-
creasing attention. Pioneering efforts include (Ying et al. 2019: Dong & Yang 2020 Klyuchnikov
et al. 2020 Siems et al.) 2020 Dong et al. 2020). Specifically, presents the first large-scale and open-source architecture dataset for NAS, in which the ground truth
(Ying et al.) (019]
test accuracy of all the architectures (i.e., 423k) in its search space on CIFAR-10 (Krizhevsky al.) 2009) are provided. Later, NAS-Bench-201 (Dong & Yang 2020) further extends NAS-Bench--O1 to support more NAS algorithm categories (c.g., differentiable algorithms) and more datasets (c.g., CIFAR- 100 (Krizhessky et al. 2009) and ImageNet16- 1engeNet16-1200 (Chrabaszez et al. (017)). Most recently, NAS-Bench-301 (Siems et al. 2020) and NATS-Bench (Dong et al. 2020) are developed to support benchmarking NAS algorithms on larger search spaces. However, all of these works either merely provide latency on the server-level GPU (e.g., GTX 1080Ti) or do not consider any hardware-cost data real hardware at all, limiting their applicability to HW-NAS (Wu et al.] 2019 Wan et al.) 2020 Cai et al.] 2018) that primarily targets commercial edge devices, FPGA (Wang et a 2020b), and ASIC (Chen et al. 2016 Lin et al.) 2017; 2016 Zhao et al. 2020a). This has motivated to develop the proposed
Benchench, which aims to make HW-NAS more accessible especially for hardwar experts and reproducible.
3

---

Published as a conference paper at ICLR 2021
A concurrent work (published after our submission) is BRP-NAS (Chau et al. 2020). which presents
a benchmark for the latency of all the networks in NAS- Bench-201 (Dogg & Yang 2020) search space. In comparison, our proposed HW-NAS-Bened includes (1) more device categories (i.e., not only commercial devices, but also FPGA (Wang et al. 2020b) and ASIC (Chen et al.] 2016)), (2)
more hardware-cost metrics (i.e., not only latency, but also energy), and (3) more search spaces (i.e., not only HAS-Bench-200 (Dong & Yang 2020) but also FBNet (Wu et al., (019)). Additionally, we (4) add detailed description of the pipeline collect the hardware-cost of various devices and (5) analyze the necessity of device-specifee HW-NAS solutions based on our collected data.
3 THE PROPOSED HW-NAS-BENCH FRAMEWORK
3.1 HW-NAS-BENCITS CONSIDERED SEARCH SPACES
To ensure a wide applicability. our -e-NNA--Becch considers two representative NAS search spaces: (1) JAS-Bench-201's 's cell-based search space and (2) FBNet search space. Both contribute valuable aspects to ensure our goal of constructing a comprehensive HW-NAS benchmark. Specif- ically, the former enables HW-NAS-Benn to naturally integrate the ground truth accuracy data of all NAS-Bench-201's considered network architectures, while the latter ensures that HW-NAS- Bench includes the most commonly recognized hardware friendly search space.
NAS-Bench-201 Search Space. Inspired from the search space used in the most popular cell-based NAS,
adopts fixed cell search space, where each architecture consists of a prede- fined skeleton with a stack of the searched cell that is represented as densely-connected directed acyelic graph (DAG). Specifically, it considers nodes and representative operation candidates for the operation set, and varies the feature map sizes and the dimensions of the final fully-connected layer to handle its considered three datasets (i.e., CIFAR-IO, CIFAR-IO0 -00 (Krizhevsky et al., 2009), and ImageNet16 6- 120 (Chrabaszez et al. 2017)) 7) leading to total of 3 x 5 46875 architectures. Training log and accuracy are provided for each architecture. However, NAS-Bench-22 directly used for HW-NAS it only includes theoretical cost metrics (i.e., FLOPs and the number of parameters (#Params)) and the latency on a server-level GPU (i.e., GTX 1080Ti). HW-NAS- Bench enhances NAS-Bench-201 by providing all the 46875 architectures' measured/estimated hardware-cost on six devices, which are primarily targeted by SOTA HW-NAS works.
can not be
FBNet Search Space. FBNet (Wu et al., 2019) constructs layer- wise search space with fixed macro-architecture, which defines the number of layers and the input/output dimensions of each layer and fixes the first and last three layers with the remaining layers be searched. In this way, the network architectures in the FBNet (Wu et al. (019) search space have more regular structures than those HAS-Bench-221, and have been shown to be more hardware friendly (Fu et al., [220aa] Ma et al.] 2018). The 9 considered pre-defined cell candidates and 22 unique positions lead to total of 922 1021 unique architectures. While HW-NAS researchers can develop their search algorithms on top of the FBNet (Wu et al. 2019) search space, tedious efforts are required to build the hardware-cost look-up tables or models for each target device. HW-NAS-Bence provides the measured/estimated hardware-cost on six hardware devices for all the 10 architectures in the FBNet search space, aiming to make HW-NAS research more friendly to non-hardware experts and easier to be benchmarked.
3.2 HARDWARE-COST COLLECTION PIPELINE AND THE CONSIDERED DEVICES
To collect the hardware-cost data for all the architectures in both the search spaces, we construct a generic hardware-cos collection pipeline (see Figure 2 to automate the process. The pipeline mainly consists of the target devices and corresponding deployment tools (e.g., compilers). Specifically, it takes all the networks as its inputs, and then compiles the net- works to (1) convert them into the device's 's required execution format and (2) optimize the execution flow, the latter of which aims to optimize the hardware performance on the target devices. For example, for collecting the hardware-cost in Edge GPU, we first set the device in the Max-N mode to fully make use of all available resources following (Wofk et al., 2019), and then set up the embedded power rail monitor (Texas Instruments Inc.) obtain the real- measuredd latency and energy via sysfs (Patrick Mochel and Mike Murphy.). averaging over 50 runs. We can see that the hardware-cost collection pipeline requires various hardware domain knowledge, includ-
and FBNet
4

---

Published as a conference paper at ICLR 2021
Possible Architectures
Toow Lioe
PyTorch
Impiersttation
PyTorch
Hardware Devioes and Implementation Tools
NIF
Edge TPU Compieer
TomsorRI
Harchaaee Pragina
Coral
Elfiiiency Performance
Lebrey
TIi-NSS-Bence
Configure Devices in Advance Using Hardware Experts Knowledge
Figure 2: Illustrating the hardware-cost collection pipeline applicable to various hardware devices.
ing machine learning development frameworks, device compilation, embedded systems, and device measurements, imposing a barrier-to-entry to non-hardware experts.
Next, briefly introduce the six considered hardware devices (as summarized in Table and the specific configuration required to collect the hardware-cost data on each device.
Edge GPU: NVIDIA Edge GPU Jetson TX2 (Edge GPU) is a commercial device with 256-core Pascal GPU and a 8GB LPDDR4, targeting IoT applications (NVIDIA Inc. a). When plugging an Edge GPU into the above hardware-cost collection pipeline, first compile the network chitectures in both NAS-Bench-201 and FBNet spaces to (1) convert them to the TensorRT format and (2) optimize the inference implementation within NVIDIA's recommended TensorRT runtime environment, and then execute them in the Edge GPU to measure the consumed energy and latency.
Raspi 4: Raspberry Pi 4 (Raspi 4) is the latest Raspberry Pi device (Raspberry Limited.), con- sisting of Broadcom BCM2711 SoC and a 4GB LPDDR4. To collect the hardware-coss oper- ating on it, we compile the architecture candidates to (1) convert them into the TensorFlow Lite (TFLite) (Abadi et al. 2016) format and (2) optimize the implementation using the official inter- preter (Google L.I.C] 2020) in Raspi 4, where the interpreter will pre-configured.
An Edge TPU Dev Board (Edge TPU) GGoole LLC. a) is a dedicated ASIC accelerator developed by Google, targeting Artificial Intelligence (AI) inference for edge applications. Similar to the case when using Raspi 4, all the architectures are converted into the TFLite format. After that, an Edge TPU compiler will be used to convert the pre-built TFLite model into a more compressed format which is compatible to the pre-confegured runtime environment in the Edge TPU.
Edge TPU:
Pixel 3: Pixel 3 is one of the latest Pixel mobile phones (Google LLC. (), which are widely used as the target platforms by recent NAS works (Xiong al. 2020 Howard et al.) 2019 Tan al. 2019) To collect the hardware-cost in Pixel 3, we first convert all the architectures into the TFLite format, then use TFLite's official benchmark binary file obtain the latency, when configuring the Pixel
device to only use its big cores for reducing the measurement variance as in (Xiong et al. 2020
Tan
al. 2019).
ASIC-Eyeriss: For collecting the hardware-cost data in ASIC, we consider SOTA ASIC acceler- ator, Eyeriss (Chen al. 2016). Specifically, we adopt the SOTA ASIC accelerator's 'ss performance simulators: (1) Accelergy (Wu et al. 20i9)+Tmmcooo (Parashar et al. 2019) and (2) DNN-Chij Pre- dictor (Zhao al., 2020b), both of which automatically identify the optimal algorithm-to soorithm-to-hhddwaan hardware mapping methods for each architecture and then provide the estimated hardware-cose of the network execution in Eyeriss.
Table I: Important details about the six hardware devices considered by our HW-NAS-Bench.
Devices
|| Edge GPU
Raspi
Edge TPU
Pixel
ASIC-Eyeriss
FPGA
Collected Metrics
Latency (ms) Energy (mJ)
Latency (ms))
Latency (ms)
Latency (ms)
Latency (iss)
Latency (ms)
Energy (mJ)
Energy (in)
Collecting Method
||
Measured
Measured
Measured
Measured
Estimated
Estimated
Runtime Environment
TensorKT
TensorFlow
Edge TPU
TensorFlow
Accelergy Tincloop/
Vivado
Lite
Runtime
Lite
DNN Chip Predictur
HLS
Customizing Hardware?
||
Category
Commercial Edge Devices
ASIC
FPGA
5

---

Published as a conference paper at ICLR 2021
Table 2: Two types of correlation coefficients (larger means more correlated) between the real- measured hardware-cost of the whole architectures and the approximated hardware-cost based on 100 randomly sampled architectures from the FBNet search space.
Correlation Coefficient Types
Datasets
Lalency on
Energy Edge GPU
Latency on
Latency on Edge TPU
Lalency on Pixel
Edge GPU
Raspi 4
CIFAR-100
0.9200
0.9116
0.9219
0.4935
0.9324
Pearson Corcelatonn Coefficient
ImageNet
0,8634
0,9640
0.9897
0.7153
0.9162
CIFAR-100
0.7373
0.7240
0.7470
0.3551
0.8593
Kendall Rank Correlation Coefficient
ImageNet
0.7111
0.8379
0.9163
0,5806
0.8064
FPGA: FPGA is a widely adopted Al acceleration platform featuring higher hardware flexibility than ASIC and more decent hardware efficiency than commercial edge devices. To collect hardware- cost data in this platform, we first develop SOTA chunk based pipeline structure (Shen et al. 2017 Zhang et al.) 2220) implementation, compile all the architectures using the standard Vivado HLS toolflow (Xilinx Inc. a). and then obtain the hardware-cost on Xilinx ZC706 board with Zynq XC7045 SoC (Xilinx Inc. by.
More details about the pipeline for each of the aforementioned devices are provided in the Ap- pendix for better understanding.
to estimate the hardware-cost of the networks in the FBNet search space In our 2019) when being executed on the commercial edge devices (i.e., Edge GPU, Raspi 4, (Wu et al.
Edge TPU, and Pixel 3), sum up the hardware-cost of all unique blocks (i.e., "block" in the FBNet space (Wu et al. (019)) within the network architectures. To validate that such an approxi- mation is close the corresponding real-measured results, conduct experiments, summarized in Table 2 to calculate two types of correlation coefficients between the measured and the approx- imated hardware-cost based on 100 randomly sampled architectures from the FBNet search space. We can see that our approximated hardware-cost is highly correlated with the real-measured one, except for the case on the Edge TPU, which we conjecture caused by the adopted in in-house Edge TPU compiler (Google I.I.C. c). More visualization results can be found in the Appendix A.
4 ANALYSIS ON HW-NAS BENCH
In this section, we provide analysis and visualization of the hardware-cost and corresponding aceu- racy data (the latter only for architectures (AS-Bench-201) for all the architectures in the two con- sidered search spaces. Specifically, our analysis and visualization confirm that (1) commonly used theoretical hardware-cost metries such as FLOPs do not correlate well with the measured/cstimated hardware-cost; (2) hardware-cost of the same architectures can differ lot when executed on differ- ent devices; and (3) device-specific HW-NAS is necessary because optimal architectures resulting from HW-NAS targeting on one device can perform poorly in terms of the hardware cost when being executed on another device.
4.1 CORRELATION BETWEEN COLLECTED HARDWARE-COST AND THEORETICAL ONES
To confirm whether commonly used theoretical hardware-cost metrics align with real- measured/essimmated ones, we summarize the calculated correlation between the collected hardware- cost in our HW-NAS-Beneh and the theoretical metrics (i.e., FLOPs and #Params), based on the data for all the architectures in both search spaces on all the six considered hardware devices where a total of four different datasets are involved.
Table 3: Kendall Rank Correlation Coefficient between sell-measured/estimate hardware-cost and theoretical ones considering the MAS-Bench-200 search space, where coefficients <0.5 are bolded.
Edge GPU
Ruspi 4 Latency
Edge TPU
Pixel 3
ASIC-Eyeriss
PPGA
Dataset
Metries
Latency
Energy
Latency
Latency
Latency
Energy
Latency
Energy
FLOPs
0.3571
0.4064
0.7394
0.1847
0.6823
0.4178
0.5359
0,8313
0,8313
CIFAR-10 10
#Params
0.3571
0.4064
0.7394
0.1847
0.6823
0.4178
0.5359
0,8313
0.8313
FLOPs
0.3589
0.4073
0,7384
0.1851
0.6844
0.4197
0.5360
0.8313
0,8313
CIFAR- 100
#Params
0.3589
0.4073
0.7384
0.1851
0.6844
0.4197
0.8313
0.8333
ImageNet16-120
FLOPs
0.3544
0.3868
0.6303
0.2635
0.7017
0.4166
0.5363
0.9205
0.9205
#Params
0.3544
0
0.6303
0.2635
0,7017
0.4166
0.5363
0,9205
0.9205

---

Published as a conference paper at ICLR 2021
Table 4: Kendall Rank Correlation Coefficient between maal-ssssurd//ssimateed hardware-cost and theoretical ones considering the FBNet search space, where coefficients <0.5 are bolded.
Edge GPU
Raspi 4 Latency
Pixel 3
AyeC-syeriis
FPGA
Dataset
Metrics
Latency
Energy
Latency
Latency
Energy
Latency
Energy
FLOPs
0.0149
0.1564
0.7713
0.8092
0.8490
0.7854
0,8710
0.8710
CIFAR-100
#Params
-0.0733
0.0202
0.4910
0.3734
0.4297
0.6455
0,5151
0,5151
FLOPs
0.4633
0.6094
0.7531
0.7678
0.8935
0,7970
0,8643
0.8643
ImageNet
#Params
0.0985
0.1840
0.2318
0.2357
0.3202
0.4140
0.4198
0.4198
As summarized in Tables 3 commonly used theoretical hardware-cost metrics (i.e., FLOPs and #Params) do not always correlate well with measured/estimated hardware-cost for the architectures in both the NAS-Bench-201 and FBNet spaces. For example, there exists at least one coefficient <0.5 on all devices, especially for the cases with reul-measured/essimated hardware-cost on com- monly considered edge platforms including Edge GPU, Edge TPU, and ASIC-Eyeriss. As such, HW-NAS based on the theoretical hardware-cost might lead sub-optimal results, motivating HW- NAS benchmarks like our HW-NAS-Benchs Note that we consider the Kendall Rank Correlation Coefficients (Abdi) 2007), which is a commonly used correlation coefficient in both recent NAS frameworks and benchmarks (You et al. 2020b Siems et al. 2020 Yang et al. 2020
4.2 CORRELATION AMONG COLLECTED HARDWARE-COST ON DIFFERENT DEVICES
To check how much the hardware-cost of the same architectures different devices correlate, visualize the correlation between the hardware-cost collected from every two paired devices based on the data for all the architectures in both the CAS-Bench-201 and FBNet search spaces with each of the architectures associated with 9 different hardware-cost metrics.
Edge GPU
Respi&
Edge TPU! ABIC-Eyerii Eyerisa
Latency
Energy
Latency
Latancy
Latency
Latency
Energy
Latency
Energy
GPU
Latency LUW oul
0:45
6:34
6:45
0:45
3.69 6.5
5:34 6.45
6.7
635
1:0
Energy (va)
0:56
Raspi
Ladency
Latenoy
0.6
0.6
025
Latency 2065 036
0.79 0.70
606
600
1.08
005
0.72
0.72
4:45 6:08
6:6
9:24
ABIC-
0.4
9.4
0:27
0.4
Eyeriss
Latency
0.29
1.00
0.2
6:23
0.2
3
02
FPGA
Energy
0:50 0:14
0:
0:57
1.00
1:00
0:45
6:54
6:14
6:34
0:33
6:54
6:44
1.00
1:00
44 44)
<47
6335
6:14
0.0
0.0
0.0
CIFAR-10
CIFAR-100
ImageNet16-120
Figure 3: Kendall Rank Correlation Coefficient between real- maaesurrrrd//ettttteeddd hardware-cost in different devices considering the NAS-Bench-209 search space.
Edge GPU
Raspi
Pizel 3 Latenoy
ASIC-Eyeriss
FPGA
Latenoy
Energy
Latency
Latenoy
Energy
Latency
Energy
1.0
1.0
Latency
1.90
200
6:07
0.01
0.13
0.83
2.51
0.15
0.12
3.43
0.44
0.44
Edge GPU
Energy 6.33
1.00
0.23
6.21
3.30
0.33
0.15
0.33
0.01
1.00
0.74
0.76
0.57
0.57
0.57
0.37
Respi
Latency-
0:35
9.10
1.00
61
0.76
0.24
0.74
0.6
0.82
9.70
0.65
0.69
0.6
ASIC- Eyeriss
Lalancy
0.4
0.4
8.53
3.00
Latency 5.5
2.18
0.74
6.74
07H
0.88
5.00
1.00
0.2
8.64
8.77
0.68
0.70
0.80
0.77
1.00
1
0.2
Energy- 6.l
9.15
0.74
6.73
7.
0.83
3,00
1
9:46
0.7
8.09
0.70
0.89
0.77
1,00
3
CIFAR-100
ImageNet
Figure 4: Kendall Rank Correlation Coefficient between real-meesured//sstmatted hardware-cost in different devices considering the FBNet search space.

---

Published as a conference paper at ICLR 2021
Test Accuracy (1)
Test Accuracy (46)
Toct Accuracy (16)
Edge Latency (ms)
13
Raspi Latency (ms)
45
25
1.
Latency (ms)
Test Accuracy (96
Test Accuracy (96)
Test Accuracy (96)
20
Lateney (ms)
ASIC-Eyeriss Latency (ms)
FPGA Latency (ms)
Figure 5: Accuracy vs. hardware-cost on different devices considering NAS-Bench-2011 where points in red denote the architectures with the optimal trade-offs between "accuracy on ImageNet16 120 vs. latency measured on Edge GPU", which the architectures represent the ground truth of HW-NAS targeting Edge GPUs.
The visualization in Figures 3 iindicates that hardware-cost the same network architectures can differ a lot when being executed different devices. More specifically, the correlation coefficients can small -0.00 (e.g., Edge GPU latency vs. ASIC-Fyeriss energy for the architectures in the FBNet search space), which is resulting from the large difference in their underlying (1) hardware sucro-architecttress and (2) available hardware resources. Thus, the resulting architecture of HW-NAS targeting device might perform poorly when being executed other devices, motivating device-specific HW-NAS; Furthermore, is crucial to develop comprehensive hardware- cost datasets like HW-NAS-BBL Bench to enable fast development and ensure optimal results of HW-NAS for different applications.
4.3 OPTIMAL ARCHITECTURES DIFFERENT HARDWARE DEVICES
To confirm the necessity of performing device-speciffs HW-NAS from another perspective, sum- marize the test accuracy vs. hardware-cost all the architectures JAS-Bench-200 considering the ImageNet16-1 6- dataset, and analyze the architectures with the optimal accuracy-cost trade-offs for different devices.
As shown in Figure such optimal architectures for different devices not the same. For example, the optimal architectures on Fdge GPU (marked as red points) can perform poorly in terms of the hardware-cost in other devices, especially in ASIC-Eyeriss and Edge TPU whose hardware-cost exactly has the smallest correlation coefficient with the hardware-cost measured in Edge GPU, which shown in Figure 3 Again, this of analysis and visualization confirms that HW-NAS targeting on one device can perform poorly in terms of the hardware-cost when being executed on another device, thus motivating the necessity of device-specific HW-NAS.
USER CASES: BENCHMARK SOTA HW-NAS ALGORITHMS
In this section, we will demonstrate the user cases of our HW-NAS-Bende to show (1) how non- hardware experts can it to develop HW-NAS solutions by simply querying the hardware-cost
8

---

Published as a conference paper at ICLR 2021
Table 5: Inference accuracy and lateney comparison of the optimal architectures resulting from ZW-NAS-Bendh when targeting different hardware devices.
Targeted Device in HW-NAS
Latency on Edge GPU (ms)
Latency on Raspi 4 (ms)
Latency on FPGA (ms)
Top-1 A%c.(%)
Edge GPU
74.11
9.96
31.01
20.19
Raspi 4
73.46
13.88
22.91
15.39
FPGA
73.51
20.65
25.43
13.96
data and (2) dedicated device-specific HW-NAS can indeed often lead to optimal accuracy-cost trade-offs, again showing the important need for HW-NAS benchmarks like our HW-NAS-Bench to enable more optimal HW-NAS solutions via device-specific HW-NAS.
Benchmark Setting. We adopt SOTA HW-NAS algorithm, ProxylessNAS CCai al. 2018)
for this experiment. As an example to use our HW-NAS-Bench we use ProxylessNAS to search over the FBNet (Wu al., 2019) search space on CIFAR-100 (Krizhevsky et al. 2009). when targeting different devices in our HW-NAS-Bench by simply querying the corresponding device's measured/essimated hardware-cost. which has negligible overhead as compared to the HW-NAS algorithm itself, without the need for hardware expertise or knowledge during the whole HW-NAS.
5.1 OPTIMAL ARCHITECTURE, RESULTING FROM DEVICE-SPECIRIC HW-NAS
Table Billuusraas that the searched architectures achieve the lowest latency among all architectures when the target devices of HW-NAS the same the one used measure the architecture's on-device inference latency. Specifically, when being executed on an Edge GPU, the searched ar- chitecture targeting Raspi during HW-NAS leads to about 50% higher latency, while the searched architecture targeting FPGA during HW-NAS introduces over a 100% higher latency, than the ar- chitecture specifically target on the Edge GPU during HW-NAS, under the same inference accuracy. This set of experiments shows that non-hardware experts can easily use our HW-NAS-Bene to develop optimal HW-NAS solutions, and demonstrates that device-specific HW-NAS is critical to guarantee the searched architectures' on-device performance.
6 CONCLUSION
We have developed HW-NAS-Bench the first public dataset for HW-NAS research aiming to (1) democratize HW-NAS research non-hardware experts and (2) facilitate unified benchmark for HW-NAS to make HW-NAS research more reproducible and accessible. Our HW-NAS -ennchh cov- representative NAS search spaces, and provides all network architectures' hardware-cost data on six commonly used hardware devices that fall into three categories (i.e., commercial edge devices, FPGA, and ASIC). Furthermore, we conduct comprehensive analysis of the collected data in HW-NAS-Bende
aiming to provide insights to not only HW-NAS researchers but also DNN ac- celerator designers. Finally, we demonstrate exemplary user cases of HW-NAS- B-Bnnch to show: (1) how HW-NAS -eenhhhh can be easily used by non-hardware experts via simply querying the collected data to develop HW-NAS solutions and (2) dedicated device-specific HW-NAS can indeed lead to optimal accuracy-cost trade-offs, demonstrating the great necessity of HW-NAS benchmarks like our proposed TIW-NAS-Bendee It expected that our IW-NAAS-Bechhaan significantly expedite and facilitate HW-NAS research innovations.
ACKNOWLEDGEMENT
The work supported by the National Science Foundation (NSF) through the CNS Division of Computer and Network Systems (Award number: 2016727).

---

Published as a conference paper at ICLR 2021
REFERENCES
Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving. Michael Isard, al. Tensorflow: A system for large-scale machine learning. In 12th [USENIX) symposium on operating systems design and implementation ({OSDI) 16), pp. 265-283, 2016.
Hervé Abdi. The kendall rank correlation coefficient. Encyclopedia of Measurement and Statistics. Sage, Thousand Oaks, CA, pp. 508-510, 2007.
Junjie Bai, Fang Lu, Ke Zhang, et al. Onnx: Open neural network exchange.
com/onna onnx onnx/onnx onnx, 2020.
Samik Basu, Mahasweta Ghosh, and Soma Barman. Raspberry pi 3b+ based smart remote health monitoring system using iot platform. In Proceedings of the 2nd International Conference on Communication, Devices and Computing, pp. 473 484. Springer, 2020.
Jacob Benesty, Jingdlong Chen, Yiteng Huang, and Israel Cohen. Pearson correlation coefficient. In Noise reduction in speech processing, pp. 1-4. Springer, 2009.
Han Cai, Ligeng Zhu, and Song Han. Proxylessnas: Direct neural architecture search on target task and hardware. arXiv preprint Xiv: 1EXX::8112.00322 2018.
Thomas Chau, Lukasz Dudziak, Mohamed Abdelfattah, Royson Lee, Hyeji Kim, and Nicholas D Lane. Brp-nas: Prediction-base based nas using gens. Advances in Neural Information Processing Systems, 2020.
Y. Chen, T. Krishna, Emer, and V. Sze. Eyeriss: An energy-efficient reconfigurable accelerator for deep convolutional neural networks. JSSC 2017, 52(1):127-138, 2017.
Yu-Hsin Chen, Joel Emer, and Vivienne Sze. Eyeriss: A spatial architecture for energy-efficient dataflow for convolutional neural networks. ACM SIGARCH Computer Architecture News, 44 (3):367-379, 2016.
Yu-Hsin Chen, Tushar Krishna, Joel Emer, and Vivienne Sze. Eyeriss: An Energy- Efficient Reconfigurable Accelerator for Deep Convolutional Neural Networks. In IEEE International Solid-State Circuits Conference, ISSCC 2016. Digest of Technical Papers, pp. 262-263, 2016.
François Chollet et al. Keras. https:://eras.it 2015.
Patryk Chrabaszez, Iya Loshchilov, and Frank Hutter. downsampled variant of imagenet as an alternative to the cifar datasets. arXiv preprint arXiv: 1707. 1707.08819, 2017.
Grace Chu, Okan Arikan, Gabriel Bender, Weijun Wang, Achille Brighton, Pieter-Jan Kindermans, Hanxiao Liu, Berkin Akin, Suyog Gupta, and Andrew Howard. Discovering hurdi-hardware mobile models via architecture search. arXiv preprint arXiv:2008.08172 2020.
Xuanyi Dong and Yi Yang. Nas-bench-22 201: Extending the scope of reproducible neural architecture search. International Conference on Learning Representations (ICLR), 2020. URL https //openzeview. wwt//eeummme.e..
2XBKDD
Xuanyi Dong, Lu Liu, Katarzyna Musial, and Bogdan Gabrys. Nats-bench: Benchmarking nas algorithms for architecture topology and size. arXiv preprint arXiv:2009.00433 2020.
Yonggan Fu, Wuyang Chen, Haotao Wang, Haoran Li, Yingyan Lin, and Zhangyang Wang. Autogan-distille: Searching to compress generative adversarial networks. arXiv preprint arXiv:2006.08198, 2020a.
Yonggan Fu, Zhongzhi Yu, Yongan Zhang, and Yingyan Lin. Auto-agent-distiller: Towards efficient deep reinforcement learning agents via neural architecture search, 2020b.
Lukas Geiger and Plumerai Team. Larq: An open-source library for training binarized neural net- works. Journal of Open Source Software, 5746). January 2020. doi: 10.21 2221100//oss017000
URL htthtt //doi org 000
10

---

Published as a conference paper at ICLR 2021
Google LLC. Edge TPU Compiler,
httpe://oocallall//cce///e-bb0d
get-started/ accessed 2020-09-01.
Google
LLC. Edge TPU Code Examples, b.
http:://ooral.ailexxxpppppp
#code-examp es accessed 2019-11-21.
Google LLC. Edge TPU
c. /ttps:///crallaa//////////ee/eee
reeuteemrrquiiimeeeeee accessed 2020-09-01.
Google LLC. Edge TPU FAQ, d. 2019-11-21.
attts://ooad.///0occeeeeeto
accessed
Google LLC. Pixel 3,
accessed 2020-09-01.
Google LLC. TensorFlow Lite: Deploy machine learning models on mobile and IoT devices, f. 2019-11-21
https://www http://www.eessrfloo...oooo wtw.eessorolooooroooleeeeee..eee
Google LLC. Tflite python quickstart.
http:::/ww..eensrrrroo.o.cc////.
python] 2020.
Andrew Howard, Mark Sandler, Grace Chu, Liang-Chich Chen, Bo Chen, Mingxing Tan, Wei- jun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for mobilenetv3, In Proceedings of the IEEE International Conference on Computer Vision, pp. 1314-1324, 2019.
Andrew Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint M.Xiv:1704.048111 2017.
Nikita Klyuchnikov, Ilya Trofimov, Ekaterina Artemova, Mikhail Salnikov, Maxim Fedorov, and Evgeny Burnaev. Nas-bench-nlp: Neural architecture search benehmark for natural language processing, 2020.
Alex Krizhevsky et al. Learning multiple layers of features from tiny images. 2009.
Jaeseong Lee, Duseok Kang, and Soonhoi Ha. S3nas: Fast npu-aware neural architecture search methodology. arXiv preprint arXiv:2009.02000 2020.
Chaojian Li, Tianlong Chen, Haoran You, Zhangyang Wang, and Yingyan Lin. Halo: Hardware- aware learning optimize. Proceedings of the European Conference on Computer Vision (ECCV), September 2020.
Y. Lin, S. Zhang, and N. R. Shanbhag. Variation-tolerant architectures for convolutional neural networks the near threshold voltage regime. In 2016 IEEE International Workshop on Signal Processing Systems (SiPS), pp. 17-22, 2016. doi: 10.
Y. Lin, C. Sakr, Y. Kim, and N. Shanbhag. Predictivenet: An energy-efficient convolutional neural network via zero prediction. In 2017 IEEE International Symposium on Circuits and Systems (ISCAS), pp. 1-4, 2017. doi: 10..119/SSAS... 2017.8050777.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. arXiv preprint Xivi 806.09055, 201 8a.
Sicong Liu, Yingyan Lin, Zimu Zhou, Kaiming Nan, Hui Liu, and Junzhao Du. On-demand deep model compression for mobile devices: A usage-driven model selection framework. MobiSys *18 pp. 389-400) Association for Computing Machinery, 2018h. ISBN 9781450357203.
Ningning Ma, Xiangyu Zhang, Hai-Tar Tao Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient enn architecture design. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 116-131, 2018.
Alberto Marchisio, Andrea Massa, Vojtech Mrazek, Beatrice Bussolino, Maurizio Martina, and Muhammad Shafique. Nascaps: framework for neural architecture search to optimize the accuracy and hardware efficiency convolutional capsule networks. arXiv preprint arXiv:2008.0847 2020.
11

---

Published as a conference paper at ICLR 2021
NVIDIA Inc.
NVIDIA Jetson TX2, matonomous--mchinnee embedded http:///ccdddeettttttttt/ttttttr accessed 2020-09-01.
https: www sividia..o com wocom//eee com/en-us/
NVIDIA Inc. Tensorrt, b.
NVIDIA Inc. Benchmark tx2 performance in googlenet with tensorrt,
Angshuman Parashar, Priyanka Raina, Yakun Sophia Shao, Yu-Hsin Chen, Victor A Ying. Anurag Mukkara, Rangharajan Venkatesan, Brucek Khailany, Stephen W Keckler, and Joel Emer. Timeloop: A systematic approach to dnn accelerator evaluation. In 2019 IEER international symposium performance analysis of systems and software (ISPASS), pp. 304-315. IEEE, 2019.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. In Advances neural information processing systems, pp. 8026-8037, 2019.
Patrick Mochel and Mike Murphy. sysfs - The filesystem for exporting kernel ob- jects.
https: /wwww//www/ kernel. http://////oooomeeaaaa////eet/te /sysfs
xtt accessed 2019-11-21.
Raspberry Pi Limited. Raspberry 4.
raspberry-pi- 4-model--------- 1-b/ accessed 2020-09-01.
Benjamin Ross. AI at the Edge Enabling a New Generation of Apps. Smart De-
vices, March 2020. URL https:://ww/tt
www.smaatttao wetss-mmart-ddvicce Bi-at-ther -i-at-thheedde--eneee enabl ing new wwwpaaaeeeeeraaiooo---ooee
Jianghao Shen, Yue Wang, Pengfei Xu, Yonggan Fu, Zhangyang Wang, and Yingyan Lin. Fractional
skipping: Towards finer-grained dynamic cnn inference. 2020.
Yongming Shen, Michael Ferdman, and Peter Milder. Maximizing cnn accelerator efficiency through resource partitioning. In Proceedings of the 44th Annual International Symposium on Computer Architecture, ISCA '17, pp. 535-547, New York, NY, USA, 2017. Association doi: 10. 01144//37995563080022 URL
for Computing Machinery. ISBN 978 4503488928
httts:://doi.rrg// wtt.:/oor/.....1///111 10411145/300 0145/3079856.300222
Mennatullah Siam, Mostafa Gamal, Moemen Abdel-Razek, Senthil Yogamani, Martin Jagersand, and Hong Zhang. A comparative study of real-time semantic segmentation for autonomous driving. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp. 587-597, 2018.
Julien Siems, Lucas Zimmer, Arber Zela, Jovita Lukasik, Margret Keuper, and Frank Hutter. Nas- bench-301 and the for surrogate benchmarks for neural architecture search. arXiv preprint arXiv:2008.0977 2020.
Dimitrios Stamoulis, Ruizhou Ding, Di Wang, Dimitrios Lymberopoulos, Bodhi Priyantha, Jie Liu, and Diana Marculescu. Single-path nas: Designing hardware-efficient convnets in less than 4 hours. Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 481-497. Springer, 2019.
Mingxing Tan and Quoe V Le. Efficientnet: Rethinking model scaling for convolutional neural networks. arXiv preprint arXiv:1905.1 1946, 2019.
Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoe V Le. Mnasnet: Platform-aware neural architecture search for mobile. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2820-2828, 2019.
Texas Instruments Inc. INA3221 Triple-Chamin High-Side Measurement, Shunt and Bus Voltage Monitor. http://wwwi http::/wwer ti. wtt.tt..ooooo...e.oooeeo
wwwm/pprodutt/tt..2222 accessed 2019-11-21.
12

---

Published as a conference paper at ICLR 2021
Alvin Wan, Xiaoliang Dai, Peizhao Zhang. Zijian He, Yuandong Tian, Saining Xie, Bichen Wu, Matthew Yu, Tao Xu, Kan Chen, et al. Fbnetv2: Differentiable neural architecture search for spatial and channel dimensions. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 12955-12974, 2020.
Chien Yao Wang, Hong- Yuan Mark Liao, Ping- Yang Chen, and Jun-Wei Hsich. Enriching variety of layer-wise learning information by gradient combination. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, Oct 2019a.
Y. Wang, Shen, T. K. Hu, Xu, Nguyen, Baraniuk, Z. Wang, and Lin. Dual dynamic inference: Enabling more efficient, adaptive, and controllable deep inference. IEEE Journal of Selected Topics in Signal Processing, (4(4):623-633 2020a. doi: 10. 10.1109//TSP222002297777
Y. Wang, J. Shen, T. K. Hu, P. Xu, T. Nguyen, R. Baraniuk, 7. Wang, and Y. Lin. Dual dynamic inference: Enabling more efficient, adaptive, and controllable deep inference. IEEE Journal of Selected Topics in Signal Processing, (33));223-33, 2020b. doi: 10.1 10.1109//TSP2222022299600
Yue Wang, Ziyu Jiang, Xiaohan Chen, Pengfei Xu, Yang Zhao, Yingyan Lin, and Zhangyang Wang- E2-Train: Training state-of-the-art cnns with over 80% energy savings. In Advances in Neural Information Processing Systems, pp. 5139-5151, 2019b.
Diana Wofk, Fangchang Ma, Tien-Ju Yang, Sertac Karaman, and Vivienne Sze. Fastdepth: Fast monocular depth estimation on embedded systems. In 2019 International Conference on Robotics and Automation (ICRA), pp. 6101-6108. IEEE, 2019.
Bichen Wu, Xiaoliang Dai. Peizhao Zhang, Yanghan Wang. Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer. Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 10734-10742, 2019.
Junru WU, Yue Wang, Zhenyu Wu, Zhangyang Wang, Ashok Veeraraghavan, and Yingyan Lin. Deep k-means: Re-training and parameter sharing with harder cluster assignments for compressing deep convolutions. arXiv preprint arXiv: 1tt.::1800.092288 2018
Yannan N. Wu, Joel S. Emer, and Vivienne Sze. Accelergy: An Architecture- Level Energy Estimation Methodology for Accelerator Designs. IEEE/ACM International Conference On Computer Aided Design (ICCAD). 2019.
In
Qingcheng Xiao, Yun Liang, Liqiang Lu, Shengen Yan, and Yu-Wing Tai. Exploring heterogencous
algorithms for accelerating deep convolutional neural networks on fpgas. In Proceedings of the
54th Annual Design Automation Conference 2017. DAC '17, New York, NY, USA, 2017, Asso- ciation for Computing Machinery. ISBN 978 450349277. doi: 10. 00.1445/9001169930624444 URL https://do/ waaoooo.orgg/000 1eee..11550900 639 3062244
Xilinx Inc. Vivado High-Level Synthesis, a.
bttps://http:///ww hwtp:/::://wwwwwt
com/
htt.:ttttte.ssaaeoo///o/o//aaaaa 2019-09-16.
accessed
Xilinx Inc. Xilinx zynq-7000 soc zc706 evaluation kit. https: wwwwwwwwwammmmmmm xi linx com/
b. (Accessed on 09/30/2020).
Yunyang Xiong, Hanxiao Liu, Suyog Gupta, Berkin Akin, Gabriel Bender, Pieter-Jan Kindermans, Mingxing Tan, Vikas Singh, and Bo Chen. Mobiledets: Searching for object detection architec- tures for mobile accelerators. arXiv preprint arXiv:2004.10 145525, 2020.
Antoine Yang, Pedro M. Esperança, and Fabio M. Carlucci. Nas evaluation is frustratingly hard. In International Conference Learning Representations, 2020. httptttss:///eeeeeeeeet
net/fooum?? dlt.orummmmmddyyookookokkk
Xuan Yang, Jing Pu, Blaine Burton Rister, Nikhil Bhagdikar, Stephen Richardson, Shahar Kvatin- sky, Jonathan Ragan-Kelley, Ardavan Pedram, and Mark Horowitz. A systematic approach to blocking convolutional neural networks, 2016.
13

---

Published as a conference paper at ICLR 2021
Chris Ying, Aaron Klein, Eric Christiansen, Esteban Real, Kevin Murphy, and Frank Hutter. Nas- bench-101: 101: Towards reproducible neural architecture search. International Conference on Machine Learning, pp. 7105-7114, 2019.
Haoran You, Xiaohan Chen, Yongan Zhang, Chaojian Li, Sicheng Li. Zihao Liu, Zhangyang Wang, and Yingyan Lin. Shiftaddnet: A hardware-inspired deep network. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.). Advances in Neural Information Processing Systems. volume 33. Pp. 2771-2783 Curran Asso- neur ps. ccs/paperr2222/F11ee
ciates, Inc., 2020a. - wbw.--apbbb------------- lcf44d7975e5c33e www.aeeecccaaaa0a55555555555
.ddf]
Shan You, Tao Huang, Mingmin Yang, Fei Wang, Chen Qian, and Changshui Zhang. Greedynas: Towards fast one-shot nas with greedy supernet. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1999-2008, 2020b.
Chen Zhang, Peng Li, Guangyu Sun, Yijin Guan, Bingjun Xiao, and Jason Cong. Optimiz- ing fpga-based accelerator design for deep convolutional neural networks. In Proceedings of the 2015 ACM/SIGDA International Symposium on Freed-Programmmale Gate Arrays, FPGA '15, pp. 161-170, New York, NY, USA, 2015. Association for Computing Machinery. ISBN doi: 10.1145/2684446.268888 2689060. URL https: http///ii..rg//...../ 1145/
0781450333153.
2684746. 2684746 2689060
Jianhao Zhang. Yingwei Pan, Ting Yao, He Zhao, and Tao Mei. dabnn: A super fast inference frame- work for binary neural networks on arm devices. In Proceedings of the 27th ACM International Conference Multimedia, pp. 2272-2275, 2019.
Xiaofan Zhang, Junsong Wang, Chao Zhu, Yonghua Lin. Jinjun Xiong, Wen-mei Hwu, and Deming Chen. Dnnbuilder: An automated tool for building high-performance dnn hardware accelerators for fpgas. In Proceedings of the International Conference on Computer-Are Aided Design, ICCAD '18, New York, NY, USA, 2018. Association for Computing Machinery. ISBN 9781450359504. doi: 10.1145/224066.3224800..
URL petps::/dci.orgeee /ttp://///...r/1111111113
3240765. 3240801
Yongan Zhang, Yonggan Fu, Weiwen Jiang, Chaojian Li, Haoran You, Meng Li, Vikas Chandra, and Yingyan Lin. Dna: Differentiable network--cccelertor co-search, 2020.
Cheah Wai Zhao, Jayanand Jegatheesan, and Son Chee Loon. Exploring iot application using rasp- berry pi. International Journal of Computer Networks and Applications, 2(1):27-34. 2015.
Y. Zhao, X. Chen, Y. Wang, C. Li. H. You, Y. Fu, Y. Xie, Z. Wang. and Y. Lin. Smartexchange: Trading higher-cost memory storage/access for lower-cost computation. In 2020 ACM/IEEE 47th Annual International Symposium on Computer Architecture (ISCA), pp. 954 2020a. doi: 10.1109/SSC44567.22200000 20.000000..
Y. Zhao, C. Li, Y. Wang. P. Xu, Y. Zhang, and Y. Lin. Dnn-chip predictor: An analytical performance predictor for dnn accelerators with various dataflows and hardware architectures. In ICASSP 2020 2020 IEEE International Conference Acousties, Speech and Signal Processing (ICASSP), pp.
1593-1597, 2020b. 202.b.
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le. Learning transferable architectures for scalable image recognition. In Proceedings the IEEE conference on computer vision and pattern recognition, pp. 8697-8710, 2018.
14

---

Published as a conference paper at ICLR 2021
A MORE VISUALIZATION ON THE MEASURED HARDWARE-COST FOR THE FBNET SEARCH SPACE
CIFAR-100
MeNsse ree Edige TPU Lateney
Cecamnn podd PaIneIo
T 0.85
JP 0.83
R 0.85
0.87
muu di Sanari Beaann
0.24
Approsimated Sdge Lateny
Approsimated Edge GPU Energy
Appmoinatedd Respl Latenyy
Aspr ocmmated Edge TPU Laisney
Approsimated Powi Lanency
ImageNet
Maneong soon oPa Energy
Assamunn weneenneeeeeee Eddge TPU Labaency
X 0.76
0.90
0.98
H
0.84
0.51
Approsimazed Edge OPÚ Latency
Approcmated Edge Energy
Appe oxmatee Lateney
Approstmind Edge TRU Laiinay
Approximiaze Latoney
Figure 6: Comparison between the approximated and measured hardware-cost on CIFAR-100 (Top) and ImageNet (Bottom), where the red line indicates the fitting line for all the measured data, and R2 represents the square of the Pearson Correlation Coefficient (Benesty et al. 2009).
Fig. 6 shows a comparison between the approximated and measured hardware-cost of randomly sampled 100 architectures when being executed on commercial edge devices using the ImageNet and CIFAR-100 datasets, which verifies that our approximation of summing up the performance of the unique blocks is simple yet quite accurate for providing the hardware-cost for networks the FBNet space and is consistent with our observation in Table 2
B COMPARING THE ESTIMATED COST EXECUTED ON EYERISS USING ACCELERGY++IMFLOOG AND DNN-CHIP REDICTOR
Both Accelergy [w et al. 2019 Timeloo (Parashar et al. 2019) and DNN-Chip Predictor (Zhao al. 2020b) are able to simulate the latency and energy cost of Hyeriss (Chen al. 2016), SOTA ASIC DNN accelerator, when giving the network architectures. From Table 6 they nearly give the same estimation for the latency and energy cost: specifically, the mean of their differences is 6.096%, the standard deviation of the differences is 0.779%. the Pearson correlation coefficient is 0.9998, and the Kendall Rank correlation coefficient is 0.9633, term of the average performance, when being benchmarked with NAS Be-S--Bech-222 on 3 datasets. Therefore, we use the average value of their predictions the estimated latency and energy on Eyeriss our proposed HW-NAS-Bende Bench.
Table 6: The differences of the hardware-cost estimation given by Accelergy (Wu et al.
2 Timeloop (Parashar al. 2019) and DNN-Chip Predictor (Zhao et al. 2020b). consider- ing NAS- Bench-201 on datasets.
Mean of Differences
Standard Deviation of Pearson Correlation
Kendall Rank Correlation Coefficient
Datasets
Hardwarc-cost
Differences
Coefficient
Latency
1.648%
0.642%
0.9999
0.9888
CIFAR-10
Energy
10.96%
0.9997
Latency
1,572%
0.611%
0.9888
CIFAR-100
Energy
10.93%
1.029%
0.9997
0.9374
ImageNNe16-120
Latency
1.338%
0.520%
0.9999
0.9888
Energy
10.13%
0.840%
0.9998
0.9388
Average Performance
6.096%
0.779%
0.9998
0.9633
15

---

Published as a conference paper at ICLR 2021
Table Left: the marco-archittctures of the search space proposed in the FBNet (Wu al. 2019) for the ImageNet classification; Right: our modified search space to fit the imput image size of the CIFAR-101 dataset. In the tables, "TBS" means the layer type needs be searched and "Stride" denotes the stride of the first block in the stage. Here the modified parameters are emphasized as bold characters.
Input Shape
Block
Filtert
Blocki
Stride
Input Shape
Block
Filtert
Block#
Stride
224
3 x 3 conv
16
2
32
3 conv
16
112 x 16
TBS
16
32a x 16
TBS
16
112º x 16
TBS
24
2
32 x 16
TBS
24
1
562 24
TBS
32
x 24
TBS
32
28a 32
16 x
TBS
14'
TBS
112
8
TBS
112
14 112
TBS
184
4
82 112
TBS
184
78 184
TBS
352
4 184
TBS
352
7 x 352
1 x 1 conv
1984
4 x 352
x 1 conv
1504
x 1984
7 x avgpool
4 x 1304
x avgpool
1000
1504
100
C MINOR MODIFICATIONS ON THE FBNET SEARCH SPACE WHEN BENCHMARKING ON CIFAR-100
Here describe modification the FBNet search space when benchmarking CIFAR-100 (i.e., the setting in Section 5) by comparing the marco- marco-architecttres before and after such modifica- tion in Table 7
D DETAILS OF THE PIPELINE USED TO COLLECT HARDWARE-COST DATA
D.1 COLLECT PERFORMANCE ON THE EDGE GPU
NVIDIA Edge GPU Jetson TX2 (Edge GPU) (NVIDIA Inc. a is a commonly used commercial
edge device, consisting of quad-core Arm Cortex-A57, dual-core NVIDIA Denver2, 256-core Pascal GPU, and a 8GB 128-bit LPDDR4, for various deep learning applications including classifi- cation (Li et al., 2020), segmentation (Siam et al.) 2018), and depth estimation (Wofk et al.. 2019)
targeting 1oT, and self-driving environments, Although widely-used TensorFlow (Abadi et al. 2016)
and PyTorch (Paszke et al. 2019) can directly used in Edge GPUs, achieve faster inference, TensorRT (NVIDIA Inc. b). C++ library for high- performance inference on NVIDIA GPUs, is more commonly used as the runtime environment Edge GPUs when only benchmarking inference performance (Wang et al. 2019a NVIDIA Inc.
We pre-set the Edge GPU the max-N mode to make full use of the resource on following Wofkk et al. 2019). When plugging Edge GPUs into the hardware-cost collection pipeline, we first com- pile the PyTorch implementations of the network architectures in both NAS-Bench, and FBNet search spaces to TensorRT format models. In this way, the resulting hardware-cost can benefit from the optimized inference implementation within the TensorRF runtime environment. And then benchmark the architectures in Edge GPUs further measure the energy and latency using the sysfs (Patrick Mochel and Mike Murphy.) of the embedded INA3221 (Texas Instruments Inc.) power rails monitor.
D.2 COLLECT PERFORMANCE ON RASPI 4
Raspherry 4 (Raspi 4) (Rasbberry Limited.) the latest Raspberry device, which is popular
hardware platform for general purpose IoT applications (Zhao et al.) 2015) Basu et al. 2020) and is able to support deep learning applications with specifical framework designs (Google LLC. Zhang
et al. 2019 Geiger & Team 2020). We choose the type of Raspi 4 with a Broadcom BCM271 SoC and a 4GB LPDDR4 (Raspberry Limited.). Similar to Edge GPUs, Raspi 4 can run architectures in the TensorFlow (Abadi et al. (016)), Py Torch (Paszke et al. 2019). TensorFlow Lite (Google LLC. [ runtime environments. We utilize TensorFlow Lite (Google LLC. as it can further boost the inference efficiency.
16

---

Published as a conference paper at ICLR 2021
To collect hardware-cost operating Respi 4, official TensorFlow Lite interpreter is pre- configured in the Raspi 4, following the settings in (Google LLC, 2020). We benchmark the possi- ble architectures in HW-NAS-Boc Beschh Raspi 4 after compiling them to the TensorFlow Lite (Abadi et al. 2016) format to measure the resulting latency.
D.3 COLLECT PERFORMANCE ON THE EDGE TPU
Edge TPU (Google LLC. is series of dedicated ASIC accelerators developed by Google, target- ing AI inference at the edge, which can be used for classification, pose estimation, and segmenta- tion (Xiong al. 2020 Google LLC. b with extremely high efficiency (e.g., 2.32x more efficient than a single SOTA desktop GPU, GTX 2080 Ti, in terms of the number of fixed-point operations per watt (Google L.C.I d). In our proposed collection pipeline, we choose the Dev Board (Google LLC.
a which provides the most functions among all products.
To collect hardware-cost in Edge TPUs, all the architectures to be benchmarked will first be con- verted to the TensorFlow Lite (Google LLC. 1t format from their Keras (Chollet et al. 2013) imple- mentation. After that, an in-house compiler (Google LLC. ())
will be used convert the TensorFlow Lite models into more compressed format. This pipeline uses the least converting tools to make sure that the most operations are supported, as compared to other options (e.g., converting from the PyTorch-ONNX (Bai et al. 2020) implementation). Only the latency is collected on the Edge TPU since it lacks accurate embedded power rails monitor. We do not consider the FBNet's search space for the Edge TPU, and more details are in the Appendix
D.4 COLLECT PERFORMANCE ON PIXEL
Pixel 3 (Google LLC. is one of the latest Pixel mobile phones that widely used as the target platform by recent NAS works (Xiong et al., 2020 Howard et al.] 2019 Tan et al.] 2019) and machine learning framework benehmark (Google LLC.TDD. In implementation, the Pixel 3 is pre-configured to use its big cores following the setting in (Xiong et al. 2020 Tan et al. 2019). Similar the case of Raspi 4, we first convert the possible architectures the search spaces of our proposed HW-NAS-Bese binary files to measure the latency for each architecture.
into the TensorFlow Lite format and then use the official benchmark
D.5 COLLECT PERFORMANCE ON ASIC-EYERISS
For hardware-coss data collection in ASIC, we consider Eyeriss (ASIC-Eyeriss) which is a SOTA ASIC accelerator (Chen al., 2016). The Eyeriss chip features 168 processing elements (PEs) which are connected through a configurable dedicated on-chip network into a 2D array. A 128KB SRAM is shared by all PEs and further divided into multiple banks, each of which can be assigned to fit the input feature maps or partial sums. Thanks to these configurable hardware settings, we can adopt the optimal algorithm-to-hant ha-ddaaare mappings for different network architectures when being executed on Eyeriss to minimize the energy or latency by maximizing data reuse opportunities for different layers.
In order to find the optimal mappings and evaluate the performance metrics on Eyeriss, adopt SOTA performance simulators for DNN accelerators (1) Accelergy (Wu al. 2019)+Timeloop (Parashar et al., 2019) and (2) DNN-Chip Predictor (Zhao et al. 2020b). Both of the simulators can characterize the Eyeriss's and predict the energy cost and latency metrics. Given the Eyeriss accelerator and layer information (e.g, layer type, feature map size, and kernel size) in both WAS-Bench-200 and FBNet, Accel- ergy+Timeloop reports the energy cost and latency characterization through an integrated mapper that finds the optimal mapping for such layer when being executed in Eyeriss. The inputs to DNN- Chip Predictor are the same as those to scclergyyTTmmelowe product. DNN-Chin Chip Predictor identifies the optimal mapping
archtt-rchitecture, perform mapping exploration,
except that we can set the optimization
metric as energy/ateency/eeegy--eee for the optimization metric and generates the estimated hardware-cost. We report the average pre- diction from the two simulators as the estimated hardware-coss of Eyeriss, and more details can be found in Appendix B
17

---

Published as a conference paper at ICLR 2021
Table 8: Our implemented FPGA accelerators for HW-NAS-Bema vs. SOTA FPGA accelerators,
considering VGGI6 on the ImageNet dataset and using Zynq XC70745 as the FPGA device.
7hang al. 2018
Xiao et al. 2017] Our Implementation
Resource Utilization
680/900 DSP
824/900 DSP
723/900 DSP
Performance (GOP/s)
262
230
291
D.6 COLLECT PERFORMANCE ON FPGA
FPGA widely adopted AI acceleration platform which can offer higher flexibility in terms of the hardware resources for accelerating AI algorithms. For collecting hardware-cost data in FPGA, we construct SOTA chunk based pipeline structure (Zhang et al. 2018 [Shen et al.) 2017 as our FPGA implementation. By configuring multiple sub-accelerators (chunks) and assigning different layers to different sub-accelerators consumption. To further free up our implantation's 's potential to reach the performance frontier across different architectures, we additionally configure hardware settings such as the number of PEs, in- terconnection method of PEs, and tiling/scheduling of the operations, which are commonly adopted by FPGA accelerators (Chen et al. 2017 Zhang et al. 2013) Yang et al. (0066). We then compile all the architectures using the standard Vivado HIS toolflow (Xilinx Inc. a) and obtain the bottleneck latency, the maximum latency across all sub-accelerators (chunks) of the architectures on Xilinx ZC706 development board with Zynq XC7045 SoC (Xilinx Inc. b).
we can balance the throughput and hardware resource
To verify our implementation, we compare our implementation's performance with SOTA FPGA accelerators (Zhang et al.) 2018 Xiao al.) 2017) given the same architecture and dataset as shown in Table 8 We can see that our implementation achieves SOTA performance and thus provides insightful and trusted hardware- estimation for the EW-NAS-Benndh
18


Reviews:
[{"review_id": "sXqr5ERL2S", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Contributions ###\n* The paper proposes a benchmark for hardware-aware neural architecture search (HW-NAS). For this, the authors adopt two popular search spaces (NAS-Bench-201 and FBNet) and measure/estimate hardware performance metrics such as energy costs and latency for six hardware devices (spanning commercial edge devices, FPGA, and ASIC) for all architectures in this search spaces.\n*The authors also study the rank correlation of the architectures from the  NAS-Bench-201 space regarding different hardware metrics (including both measured ones and theoretic ones like FLOPs) and find several pairs with low rank correlation. This justifies that a generic theoretic hardware metric like FLOPs is not sufficient as proxy for all practically relevant metrics.\n\n### Significance ###\nHW-NAS is an important area of research, in particular for bringing powerful DL models to edge devices and for reducing energy consumption. While the lack of generic NAS benchmarks has been addressed recently, the same did not hold true for HW-NAS. Thus, the proposed HW-NAS-Bench fills an important gap and can prove to be very useful for practitioners and HW-NAS researchers.\n\n### Originality ###\nThe design of HW-NAS-Bench is mostly straight-forward in that it builds upon established search spaces and NAS benchmarks and \"only\" estimates hardware metrics such as latency and energy consumption on target hardware. However, this \"only\" of course encompasses a significant effort, in particular since six very different target hardwares are covered. I highly appreciate this effort! I hope that the author's statement \"All the codes and data will be released publicly upon acceptance\" also includes the code for conducting the measurements. This code would be potentially very valuable for practitioners that plan to estimate hardware costs for different search spaces or devices.\n\n### Clarity ###\nIn general, summary of the design of HW-NAS-Bench and how hardware metrics are measured is outlined very clearly.\nThe clarification on how hardware costs for the huge FBNet search space are estimated (Appendix A) should be part of the main paper, however. Table 5 contains also relevant justification for this way of estimating. However, it is unclear to me why the authors use Pearson correlation rather than rank correlation here.\n\n### Quality ###\nThe authors make a convincing case that is is not sufficient to consider theoretic hardware metrics like FLOPs for ranking different architectures since the rank correlation with respect to FLOPS and practical hardware metrics such as latency can be quite low. \n\nHowever, for a NAS benchmark, the point is not so much on comparing individual architectures but rather comparing different NAS methods (that is the architectures they select from the search space). And from the paper, it is not clear that the ranking of different NAS methods would be different when using FLOPs as hardware metric compared to using  latency or energy consumption. HW-NAS-Bench is a good basis for analysing this and the paper would be strengthened by some initial results on comparing HW-NAS methods on the benchmark.\n\nMoreover, and related to the point above, it is not really clear how to rank different NAS methods in the proposed benchmark since there is no full evaluation protocol. Two things would need clarification: (a) how would one compare Pareto fronts of different HW-NAS methods in the accuracy-hardware metric space, in particular when they intersect? (b) since there are now very many hardware metrics (latency + energy consumption for six different target devices), a way to aggregate these metrics into a single \"average hardware metric\" would be helpful. Without (a) and (b) it is not clear how one could actually benchmark HW-NAS methods on HW-NAS-Bench.\n\n### Recommendation ###\nIn summary, I think the proposed HW-NAS-Bench will prove useful for HW-NAS development. I thus lean towards accepting the paper, in particular if the points raised above would be adressed.\n\n### Recommendation after Author Response ###\nI have read the author response and appreciate the effort spent by the authors on this response. My main criticism was addressed and the author's feedback is very convincing. The authors have not yet added this additional content to the paper. Assuming they will include it in the final version,  I am confident that this paper will meet all standards of ICLR and recommend acceptance. I increase my score accordingly to 7.", "extra_scores": {}, "extra_text": {}}, {"review_id": "l8fr_1eQPSf", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents a benchmark / dataset, HW-NAS-Bench, for evaluating various neural architecture search algorithms. The benchmark is based on extensive measurements on real hardware. An important goal with the proposal is to support neural architecture searches for non-hardware experts. Further, the paper provides a good overview of related work in the domain. \n\nThe paper has a very good intention, i.e., to help and support non-hardware experts in the neural architecture search process. I think the paper contributes a lot to that ambition, by providing a benchmark / dataset of hardware-aware measurements / predictions that can be queried either by a person or a NAS algorithm.\n\nThe network architectures that provide the search space are NAS-Bench-201 and FBNet, and the measurements/predictions are obtained from three categories of devices, i.e., commercial edge devices, FPGAs, and ASICs). Thus, my belief is that the proposed dataset / benchmark covers a significant and representative part of the most common targets for NAS algorithms. Further, I like that the authors will publish their measurement results / estimated hardware costs for over 46000 network/hardware combinations.\n\nThe work presented in the paper is important and can potentially have a significant impact, both in industry as well as in academia. \n\nSome other comments / questions:\n* I really don't understand the idea behind Table 4. The performance is best when we run on the same hardware as we have optimized for? Am I missing something here?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "G7Ei7m0q13S", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Note**: I updated my review score after the original review was written. See comments below for details.\n\n## Summary\nOne important application of Neural Architecture Search is to find neural network architectures with good accuracy/inference time or accuracy/energy tradeoffs on a specific hardware device. However, the submission convincingly argues that many existing NAS benchmarks focus only on accuracy, or only provide very limited data about inference times.\n\nIn my view, the submission's main contribution is a promise to publicly release inference time and power usage measurements and code for 5-6 different hardware devices on two existing NAS benchmark tasks: NASBench-201 and FBNet. The submission also provides some analyses on this data. For example: the authors measure the correlation between inference times for the same network architectures on different hardware devices.\n\nThe authors convincingly argue that properly performing on-device inference time/energy benchmarks properly is challenging for practitioners because it \"requires various hardware domain knowledge including machine learning development frameworks, device compilation, embedded systems, and device measurements.\" This is a major motivation for the paper and associated datasets, which present the use of pre-computed latency measurements as an easier alternative for ML researchers.\n\n**Pros**\n* **The proposed dataset seems useful for research on hardware-aware NAS algorithms.** The authors' datasets, which contain inference time measurements for several different hardware devices, should make it easier for researchers to experiment with NAS algorithms for finding better accuracy/inference time tradeoffs.\n* **The paper contains some interesting analyses.** I particularly liked Figure 5; the authors identify Pareto-optimal architectures on Edge GPU and show the accuracy/inference time tradeoffs for these same devices on other hardware platforms.\n\n**Cons**\n* **The submission calls itself a \"unified benchmark for HW-NAS,\" which may be a bit misleading.** Other NAS benchmark papers like NASBench-101 and NASBench-201 try to ensure that benchmark results from different researchers are comparable to each other; the submission does not. For example: two papers could both use the data from HW-NAS-Bench but produce incomparable results if searched for network architectures with different inference times.\n* **I'm not sure what FBNet-related code is publicly available / will be released, and would like a clarification.**  While I was able to find official reproductions of specific FBNet models on github, I'm not sure whether there's an official open-source implementation of the search space itself. I'm hoping the authors can clarify, since releasing raw benchmark numbers for FBNet may not be very useful unless they're accompanied by code that can train/evaluate any architecture in the FBNet search space. If there's an official implementation, I hope the authors can provide a pointer. If the authors are using their own reproduction of the search space, I'd like to understand what they've done to verify the correctness of their implementation. (Ditto for NASBench-201 if the authors are not using the official implementation.)\n* **For the FBNet search space, the information about correlations between predicted and true latency measurements is quite limited.** The author provide Pearson correlations on a random sample of architectures in Appendix A, but additional information (e.g., plotting predicted vs. true latency for a random sample of architectures) would strengthen the results. In addition, Appendix A only includes one \"Pearson correlation\" measurement per hardware device, and it's not clear to me whether this number is for the authors' CIFAR-10 benchmark, their ImageNet benchmark, or the union of the two. Breaking down the measurements and providing separate CIFAR-10 and ImageNet numbers would make this analysis stronger.\n\nIn addition: the usefulness of the authors' code/dataset in practice will largely depend on how easy-to-use/well-designed the code library for querying inference times is, and the paper doesn't contain enough information for me to evaluate this. This is a limitation of the review process.\n\n**Notes on Rating:** I've given the paper a borderline score (5) in my initial review, due to the open questions mentioned in the \"cons\" section above. However, I believe proposed dataset could be a valuable contributions to the ML research community, and would lean toward accepting the paper if the concerns are suitably addressed.\n\n## Experiments presented in paper\nThe paper promises to release on-device inference time measurements for NASBench-201, as well as a lookup-table based inference time prediction model for FBNet. For NASBench-201, measurements are provided on Edge GPU (NVIDIA Jetson TX2), Raspberry Pi 4, Edge TPU, Pixel 3, and ASIC-Eyeriss). For FBNet, latency it appears that the same devices are used, except that Edge TPU is omitted. (Although I could not find a direct explanation, Appendix A suggests that Edge TPU was excluded because a latency table-based model was not very predictive of on-device measurements.)\n\nIn addition to the raw benchmark numbers, the submission presents some experiments / sanity checks on these benchmarks (Section 4):\n* Table 2: Rank correlations between model FLOPS/Parameter counts and on-device latency/energy measurements.\n* Figure 3: Rank correlations for the inference latencies of the same model on different hardware devices.\n* Figure 5: Taking network architectures which have pareto-optimal accuracy/latency tradeoffs on Edge GPU, and evaluating how close to optimal the network architectures are on Edge GPU in the NASBench-201 search space.\n\nIn addition, the authors present results from running three architecture searches using ProxylessNAS with different target hardware devices. (Section 5.1).\n\nWhile Figures 2 and 3 and Section 5.1 mirror similar results from earlier papers like ProxylessNAS and FBNet, I still think they're valuable because they successfully validate earlier experimental claims on new search spaces and target hardware devices.\n\n## Clarity\nIn general, the paper seems clear and well-organized. While the authors generally did a good job of proof-reading, I did notice a few minor typos. For example: the Section 2.1 title says \"HareWare\" instead of \"Hardware\"; in Section 3.2 under \"Edge TPU\", \"runitime\" should be changed to \"runtime\"; and in Appendix D, \"TensorFLow\" should be \"TensorFlow\".\n\n## Additional Comments\nThe authors provide detailed information about their experimental setups in in Appendix D. I did my best to spot-check these descriptions, and the descriptions looked reasonable to me. However, I don't have enough experience with on-device benchmarks to independently certify that the benchmarks were performed correctly.\n\nThe submission includes a promise that \"all the codes and data will be released publicly upon acceptance.\" I consider this to be a major contribution of the paper, and the paper would need to be reviewed again if this promise cannot be fulfilled for any reason.", "extra_scores": {}, "extra_text": {}}, {"review_id": "5GyZz4Jxb-f", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths\n1. Analysis of different NAS algorithm and search space\n2. Comparison of measurement vs. estimation for different hardware systems\n\nWeaknesses\n1. The analyzed parameters are absolute\n2. The set of analyzed hardware is limited\n3. There are few mentions of the analyzed Deep Learning models\n\nMajor Comments\n1. How would a configuration not present in the benchmark be handled? Via interpolation or returning an error?\n2. In Section 3, how long does it take to run all the measurements and estimations? While this is a one-time cost, if the benchmark requires improvements this cost is paid again.\n3. In Section 3, there are very few mentions of the analyzed models: it seems the focus is on vision networks, mostly using convolutional layers, but there are other network types which are rising to prominence and for which the hardware is optimized, such as transformers. It is suggested to add more details regarding the use of other network types, or at least analyze this different domain of NAS to provide a proper justification.\n4. In Section 3.2, while the set of chosen hardware spans multiple devices and targets, it may be limited towards the “fixed” devices, such as mobile phones and edge PCs, as only a handful of them are analyzed. While these examples may be representative, they could not cover the whole search space and characteristics, limiting the applicability of the benchmark in real-world scenarios.\n5. In Section 4, when analyzing the different hardware systems, there is the usage of absolute characteristics, such as FLOPs and latency, why are other relative characteristics, such as arithmetic intensity, not being considered? They could provide a better estimate and means of comparison, especially since the set of hardware is very wide, covering the whole intensity spectrum.\n\nRelated Work Suggestions\n1. A. Marchisio, A. Massa, V. Mrazek, B. Bussolino, M. Martina, M. Shafique, “ NASCaps: A Framework for Neural Architecture Search to Optimize the Accuracy and Hardware Efficiency of Convolutional Capsule Networks”, to appear at The IEEE/ACM 2020 International Conference On Computer Aided Design (ICCAD), November 2020\n\nMinor Comments\n1. In Section I, in Figure 1, the text and the figures are difficult to read, as they should have a slightly bigger font size.\n2. In Section 4.2, Figures 3 and 4, the axes are difficult to follow, especially since they are not repeated for the other graphs in the figures.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a new NAS benchmarks for hardware-aware NAS. For each of the architectures in the search space of NAS-Bench-201, it measures hardware performance (energy cost and latency) for six different hardware devices. This is extremely useful for the NAS research community, since it takes very specialized hardware domain knowledge (including machine learning development frameworks, device compilation, embedded systems, and device measurements) as well as the hardware to make these hardware-aware measurements on as many as six (very different) devices. 

The code has been made available to the reviewers during the author response window and has been checked by the reviewers in the meantime. All reviewers appreciated the paper and gave (clear) acceptance scores. 

Before this work, it was very hard for the average NAS researcher to assess their method properly in a hardware-aware setting, and I expect this work to change this, and to open up the very important field of hardware-aware NAS to many more researchers. For this reason I recommend to accept this paper as a spotlight.


Decision: Accept (Spotlight)