Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: The paper presents a benchmark / dataset, HW-NAS-Bench, for evaluating various neural architecture search algorithms. The benchmark is based on extensive measurements on real hardware. An important goal with the proposal is to support neural architecture searches for non-hardware experts. Further, the paper provides a good overview of related work in the domain. 

The paper has a very good intention, i.e., to help and support non-hardware experts in the neural architecture search process. I think the paper contributes a lot to that ambition, by providing a benchmark / dataset of hardware-aware measurements / predictions that can be queried either by a person or a NAS algorithm.

The network architectures that provide the search space are NAS-Bench-201 and FBNet, and the measurements/predictions are obtained from three categories of devices, i.e., commercial edge devices, FPGAs, and ASICs). Thus, my belief is that the proposed dataset / benchmark covers a significant and representative part of the most common targets for NAS algorithms. Further, I like that the authors will publish their measurement results / estimated hardware costs for over 46000 network/hardware combinations.

The work presented in the paper is important and can potentially have a significant impact, both in industry as well as in academia. 

Some other comments / questions:
* I really don't understand the idea behind Table 4. The performance is best when we run on the same hardware as we have optimized for? Am I missing something here?

review_title: The paper presents a dataset, HW-NAS-Bench, for evaluating neural architecture search algorithms. Based on real measurements, HW-NAS-Bench provides a valuable tool for NN designers.

final_score: rating: 7: Good paper, accept
review_content: Strengths
1. Analysis of different NAS algorithm and search space
2. Comparison of measurement vs. estimation for different hardware systems

Weaknesses
1. The analyzed parameters are absolute
2. The set of analyzed hardware is limited
3. There are few mentions of the analyzed Deep Learning models

Major Comments
1. How would a configuration not present in the benchmark be handled? Via interpolation or returning an error?
2. In Section 3, how long does it take to run all the measurements and estimations? While this is a one-time cost, if the benchmark requires improvements this cost is paid again.
3. In Section 3, there are very few mentions of the analyzed models: it seems the focus is on vision networks, mostly using convolutional layers, but there are other network types which are rising to prominence and for which the hardware is optimized, such as transformers. It is suggested to add more details regarding the use of other network types, or at least analyze this different domain of NAS to provide a proper justification.
4. In Section 3.2, while the set of chosen hardware spans multiple devices and targets, it may be limited towards the “fixed” devices, such as mobile phones and edge PCs, as only a handful of them are analyzed. While these examples may be representative, they could not cover the whole search space and characteristics, limiting the applicability of the benchmark in real-world scenarios.
5. In Section 4, when analyzing the different hardware systems, there is the usage of absolute characteristics, such as FLOPs and latency, why are other relative characteristics, such as arithmetic intensity, not being considered? They could provide a better estimate and means of comparison, especially since the set of hardware is very wide, covering the whole intensity spectrum.

Related Work Suggestions
1. A. Marchisio, A. Massa, V. Mrazek, B. Bussolino, M. Martina, M. Shafique, “ NASCaps: A Framework for Neural Architecture Search to Optimize the Accuracy and Hardware Efficiency of Convolutional Capsule Networks”, to appear at The IEEE/ACM 2020 International Conference On Computer Aided Design (ICCAD), November 2020

Minor Comments
1. In Section I, in Figure 1, the text and the figures are difficult to read, as they should have a slightly bigger font size.
2. In Section 4.2, Figures 3 and 4, the axes are difficult to follow, especially since they are not repeated for the other graphs in the figures.
review_title: This paper analyzes the existing landscape of Neural Architecture Searches and summarizes a set of hardware and software parametrization with the aim of creating a standardized benchmark for NAS algorithms. 

final_score: rating: 6: Marginally above acceptance threshold
review_content: **Note**: I updated my review score after the original review was written. See comments below for details.

## Summary
One important application of Neural Architecture Search is to find neural network architectures with good accuracy/inference time or accuracy/energy tradeoffs on a specific hardware device. However, the submission convincingly argues that many existing NAS benchmarks focus only on accuracy, or only provide very limited data about inference times.

In my view, the submission's main contribution is a promise to publicly release inference time and power usage measurements and code for 5-6 different hardware devices on two existing NAS benchmark tasks: NASBench-201 and FBNet. The submission also provides some analyses on this data. For example: the authors measure the correlation between inference times for the same network architectures on different hardware devices.

The authors convincingly argue that properly performing on-device inference time/energy benchmarks properly is challenging for practitioners because it "requires various hardware domain knowledge including machine learning development frameworks, device compilation, embedded systems, and device measurements." This is a major motivation for the paper and associated datasets, which present the use of pre-computed latency measurements as an easier alternative for ML researchers.

**Pros**
* **The proposed dataset seems useful for research on hardware-aware NAS algorithms.** The authors' datasets, which contain inference time measurements for several different hardware devices, should make it easier for researchers to experiment with NAS algorithms for finding better accuracy/inference time tradeoffs.
* **The paper contains some interesting analyses.** I particularly liked Figure 5; the authors identify Pareto-optimal architectures on Edge GPU and show the accuracy/inference time tradeoffs for these same devices on other hardware platforms.

**Cons**
* **The submission calls itself a "unified benchmark for HW-NAS," which may be a bit misleading.** Other NAS benchmark papers like NASBench-101 and NASBench-201 try to ensure that benchmark results from different researchers are comparable to each other; the submission does not. For example: two papers could both use the data from HW-NAS-Bench but produce incomparable results if searched for network architectures with different inference times.
* **I'm not sure what FBNet-related code is publicly available / will be released, and would like a clarification.**  While I was able to find official reproductions of specific FBNet models on github, I'm not sure whether there's an official open-source implementation of the search space itself. I'm hoping the authors can clarify, since releasing raw benchmark numbers for FBNet may not be very useful unless they're accompanied by code that can train/evaluate any architecture in the FBNet search space. If there's an official implementation, I hope the authors can provide a pointer. If the authors are using their own reproduction of the search space, I'd like to understand what they've done to verify the correctness of their implementation. (Ditto for NASBench-201 if the authors are not using the official implementation.)
* **For the FBNet search space, the information about correlations between predicted and true latency measurements is quite limited.** The author provide Pearson correlations on a random sample of architectures in Appendix A, but additional information (e.g., plotting predicted vs. true latency for a random sample of architectures) would strengthen the results. In addition, Appendix A only includes one "Pearson correlation" measurement per hardware device, and it's not clear to me whether this number is for the authors' CIFAR-10 benchmark, their ImageNet benchmark, or the union of the two. Breaking down the measurements and providing separate CIFAR-10 and ImageNet numbers would make this analysis stronger.

In addition: the usefulness of the authors' code/dataset in practice will largely depend on how easy-to-use/well-designed the code library for querying inference times is, and the paper doesn't contain enough information for me to evaluate this. This is a limitation of the review process.

**Notes on Rating:** I've given the paper a borderline score (5) in my initial review, due to the open questions mentioned in the "cons" section above. However, I believe proposed dataset could be a valuable contributions to the ML research community, and would lean toward accepting the paper if the concerns are suitably addressed.

## Experiments presented in paper
The paper promises to release on-device inference time measurements for NASBench-201, as well as a lookup-table based inference time prediction model for FBNet. For NASBench-201, measurements are provided on Edge GPU (NVIDIA Jetson TX2), Raspberry Pi 4, Edge TPU, Pixel 3, and ASIC-Eyeriss). For FBNet, latency it appears that the same devices are used, except that Edge TPU is omitted. (Although I could not find a direct explanation, Appendix A suggests that Edge TPU was excluded because a latency table-based model was not very predictive of on-device measurements.)

In addition to the raw benchmark numbers, the submission presents some experiments / sanity checks on these benchmarks (Section 4):
* Table 2: Rank correlations between model FLOPS/Parameter counts and on-device latency/energy measurements.
* Figure 3: Rank correlations for the inference latencies of the same model on different hardware devices.
* Figure 5: Taking network architectures which have pareto-optimal accuracy/latency tradeoffs on Edge GPU, and evaluating how close to optimal the network architectures are on Edge GPU in the NASBench-201 search space.

In addition, the authors present results from running three architecture searches using ProxylessNAS with different target hardware devices. (Section 5.1).

While Figures 2 and 3 and Section 5.1 mirror similar results from earlier papers like ProxylessNAS and FBNet, I still think they're valuable because they successfully validate earlier experimental claims on new search spaces and target hardware devices.

## Clarity
In general, the paper seems clear and well-organized. While the authors generally did a good job of proof-reading, I did notice a few minor typos. For example: the Section 2.1 title says "HareWare" instead of "Hardware"; in Section 3.2 under "Edge TPU", "runitime" should be changed to "runtime"; and in Appendix D, "TensorFLow" should be "TensorFlow".

## Additional Comments
The authors provide detailed information about their experimental setups in in Appendix D. I did my best to spot-check these descriptions, and the descriptions looked reasonable to me. However, I don't have enough experience with on-device benchmarks to independently certify that the benchmarks were performed correctly.

The submission includes a promise that "all the codes and data will be released publicly upon acceptance." I consider this to be a major contribution of the paper, and the paper would need to be reviewed again if this promise cannot be fulfilled for any reason.
review_title: Paper Review

final_score: rating: 7: Good paper, accept
review_content: ### Contributions ###
* The paper proposes a benchmark for hardware-aware neural architecture search (HW-NAS). For this, the authors adopt two popular search spaces (NAS-Bench-201 and FBNet) and measure/estimate hardware performance metrics such as energy costs and latency for six hardware devices (spanning commercial edge devices, FPGA, and ASIC) for all architectures in this search spaces.
*The authors also study the rank correlation of the architectures from the  NAS-Bench-201 space regarding different hardware metrics (including both measured ones and theoretic ones like FLOPs) and find several pairs with low rank correlation. This justifies that a generic theoretic hardware metric like FLOPs is not sufficient as proxy for all practically relevant metrics.

### Significance ###
HW-NAS is an important area of research, in particular for bringing powerful DL models to edge devices and for reducing energy consumption. While the lack of generic NAS benchmarks has been addressed recently, the same did not hold true for HW-NAS. Thus, the proposed HW-NAS-Bench fills an important gap and can prove to be very useful for practitioners and HW-NAS researchers.

### Originality ###
The design of HW-NAS-Bench is mostly straight-forward in that it builds upon established search spaces and NAS benchmarks and "only" estimates hardware metrics such as latency and energy consumption on target hardware. However, this "only" of course encompasses a significant effort, in particular since six very different target hardwares are covered. I highly appreciate this effort! I hope that the author's statement "All the codes and data will be released publicly upon acceptance" also includes the code for conducting the measurements. This code would be potentially very valuable for practitioners that plan to estimate hardware costs for different search spaces or devices.

### Clarity ###
In general, summary of the design of HW-NAS-Bench and how hardware metrics are measured is outlined very clearly.
The clarification on how hardware costs for the huge FBNet search space are estimated (Appendix A) should be part of the main paper, however. Table 5 contains also relevant justification for this way of estimating. However, it is unclear to me why the authors use Pearson correlation rather than rank correlation here.

### Quality ###
The authors make a convincing case that is is not sufficient to consider theoretic hardware metrics like FLOPs for ranking different architectures since the rank correlation with respect to FLOPS and practical hardware metrics such as latency can be quite low. 

However, for a NAS benchmark, the point is not so much on comparing individual architectures but rather comparing different NAS methods (that is the architectures they select from the search space). And from the paper, it is not clear that the ranking of different NAS methods would be different when using FLOPs as hardware metric compared to using  latency or energy consumption. HW-NAS-Bench is a good basis for analysing this and the paper would be strengthened by some initial results on comparing HW-NAS methods on the benchmark.

Moreover, and related to the point above, it is not really clear how to rank different NAS methods in the proposed benchmark since there is no full evaluation protocol. Two things would need clarification: (a) how would one compare Pareto fronts of different HW-NAS methods in the accuracy-hardware metric space, in particular when they intersect? (b) since there are now very many hardware metrics (latency + energy consumption for six different target devices), a way to aggregate these metrics into a single "average hardware metric" would be helpful. Without (a) and (b) it is not clear how one could actually benchmark HW-NAS methods on HW-NAS-Bench.

### Recommendation ###
In summary, I think the proposed HW-NAS-Bench will prove useful for HW-NAS development. I thus lean towards accepting the paper, in particular if the points raised above would be adressed.

### Recommendation after Author Response ###
I have read the author response and appreciate the effort spent by the authors on this response. My main criticism was addressed and the author's feedback is very convincing. The authors have not yet added this additional content to the paper. Assuming they will include it in the final version,  I am confident that this paper will meet all standards of ICLR and recommend acceptance. I increase my score accordingly to 7.
review_title: Review for HW-NAS-Bench

Meta-review: This paper presents a new NAS benchmarks for hardware-aware NAS. For each of the architectures in the search space of NAS-Bench-201, it measures hardware performance (energy cost and latency) for six different hardware devices. This is extremely useful for the NAS research community, since it takes very specialized hardware domain knowledge (including machine learning development frameworks, device compilation, embedded systems, and device measurements) as well as the hardware to make these hardware-aware measurements on as many as six (very different) devices. 

The code has been made available to the reviewers during the author response window and has been checked by the reviewers in the meantime. All reviewers appreciated the paper and gave (clear) acceptance scores. 

Before this work, it was very hard for the average NAS researcher to assess their method properly in a hardware-aware setting, and I expect this work to change this, and to open up the very important field of hardware-aware NAS to many more researchers. For this reason I recommend to accept this paper as a spotlight.


Decision: Accept (Spotlight)