Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper provides a large-scale study of hardware metric predictors on 2 recent tabular benchmarks: Trans-NAS-Bench-101 and HW-NAS-Bench. The authors compare the predictors in terms of rank correlation and their ability to simulate the true pareto front and demonstrate that model-based predictors outperform Lookup Tables with very few training data points and that simple MLPs can outperform the more specialized models such as GNNs.
main_review: While I think that the motivation to have a large-scale study of hardware metric predictors similar as White et al. do for performance predictors is a reasonable one, however I think that this paper needs to improve its presentation and clarity. There is a lot of text (e.g. in Sections 3, 4 and 5) which might be compressed or moved to the appendix, adding space for additional experiments to be conducted. Considering that the novelty and the results presented in the paper are not surprising, I think a more thorough empirical evaluation is necessary.

These are some of my questions and recommendations:

- I am quite confused by the experiments and results in Section 6. In Figure 4, how do you obtain the simulated pareto front? Do you evaluate all the architectures in the space using the predictor (for both accuracy and hardware metric)? 
- What are the "selected architectures"? Are they the architectures in the "simulated pareto front" evaluated on the true HW-NAS-Bench?
- The main purpose of tabular NAS benchmarks, including HW-NAS-Bench that is for hardware metrics, is to provide a fast way to prototype and evaluate NAS algorithms, which will ultimately be evaluated on real world benchmarks. Therefore, I think, for the case at hand, it would be useful to evaluate the trained predictors on a setting where they are used in a multi-objective NAS algorithm together with the performance predictors.
- I found many of the plots to be redundant, e.g. the right-hand side plots in Fig 1 and 2. I think the authors can gain some space by moving them to the appendix and adding additional experimental results in the main paper.
summary_of_the_review: I think that this paper is useful since it evaluates model-based predictors on the same open-source framework, enabling a fair comparison of different predictors. However, I think that the experimental evaluation conducted by the authors is not enough in order for me to give an acceptance score. There is also quite some work to be done in order to improve the structuring and clarity of the text in my opinion.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work provides a compressive analysis across multiple (18) different hardware performance predictors by 
- (1) collecting their performance under different amounts of training data and different input network structures and showing each prediction method’s advantageous/disadvantageous scenarios
- (2) analyzing the  prediction accuracy's influence on selecting subsequent hardware architecture and giving the insights on how to pick/design hardware performance predictors for the NAS.

Based on the observation drawn, MLP ones are found to be the most promising predictors in terms of the accuracy with limited training samples, while the Lookup Table model serves as a very cheap and straightforward guidance. In terms of the architecture selection guided by the model-based predictor, the work conjectures by verifying some of the selected network  structures’  hardware performance and increase the number of explored network structures can lead to better pareto front of the selected network architecture
main_review: Strength:
- This work did a very comprehensive survey across different kinds of the performance predictors and across two existing hardware performance dataset. The network architectures considered are also well rounded.
- This work gives reasonable explanations/insights for most of the phenomena in the conducted experiments.

Weakness:
- Novelty is limited
- The insights are shallow and obvious
- Unclear on the aspects of how to counter the imperfect predictions on guiding the network architecture selection


summary_of_the_review: - **Novelty is limited**: Most of this work is simply using the existing hardware performance predictors and running them to collect results on the existing hardware dataset. Too much focus on introducing the setup and configurations of these hardware performance predictors and dataset, and simply numerating the results from the conducted experiment, while the proposed insights are overshadowed. 

- **The insights are shallow and obvious**: Even though the paper presents multiple insights from the experiments, most of them are rather shallow and obvious. For instance, the paper stressed that the model-based can benefit more from more training data as compared to the lookup table based predictors, which is rather obvious as model based is data driven while lookup table is deterministic. However, some specific necessary and useful  insights are missing. For instance for page 5, most of the texts are devoted to reporting which methods perform better and which methods scale better  with more data. The underlying reasons or hypotheses are not provided.

- **Unclear on the aspects of how to counter the imperfect predictions on guiding the network architecture selection**: One of the most interesting aspects about this work is that it proposes imperfect predictions from the hardware predictors can also lead to decent selected architectures. However, it is rather unclear how to achieve this and why. For instance, in page 7, the authors mention “verifying the hardware metric predictions of the 13 architectures”; which 13 architectures? It was never mentioned elsewhere. I really hope the author can stress this part more.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper systematically evaluates a wide range of hardware performance predictors across different networks/devices/tasks and analyzes the influence of such predictors to the architecture selection process. It provides insights for the community about how to select a proper hardware performance predictor in different scenarios.
main_review: **Strength:**
1. The paper is well-written with clear logical flow and easy to follow.
2. The evaluation across different predictors and tasks is solid and the codes are provided.
3. The simple but effective insights can benefit the NAS community.

**Weakness:**
1. The major concern of this paper is the technical novelty since no new technique is proposed and the provided insights are relatively intuitive. In particular, a less accurate but fast predictor can intuitively lead to better results if it can see more architectures, where the measurement cost is assumed to be the bottleneck. More novel insights are expected.
2. The averaged correlation in Fig. 1 across different devices (and tasks) may not be the only metric for evaluating the predictors, i.e., the device-specific correlation is also important. For example, as demonstrated in HW-NAS-Bench, for some search spaces (i.e., a sequential space like FBNet space) and many devices, the operation-wise hardware performances are additive, indicating a lookup table will be accurate enough as a predictor. For most deployment scenarios with determined types of devices, such device-specific discussions could strength the contributions of this paper and provide more detailed insights for the community.
3. The discussions about the influences of the predictors to the architecture selection should be NAS-algorithm-specific while such discussions are missing in the paper. For example, for reinforcement learning based NAS methods, the hardware measurement is independent of the subnetwork training, while the hardware performance will participate in the training process of FBNet as a regularization term, where an inaccurate hardware performance predictor will more directly affect the results after convergence. Such NAS-algorithm-specific discussions may also guide the NAS algorithm selection given affordable hardware performance predictors in addition to predictor selections.




summary_of_the_review: Given the limited novelty and lacked discussions elaborated in the weakness part, I tend to deem this paper marginally below the acceptance threshold. I'm willing to adjust my scores if the concerns are properly addressed.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In neural architecture search (NAS), performance predictors are an important tool, because predicting attributes such as accuracy can save costly measurements. Hardware metric predictors predict metrics such as latency in addition to accuracy. This paper gives an empirical study of 18 different performance predictors on NAS-Bench-201 and TransNAS-Bench-101, evaluating the rank correlation of the predictors on different datasets and predicted metrics. Then they evaluate how the inaccuracy of the predictors affect the predicted Pareto-front of architectures, compared to the true Pareto front. The authors run this experiment by simulating the errors in predictors. The authors find that MLP models perform the best on average across all settings they tried, and leads to an acceptable predicted Pareto-front.
main_review: Strengths
- This is an important problem. Although there have been experimental studies on regular performance predictors, not much was known about hardware metric predictors until this paper, which is a more realistic setting.
- The analysis is good. For example, Fig 1 (right) and Fig 2 (right) give easily visualizable data by centering on predictor-average. Also, they focused on analyzing only the most challenging prediction tasks, which seems beneficial.
- The authors have thorough appendices where they release all of their code, raw results, and methodologies. So reproducibility and ease of follow-up is high.
- It is interesting that in Section 6, the authors simulate the predictor (now there is a prediction of a prediction) to speed up results. However, I have some follow-up questions about this in “weaknesses”.

Weaknesses
- The authors state a few interesting insights at the end of Section 4, 5, 6. For example, MLP works great on HW-NAS-Bench and TransNAS-Bench-101, and the widely used lookup table does not perform nearly as well. Although experimental survey papers are very useful, the best ones follow up on something that is surprising, interesting, or insight-provoking. For example, the authors could go a step further and look into why MLP performed so well, or, find some other type of insight in their study that leads to new directions.

I also have a few smaller nitpicks and questions about the experiments:
 - Section 6 is nice, but don’t a lot of NAS methods use a predictor-guided search such as BO, where the predictor is constantly being updated (even for hardware aware NAS algorithms)? So it would also be interesting to see how the predictors compare within NAS.
 - Section 6 is hard to follow. I don’t fully understand how the predictors are simulated even when reading the figures and Appendix G. It seems like the authors are focusing on approximating the distribution of mistakes. But then do you draw from the distribution on each prediction? What if the mistakes are one-sided based on properties of the architecture? Maybe I am misunderstanding.
 - In Section 6, it sounds like the authors estimated the hardware metrics but “All accuracy values remain accurate.” In the context of real NAS, isn’t this unrealistic because hardware metrics are much easier to get than the accuracies? Why not also use a predictor for the accuracy, and then compute the error of the Pareto-front after that? Or maybe I am misunderstanding?
 - When the authors explained in Section 4 that they only selected five out of 27 datasets, I thought this was totally arbitrary, especially when looking that they are imbalanced (e.g., CIFAR 10 only used once, latency used three times while energy and intensity are used once). But then eventually I got to Appendix E and realized that it is a principled selection, and some of the 27 are trivial and not worth analyzing. So I suggest that the authors stress this methodology more in Section 4. Also, it could still be useful to add some analysis on all 27, in addition to keeping the experiments that are already there. Or, some aggregate analysis about which metrics are hardest and easiest, which edge devices, which image datasets (e.g., like a summary of Table 2).
- Also on page 6, there are a few statements that the authors should have citations for: "Latency is generally the most crucial hardware metric" and "the macro-level search space is more applicable to most real-world problems than a micro-level search space."
summary_of_the_review: Overall, this paper is important and useful for the community, and has pretty good and thorough analysis. However, it could have more interesting insights, and there are some questions about Section 6. Overall, I currently will give a weak reject and look forward to the authors’ response.
review_title: null

Meta-review: This paper comprehensively evaluated 18 different performance predictors on ten combinations of metrics, devices, network types, and training tasks for NAS. While evaluating and comparing different prediction models is not itself novel, the authors provided many insights that are potentially interesting to future NAS developments. 

Reviewer reactions to this paper are rather mediocre and lukewarm. It is in general consensus that this work gives a good empirical analysis on hardware metric predictors for NAS, but the novelty is low and it is perhaps a bit incremental (e.g., nothing "shockingly new" was revealed, and observations are mostly "as expected"). Despite the authors improving the paper during rebuttal with new plots/tables, there remain to be unaddressed comments, e.g., adding experiments that run BO / evolution / etc with different hardware predictors and comparing the quality of the Pareto front. Those missed points were also raised in the private discussion. 

After personally reading this paper, AC sides with most reviewers that this paper lacks true novelty nor technical excitement. While the empirical study is valuable, it perhaps suits venues other than ICLR, e.g., the NeurIPS benchmark track.
Decision: Reject