Venue: AutoML 2022 Track Main

Reviews:

final_score: rating: 5: Accept, good paper
review_content: summary_of_contributions: n/a
overall_review: n/a
review_summary: n/a
review_title: null

final_score: rating: 4: Marginally above the acceptance threshold (use sparsely)
review_content: summary_of_contributions: The paper provides a empirical analysis of the efficacy of several hardware predictors and their downstream impact on the end to end NAS algorithm performance on the HW-NAS-Bench and Trans-NAS-Bench-101 benchmarks. 

The authors evaluate 18 prediction algorithms on several hardware and task scenarios from these benchmarks and provide some insights into the relative advantages and disadvantages of these predictors in different scenarios. The authors then evaluate the impact of predictor sub-optimality on the end output - comparing the discovered Pareto-front with the optimal Pareto-front. Using these experiments, the authors provide insights and recommendations on how to choose the best predictor for a given task.
overall_review: Strengths :

1) With the increasing growth and focus on Efficient ML and Hardware-Aware NAS, understanding HW performance prediction is a fairly important problem. While there have been plenty of papers that study performance prediction for accuracy, this work is (to the best of my knowledge) the first work to study performance prediction specifically in the HW context.
2) The paper is well written and provides simple and actionable insights for future work backed by data and easy to understand visualizations.
3) The authors provide the code and a detailed appendix, making reproducibility and future work on this topic accessible. 

Weaknesses :

1) Novelty : While this work provides some insights, the novelty is limited, considering that this work is mostly an adaptation of [1] to the hardware prediction domain. Some of insights presented as well (improved performance with more training data, efficacy of MLPs and better Pareto fronts with more samples evaluated) are not particularly new.

2) Transferability : (elaborated in the questions above).

review_summary: This paper provides some interesting insights and analysis about a problem (HW Performance prediction) that is of importance to the AutoML community. However, the current insights lack depth and there are a few concerns regarding the transferablity/generalization of these recommendations. I currently give this paper a weak accept, but I'm willing to change my evaluation if the mentioned concerns are addressed.
review_title: null

final_score: rating: 5: Accept, good paper
review_content: summary_of_contributions: This paper evaluates 18 different predictors for hardware metrics on 2 NAS benchmarks, NAS-Bench-201 and TransNAS-Bench-101. These predictors are compared based on the rank correlation and the authors investigate how the simulated Pareto-front based on these hardware metric predictors compares to the true Pareto-front. Their findings show that model-based predictors outperform lookup tables with few data points and that simple MLPs perform the best on average across all their evaluated settings.


overall_review: Positive:
- Useful insights on hardware metric predictors for NAS, which will be useful for a lot of researchers in the AutoML community.
- 18 different performance predictors are utilized for the analysis and are evaluated across 2 NAS tabular benchmarks.
- All the code is available on the same framework.
- The paper is well-structured and mostly easy to follow

Negative and other comments:
- Without belittling the importance of the large-scale evaluation conducted in this paper, the novelty aspect is lacking. To circumvent this my suggestion would be to go along the lines of for instance combining predictors as done in White et al. (2021) with the OMNI predictor. 
- I would also be curious to see how some multi-objective or hardware-aware NAS algorithms perform in the 2 settings: (1) when using the predicted hardware metric and (2) using the true hardware metric.
review_summary: In general, this paper provides some useful insights on hardware metric predictors in NAS and can be of practical relevance to the community since everything is implemented in NASLib and it can be used for benchmarking purposes in multi-objective NAS. I would therefore recommend acceptance.
review_title: null

final_score: rating: 5: Accept, good paper
review_content: summary_of_contributions: This paper presents a wide experimental evaluation of performance predictions in the context of NAS (but could be applicable more generally). The experiments show that MLP-based performance models are generally the most accurate, and on the opposite common-used lookup tables are the most inaccurate if compared to models trained on a reasonable amount of data. The authors then measure the discrepancy between model performance predictions and real performances. They show that for good predictors a subset of the best candidates of the pareto frontier identified using the performance predictions corresponds closely to the best candidates of the real pareto frontier. They measured this discrepancy with an hypervolume, the space between the estimated pareto frontier and the true pareto frontier.
overall_review: + Is it a relevant topic for the conference
+ The message is clear and well supported

- Computational cost of performance estimation is minor compared to accuracy estimation and therefore the improvements possible are limited.
review_summary: I believe the message of this paper is relatively important. The experiments are broad supports well the message as well. The impact of this work may be limited however because performance estimation is quite inexpensive compared to accuracy estimation.
review_title: null

final_score: rating: 5: Accept, good paper
review_content: summary_of_contributions: This paper, inspired by the study by [White et al., 2021], empirically reviews the performance of different hardware metric predictors on HW-NAS-Bench and TransNAS-Bench-101. It further evaluates the effect of predictors on multiobjective/hardware-aware architecture search.
overall_review: Pros:
-	the extensive empirical study on different hard-ware metric predictors are useful for future hardware-aware NAS research
-	the empirical analyses are thorough and well structured. I appreciate the authors’ efforts to evaluate the effect of predictors on multi-objective search on top of comparing their fitting results, which gives better idea on how well predictors work in practice.
-	I quite like the discussion section in the paper which summarises the key insights and provide useful recommendations for the readers.


Cons:
-	The novelty of the paper is quite limited because only existing predictors are evaluated and the empirical analyses mainly follow the structure in [White et al., 2021]. It would make the paper more interesting if the authors can leverage the insights learnt and propose a new type of hybrid predictor as in [White et al., 2021].
-	The conclusion from the empirical analyses is kind of expected without much surprising insights.  

review_summary: While the novelty of the paper is limited, such extensive empirical study on hard-ware metric predictors is still useful and necessary for NAS community. The authors do a good job in conducting this empirical study and in delivering/deriving the useful insights/recommendations. Thus, I would recommend to accept this paper.
review_title: null

Meta-review: metareview: The authors conduct a comparison of several hardware predictors inspired by the work of White et al. 2021. All reviewers consider this study useful but also identified some areas that require deeper investigation. It remains unclear whether the results transfer to different search spaces or NAS algorithms that differentiate through the predictor. There is a clear agreement to accept this work at the conference.
confidence: 5: The area chair is absolutely certain
Decision: Accept