Venue: AutoML 2023 Conference

Reviews:

final_score: rating: 8: Accept: Technically sound paper with major impact and strong evaluation, with perhaps some minor flaws.
review_content: summary_of_contributions: This paper proposed Multi-Predict NAS algorithm that can be used for multiple search spaces and multiple hardware devices. Multi-Predict uses the search space agnostic architecture encoding, ZCP. ZCP is an ensemble of multiple zero-cost proxies so that ZCP is independent of the search space but still can encode architecture information. Multi-Predict applies learnable embeddings to represent different devices, named HWL. Thanks to the novel encoding of architectures and hardware, Multi-Predict can train one predictor for multiple search spaces and different hardware devices. Furthermore, the encoding algorithms enable Multi-Predict to transfer knowledge to new search spaces or add new hardware devices. It is demonstrated that Multi-Predict can transfer knowledge from FBNet to NAS-Bench-201. 
overall_review: This paper focuses on an important NAS topic, transferring knowledge between different search spaces. A very recent work [R1] uses meta-learning to solve this problem and transfers knowledge between different datasets. Multi-Predict is able to train a predictor for multiple search spaces and different devices. Multi-Predict proposes a very simple yet effective architecture encoding (ZCP) that is agnostic to search spaces. I really like the idea of ZCP. I suggest the authors discuss [R1] in the literature review because [R1] is the most recent paper. 

This paper definitely should be accepted. 

[R1] Transfer NAS with Meta-learned Bayesian Surrogates, ICLR 2023. https://openreview.net/forum?id=paGvsrl4Ntr 
review_summary: Multi-Predict makes it possible to transfer knowledge between different search spaces.  ZCP encoding is simple, smart, and effective. Although ZCP does not have a very strong algorithm contribution, I really like the idea. In my opinion, ZCP is an ensemble of multiple zero-cost proxies. In that case, it can represent the architecture but is not dependent to search space. Furthermore, Multi-Predict makes it easier to adapt predictors to new search spaces or new hardware. 
actions_required_to_increase_overall_recommendation: I suggest the conference accept this paper.
review_title: null

final_score: rating: 5: Borderline Leaning Reject: Technically sound paper where reasons to reject nonetheless outweigh reasons to accept. Please use sparingly.
review_content: summary_of_contributions: The paper proposes a method that encodes zero-cost proxies or hardware latencies to make sample-efficient predictions about the considered task, specifically accuracy or latency, of an architecture. Majority of the paper focuses on evaluation using latency prediction, but accuracy prediction is also evaluated. Within latency prediction, both standard and adversarial tasks are considered, with strong improvements shown on the adversarial task.
overall_review: Positives:
* The paper introduces a novel search-space independent NN encoding strategy that is the first step towards search-space, task and device agnostic predictors for NAS
* The method is sample efficient and has performance competitive with current approaches, in some settings (adversarial task) leading to noticeable gains
* Evaluation is done across various types of predictors: latency and accuracy, variety of benchmarks and settings
* Extensive analyses are provided that study the method from various points of view, e.g. in terms of pre-training, training from scratch scenarios, etc.

Negatives:
* Inconsistent evaluations that make it challenging to make sound conclusions about the method - more details in the technical quality section
* The paper is difficult to read and understand - there are many settings considered, without proper explanations - more details in the clarity section
* Accuracy predictors can be of particular interest to many researchers, but the evaluation in the paper studies them in significantly less depth than latency predictors  - one would likely expect more balanced evaluation for both types of predictors

review_summary: The paper provides an interesting method that is a step towards more general NAS predictors. However, the paper would strongly benefit from more consistent evaluations and clearer writing so that it is easier to see the advantages of the method.

actions_required_to_increase_overall_recommendation: Make the evaluations more consistent and also improve the clarity of the paper.
review_title: null

final_score: rating: 4: Weak Reject: For instance, a paper with minor technical flaws, limited impact, and/or weak evaluation.
review_content: summary_of_contributions: This paper proposes a novel NN encoding called Multi-Predict. The encoding is based on the combination between Zero Cost Proxies (ZCP), and Hardware Latencies (HWL) in order to improve Neural Architecture Search (NAS). The main differences between this paper and prior works is that this one uses a combined approach between ZCPs and HWLs to generate a search space agnostic method to predict neural architecture outcomes. The authors did a variety of comparisons against the BRP-NAS and HELP methods, using NASBench-201 as the primary comparison framework.
overall_review: Strengths:

The authors introduce a new method for creating neural network encodings, and leveraging it to create better neural architectures. The motivation is clear, where we are looking to improve model latencies and performance simultaneously, while also keeping the usually large NAS costs down. To accomplish this, they encode information from a ZCP and a HWL in order to introduce a more efficient NAS search space.

Weaknesses:

Taking from the definition of this paper, there is a Zero Cost Proxy that should be used in this final method to perform NAS. While there is a basic definition of what a ZCP is, and how they are used, there is no explanation of these ZCPs outside of the appendix. There is also a lack of experimentation for the ZCPs. One such example would be to compare the base ZCP performance against your model. While we could assume that your model would perform better due to being an ensemble method, this is not explicitly mentioned in the paper. 

I also feel that even though the Spearman's rank correlation is useful in describing the effectiveness of this method, showcasing test dataset accuracies on NAS tasks would be very useful for future comparison against this work.

review_summary: Overall, this paper has concise experiments that prove the major points layed out in the abstract. The paper also clearly identifies its own contributions to the field, and how they are useful. However, this new NN encoding does not appear to have a high potential within the field of AutoML.
actions_required_to_increase_overall_recommendation: I would recommend adding further experiments on the ZCPs in order to give a more holistic comparison against the baseline for this algorithm. Also, including accuracy values for some experiments will improve the paper’s ability for future comparisons.
review_title: null

final_score: rating: 6: Borderline Leaning Accept: Technically sound paper where reasons to accept outweigh reasons to reject. Please use sparingly.
review_content: summary_of_contributions: This paper tackles the problem of search space-dependent performance prediction models used so far in NAS. This paper proposes a search space-independent neural network encoding based on zero-cost proxies and hardware embeddings. This way, the proposed method shows an improved correlation for latency prediction in comparison to SOTA baselines. Furthermore, this method enables transfer to other unseen devices and search spaces as well as tasks for latency and accuracy prediction. The paper evaluates the proposed method on NAS-Bench-201 and FB-NET for different devices.
overall_review: This paper introduces an important research aspect.
It is mainly well-written and motivates the proposed approach in a good way.

However, there are some open question, which would need to be answered. 
Especially, my main concern is, that there is no actual search result reported in this paper, but only correlations, which does not show the ability to find high-performing architectures. 

review_summary: The idea of taking a step forward and using search space-independent predictors is great, I believe more results on actual search results and more common NAS-Benchmarks are needed.


actions_required_to_increase_overall_recommendation: In order to increase the overall recommendation, this paper would need to answer the open questions in Technical Quality And Correctness. 

**After Rebuttal:** I increase my score from 4 to 6.
review_title: null

final_score: rating: 8: Accept: Technically sound paper with major impact and strong evaluation, with perhaps some minor flaws.
review_content: summary_of_contributions: The paper provides a few-shot multi-search-space latency and accuracy predictors that could be used to expedite NAS methods. The paper focuses on generalizing the task of prediction such that it is agnostic to task, search space, and hardware. I feel this is the most interesting contribution from the paper - Instead of using a vector for representing NNs, authors leveraged a search-space independent network embedding vector that is based on either zero-cost proxies or hardware latency measurements.
overall_review: •	Strengths:

o	It is a relevant topic for the conference.

o	The message of the paper is well supported.

o	This paper addresses a SoTA problem in the AutoML domain.  


•	Weaknesses:

o	Computational cost and carbon footprint of the accuracy and latency predictors have not been reported.

o	The manuscript contains a few typos.

o	A multi-device latency predictor may not work in practice.

review_summary: Overall the idea of using a novel network embedding for building design-space and task-agnostic accuracy/latency predictor is quite interesting. I give this paper a weak accept, but I'm willing to change my evaluation if the mentioned concerns are addressed.
actions_required_to_increase_overall_recommendation: Please consider addressing my concerns mentioned in the previous sections.
review_title: null

Decision: Accept(Main Track)