Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: Although NAS automates the manual design process of networks, it still requires expensive computing costs for evaluating multiple models. Zero cost proxies have been studied for alleviating the issues but it has several limitations, low productivity and poor ability of generalization. To tackle the current limitation of it, this paper introduces a genetic programming driven methodology, EZNAS, to automatically discover Zero-Cost Neural Architecture Scoring Metrics (ZC-NASMs)  that are interpretable, generalizable and deliver state of the art score-accuracy correlation.

strengths_and_weaknesses: Clarity: This paper is well-organized and easy-to-read. The authors smoothly develop their logic and key concepts throughout the paper. The concept figures are illustrated clearly. 

Quality: This paper smoothly develops its logic throughout the paper. The experiments seem carefully designed.  

Originality: The contribution of this work seems solid, since unlike the existing methods that use ZC-NASMs that are driven by human intuition or theoretical inspiration, this work devises a method for automatically discovering ZC-NASMs which lead to state-of-the-art score-accuracy correlations. This work has the potential to reduce human bias in the design of ZC-NASMs. 

Significance: The achievement of this work seems significant, sinc it shows better score-accuracy correlation compared to the existing metrics and methods, i.e. DARTS, Amoeba, etc. 

questions: I can only see the score-accuracy correlation in the main paper and the supplementary documents. 

If possible, please provide actual test accuracy and statistics, and detailed information, (i.e. training times, search times, FLOPs, model sizes, etc) of sampled networks for Table 1-5. 

limitations: No negative societal impact is expected.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper automates the discovery of zero-cost proxies in NAS. Very low-level mathematical operations are selected and combined through genetic programming to form proper zero-cost proxies. The effectiveness of the proposed method is validated on NASBench-201 and NDS.
strengths_and_weaknesses: Strength:

1.	The paper is well motivated and clearly written. The paper is clear about limitations too.

2.	I think this paper will be valuable to the community. Zero Cost Proxies discovery is important for efficient neural architecture search. Evolving the proxies automatically from low-level mathematical operations also follows the trend of auto-ml. 

3.	The searched zero Cost Proxy can better capture the order of candidate models than previous hand-crafted proxies, and is reported can generalize better on different search spaces.

Weakness:

1.	The proposed method looks quite complicated, considering the program representation, NN statistics generation as well as the evolutionary scheme. The inclusion of code would be a large benefit to this work. 

2.	This paper borrows ideas from [1], providing the program synthesis is also considered before, which weakens the novelty of this paper. So I recommend a weak accept.

References:

E. Real et.al. Automl-zero: Evolving machine learning algorithms from scratch. ICML ‘20.


questions: None
limitations: Yes
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This work concentrates on crafting training-free measurement for neural architecture at initialization. Instead of being designed by human experts, the authors propose a genetic programming framework to automate the discovery of zero-cost proxies. Each candidate is represented by a computational graph of operation trees. The initial population is randomly constructed. At each iteration, an evolution task subset is sampled from the full datasets to evaluate, and the candidates with higher ranking correlation with the final performance is preserved. Experiments on a series of NAS benchmarks shows that the searched zero-cost proxies are more robust than the manually designed scoring functions.
strengths_and_weaknesses: # Strengths:
1. I believe that this paper provides an insightful idea for the AutoML community that we can adopt evolutionary search on computational graphs for a specific operation with infinite searching space. Zero shot NAS is one example.
2. Experiments show that the searched proxy can generalize to different datasets, with extremely small searching cost (a single CPU in under 24 hours)
# Weaknesses:
1. Actually, this paper can be regarded as a new predictor-based NAS method, where the predictor in this work is searched by genetic algorithm. A general framework for predictor-based NAS is to construct a surrogate predictor (e.g. a meta network) to predict the final performance. However, none prior work on predictor-based NAS  has been referenced in this paper, nor does the author discuss its difference with them. I suggest the author should include them in your discussion.
2. Some of the very important baselines are missing. There exists a new trend for zero-cost NAS on neural tangent kernels[1, 2], which is cited in the paper but not presented in the quantitative comparison. I highly recommend the authors to compare with them.
3. The search space is quite restricted, "ReLU-Conv2D-BatchNorm2D", which hinders its application to more advanced architecture nowadays. 
[1] Shu Y, Cai S, Dai Z, et al. NASI: Label-and Data-agnostic Neural Architecture Search at Initialization[J]. arXiv preprint arXiv:2109.00817, 2021.
[2] Chen W, Gong X, Wang Z. Neural architecture search on imagenet in four gpu hours: A theoretically inspired perspective[J]. arXiv preprint arXiv:2102.11535, 2021.

questions: What does the searched EZNAS-A and EZNAS-B tree look like? I think it is important to visualize them or comprare them with typical zero-shot proxies. 
limitations: No limitation is discussed in the paper.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This work proposes a framework, namely EZNAS, searching for zero-cost proxies based on evolutionary algorithm. Specifically, the authors utilize an expression tree to represent the computation of a zero-cost proxy and adopt DEAP evolutionary algorithm framework. Extensive experiments on NAS benchmarks show the effectiveness of the method.
strengths_and_weaknesses: Strengths:
+ The motivation is interesting and sound. This work attempts to search an effective zero-cost proxy rather than manually design one like prior works.
+ The experimental results are pretty good.

Weaknesses:
- The expression tree representation and the evolutionary algorithm is not new, which slightly limits the novelty.
- The layout is informal. 
a) Fig. 1 should be placed behind the Fig. 2.
b) I suggest the authors providing a detailed legend in Fig. 2 to explain the meanings of T1(G), T2(G), T3(G), and T4(G).
c) Figure 5 should be Table 5.

questions: 1. How do you obtain the accuracy of architectures sampled from NDS (Line 253-256)? Do you train theses networks from scratch or directly use the accuracy in NAS-Bench-301?
limitations: The limitations are addressed.
review_title: null

Meta-review: metareview: This paper proposes a genetic algorithm for discovering zero-cost scoring metrics for neural architectures, that can generalize to neural networks in unseen search spaces. The proposed genetic algorithm based on the DEAP evolutionary algorithm can discover programs in the form of expression trees to represent the computation of a zero-cost proxy, and thus is also interpretable. The proxy found using the proposed algorithm is validated on the benchmark datasets such as NASBench-201, and is shown to outperform previous methods in the estimation of architecture’s accuracy, in terms of the Spearman’s correlation score. 

All reviewers gave the paper positive ratings, due to the importance of discovering zero-cost proxy for NAS, its superior performance estimation ability over existing proxies, and its quality of writing. However, reviewers were also concerned about possibly limited practical impact due to the complicated methods, lack of methodological novelty as it could be considered as a straightforward application of an existing genetic algorithm, and missing comparison against predictor-based NAS methods as well as state-of-the-art methods. 

I also agree that while the proposed work could be a valuable addition to researchers working on zero-cost NAS, the missing discussion and comparison against relevant baselines such as predictor-based NAS is the main drawback of the method. In addition to methods that are based on Neural Tangent Kernels mentioned by Reviewer DDGZ, meta-NAS methods [Lee et al. 21] [Jeong et al. 21] that can generalize to new datasets should be also further discussed and compared against, considering both the performance estimation accuracy and training efficiency. Lack of the results and analyses of end-to-end NAS results is another drawback. 

The author responses do not provide experimental results on these points that are raised by the reviewers, perhaps due to good initial scores. However, these missing discussions and comparisons makes it difficult to understand the pros and cons of the proposed work over theirs, which makes it a borderline paper at its current state. I strongly suggest the authors discuss existing works and what advantage the proposed method has over theirs (e.g. generalization to new search spaces, explainability) as well as perform experimental comparison over theirs. 

[Lee et al. 21] Rapid Neural Architecture Search by Learning to Generate Architectures from Datasets, ICLR 2021
[Jeong et al. 21] Task-Adaptive Neural Network Search with Meta-Contrastive Learning, NeurIPS ICML. 
confidence: Less certain
Decision: Accept