Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The paper give a large-scale comparison of performance predictors in NAS. The results act as recommendations for the best predictors to use in different settings. The author also show that certain families of predictors can be combined to achieve even better predictive power.
main_review: (a) The paper proposed a large-scale comprehensive comparison of performance predictors in NAS, the choice of predictors range from learning curve extrapolation, to weight-sharing, to 10 supervised learning and zero-cost proxies. The survey itself is beneficial to the NAS community.
(b) The paper proposed a new predictor OMNI, to combine complementary information from three families of performance predictors, leading to substantially improved performance.
limitations_and_societal_impact: (a) The paper has well-study the ranking performance of predictors, however I believe in NAS, sampling (of arch and peformmance pairs) also plays a important role and can largely affect the final performance of NAS predictors. As a result I believe sampling should be decoupled in the evaluation of predictor-based NAS, e.g. BANANNAS/ BONAS which use guided sampling method, should fall into a different category and compared seperately.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper compares 3 kinds of performance predictors, model based predictors, learning curve based predictor and zero nas predictors. They benchmark the performance of all 31 predictors against 5 benchmarks: NAS-Bench101, DARTS, NASBench201 and NLP.

In addition to this, they also incorporate SOTL-E and jacobian variance as addition features to NGBOOST and SemiNAS predictors and demonstrate that it improves the performance over all 3 of them. 
main_review: Comparing all the predictors and analyzing about how they perform with respect to initialization time and query time is definitely useful for the community. 

They were able to empirically demonstrate some hunches that the community had such as 1) Zero cost NAS techniques do not perform well on larger DARTS space 2) Surrogate models using latent features of architectures would generalize well when one does not have a lot of training data. 

While the experiments and the insights provided in the paper are useful, none of it is a revelation or surprising. For instance, the tree based methods developed recently are able to outperform model based methods but it takes a lot longer to train them. This paper does not have enough novelty to be accepted as a conference paper. However, this is a good starting point for people who are new to the field. It might be good to expand this paper by adding more algorithmic details of various flavors of performance predictors and make it a survey paper instead.




limitations_and_societal_impact: 1) While SoTL and SoTL-E perform well empirically, it is theoretically not valid to use sum of training losses to predict the performance of a network.  A network with lower training loss might also be overfitting and not generalize, while a network with regularization such as dropout, l2 etc might have a higher training loss but still generalize well.

2) It might be better if you didn't include SoTL and SoTL-E in your list of 31 predictors as well.

3) Given that the we know the learning curves of all the architectures in the NAS benchmarks, why not include all the architectures in the test set, rather than only 200? That will give us an actual overview of how these predictors perform on that search space.

4) Given that OMNI has a SEMINAS variant as well, please generate figures corresponding to Figure 3 and 10 to quantify how much it improves over SEMINAS too.

5) Please include OMNI in figure 4 to show its efficacy in ranking architectures that are close to each other.

6) Another useful experiment to perform would be to test how stable the predictors are to various train/test splits and random seeds. 

7) I understand that the predictors included in the paper are all those which have open source repositories. It is useful to include other state of the art (SOTA) learning curve predictors ([1] ; [2]; 3) and SOTA model based predictors [4] to a more complete analysis of the performance predictors.

[1] Learning curve prediction with Bayesian neural networks, Klein et al.
[2] Learning to Rank Learning Curves, Wistuba et al.
[3] Speeding up hyperparameter optimization by extrapolation of learning curves using previous builds, Chandrasekhar et al.
[4] Neural Architecture Search in A Proxy Validation Loss Landscape, Li et al.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: Neural Architecture Search (NAS) aims to find the network architectures with the best accuracies -- or best size/accuracy tradeoffs -- from within a human-defined search space. The submission evaluates a number of different heuristics for ranking different candidate architectures in a search space. The submission claims to evaluate 31 different techniques which fall into several categories (e.g., early stopping, learning curve extrapolation, supervised learning, one-shot models, zero-shot metrics). The submission claims that different techniques work better for different compute budgets (e.g., low setup cost/low search cost vs. high setup cost/high search cost). It also claims that combining techniques from different families (e.g., supervised learning + zero-shot metrics) can lead to better results than using any one technique on its own. Experimental results are reported on NASBench-101, NASBench-201, the NASBench-301/DARTS search space, and NASBench-NLP.
main_review: **Contributions:** The submission evaluates/compares a number of previously proposed NAS techniques from several popular categories:
1. Supervised training methods which sample a set of (random) architectures, measure the performance (accuracy) of each one, and then use the resulting  `<architecture, accuracy>` pairs to train a regression model which can predict the accuracies of unseen architectures.
2. Curve extrapolation ("learning-curved based") methods which try to predict a network's accuracy at the end of training by measuring its accuracy at each of the first `K` epochs of training.
3. One-shot NAS methods which train a single set of shared weights which can be used to evaluate/rank many different candidate architectures in the search space.
4. Zero-shot NAS methods which try to rank different candidate architectures based on a few batches of data using randomly initialized weights.

**Results:** The submission (not surprisingly) claims that different techniques are optimal in different ranges. For example, it claims that Jacobian Covariance, a zero-shot technique, performs best in the very low-cost regime, whereas simply training network architectures from scratch for a shortened number of steps performs best in the high-cost regime. Both of these claims seem plausible.

Perhaps more interesting -- and more surprising -- is the submission's claim that if the goal is to maximize the correlation between predicted and observed (ground-truth) model accuracies, "Of the 31 predictors we tested, we found that just seven of them are Pareto-optimal with respect to Kendall Tau, initialization time, and query time. That is, only seven algorithms have the highest Kendall Tau value for at least one of the 154 query time/initialization time budgets on NAS-Bench-201 CIFAR-10" (L217). Similar trends are reported for the other datasets.

In addition to evaluating different techniques' abilities to rank different candidate architectures in a search space (e.g., rank-correlation between predicted and ground-truth accuracies), the submission also evaluates: (i) each technique's ability to estimate which of two similar-looking architectures will have a higher accuracy, and (ii) the effects of ensembling different techniques, e.g., a supervised technique + zero-shot NAS.

**Significance:** The findings, if valid, can be potentially useful for practitioners who want to know which classes of NAS techniques to focus on. One risk in trying to evaluate so many different techniques in a single paper is that the authors might not spend enough time ensuring that each individual technique is properly implemented and well-optimized. The authors take a number of reasonable steps to try to mitigate this concern:
1. They try to use the authors' original implementations of different techniques in many cases (L186). However, looking at the Appendix and included source code, I have the impression that some techniques are evaluated using snippets of code from the original authors' implementation whereas others are reimplemented from scratch.
2. Hyper-parameters are re-tuned by the submission's authors to help control for the fact that different techniques probably received different amounts of hyper-parameter tuning in the past.
3. The final evaluation uses a separate held-out test set. (Good ML practice.)

Given the sheer number of architectures evaluated by the submission, this seems like a reasonable evaluation protocol. However, it comes with the caveat that if a technique is found *not* to work well by the submission's authors, it can be unclear whether that's because of a flaw in the technique itself or whether the submission's implementation of that technique is implemented incorrectly / insufficiently tuned.

The submission's claim that ensembling different NAS techniques (e.g., NGBoost + Jacobian covariance) can lead to significantly improved performance seems intuitive and potentially relevant for practitioners. This seems like a direction that's received relatively little attention for the NAS community. While it comes with the downside of making the NAS implementation significantly more complex, I still think reporting these results is a nontrivial research contribution.

**Clarity:** The submission seems well-written, polished, and easy to understand.

**Post-rebuttal update:** The discussion and reproducibility table provided by the authors does a good job of addressing my concerns about whether the original methods were reproduced faithfully, and I've increased my rating from 6 to 7.
limitations_and_societal_impact: The quality of ML algorithms tends to be sensitive to hyper-parameters and small implementation details, so the reliability of the submission's claims may depend on the quality of the submission's authors' implementations of different NAS techniques. However, this is as much a general concern about the type of work (trying to evaluate/compare/rank 30+ techniques in a single paper) than about authors' specific implementation.

The most effective mitigation I can think of is to verify that the submission's implementation is able to reproduce the results reported by the original authors of a given technique. I don't think the submission's authors are doing this, but I might've missed something. (I haven't checked the paper's appendix too closely.)
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors identify and benchmark a plethora of recently proposed techniques for predicting neural network architecture performance on a task without doing a full training run. 

They begin by grouping each technique into categories: model-based, learning-curve-based, zero-cost, and weight-sharing. When viewed like this it becomes clear that there is a need for a common benchmark to distinguish the performance of the different approaches. 

To compare under one framework, evaluation is done with respect to two variables: initialization time (time spent preparing the model) and query time (time spent retrieving predictions from the model).

The authors then present their own method, OMNI, which performs NAS using a blend of model-based, learning-curve-based, and zero-cost predictors. 

main_review: I think this is a solid paper and should be accepted. There is nothing hugely surprising or significant in it, and there are no real deep insights uncovered, so I'm not sure it merits being a "seminal paper". However, the experiments are thorough, logically conducted and answer previously unanswered questions. I think overall this work will be helpful and insightful to the neural architecture search (NAS) subfield. 

**Originality:** This is a novel investigation into (and combination of) pre-existing techniques. Analysing different predictors under a single framework (via query/init times) is a new way of looking at NAS and allows us to ask interesting questions.  Related work is adequately cited.

**Quality:** The submission is technically sound, and the reader leaves the paper with the impression that each claim has been thoughtfully evaluated. The choice of methods seems to give a reasonable coverage of the space of existing work and the authors are careful to address problems with comparative analysis (especially with regard to hyperparameter setup). 

**Clarity:** The submission is very clearly written and organized. It's easy to glean the main takeaways and the logical flow of the paper makes sense. I feel confident that there is enough detail in the paper that would allow me to reproduce each of the results. 

**Significance:** The results of the paper come in the form of clear takeaways which I'm sure will be of practical use to anyone hoping to use neural architecture search. The results from the experiments are a unique dataset which will allow for the exploration of further lines of research. 

## Positive points:
- The paper is very well written and structured, and comes with clear, robustly verified takeaways
- Open sourced code 
- This is an experiment-heavy paper, which I suppose you would expect from a characterization task like this. The experiments cover a wide range of datasets and techniques, and at the end the authors present a combined method which appears to robustly outperform using a single approach. 
- There are many interesting visualisations with pleasantly surprising results — the fact that the benefits of using different predictors can be combined and are complementary is both interesting and useful as an insight.

## Points to address:
1. I found the use of the terms: initialization, query and update a little confusing. In the case of a zero-cost proxy, does “initialization time” ever change? Whereas for something like XGBoost, does “initialization” really mean “training”?
2. When do we explicitly want to trade off query and initialization time? At the end of section 4 the authors state “we recommend deciding on a target initialization time and query time budget”. I understand that the two aspects were needed to design a comparison framework for various methods, but I’m struggling to think of a scenario in which this is a trade-off that’s useful to be able to make, or what might motivate me to have a preference for one or the other? 
3. It’s an enormous topic, but I would have liked to see a slightly more fundamental investigation of what drives the various performance predictors. Is there anything that can be extracted from the best performing methods on various NAS-benchmarks to address what exactly each one is measuring? Since the different techniques appear to be complementary, is it likely that they are measuring different things? I suspect this question is out of scope for this paper (which is already packed with insights), but it would be great to have some quantification of this.
4. I would also have liked to see some more “baseline” methods. For instance, what if you use number of parameters/flops as a zero-cost proxy? 
5. Minor visual note: the plot titles in figures 3 and 5 aren’t aligned properly.

## General questions

I think this is a really interesting piece of work that leaves the reader with many things to ponder! I therefore have some more general questions for the authors, that I don’t think necessarily need to be addressed in the paper but would be interested to hear about (I will continue numbering them in case you would like to reply to them specifically):

6. Regarding NAS-Bench-201, the authors state "6 466 are unique after removing isomorphisms” — do they have a citation for this? 
7. On line 279 the authors state: “The NAS-Bench-101 API only gives the validation accuracy at four epochs”. Is this true? This seems surprising!
8. The authors mention that the training and test sets are architectures drawn uniformly at random from the search space — could they confirm that these are two separate sets? I.e. there’s no chance of resampling the same architecture?
9. During the mutation-based test set experiments the authors limit the test architectures to be “at most an edit distance of three from a seed architecture”. How is edit distance calculated? Is it a distance between binary adjacency matrices? If so, is the edit distance of replacing e.g. a conv1x1 with a conv3x3 the same as the edit distance of replacing a conv3x3 with an identity? Again, it’s not necessary to include anything in the paper, but I would be intrigued to hear the authors thoughts on how different measurements of “distance” might impact this set of experiments. Can we use e.g. the distance between scalar outputs of zero-cost measures as “edit distances” between architectures? 



limitations_and_societal_impact: I have no concerns about societal impact beyond any other work on neural networks.
review_title: null

Meta-review: This is an in-depth investigation of existing techniques. The experiments support the claims/questions in the paper well. The authors engaged well during the discussion phase to clarify all reviewers' concerns. The findings are likely going to be very useful to the NAS community.
Decision: Accept (Poster)