Venue: AutoML 2023 Conference

Reviews:

final_score: rating: 4: Weak Reject: For instance, a paper with minor technical flaws, limited impact, and/or weak evaluation.
review_content: summary_of_contributions: The paper is about methods for training supernets, ie network architecture selection. It is mostly a detailed comparison of existing methods across existing benchmarks, with key parameters varied in order to establish performance ranking of methods and relate this to the underlying setting. Some rather subtle findings emerge and there is a reasonable attempt to relate this to salient features of the methods compared.
overall_review: The paper is a large and thorough comparative assessment of some techniques that certainly fall within AutoML.

The paper is somewhat justified by a discussion of the context of the methods and setting, which go some way to motivating the study.

Experiments are presented clearly and code is provided in support.

On the other hand,

The findings of the paper appear to be subtle.

The abstract does not highlight a clear finding.

The experiments would be best described as explorative, rather than designed carefully to assess a distinct hypothesis. This means that conclusions (where attempted) tend to be 'just-so', and are not then tested further. This makes the paper less valuable.
review_summary: My recommendation is based on decades of experience in ML and related areas, and an assessment largely of how clearly the authors have justified and presented their findings, and my estimation of how much interest/impact that will have. I accept that I could be wrong because this specific area of AutoML is not my area.
actions_required_to_increase_overall_recommendation: The authors should improve the abstract to deliver a clearer finding / improve the chances of impact.

The authors should more clearly present testable hypotheses, or if this is not possible, then suggest ways in which findings could be tested in future.
review_title: null

final_score: rating: 6: Borderline Leaning Accept: Technically sound paper where reasons to accept outweigh reasons to reject. Please use sparingly.
review_content: summary_of_contributions: The authors of this paper conducted an analysis of NAS algorithms to better understand the conceptual structure of supernet NAS algorithms and their performance. They conducted a literature review and an experimental case study to highlight the need for architecture selection and supernet training to be studied as interdependent processes. They then performed comparative evaluations of these algorithms. The authors stress the importance of systematic evaluations of NAS algorithms using equivalent amounts of training for compared algorithms and across multiple search spaces. The insights gained from this study can guide the development of future differentiable search methods.
overall_review: NAS-Bench-201 is small and inexpressive as it contains only 15k architectures. The results on this benchmark are not generalizable and may not transfer to larger benchmarks. Therefore, larger NAS Benchmarks should be considered for evaluation. Or else, the authors should give justification for choosing only this benchmark. 

There are many similar works in the past targeting architecture selection methods. It is difficult to understand the differences with other papers. A comparison table in Section 2.3 (or in the appendix) could provide a better understanding of the main contribution of this paper over others. 

It may be good to plot the searched models for a few cases and compare the architectural specifications for different methods. 

The size of the sampled/searched model matters. So, a deeper analysis of how model size varies with the architecture selection method may be potentially useful along with accuracy. 

The authors considered only prune, valid, synflow, jacob, perturb for evaluation. What is the reason for choosing only these five? Are these the only methods available? 

This work says “We did these experiments and here are the results with the interpretation of results” It may be good to provide any further guidelines for future researchers to improve the selection methods. 

review_summary: The reviewer mainly focused on interpreting the results of benchmarking different architecture selection methods. The reviewer has gone through the methods chosen as a part of the study and provided a few steps to improve the article. 


actions_required_to_increase_overall_recommendation: As suggested before, the paper can be enhanced by conducting experiments on larger benchmarks and more architecture selection methods. The presented insights from the results on NAS-Bench-201 are confusing in a sense. It may be good separate zero-shot methods and give an in-depth analysis separately from the DARTS-based methods. 


The following latest papers should be considered in references to support the literature review: 
Weight-sharing neural architecture search: A battle to shrink the optimization gap
SuperNet in Neural Architecture Search: A Taxonomic Survey  
Neural Architecture Search Survey: A Hardware Perspective
Neural Architecture Search: Insights from 1000 Papers

Minor issues: 

Reference to Table 3 is missing in lines 548 and 549
Line 187: Here we focus in on the -> Here we focus on the
review_title: null

final_score: rating: 5: Borderline Leaning Reject: Technically sound paper where reasons to reject nonetheless outweigh reasons to accept. Please use sparingly.
review_content: summary_of_contributions: This paper broadly considers the task of architecture selection from supernet based methods for neural architecture search. The authors study DARTS- and DARTS -PT to claim that search space design is the most significant factor in NAS, with epochs required to train the supernet being more important than the data-set used. The paper makes a case for the importance of systemic evaluation of NAS algorithms. 
overall_review: This is a timely, well motivated paper with interesting insights. The case study provides a clear understanding of the limitations of DARTS as well as the differences in methods of addressing the instability of architecture weights in DARTS. The results are well presented and conclusions are well supported by available evidence. However, the diversity of data-sets and search spaces brings into question the quality of available evidence. The main negative aspect of the paper is the lack of diversity in the architectural search space, making their comments on the importance and trade-offs of architecture design, architecture selection methods and training lengths inadequately supported. 
review_summary: I recommend a weak reject. I do not believe there are any technical flaws in the study, but its impact would be limited due to the weak evaluation. The paper addresses an important problem in differentiable NAS and conducts an effective literature survey to identify aspects of differentiable NAS that requires further study. However, a greater diversity of search spaces and data-sets are required to convince me of the conclusion and contributions.
actions_required_to_increase_overall_recommendation: 1. I would appreciate the authors insights on whether their selection of search spaces and data-sets are sufficient to support the claims made in the paper.
2. Why are full-ImageNet scale evaluations not included in the paper? I believe they would make the paper more insightful.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak impact, and/or weak evaluation.
review_content: summary_of_contributions: The paper presents a comprehensive assessment of architecture selection techniques along with various supernet training methods, highlighting the previously overlooked interdependence among them. It also showcases the effective integration of zero-shot NAS methods as new architecture selection mechanisms in supernet NAS training while evaluating the aforementioned techniques
overall_review: I want to first of all highlight the effort for designing the framework by taking the search space, the datasets and the number of training epochs to evaluate different pairs of supernet training and architecture selection mechanisms. Running all possible combinations resulted in a very high computational cost (25400 GPU hours). This very high cost is the actual reason that pushes practitioners to focus on a single aspect which is mostly the supernet training. Obviously, they are aware of the issues related to the architecture selection mechanisms and the interplay between those and the supernet training strategies and not as claimed by the authors. The whole point of AutoML is not only to reduce the manual workload but also to keep the computational load under control. Apart from this major critisism, the take home-messages of the paper regarding the role of the search space, the training length and the dataset are not novel.      
review_summary: Given the lack of novelty and the minor expected impact on the field of AutoML, this paper is not eligible for acceptance at this venue. 
actions_required_to_increase_overall_recommendation: it is hard to improve the paper since it is an empirical study of a reasonably large search space. The lack of consistent outcomes and generic take home messages across setups makes it just not worth acceptance.
review_title: null

Decision: Accept(Workshop)