Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes to evaluate some standard NAS algorithms over various tasks, ranging from image classification, sattllite image time series analysis or ECG detecting hard diseases. 


main_review: This paper proposes a 'so-called' NAS-Bench-360, which is quite misleading. The original NASBench idea (brought by NASBench-101 Ying et al.) is to exhaustively train all (or a substantial amount of) architectures within a search space and record their training / evaluation statistics for researcher or NAS practitioners to evaluate their search algorithm in a fair manner. However, this work mis-use the concept. After reading the entire paper for multiple times, it seems this work is merely evaluating some standard NAS algorithms on various datasets and present some empricial analysis.  

In my humble opinion, the usefulness of NAS-Bench is to leverage some groups' massive compute-power to obtain a massive pool of architecture statistics on various tasks, to benefit people without such power to conduct NAS research. Evaluting some algorithms, even across drastically different tasks, is interesting but not enough to be accepted to a top tier conference. 


summary_of_the_review: See above, this paper mis-use the concept of NAS-Bench that is common to NAS researcher. After reading the entire paper, I do not find it interesting as a NAS practitioner. Evaluating some algorithms on different tasks is a good empirical report but cannot be qualified as a top-tier conference paper due to lack of novelty and effort.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents NAS-Bench-360, which consists of collection of 10 diverse tasks, carefully selected to include datasets with different number of classes, total number of datapoints, dimensionality and application domain. By using a fair evaluation protocol, the authors benchmark different NAS methods on various search spaces, with various degrees of inductive bias, and provide some interesting insights on the effectiveness of NAS under time constraints, showing that there is almost no benefit compared to using a fixed manually engineered architecture.
main_review: In general this paper is really well-written and the motivation behind its idea is quite valid in my opinion. In general I think the NAS community will benefit from this suite of different tasks to run their NAS methods. Below I list some more details comments:

**Pros:**

(+) I think the set of collected tasks will be a useful contribution to the NAS community in order to shift the attention to other tasks that do not only involve image classification.

(+) Useful insights on the usefulness of NAS by evaluating different methods or/and search space across 10 different diverse tasks.

(+) The paper is well-structured and easy to follow.

(+) Available codebase that allows reproducing the experiments and evaluating new methods by using the same settings.

**Cons:**

(-) I think the paper could benefit more by experiments that provide further insights for the NAS practitioner. For instance, performance predictors are commonly used nowadays in NAS [1]. It would be useful to evaluate a couple of these methods (e.g. some of the ones used in [1]) on the proposed suite, both in isolation (reporting the rank correlation coefficient) and inside a black-box algorithm (e.g. Bayesian optimization).

(-) I think there is some text and results in the main paper that can be moved to the appendix and make some space for additional empirical results. For instance, the authors can move Table 3 to the appendix, since it is somewhat redundant with Table 2. Some examples of such additional experiments can be for instance:
- A more detailed investigation on why the NAS methods fail in many of the tasks in NAS-Bench-360; 
- Why some of these NAS methods perform better in some tasks and some others not? Is this because of a sub-optimal search space (operation choices and topology) or an underperforming NAS optimizer?
- How much does the inductive bias affect the ranking of different NAS methods evaluated throughout the 10 tasks? By NAS methods I mean: Different NAS optimizers evaluated on a fixed search space.

**Other comments:**

- I think the term "NAS-Bench" is reserved for tabular/surrogate benchmarks and using it for this submission might be confusing at first for many readers, since NAS-Bench-360 in principle is only a collection of datasets and scripts for running methods (search space + NAS algorithm) on them. I would suggest the authors to reconsider changing the name of their suite.

- I would also add plots showing the performance over time of the different evaluated NAS methods on every task.

- I agree with the authors that as an AutoML practitioner, in general I am also concerned which will provide the best output: 1) optimizing the architecture when keeping the hyperparameters fixed or 2) the hyperparameters of a fixed well-performing architecture. However, one can also do both, i.e. given a fixed computational budget, start with 1) and then move to 2), or the other way around. It would be interesting to see this experiment with a NAS search space that includes the WRN architecture (does the DenseNAS space include WRN? If yes, one can very well use that.).

- Why Table 5 is in the Appendix and not in the main paper instead of Table 2?

**References**

[1] White et al. How powerful are performance predictors in neural architecture search?, In NeurIPS 2021
summary_of_the_review: I really like the contributions and the insights offered by this paper. Even though there is no novelty per se, I think the NAS community will benefit from this curated collection of diverse tasks. I lean towards acceptance and I will increase my score after most of my concerns are addressed.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors argued that existing NAS benchmarks targets the well-studied tasks, and thus proposed that we should evaluate NAS on diverse tasks. And thus the authors collected 10 tasks and did some analysis on these ten tasks/datasets to compare several NAS methods and some strong baseline DL models, such as wide resnet.
main_review: In this work, the authors thrown out an issue that the existing NAS benchmarks focused on the popular tasks -- image classification and needs more diverse tasks for the NAS evaluation. First of all, the existing NAS benchmarks have covered at least 6 different tasks/datasets and if we counting the dataset/tasks explored in the HPO works (which also try to find some architecture options), we have more architecture datasets/benchmarks in the NAS community, which is strong and diverse. So that some claims in the paper may be exaggerated.

As for the proposed "benchmark", one important feature for the NAS benchmark is the associated architecture datasets, which provides tabuler or suggrate performance data for the architectures. Unfortunately, this work does not provide such architecture dataset.

As for the collected tasks, some may be too simple and not suitable for evaluating the DL or NAS methods. For example, some datasets only has about 1K data but try to do some 2D dense prediction. It is not supuring some approaches, which require enough data, perform poorly on such dataset.

From the technical part, the authors compared a few NAS method and basic DL models. As the NAS techniques have been applied in many many different domains/tasks. If we collect the results from the papers that use DARTS in a non-CIFAR/classification application, the effect seems to be the similar. Compared to this option, the only benefit of this NAS-Bench-360 would be the calibration the NAS methods in the existing application papers.
summary_of_the_review: This NAS benchmark lacks architecture datasets for the collected ten datasets and technical novelty.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new benchmark for NAS methods, which is called NAS-Bench-360. Unlike the existing benchmark datasets for NAS, the proposed benchmark contains ten diverse tasks derived from various fields of research. This paper has tested several standard NAS methods on the proposed benchmark and confirmed that there are many gaps among the ten tasks and NAS methods.
main_review: + The attempt of creating new benchmarks or datasets is very promising for the NAS field. 
+ The proposed benchmark is much diverse compared with the existing ones.

Concerns:
- The attempt is important but this paper seems to be just providing the existing datasets. For NAS evaluation, it is also important to provide the rank of each candidate architecture in the search space because recent studies show that NAS methods find architectures achieving good performance, but their ranks in the search space are far from the best [Yu+, ICLR'20]. It indicates that it is necessary to provide the rank or something related to the rank to appropriately evaluate NAS methods, not only performance.

- Depending on a given task, an optimal search space would change. But this paper uses the limited search space for the tasks, thus I am wondering if this benchmark can appropriately evaluate NAS methods. It seems that designing search space itself is needed for diverse tasks and evaluation of NAS methods on them.

- It would be better to describe the difference between the existing NAS benchmarks and the proposed one in more detail.
- It would be nice to provide the detail of the search space in the main paper. 
summary_of_the_review: The attempt of creating a new benchmark is very important and this paper shows promising results. However, my major concern is about the design of the proposed benchmark (see concerns above). Hopefully, the authors can address my concern in the rebuttal period.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a benchmark to test the performance of NAS algorithms and search spaces on a diverse set of tasks. The benchmark consists of 10 different datasets across different modalities. On these tasks, a variety of NAS algorithm as well as search spaces are allowed a fixed amount of compute resources (in terms of GPU hours) to explore and train, and reach the final performance. The search space are not limited to architecture topologies, but also hyper-parameters. On this new benchmark, authors find that existing SoTA NAS methods may not generalize to different tasks, especially with low compute budgets. 
main_review: Strength:

1/ I think the problem this paper raises is very meaningful. Currently, NAS has been over-focused on a few problems, especially, image classification on ImageNet or CIFAR10 datasets. It is not clear wether progress achieved on NAS will transfer to other problems and applications.
 
2/ The NAS benchmark proposed in this paper is flexible and general enough to include architecture search, hyper-parameter tuning, and different base architectures. This is better than previous NAS Benchmarks, where the scope is limited to cell-level architecture space on a specific problem. Also, the computational cost for this NAS benchmark is much more manageable. 

3/ The conclusion or take-away message of this paper is useful for practitioners. Especially, depending on the computational budgets, whether to choose to use an off-the-shelf model with HP tuning or to search architectures. 

Weaknesses:
1/ Despite that authors clarified their principles of choosing different tasks in this benchmark, the exact choice is still quite arbitrary. Eventually, the reason to have a benchmark is that 1) the benchmark itself measure some metrics we care about, or 2) performance on this benchmark can generalize to a wider range of tasks that may arise in practical applications. In this paper, scenario-1) obviously does not hold. But for scenario-2), this has not been demonstrated in the paper. Specifically, does a NAS method that performs well on this benchmark can generalize better to new unseen tasks? There is no experimental or theoretical support for this, which limits the contribution of this work.

2/ The technical contribution of this paper is limited. This paper did not propose new methods or other novel contributions. An example of a more significant contribution might be to propose a better NAS algorithm that generalizes better than existing ones. 

3/ The paper is not well written and organized. For example, the metrics in Figure 1 is not clearly explained. The caption explains the metric as "A larger value indicates a larger fraction of tasks on which the method is within a multiplicative factor \tau of the best." The question is, the best of what? The best performance in terms of accuracy or error rate? What is "multiplicative factor \tau of the best"? I don't think this is clearly explained anywhere in the paper, which makes the results confusing. 
summary_of_the_review: The paper proposes a meaningful direction and benchmark for NAS. But whether this benchmark can measure the generalization of NAS method is not clear, the technical contribution of the paper is limited, and the writing of the paper should be improved.
review_title: null

Meta-review: This paper at first used the name NAS-Bench-360 for the benchmark, which confused several reviewers (who expected a tabular benchmark behind this name). The authors renamed the benchmark, which removed this issue, emphasizing that the contribution does not lie in proposed a new tabular NAS benchmark, but a new performance evaluation of NAS on different data sets.
One reviewer recommended acceptance, but 3 reviewers stuck with their rejection scores, the reasons being that 
- there are by now several papers applying NAS outside of computer vision, with a seemingly more comprehensive analysis 
- more analysis would be useful
- it is unclear how general the conclusions are that can be drawn from performance on the included datasets.
(Low technical novelty was also mentioned, but I do not believe that this type of paper can be very impactful even if it has no technical novelty.)

Overall, although I agree with the accepting reviewer that this type of work can be very useful to the community, the rejecting reviewers have too many criticisms to accept the paper in its current form. I encourage the authors to address them and to resubmit.
One note (which did not affect the decision, but which I'd like to notify the authors about) is that a reviewer found that the author identity was revealed in the anonymous codes provided by the authors (https://anonymous.4open.science/r/NAS-Bench-360-26D1).
Decision: Reject