Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: null
review_content: Dear reviewers, I am very curious about the positive opinion of most of you about this very common work (rejected many times).
The NAS community hasn't had a breakthrough for a long time, and some benchmarks NAS101/201/301 and so on have been plentiful, and work similar to this one exists ([1] TransNAS-Bench-101: Improving Transferability and Generalizability of Cross-Task Neural Architecture Search(TransNAS-Bench-101)). 
I think this work is very common and needs to be promoted for the following reasons：

1. I don't think benchmarking tasks other than computer vision is a great contribution: as we all know, computer vision is a more difficult task than other tasks such as language, text, signal processing, etc. Deep learning methods have been doing very well on these signal processing tasks for a long time. Evaluating NAS methods on these simple tasks is not convincing and does not have much implications.
2. this benchmark is too simple and does not provide a valid assessment of the NAS approach. This data is mostly classification tasks on simple datasets. But the more difficult ones include regression, and reconstruction tasks are not included.
3. very inadequate evaluation of NAS methods on this dataset: although this work provides a few NAS methods, this is not enough, please provide more methods to test: e.g. One-Shot NAS: SPOS, OFA, BigNAS, FairNAS, etc; Zero-NAS: FLOPs. NWOT, epe-nas，fisher，flops，grad-norm，grasp， l2-norm，jacov ，nwot，params ，snip， synflow ，zen-score, etc;
4. please provide the results of random NAS search with FLOPs as an indicator, if this result is already better than Darts method, then this benchmark is useless.
5. this work has been rejected several times, but we don't see much improvement in the revisions.

As a researcher in the NAS community, I am very eager for breakthrough work to appear. Ordinary work like this is not good enough to help the development of NAS.

review_title: The NAS community needs new breakthrough approaches and difficult benchmarks, and this ordinary work is not enough.

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: This paper introdcues NAS-Bench-360, which evaluates different architectures and NAS methods on in total 10 different tasks. This evaluation shows, that different NAS methods, which perform very well on computer vision tasks do not transfer with a high performance to other domains. This paper also evaluates the architectures from the NAS-Bench-201 search space on 2 diverse tasks, showing that these architectures do not generalize well. Lastly, this paper compares TE-NAS, a zero-shot search method, with GAEA on several tasks, resulting in varied performance over these considered diverse tasks. 
strengths: + This paper tackles an important fact, that NAS methods should be evaluated on different domains to indeed show superiority of a proposed NAS method. 
+ The experiments underline the importance of this tasks and the claimed contribution.
+ Most of the selected tasks are reasonable and sufficiently different for this evaluations
+ The paper is well written and clearly structured
+ The provided code and URL are clear. 
weaknesses: - The main concern, is that this paper claims to introduce a benchmark, whereas this paper mainly introduces an evaluation of different NAS methods on different task. 
-  A benchmark should provide detail information about the found architectures and their performance in their provided documentation. In the best case also the learning curves or the weights for future research. 
- Some of the considered datasets only have a small size, especially the datasets, which are used for the NAS-Bench-201 evaluations. 

correctness: The construction of this paper is clear and reasonable and the made claims are correct given the shown experiments. 
However some of the claims (see Additional Feedback) are not clear to me. 

clarity: This paper is clearly and well written. 
relation to prior work: This paper includes all important related work and clearly states the difference to prior work and previous contributions. 
additional feedback: As stated in the weakness part some of the considered datasets are small. Does this small dataset size affect the performance of the NAS-Bench-201 architectures in any way? 
Also due to the different NAS methods also different search spaces are compared with each other. How do the actual underlying predefined search spaces influce the overall ability to transfer methods to other domain tasks? Do some of the NAS methods perform better, because of the search method and optimizer or because of the search space? That is not clear to me.
Where are the performances of networks using FNO or XD-operations in the paper?
In addition it is not clear to me how XGBoost is included? What is the underlying architecture search space on which XGBoost is applied to as a regression task?

Given the weaknesses and the additional feedback, I tend to reject this paper. 
However, I hope that the authors can address my concersn in the discussion phase.
review_title: Important idea with missing insights

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper presents a benchmark suite which is a collection of datasets / tasks to enable evaluate of different search spaces and search algorithms. The datasets also include precomputed performance of NAS-Bench-201 architectures on two of the tasks.
Based on the tasks collected, the paper presents experimental results of several NAS algorithms on multiple search spaces and tasks. The authors conclude that the analysis by previous papers on computer vision tasks do not generalize to diverse tasks. In addition, the zero-cost proxies perform inconsistently as well.
strengths: - A diverse collection of tasks with scripts provided to evaluate and reproduce results of different NAS algorithms and search spaces.
- Comprehensive experimental results showing the shortfall of exisiting algorithms and search space on generalization.
weaknesses: - As the authors already mentioned, this 'benchmark' is an organization of tasks such that users can evalute different NAS algorithms and search spaces. Since it is not a pre-computed tabular benchmark, I wonder if this work can be called a benchmark. 
- Also I am interested to see if the authors have provided an API or standardization to allow people to contribute new search algorithms, search spaces and precomputed data.

correctness: - To the best of my knowledge, the dataset is constructed in a sound way, together with well structured evaluation methods.
clarity: The paper is well structured and clearly written.
relation to prior work: It is clearly discussed.
additional feedback: - In the experiment, how do you define a good set of hyperparameters for the constrained setting?
- It is expected that a search space for one task does not perform well on another task. Are there any other search spaces specifically designed for those non-computer vision tasks?
- Section 5 is referencing Table 5, should it be Table 4?
review_title: Well motivated work 

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: This work presents a benchmark for evaluating NAS methods on different tasks. This work has done more evaluation and analysis. However similar work already exists and some of their tasks are more reasonably selected.
strengths: 1. This article is very well organized and written.
2. The article gives a more detailed analysis and experiments.
weaknesses: The biggest problem idea is similar to Trans-bench by Huawei in CVPR[1]：
1. The multitasking selection of this work seems less scientific compared to Trans-Bench.
2. Lack of evaluation of One-shot NAS, zero-shot NAS.
3. This work has actually been proposed for some time, but no other NAS work has been reviewed on it.

[1]  TransNAS-Bench-101: Improving Transferability and Generalizability of Cross-Task Neural Architecture Search(TransNAS-Bench-101)
correctness: Yes
clarity: Yes
relation to prior work: No
additional feedback: None
review_title: An ordinary research

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This work raises a concern about recent NAS research overfitting on Computer Vision (CV) tasks and attempts to alleviate the issue by providing NAS researchers a benchmark that assesses their work's viability for tasks beyond widely popular CV tasks and instead for tasks generally conducive to the use of Convolutional Neural Networks instead. The primary contribution of the authors is a collection of tasks, including datasets and performance metrics, which they have named "NAS-Bench-360", that aims to provide a standardized baseline for comparison of any NAS algorithm's performance across multiple task domains. A secondary component of their work is a dataset of pre-computed neural network configuration performances for two of the 10 tasks included NAS-Bench-360, in the same vein as and supplemental to that provided by NAS-Bench-201 on 3 CV tasks.

The authors use their proposed benchmark to evaluate a number of NAS search spaces and algorithms in order to demonstrate that many existing state-of-the-art works in the field fail to generalize well to tasks other than CV. They also present an analysis of the viability of using NAS for researchers on constrained budgets working on specific research tasks that could potentially benefit from employing NAS but do not necessarily need to. Currently, some further analyses have been mentioned in the paper but are missing from the appendix.

Overall, a well written, coherent paper which makes a valuable contribution to the scientific community. Once the missing analyses have been added in, I would be happy to increase the rating of the paper.
strengths: - Correctly identifies and addresses an issue with existing NAS research, namely, overfitting on a handful of CV tasks.
- Sensible and valid analyses of the presented work and the proposed hypotheses.
- Open-source code has been made available for facilitating reproducibility.
- Benchmark/dataset hosting appears reliable and the licenses are open.
- The paper has been written well and is easy to understand.
weaknesses: - There are some incomplete and missing analyses in the paper. Unless I somehow missed it, the analyses mentioned in Section 5.2.2 that should have been present in the appendix were absent.
- There is room for improvement by conducting an internal validation of the diversity of the chosen tasks, thereby justifying their applicability to the experiments conducted here, i.e. to testing how badly NAS algorithms generalize. The "expert" models mentioned in appendix D would have made for a very decent baseline for demonstrating this, by illustrating the drops in performance in an "expert" model when it is used for a task that it has not been designed for. I must admit, however, that I am unaware of the computational costs involved in conducting such an analysis.
- The authors have a very rich dataset at hand which could have been used for a much more interesting and in-depth statistical analysis than has been presented, missing out on a lot of potential for the paper, going beyond comparing only the final performances of NAS algorithms across tasks but also including an analysis of their behaviours in multiple dimensions. (See "Additional Feedback" for an example)
correctness: To the best of my understanding, the evaluations and experiments carried out in the paper are justified and valid.
clarity: The paper is well-written, easy to follow, cohesive and logical.
relation to prior work: A related works section has been presented with appropriate contextualization of the presented work.
additional feedback: 1. The claims made in section 5.2.2 hinge on the presence of an accompanying analysis in the appendix, which I was unable to locate.
2. Have you considered using a log-scale on at least one of the axes in Figure 2 for the sake of visual clarity?
3. Line 180-181: The last sentence here probably needs to be rephrased or is missing a phrase, that "since" in there makes no sense to me.

I am aware that the following apply only to the Appendix, and hence am including this here only as possible "bonus points":
- Appendix F looks very much incomplete.
- Possible extra statistical analysis: As tables 8 and 9 demonstrate, there exists both a significant disparity in the number of FLOPS for the models being searched by different algorithms in different search spaces and for different tasks. This warrants a possibly very interesting statistical analysis of the impact this has on the final performance for a given task as well as changes in behaviour across tasks. How much disparity does exist in the size/computational capacity (FLOPS) of the models searched by the same search algorithm/search space across tasks? How correlated are model capacity (FLOPS) and model performance (figure 5 only show model size vs performance)? I imagine the results of such an analysis on behavioural disparities across tasks would actually also fit and complement the main paper very well. Moreover, you should technically already have all the data needed for such an analysis.
review_title: An important step towards generalizing NAS beyond Computer Vision

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The paper proposes NAS-Bench-360 (NB360), a diverse set of tasks and a collection of metrics for Neural Architecture Search (NAS). NB360 extends beyond popular computer vision benchmarks and some precomputed performance is provided for two tasks. The paper empirically shows inconsistent performance of several modern NAS algorithms across tasks of NB360, demonstrating its usefulness in evaluating the robustness of NAS methods.
strengths: 1. The question of how well NAS algorithms generalize across tasks is an important one for both practical applications and scientific research.

2. The dataset covers a range of applications, dimensionality, learning objectives, as well as scales.

3. Code, data, models, seeds, and environment are released to facilitate reproduction.

4. Empirical results show many modern NAS methods are not robust as they perform inconsistently across tasks. It identifies an important limitation of many modern NAS methods which may facilitate future research to address them

5. Resource-constrained experiments provide some (potentially counterintuitive) insights for practitioners that they may be better off using a fixed handcrafted domain-specific model.

6. The paper discusses some limitations of the work and provides some insights regarding future improvement.

weaknesses: 1. Perceiver IO needs a certain scale to perform well. While one can argue there is value in testing it at a very small scale, it is not entirely a fair comparison.

2. It is helpful to spell out that the AWS p3.2xlarge has the 16GB version of V100.

3. Natural language processing (NLP) is another important and popular field for modern machine learning which includes NAS research. Is there a good argument for not including NLP tasks?

4. Figure 4 left panel is misaligned

correctness: 1. The datasets of this work is a collection of existing datasets.

2. The authors proposed a few evaluation metrics including median rank, performance improvement over baseline, and performance profiles. Each of them has pros and cons but overall it is a good set of metrics.

3. The computation restriction of requiring a model to fit on a single V100 GPU, although could be justified, is not justified in the paper. On one hand some labs can not afford V100 GPUs, on the other hand many architectures (like Perceiver IO in the paper) really require a certain scale to perform well which needs multi-GPU training. In that regard, it is unsurprising to see Perceiver IO underperform in the paper.

clarity: The paper is generally well written and easy to follow.

relation to prior work: The authors provide a good discussion of related work as well as how this work differs from them.

additional feedback: Detailed suggestions for improvement can be found in “weakness” and "correctness" sections.
review_title: A good tool for probing NAS robustness

Meta-review: metareview: NAS-Bench-360 enables the evaluation of NAS methods across a diverse set of tasks and domains and the author's experiments demonstrate that NAS methods are not generally robust across all domain/tasks. 

The authors also make a strong argument that a NAS benchmark should be a combination of a search space and a search algorithm, challenging pre-conceived notions of what a benchmark is for this research community.  (Existing pre-computed tabular benchmarks do not necessarily allow for innovation of the search space.)

Overall, the benchmark and evaluations behind NAS-Bench-360 will benefit the broader research community and are significant improvement over existing work. The authors should consider adding NLP tasks to strengthen the paper.

confidence: 3: The area chair is somewhat confident
Decision: Accept