Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes another benchmark dataset for NAS, NAS-Bench-x11. Compared to the original benchmarks NAS-Bench-101, NAS-Bench-301 and NAS-Bench-NLP, NAS-Bench-x11 contains the full training
information for each candidate architecture especially the learning curves so that it allows multi-fidelity NAS algorithms

NAS-Bench-x11 is a surrogate benchmark, which trains a surrogate model to predict the learning curve of the vast amount of candidate architectures. Prediction of the learning curve can be accomplished by singular value decomposition and noise modeling. 

main_review: Strength:

The paper is well motivated and clearly written. The paper is clear about limitations too.

I think this paper will be valuable to the community.  The created NAS-bench-x11 contains the full training information of different architectures, which may make it more convenient to analyse NAS algorithms and promote further NAS algorithm design. The idea of predicting the learning curve is very interesting to me. 

The authors conducted extensive experiments to demonstrate the effectiveness of the proposed method.  

Weakness:

This paper is a completion of the previously created benchmarks. Providing learning curves is also considered before in NAS-Bench-201, which weakens the novelty of this paper. So I recommend a weak acceptence.

limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: Existing NAS benchmark datasets (e.g., NAS-Bench-101) do not provide full training information of architectures, which make it infeasible to evaluate the multi-fidelity algorithms (e.g., learning curve extrapolation). To address these issues, this paper create surrogate benchmark datasets, namely NAS-Bench-111, NAS-Bench-311 and NAS-Bench-NLP11, with singular value decomposition and noise modeling methods. Extensive experimental results with the full training information on these datasets show the superiority of the proposed method.
main_review: ## Positive points:
1. The paper presents a method to extend the existing single-fidelity benchmarks to multi-fidelity ones with full training information.
2. The authors create NAS-Bench-111, NAS-Bench-311, and NAS-Bench-NLP11 benchmark datasets, which may be beneficial to develop new NAS algorithm.

## Negative points:
1. Does the proposed method apply to real world NAS application? In practice, given a dataset, we may have no available training information of any architectures. In this case, how to apply the proposed method? It would be limited if the proposed method is only able to extend the existing NAS benchmarks.
2. In line 220, why do the authors sample architectures from BANANAS, local search and regularized evolution?
3. In line 225, the authors mention that the method is able to improve the performance with the accuracies from the tabular benchmark (at epochs 4, 12, 36, 108) as additional features. More details of this should be provided.

limitations_and_societal_impact: The authors have addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The authors describe a mechanism to predict learning curves which they use to generate surrogate benchmarks from existing NAS-Bench versions without learning curves. The authors explore different ways to create learning curve embeddings, different ways to predict these embeddings using the architecture representation as an input and different ways to model the training noise. Furthermore, the authors discuss a short study on using learning curve extrapolation in context of speeding-up NAS.
main_review: The authors provide a good description of their method, alternative ways to solve each problem faced and support their decisions with empirical evidence.

My main concern is about the usefulness of this surrogate benchmark which has problems as partly discussed by the authors. It remains unclear what impact these problems would have.

The noise modelling in this paper leads to artifacts we would not see in real learning curves. First, it is very unlikely that the noise is architecture- and hyperparameter-independent. Second, given just the very few examples in Figure 1 we observe that the predicted learning curve in early epochs can already exceed to highest value of the entire learning curve (possibly due to very high noise levels).
In general, all surrogate benchmarks have the problem that they introduce some sort of bias which will not match what you would see in reality. Therefore, these datasets are more similar to synthetic datasets than real ones. They might be nice to prototype but I wouldn’t use them to evaluate NAS methods.

The experiment on combining learning curve extrapolation is a nice reminder for the NAS community to keep these approaches in mind when comparing different approaches. However, it provides no novelty since these methods have been used in the context of NAS already (see [1]).

limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: Existing NAS benchmarks often only include the architecture performance at the final or some selected checkpoint epochs. However, per-epoch information is essential for developing and evaluating multi-fidelity NAS algorithms. This work proposed to fit a surrogate model to predict the learning curve given an architecture encoding. Utilizing the proposed predictor, the author extends three existing benchmarks NASBench-101, NASBench-301, and NASBench-NLP to include per-epoch accuracies. Further, to demonstrate the necessity of developing multi-fidelity algorithms for NAS, the author also describes a procedure to extend the single-fidelity algorithms to multi-fidelity ones by using learning curve extrapolation to filter out candidate proposals with partial training. Despite possible room for improvements on the performance of the learning curve predictor used, the proposed method would enable researchers to develop and rapidly test multi-fidelity NAS algorithms. I vote for acceptance.
main_review: Strength:
1. The contribution of this paper is relevant to the NAS community and would accelerate research on multi-fidelity NAS algorithms.
2. The performance of the proposed learning curve predictor model is overall satisfactory.
3. The author adopted the proposed method to extend three existing benchmarks to include per-epoch information.

Weakness:
1. The author reported ranking correlation and KL divergence between the predicted learning curve and ground truth. The numbers seem reasonable, but the errors are still non-negligible. It is unclear how misleading these errors would be if someone tries to develop multi-fidelity algorithms by querying the proposed extended benchmark. How would the existing multi-fidelity NAS algorithms perform under the proposed benchmark, over ground-truth data? This might be doable for NASBench201 or NASBench111 (the version created without using information about the given 4 checkpoint epochs), as they provide ground-truth accuracies on at least some checkpoints?

Comments and Questions:
1. For NASBench-NLP, using the accuracy from the first few epochs as features significantly improves the learning curve predictor's performance. Can we also expect visible improvements on other spaces, if the info of the first few epochs is also provided?

2. You mentioned in para 355 that future surrogate works should save full learning curve info to avoid the need for any extra extensions. But surrogate benchmarks like NASBench-301 only train a small portion of the architectures fully, and the large majority part of the search space will be benched by the predictor. So it seems that such surrogate benchmark would still need to be extended with learning curve prediction anyway. Could you elaborate more on this?
limitations_and_societal_impact: Yes
review_title: null

Meta-review: This paper adds learning curve information to popular NAS benchmarks via a surrogate model. This is valuable to the community since it opens up current NAS benchmarks to multi-fidelity search approaches. Furthermore it demonstrates a compeling 'template' that can be used for creating new NAS benchmarks on new search spaces where it is not feasible to create tabular benchmarks due to the sheer size of the search space. 

During the review process a number of improvements have come about with the reviewers, especially when engaging with NYyv with respect to top architecture predictions. It is strongly recommended that the authors include this new information in the paper. In this vein, the paper should clearly mark the limitations of this benchmark and how NAS researchers should consume this benchmark and interpret the outcomes of their search algorithms.

This benchmark is going to be widely used by the community. So the quality of the software release and ease-of-usage is quite critical. Please pay a lot of attention to making sure that the package can be installed and run easily via standard Python package distribution channels like pip/conda and the associated data/surrogate models can be installed in one-click and there are plenty of examples to showcase how to use the benchmark. 

Decision: Accept (Poster)