Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This work proposes a transferrable surrogate for NAS based on Bayesian Optimization with deep-kernel Gaussian Processes. The proposed predictor can be adapt to unseen datasets rapidly by significantly reducing the search cost of NAS. On the NAS-Bench-201 and multiple unseen datasets, this work outperformed recent NAS methods including MetaD2A for the performance of the obtained architecture and search efficiency.
strength_and_weaknesses: - Strengths
   - I think this work well addressed the main limitation of MetaD2A which is the meta-learning-based transferrable NAS method. As this paper said, the meta-learning-based predictor proposed by MetaD2A can be exploited to unseen datasets after once meta-training, which significantly reduces the search time for unseen datasets. However, MetaD2A only exploits the meta-learned predictor, can not adapt to the unseen dataset even if the unseen task provides few-shot samples. This work combines the BO method to tackle the problem of transferrable predictors by allowing them to reflect feedback from unseen tasks.

- Weaknesses
  - I think while the contribution of this work in the structure of predictor is limited, this work assigned the part of the paper for them too much. For example, using both transformer-based set encoder and graph encoder together is almost same with MetaD2A, this paper described it  as the figure and performed ablation study about that. I think it would be better to emphasis the difference between MetaD2A and this work or BO + predictor as a Figure.
clarity,_quality,_novelty_and_reproducibility: I think clarity of this paper is needed to be improved as there are mixing between their contributions and previous works' contributions.
Actually, applying BO to NAS is not new, yet, to my knowledge, this work is the first to use BO for meta-learning-based predictor for NAS and addresses the important problem in meta-learning-based NAS. 
On the NAS-Bench-201 benchmark and multiple unseen datasets, their experiments are solid to support this work. Thus, i think the overall quality is good, yet, one caveat is that NAS-Bench-201 is rather small and narrow benchmark.


summary_of_the_review: The good points are that this work successfully addressed the important problem of the meta-learning-based prediction model in NAS with solid experiments. The bad points are that BO is not new in NAS domain, the composition of the paper is needed to be improved, and the benchmark that they used is small.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: As most NAS algorithms search for well-performing architectures from scratch given the target dataset, this paper follows the existing line of transferring information across different datasets to accelerate the search process. Specifically, this paper follows (Lee et al., 2021) to learn the representation of datasets and architectures from many other datasets (i.e., as training data) and then propose to leverage Bayesian Optimization (BO) with deep-kernel Gaussian Processes to find the optimal architecture for a specific dataset. Empirical results show that this method can improve NAS from scratch and also the method in (Lee et al., 2019).
strength_and_weaknesses: ## Strengths
1. Different from (Lee et al., 2019), this paper introduces deep-kernel Gaussian Processes as a probabilistic performance surrogate and intends to adaptively update this surrogate using newly evaluated architectures on the test task.
2. This paper introduces BO to achieve query-efficient (i.e., search-efficient) optimization.


## Weaknesses
1. From my understanding, this paper follows the idea of (Lee et al., 2019) (i.e., meta-learn the representation of datasets and architectures and then predict the performance of architectures based on this learned representation) and only makes a few changes to the method, i.e., replacing its deterministic performance predictor with a probabilistic performance surrogate (i.e., the deep-kernel Gaussian Processes) and using BO for the optimization based on existing works. In this view, this paper mainly combines techniques from different fields for its method and therefore the novelty of this paper is kind of weak to me.
2. In the abstract and introduction section, this paper doesn't provide a clear and detailed demonstration of its motivations, which not only makes me feel confused about why this paper is needed in the literature but also makes the contributions and the importance of this paper less convincing to me. Specifically, this paper only gives a brief introduction to the line of NAS using transferred information across datasets without elaborating more on why these existing works can not well satisfy the demand in practice and why the method in this paper is needed. Moreover, the introduction section of this paper lacks a comparison with its most related work (Lee et al., 2019) to justify the necessity of proposing the method in this paper.
3. Some claims of this paper are not well supported. For example, in the related work section, this paper does not show (a) why the missing trade-off between exploration and exploitation in (Lee et al., 2019) is important and desirable for NAS using transferred information across datasets, and (b) why deep-kernel Gaussian Processes (GP) rather than GPs using the existing kernels are needed in this paper. Moreover, the result of Table 2 in this paper (a) can not unveil the drawbacks of MetaD2A, i.e., the missing exploration vs. exploitation tradeoff and the non-adaptive update in the optimization process and (b) may not support that these drawbacks lead to the stagnated performance of MetaD2A (located at the "Results for Hypothesis 2" of Sec. 5).
4. While this paper claims that the method in this paper can adapt to function evaluations on the test task, Sec. 3 can not well support it because of the missing details of BO. I highly recommend the authors provide the details of how BO works to further justify this point.
5. The empirical results in this paper show that in most cases the proposed method can only marginally improve over MetaD2A. More experiments that can distinguish these two methods may help to justify the superiority of this newly proposed method.

## Questions
1. What's the training cost for w in your proposed method? While there is only marginal improvement over existing NAS from scratch in Figure 2, the total cost (including the training cost for w and the search cost in the x-axis of Figure 2) for the method in this paper may become a problem in practice since it may be more time-consuming.
3. As the w in this paper is trained based on the meta-datasets M, how does this method perform for the NAS problem with out-of-distribution datasets? Will it become a big issue for the proposed method? Because when only considering in-distribution datasets, directly transferring the selected architectures from other datasets can already provide competitive performances with compelling search costs as shown in Figure 2. So, the proposed method in this paper may not be necessary for this scenario. Meanwhile, if the proposed method can not guarantee its performance for the out-of-distribution datasets, this may also hinder the application of this method in practice.

clarity,_quality,_novelty_and_reproducibility: Overall, some parts of this paper need to be clarified further (see weaknesses above), and the novelty seems to be limited to me. 
summary_of_the_review: In general, my major concerns lie in the novelty and clarity of this paper. I hope the authors can address my concerns during the rebuttal period.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: 
Summary:
This paper proposes a neural architecture transfer technique. Although this problem is not new, this paper takes some reasonable measures. Its performance is worthy of recognition. This article is an above-average article and deserves a weak acceptance.


strength_and_weaknesses: 
(Negative) The neural frame transfer technique is not a new technique. Prior arts have had some exploration.

(Positive) This article provides the code and also provides the "NAS Best Practices Checklist." This is very positive because the reproducibility of NAS is widely questioned.


(Positive) The method of this article is very simple.


(Positive) This article introduces Dataset encoding, which is very reasonable.


(Negative) In the article, the definition of architecture encoding is very vague. Please give more details of it.


(Negative) During the acquisition of dataset encoding, positive and negative samples need to be randomly sampled. Is dataset encoding random? If so, please quantify this randomness or expectation.

(Positive) The experimental results in this paper are very good (Figure 2 and Figure 3).


(Negative) It would be better to add experimental validation in real open environments, such as the original ImageNet dataset and the open neural architecture search space. After all, neural architecture transfer and search techniques are meant to be used in real open environments. If the search can only be carried out in these fixed search spaces, its practical significance will be greatly reduced.


(Positive) This article performs a full ablation analysis.

clarity,_quality,_novelty_and_reproducibility: 
The quality and clarity are good. The originality is median.


summary_of_the_review: 
See "Summary Of The Paper." This article is an above-average article and deserves a weak acceptance.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a transfer NAS approach by leveraging the previous knowledge, i.e., (architecture, dataset, accuracy) triples.
It uses a graph neural network to encode the architecture information, a set transformer to encode the dataset information, and then learning a kernel function with an MLP. Empirical results on NAS-Bench-201 are strong.
strength_and_weaknesses: ### Strengths
- The motivation of this paper is great, the idea of transfer previous knowledge is straightforward and lack of exploration. To my knowledge, there are previous methods trying to learn from history [1], but using on NAS is novel.
- The results on NAS-Bench-201 are pretty robust across six datasets, and it's good to see from the ablation study that both the architecture and the dataset embedding have contributions to the final performance.

### Weaknesses
- When encounter a new dataset / architecture space, how do you propose a new architecture? Do you need to traverse lots of architectures and compare their score? I don't see discussions of this part in the paper.
- Lack of comparison to predictor based methods [2, 3], which also learns an architecture embedding and directly predict the accuracy. It could be better if the authors can show that learning the Gaussian kernel function rivals.
- For the learning process of the architecture / dataset embedding, do you just leverage the previous checkpoints or train them by yourselves? If it does, do you include the training cost into the NAS search cost?
- Another concern is that all the experiments are shown in the NAS-Bench-201 space, which is pretty small. Is it possible to show the result on larger space, e.g., DARTS space. This can also show that the learnt embedding can transfer across spaces.

[1] Yang et al. "OBOE: Collaborative filtering for AutoML model selection." KDD 2019.  
[2] Yan et al. "Does Unsupervised Architecture Representation Learning Help Neural Architecture Search?" NeurIPS 2020.  
[3] Ning et al. "A Generic Graph-based Neural Architecture Encoding Scheme for Predictor-based NAS." ECCV 2020.
clarity,_quality,_novelty_and_reproducibility: ### Quality
The writing and organization of this paper are great.

### Clarity
All contents are clear and easy to follow.

### Novelty
It's novel to leverage the previous knowledge in NAS, although each component, i.e., learning architecture and dataset embedding, BO are not new.

### Reproducibility
Code is provided, and it provides a NAS Best Practices Checklist.



summary_of_the_review: The quality of this paper is generally high. However, there are some problems in the method (how to propose new architecture) and the empirical evaluation part (lack of comparison, small search space). I would like see the rebuttal and then adjust my rating.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: All reviewers were positive about the paper. Hence, the decision is to recommend the paper for acceptance. The authors are encouraged to make the necessary changes to the paper to the best of their ability following the reviewers' comments. 

justification_for_why_not_higher_score: Highest already
justification_for_why_not_lower_score: Three out of four reviewers were excited about the paper.
Decision: Accept: notable-top-5%