Stronger NAS with Weaker Predictors

Abstract: Neural Architecture Search (NAS) often trains and evaluates a large number of architectures. Recent predictor-based NAS approaches attempt to alleviate such heavy computation costs with two key steps: sampling some architecture-performance pairs and fitting a proxy accuracy predictor. Given limited samples, these predictors, however, are far from accurate to locate top architectures due to the difficulty of fitting the huge search space. This paper reflects on a simple yet crucial question: if our final goal is to find the best architecture, do we really need to model the whole space well?. We propose a paradigm shift from fitting the whole architecture space using one strong predictor, to progressively fitting a search path towards the high-performance sub-space through a set of weaker predictors. As a key property of the weak predictors, their probabilities of sampling better architectures keep increasing. Hence we only sample a few well-performed architectures guided by the previously learned predictor and estimate a new better weak predictor. This embarrassingly easy framework, dubbed WeakNAS, produces coarse-to-fine iteration to gradually refine the ranking of sampling space. Extensive experiments demonstrate that WeakNAS costs fewer samples to find top-performance architectures on NAS-Bench-101 and NAS-Bench-201. Compared to state-of-the-art (SOTA) predictor-based NAS methods, WeakNAS outperforms all with notable margins, e.g., requiring at least 7.5x less samples to find global optimal on NAS-Bench-101. WeakNAS can also absorb their ideas to boost performance more. Further, WeakNAS strikes the new SOTA result of 81.3% in the ImageNet MobileNet Search Space. The code is available at: https://github.com/VITA-Group/WeakNAS.


Reviews and author responses:

 Q1: Lack of theoretical justification. Assume that the search space is uniformly at random distributed, what value is selected to be the cutoff threshold? In this paper the number of iterations is used as shown in Figure 3, however, this is still implicitly shrink the search space rather than explicitly, making it hard to explain.

- Sorry, your assumption appears to be impractical and wrong. Apparently, if NAS search space is indeed randomly distributed, any other NAS methods would not be better than random search. In fact, the highly structured NAS space is revealed to be true by many NAS prior works [1,2]. Interestingly, those include one you brought up in your review: arxiv 2005.02960. You are also welcome to check our  visualized distribution of NAS-Bench-101/201 in the supplementary. Therefore, we respectfully disagree with you on your “justification” criticism, and we hope you can take a look at other reviewers unanimously agreeing on and appreciating our rationale. 

[1] Exploring the Loss Landscape in Neural Architecture Search, Colin White, Sam Nolen, Yash Savani

[2] Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, Piotr Dollár, Designing Network Design Spaces

Q2: Lack of novelty

- We respectfully disagree on this. We are familiar with the literature and we never claimed to have invented “ iteratively refining predictors” or so. The principled re-formulation of predictor-based NAS, and then the weak predictor solution inspired by the insight of specially structured NAS space (explained above), makes our actual core contribution. In the paper, line 301-305 clearly explains the difference and advantage of our WeakNAS with regard to the related BO-type NAS, showing that “an oversimplified version of BO actually outperforms BONAS, showcasing that the simplification using weak predictor indeed does not compromise NAS performance. The claim has been well-validated by our complete empirical comparison on both NAS-benches and open-domain search.

- We also appreciate the fact that our novelty has been extensively positively recognized by all other reviewers, e.g. “scheme is simple and novel in NAS literature” (Lp96), “claims are well-supported by the experimental results” (NXMZ) and “general and can be used with various predictors or architecture encodings” “contribution clear to comprehend” ( vUV5)

Q3: Lack of comparison. There are so many performance prediction works missing.
- We are happy to cite all the mentioned works in the final version to help additional literature comprehensiveness. Yet below, we summarize the results from all of your mentioned to-compare papers in the following table.  “-” denotes that the paper has no result on the corresponding nas-bench or search space. In case some papers have multiple implementation ways and results, we always report the highest performance one in each setting, e.g. for [6], we show the arch2vec-RL baseline on NASBench101 and arch2vec-BO baseline on NASBench201, For [7], we show the BANANAS baseline. *are results extracted from Figures. According to the results in the Table below, our WeakNAS consistently achieves SoTA over all other methods on NAS-Bench-101 Validation/ Testing Set, NAS-Bench-201, NASNet and MobileNet search spaces.

| Method | NAS-Bench-101 (Val) | NAS-Bench-101 (Test) | NAS-Bench-201 (CIFAR10) | NASNet (ImageNet)| MobileNet (ImageNet) |
|:----------|:-----------:|:-----------:|----------:|------------:|------------:|
| WeakNAS | 94.77@200 queries| 94.05@100 queries 94.18@200 queries| 91.54@100 queries 91.58@200 queries | 77.1@800 queries  |81.2@800 queries  |
| [1] (ICLR Workshop 2018) | - | - | - | -  | - | - |
| [2] BANANAS (ArXiv Preprint, Oct 2019) | - | *93.97@40 TPU hrs (~100 queries) | *91.55@48 GPU hrs (~100 queries) | -  | - |
| [3] (CVPR 2020) | - | 94.01@1000 queries | - | -  | - |
| [4] (ArXiv Preprint, Apr 2021) | - | 94.01@1000 queries | - | -  | - |
| [5] (ArXiv Preprint, July 2020) | - | 94.15@1000 queries | - |  - | 76.6@1000 queries  |
| [6] (NeurIPS 2020) | - | 94.10@400 queries | 91.41@$10^6$s (~500 queries)  | - | - | - |
| [7] (ArXiv Preprint, Feb 2021) | - | *94.01@40 TPU hrs (~100 queries) |  - | - | - | - |
| [9] Local Search (Arxiv Preprint, Apr 2020) | - | - |  - | - | - | - |
| [10] (ICML Workshop 2020) | - | - |  - | - | - | - |
| [11] (ECCV 2020) | 94.70@200 queries  | - |  - | - | - | - |

- We noticed some of your mentioned work focus on designing better architecture encoding for NAS. They yield appealing results, but clearly their focus is orthogonal to ours (better predictor scheme). Hence, there is no conflict or inherent tension between the two directions: we like these papers and we are curious to see if WeakNAS can be combined with their novel encodings.

- We conduct preliminary results and report in the Table below. Combining WeakNAS with one of such examples, CATE embedding [8], can further exceed the best performance of CATE (cate-DNGO-LS) reported on NAS-Bench-101, especially improving avg. rank by over 60%. We will try more advanced encodings and we plan to dig further collaborations along this direction. Meanwhile, we emphasize again this makes an orthogonal effort and does not compromise the original contribution of WeakNAS.

- | Method |  \#Queries  | Test Acc.(\%) | SD(\%) | Test Regret(\%) | Avg. Rank |
|:----------|----------:|------------:|------------:|----------:|---------:|
| CATE (cate-DNGO-LS) | 150 | 94.10 | - | 0.22 | 12.3 |
| WeakNAS | 150 | 94.10 | 0.19 | 0.22 | 12.3 |
| WeakNAS + CATE | 150| 94.19 | 0.12 | 0.13 | 5.24 |

- In the current paper, we already compare with many SOTA predictor-based NAS approaches, including the latest updated results of LaNAS after their TPAMI 2021 acceptance (much stronger than their initial version in 2019), as well as other well-recognized and most related baselines such as SemiNAS (NeurIPS 2020) and BONAS (NeurIPS 2020). We are not sure if it is really fair to ask a NAS paper to compare with every other work exhaustively, even ones only loosely related or not yet peer-reviewed. But we believe our existing comparison is already robust enough to support our method, and all other reviewers agree that “The empirical results are very strong and comprehensive” and “significant” “well support the claims”.

[1] Bowen Baker, Otkrist Gupta, Ramesh Raskar, Nikhil Naik. Accelerating Neural Architecture Search using Performance Prediction, ICLR Workshop 2018

[2] Colin White, Willie Neiswanger, Yash Savani. BANANAS: Bayesian Optimization with Neural Architectures for Neural Architecture Search, arXiv Preprint

[3] Yehui Tang, Yunhe Wang, Yixing Xu, Hanting Chen, Chunjing Xu, Boxin Shi, Chao Xu, Qi Tian, Chang Xu, A Semi-Supervised Assessor of Neural Architectures, CVPR 2020

[4] Colin White, Arber Zela, Binxin Ru, Yang Liu, Frank Hutter. How Powerful are Performance Predictors in Neural Architecture Search? arXiv Preprint

[5] Renqian Luo, Xu Tan, Rui Wang, Tao Qin, Enhong Chen, Tie-Yan Liu. Accuracy Prediction with Non-neural Model for Neural Architecture Search. arXiv Preprint

[6] Shen Yan, Yu Zheng, Wei Ao, Xiao Zeng, Mi Zhang. Does Unsupervised Architecture Representation Learning Help Neural Architecture Search? NeurIPS 2020

[7] Colin White, Willie Neiswanger, Sam Nolen, Yash Savani. A Study on Encodings for Neural Architecture Search. NeurIPS 2020

[8] Shen Yan, Kaiqiang Song, Fei Liu, Mi Zhang. CATE: Computation-aware Neural Architecture Encoding with Transformers, ICML 2021

[9] T. Den Ottelander, A. Dushatskiy, M. Virgolin, P. A. N. Bosman. Local Search is a Remarkably Strong Baseline for Neural Architecture Search. arXiv Preprint

[10] Sheheryar Zaidi, Arber Zela, Thomas Elsken, Chris Holmes, Frank Hutter, Yee Whye Teh. Neural Ensemble Search for Uncertainty Estimation and Dataset Shift. ICML Workshop 2020

[11] Xuefei Ning, Yin Zheng, Tianchen Zhao, Yu Wang, Huazhong Yang. A Generic Graph-based Neural Architecture Encoding Scheme for Predictor-based NAS. ECCV 2020







This paper proposes a novel method for the Neural Architecture Search problem. The authors reinterpret the NAS problem as a bilevel optimization problem, and show that the presented solution follows naturally from this formulation. Its main idea is to iteratively train a proxy model (performance predictor) based on the current iteration's best knowledge about the search space, and make new predictions on the search space, adding them to the training set, motivated from Bayesian Optimization. This allows evaluating fewer samples to reach the same accuracy, while also being less dependent on the design and performance of the proxy model. They show that the proposed method works well on NAS-Bench-101, -201, and in OFA search space, using fewer architecture samples and achieving higher accuracy than SOTA methods.  The proposed neural architecture search method (WeakNAS) offers an alternative approach to proxy model-based NAS. Pros:
- Better sample efficiency
- Finds higher accuracy models
- Lower variance in performance over multiple runs

The claims are well-supported by the experimental results. The paper is generally well-written, but it would have been better if the algorithm was also presented concisely in the form of pseudocode.

It is not clear if the results are meaningful, though. In Table 2, It shows that with the same number of queries, WeakNAS shows nearly the same performance as Local Search, a very simple baseline. This is perhaps because the NAS-Bench-101 space is very small, but given no comparison between WeakNAS and Local Search in other settings, I have no other choice to believe that WeakNAS is no better than Local Search. It is not clear how the predictor models are trained. In the case of MLP, do the authors start SGD from random weights at each iteration? If so, it seems to me that the algorithm needs at least $O(n)$ SGD steps at each iteration, where $n$ is the number of samples in the training set so far. This makes the entire algorithm have the time complexity of $O(n^2)$. This means that the cost of training the proxy model could quickly outweigh the cost of evaluating the samples if $n$ gets large enough.

To me, it seems vital in proving WeakNAS' superiority to show how much actual (wall-clock) time is saved compared to other methods in the overall search process. However, there is no plot with the elapsed wall-clock time. Instead, only the iteration number or the number of samples is provided as the x-axis. This raises the question of whether this algorithm is realistic in saving the actual search time. It would also be helpful to provide how much time is spent on training the proxy models and predicting the samples' performances vs. actually evaluating the selected architectures' performances. I suspect that running the predictor on all of the architectures is somewhat costly, in addition to the training cost.

In the footnote of page 3, the authors reveal that only 1,000 random samples' performances are predicted at each iteration, from which 100 best-performing samples are evaluated. They also state that (1) "the probability of some good examples in the total 1000 samples is very high", and (2) "progressive evolving will make this probability even larger". However, the size of the OFA search space is on the order of $10^{18}$. From my understanding of the method, this means that even if after 10,000 iterations, the algorithm only "sees" $10^{-9}$ percent of the entire search space. It is extremely likely that the optimal architecture lies outside this tiny subspace since the candidate samples were sampled entirely randomly. I suspect that Local Search might even do better than WeakNAS in this case, since unlike WeakNAS, Local Search utilizes the knowledge of the current samples to decide which architectures to sample next. Therefore I am not convinced that assertion (1) is true. Moreover, since the method of sampling the candidates is entirely random, I do not see how (2) is true, either. From my understanding, the progressive evolving should only increase the probability of selecting the 100 samples that perform better than the other 900 from the 1000 candidates, not the probability of there being a better model in the 1000 samples from the $10^{18}$ search space. I hope to get some clarification on this part.

 All my concerns are resolved now. I will raise my score to 7.

 Q1: Does this mean that you train for 200 mini-batch SGD steps, regardless of how much training data there is?

- Yes, we inherit the hyperparameter from python package sklearn [1], our predictor will only go through 200 iterations *200 batchsize = 400K samples. Since at each iteration we only have 100 new training samples and a total sample budget of merely 1000, enumerating 400K samples is well enough in our setting.

[1] https://scikit-learn.org/stable/modules/generated/sklearn.neural_network.MLPRegressor.html

Q2: The search space does not seem "highly structured"

- We found the hyperparameter for t-SNE and plotting is not ideal to showcase the structured NAS search space.
As a result, we did a few simple fixes to resolve the issue
  - In the t-SNE function [1]
    - We tune down the perplexity from 50 to 20, which makes the samples less “squeeze” together and allows the space between natural clusters to be larger in the embedded space.
  - In the Visualization plot
    - We reduce the scatter point size and adjust the ordering of samples to make sure good samples (yellow one) are “popped out” and are not occluded by the bad samples (green and blue ones)

- We have updated the visualization in this link (https://tinyurl.com/he7zp8be). To ensure reproducibility, we also attached an anonymized jupyter notebook to easily reproduce this visualization in this link (https://tinyurl.com/2xhpvz8y). The updated visualization shows a structured NAS search space where the good architectures are clustered together (shown as the yellow regions), interestingly we found this also aligns with recent study [2] on NAS search space, where they show architectures with similar accuracy are more likely to be neighbors.

[1] sklearn.manifold.TSNE: https://scikit-learn.org/stable/modules/generated/sklearn.manifold.TSNE.html

[2] Exploring the Loss Landscape in Neural Architecture Search, Colin White, Sam Nolen, Yash Savani

Q3: uniformly sampling the 10K evaluation pool. Do you plan to include it in the final draft?

- We promise to include clarification of the 10K evaluation pool in the final draft.

This paper proposes a progressive method to implicitly shrink the search space by updating the new weak predictors given the prediction results from the previous weak predictors. Experiments show that the proposed method costs fewer samples to find top-performance architectures on various search spaces.  +: Results on NAS-Bench-101, NAS-Bench-201 are good. Results on NASNet and MobileNet look promising. 
+: Simple and effective. 

-:
Novelty. Since one-time fitting as used in neural predictor costs much initialization time (since it needs k = 172 architectures to fit a strong GCN predictor) and easily causes overfitting, iteratively refining predictors starting with a small portion of initialized architectures in active learning way has been used in many previous works. Could the author[s] state their novelty more clearly?

Related works and experimental comparison. There are some performance prediction works are missing, from the seminal work https://arxiv.org/pdf/1705.10823.pdf to https://arxiv.org/pdf/1910.11858.pdf, https://arxiv.org/pdf/2005.06821.pdf,  https://arxiv.org/pdf/2007.04785.pdf, https://arxiv.org/pdf/2104.01177.pdf. Architecture encodings/representation works https://arxiv.org/pdf/2006.06936.pdf, https://arxiv.org/pdf/2007.04965.pdf, https://arxiv.org/pdf/2102.07108.pdf. Local search work is proposed in https://arxiv.org/pdf/2004.08996.pdf, https://arxiv.org/pdf/2005.02960.pdf. 1. Show proof of concepts of Figure 2 in the paper, and why it is novel compared to previous related works. 

2. Provide more discussions/ablation results on different predictors with different encodings in the main text and Table 3, 4, Figure 4, 5. 

3. Add results comparison or discussion with https://arxiv.org/pdf/2004.01899.pdf and https://arxiv.org/pdf/2006.08573.pdf.


 Dear Authors,

Please provide answers to Reviewer NXMZ's questions. It would be nice if you provide more explanations about the new visualization you provided in the response, and in what sense the search space is "highly-structured". 

Thanks,
AC

 Thank you for the further clarifications about the training time and your efforts for the response.

1. > We trained it for only 200 iterations with a batch size of 200  

  Just to be sure I understood correctly: Does this mean that you train for 200 mini-batch SGD steps, regardless of how much training data there is? Or do you mean that you train for 200 epochs over the entire training set?

2. Thank you for the explanation about the search space and the sampling method. 

  A lot of my concerns have been addressed, and I understand why your method works well much better now. The visualization is also very interesting. I have one question regarding the visualization: You say that the search space is "highly structured" as seen by the T-SNE visualization, but looking at the figure on the top-left, there seems to be no correlation between the "structure" and the ranking: The ranking seems to be distributed uniformly regardless of where in the structure a point is located at. So I ask, from what do you conclude that the search space is "highly structured"?

Also, your paper doesn't seem to mention the method of uniformly sampling the 10K evaluation pool. Do you plan to include it in the final draft?

 Thanks for your responce. I will keep my score.

 Thanks for your reply. The rebuttal have addressed all of my concerns.
After reading the comments of other reviewers, I will keep my rating.




 Q1: $n$ is not constant, but increases at each iteration $(n_i=M_{0}+M_{i})$, where $M_{0}$ is the number of initial training samples, is the iteration index, and  is the number of samples added to the training set at each iteration), so you cannot simply multiply $k$ and $O(n)$ and say that the overall time complexity is $kO(n)$. For total  iterations, you need $\sum_{i=1}^{k} {n}_{i} = O(k^{2})$ SGD steps. I realize that the notation was not clear in my initial review, but my point still stands.

- Thanks for the additional clarifications of notations! Now we get your question, and you are right that in this case the time complexity could be noted as $O(k^{2})$. While you are correct if $k$ becomes very large, the cost of training the proxy model can grow to become notable: that doesn’t happen in our practice since $k$ is tied to the total sample budget and is kept rather small. For example, given the 1000 sample budget on ImageNet MobileNet search Space, we can only set the K as small as 10. Consequently, the time complexity would remain small.

Q2: A simple MLP is indeed faster to train compared to other models. However, that seems a lot faster than I expected. How many epochs do you train the MLP for?

- We trained it for only 200 iterations with a batch size of 200, and we empirically found it already achieved good performance. We use the MLP implementation based on PyTorch.

Q3: Assume there are 5000 clusters in total and the whole space is uniformly distributed in such 5000 clusters. This assumption does not seem realistic. If this was true, then we can find 'good' architectures with 0.865 probability in any search space by just randomly sampling 1,000 samples, and testing all of them.

- We extend WeakNAS to open domain settings by (a) Construct the evaluation pool $\bar{X}$ by uniform sampling the whole search space $X$ (b) Apply WeakNAS in the evaluation space $\bar{X}$ to find the best performer. For instance, when working with MobileNet search space that includes ${10}^{18}$ architectures, we uniformly sample 10K models as an evaluation pool and further apply WeakNAS with a sample budget of 1K, it achieves 81.3% accuracy on ImageNet.


(a) Construct the evaluation pool $\bar{X}$ from the search space $X$
- We uniformly sample an evaluation pool $\bar{X}$ to handle the extremely large MobileNet search space ($|X|$=${10}^{18}$), since its not doable to predict the performance of all architectures in $X$. We use uniform sampling due to a recent study [1] reveal that human-designed NAS search spaces usually contain a fair proportion of good models compared to random design spaces, for example, in Figure 9 of [1], it shows that in NASNet/Amoeba/PNAS/ENAS/DARTS search spaces, Top 5% of models only have a <1% performance gap to the global optima. In practice, the uniform sampling strategy has been widely verified as effective in other works of predictor-based NAS such as [2-4], For example, [2][3][4] set $|\bar{X}|$ to be 112K, 15K, 20K in a search space of ${10}^{18}$ networks, respectively. In our case, we set $|\bar{X}|$ = 10K.

(b) Apply WeakNAS in the evaluation space $\bar{X}$
- We then further apply WeakNAS in the evaluation pool $\bar{X}$. This is because even with the evaluation pool $|\bar{X}|$ = 10K, it still takes days to evaluate all those models on ImageNet (in a weight-sharing SuperNet setting). Since the evaluation pool $\bar{X}$ was uniformly sampled from NAS search space $X$, it preserves the highly-structured nature of $X$. As a result, we can leverage WeakNAS to navigate through the highly-structured search space. WeakNAS build a iterative process, where it searches for some top-performing cluster at the initial search iteration and then “zoom-in” the cluster to find the top performers within the same cluster (as shown in this visualization https://tinyurl.com/he7zp8be). At $k$-th iteration, WeakNAS balance the exploration and exploitation trade-off by sampling 100 models ${S}^{k}$ from the Top 1000 predictions of the predictor $\tilde{f}^{k}$ and use the promising samples ${S}^{k}$ to further improve performance of the predictor $\tilde{f}^{k+1}$ in the next iteration .

- We leverage WeakNAS to further decrease the number of queries to find the optimal in $\bar{X}$ by 10 times, the search cost has dropped from 25 GPU hours (evaluate all 10K samples in the evaluation pool $\bar{X}$) to 2.5 GPU hours (apply WeakNAS to further subsample 1K in the 10K evaluation pool $\bar{X}$), while still achieving a solid performance of 81.3% on ImageNet.

[1] Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, Piotr Dollár, Designing Network Design Spaces

[2] Neural Predictor for Neural Architecture Search

[3] Accuracy Prediction with Non-neural Model for Neural Architecture Search

[4] FBNetV3: Joint Architecture-Recipe Search using Predictor Pretraining

 Thanks for your response!

Q1: Please consider putting the tsne plot into the paper (remember to change top row a-b-b-b to a-b-c-d)
- We promise to include the t-SNE visualization in the final version. We've fixed the typos and change the captions from row a-b-b-b to a-b-c-d. Here is an updated version of visualization: https://tinyurl.com/he7zp8be

Q2: [6] in the above table should be around 94.10 given 400 queries (see Table 2)
- We've fixed the typos in the above Table.



 We are happy to see that our last response seems to clear the air a bit, e.g. “negative social impact”“lack of theoretical justification” and so on. We will also update our responses once our discussion reaches more effective consensus.  

Our response does not draw any clue or guess on anybody's identity. We absolutely respect the double-blind rule and focus our discussions to be professional and fact-based. We hope you will honor the same.

Regarding your (updated) questions: 

Q1: Why do you state that the proposed method is a principled way to guide the search trajectory for good and bad regions? Could author[s] show some illustrations/figures for the proof of concept from Figure 2 in the paper.

- By “principal”: we actually borrowed this word from Reviewer “vUV5” that describes our unified optimization form for predictor NAS. We quote his/her full words here for context: “This paper used a principled re-formulation of predictor-based NAS to motivate their method. The bi-level form leads to a coordinate descent-like heuristic algorithm, that is further interpreted as the alternating iteration of sampling and local fitting.” We think this explains well the principal aspect of our work.

- For the proof of concept in Figure 2, we show a 2D visualization of search dynamics in NAS-Bench-201 Search Space (CIFAR100) using t-SNE here: [https://tinyurl.com/he7zp8be](https://tinyurl.com/he7zp8be).

    - At $k$-th iteration, we randomly sample $M$=$40$ new architectures from the $Top N$=$200$ ranked architectures in $\tilde{P}^{k}$, top row from (a)-(d) show the sampling space $\text{Top}_N(\tilde{P}^{k})$, while bottom row from (e)-(h) show the sampled architectures $S^k$. The Ranking of architectures is encoded by color, those not-sampled architectures are greyed out. Based on the visualization, we have the following observations:
        - The NAS-Bench-201 search space is high-structured, good performing architectures are usually grouped in clusters.
        - The sampling space $\text{Top}_N(\tilde{P}^{k})$ and sampled architectures $S^k$ are consistently evolving towards more promising regions, this can be noticed by the increasing more yellowish color of the highlighted samples.

    - We will include this visualization in the final version of the paper.

Q2: The proposed method is more like a learning approach. As a fair comparison, I would suggest using the same predictors (e.g. GCN, MLP) for an apple-to-apple comparison with BONAS and LaMCTS. From the reply to reviewer Lp96 I can rarely see an advantage. It would be also good if author[s] do the same for latent action MCTS and make a comparison instead of directly reporting their number since you emphasize these works throughout the paper.

-  We have shown our apple-to-apple comparison using the same GCN BONAS in the response to Lp96. WeakNAS has superior performance to BONAS using exactly the same predictor. At 1000 queries, we reduce regret from 0.07 to 0.04 and average rank from 1.7 to 1.4, both of which are often regarded as more meaningful metrics than test accuracies.

-  We are however unable to do the same for LaNAS, since it is also a general framework to do exploration and exploitation trade-off by constructing a series of binary partition trees. Unlike BONAS, we cannot simply replace WeakNAS predictors with binary partition trees in LaNAS, because binary partition trees do not keep ranking information and are hence not directly compatible with our “TopN” sampling methods. If the reviewer has better suggestions, we’ll be happy to implement and report.

Q3: Regarding the two works you raised for comparison:

- [1] https://arxiv.org/pdf/2004.01899.pdf

  - The comparison result is added in the Table of the previous response.

- [2] https://arxiv.org/pdf/2006.08573.pdf.

    -  A direct comparison cannot be made between WeakNAS and [2]. We want to clarify that [2] (deep ensemble NAS) aims to find the deep ensemble of the best architectures in the NAS search space, while our WeakNAS (Traditional NAS) focuses on finding the single best architecture in the NAS search space. It is obvious that deep ensemble NAS has a larger and more diverse search space and an inherently different objective, e.g. deep ensemble NAS would explicitly enforce diversity in architectures while traditional NAS does not, this makes it infeasible to make an apple to apple comparison between those two.

Q4: Regarding the discussion on different predictors with different encodings:
- Yes, results and discussions in our last response are promised to be included in the final version. That also won’t hamper our contribution or novelty




 Thanks author[s] for their effort, I really appreciate it. The reply addresses my questions. Please consider putting the tsne plot into the paper (remember to change top row a-b-b-b to a-b-c-d). [6] in the above table should be around 94.10 given 400 queries (see Table 2). I'm not sure if it is fair enough to other paper which is already in a good shape in the submission. Nevertheless, I will update my comments and raise the score. Thanks again for the great rebuttal and hard work.

 Sorry for late response.
Thanks for thoughtful response.
I decided to raise my rating to 6.

Here are updated review:  
### Pros
- Suggested search strategy is simpler but shown to be work better than ones of other BO-base NAS methods.

### Cons
- Paper could be misleading for making relationship of this work and BO.

### Comments
I vote to accept this paper for sharing, 1) vital intuition of search strategy in NAS and 2) its empirical superiority over other search methods, to NAS community.

This submission suggests a novel search strategy for NAS.  
While suggested search method takes the form of BO-like iterative procedure, exploration is done by acquiring only a portion of top-K architectures scored by an on-the-fly accuracy predictor.  
Such a search scheme is simple and novel in NAS literature, but achieved SOTA performance and search efficiency in NAS-Bench-101, NAS-Bench-201, and ImageNet@NASNet search space.  ### Summary of Review
I still have unresolved questions about the relationship vs BO-based NAS.
I'm willing to vote for acceptance if my questions are resolved.

### Pros
- Proposed search strategy is simple but shown to be powerful and efficient in NAS benchmarks

### Cons
- The relationship with respect to BO-based NAS is not clearly written.
- I have remaining questions which can be resolved during the rebuttal period. 

### Questions
1. According to Line117, the newly sampled architectures are accumulated. But doesn't it contradict the main "weak predictor" idea and Figure2? If the predictor is considered to be getting "weaker" because architectures will be sampled more and more only from the high accuracy predicted region, how does it differ from other BO-based methods (ex. BONAS)?

2. Because I'm not understanding the gap vs BO-based NAS in terms of its iterative procedure as in Question 1, the part that sounds novel is its acquisition strategy.
This simple acquisition strategy can be solely enough contribution if the effectiveness is justified.
However, I can't find the full justification of why the new exploration strategy is better than others.
   - Section 2.3 Line148-161 seems to be justifying all BO for me because of Question 1.
   - Section 2.3 Line162-167 is so finally what I have found for the only difference from other BO methods.
The lines (162-167) are explaining the substantial idea, but not addressing why it is superior.
   - Comparisons to LaNAS and BONAS in Section 3.4 are addressing the advantage, but there's no strict experimental support.
ex. What if only UCB acquisition is adopted while keeping every other setup (ex. predictor architecture, architecture representation)?   

3. Is there an alternative version of table 3 but with NAS-Bench-201 results? Table 4 reports the required number of sampling only if each method had reached the global optimum.

## Minor Comments
- Algorithm would be helpful for readers. (ex. how was $\tilde{f}^{k+1}$ initialized. Random or $\tilde{f}^{k}$?)
- BONAS reference should be updated to NeurIPS 2020 publication. ### Limitations
Optimal convergence is not guaranteed and well-designed architecture encoding is necessary, although those are general limitations of NAS literature.

### Societal Impact
I agree with the impact what the authors mentioned: better NAS can lead GreenAI.

 1. Thank you for the clarification. The writing can definitely be improved regarding this point.

2. > To clarify the time complexity, at each iteration, SGD will do $O(n)$ steps, If we have k iteration in total, the time complexity of the iterative process would be $kO(n)$ instead of $O(n^2)$.

   $n$ is not constant, but increases at each iteration ($n_i = M_{0} + Mi$, where $M_{0}$ is the number of initial training samples, $i$ is the iteration index, and $M$ is the number of samples added to the training set at each iteration), so you cannot simply multiply $k$ and $O(n)$ and say that the overall time complexity is $kO(n)$. For total $k$ iterations, you need $\sum_{i=1}^k n_i =  O(k^2)$ SGD steps. I realize that the notation was not clear in my initial review, but my point still stands.

3. A simple MLP is indeed faster to train compared to other models. However, that seems a lot faster than I expected. How many epochs do you train the MLP for?

4. > assume there are 5000 clusters in total and the whole space is uniformly distributed in such 5000 clusters

   This assumption does not seem realistic. If this was true, then we can find 'good' architectures with 0.865 probability in any search space by just randomly sampling 1,000 samples, and testing all of them.

 Q1: It is not clear if the results are meaningful, though. In Table 2, It shows that with the same number of queries, WeakNAS shows nearly the same performance as Local Search, a very simple baseline. This is perhaps because the NAS-Bench-101 space is very small, but given no comparison between WeakNAS and Local Search in other settings, I have no other choice to believe that WeakNAS is no better than Local Search.

- Sorry for the confusion caused by Table 2’s presentation - we will improve the clarity here. Rather than directly using local searchin NAS search space, in Table 2 we actually integrate local search as a meta sampling method in WeakNAS (e.g., on top of it, rather than independently) to compare with other sampling methods such as uniform sampling. At each iteration, we choose the Top1 prediction and use it as the initialization for local search, and after which we use local search to explore more samples, and those samples are used to train the NAS predictor in the next iteration.
To address this concern, we also include a Table of directly using local search and Semi-NAS method on NAS-Bench-101, we can see that directly applying local search and Semi-NAS in NAS is clearly inferior than WeakNAS+local search or WeakNAS + Semi-NAS.

- | NAS Predictor |  \#Queries  | Test Acc.(\%) | SD(\%) | Test Regret(\%) | Avg. Rank|
|:----------|----------:|------------:|------------:|----------:|----------:|
| WeakNAS |1000 | 94.25 | 0.04 | 0.07 |1.7 |
| WeakNAS + Local Search |1000 | 94.24 | 0.03 | 0.08 | 1.9 |
| WeakNAS + Semi-NAS | 1000 | 94.26 | 0.02 | 0.06 | 1.6 |
| Local Search |1000 | 94.15 | 0.08 | 0.17 | 7.8 |
| Semi-NAS |1000 | 93.95 | 0.11 | 0.37 | 94.0 |

Q2: It is not clear how the predictor models are trained. In the case of MLP, do the authors start SGD from random weights at each iteration? If so, it seems to me that the algorithm needs at least O(n) SGD steps at each iteration, where is the number of samples in the training set so far. This makes the entire algorithm have the time complexity of O(n2). This means that the cost of training the proxy model could quickly outweigh the cost of evaluating the samples if gets large enough.

- Yes, We start SGD from random weight. To clarify the time complexity, at each iteration, SGD will do $O(n)$ steps, If we have k iteration in total, the time complexity of the iterative process would be $kO(n)$ instead of $O(n_{2})$.

Q3: This raises the question of whether this algorithm is realistic in saving the actual search time. It would also be helpful to provide how much time is spent on training the proxy models and predicting the samples' performances vs. actually evaluating the selected architectures' performances. I suspect that running the predictor on all of the architectures is somewhat costly, in addition to the training cost.

- In the below table, we showed the comparison of two costs on NAS-Bench-101 models, and we found training and evaluation time of proxy models is negligible compared to the cost of training the neural architecture itself. 
As a result, plotting against number of samples can be seen as a good approximation of plotting against actual wall-clock time, which has been widely adapted in sample-efficient NAS papers [1][2][3][4][5].

- | Weak NAS Predictors | Config | train proxy model (s/arch) | eval proxy model (s/arch) | Arch Evaluate (s/arch) 
|:----------|:-----------|:----------:|:------------:|:------------:|
| MLP | 4 layers @1000 hidden| $8.59 ×10^{-5}$  | $3.53×10^{-5}$ | $1.76×10^{3}$ |
| Gradient Boosting Tree | 1000 Trees             | $5.70 ×10^{-4}$ | $5.54 ×10^{-7}$ | $1.76×10^{3}$ |
| Random Forrest |1000 Forrests                     | $3.20 ×10^{-3}$ | $1.77 ×10^{-4}$ | $1.76×10^{3}$ |

[1] "AlphaX: eXploring Neural Architectures with Deep Neural Networks and Monte Carlo Tree Search." AAAI 2019.

[2] "Sample-efficient neural architecture search by learning action space." TPAMI 2021.

[3] Shen Yan, Yu Zheng, Wei Ao, Xiao Zeng, Mi Zhang. Does Unsupervised Architecture Representation Learning Help Neural Architecture Search? NeurIPS 2020

[4] Shen Yan, Kaiqiang Song, Fei Liu, Mi Zhang. CATE: Computation-aware Neural Architecture Encoding with Transformers, ICML 2021

[5] Xuefei Ning, Yin Zheng, Tianchen Zhao, Yu Wang, Huazhong Yang. A Generic Graph-based Neural Architecture Encoding Scheme for Predictor-based NAS. ECCV 2020

Q4: In the footnote of page 3, the authors reveal that only 1,000 random samples' performances are predicted at each iteration, from which 100 best-performing samples are evaluated. They also state that (1) "the probability of some good examples in the total 1000 samples is very high", and (2) "progressive evolving will make this probability even larger". I am not convinced that assertion (1) is true. Moreover, since the method of sampling the candidates is entirely random, I do not see how (2) is true, either.

- Why "the probability of some good examples in the total 1000 samples is very high"? According to recent work [1] and section 2.3, they suggest NAS search space is highly structured where architectures with similar accuracy are more likely to be neighbors. As a result, our goal of WeakNAS is reduced to finding samples that reside in “any” of those good-performing clusters. For example, given a very large search space, if we want to find samples residing in the Top 0.02% clusters (i.e. assume there are 5000 clusters in total and the whole space is uniformly distributed in such 5000 clusters) and sample 100 samples each time (sample 100 times in total), then the probability that we do not find any sample within the Top 0.02% cluster is only ${((5000-1)/5000)}^{100*t}$=0.135. In other words, we have a 0.865 probability that we would find some samples in the good-performing clusters.
- Further, as stated in Response 1, local search is a sampling strategy that is orthogonal to our WeakNAS framework,we can integrate local search as a meta sampling method in WeakNAS, and it has comparable performance to other sampling methods such as uniform sampling. Besides, for Local Search, it heavily depends on initialization and can easily drop into local minimals, while WeakNAS can achieve a good trade-off between exploration and exploitation by controlling Top N and M in Table 2.
- "progressive evolving will make this probability even larger" The claim indeed needs clarification. Take the above case as an example,  the probability that we can find at least one sample from the Top 0.02% cluster in $t$ iterations is $1-{((5000-1)/5000)}^{100*t}$, which increases along with $t$.


 To be clear, since an early version of this draft was published in the arXiv on June, I know the identity of the author[s], but I have no idea that author[s] seem already know my identity and the reply is specifically defense some work related to me even though most of works I have mentioned are not related to me. Some clarifications are needed.

I am not mean or conflict of interest to this work. As a matter of fact, there are several important work are missing. I like the results, but I don't understand why it works so well. There are two key questions: 1). Why do you state that the proposed method is a principled way to guide the search trajectory for good and bad regions? Could author[s] show some illustrations/figures for the proof of concept from Figure 2 in the paper. 2). The proposed method is more like a learning approach. As a fair comparison, I would suggest using the same predictors (e.g. GCN, MLP) for an apple-to-apple comparison with BONAS and LaMCTS. From the reply to reviewer Lp96 I can rarely see an advantage. It would be also good if author[s] do the same for latent action MCTS and make a comparison instead of directly reporting their number since you emphasize these works throughout the paper.

These are my suggestions. I apologize to author[s] if they think I gave unreasonable feedback. I would like to update my previous comments.

 Q1: The experimental results in Table 6 (81.3% Top-1 accuracy on ImageNet) are quite impressing. However, some implementation details are missing, including the details of the search space, the details of retraining strategy and so on. These details have a huge influence on the performance.

- We adopt PyTorch and image models library (timm)[1] to implement our models.  We use Random Erase and RandAug, a drop out rate of 0.3 and a drop path rate of 0.0, we also use exponential moving average (EMA) with a decay rate of 0.9999. During training and evaluation, we set the image size to be 236x236. We train for 300 epochs with warm-up of 3 epochs, we use a batch size of 1024 and RMSprop as the optimizer. We use a cosine decay learning rate scheduler with a starting learning rate of 1e-02 and a terminal learning rate of 1e-05. Are those details wil be included in final version for clarity.

[1] https://github.com/rwightman/pytorch-image-models

Q2: Will the authors publish the code and pretrained models for better reproduction?
- We attach an anonymized version of the code in this anonymous link below. We promise to release the code on GitHUb for 100% reproducibility after acceptance.

https://tinyurl.com/3xm2fzzb

Q3: In Table 6, it would be better to report the search cost for clearer comparisons
- The search cost of WeakNAS in Table 6 is 2.5 GPU hrs at 1000 queries, while in comparison OFA [2] need 40GPU hrs at 16000 queries. We will add the the search cost comparsion in the final version.

[2] Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, Song Han, Once-for-All: Train One Network and Specialize it for Efficient Deployment, ICLR 2020


 Q1: According to Line117, the newly sampled architectures are accumulated. But doesn't it contradict the main "weak predictor" idea and Figure2?
- Sorry for the confusion. We will fix the confusion in Figure2. Note that putting previously sampled architectures into the training set does not contradict our “Weak” predictor idea and incurs no extra overhead. In fact, our weak predictor learns to connect a path towards the high-performance sub-space. Empirically, we found discarding the history architectures from the training set causes search instability, since the NAS predictor would easily collapse given each iteration we only sample 10 new training samples. Accumulating historical architectures here acts like a “momentum” operation that smoothens the optimization trajectory, but still it allows new predictors to increasingly focus on the high-accuracy regions due to the sampling trend. 

Q2: If the predictor is considered to be getting "weaker" because architectures will be sampled more and more only from the high accuracy predicted region, how does it differ from other BO-based methods (ex. BONAS)? I can't find the full justification of why the new exploration strategy is better than others.

- Great question. To summarize in brief: (1) our method shares high-level similarity to BO, but does not take a precise BO form and avoids the explicit uncertainty modeling. Note that explicit uncertainty modeling is not only expensive, but also bears its own variations/inaccuracy; (2) we use a different and much simpler/implicit exploration mechanism, motivated by the specially structured NAS space. This new simpler method is much cheaper and also stable; (3) thanks to this problem-specific insight, we actually benefit from our simplified adaptive balancing of exploration and exploitation during search, and our method outperforms BONAS.

- In more details, our method can be regarded as a purposely simplified variant of Bayesian Optimization (BO): the weak predictors take a similar role to typical acquisition functions, but ours refer to no explicit uncertainty-based modeling such as Gaussian Process (which are often inefficient to scale up). Section 2.3 has analyzed the connection and differences. Our current approach does not rely on any explicit uncertainty-based modeling such as Gaussian Process, mainly motivated by reducing the search cost (uncertainty modeling is heavier). However, our approach is not free of exploration – just that we have a different, simpler build-in exploration mechanism.

- We find our simplified exploration in this current setting works well for this application (in fact, better than BONAS), presumably due to the structured NAS search space. Overall, NAS space has a highly non-uniform distribution of architectures, whose density is also heterogeneous (although a precise definition of architecture distribution or density hinges on the concrete definition of distance). That is because the candidate architectures were created from varying operators (often causing similar performers), to varying width or depth (often causing bigger performance gaps). Therefore, recent work [1] suggests the architectures display highly clustered distribution (as can be visualized by histogram on NAS-Bench-101/201 in Figure 1/2 of Supplementary); and the best performers are often gathered close [1] or follow some specific patterns [2]. This also leads to our work’s underlying assumption: we can progressively connect the line from the initialization to the finest subspace, where the best architecture resides.

- Based on the insight of the non-uniform or heterogeneous search space, the demand for the exploitation-exploration trade-off is also heterogeneous across different search stages. At the beginning stage, more exploration will be needed to identify the promising direction towards the finer space. That is naturally achieved because the weak predictor at the initial stage only roughly fits the whole space (in other words, it CANNOT fit too well by then, and we intentionally make so). As we keep zooming in to the good-performing subspace, the weak predictor is also gradually refined and better fit, therefore making stronger exploitation. Eventually, our weak NAS predictor is made to adaptively balance the exploration and exploitation throughout the search process, which echoes the findings by many prior works such as [3][4][5].

[1] Exploring the Loss Landscape in Neural Architecture Search, Colin White, Sam Nolen, Yash Savani

[2] Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, Piotr Dollár, Designing Network Design Spaces

[3] "AlphaX: eXploring Neural Architectures with Deep Neural Networks and Monte Carlo Tree Search." AAAI 2019.

[4] "Sample-efficient neural architecture search by learning action space." TPAMI 2021.

[5] "Revisiting Neural Architecture Search." arXiv 2020.

Q3: Comparisons to LaNAS and BONAS in Section 3.4 are addressing the advantage, but there's no strict experimental support. ex. What if only UCB acquisition is adopted while keeping every other setup (ex. predictor architecture, architecture representation)?
In UCB there are two terms, one is the reward of the sample (accuracy), the other is the uncertainty of the sample, since we do not use Bayesian sigmoid regression (BSR) model to calculate mean and variance (uncertainty) as in BONAS, there is no uncertainty measure in our WeakNAS predictor model, as a result, UCB cannot directly apply to WeakNAS.

- We did an ablation study where WeakNAS uses the same architecture predictor (GCN) and architecture representation as BONAS, the only difference is exploration-exploitation balance, where BONAS use UCB and ours use TopK uniform sampling. We include the comparison on NAS-Bench-101 in the table below and found WeakNAS still performs notably better.

- | Method | Predictor |  \#Queries  | Test Acc.(\%) | SD(\%) | Test Regret(\%) | Avg. Rank |
|:----------|:-----------:|----------:|------------:|------------:|----------:|---------:|
| BONAS | GCN   |1000 | 94.25 | - | 0.07 | 1.7 |
| **WeakNAS** | **GCN** |**1000** | **94.28** | **0.06** | **0.04** | **1.4** |


 Q1: Line 294: “Table 2 shows that the combination of our method with Semi-NAS can further boost the performance of WeakNAS”. I never found this result. I suspect there might be some typo, but this combination makes a critical claim and the authors must clarify that.

- The combination of WeakNAS with Semi-NAS result is shown in Table. 2 bottom row. Here Semi-NAS is used as a meta sampling method, where at each iteration we further train a Semi-NAS predictor with pseudo labeling strategy to augment the training set of our weak predictors. We show that the combination of our method with Semi-NAS can further boost the performance of WeakNAS. We will highlight this result to be more visible in revised version.

Q2: The authors in general confused the usage of “less” and “fewer”, e.g. “less samples” should be “fewer samples”. Similar happens many times in the paper.

- Thanks for pointing out those typos. We will fix them in the final version

This paper describes a predictor-based NAS scheme called weak NAS. Rather than finding a sophisticated predictor to fit the whole architecture space, it is a better idea to have a series of weaker predictors, that progressively narrows down the search and move towards the best sub-space. The empirical results are very strong and comprehensive, outperforming some very recent strong SOTAs such as LaNAS (TPAMI 2021).   The main insight of this paper is that, due to the special structure of NAS search space, its search strategy can be accordingly simplified. Specifically, a line of progressively evolving weak predictors can be found to connect a path to the best architecture. Hence, WeakNAS requires only to estimate a series of weak predictors to fit small local spaces and to progressively move the search space towards the subspace where good architecture resides.

This paper used a principled re-formulation of predictor-based NAS to motivate their method. The bi-level form leads to a coordinate descent-like heuristic algorithm, that is further interpreted as the alternating iteration of sampling and local fitting. The framework is general and can be used with various predictors or architecture encodings. Section 2.3 also discussed its relationship to BO and shows WeakNAS to be an efficient BO variant with a simple implicit acquisition function. It can work thanks to the special structure-property of NAS space. 

As an empirical paper, this paper benefits from the high completeness of its empirical comparison. The comparison is  comprehensive and covers the latest predictor-based and other NAS methods. Particularly, Section 3.4 presents a very helpful discussion on the similarities and differences, between WeakNAS and Semi-NAS, LaNAS, and BONAS. It makes WeakNAS’s contribution clear to comprehend. 

The results surpasses the recent SOTAs. In general, WeakNAS can find better models with fewer queries; it also has a significant speedup on the search time, compared to SOTAs. In the open-domain NASNet space, WeakNAS is able to reduce 0.6% top-1 error from the latest LaNAS [21], using much fewer GPU hours and at similar parameter counts. When it comes to the more challenging MobileNet open space, WeakNAS also beats LaNAS by 0.5% top-1 error, using even fewer FLOPs.

The paper writing is also good.  Fig 2 and Fig 3 are  informative and extensive ablation studies are conducted. 

Questions:
-	Line 294: “Table 2 shows that the combination of our method with 295 Semi-NAS can further boost the performance of WeakNAS”. I never found this result. I suspect there might be some typo, but this combination makes a critical claim and the authors must clarify that.
-	The authors in general confused the usage of “less” and “fewer”, e.g. “less samples” should be “fewer samples”. Similar happens many times in the paper.
 The authors have sufficiently discussed this aspect.

This paper presents a method that trains neural architecture predictors by focusing on the high-performing sub-space. Specifically, the authors start from a weak predictor and sample a set of promising architectures from the previous predictor to learn a better predictor. Extensive experiments on NAS-Bench-101, NAS-Bench-201, NASNet and MobileNet search spaces demonstrate the effectiveness of the proposed method. However, some important details are missing. My details comments are as follows.  Positive points:
1. The authors point out that it is not necessary to fit all architectures in the whole search space while training predictors, which is very interesting.
2. The authors proposed an iterative training scheme to learn a set of gradually improved predictors.
3. The experimental results show the superiority of the proposed method over the existing ones.

Negative points:
1. The experimental results in Table 6 (81.3% Top-1 accuracy on ImageNet) are quite impressing. However, some implementation details are missing, including the details of the search space, the details of retraining strategy and so on. These details have a huge influence on the performance. 
2. Will the authors publish the code and pretrained models for better reproduction?
3. In Table 6, it would be better to report the search cost for clearer comparisons.
 The authors have addressed the limitations and potential negative societal impact of their work.

Meta-review: This paper proposes a Neural Architecture Search method which utilizes a series of weak performance predictors, that are trained to focus on the samples that are more likely to perform better by the previous weak predictor. This is done by alternating between the learning of the weak predictor based on the sampled architectures, and the sampling with this weak predictor. The authors validate their method with a proof-of-concept experiment, as well as experimental validation on the NAS-Bench-101 and NAS-Bench-201 benchmarks, against relevant predictor-based and BO-based NAS baselines. The results show that the proposed alternating scheme with the weak predictors are significantly more effective and sample-efficient.

The reviewers in general found the high-level idea of using a series of weak predictors to narrow down the search space, and the proposed alternating search scheme, to be novel and reasonable. Also, they find the experimental validation convincing, and the proposed scheme to be highly effective and efficient. 

However, at the same time, the reviewers had the following concerns.
- The paper lacks many details, such as the pseudo-code of the algorithm, complexity analysis, implementation details, and the details of the retraining strategy.
- The proposed method is not validated against some of the SOTA methods for NAS. 
- The novelty of the method over existing iterative search space refining strategies, or NAS based on Bayesian optimization, has not been sufficiently discussed.

During the discussion period, the authors provided the code, discussion on the computational complexity, the results of the SOTA methods, and the discussion on the novelty of the proposed method over existing methods. This cleared away most of the concerns from the reviewers, resulting in a unanimous agreement to accept the paper. 

However, I have additional concerns, which I want the authors to clearly address in the revision. Most come from the lack of formal definitions and analysis. 

- There is no formal definition of “weak” and “strong” performance predictors. This should be formally defined in order to clarify what are weak and what are strong predictors. 
- There is no visualization of the actual search space while running the proposed algorithm, that is similar to Figure 2, which makes it difficult to see what happens there during the iterative search process.
- There is no analysis of the failure case - in what cases does the weak predictor miss out high-performing architectures?
- There is no theoretical or empirical analysis on how many steps, or weak predictors, are necessary to find an optimal architecture from a pool of $N$ architectures, which seems crucial in the use of the proposed algorithm.

I recommend accepting the paper since the proposed idea is interesting and seems to work well in practice, and all reviewers are unanimously positive toward it. Yet, I believe that a lot of effort should be put into analyzing how the proposed method works. The paper is too informal and looks half-baked at its current state, and I strongly recommend the authors to incorporate the discussions with the reviewers, as well as the answer to my concerns above.

Decision: accept