Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: 
The paper introduces a new encoding for cell structures to improve efficiency of neural architecture search methods.
As a second contribution the paper proposes Bayesian optimization with an ensemble of neural networks as probabilistic model for the objective function.
Both contributions combined show superior performance on the Nasbench101 benchmark as well as competitive performance on the DARTS search space.

While the paper identifies an important problem - encoding of architectures - I do not think the paper is ready for acceptance.

About the encodings:

First, using different encodings to enable better architecture search has been investigated by others before.
For example, Ying et al. also provided different encodings of the adjacency matrix for Nasbench101 besides the used binary encoding and it seems that different methods work well with different encodings.
Also in the work by Kandasamy et al. they presented an encoding for architecture, such that Bayesian optimization can be applied.

Second, the encoding described in the paper lacks some intuition.

 - How does enumerating all paths and encoding them as a binary vector convey more information than just using the adjacency matrix?

 - Lead isomorphic graphs, which for example occur in Nasbench101, to the same encoding?

 - It seems somewhat counter intuitive to use a large binary vector (more than 18000 dimensional vector for the DARTS space) as encoding for Bayesian optimization which is known to struggle with high dimensional input spaces.


About the Bayesian optimization strategy:

The proposed probabilistic model for Bayesian optimization seems straight forward and simple. Also here, previous work (Snoek et al, Springenberg et al., Perrone et al)  already proposed to use neural networks and ,in order to be more convincing, the paper should include a comparison to these methods.
Furthermore, the paper should clarify how the diversity in the neural network ensemble is enforced. Are the neural networks trained with different random initialization? How does it compare to the method proposed by Lakshminarayanan et al. which showed better performance for neural network ensembles based only on different random initialization?


Minor comments:


- In the Nasbench101 paper other Bayesian optimization strategies (e.g SMAC, BOHB, TPE) showed strong performance. The results would be more convincing if these methods are included in the comparison.

- Following the empirical protocol by Ying et al. the results would be easier to parse if the Figure 3  could report the log test regret on the y-axis. I am also missing a figure that shows the robustness of the method across independent runs.

- How are invalid architectures in the Nasbench101 (e.g architectures that violate the max edge constraint) treated in the experiments?

- I think the paper is missing the following references:

Simple and scalable predictive uncertainty estimation using deep ensembles
B Lakshminarayanan, A Pritzel, C Blundell
Advances in Neural Information Processing Systems, 6393-6395

Scalable hyperparameter transfer learning
V Perrone, R Jenatton, M Seeger, C Archambeau
Advances in Neural Information Processing Systems, 6845-6855



post rebuttal
------------------


I thank the authors for answering my questions regarding the path encoding and taking the time to improve the empirical evaluation of the paper. While I think that the paper has improved, I am afraid that the contributions of the paper are still not strong enough to reach the bar of acceptance because of the following reasons: 

- The path encoding is an interesting approach and seems to improve upon just using the adjacency matrix directly, it doesn't scale and, hence, it remain somewhat unclear how valuable it is in practice. 

- More importantly, I am not convinced that the proposed neural network model represents a sufficient contribution. After some discussion with the authors, they agree that existing BO methods based on neural networks could also be applied to this setting and even say that they may perform well with the path encoding. However, they are not include them in the comparison and only promise to add them for the final evaluation.  I am concerned that if it turns out that other methods perform as well or even better, it would dramatically lower the contribution of the paper.
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This paper develop a path-based encoding scheme to featurize the neural architectures that are used to train the neural network model, and design a NAS algorithm that performs Bayesian optimization using a neural network model. The experiments show the priority of the proposed method.

In general, this paper is easy to follow, but the contribution is limited. The author did not give a clear explanation of why does this method work. There are several problems that exist in the paper:

1.	The paper introduced a path-based encoding scheme, which seems have nothing different from enumerating all possible paths. Any additional operations should be clarified in the paper.
2.	The method retains new architectures with high UCB value. However, the author did not prove that a higher UCB value leads to a better architecture. Eq.(1) trained several different networks to predict the accuracy. However, when using early stop stragegy, the intermediate accuracy is not convincing, and the new architecture selected based on UCB may not perform well when training with full epochs. If early stop is not used, there is no need  to predict the accuracy with different networks.
3.	In my opinion, Algorithm 1 is a simplified traditional Evolutionary Algorithm, which only have mutation operation and do not have selection and crossover operation, and has limited novelty.


review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: The paper considers the neural architecture search using Bayesian optimisation. The paper first propose a path-based encoding scheme to featurise the neural architectures. A path encoding of a cell is created by enumerating all possible paths from the input node to the output node. Then, they propose to train 5 neural networks and ensemble these networks to get the prediction (including the predictive value and uncertainty). The paper optimises the acquisition function via a mutation procedure, where we randomly mutate the best architectures that we have trained so far, many times, and then select the architecture from this set which minimizes the acquisition function.

While the writing is readable and the experiments seem promising, the reviewer thinks that the novelty and contribution of the paper are limited. Particularly, using 5 neural networks for estimating the uncertainty is less convincing. This is because it would have been better if we can train a Bayesian neural network to provide the uncertainty quantification directly.


Minor:
Page 5: “randomly randomly”

review_title: Official Blind Review #3

Meta-review: This paper uses Bayesian optimization with neural networks for neural architecture search. 
One of the contributions is a path-based encoding that enumerates every possible path through a cell search space. This encoding is shown to be surprisingly powerful, but it will not scale to large cell-based search spaces or non-cell-based search spaces. The availability of code, as well as the careful attention to reproducibility is much appreciated and a factor in favor of the paper.

In the discussion, it surfaced that a comparison to existing Bayesian optimization approaches using neural networks would have been possible, while the authors initially did not think that this would be the case. The authors promised to include these comparisons in the final version, but, as was also discussed in the private discussion between reviewers and AC, this is problematic since it is not clear what these results will show. Therefore, the one reviewer who was debating about increasing their score did in the end not do so (but would be inclined to accept a future version with a clean and thorough comparison to baselines). 

All reviewers stuck with their score of "weak reject", leaning to borderline. I read the paper myself and concur with this judgement. I recommend rejection of the current version, with an encouragement to submit to another venue after including a comparison to BO methods based on neural networks.
Decision: Reject