Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The paper describes a new neural architecture search method based on monte-carlo tree search that dynamically adapts the action space.
The methods consists of two stages. In the first stage, the learning stage, the action space is divided into good and bad regions based on a tree structure. In the seconds stage, new data is generated by sampling new architecture with monte-carlo tree search. Those two stages are iterated with new incoming data.

The paper proposes an interesting approach which achieves competitive results compared to state-of-the-art methods.
In general the paper is well written and easy to follow. However,  I haven't fully understood  how the model space is divided at different nodes. What exactly is the splitting criterium? Are the splits axis aligned?

Further comments:

- Figure 4 a and b seemed to be flipped?

- Figure 5 top row would be easier to parse if the x-axis is on a log scale.

- Could you also include other Bayesian optimization methods, such as SMAC or TPE, which should competitive performance on NASBench101 and do not suffer from a cubic scaling



post rebuttal
------------------


I thank the authors for performing additional experiments and clarifying my questions. First of all, I would like to stress that I still think the approach seems promising.
However, I am not entirely convinced by the empirical results.  While the proposed method converges faster to the global optimum than other methods, it is only able to improve by a little epsilon in terms of validation accuracy,  which could indicate overfitting on these tabular benchmarks. Furthermore, in the beginning Bayesian optimisation based methods consistently perform better which might be in practice more relevant for these highly expensive optimisation problems.  Because of that, I keep my initial score. 
review_title: Official Blind Review #4

final_score: rating: 3: Weak Reject
review_content: This paper introduces a Neural Architecture Search algorithm that attempts to solve the problems of the existing NAS only utilizing manually designed action space (not related to the performance). The paper proposes LaNAS which is based on an MCTS algorithm to partition the search space into tree nodes by the performance in the tree structure. The performance of the method is shown in the NASBench-101 dataset and Cifar-10 open domain search.

I lean to reject this paper because (1) the motivation is not well justified by the experiments, (2) the comparison on NASBench-101 is not convincing, (3) some important explanation of the method is missing.

Main arguments
The main contribution of the paper is using a learned action space in MCTS rather than a manually designed MCTS algorithm for NAS. However, as far as I know, the MCTS approach for the NAS problem is not a standard solution for NAS (which is not proved to be practically useful in other people’s papers) which diminishes the contribution of the improvements of MCTS in NAS.

Lack of the main comparison. For the motivation of the proposed method, the authors mention the drawbacks of other NAS methods used fixed action space in their RL or MCTS module. However, the authors only show that using a learned action space in MCTS is better than a fixed MCTS algorithm in the experiments. What about using a learned action space in the RL module such as PPO in NAS comparing to the fixed one?

The comparison of NASBench-101 is not convincing. The authors compared with the BO method and claimed that the method is 16.5x more efficient than BO. However, the recently released paper “BANANAS: Bayesian Optimization with Neural Networks for Neural Architecture Search” said that their method is 3.8x more efficient than the one proposed here, which is quite confusing.

Some important explanation of the methods is missing. Throughout the paper, the method to sample from a leaf node is only mentioned in 3.3. However, the corresponding sampling method is unclear. The paper only mentions that MCMC has been adopted to sample from the target subspace. In my opinion, it is not so trivial to use MCMC here and should be elaborated in more detail. Otherwise, people cannot use it.

As given in figure 3, if c is set to be a very small number, the search is similar to simply using a series of predictors and always samples the models with better-predicted accuracies. Is MCTS really useful here? To show the effectiveness of MCTS, it is recommended to experiment on different values of c.

Results given in the upper row of Fig.5 is not useful. In practice, it is already painful to sample about 1000 models and train them for the Cifar-10 dataset. It is more useful to see how this method behaves in the range of (0,1000). However, in Fig.5, different methods are all overlapping in this range and hard to tell whether this method is better than other methods

review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: In this paper, the authors propose a strategy for neural architecture search. The basic idea is effectively to model the accuracy of architectures in the search space, and use this model to select subsequent architectures with a MCTS-like procedure.

Overall, my primary concern with the paper is a lack of context in the larger field of model based optimization. To be clear, the authors' method is clearly an instantiation of model based optimization. However, much of the paper is arguably written as though this needs to be invented from first principles. For example, much of section 3 is arguably a specific instantiation of the basic model based optimization loop, and much of the discussion on global versus sequential search exists in this literature as well.

I believe the paper would be greatly improved by (1) providing this context, and (2) explaining the authors' approach within this context. Much of the discussion contrasting arbitrary action spaces with handcrafted ones are somewhat lost in the actual experimental setup: For example, the ConvNet-60K and LSTM-10K datasets have well specified parameter spaces. Beyond this, I'd like the authors to contrast the surrogate tree model used with simple CART trees: the fitting procedure in section 3.1 is quite similar to standard methods used to fit regression trees. 

Additionally in the same context, a significant amount of related work is missing. The use of tree models for model based optimization have been considered before (e.g., SMAC), although the MCTS acquisition with a single tree surrogate is novel as far as I am aware. Other recent methods in model based optimization exist, including those with specific application to architecture search that explicitly outperform the basic Bayesian optimization algorithm (e.g. NASBot), and I'm therefore not sure if the comparison to the most basic instantiation of BayesOpt is appropriate. 

Beyond this, the experimental performance of the authors' method seems quite good on the tasks considered, and at least a substantial subset of the baselines considered are recent. I would therefore not be upset to see the paper published and would be willing to increase my score; however, I believe the framing of the paper needs substantial imrpovement.
review_title: Official Blind Review #1

Meta-review: This paper proposes an MCTS method for neural architecture search (NAS). Evaluations on NAS-Bench-101 and other datasets are promising. Unfortunately, no code is provided, which is very important in NAS to overcome the reproducibility crisis.

Discussion:
The authors were able to answer several questions of the reviewers. I also do not share the concern of AnonReviewer2 that MCTS hasn't been used for NAS before; in contrast, this appears to be a point in favor of the paper's novelty. However, the authors' reply concerning Bayesian optimization and the optimization of its acquisition function is strange: using the ConvNet-60K dataset with 1364 networks, it does not appear to make sense to use only 1% or even only 0.01% of the dataset size as a budget for optimizing the acquisition function. The reviewers stuck to their rating of 6,3,3.

Overall, I therefore recommend rejection. 
Decision: Reject