Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: --Summary--
The authors present an algorithm BOGCN-NAS which combines bayesian optimization and GCNs for searching over NN architectures. The authors emphasize that this method can be used for multi-objective optimization and run experiments over NAS-Bench, LSTM-12K and ResNet models. 

--Method--

Methodologically, the contribution is somewhat weak. The main technical contribution is to use a GCN to get a global representation of a graph, which can then be used for downstream regression tasks such as predicting accuracy. It’s not clear how much the GCN generalizes in being able to encode arbitrarily large architectures. The two main examples offered are NAS-Bench and LSTM-12K focus on optimizing cell architectures which contain a handful of nodes e.g. (5 in the case of NAS-101).

Graph embeddings:
The authors have not considered other graph embeddings to use in their Bayesian regression setup. E.g. see https://arxiv.org/pdf/1903.11835.pdf for a list.

Bayes-opt:
In Algo1, step 5. The authors randomly sample a number of architectures in order to calculate EI scores on them. For large discrete combinatorial search spaces, this approach will not scale.

Multi-objective optimization:
It’s not clear why GCNs or BO is required for this. Any predictor that generates multiple metrics could substitute in order to create a pareto-curve. Even multi-objective RL based approaches could suffice. Thus multi-objective opt only seems like a minor/tangential contribution.

--Experiments--
The main claim of the paper is that this approach works well  for the multi-objective case. However, the results only look at two objectives #params vs accuracy. There’s a pretty strong correlation between the two. It’s unclear how the method generalizes when objectives are not correlated. The authors need to thoroughly demonstrate other objectives/find suitable benchmarks for the same as clearly NAS-101 will not suffice.

Other concerns:
 - Table 1 has correlations using 1000 training architectures. Why 1000? Why not 50 (that’s how sec 4.2 is initialized). Also, the correlation results are less impressive in Figure 9.
 - Table 1 lists the number of params that the predictor uses. Why is this important? How about comparing with a linear regressor?
 - The results in Sec 4.3 are using random as the only baseline. This is a pretty weak baseline.
 - In sec 4.4, the authors pick models M1, M2 and M3 as candidate examples.. How were these chosen ?
 - Sec 4.5 transfer learning results are pretty weak. Transfer across datasets is much more interesting e.g. between ImageNet and Cifar-10.

Overall, this paper has some interesting results, which show that GCNs can be useful models to encode graph structured inputs. However, the methodological and experimental results can definitely be strengthened. The authors may consider the following:
Address how GCNs can model and scale to general architecture spaces than a small number of nodes in a cell.
Address how to sample better over combinatorial search spaces than random in the inner loop of BO.
Strengthen MO-opt results. Use better baselines than random and different objectives than accuracy vs #params.

review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This paper proposed BOGCN-NAS that encodes current architecture with Graph convolutional network (GCN) and uses the feature extracted from GCN as the input to perform a Bayesian regression (predicting bias and variance, See Eqn. 5-6). They use Bayesian Optimization to pick the most promising next model with Expected Improvement, train it and take its resulting accuracy/latency as an additional training sample, and repeat. 

They tested the framework on both single-objective and multi-objective tasks.  On the single-objective (accuracy task). They tested it on NasBench and LSTM-12K, two NAS datasets with pre-trained models and their performance. They obtained very good performance on both, beating LaNAS (previous SoTA) by 7.8x higher sample efficiency. On multiple-objective, they show higher efficiency in finding Pareto frontier models, compared to random search. 

One main question I have is whether the next model is chosen given the current prediction model? For NasBench, did you run your predictor for all (420k minus explored) models and pick the one that maximizes Expected Improvement? Note that LaNAS is more efficient in that manner by sampling directly on polytopes formed by linear constraints. If so, how do you pick the next candidate models in open domain setting?

It looks like Eqn. 9 biases the training heavily towards high accuracy region, which is a hack. Although in the Appendix (Fig. 7) the authors have already perform some analysis on the effect of different weight terms, I wonder whether there is a more problem-independent way of doing it. The MCTS in LaNAS is one way that automatically focuses the search on the important regions. Currently, the proposed approach might limit the usability of the proposed method to other situations when accuracy is no longer that important. 

The performance is really impressive in the NasBench and LSTM dataset. The paper mentioned that “BO can really predict the precious model to explore next” but didn’t provide an examples in the paper. I would whether the author could visualize it in the next revision, which would be very cool. 

Do you have open domain search research in single-objective search? 

Why not use NasNet architecture for a fair comparison with other NAS papers?

In Appendix, Fig. 6 shows that even without GCN and BO, a single MLP already achieves global optimal solution in LSTM-12K dataset with ~850 samples, already beating all the previous methods (Random, Reg Evolution, MCTS and LaNAS). If that’s the case, I wonder how much roles the proposed methods (BO + GCN) play during search? Also what do you mean by “without BO”? Do you only predict the mean and assume all variance is constant? 

=====Post Rebuttal======
I have read other reviewers' comments and the rebuttal. 

One of the main problems in this paper is an unfair comparison against LaNAS. LaNAS only uses a single sample at each leaf, while they sample 100% to 0.1% of the models, evaluate them with the current BO model and find the best. For NasBench-101 with 420K models, even sampling 0.1% each time means ~400 samples and the performance (from the rebuttal) seems to degrade substantially from 100% case (1464.4->4004.4). This means that almost 3x more samples are needed, compared to what they claimed. 

I agree with the authors that calling BO function is super fast so maybe this is fine. However, on the open domain experiments, their performance is also not better than LaNAS+c/o, which they didn't list in the rebuttal. I listed it here:

Model                      	Params       	Top-1 err	No. of samples truly evaluated
-----------------------------------------------------------------------------------------------------------------
BOGCN+cutout (V1)	   3.1M 	          2.74	                      200
BOGCN+cutout (V2)	   3.5M 	          2.61	                      400
LaNAS+c/o                       3.2M              2.53±0.05                    803

Overall this paper is on the borderline. I don't mind if the paper gets rejected. For now I lower the score to 3. 
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: This paper provide a NAS algorithm using Bayesian Optimization with Graph Convolutional Network predictor. The method apply GCN as a surrogate model to adaptively discover and incorporate nodes structure to approximate the performance of the architecture. The method further considers an efficient multi-objective search which can be flexibly injected into any sample-based NAS pipelines to efficiently find the best speed/accuracy trade-off.

The paper is well-written. The experiments are abundant. However, the paper has following drawbacks that need to be further concerned:

1.	In my opinion, the key point of the paper has nothing to do with GCN or multi-objective. The important part is to use BO and EI to sample new architecture. However, no theoretical proof is provided to guarantee that the performance is getting better during while loop in Algorithm 1.
2.	Eq.(9) focuses more on models with higher accuracy. However, those models with bad performance will be predicted inaccurately and may have a higher score than good models. For model with ground-truth near 0, arbitrary predicted score results in the same loss. Eq.(9) seems cannot prevent this situation from happening.
3.	Table 1 compare different architectures with GCN. However, the LSTM is the worst architecture used among the 3 different architectures in the original paper (Alpha-X), which makes the comparison unfair.
4.	Table 2 shows the number of architectures trained. However, the proposed method need to update GCN multiple times during searching, which makes the comparison unfair.
5.	Table 2 shows the number of training models before finding the best model. It is meaningless when used in reality, which often contains more than 10^10 different architectures and the best architecture is unknown. In my opinion, the performance of the top1 architecture predicted by the proposed method is much important.
6.	Algorithm 1 uses Pareto front, which does not exist when doing experiments on single-objective search. More details should be clarified.
review_title: Official Blind Review #2

Meta-review: This paper proposes to use Graph Convolutional Networks (GCNs) in Bayesian optimization for neural architecture search. While the paper title includes multi-objective, this component appears to only be a posthoc evaluation of the Pareto front of networks evaluated using a single-objective search -- this could be performed for any method that evaluates more than one network. Performance on NAS-Bench-101 appears to be very good. 

In the private discussion of reviewers and AC, several issues were raised, including whether the approach is compared fairly to LaNAS and whether the GCN will predict well for large search spaces. Also, unfortunately, no code is provided, making it unclear whether the work is reproducible. The reviewers unanimously agreed on a weak rejection score.

I concur with this assessment and therefore recommend rejection.

Decision: Reject