Differentiable Graph Optimization for Neural Architecture Search
Authors: Chengyue Huang
Lingfei Wu
Yadong Ding
Siliang Tang
Fangli Xu
Chang Zong
Chilie Tan
Yueting Zhuang
Venue:  (2021)
Keywords: Neural Architecture Search
Graph Structure Learning

Abstract: In this paper, we propose Graph Optimized Neural Architecture Learning (GOAL), a novel gradient-based method for Neural Architecture Search (NAS), to find better architectures with fewer evaluated samples. Popular NAS methods usually employ black-box optimization based approaches like reinforcement learning, evolution algorithm or Bayesian optimization, which may be inefficient when having huge combinatorial NAS search spaces. In contrast, we aim to explicitly model the NAS search space as graphs, and then perform gradient-based optimization to learn graph structure with efficient exploitation. To this end, we learn a differentiable graph neural network as a surrogate model to rank candidate architectures, which enable us to obtain gradient w.r.t the input architectures. To cope with the difficulty in gradient-based optimization on the discrete graph structures, we propose to leverage proximal gradient descent to find potentially better architectures.
Our empirical results show that GOAL outperforms mainstream black-box methods on existing NAS benchmarks in terms of search efficiency.

Under review as a conference paper at ICLR 2021
DIFFERENTIABLE GRAPH OPTIMIZATION FOR NEU- RAL ARCHITECTURE SEARCH
Anonymous authors Paper under double-blind review
ABSTRACT
In this paper. we propose Graph Optimized Neural Architecture Learning (GOAL), novel gradient-based method for Neural Architecture Search (NAS). to find better architectures with fewer evaluated samples. Popular NAS meth- ods usually employ black-box optimization based approaches like reinforcement learning, evolution algorithm or Bayesian optimization, which may be inefficient when having huge combinatorial NAS search spaces. In contrast, we aim to ex- plicitly model the NAS search space as graphs, and then perform gradient- brased optimization to learn graph structure with efficient exploitation. To this end, we learn a differentiable graph neural network as surrogate model to rank candidate architectures, which enable us to obtain gradient w.r. the input architectures. To cope with the difficulty in gradient-based baaeed optimization the discrete graph struc- tures, we propose to leverage proximal gradient descent to find potentially better architectures. Our empirical results show that GOAL outperforms mainstream black-box methods on existing NAS benchmarks in terms of search efficiency.
INTRODUCTION
Neural Architecture Search (NAS) methods achieve great success and outperform hand-crafted mod- els in many deep learning applications, such image recognition, object detection and natural lan- guage processing (Zoph et al.) 2017 Liu et al. 2019 Ghiasi et al., 2019 Chen et al. 2020). Due to the expensive cost of laaiiingevaluaaing a neural architecture, the key challenge of NAS is to ex- plore possible good candidates effectively. To cope with this challenge, various methods have been proposed, such as reinforcement learning (RI.), evolution algorithm (EA), Bayesian optimization (BO) and weight-sharing strategy (WS), to perform efficient search (Zoph & Le 2016 Real et al.) 2019 Hutter et al. 2 Liu et al. 2019 Guo et al. 2019).
While the weight-sharing strategy improves overall efficiency by reusing trained weights to reduce the total training cost, zeroth-order algorithms like RL, EA and BO employ black-box optimization, with the goal of finding optimal solutions with fewer samples. However, the search space of NAS is exponentially growing with the increasing number of choices. As a result, such huge combinatorial search spaces lead to insufficient exploitation of black-box learning framework (Luo et al. 2018).
Another line of research has been focused on formulating the NAS search space as graph structures, typically directed acyclic graphs (DAGs), and then the search target is cast as choosing an optimal
combination of the nodes and edges in the graph structure (Pham et al. 2018 Liu et al., 2019 Xie al. 2019). However, existing methods tend to perform the optimization in the indirect manner ing box optimization. In contrast, we aim to explicitly model the search space as graphs and optimize graph structures directly. We thus propose Graph Optimized Neural Architecture Learning (GOAL), a novel NAS approach combined with graph learning for efficient exploitation, as briefly shown in Fig T Unlike other black-box approaches, we use a differentiable surrogate model to di- rectly optimize the graph structures. The surrogate model takes graph structure corresponds to a neural architecture as input, and predicts a relative ranking score as the searching signal. We then apply gradient descent on the input graph structure to optimize the corresponding architecture, which attempts to obtain a better predicted ranking score. As we optimize the surrogate model and the architectures iteratively, the optimal architectures could be typically obtained after few itera- tions. In particular, to cope with the difficulty of using gradient-basee optimization on the discrete graph structure, adapt the proximal algorithm for allowing optimize discrete variables in

---

Under review as a conference paper at ICLR 2021
GNN propagation
original architecture
predicted ranking g f(a))
back-propagation Vaz ((t)
gradicnt desccnt
better architeeture
Figure I: Overview of the GOAL steps. GNN-based surrogate model f predicts the ranking score ÿ of the original architecture Ote, then back propaprrogg through the GNN model to compute the gradient w.r.t ag.. better architecture a4+1 obtained by proximal gradient descent.
differentiable manner (Parikh et al. 2014 Bai et al. 2019 Yao et al. 2019). We build the surrogate
model with Edge Conditional Convolution (ECC) (Simonovsky & Komodakis 2017). D. variant of Graph Convolutional Network (GCN) (Kipf & Welling 2017), to handle the graph representation of various search spaces.
Our empirical results on existing NAS benchmarks with different search spaces demonstrate that, GOAL outperforms exist state-of-the-art black-box optimization baselines and neural surrogate model based methods by a large margin in terms of the search efficiency.
The main contributions of this paper are summarized as follows:
We propose differentiable surrogate model for ranking neural architectures based on GNN, which takes advantage of the graph structure of neural architectures and guides the architecture search efficiently.
We present GOAL, a novel gradient-based NAS sample-efficient approach with the assis- tance the proposed surrogate model. Comparing exist algorithms with GNN sur- rogates, GOAL makes full use of the learned representation by jointly optimizing the GNN model and candidate architectures and performing efficient exploitation within graph- structured search spaces.
Our empirical results demonstrate that the GOAL significantly outperforms existing state- of-the-art methods in various search spaces settings.
RELATED WORKS
2.1 NEURAL ARCHITECTURE SEARCH
From the earliest explorations on automatic neural network designing to recent NAS trends, the NAS problem developed from hyper-parameter optimization, becoming
challenging task due to
the inherent complexity of its search space (Bergstra et al.) 2013 Elsken et al.] 2018). Existing
popular approaches include various kinds of algorithms: reinforcement learning (Zoph & Le 2016
Zoph et al. 2017 Pham et al.] 22188) evolution algorithm (Real et al. 2019), Bayesian optimization Falkner al. 2018 White et 2019). monte carlo tree search (Wang al. 2019b a). gradient based methods (Lu et al.) 2019: Luo et al. 2018). etc. There are also some works employ surrogate models to predict the nerformance of architectures before training to reduce the cost of architecture evaluation (Liu et al. 2018 Wen et al.) 2019; Wang et al.) 2019b). Some most recently parallel works even tries to improve black-br box optimization methods like Bayesian optimization by efficient surrogate predictors (White et al. 2019 Shi et al. 2019). Existing gradient-based methods usually employ shightt-haring based relaxation or use encoder-decoder to optimize continuous hidden space (Liu et al. 2019 Luo et al. 2018). These approximations cause biased model criterion and generation, which can harm the final performance (Yu et al. 2019, Yang et al., 2020). In contrast, method directly optimizes the discrete architectures, avoids the biased model criterion.
2

---

Under review as a conference paper at ICLR 2021
Due to the complex search settings and expensive evaluation cost, NAS works are hard to evaluate and reproduce fairly (Li & Talwalkar 2019 Yang et al. 2020). NAS- MAS-Bench-100 101 and NAS- Bench 201
proposed to provide fair and easy benchmarks for NAS algorithms (Dong & Yang. 2020 Ying et al. 2019).
Taking advantage of the development of graph neural networks (GNNs), several recent works show that the power of GNNs can also benefit the NAS task (Shi et al. 2019 Wen et al.] (2199). However, these works only employ GNNs as a powerful black-box predictor. In contrast, we make full of the learned representation of GNN by back-propagating through the GNN model and performing
gradient- optimization on the input graphs.
2.2 GRAPH STRUCTURE LEARNING
Graph neural network (GNN) is kind of neural model for extracting features from graph-structured data (Zhou et al. 2018). The most popular family of GNNs is the Graph Convolutional Network (GCN), which propagates the features of nodes in the spectral monomorphic graphs with single 2017 Morris et al. 2019). Since the vanilla GCN can only handle
spatial domain (Kipf & Welling
type of edge, many variants have been proposed handle more complex graphs (Schlichtkrull al., 2018 Simonovsky & Komodakis 2017).
As the normal GNN pipeline requires available fine-grained graph structure, recent works present approaches to optimize the graph structure and GNN model jointly (Chen et al.) 2019; Franceschi al. 2019). The graph structures could be either constructed from scratch or fine-tuned from sub-optimal graph with the GNN model. We follow this manner, iteratively optimize the graph of neural architectures, guided by the GNN model.
PRELIMINARY: NEURAL ARCHITECTURE SEARCH SPACES
While our method can be potentially applied more generally, in this paper we focus on the most commonly used cell-based search spaces of convolutional neural networks (CNNs) (Zoph et al., 2017 Pham et al.) 2018, Liu et al., 20199) Ying al., 2019: Dong Yang 2020). A cell basic building block of architectures represented as a DAG, which consists of several basic neural operations and their connectivity relationship as shown
Fig 2 Popular basic operations include different kinds of convolutions and poolings, such as 1 x I convolution, 3 x 3 convolution, 3 x max pooling, etc. A complete CNN architecture is thus formed by stacking several repeats of the same cell.
There are different kinds of representations to form an architecture by a DAG. Fig shows two typical representations. It is non-trivial convert from each other, since the corresponding architectures formed by different representations do not coincide.
input/ouiput
intermediate outputs
operations
operations
disconnected counected
Figure 2: Two typical architecture representations as DAGs. Left: nodes as operations, edges as connections; right: nodes as intermediate outputs, edges as operations.
The cell-based search spaces usually heuristically constrained by limiting the max number of nodes and edges, or the degree each node, which corresponds to the number of operations and connections in a cell (Liu et al. 2019: Ying al. 2019). This brings different feasible sets in optimization in Sec 5.1
4 LEARNING TO RANK ARCHITECTURES BY GRAPH
We first design the GNN model which guides our optimization on neural architectures. Denoting a neural architecture setting a, train differentiable model f(a:) which predicts score ý to
3

---

Under review as a conference paper at ICLR 2021
indicate the real performance y, e.g. accuracy the image classification task, of a. We build the model as GNN by Edge Conditional Convolution (ECC) layers (Simonovsky & Komodakis 2017) While the GNN model itself is fully differentiable, it possible to obtain gradient w.r.t from f(a) to optimize the architecture.
4.1 GRAPH REPRESENTATION OF NEURAL. ARCHITECTURES
As described in Sec mainstream NAS works search architectures in search spaces which could be represented directed acyclic graphs (DAGs). We form these search spaces unified perspective.
Denoting G = (V, E, T. as the graph corresponding to an architecture a, where V the set of nodes, E is the set of edges, T and A denote the types of nodes and edges respectively. For predefined search space, we can form each architecture in the space by fixing V and E as fully- connected DAG, where each node is connected each node for any < j S [V], and assign proper T and A. As the connectivity between nodes can also be denoted as an edge type (i.e. denoting as disconnected type, as shown in the left of Fig 22, we can determine both operation types and connectivity of the graph by T and A. The neural architecture setting ar is therefore formed by a AT, consisting of the types of nodes and types of edges for graph structure. In particular, we form each component of 'T and A as one-hot vector, which encodes categorical selection.
To build the surrogate model dealing with various nodes and edges types in the graph representation of neural architectures, we employ the ECC, a variant of GCN, to extract the features h, of each node i in the graph (Simonovsky & Komodakis 2017). For each node i in the directed graph G, the I-th layer of a L-layers ECC propagates features h 2-1 of the previous layer from node j E (i) to node i, where (i) denotes the predecessors set of node i. The ECC learns kernel generator ke + Rd 2, which maps the edge attribute space A to a veal-valued de x de-1 filter for the graph convolution, where de indicates the number of channels in the l-th convolution layer. Thus the graph convolution is formed by:
h = O'
Σ k((a)))))
JEN(1)
where ay is the attribute, i.e. categorical selection as one-hot vector, of the edge from node j to node ② is learnable parameter.
For high-level prediction upon the whole graph, we compute the feature vector he of the whole graph G by taking the weighted sum of each node features of the last layer (Li et al. 2016):
ha = Σ exp(wh:) E, exp(wh,) h;
(2)
where is a learnable parameter to compute the weight for features of each node. Followed by fully-connected layers that take hG as input, we can get the final prediction ÿ as an indicator score of the input architecture.
4.2 OBJECTIVE FUNCTION
Since only care about the relative order to gain better architectures, use the pair-wise rank loss for our training objective function (Burges et al. 2005). Let = {(a,, yı)} denotes the dataset which contains all evaluated architecture a; and corresponding performance ys. ÿr f(a,) be the predicted score, the loss L is formed by:
L Σ
log (1 + - (Te- Siggn(y: w))))
(3)
(1 i
Thus the prediction ÿ could be used select the best architectures.
4

---

Under review as a conference paper at ICLR 2021
5 SEARCH BY JOINT OPTIMIZATION
We thus present our method for searching optimal architectures by gradient. We jointly optimize the surrogate model and the candidate architectures alternately find an optimal final result. For allowing gradient- based optimization on the discrete graph structures corresponding to neural archi- tectures, we introduce the proximal gradient descent on the graph structures. Then we present the complete search algorithm.
5.1 PROXIMAL GRADIENT DESCENT ARCHITECTURES
Since the learned surrogate f in Sec 4 is differentiable, we want to perform gradient descent or by the predicted Y to find better architecture. However, the edge type selection the searching phase is discrete, bringing hardness to optimize directly by gradient descent. We thus introduce the proximal optimization to perform gradient-basee optimization on a discrete feasible set (Parikh) al., 2014 Yao et al. 2020 Bai et al.] 2019). Assume the learnt f outputs lower ÿ for better architecture ar, our search could be formed as a constrained optimization arg mina f(a), s.t. a E the search space. Typically, for the graph representation G G= (V, A, R. corresponds to a = (A, R). S requires the A and R to be discrete categories for operation selection. Recall that form each component in A and R one-hot vectors, update a iteratively with a continuous approximation o, whose components are real-valued continuous vectors, by following iterative steps:
S, where S is the feasible set depends
a2 =(((r) Ø141 - Va.f(a))
where the gradient step size; q(-) is the proximal operator maps continuous variables to the feasible set S, which satisfies ((a) arg min ail, s.t. a E S. Eqn. is known as a limiting case of lazy proximal step (Bai et al., 2019 Xiao 2010). Practically we harden each continuous component vector in to the nearest one-hot categorical vector by setting the max- valued component to 1 and others to O.
5.2 SEARCH PROCEDURE
Algorithm 1: GOAL: Graph Optimized Neural Architecture Learning
Input:
Random initialized architecture candidates pool P = {āg. ai...., n }: max time cost budget T; max number K for new produced architectures each step; random initialized surrogate model f-
Train and evaluate each (()) corresponds to â E P; while unsatisfied terminating condition do Train f with P using loss function C in Eqn.
Pnow Obtain new & for each o; P by Eqn.
Pnew + Pnew with the best predicted ranking score;
for ã; Puew do
Train and evaluate the architecture corresponding to g(ā,);
end
P -- PUPPU new:
end
Result: The best architecture in P.
The essential of the search is a joint optimization process on the architectures and the surrogate model A complete algorithm description is presented in Alg
The search process starts at a small batch of random sampled architecture continuous approxima- tions, which we denote the architecture candidates pool P {ā.). We train and evaluate each architecture a corresponds to g(ā)), then train the initial surrogate model f with the evaluated architectures as training data.
5

---

Under review as a conference paper at ICLR 2021
In each iteration of the search loop, optimize each a P Equ. to obtain potentially better architecture approximation. The new generated approximations with the best K predicted ranking scores will be evaluated and collected to enlarge the candidates pool P. Then the enlarged pool used to further improve the surrogate model f. which enable more precise prediction and gradient for architecture exploitation.
The process terminate at a pre-defined condition, e.g. the total training cost budget. We choose the best architecture in P the final result.
5.3 WEIGHTT-HHARING STRATEGY
Weight-sharing is powerful approach for reducing the total training time on architecture evaluation (Pham et al. 2018). GOAL can easily take benefits from weight-sharing to further improve the overall efficiency.
Our weight-sharing pipeline follows Li & Talwalkar (2019) and Guo et al. (2019) which first train a super-net model, then evaluate architectures as substructures in the super-net to search the best architecture. The super-net consists of all the possible modules of the architectures in the search space. To train a super-net for weight-sharing search, a substructure is uniformly sampled and separated from the super-net in each training iteration. The weights are inheritance from the super- net, and only the weights of the selected substructure will be trained in this iteration.
Specially, we adapt the algorithm Alg to find the best architecture in the trained super-net. The only difference here is that we do not need to train and evaluate an architecture from scratch; instead we use the substructure separated from the super-net, run a large validation batch and calculate the accuracy as the evaluation of the architecture.
EXPERIMENTS
In this section, we evaluate the effectiveness and generality of the GOAL method. The code for our approach is provided for research purpose
6.1 DATASETS
We run experiments on two NAS benchmark datasets: MMS-Bench-1O1 and NAS-Bench-201 (Ying ct al. 2019 Dong & Yang 2020). The datasets obtain validation and testing accuracy of all architec- tures in their search spaces under consistent training and testing settings. We use the validation and test accuracy of the image classification task CIFAR-10, provided by both datasets, as our search target. A brief description of the search spaces provided by the datasets is presented in Appendix A
6.2 STANDALONE SURROGATE MODEL. ANALYSIS
We first evaluate the standalone surrogate model. To evaluate the ability of ranking architectures of the proposed surrogate, we train the model to rank the architectures in NAS-Bench-201 by the validation accuracy. We split 3k of 6k unique structures in NAS-Bench-201 For the training set, we sample (128, 256, 512} architectures with their final validation accuracy to evaluate the models with different sizes of training data. The GNN model consists of 4 ECC layers of 64 hidden layer size. For comparison, we build an alternative multilayer perceptron model following AlphaX (Wang al. 2019b), which stacked by 4 layers of 64 hidden units. We train each model with batch size of 32 and Adam optimizer with 2 x 10-3 learning rate and 5 x 10-1 weight decay. We use the Spearman's 's rank correlation coefficient p between the ground truth and the predicted score the evaluation metric. We rank the results in ascending order of validation error, thus the lower rank order donates a better architecture.
the test set.
As shown in Fig 3 the GNN model suppresses the MLP model constantly. Note that during the actual searching process, training data of the surrogate is dynamically selected by gradient descent. We present the surrogate performance in such situation in Appendix D
Code included with the supplementary material. Code will be released upon the paper acceptance.

---

Under review as a conference paper at ICLR 2021
model samples, 0.9595
model w/
EMN model
MLP model
3000
3000
2500
2500
2000
2000
5. 0.90
0.88
500
500
0.86
0.84
3000
2000
3000
200
3.00
400
500
training data size
Figure 3: Ranking results on architectures in NAS-Bench-200 where stands for Spearman's 's rank correlation coefficient, lower rank order stands for better architecture. Left: ranking results of dif- ferent models; right: comparison between models with varying training data sizes.
6.3 SEARCH RESULTS
Settings For search evaluation, we follow the original benchmark protocols of NAS- NAS-Bench- 101 and NAS-Bench-201. We use the final validation accuracy of NAS-Bench-101 and the 12th training epoch validation accuracy of NAS-Bench-20/ as the search signal, then report the final valid and test accuracy of the searched architectures. Both the validation and test accuracy are provided by the benchmark datasets. We keep the stochastic accuracy setting of both datasets, which returns random accuracy value for each query simulate the randomness in architecture training and evaluation. We run all algorithms for 50 trails with different random seeds, then take the mean and standard deviation of the results. We set the terminating condition of all methods the max budget for the total training time of all evaluated architectures. for 5 x 10y seconds on MAS-Bench- -01 (for about 300 samples) and x 10a seconds on NAS-Bench-201 (for about 240 samples). For experiments on GOAL, we set K = 5, gradient step size μ = 1, initial pool size N = 64 for NAS- Bench-101. 101, iteration in Alg I
32 for PAS-Bench-201. The surrogate model f is trained for 25 epochs for each Other model settings keep the same in Sec 6.2
Baselines We use the following state- socec-offtheeeae the-art art baselines for comparison on both datasets: Ran- dom Search (RS) (Ying et al.] 2019, Dong & Yangl 2020). Reinforcement Learning (RL) Zooph
& Le 2016 Ying et al.) (219). Regularized Evolution Algorithm (REA) (Real al.
2019).
Bayesian Optimization Hyperband (BOHB) (Falkner et al.) 2018). Sequential Model-based Al-
gorithm Configuration (SMAC). Neural Predictor for Neural Architecture Search (NPNAS) (Wen)
et al. 2019). For fair comparison, we equip NPNAS with the GNN surrogate model we proposed.
To validate the contribution of GNN surrogate in GOAL, we additionally includes a variant, namely GOAL-MLP, which employs MLP model instead of GNN as the surrogate model in the GOAL pipeline.
On WAS-Bench-IWe 101, we additionally run AlphaX (Wang et al. 2019b) and Neural Architecture Op- timization (NAO) (Luo al., 2018).
We provide detailed settings of the baselines in Appendix
Results Fig shows the mean accuracies of the searched best architectures of the compared al- gorithms. We also appear the version with standard deviation error bar Appendix We show both validation and test curve with the total time cost growing. The Oracle on NAS-Bench-20) shows the upper bound, which indicates the average and variance of the results of the global optimal architectures selected by the best 12th valid accuracy.
GOAL shows promising efficiency on both datasets. Comparing to the small search space NAS- Bench-201, advantages of GOAL are more obvious on NAS-Bench- 101, showing that our method is more efficient to handle complex graph structures in large search space. In the smaller search space of WAS-Benche
our GOAL quickly assess the range of Oracle after about 1.2 x 10 seconds, and robustly stays at the global optima after 1.7 10 seconds, showing that GOAL can quickly find the global optimal solutions small search space.

---

Under review as a conference paper at ICLR 2021
Note that though GOAL-MLP marginally better than other baselines, the comparison between GOAL and GOAL-MLP shows the importance of the high-quality graph representation of GOAL. However, both GOAL-MLP and GOAL significantly suppress NPNAS, the offline GNN predictor based method, and NAO, the encoder-decoder based optimization method, shows that our proximal optimization based pipeline can improve the exploitation efficiency.
further case study of the searched architectures is presented in Appendix
GOAL
Random
GOAL
Random
6.0
GOAL MLP
BHOB
9.8
GOAL MIP
BOHB
PNNA
9.6
N
AlphaxX
REA
Gracle
valla or (80
RL
NAD
9.4
RL
5.6
9.0
5.4
8.8
8.4
5.0
2
0.5
1.0
1.5
2.0
2.5
time cost (5)
1e5
time cost (5)
106
7.0
7.0
GOAL
Random
GOAL
Random
GOAL-MIP
BHD8
6.8
GOAL-MLP
60HB
REA
Alphax
6.6
REA
Oracle
1052 orr (N)
RL
NAO
6.4
6.2
6.4
6.0
5.8
5.6
6.0
5.4
2
0.5
1.0
1.5
2.0
2.5
time cost (5)
1e5
Lime cost
1e6
NAS-Bench-101
(b) NAS-Bench-201
Figure 4: Comparisons of search efficiency. Left: validation (top) and test (bottom) error of archi- tectures searched on NAS-Bencch-001 right: validation (top) and test (bottom) error of architectures searched on NAS-Bench-201
6.4 WEIGHT-SHARIE RESULTS
NAS Bench 201 Space We run the weight-sharing search on NAS-Bench-201 to evaluate the per- formance of GOAL with weight-sharing strategy. Following the settings in NAS- BeS-Bench-20,, we train super-net for 250 epochs CIFAR-10, then the search process for trails with different seeds to evaluate the mean performance. For comparison, we use random-search (Li & Talwalkar 2019) and SPOS (Guo et al. 2019), state-of-the-arr based on evolution algorithm, baselines. As shown in table[[] GOAL leads better and more robust search results comparing to the baselines, suggesting the fitness with weight-sharing strategy.
DARTS Search Space To validate GOAL on larger search space, we perform weight-sharing search in the DARTS space (Liu et al. 2019) on CIFAR-10 dataset. We use the same size of proxy super-net DARTS, following the super-net training settings Li & Talwalkar (2019) For GOAL search stage, we set = 16, initial pool size N 256, gradient step size j evaluated 1024 architectures in the search stage. The found architecture is validated CIFAR-10 following the settings in Liu et al. (2019).
We totally
The final performance is summarized in Table GOAL finds more competitive architecture in iden- tical search space comparing to other weight-sharing methods based on different search strategies. The architecture searched by GOAL is visualized in Appendix F
8

---

Under review as a conference paper at ICLR 2021
Table 1: Results on weight-sharing search on WAS-Bench-201. Reported by Dong Yang 2020 DARTS failed on Mas--ench-200 benchmark due to its sensibility search space and hyperparam- eters. I 100 samples evaluated on 250 epoch trained super-net.
Method
CIFAR-10 validation (%)
CIFAR-10 test (%)
DARTS'
(Luu et al.
2019
39.77=00.00
54.30:10.00
Random 1#
(Li & Talwalkar
2019
8..16::1..9
87.66:1 1,69
SPOS Guo al.
2019)
88.91
92.32:10.89
GOAL*
89.33
92.84.10.51
Table 2: DARTS space result CIFAR-10 dataset.
Architecture
Test Err. (%)
Search Method
ENAS (Pham ct al. 2018)
2.89
RL
NAO-WS (Lo 2018
2.93
NAO gradient
DARTS
Liu al.
2019
2.76
Random & Talwalkar
2019
2.85
random
GOAL
2.64
GOAL
7 CONCLUSION
In this paper, we proposed Graph Optimized Neural Architecture Learning (GOAL), a novel gradient-based approach for neural architecture search combined with graph structure learning. Dif- ferent from popular black-box optimization based methods, we explicitly model the NAS as the optimization on graph structure, and apply proximal gradient descent discrete graph structures with assistance of a GNN-based surrogate model. Our GOAL outperforms mainstream SOTA NAS and syper-parameter optimization methods on NAS benchmarks, shows promising efficiency on the NAS problem.
REFERENCES
Yu Bai, Yu-Xiang Wang, and Edo Liberty. Proxquant: Quantized neural networks via proximal operators, In International Conference on Learning Representations, 2019.
J. Bergstra, D. Yamins, and D. D. Cox. Making science of model search: Hyperparameter opti- mization in hundreds of dimensions for vision architectures. In Proceedings of the 30th Interna- fional Conference on International Conference Machine Learning- Volume 28, ICML'13, pp- 1-115-1-123. .org 2013.
Chris Burges, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamilton, and Greg Hul- lender. Learning to rank using gradient descent. In Proceedings of the 22nd International Con- ference on Machine Learning, ICML '05, pp. 89-96, New York, NY, USA, 2005. Association for Computing Machinery. ISBN 1595931805, doi: 10. 1 144/1 100//1002311110000 1103663.
Daoyuan Chen, Yaliang Li, Minghui Qiu, Zhen Wang, Bofang Li, Bolin Ding, Hongbo Deng, Jun Huang, Wei Lin, and Jingren Zhou. Adabert: aask-adaptive bert compression with differentiable neural architecture search, 2020.
Yu Chen, Lingfei Wu, and Mohammed J. Zaki. Deep iterative and adaptive learning for graph neural networks, 2019.
Xuanyi Dong and Yi Yang. Nas-bench-200 Extending the scope of reproducible neural architecture search. In ICLR, 2020.
Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey, 2018.
Stefan Falkner, Aaron Klein, and Frank Hutter. Bohb: Robust and efficient hyperparameter opti- mization at scale, 2018

---

Under review as a conference paper at ICLR 2021
Luca Franceschi, Mathias Niepert, Massimiliano Pontil, and Xiao He. Learning discrete structures for graph neural networks. In Proceedings of the 36th International Conference on Machine Learning. 2019.
Golnaz Ghiasi, Tsung-Yi Yi Lin, and Quoc Le. Nas-fpn: Learning scalable feature pyramid chitecture for object detection. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Jun 2019. doi: 10.1 10.11099pp:2200100077..
Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng. Zechun Liu, Yichen Wei, and Jian Sun. Single path one-shot neural architecture search with uniform sampling. arXiv preprint OrXiv:1904.0044 00420,, 2019.
Frank Hutter, Holger H. Hoos, and Kevin Leyton-Browe Sequential model-based optimization for general algorithm configuration. In Proceedings the 5th International Conference Learning and Intelligent Optimization. LION'OS, pp. 507-523, Berlin, Heidelberg, 2011. Springer- Verlag.
ISBN 9783642255656. doi: 10.
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional net- works. In International Conference on Learning Representations (ICLR), 2017.
Liam Li and Ameet Talwalkar. Random search and reproducibility for neural architecture search, 2019.
Yujia Li, Richard Zemel, Mare Brockschmidt, and Daniel Tarlow. Gated graph sequence neural networks. In Proceedings of ICLR' 16, April 2016.
Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy. Progressive neural architecture search. Lecture Notes in Computer Science, pp. 19-35, 2018. ISSN 1611-3349. doi: 10.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable architecture search, In International Conference on Learning Representations, 2019.
Renqian Luo, Fei Tian, Tao Qin, En-Hong Chen, and Tie- Liu. Neural architecture optimization. In NIPS, 2018.
Renqian Luo, Xu Tan, Rui Wang, Tao Qin, Enhong Chen, and Tic- Yan Liu. cemi-supervised neural architecture search, 2020.
Christopher Morris, Martin Ritzert, Matthias Fey, William L. Hamilton, Jan Eric Lenssen, Gaurav Rattan, and Martin Grohe Weisfeiler and leman go neural: Higher-order graph neural networks. Proceedings of the AAAI Conference on Artificial Intelligence. 33:4602-4609, Jul 2019. ISSN 2159-5399, doi: 333001330
Neal Parikh, Stephen Boyd, et al. Proximal algorithms. Foundations and Trends in Optimization, 1(3):127-23,, 2014.
Hieu Pham, Melody Guan, Barret Zoph, Quoc V. Le, and Jeff Dean. Efficient neural architecture search via parameter sharing, 2018.
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoe V Le. Regularized evolution for image classifier architecture search. In Proceedings of the aaai conference on artificial intelligence, volume 33, pp. 4780-4789, 2019.
Michael Schlichtkrull, Thomas N. Kipf. Peter Bloem, Rianne van den Berg, Ivan Titov, and Max Welling. Modeling relational data with graph convolutional networks. Lecture Notes in Computer Science, pp. 593-607, 2018. ISSN 1611-3348 doi: 000079978-3-319-9441---
Han Shi, Renjie Pi, Hang Xu, Zhenguo Li, James T. Kwok, and Tong Zhang. Efficient sample-based neural architecture search with learnable predictor, 2019.
Martin Simonovsky and Nikos Komodakis. Dynamic coneitiondt filters in convolutional neu- ral networks on graphs. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3693-3702, 2017.
10

---

Under review as a conference paper at ICLR 2021
Linnan Wang, Saining Xie, Teng Li, Rodrigo Fonseca, and Yuandong Tian. Sample-effiiient neural architecture search by learning action space, 2019a.
Linnan Wang, Yiyang Zhao, Yuu Jinnai, Yuandong Tian, and Rodrigo Fonseca. Alphax: exploring neural architectures with deep neural networks and monte carlo search, 2019b.
Wei Wen, Hanxiao Liu, Hai Li, Yiran Chen, Gabriel Bender, and Pieter-Jan Kindermans. Neural predictor for neural architecture search, 2019.
Colin White, Willie Neiswanger, and Yash Savani. Bananas: Bayesian optimization with neural architectures for neural architecture search, 2019.
Lin Xiao. Dual averaging methods for regularized stochastic learning and online optimization. Journal of Machine Learning Research, 1 11(88)2553-2596 2010.
Saining Xie, Alexander Kirillov, Ross Girshick, and Kaiming He. Exploring randomly wired neural networks for image recognition. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Oct 2019. doi: 10. 10.11099cxxx222190001333
Antoine Yang, Pedro M. Esperança, and Fabio M. Carlucci. Nas evaluation is frustratingly hard. In International Conference Learning Representations, 2020.
Q. Yao, J. Xu, W.W. Tu, and Z. Zhu. Efficient neural architecture search via proximal iterations. Technical report, 2020.
Quanming Yao, Ju Xu, Wei- Wei Tu, and Zhanxing Zhu. Efficient neural architecture search via proximal iterations, 2019.
Chris Ying, Aaron Klein, Eric Christiansen, Esteban Real, Kevin Murphy, and Frank Hutter. NAS- bench- 101: Towards reproducible neural architecture search. In ICML, 2019.
Kaicheng Yu, Christian Sciuto, Martin Jaggi, Claudiu Musat, and Mathieu Salzmann. Evaluating the search phase of neural architecture search, 2019.
Jie Zhou, Ganqu Cui, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun. Graph neural networks: A review of methods and applications. arXiv preprint arXiv:1812.08434, 2018.
Barret Zoph and Quoc V. Le. Neural architecture search with reinforcement learning, 2016.
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V. Le. Learning transferable architectures for scalable image recognition, 2017.

---

Under review as a conference paper at ICLR 2021
A
SEARCH SPACES OF THE DATASETS
The NAS-Bench- and NAS-Bench-201 form their search space differentl::
NAS-Bench-101 donates graph nodes as neural operations, and graph edges as input/output connection, as the left of Fig shows. The dataset provides a search space with at most 7 nodes and 9 edges. Except for input and oupur nodes, intermediate nodes are one of the operations in {1 1 convolution. 3 x 3 convolution, 3 3 max pooling). There are totally 423,624 unique graphs in the search space.
NAS-Bench-201 uses graph edges represent both connections and operations, nodes to aggregate intermediate outputs, just like the right of Fig 2 The nodes donate tensor element-wise addition, while edges are operations {1 convolution, lution, 3 x 3 average pooling, skip connection, zeroize]. This search space consists of 15,625 unique models without considering graph isomorphism. After de-duplicating by graph isomorphism, there are 6,466 unique topology structures.
convo-
B DETAILED BASELINE SETTINGS
Following SOTA of NAS and hyper-parameter optimization methods are used in both NAS-Bench- 101 and MAS--Bnch-200 experiments:
Random Search (RS) (Ying et al. 2019 Dong & Yang 2020). As reported by many prior works, RS is competitive algorithm for NAS. We use the implements and setrings in Ying et (019)) and Dong & Yang (2020).
2019). For the RL method, we use the implementation in (Ying et al. 2019), which directly learns the architecture distribution as action policy, instead of learning an RNN controller as in (Zoph et al., 2017).
Reinforcement Learning (RL) (Zoph & 2016 Ying et al.
Regularized Evolution Algorithm (REA) (Real et al. 2019). The original benchmarks of
both datasets show that REA is the state-of-the-an algorithm among the benchmarks (Ying et al. 2019 Dong & Yang 2020). We inherit the settings from Ying et al. (2019); Dong &
Yang 2020).
Bayesian Optimization Hyperband (BOHB) Falkner et al. 2018). BOHB a Bayesian optimization method combined with hyperband for fast hyperparameter optimization. We inherit the settings from Ying et al. (2019): Dong Yang (2020).
Sequential Model-based Algorithm Configuration (SMAC) (Hutter et al. 201 SMAC is a sequential model-based Bayesian optimization method which employs random forest the predictor. We inherit the settings from Ying al. (2019).
Neural Predictor for Neural Architecture Search (NPNAS) (Wen et al. 2019). The NPNAS samples fixed size of architectures from the search space as training data for train- ing a GCN-based surrogate model for once, then freeze the surrogate model and predict the performance of all architectures in the search space to select the best ones. We use the surrogate model we proposed to imitate the method of NPNAS for fair comparison on the searching phrase. We set the training set size to 172 on NAS-Bench- Bench-201, which are reported as the optimal values for our search constraint by (Wen ct al. 2019).
and 128 on NAS-
Following baselines are additionally run on NAS-Bench-
Neural Architecture Optimization (Luo et al. 2018). NAO employs an auto-encoder and perform searching in the learnt embedding space of architectures. We follow the NAS- Bench settings in Luo et al. (2020).
AlphaX (Wang et al.) 2019b). AlphaX adapts monte-care tree search on NAS problem. We keep the settings unchanged from Wang al. (2019h).
12

---

Under review as a conference paper at ICLR 2021
C EXPERIMENT RESULTS WITH VARIANCE
The experiment results on NAS-Bench-100 and TAS-Bench-200 with standard deviation is shown in Fig Note that the variance of the test error on WAS-Bench-101 is higher than the variance of validation error, since our direct search target is the validation enror, and the correlation between validation eтror and test error not perfect. Comparing other baselines, the better mean results of GOAL do not cause higher variance, even lower, showing the efficiency of GOAL is robust and promising.
6.2
10.0
GOAL
Random
Oracie
-
6.0
GOAL-MIP
BHDB
GOAL
Random
9.6
REA
AlphaX
N NPNNS
- SMAC
volld (86)
RL
NAO
9.4
REA
9.0
8.8
2
0.5
1.0
1.5
2.0
2.5
time cost (5)
1e5
Lime cost (s)
166
7.0
7.0
GOAL
Random
Oracle
- RL
6.8
GOAL-MIL
BHO0
6.8
GOAL
---
Random
NPNAS
6.6
RFA
Alphax
6.6
NNNAS
SMAC
RL
NAO
6.4
REA
1052 cr
6.2
6.0
5.8
5.6
5.4
2
0.5
1.0
1.5
2.0
2.5
time (s)
1e5
time cost
1c6
Figure 5: Comparisons of search efficiency with standard deviation. Left: validation (top) and test (bottom) error of architectures searched on NAS-Bench-1011 right: validation (top) and test (bottom) error of architectures searched on NAS- AAS-Bench-221.
D SURROGATE MODEL PERFORMANCE ACCOMPLISHED SEARCHING
To analysis the behavior of our surrogate model the search process, we show the ranking results on all unique architectures in NAS-Bench-200 of the surrogate model accomplished search process in Fig 6al For comparison, we also retrained the model with uniformly sampled training data and show the results in Fig Comparing the ranking results in Fig 6b the model accomplished searching performs better on the left-bottom corner, which consists of the best models the whole search space. Since the architectures with good performance
what actually concern in the NAS problem, our alternately training strategy in the searching process could enhance the model comparing with the training-once approaches like NPNAS.
E CASE STUDY OF THE ARCHITECTURE TRANSITION
7 shows the evolution history of an architecture found by GOAL on WAS-Bench-1O1. The case starts at extremely bad initialization, which directly sends the input to the output. The first step adds several convolution nodes in the company ResNets-like skip-connections, leading to meaningful structure. The following steps shrink the architecture to be smaller and fine-tune the connections. As presented in the transition steps, in the NAS Bench-101 space, more operations do not always lead better performance, and the connectivity impacts lot on the final validation
13

---

Under review as a conference paper at ICLR 2021
100g
1000
800
arch Recturee
ground butt rank
800
4000
000
4000
600
ground
400
400
200
200
2000
prediction
Figure 6: Left: the ranking result of the model accomplished search process on WAS-Bench-22 right: the ranking results of the model retrained with the same size uniformly sampled training data.
accuracy. Notice that only 3 x convolution operations are selected these architectures, which shows that other operations like max pooling are less helpful for improving accuracy the tasks in NAS-Bench-101.
valid err 16.84% valid 6.88%
valid err 5.94%
valid 5.19%
Figure 7: The 3-step evolution history of a found architecture NAS-Bende Bench- ', 'O', and 'C3' mean for input node, output node and x 3 convolution node respectively. The green parts are newly added, while dashed parts indicate the deletion.
F ARCHITECTURE SEARCHED ON DARTS SPACE
Ap-ousy 3x3
men 363
xp_ourv_3x3
M2.
ony a6
kkkk]
mas poel 3d3
cuev 515
(b)
(a)
Figure 8: Normal (left) and reduction (right) cell found by GOAL DARTS space.
14


Reviews:
[{"review_id": "OcGDojF6myE", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "#### Summary:\nThis work propose Graph Optimized Neural Architecture Learning, that uses a differentiable surrogate model to directly optimize the graph structures. More specifically, the surrogate model takes a graph structure as the neural architecture embedding and predicts a relative ranking, then applies gradient descent on the input graph structure to optimize the neural architecture. GOAL demonstrates superior performance compared to SoTAs.\n\n#### Weakness:\n\n-First, the method is quite similar to NAO, where an encoder and decoder approach the maps neural architectures into a continuous space and builds a predictor based on the latent representation. The only difference is that NAO is use a decoder to decode the optimized latent representation back to architecture representation while here GOAL applies gradient descent on a graph neural network. Also, how to decode the parameters back to the neural architecture discrete representation is not clearly explained in the paper.\n\n-Second,  this method can work well for small models and small search spaces, but can be hardly applied to larger models. Training the surrogate function for a larger search space or larger models can take more samples and more training time (e.g. large models takes much longer time to train, thus even a proxy accuracy should take more time to evaluate). A parameter sharing scheme can be very inaccurate in the beginning therefore results in suboptimal architecture selection. The inaccuracy is compounded when using the same model for both a surrogate function and neural architecture search. The evaluation on only NAS-bench partially verifies the reviewers concerns. \n\n#### Detailed feedback:\nThe reviewer would like to suggest several fixes to this paper:\n-First, try to compare to better baselines (e.g. NAO, more recent differentiable search work) on not only NASBench, but on real CIFAR10 or ImageNet workloads. \n\n-Second, compare a non graph neural network based approach with GOAL and show the necessity of using a graph neural network.\n\n-Third, be more clear about decoding back the graph neural network parameterizations to the neural architecture representation. Include more details on the number of samples used to train the surrogate function and hyperparameteers used in algorithm 1. \n\n\n[1] \"Neural Architecture Optimization\", https://arxiv.org/abs/1808.07233", "extra_scores": {}, "extra_text": {}}, {"review_id": "O61hSt0wp9", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors address the Neural Architecture Search problem. At the core of their contribution is an architectural improvement; performance prediction of a considered architecture is much better when using a particular graph neural network on (softened) architecture topology. The rest of the NAS pipeline is naturally built around this observation and the final performance looks quite strong.\n\nI think the topic is of sufficient significance to the community and I find the proposed method well-tailored to the problem and sufficiently original. Also, the writeup is easy to follow (up to minor issues listed below).\n\nSince the contribution is mainly in finding a good architecture for the surrogate model, I find it very unsatisfying that all experiments are carried out on CIFAR-10. The ML community has had some bad experience with architectures overfitting to datasets and more thorough evaluation is needed. Also, the prime purpose of the NAS line of work isn't to find a good architecture for CIFAR-10, and if this method aspires to have a broader impact in the NAS community (which I think it should), the experimental section needs to be expanded.\n\nMinor points:\nEq 3: I don't understand. Due to the $\\textrm{sign}$ function, the loss does not seem differentiable w.r.t $\\tilde{y}_i$. Or do the brackets need some regrouping?\nThe end of Sec 5.1: Currently $\\tau$ cancels in the definition of $q$. I guess it should be inside the brackets. Also, what does it mean to \"practically\" take a limit? The point seems to be that softmax with very low temperature reduces to a hard one-hot vector but I do not understand how the authors use this precisely. Why not \"round\" to the one-hot vector right away?\nSec 2.2: models -> model\nSec 4.1 donated -> denoted?\nExperiments: One ablation that I think would be great for giving insight to inner working of the method would be the following: In Figure 4, include also the performance curve of GOAL with the MLP architecture for rank prediction. This should nicely demonstrate where your main technical point is.", "extra_scores": {}, "extra_text": {}}, {"review_id": "8Xy7qcOjBBE", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposed (1) to use a graph neural net to predict the performance of network architectures, (2) to query new architecture proposals to try next from the predictor, and (3) to iteratively refine the predictor using the collected dataset using gradient descent. Overall, the method is well-motivated and the paper is easy to follow.\n\nMy main concerns are as follows:\n* In section 2.1 the authors claim that \"In contrast, our method directly optimizes the discrete architectures, avoids the drawbacks of continuous relaxation\". This is not true—the method still relies on continuous relaxation because of \\bar{\\alpha} in equation (4), and is still subject to discretization discrepancies due to the quantization function q.\n* Predictive methods with GCNs are not new, especially considering that Wen et al., 2019 also trained GCN using gradient descent (although in an offline fashion). The only additional ingredients here seem to be (a) interleaving GCN with the data collection process in an online manner, and (b) weight sharing. However, the advantage of (a) has not been empirically verified with ablation studies and (b) is already common nowadays.\n* It seems a bit unconventional that the authors reported the best architecture in the main paper but the mean & variance in the appendix. IMO Figure 5 in Appendix C provides a much more accurate picture of the usefulness of different algorithms than Figure 4. In practice, it can be tricky to tell the \"best\" architecture for new tasks where test sets are not available.\n* The improvements are within the range of variance as compared to SPOS. While the results are still overall positive, it may not well justify the additional implementation complexity of the method.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work was deemed interesting by the reviewers, but they highlighted the following weaknesses in this version of the paper:

- Lack of comparison to other methods.

- Lack of novelty compared to previous work.

- Fundamental problem with training only on one dataset (MNIST), issue with possible overfitting.
Decision: Reject