Title: GRAPH HYPERNETWORKS FOR NEURAL ARCHITECTURE SEARCH

Abstract: Neural architecture search (NAS) automatically finds the best task-specific neural network topology, outperforming many manual architecture designs. However, it can be prohibitively expensive as the search requires training thousands of different networks, while each can last for hours. In this work, we propose the Graph HyperNetwork (GHN) to amortize the search cost: given an architecture, it directly generates the weights by running inference on a graph neural network. GHNs model the topology of an architecture and therefore can predict network performance more accurately than regular hypernetworks and premature early stopping. To perform NAS, we randomly sample architectures and use the validation accuracy of networks with GHN generated weights as the surrogate search signal. GHNs are fast -they can search nearly 10× faster than other random search methods on CIFAR-10 and ImageNet. GHNs can be further extended to the anytime prediction setting, where they have found networks with better speed-accuracy tradeoff than the state-of-the-art manual designs.

1. INTRODUCTION:

The success of deep learning marks the transition from manual feature engineering to automated feature learning. However, designing effective neural network architectures requires expert domain knowledge and repetitive trial and error. Recently, there has been a surge of interest in neural architecture search (NAS), where neural network architectures are automatically optimized. One approach for architecture search is to consider it as a nested optimization problem, where the inner loop finds the optimal parameters w * for a given architecture a w.r.t. the training loss L train , and the outer loop searches the optimal architecture w.r.t. a validation loss L val : w * (a) = arg min w L train (w, a)

(1)

EQUATION

Traditional NAS is expensive since solving the inner optimization in Eq. 1 requires a lengthy optimization process (e.g. stochastic gradient descent (SGD)). Instead, we propose to learn a parametric function approximation referred to as a hypernetwork (Ha et al., 2017; Brock et al., 2018) , which attempts to generate the network weights directly. Learning a hypernetwork is an amortization of the cost of solving Eq. 1 repeatedly for multiple architectures. A trained hypernetwork is well correlated with SGD and can act as a much faster substitute.

Yet, the architecture of the hypernet itself is still to be determined. Existing methods have explored a variety of tactics to represent architectures, such as an ingenious 3D tensor encoding scheme (Brock et al., 2018) , or a string serialization processed by an LSTM (Zoph & Le, 2017; Zoph et al., 2018; Pham et al., 2018) . In this work, we advocate for a computation graph representation as it allows for the topology of an architecture to be explicitly modeled. Furthermore, it is intuitive to understand and can be easily extensible to various graph sizes.

To this end, in this paper we propose the Graph HyperNetwork (GHN), which can aggregate graph level information by directly learning on the graph representation. Using a hypernetwork to guide architecture search, our approach requires significantly less computation when compared to state-ofthe-art methods. The computation graph representation allows GHNs to be the first hypernetwork to generate all the weights of arbitrary CNNs rather than a subset (e.g. Brock et al. (2018) ), achieving stronger correlation and thus making the search more efficient and accurate.

While the validation accuracy is often the primary goal in architecture search, networks must also be resource aware in real-world applications. Towards this goal, we exploit the flexibility of the GHN by extending it to the problem of anytime prediction. Models capable of anytime prediction progressively update their predictions, allowing for a prediction at any time. This is desirable in settings such as real-time systems, where the computational budget available for each test case may vary greatly and cannot be known ahead of time. Although anytime models have non-trivial differences to classical models, we show the GHN is amenable to these changes.

We summarize our main contributions of this work:

1. We propose Graph HyperNetwork that predicts the parameters of unseen neural networks by directly operating on their computational graph representations. 2. Our approach achieves highly competitive results with state-of-the-art NAS methods on both CIFAR-10 and ImageNet-mobile and is 10× faster than other random search methods. 3. We demonstrate that our approach can be generalized and applied in the domain of anytimeprediction, previously unexplored by NAS programs, outperforming the existing manually designed state-of-the-art models.

2. RELATED WORK:

Various search methods such as reinforcement learning (Zoph & Le, 2017; Baker et al., 2017a; Zoph et al., 2018) , evolutionary methods (Real et al., 2017; Miikkulainen et al., 2017; Xie & Yuille, 2017; Liu et al., 2018b; Real et al., 2018) and gradient-based methods (Liu et al., 2018c; Luo et al., 2018) have been proposed to address the outer optimization (Eq. 2) of NAS, where an agent learns to sample architectures that are more likely to achieve higher accuracy. Different from these methods, this paper places its focus on the inner-loop: inferring the parameters of a given network (Eq. 1). Following Brock et al. (2018) ; Bender et al. (2018) , we opt for a simple random search algorithm to complete the outer loop.

While initial NAS methods simply train candidate architectures for a brief period with SGD to obtain the search signal, recent approaches have proposed alternatives in the interest of computational cost. Baker et al. (2017b) propose directly predicting performance from the learning curve, and Deng et al. (2017) propose to predict performance directly from the architecture without learning curve information. However, training a performance predictor requires a ground truth, thus the expensive process of computing the inner optimization is not avoided. Pham et al. (2018) ; Bender et al. (2018) ; Liu et al. (2018c) use parameter sharing, where a "one-shot" model containing all possible architectures in the search space is trained. Individual architectures are sampled by deactivating some nodes or edges in the one-shot model. In this case, predicting w * (a) can be seen as using a selection function from the set of parameters in the one-shot model.

Prior work has shown the feasibility of predicting parameters in a network with a function approximator (Denil et al., 2013) . Schmidhuber (1992; 1993) proposed "fast-weights", where one network produces weight changes for another. HyperNetworks (Ha et al., 2017) generate the weights of another network and show strong results in large-scale language modeling and image classification experiments. SMASH (Brock et al., 2018) applied HyperNetworks to perform NAS, where an architecture is encoded as a 3D tensor using a memory channel scheme. In contrast, we encode a network as a computation graph and use a graph neural network. While SMASH predicts a subset of the weights, our graph model is able to predict all the free weights.

While earlier NAS methods focused on standard image classification and language modeling, recent literature has extended NAS to search for architectures that are computationally efficient (Tan et al., 2018; Dong et al., 2018; Hsu et al., 2018; Elsken et al., 2018; Zhou et al., 2018) . In this work, we applied our GHN based search program on the task of anytime prediction, where we not only optimize for the final speed but the entire speed-accuracy trade-off curve.

3. BACKGROUND:

We review the two major building blocks of our model: graph neural networks and hypernetworks.

Graph Neural Network: A graph neural network (Scarselli et al., 2009; Li et al., 2016; Kipf & Welling, 2017 ) is a collection of nodes and edges (V, E), where each node is a recurrent neural network (RNN) that individually sends and receives messages along the edges, spanning over the The GHN is trained to minimize the training loss of the sampled network with the generated weights.

Random networks are ranked according to their performance using GHN generated weights.

horizon of message passing. Each node v stores an internal node embedding vector h

(t)

v ∈ R D , and is updated recurrently:

EQUATION

)

where U is a recurrent cell function and m

(t)

v is the message received by v at time step t:

EQUATION

with M the message function and N in (v) the set of neighbors with incoming edges pointing towards v. U is often modeled with a long short-term memory (LSTM) unit (Hochreiter & Schmidhuber, 1997) or gated recurrent unit (GRU) (Cho et al., 2014) , and M with an MLP. Given a graph A, we define the GNN operator G A to be a mapping from a set of initial node embeddings {h

(0) v } to a set of different node embeddings {h (t)

v }, parameterized by some learnable parameters φ:

EQUATION

Throughout propagation the node embeddings h

(t)

v continuously aggregate graph level information, which can be used for tasks such as node prediction and graph prediction by further aggregation. Similar to RNNs, GNNs are typically learned using backpropagation through time (BPTT) (Werbos, 1990) .

Hypernetwork: A hypernetwork (Ha et al., 2017) is a neural network that generates the parameters of another network. For a typical deep feedforward network with D layers, the parameters of the j-th layer W j can be generated by a learned function H:

EQUATION

where z j is the layer embedding, and H is shared for all layers. The output dimensionality of the hypernetwork is fixed, but it's possible to accommodate predicting weights for layers of varying kernel sizes by concatenating multiple kernels of the fixed size. Varying spatial sizes can also be accommodated by slicing in the spatial dimensions. Hypernetworks have been found effective in standard image recognition and text classification problems, and can be viewed as a relaxed weight sharing mechanism. Recently, they have shown to be effective in accelerating architecture search (Brock et al., 2018) .

4. GRAPH HYPERNETWORKS FOR NEURAL ARCHITECTURAL SEARCH:

Our proposed Graph HyperNetwork (GHN) is a composition of a graph neural network and a hypernetwork. It takes in a computation graph (CG) and generates all free parameters in the graph.

During evaluation, the generated parameters are used to evaluate the fitness of a random architecture, and the top performer architecture on a separate validation set is then selected. This allows us to search over a large number of architectures at the cost of training a single GHN. We refer the reader to Figure 1 for a high level system overview.

4.1. GRAPHICAL REPRESENTATION:

We represent a given architecture as a directed acyclic graph A = (V, E), where each node v ∈ V has an associated computational operator f v parametrized by w v , which produces an output activation tensor x v . Edges e u →v = (u, v) ∈ E represent the flow of activation tensors from node u to node v.

x v is computed by applying its associated computational operator on each of its inputs and taking summation as follows

EQUATION

4.2. GRAPH HYPERNETWORK:

Our proposed Graph Hypernetwork is defined as a composition of a GNN and a hypernetwork. First, given an input architecture, we used the graphical representation discussed above to form a graph A. A parallel GNN G A is then constructed to be homomorphic to A with the exact same topology. Node embeddings are initialized to one-hot vectors representing the node's computational operator.

After graph message-passing steps, a hypernet uses the node embeddings to generate each node's associated parameters. Let h

(T ) v

be the embedding of node v after T steps of GNN propagation, and let H (•; ϕ) be a hypernetwork parametrized by ϕ, the generated parameters wv are:

EQUATION

)

For simplicity, we implement H with a multilayer perceptron (MLP). It is important to note that H is shared across all nodes, which can be viewed as an output prediction branch in each node of the GNN. Thus the final set of generated weights of the entire architecture w is found by applying H on all the nodes and their respective embeddings which are computed by G A : The computation graph of some popular CNN architectures often spans over hundreds of nodes (He et al., 2016a; Huang et al., 2017) , which makes the search problem scale poorly. Repeated architecture motifs are originally exploited in those architectures where the computation of each computation block at different resolutions is the same, e.g. ResNet (He et al., 2016b) . Recently, the use of architectural motifs also became popular in the context of neural architecture search, e.g. (Zoph et al., 2018; Pham et al., 2018) , where a small graph module with a fewer number of computation nodes is searched, and the final architecture is formed by repeatedly stacking the same module. Zoph et al. (2018) showed that this leads to stronger performance due to a reduced search space; the module can also be transferred to larger datasets by adopting a different repeating pattern.

EQUATION

Our proposed method scales naturally with the design of repeated modules by stacking the same graph hypernetwork along the depth dimension. Let A be a graph composed of a chain of repeated modules {A i } N i=1 . A graph level embedding h Ai is computed by taking an average over all node embeddings after a full propagation of the current module, and passed onwards to the input node of the next module as a message before graph propagation continues to the next module.

EQUATION

h Ai = 1 |V i | v∈Vi h (T ) v |v ∈ V i (13) = 1 |V i | G (T ) Ai h (0) v |v ∈ V i , h Ai-1 ; φ ∀i > 0 (14)

Note that G Ai share parameters for all A i . Please see Figure 2 for an overview. Standard GNNs employ the synchronous propagation scheme (Li et al., 2016) , where the node embeddings of all nodes are updated simultaneously at every step (see Equation 3). Recently, Liao et al. (2018) found that such propagation scheme is inefficient in passing long-range messages and suffers from the vanishing gradient problem as do regular RNNs. To mitigate these shortcomings they proposed asynchronous propagation using graph partitions. In our application domain, deep neural architectures are chain-like graphs with a long diameter; This can make synchronous message passing difficult. Inspired by the backpropagation algorithm, we propose another variant of asynchronous propagation scheme, which we called forward-backward propagation, that directly mimics the order of node execution in a backpropagation algorithm. Specifically, let s be a topological sort of the nodes in the computation graph in a forward pass,

EQUATION

(

EQUATION

The total number of propagation steps T for a full forward-backward pass will then become 2|V|-1.

Under the synchronous scheme, propagating information across a graph with diameter |V| would require O(|V| 2 ) messages. This is reduced to O(|V|) under the forward-backward scheme.

4.5. LEARNING:

Learning a graph hypernetwork is straightforward since w are directly generated by a differentiable network. We compute gradients of the graph hypernetwork parameters φ, ϕ using the chain rule:

∇ φ,ϕ L train ( w) = ∇ wL train ( w) • ∇ φ,ϕ w (16)

The first term is the gradients of standard network parameters, the second term is decomposed as

EQUATION

EQUATION

where (Eq. 17) is the contribution from GNN module G and (Eq. 18) is the contribution from the hypernet module H. Both G and H are jointly learned throughout training.

5. EXPERIMENTS:

In this section, we use our proposed GHN to search for the best CNN architecture for image classification. First, we evaluate the GHN on the standard CIFAR (Krizhevsky & Hinton, 2009) and ImageNet (Russakovsky et al., 2015) architecture search benchmarks. Next, we apply GHN on an "anytime prediction" task where we optimize the speed-accuracy tradeoff that is key for many real-time applications. Finally, we benchmark the GHN's predicted-performance correlation and explore various factors in an ablation study.

5.1. NAS BENCHMARKS:



5.1.1. CIFAR-10:

We conduct our initial set of experiments on CIFAR-10 ( Krizhevsky & Hinton, 2009) , which contains 10 object classes and 50,000 training images and 10,000 test images of size 32×32×3. We use 5,000 images split from the training set as our validation set. Search space: Following existing NAS methods, we choose to search for optimal blocks rather than the entire network. Each block contains 17 nodes, with 8 possible operations. The final architecture is formed by stacking 18 blocks. The spatial size is halved and the number of channels is doubled after blocks 6 and 12. These settings are all chosen following recent NAS methods (Zoph & Le, 2017; Pham et al., 2018; Liu et al., 2018c) , with details in the Appendix.

Training: For the GNN module, we use a standard GRU cell (Cho et al., 2014) with hidden size 32 and 2 layer MLP with hidden size 32 as the recurrent cell function U and message function M respectively. The shared hypernetwork H (•; ϕ) is a 2-layer MLP with hidden size 64. From the results of ablations studies in Section 5.4, the GHN is trained with blocks with N = 7 nodes and T = 5 propagations under the forward-backward scheme, using the ADAM optimizer (Kingma & Ba, 2015) . Training details of the final selected architectures are chosen to follow existing works and can be found in the Appendix.

Evaluation: First, we compare to similar methods that use random search with a hypernetwork or a one-shot model as a surrogate search signal. We randomly sample 10 architectures and train until convergence for our random baseline. Next, we randomly sample 1000 architectures, and select the top 10 performing architectures with GHN generated weights, which we refer to as GHN Top. Our reported search cost includes both the GHN training and evaluation phase. Shown in Table 1 , the GHN achieves competitive results with nearly an order of magnitude reduction in search cost.

In Table 2 , we compare with methods which use more advanced search methods, such as reinforcement learning and evolution. Once again, we sample 1000 architectures and use the GHN to select the top 10. To make a fair comparison for random search, we train the top 10 for a short period before selecting the best to train until convergence. The accuracy reported for GHN Top-Best is the average of 5 runs of the same final architecture. Note that all methods in Table 2 use CutOut (Devries & Taylor, 2017) . GHN achieves very competitive results with a simple random search algorithm, while only using a fraction of the total search cost. Using advanced search methods with GHNs may bring further gains.

5.1.2. IMAGENET-MOBILE:

We also run our GHN algorithm on the ImageNet dataset (Russakovsky et al., 2015) , which contains 1.28 million training images. We report the top-1 accuracy on the 50,000 validation images. Following existing literature, we conduct the ImageNet experiments in the mobile setting, where the model is constrained to be under 600M FLOPS. We directly transfer the best architecture block found in the CIFAR-10 experiments, using an initial convolution layer of stride 2 before stacking 14 blocks with scale reduction at blocks 1, 2, 6 and 10. The total number of flops is constrained by choosing the initial number of channels. We follow existing NAS methods on the training procedure of the final architecture; details can be found in the Appendix. As shown in 

5.2. ANYTIME PREDICTION:

In the real-time setting, the computational budget available can vary for each test case and cannot be known ahead of time. This is formalized in anytime prediction, (Grubb & Bagnell, 2012) the setting in which for each test example x, there is non-deterministic computational budget B drawn from the joint distribution P (x, B). The goal is then to minimize the expected loss

L(f ) = E [L (f (x), B)] P (x,B)

, where f (•) is the model and L(•) is the loss for an f (•) that must produce a prediction within the budget B.

We conduct experiments on CIFAR-10. Our anytime search space consists of networks with 3 cells containing 24, 16, and 8 nodes. Each node is given the additional properties: 1) the spatial size it operates at and 2) if an early-exit classifier is attached to it. A node enforces its spatial size by pooling or upsampling any input feature maps inputs that are of different scale. Note that while a naive one-shot model would triple its size to include three different parameter sets at three different scales, the GHN is negligibly affected by such a change. The GHN uses the area under the predicted accuracy-FLOPS curve as its selection criteria. The search space, contains various convolution and pooling operators. Training methodology of the final architectures are chosen to match Huang et al. (2018) and can be found in the Appendix.

Figure 3 shows a comparison with the various methods presented by Huang et al. (2018) . Our experiments show that the best searched architectures can outperform the current state-of-the-art human designed networks. We see the GHN is amenable to the changes proposed above, and can find efficient architectures with a random search when used with a strong search space.

5.3. PREDICTED PERFORMANCE CORRELATION (CIFAR-10):

In this section, we evaluate whether the parameters generated from GHN can be indicative of the final performance. Our metric is the correlation between the accuracy of a model with trained weights vs. GHN generated weights. We use a fixed set of 100 random architectures that have not been seen by the GHN during training, and we train them for 50 epochs to obtain our "ground-truth" accuracy, and finally compare with the accuracy obtained from GHN generated weights. We report the Pearson's R score on all 100 random architectures and the top 50 performing architectures (i.e. above average architectures). Since we are interested in searching for the best architecture, obtaining a higher correlation on top performing architectures is more meaningful.

To evaluate the effectiveness of GHN, we further consider two baselines: 1) training a network with SGD from scratch for a varying number of steps, and 2) our own implementation of the one- Pham et al. (2018) , where nodes store a set of shared parameters for each possible operation. Unlike GHN, which is compatible with varying number of nodes, the one-shot model must be trained with N = 17 nodes to match the evaluation. The GHN is trained with N = 7, T = 5 using forward-backward propagation. These GHN parameters are selected based on the results found in Section 5.4.

Table 4 shows performance correlation and search cost of SGD, the one-shot model, and our GHN. Note that GHN clearly outperforms the one-shot model, showing the effectiveness of dynamically predicting parameters based on graph topology. While it takes 1000 SGD steps to surpasses GHN in the "Random-100" setting, GHN is still the strongest in the "Top-50" setting, which is more important for architecture search. Moreover, compared to GHN, running 1000 SGD steps for every random architecture is over 1000 times more computationally expensive. In contrast, GHN only requires a pre-training stage of 6 hours, and afterwards, the trained GHN can be used to efficiently evaluate a massive number of random architectures of different sizes.

5.4. ABLATION STUDIES (CIFAR-10):

Number of graph nodes: The GHN is compatible with varying number of nodes -graphs used in training need not be the same size as the graphs used for evaluation. Figure 5a shows how GHN performance varies as a function of the number of nodes employed during training -fewer nodes generally produces better performance. While the GHN has difficulty learning on larger graphs, likely due to the vanishing gradient problem, it can generalize well from just learning on smaller graphs. Note that all GHNs are tested with the full graph size (N = 17 nodes).

Number of propagation steps::

We now compare the forward-backward propagation scheme with the regular synchronous propagation scheme. Note that T = 1 synchronous step corresponds to one full forward-backward phase. As shown in Figure 5b , the forward-backward scheme consistently outperforms the synchronous scheme. More propagation steps also help improving the performance, with a diminishing return. While the forward-backward scheme is less amenable to acceleration from parallelization due to its sequential nature, it is possible to parallelize the evaluation phase across multiple GHNs when testing the fitness of candidate architectures. Stacked GHN for architectural motifs: We also evaluate different design choices of GHNs on representing architectural motifs. We compare 1) individual GHNs, each predicting one block independently, 2) a stacked GHN where individual GHN's pass on their graph embedding without sharing parameters, 3) a stacked GHN with shared parameters (our proposed approach). As shown in Table 5 , passing messages between GHN's is crucial, and sharing parameters produces better performance.

6. CONCLUSION:

In this work, we propose the Graph HyperNetwork (GHN), a composition of graph neural networks and hypernetworks that generates the weights of any architecture by operating directly on their computation graph representation. We demonstrate a strong correlation between the performance with the generated weights and the fully-trained weights. Using our GHN to form a surrogate search signal, we achieve competitive results on CIFAR-10 and ImageNet mobile with nearly 10× faster speed compared to other random search methods. Furthermore, we show that our proposed method can be extended to outperform the best human-designed architectures in setting of anytime prediction, greatly reducing the computation cost of real-time neural networks.

7. APPENDIX:



7.1. SEARCH SPACE:

Standard image classification on CIFAR-10 and ImageNet The search space for CIFAR-10 and ImageNet classification experiments includes the following operations:

• identity

• 1 × 1 convolution • 3 × 3 separable convolution • 5 × 5 separable convolution • 3 × 3 dilated separable convolution • 5 × 5 dilated separable convolution • 1 × 7 convolution followed 7 × 1 convolution • 3 × 3 max pooling • 3 × 3 average pooling

A block forms an output by concatenating all leaf nodes in the graph. Blocks have 2 input nodes which ingest the output of block i -1 and block i -2 respectively. The input nodes are bottleneck layers, and can reduce the spatial size by using stride 2.

Note that while ENAS supports only 5 operators due to memory constraints, GHNs can search for more operators. This is because ENAS (and other methods which use one-shot models) must store all the parameters in memory because it finds paths in a larger model. Thus the memory requirements are O(KN ) where K is the number of operations and N is the number of nodes in the candidate architecture. In contrast, the memory requirement for GHNs is O(N ) + O(K) for the candidate architecture and GHN respectively.

Anytime prediction on CIFAR-10 The search space for the CIFAR-10 anytime prediction experiments includes the following operations:

• 1 × 1 convolution

• 3 × 3 convolution • 5 × 5 convolution • 3 × 3 max pooling • 3 × 3 average pooling

In the anytime setting, nodes concatenate their inputs rather than sum. Thus, the identity operator was removed as it would be redundant. The search space does not include separable convolutions so that it is comparable with our baselines (Huang et al., 2018) Standard image classification on CIFAR-10 and ImageNet While node embeddings are initialized to a one-hot vector representing computational operator of the node, we found it helpful to pass the sparse vector through a learned embedding matrix prior to graph propagation. The GHN is trained for 200 epochs with batch size 64 using the ADAM optimizer with an initial learning rate 1e-3 that is divided by 2 at epoch 100 and 150. A naive hypernet would have a separate output branch for each possible node type, and simply ignore branches that aren't applicable to the specific node. In this manner, the number of parameters of the hypernetwork scale according to the number of possible node computations. In contrast, the number of parameters for a one-shot model scale according to the number of nodes in the graph. We further reduce number of parameters by obtaining smaller sized convolutions kernels through the slicing of larger sized kernels. 

:



Figures and Tables:

Figure 1: Our system diagram. A: A neural network architecture is randomly sampled, forming a GHN. B: After graph propagation, each node in the GHN generates its own weight parameters. C:The GHN is trained to minimize the training loss of the sampled network with the generated weights. Random networks are ranked according to their performance using GHN generated weights.

Figure 2: Stacked GHN along the depth dimension.

Figure 3: Comparison with state-of-the-art human-designed networks on CIFAR-10.

Figure5: GHN when varying the number of nodes and propagation scheme shot model proposed byPham et al. (2018), where nodes store a set of shared parameters for each possible operation. Unlike GHN, which is compatible with varying number of nodes, the one-shot model must be trained with N = 17 nodes to match the evaluation. The GHN is trained with N = 7, T = 5 using forward-backward propagation. These GHN parameters are selected based on the results found in Section 5.4.

Figure 7: Best block found for classification

Comparison with image classifiers found by state-of-the-art NAS methods which employ a random search on CIFAR-10. Results shown are mean ± standard deviation.

Table content: <table><tr><td>Method SMASHv1 (Brock et al., 2018) SMASHv2 (Brock et al., 2018) One-Shot Top (F=32) (Bender et al., 2018) One-Shot Top (F=64) (Bender et al., 2018)</td><td colspan="2">Search Cost (GPU days) Param ×10 6 ? 4.6 3 16.0 4 2.7 ± 0.3 4 10.4 ± 1.0</td><td>Accuracy 94.5 96.0 95.5 ± 0.1 95.9 ± 0.2</td></tr><tr><td>Random (F=32) GHN Top (F=32)</td><td>-0.42</td><td>4.6 ± 0.6 5.1 ± 0.6</td><td>94.6 ± 0.3 95.7 ± 0.1</td></tr><tr><td colspan="2">4.4 FORWARD-BACKWARD GNN MESSAGE PASSING</td><td/><td/></tr></table>

Comparison with image classifiers found by state-of-the-art NAS methods which employ advanced search methods on CIFAR-10. Results shown are mean ± standard deviation.

Table content: <table><tr><td>Method NASNet-A (Zoph et al., 2018) ENAS Cell search (Pham et al., 2018) DARTS (first order) (Liu et al., 2018c) DARTS (second order) (Liu et al., 2018c)</td><td colspan="2">Search Cost (GPU days) Param ×10 6 1800 3.3 0.45 4.6 1.5 2.9 4 3.4</td><td>Accuracy 97.35 97.11 97.06 97.17 ± 0.06</td></tr><tr><td>GHN Top-Best, 1K (F=32)</td><td>0.84</td><td>5.7</td><td>97.16 ± 0.07</td></tr></table>

Comparison with image classifiers found by state-of-the-art NAS methods which employ advanced search methods on ImageNet-Mobile.

Table content: <table><tr><td>Method NASNet-A (Zoph et al., 2018) NASNet-C (Zoph et al., 2018) AmoebaNet-A (Real et al., 2018) AmoebaNet-C (Real et al., 2018) PNAS (Liu et al., 2018a) DARTS (second order) (Liu et al., 2018c)</td><td colspan="3">Search Cost Param FLOPs (GPU days) ×10 6 ×10 6 1800 5.3 564 1800 4.9 558 3150 5.1 555 3150 6.4 570 225 5.1 588 4 4.9 595</td><td colspan="2">Accuracy Top 1 Top 5 74.0 91.6 72.5 91.0 74.5 92.0 75.7 92.4 74.2 91.9 73.1 91.0</td></tr><tr><td>GHN Top-Best, 1K</td><td>0.84</td><td>6.1</td><td>569</td><td>73.0</td><td>91.3</td></tr></table>

Table 3 the transferred block is competitive with other NAS methods which require a far greater search cost.

Table content: <table><tr><td/><td>96</td><td/><td/><td/></tr><tr><td/><td>95</td><td/><td/><td/></tr><tr><td/><td>94</td><td/><td/><td/></tr><tr><td>accuracy (%)</td><td>0.0 87 88 89 90 91 92 93</td><td>0.2</td><td>0.4 budget (in MUL-ADD) x 10e8 0.6 0.8 1.0 MSDNet Densenet MC DenseNets Ensemble 1.2 of Varying Depths GHN Best (ours)</td><td>1.4</td></tr></table>

Benchmarking the correlation between the predicted and true performance of the GHN against SGD and a one-shot model baselines. Results are on CIFAR-10.

Table content: <table><tr><td>Method SGD 10 Steps SGD 100 Steps SGD 200 Steps SGD 1000 Steps One-Shot</td><td colspan="4">Computation cost Initial (GPU hours) Per arch. (GPU seconds) Random-100 Top-50 Correlation -0.9 0.26 -0.05 -9 0.59 0.06 -18 0.62 0.20 -90 0.77 0.26 9.8 0.06 0.58 0.31</td></tr><tr><td>GHN</td><td>6.1</td><td>0.08</td><td>0.68</td><td>0.48</td></tr></table>



Table content: <table><tr><td>: Stacked GHN Correlation. SP denotes sharing parameters and PE denotes passing embeddings</td></tr></table>

. Block 1 contains nodes which may operate on any of the 3 scales (32 × 32, 16 × 16, 8 × 8). Block 2 contains nodes which can only operate on scales 16 × 16 and 8 × 8. Block 3 only contains nodes which operate on the scale 8 × 8. We fix the number of exit nodes. These choices are inspired byHuang et al. (2018) 7.2 GRAPH HYPERNETWORK DETAILS

Table content: <table/>

Peer review (iclr2019):
 This paper proposes to accelerate architecture search by replacing the expensive inner loop (wherein candidate architectures are trained to completion) with a HyperNetwork which predicts the weights of candidate architectures, as in SMASH. Contrary to SMASH, this work employs a Graph neural network to allow for the use of any feedforward architecture, enabling fast architecture search through parameter prediction using highly performant search spaces. The authors test their system and show that performance using Graph HyperNet-generated weights correlates with performance when trained normally. The authors benchmark their method against competing approaches ("traditional" NAS techniques which incur the full expense of the inner loop, and one-shot techniques which learn a large model then select architectures by searching for paths in said model) and show competitive performance. This is a solid technical contribution with a well-designed set of experiments. While the novelty is not especially high, the paper does a good job of synthesizing existing tools and achieves reasonably strong results with much less compute, making for a strong entry into the growing table of fast architecture search methods. I argue in favor of acceptance. Notes: -Whereas SMASH is limited to architectures which can be described with its proposed encoding scheme, GHNs only requires that the architecture be represented as a graph (which, to my knowledge, means it can handle any feedforward architecture). -Section 4.2: It*s not entirely clear how this setup allows for variable sized kernels or variable #channels. Is the output of H simply as large as the largest allowable parameter tensor, and sliced as necessary? A snippet of code might be more illuminating here than a set of equations. Additionally (I may have missed this in the text) is the #channels in each node held fixed with a predfined pattern, or also searched for? Are the channels for each node within a block allowed to vary relative to one another? -Do you sample a new, random architecture at every SGD step during training of the GHN? -I have no expertise in graph neural networks, and I cannot judge the efficacy of this scheme wrt other GNN techniques, nor can I judge the forward-backward message passing scheme of section 4.4. If another reviewer has expertise in this area and can provide an evaluation that would be great. -GPU-days is an okay metric, but it*s also problematic, since it will of course depend on the choice of GPU (e.g. you can achieve a 10x speedup just from switching from a 600-series to a V100! How does using 4 GPUS for 1 hour compare to 1 GPU for 4 hours? How does this change if you have more CPU power and can load data faster? What if you*re using a DL framework which is faster than your competitor*s?) Given that the difference here is an order of magnitude, I don*t think it matters, but if authors begin to optimize for GPU-milliseconds then it will need to be better standardized. -Further empirical evidence showing the correlation between approximate performance and true performance is also strong. I very much like that this study has been run for a method based on finding paths in a larger model (ENAS) and shows that ENAS* performance does indeed correlate with true performance, *but* not perfectly, something which (if I recall correctly) is not addressed in the original paper. -It is worth noting that for ImageNet-Mobile and CIFAR-10 they perform on par with the top methods but tend to use more parameters. -I like figures 3 and 4, the comparisons against MSDNet and random networks as a function of op budget is good to see. -Table 4 shows that the correlation is weaker (regardless of method) for the top architectures, which I don*t find surprising as I would expect the variation in performance amongst top architectures to be lower. It would be interesting to also see what the range of error rates are; I would expect that the correlation is higher when the range of error rates across the population of architectures is large, as it is easier to distinguish very bad architectures from very good architectures. Distinguishing among a set of good-to-very-good architectures is likely to be more difficult. -For Section 5.3, I found the choice to use unseen architectures a little bit confusing. I think that even for this study, there*s no reason to use a held-out set, as we seek to scrutinize the ability of the system to approximate performance even with architectures it *does* see during training. -How much does the accuracy drop when using GHN weights? I would like to see a plot showing true accuracy vs. accuracy with GHN weights for the random-100 networks, as using approximations like this typically results in the approximated weights being substantially worse. I am curious to see just how much of a drop there is. -Section 5.4: it*s interesting that performance is stronger when the GHN only sees a few (7) nodes during training, even though it sees 17 nodes during testing. I would expect that the best performance is attained with training-testing parity. Again, as I do not have any expertise in graph neural networks, I*m not sure if this is common (to train on smaller graphs and generalize to larger ones), so if the authors or another reviewer would like to comment and further illuminate this behavior, that would be helpful. Some typos: Abstract: "prematured" should be "premature" Introducton, last paragraph: "CNN networks." CNN already stands for Convolutional Neural Network.