Parameter Prediction for Unseen Deep Architectures
Authors: Boris Knyazev
Michal Drozdzal
Graham W. Taylor
Adriana Romero
Venue: NeurIPS 2021 Poster (2021)
Keywords: parameter prediction
hypernetworks
graph networks
computational graphs
optimization
meta-learning
deep networks

Abstract: Deep learning has been successful in automating the design of features in machine learning pipelines. However, the algorithms optimizing neural network parameters remain largely hand-designed and computationally inefficient. We study if we can use deep learning to directly predict these parameters by exploiting the past knowledge of training other networks. We introduce a large-scale dataset of diverse computational graphs of neural architectures - DeepNets-1M - and use it to explore parameter prediction on CIFAR-10 and ImageNet. By leveraging advances in graph neural networks, we propose a hypernetwork that can predict performant parameters in a single forward pass taking a fraction of a second, even on a CPU. The proposed model achieves surprisingly good performance on unseen and diverse networks. For example, it is able to predict all 24 million parameters of a ResNet-50 achieving a 60% accuracy on CIFAR-10. On ImageNet, top-5 accuracy of some of our networks approaches 50%. Our task along with the model and results can potentially lead to a new, more computationally efficient paradigm of training networks. Our model also learns a strong representation of neural architectures enabling their analysis.

Parameter Prediction for Unseen Deep Architectures
Boris Knyazev 1.2- Michal Drozdzal
Graham W. Taylor 22331t
Adriana Romero-Soriano 4.5,f
University of Guelph
Vector Institute for Artificial Intelligence
3 Canada CIFAR AI Chair
Facebook AI Research
5 McGGill University
Tequal advising
Abstract
Deep learning has been successful in automating the design of features in machine learning pipelines. However, the algorithms optimizing neural network parameters remain largely hand-designed and computationally inefficient. We study if we can use deep learning to directly predict these parameters by exploiting the past knowl- edge of training other networks. We introduce a large-scale dataset of diverse com- putational graphs neural architectures DEEPNETS- and use it to explore parameter prediction CIFAR- 10 and ImageNet. By leveraging advances in graph neural networks, we propose a hypernetwork that can predict performant parameters in single forward pass taking fraction of second, model achieves surprisingly good performance on unseen and diverse networks. For example, is able predict all 24 million parameters of ResNet-50 achieving 60% accuracy on CIFAR-IO. On ImageNet, top-5 accuracy of some of our networks approaches 50%. Our task along with the model and results can potentially lead a new, more computationally efficient paradigm of training networks. Our model also learns strong representation of neural architectures enabling their analysis.
CPU. The proposed
Introduction
Consider the problem of training deep neural networks on large annotated datasets, such as ImageNet [ This problem can formalized as finding optimal parameters for a given neural network a, parameterized by w.r.. a loss function L on the dataset D = {(x, w:)la of inputs X; and targets y::
Σ [(S(x;a w),
arg min
W
where f(x;; a, w) represents forward pass. Equation optimization algorithms - e.g. SGD (2) and Adam K that converge to performant parameters Wp of the architecture a. Despite the progress in improving the training speed and convergence (4H7), obtaining w, remains a bottleneck in large-scale machine learning pipelines. For example, training ResNet-50 [ on ImageNet can take many GPU hours
1 usually minimized by iterative
9 With the ever growing size of net- works [10] and necessity of training the networks repeatedly (e.g. for hyperparameter or architecture search), the classical process of obtaining W, is becoming computationally unsustainable [I 13].
new parameter prediction task. When optimizing the parameters for a new architecture a, typical optimizers disregard past experience gained by optimizing other However, leveraging this past experience can be the key to reduce the reliance on iterative optimization and, hence the high computational demands. To progress in that direction, propose task where iterative optimization is replaced with a single forward pass of a hypernetwork [[4] Hp. To tackle the task,
35th Conference Neural Information Processing Systems (NeurIPS 2021).
Part of the work done while interning at Facebook Al Research.

---

Hd is expected to leverage the knowledge of how to optimize other networks F. Formally, the task to predict the parameters of an unseen architecture a & F using H. parameterized by 8p: Wp = HD (a; pp) The task is constrained to a dataset 'D2 so Wp are the predicted parameters for which the test set performance of [(x; a, wp) is similar to the one of f(x; a, Wp). For example, we consider CIFAR-10 [T5] and ImageNet image classification datasets D, where the test set performance is classification accuracy on test images.
Approaching our task. A straightforward approach to expose HD to the knowledge of how to optimize other networks to train it on large training set of {(a,, Wp.s)) pairs, however, that is prohibitive Instead, we follow the bi-level optimization paradigm common in meta-learning 116 but rather than iterating over M tasks, we iterate over M training architectures F = {ai)==
arg min
Σ
i=1 ((1()((()))))) (x):0s HD(a;:)),
8
By optimizing Equation 2 the hypernetwork HD gradually gains knowledge of how to predict performant parameters for training architectures. It can then leverage this knowledge at test time when predicting parameters for unseen architectures. To approach the problem in Equation 2 we need to design the network space F and Hp. For F, we rely on the previous design spaces for neural architectures [19] that we extend in two ways: the ability to sample distinct architectures and an expanded design space that includes diverse architectures, such as ResNets and Visual Transformers [20]. Such architectures can be fully described in the form of computational graphs (Fig. D. So, to design the hypernetwork Ho, we rely on recent advances in machine learning on graph-structured data [21[24]] In particular, we build on the Graph HyperNetworks method (GHNs) [24] that also optimizes Equation? 2 However, GHNs do not aim to predict large-scale performant parameters as we do in this work, which motivates us to improve on their approach.
By designing our diverse space F and improving on GHNs, we boost the accuracy achieved by the predicted parameters on unseen architectures to 77% (top-1) and 48% (top-5) on CIFAR- 10 [I5] and ImageNet [.. respectively. Surprisingly, our GHN shows good ontf---disttitttion generalization and predicts good parameters for architectures that are much larger and deeper compared to the ones seen in training. For example, we can predict all 24 million parameters of ResNet-50 less than a second either on GPU or CPU achieving 60% on CIFAR-10 10 without any gradient updates (Fig 1 (b)).
Overall, framework and results pave the road toward new and significantly more efficient paradigm for training networks. Our contributions are as follows: (a) we introduce the novel task of predicting performant parameters for diverse feedforward neural networks with single hypemetwork forward pass; (b) we introduce DEEPNETS- M a standardized benchmark with in-distribution and -it--ii-ittiittionn architectures to track progress on the task ($ 3 (c) we define several baselines and propose GHN model ($4) that performs surprisingly well on CIFAR- 10 and ImageNet ($ 5511)
(d) we show that our model learns a strong representation of neural network architectures ($ 5.2)
and our model is useful for initializing neural networks 5.3). Our DEEPNETS- M dataset, trained
GHNs and code is available at https:: //gi wttpttt...oom..ccooeeeeeeearrraa
Example of evaluating on unseen architecture & F
Meta-bntching
Virtual edges
Parameter normalization
Images D
db
~
HD
(ResNei-50)
GatedCAN
Compurational graph
D Foraad pass
Bestowrd piss
Lose L
1.apass 1 pass
(b) Figure 1: (a) Overview of our GHN model ($ trained by backpropagation through the predicted parameters (Wr) on a given image dataset and our DEEPNETS- 1M dataset of architectures. Colored captions show our key improvements to vanilla GHNs ($ 2.2 The red one is used only during training GHNs, while the blue ones are used both at training and testing time. The computational graph of a is visualized as described in T (b) Comparing classification accuracies when all the parameters of ResNer-50 are predicted by GHNs versus when its parameters are trained with SGD (see full results in $5)..
(a)
Training single network a; take several GPU days and thousands of trained networks may be required.

---

2 Background
We start by providing a brief background about the network design spaces leveraged in the creation of our DEEPNETS-IN M dataset of neural architectures described in $ $3 We then cover elements of graph hypernetworks that we leverage when designing our specific GHN Ho in $
2.1 Network Design Space of DARTS
DARTS 19 is a differentiable NAS framework. For image classification tasks such as those considered in this work, its networks are defined by four types of building blocks: stems, normal cells, reduction cells, and classification heads. Stems are fixed blocks of convolutional operations that process input images. The normal and reduction cells the main blocks of architectures and are composed of: 3x3 and 5x5 separable convolutions, 3x3 and 5x5 dilated separable convolutions, x33 max pooling. xx3 average pooling, identity and zero (to indicate the absence of connectivity between two operations). Finally, the classification head defines the network output and is built with a global pooling followed by a single fully connected layer.
Typically, DARTS networks have one stem block, 14-20 cells, and one classification head, altogether forming deep computational graph. The reduction cells, placed only at 1/3 and 2/3 of the total depth, decrease the spatial resolution and increase the channel dimensionality by a factor of 2. Summation and concatenation are used aggregate outputs from multiple operations within each cell. To make the channel dimensionalities match, 1x convolutions are used as needed. All convolutional operations use the ReLU-Conv-Biie conv-Batch Norm (BN) 77 order. Overall, DARTS enables defining strong architectures that combine many principles of manual (25) 8 26 27] and automatic (24] 23 design of neural architectures. While DARTS learns the optimal task-specific cells, the framework can be modified to permit sampling randomly-structured cells. We leverage this possibility for the DEEPNETS- M construction in Please see $ A. for further details on DARTS.
2.2 Graph HyperNetwork: GHI-1
Representation of architectures. GHNs 24 directly operate the computational graph of neural architecture a. Specifically, a is a directed acyclic graph (DAG), where nodes V (u, | operations (e.g. convolutions, fully-connected layers, summations, etc.) and their connectivity is described by binary adjacency matrix A E 10, VIX[VI, Nodes are further characterized by matrix of initial node features H = [h. ; h ..., hiyll. where each h² a one-hot vector representing the operation performed by the node. We also use such one-hot representation for H but addition encode the shape of parameters associated with nodes as described in detail in $ B.I
Design of the graph hypernetwork. In [24], the graph hypernetwork Hp consists of three key modules. The first module takes the input node features HD and transforms them into d'dimennsinal node features H RIV[xd through an embedding layer. The second module takes H together with A and feeds them into a specific variant of the gated graph neural network (GatedGNN) (4). In particular, their GatedGNN mimics the canonical order æ of node execution in the forward (fw) and backward (bw) passes through computational graph. To do so, it sequentially traverses the graph and performs iterative message passing operations and node feature updates as follows:
Vt E [1, VF E [fw,bw] bw] (Ve E : m = Σ MLP(h()) h. GRU(h ] GRU( (3)
WENT
where T denotes the total number of forward-backwart passes; h² corresponds the features of node v in the (-th graph traversal; MLP(-) )) is multi-layer perceptron; and GRU(-) is the update function of the Gated Recurrent Unit 35 In the forward propagation (I fw), N/ corresponds to the incoming neighbors of the node defined by A, then in the backward propagation (х bw) it similarly corresponds to the outgoing neighbors of the node. The last module uses the GatedGNN output hidden states h. to condition a decoder that produces the parameters w (c.g. convolutional weights) associated with each node. In practice, to handle different parameter dimensionalities per operation type, the output of the hypernetwork is reshaped and sliced according to the shape of parameters in each node. We refer to the model described above as GHN-1 (Fig. I. Further subtleties of implementing this model in the context of our task are discussed in $ B.
3

---

Table 1: Examples of computational graphs (visualized using NetworkX (44)) in each split and their key statistics, to which we add the average degree and average shortest path length often used to measure local and global graph properties respectively [45] 46]. the visualized graphs, a node is one of the 15 primitives coded with markers shown at the bottom, where they are sorted by the frequency in the training set. For visualization purposes, a blue triangle marker differentiates xx1 convolution (equivalent to fully-connected layer over channels) from other convolutions, but its primitive type still just convolution. CComputed based CIFAR-10 10.
IN- IN-DISTRUE DISTRIBUTION
OUT-OF- OUT-OF-DISTRIBUTIO DISTRIBUTION
TRAIN VAL/TEST
WIDE
DEEP
DENSE
BN-FREE
RESNET/VIT
Agraphs
10"
500/500
100
100
100
100
1/1
Hoells
4-18
4-18
4-18
10-36
4-18
4-18
16/12
Nchannels
16-128
32-128
128-1216
32-208
32-240
32:336
64/128
Anodes ([V)
21-827
33-579
33-579
74-1017
57-993
33-503
161/114
w/o BN
3.5%%
0%/100%
#params(M)*
0.01-3.1
2,5-35
39.101
2.5-15.3
2.5-8.8
2.5-7.7
23.5/1. OO
avg dogree
2.3---0.1
2.3--0.11
2.3--0.11
2.3+0.11
2.440.11
2.4+0.1
2.2/2.3
poth
14.3+=4.8
14.5#:4.9
14.7=
26.2#99.3
15.1=4.1
10.0=288
11.2/10.7
marker
primitive
CODV
BN
sum
bias
group
concat dilated
LN
MaX
avg
MSA
SE
input
glob
pos
COBV
pool
pool
avg
enc
fraction in TRAIN (%)
36.3
25.5
11.1
6.5
5.1
3.8
2.5
2.5
1.8
1.7
1.2
1.0
0.5
0.5
0.2
DEEPNETS-IM
The network design space of DARTS is limited by the number of unique operations that compose cells, and the low variety of stems and classification heads. Thus, many architectures are not realizable within this design space, including: VGG (25), ResNets (8). MobileNet [33 or more recent ones such as Visual Transformer (ViT) [20] and Normalization-fiee networks [6 37). Furthermore, DARTS does not define a procedure to sample random architectures. By addressing these two limitations we aim to expose hypernetwork to diverse training architectures and permit its evaluation on common architectures, such as ResNet-50. We hypothesize that increased training diversity can improve hyper- networks' generalization unseen architectures making it more competitive to iterative optimizers.
Extending the network design space. We extend the set of possible operations with non-separable 2D convolution Squecze&Excite (SE) [4)] and Transformer-based operations 41 20 multihead self-attention (MSA), positional encoding and layer norm (LN) 12]. Each node (operation) in our graphs has attributes: primitive rype (e.g. convolution) and shape (e.g. 3x3x512x511 Overall, our extended set consists of 15 primitive types (Table [T]. We also extend the diversity of the generated architectures by introducing VGG-style classification heads and ViT stems. Finally, to further increase architectural diversity, we allow the operations to not include batch norm (BN) and permit networks without channel width expansion (e.g. as in 20
Architecture generation process. We generate different subsets of architectures (see the description of each subset in the next two paragraphs and in Table For each subset depending its purpose, we predefine a range possible model depths (number of cells), widths and number of nodes per cell. Then, we sample a stem, normal and reduction cell and a classification head. The internal structure of the normal and reduction cells is defined by uniformly sampling from all available operations. Due to a diverse design space is extremely unlikely to sample the same architecture multiple times, but a sanity check using the Hungarian algorithm [3 to confirm that (see Figure 6 in $ A.2 for details).
In-distribution (ID) architectures. We generate training set of | 106 architectures and
validation/test sets of 500/500 architectures that follow the same generation rules and are considered to be ID samples. However, training on large architectures can be prohibitive, e.g. in terms of GPU memory. Thus, in the training set we allow the number of channels and, hence the total number of parameters, to be stochastically defined given computational resources. For example, to train
Sepasabe convolutions have weights of e.g. shape 3x3x512x512 in ResNet-50. NAS works, such DARTS GHN, avoid such convolutions, since the separable ones [38 are more efficient. Non-separable convolutions are nevertheless common in practice and can often boost the downstream performance.
'The & operation common in many efficient networks 39 12]

---

our models we upper bound the number of parameters in the training architectures to around 3M by sampling fewer channels if necessary. In the evaluation sets, the number of channels fixed. Therefore, this pre-processing step prior to training results in some distribution shift between the training and the validation/test sets, However, the shift not imposed by our dataset.
(OOD) architectures. We generate five OOD test sets that follow different generation rules. In partieular, we define WIDE and DEEP sets that are of interest due the stronger downstream performance of such nets in large-scale tasks 47
48 10] These nets are often more challenging to train for fundamental 44 50 or computational 51 reasons, so predicting their
parameters might ease their subsequent optimization. We also define the DENSE set, since networks with many operations per cell and complex connectivity are underexplored in the literature despite their potential [27]. Next, we define the BN-FREE set that is of interest due to BN's potential negative
side-effects [52 53 and the difficulty or unnecessity of using it in some cases [54) 36 37 We
finally add the RESNET/VIT set with two predefined image classification architectures: commonly-
used ResNet-50 X and smaller 12-layer version of the Visual Transformer (ViT) 20 that has
recently received lot of attention in the vision community. Please see $ A. and A.2 for further
details and statistics of our DEEPNETS- MM dataset.
4 Improved Graph HyperNetworks: GHN-2
In this section, we introduce our three key improvements to the baseline GHN-1 deseribed in $ 2.2 (Fig. [T. These components are essential to predict stronger parameters our task. For the empirical validation of the effectiveness of these components see ablation studies in $ [51 and C.2.1
4.1 Differentiable Normalization of Predicted Parameters
When training the parameters of a given network from seratch using iterative optimization methods, the initialization of pa- rameters is crucial. A common approach is to use He [57] or
Table 2: Parameter normalizations.
Type of node v
Normalization
Glorot (58) initialization to stabilize the variance of activa-
Conv fouv//fullyccon. Norm. weights Biases
< g/lCi,. (WW 2 sigmoid( (TTTT) Lanb( ('T)
tions across layers of the network. Chang et al. [59] showed that when the parameters of the network are instead predicted by a hypernetwork, the activations in the network tend to
explode or vanish. To address the issue of unstable network activations especially for the case of pre- dicting parameters of diverse architectures, we apply operation-dependen normalizations (Table 2
We normalize convolutional and fully-connected weights by following the fan-in scheme of [57]
(see the comparison to fan-out in $ C.2.1 w / /B/(C HWW), where Can. H., W are the number of input channels and spatial dimensions of weights wp respectively; and 3 is a nonlinearity specific constant following the analysis in [57]. The parameters of normalization layers such as BN and LN, as well as biases typically initialized with constants, are normalized by applying a squashing function with temperature 7 to imitate the empirical distributions of models trained with SGD (see Table 2 These are differentiable normalizations, so that they are applied at training (and testing) time. Further analysis of our normalization and its stabilizing effect on activations is presented in B.2.2
4.2 Enhancing Long-range Message Propagation
Computational graphs often take the form of long chains (Table I with only a few incoming/outeoming edges per node. This structure might hinder long-range propagation of information between nodes [60] Different approaches to alleviate the long-range propagation problem exist [61-633 including stacking GHNs in [24] Instead we adopt simple graph-based heuristics in line with recent works [64 65). In particular, we add virtual edges between two nodes and u and weight them based on the shortest path s.u between them (Fig. 2.. To avoid interference with the real edges in the computational graph, we introduce separate MLP,, sp to transform the features of the nodes connected through these virtual edges, and redefine the message passing of Equation as:
Figure 2: Virtual edges (in green) allow for better capture of global context.
m E.ENy MLP(h())
Σ ( Sen
where .g(e are neighbors satisfying 1 < Smu S s((ax) and s (max) is hyperparameter. To maintain the same number of trainable parameters as in GHN-1, we decrease MI.s'' sizes appropriately. Despite its simplicity, this approach is effective (see the comparison to stacking GHNs in $ C.2.1
5

---

4.3 Meta-batching Architectures During Training
GHN-1 1 updates its parameters 0 based on a single architecture sampled for each batch of images (Equation 2) In vanilla SGD training. larger batches of images often speed up convergence by reducing gradient noise and improve model's performance [66]. Therefore, we define meta-batch bm as the number of architectures sampled per batch of images. Both the parameter prediction and the forward/backward passes through the architectures in meta-batch can be done in parallel. We then average the gradients across bm to update the parameters 0 of Hp: VoL = 1 /0mm Σ VoL, Further analysis of the meta-hatching effect the training loss and convergence speed is presented in B.2.3
Experiments
We focus the evaluation of GHN-2 on our parameter prediction task ($ 5.T). addition, we show beneficial side-effects of i) learning stronger neural architecture representation using GHN-2 in analyzing networks ($ 5.2) and ii) predicting parameters for fine-tuning (.3)). We provide further
experimental and implementation details, as well as more results supporting our arguments in $
Datasets. We use the DEEPNETS- 1 M dataset of architectures ($ 3 as well as two image classification datasets D2 (CIFAR-10 (15) and 'D2 (ImageNet (I). CIFAR- TO consists of 50k training and 10k test images of size 32x32x3 and 10 object categories. ImageNet is larger scale dataset with 1.28M training and 50k test images of variable size and 1000 fine- ge-iird object categories. We resize Ima- geNet images to x24x222xx following 19 24]. We use 5k/50k training images as a validation set in CIFAR--00/mageNee and 500 validation architectures of DEEPNETS- 1M for hyperparameter tuning.
Our baselines include GHN-1 and a simple MLP that only has access to operations. but not to the connections between them. This MLP baseline is obtained by replacing the GatedGNN with an MLP in our GHN-2. Since GHNs were originally introduced for small architectures of ~ 50 nodes and only trained on CIFAR-10 we reimplement them and scale them up by introducing minor modifi- cations to their decoder that enable their training on ImageNet and on larger architectures of up to 1 1000 nodes (see B.I for details). We use the same hyperparameters to train the baselines and 2
Baselines.
Iterative optimizers. In the parameter prediction experiments. we also compare our model stan- dard optimization methods: SGD and Adam 3. We use off-the-shelf hyperparameters common in the literature 24 19 32
On CIFAR-10. 10, we train evaluation architectures with SGD/Adam, ini- tial learning rate 7 = 0.025 / n 0.001, batch size b = 96 and up to 50 epochs. With Adam, we train only 300 evaluation architectures as a rough estimation of an average performance. On ImageNet, we train them with SGD, 17 = 0.1 and 6 = 128, and, for computational reasons (given 1402 evaluation ar- chitectures in total), we limit training with SGD I epoch. We have also considered meta-optimizers. such as 17 18). However, we were unable to scale them to diverse and large architectures of our DEEPNETS- MM, since their LSTM requires a separate hidden state for every trainable parameter in the architecture. The scalable variants exist [70 71], but are hard to reproduce without open source code.
67-69 69
Additional experimental details. We follow 24 and train GHNs with Adam, 1/ = 0.001 and batch size of 64 images for CIFAR- 10 and 256 for ImageNet. We train for up to 300 epochs, except for one experiment in the ablation studies, where we train one GHN with bn. eight times longer, i.c. for 2400 epochs. All GHNs in our experiments use T = 1 propagation (Equation 3). as we found the original T 5 of (24] to be inefficient and did not improve the accuracies our task. GHN-2 uses s (max) 50 and by 8 and additionally uses LN that slightly further improves results (see these ablations in 8 $/C.2.1 p. Model selection is performed on the validation sets, but the results in our paper are reported on the test sets to enable their direct comparison.
5.1 Parameter Prediction
Experimental setup. We trained our GHN-2 and baselines on the training architectures and training images, i.e. separate model is trained for CIFAR-I0 and ImageNet. According to our DEEPNETS- 1M benchmark, we assess whether these models can generalize to unseen in-distribution (ID) and generalization by predicting parameters for the test architectures and computing their classification accuracies on the test images of CIFAR- 10 (Table 3 and ImageNet (Table 4. The evaluation architec- tures with batch norm (BN) have running statistics, which are not learned by gradient descent 7. and
(OOD) test architectures from our DEEPNETS- 1M. We measure this
While source code for GHNs 24 is unavailable, appreciate the authors' help in implementing some steps.

---

hence are not predicted by our GHNs. To alleviate that, we follow (24] and evaluate the networks with BN by computing per batch statisties with batch size of 64 images. This is further discussed in $ C.
Results. Despite GHN-2 2 never observed the test architectures, GHN-2 predicts good parameters for them making the test networks perform surprisingly well on both image datasets (Tables 3 and + Our results are especially strong on CIFAR-I0, 10, where some architectures with predicted parameters achieve up to 77.1%, while the best accuracy of training with SGD for 50 epochs is around 15% more. We even show good results on ImageNet, where for some architectures we achieve a top-5 accuracy of up to 48.3%. While these results are low for direct downstream applications, they are remarkable for three main reasons. First, to train GHNs by optimizing Equation we do not rely on the prohibitively expensive procedure of training the architectures F by SGD. Second, GHNs rely on single forward pass to predict all parameters. Third, these results are obtained for unseen architectures, including the OOD ones. Even in the case of severe distribution shifts (e.g. ResNet-5( and underrepresented networks (e.g. V.T our model still predicts parameters that perform better than random ones. On CIFAR-10, generalization of GHN-2 particularly strong with a 58.6% accuracy on ResNet-50.
On both image datasets, our GHN-2 significantly outperforms GHN-1 on all test subsets of DEEPNETS-IM with more than 20% absolute gain in certain cases, e.g. 36.8% vs 13.7% on the BN-FREE networks (Table 33. Exploiting the structure of computational graphs is critical property of GHNs with the accuracy dropping from 66. 66.9% to 42.2% on ID (and even more on OOD) architectures when we replace the GatedGNN of GHN-2 with an MLP. Compared to iterative optimization methods, GHN-2 predicts parameters achieving an accuracy similar 2500 and 5500 iterations of SGD on CIFAR-10 10 and ImageNet respectively. In contrast, GHN-1 performs similarly to only ~50 and 2000 (not shown in Table iterations respectively. Comparing SGD to Adam, the latter performs worse in general except for the ViT architectures similar to 722 20
To report speeds on ImageNet in Table we use a dedicated machine with a single NVIDIA V V100-32GB and Intel Xeon CPU E5-1620 v4 @ 3.50GHz. So for SGD these numbers can be reduced by using faster computing infrastructure and more optimal hyperparameters 77 Using our setup,
Table 3: CIFAR-10 results of predicted parameters for unseen ID and OOD architectures of DEEPNETS- 1M. Mean (istandard error of the mean) accuracies are reported (random chance =10%) Thee number of parameter updates.
METHOD
#upd
ID-TEST
OOD-TEST DENSE
avg
max
WIDE
DEEP
BN-FREE RESNET/VIT
MLP
1
42.2+0.6
60.2
22.3+0.9
37.9+12
44.8+1.1
23.9+0.7
17.7/10.0
GHN-1
51.4+104
59.9
43.1
48.3108
51.8109 60.5+112
13.720. 65.84077
19.2/18.2
GHN-2
1
66.9+0.3
77.1
64.0=1.1
36.8+15
58.6/11.4
Iterative optimizers (all architectures are ID) in this case)
SGD (1 epoch)
0.3x10
46.1
66.5
47.2+1.1
34.2+111
45.3+0.
18,0=11
61.8/34.5
SGD epochs)
2
69.2+0.
82.4
71.2+0.3
56.7+16
67.8+09
29.0=20
78.2/52.5
SGD (50 epochs)
2
88.5+003
93.1
88,911.2
84.5+11
87.3108
45.6136
93.5/75.7
Adam (50 epochs)
25x 10
84.0#08
89.5
82.0=1.6
76.2+26
84.8+04
38.8
91.5/79.4
Table 4: ImageNet results on DEEPNETS-IM. Mean (1standard error of the mean) top-5 accuracies are reported (random chance =0.5%). "Estimated on ResNet-50 with batch size 128.
METHOD
#upd GPU
CPU sec. avg
ID-TEST
OOD-TEST
avg
avg
max
WIDE
DEEP
DENSE BN-FREE RESNET/VIT
GHN-1
1 0.3
0.5
17.2+04 32.1
15.8=09 15.9408 15.1 0.5
6.9/0.9 5.3/4.4
GIIN-2
1
0.3
0.7
27.2:006 48.3
19,4=14 1.4 24.7 2.2 7.210.6
Iterative optimizers (all architectures are ID in this case)
SGD (1 step)
1
0.4
6.0
0.5=0.0 0.7 0.5+00 0.5+00 0.5-0.0
0.5+0.0
0.5/0.5
SGD (5000 steps) 5k
2x 10
3x104
25.6 50.7
26.2=1.4 13.2+1.1 25.4 4,8408
34.8/24.3
SGD (10000 steps) 10k
4x 10
6x
37.7 62.0
38.7=1.6 22.1+114 36.3+1. 8.0
49.0/33.4
SGD (100 epochs) 1000k
1
6x T0
92.9/72.2
'Large architectures with bottleneck layers such as ResNet-50 do not appear during training. AArciteccturss such as VIT do not include BN and, except for the first layer, convolutions tte two most freeuent operations in the training set.
7

---

SGD requires on average 10* x more time on a GPU (10%x on CPU) to obtain parameters that yield performance similar to GHN-2. As concrete example, AlexNet 74 requires around 50 GPU hours (on our setup) to achieve 81.8% top-5 accuracy, while some architectures we achieve 248.0% in just 0.3 GPU seconds.
Table 5:
Ablating GHN-2 on CIFAR-10. An
average rank of the model is computed across all ID and OOD test architectures.
Train sccuncy
80-
-50
D
40
MODEL
ID-TEST OOD-TEST AvG. RANK
70
30
(IB)
-20 2
BB
(1D)
GHN-2
66.92-0.3
56,8
1,9
1000 training architectures
65.1+00.5
52.51100
2.6
normulization ($4..
62.61:..6
47.11:12
3.9
10 107
10'
10"
10'
10*
No virtual edges (5
61.5.10.4
53.910.66
4.1
Training architecuurss
meta-hatch (h...
54.34.03
47.52:06
5.5
Figure 3: GHN-2 with meta batch by
bm train 8x longer
62.4.10.5
51911.
3.7
No GatedGNN (MIPP)
42.24106
32.2410.7
7.4
8 versus 6m = 1 for different numbers of training architectures on CIFAR-IO.
GHN-1
51.41104
39.210.99
6.8
Ablations (Table 5) show that all three components proposed in $ are important. Normalization is particularly important for OOD generalization with the largest drops on the WIDE and BN-FREE networks (see $ C.2.I Using meta-batching (bm = 8) is also essential and helps stabilize training and accelerate convergence (see B.2) We also confirm that the performance gap between b,, = and 6m = 8 is not primarily due to the observation of more architectures, since the ablated GHN-2 with 6m trained eight times longer is still inferior. The gap between 6m = 8 and 6m 1 becomes pronounced with at least 1k training architectures (Fig. 3). When training with fewer architectures (e.g. 100), the GHN with meta-batching starts to overfit to the training architectures. Given chal- lenging setup with unseen evaluation architectures, is surprising that using 1k training architectures already gives strong results. However, OOD generalization degrades in this case compared to using all IM architectures, especially on the BN- EREE networks (see $ $B.2). When training GHNs on just few architectures, the training accuracy soars the level of training them with SGD. With more architectures, it generally decreases indicating classic overfitting and underfitting cases.
5.2 Property Prediction
Representing computational graphs of neural architectures a challenging problem 77 79). We verify if GHNs are capable of doing that out-of-the-boo in the property prediction experiments. We also experiment with architecture comparison in $ C.2.4 Our hypothesis that by better solving our parameter prediction task, GHNs should also better solve graph representation tasks.
Experimental setup. We predict the properties of architectures given their graph embeddings
obtained by averaging node features 8 We consider four such properties (see $ C.2.3 for details):
Accuracy on the "clean" (original) validation set of images;
Accuracy on a corrupted set (obtained by adding the Gaussian noise to images following 53
Inference speed (latency or GPU seconds per a batch of images);
Convergence speed (the number of SGD iterations to achieve a certain training accuracy).
Estimating these properties accurately can have direct practical benefits. Clean and corrupted accu- racies can be used to search for the best performing architectures (e.g. for the NAS task); inference speed be used to choose the fastest network, so by estimating these properties we can trade-off ac- curate, robust and fast networks [12]. Convergence speed can be used to find networks that are easier optimize. These properties correlate poorly with each other and between CIFAR-10 and ImageNet ($ (.2.3). so they require the model to capture different regularities of graphs. While specialized meth- ods to estimate some of these properties exist, often as a NAS task 82 30 75), our GHNs provide a generic representation that can be easily used for many such properties. For each property, we train simple regression model using graph embeddings and ground truth property values. We use 500 validation architectures of DEEPNETS- 1 M for training the regression model and tuning its hyperpa- rameters (see S for details). We then use 500 testing architectures DEEPNETS- M to measure Kendall's 's Tau rank correlation between the predicted and ground truth property values similar to 80)
A tixed size graph embedding for the architecture can be computed by averaging the output node features: [V Evey hy where ha dd and di is the dimensionality of node features.
ha

---

Additional baseline. We compare to the Neural Predictor (NeuPred) [0] NeuPred is based on directed graph convolution and is developed for accuracy prediction achieving strong NAS results. We train a separate such NeuPred for each property from scratch following their hyperparameters.
Results. GHN-2 consistently outperforms the GHN-1 and MLP baselines as well as NeuPred (Fig. 4). In $ C.2.3] we also provide results veritying if higher correlations
Neutted GHN
0.5
translate to downstream gains. For example, on CIFAR-10 10 by choosing the most accurate architecture according to the regression model and training it from scratch following [19 (4), we obtained 97.26% (=n.99) accuracy, which
0.2
Gith-1
GHN-2
Glaan image accuracy
Nolty image aecurwcy
Inferincc speed
Convergencce speed
Figure 4: Property prediction of neural networks in terms of correlation (higher is better). Error bars denote the standard deviation across 5 runs.
is competitive with leading NAS approaches,
e.g. [19]
24 32
67 69. In contrast, the
network chosen by the regression model trained on the GHN-1 embeddings achieves 95.90%
5.3
Fine-tuning Predicted Parameters
Neural networks trained on ImageNet and other large datasets have proven useful in diverse visual tasks in the transfer learning setup [3 87 20). Therefore, we explore how predicting parameters on ImageNet with GHNs compares to pretraining them on ImageNet with SGD such a setup. We consider low-data tasks as they often benefit more from transfer learning 86 87
Experimental setup. We perform two transfer-learning experiments. The first experiment is fine- tuning the predicted parameters on 1,000 training samples (100 labels per class) of CIFAR-10. We fine-tune ResNet-550, Visual Transformer (ViT) and a 14-cell architecture based on the DARTS best cell [9]. The hyperparameters of fine-tuning (initial learning rate and weight decay) are tuned on 200 validation samples held-out of the 1,000 training samples. The number of epochs is fixed to 50 as in $ [. for simplicity. In the second experiment, we fine-tune the predicted parameters on the object detection task. We closely follow the experimental protocol and hyperparameters from (88] and train the networks on the Penn-Fudan dataset [89. The dataset contains only 170 images and the task is to detect pedestrians. Therefore this task is also well suited for transfer learning. Following [8, we replace the backbone of Faster R-CNN with one of the three architectures. To perform transfer learn- ing with GHNs, in both experiments we predict the parameters of a given architecture using GHNs trained on ImageNet. We then replace the ImageNet classification layer with the target task-specific layers and fine-tune the entire network on the target task. We compare the results of GHNs to He's initialization [57] and the initialization based on pretraining the parameters on ImageNet with SGD,
Table 6: CIFAR-10 test set accuracies and Penn-Fudan object detection average precision (at loU=0.50) after tuning the networks using SGD initialized with different methods. Average results and standard deviations for 3 runs with different random seeds are shown. For each architecture, similar GHN-2-based and ImageNet-basedd results are bolded. Estimated on ResNet-50.
INITIALIZATION
GPU sec.
100-SHOT CIFAR-10
PENN-FUDAN ORJECT DETECTION
METHOD
to init.'
RESNET-50
VIT
DARTS
RESNET-50
VIT
DARTS
He's 57
0.003
41.010.4
33221033
45.4-0.4
0.197:10042
0.14410.010
0.066-0.035
GHN-1 (trained on ImugeNet)
0.6
46.6::00
23.3:t0.1
49.2=0.1
0.433:100013
0.0::0.0
0.028
GHIN-2 (trained ImageNet)
0.7
56.410.11
41.4.0.66
0..7
0.509
0.436
0.785 0.032
ImageNet pretraining steps)
6x10
45.410.3
44.3.1..1
6.34-0.3
0.302.1..022
0.182.10.046
0.033
ImageNer (2.5k pretrnining steps)
1.5x 10"
55.410.2
50.4+1..3
7..4=0.2
0.571
0.322-0.073
0.023-0.022
ImageNet (5 pretraining epochs)
3x10"
84,6+0.2
70.2+:0.5
83.9=0.1
0.723++0.045
0.002
0.827=0.053
ImageNet (fiaal opoch)
6x10
89,2+0022
74.5:+0.22
85.6=0.2
0.876+000011
0.468=000023
0.881=0.023
Results. The CIFAR-10 image classification results of fine-tuning the parameters predicted by our GHN-2 are 2 10 percentage points better (in absolute terms) than fine-tuning the parameters predicted by GHN-1 or training the parameters initialized using He's method (Table/6). Similarly, the object de- tection results of GHN-2- basee initialization are consistently better than both GHN-1 and He's 's initial- izations. The GHN-2 results are a factor of 1.5-3 improvement over He's for all the three architectures. Overall, the two experiments clearly demonstrate the practical value of predicting parameters using our GHN-2. Using GHN-1 for initialization provides relatively small gains or hurts convergence (for ViT). Compared to pretraining on ImageNet with SGD, initialization using GHN-2 leads to performance sim-

---

ilar to 1k-2.5k steps of pretraining on ImageNet depending on the architecture in the case of CIFAR- -O0. In the case of Penn-Fudan, GHN-2's performance is similar to 2 lk steps of pretraining with SGD. In both experiments, pretraining on ImageNet for just 5 epochs provides strong transfer learning performance and the final ImageNet checkpoints are only slightly better, which aligns with previous works [85]. Therefore, further improvements in the parameter prediction models appear promising.
Related Work
Our proposed parameter prediction task, objective in Equation 2 and improved GHN are related to wide range of machine learning frameworks, in particular meta-learning and neural architecture search (NAS). Meta-learning is a general framework []6, [90] that includes meta-optimizers and meta-models, among others. Related NAS works include differentiable [19] and one-shot
methods 12]. See additional related work in $ D
Meta-optimizers. Meta-optimizers [17 18 91 [2] define problem similar to our task, but where HD is an RNN-based model predicting the gradients Vw, mimicking the behavior of iterative optimizers. Therefore, the objective of meta-optimizers may be phrased as learning to optimize as opposed to our learning to predict parameters. Such meta-optimizers can have their own hyperparameters that need to be tuned for given architecture a and need to be run expensively (on the GPU) for many iterations following Equation [
Meta-models Meta-models include methods based on MAML [93]. ProtoNets [94] and auxiliary nets predicting task-specific parameters [95/98] These methods are tied to a particular architecture and need to be trained from scratch if it is changed. Several recent methods attempt to relax the choice
of architecture in meta- meta-learning. T-NAS [99 combines MAML with DARTS 19 learn both the optimal architecture and its parameters for given task. However, the best network, a, needs to be trained using MAMI. from scratch. Meta-NAS [T00] takes step further and only requires fine-tuning of a on given task. However, the a is obtained from single meta-architecture and so its choice is limited, preventing parameter prediction for arbitrary a. CATCH [00] follows a similar idea, but uses reinforcement learning to quickly search for the best a on the specific task. Overall meta-learning mainly aims at generalization across tasks, often motivated by the few-shot learning problem. In contrast, our parameter prediction problem assumes single task (here an image dataset), but aims at generalization across architectures a with the ability to predict parameters in a single forward pass.
One-shot NAS. One-shot NAS aims to learn a single "supernet" [02 12 103 that can be used to estimate the performance of smaller nets (subnets) obtained by some kind of pruning the supernet, fol- lowed by training the best chosen a from scratch with SGD. Recent models, in particular BigNAS [12] and OnceFor (OFA) 102), eliminate the need to train subnets. However, the fundamental limita- tion of one-shot NAS is poor scaling with the number of possible computational operations (24). This limits the diversity of architectures for which parameters can be obtained. For example, all subnets in OFA are based on MobileNet-v3 [33), which does not allow to solve our more general parameter prediction task. To mitigate this, SMASH [04] proposed to predict some of the parameters using hypernetworks [14] by encoding architectures as 3D tensor. Graph HyperNetworks (GHNs) [24 further generalized this approach to "arbitrary" " computational graphs (DAGs). which allowed them to improve NAS results. GHNs focused on obtaining reliable subnetwork rankings for NAS and did not aim to predict large-scale performant parameters. We show that the vanilla GHNs perform poorly on our parameter prediction task mainly due to the inappropriate scale of predicted parameters, lack of long-range interactions in the graphs, gradient noise and slow convergence when optimizing Equation 2 Conventionally NAS, GHNs were also trained in quite constrained architecture
space 105
We expand the architecture space adopting GHNs for a more general problem.
7 Conclusion
We propose novel framework and benchmark to learn and evaluate neural parameter prediction models. Our model (GHN-2) is able to predict parameters for very diverse and large-scale architectures in a single forward pass in a fraction of second. The networks with predicted parameters yield surprisingly high image classification accuracy given the extremely challenging nature of our parameter prediction task. However, the accuracy is still far from networks trained with handcrafted optimization methods. Bridging the gap is a promising future direction. As a beneficial side-effect, GHN-2 learns a strong representation of neural architectures as evidenced by our property prediction evaluation. Finally, parameters predicted using GHN-2 trained on ImageNet benefit transfer learning in the low-data regime. This motivates further research towards solving our task.
10

---

Acknowledgments
BK is thankful to Facebook AI Research for funding the initial phase of this research during his internship and to NSERC and the Ontario Graduate Scholarship used to fund the other phases of this research. GWT and BK also acknowledge support from CIFAR and the Canada Foundation for Innovation. Resources used in preparing this research were provided, in part, by the Province of Ontario, the Govermment of Canada through CIFAR, and companies sponsoring the Vector Institute: htttp://wwz vectorinsts tute. ai/ htt/ppattners editorial help. We are thankful to the Vector AI Engineering team (Gerald Shen, Maria Koshkina and Deval Pandya) for code review. We are also thankful to the reviewers for their constructive feedback.
We are thankful to Magdalena Sobol for
References
[1] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211-22 1521122522222 2015.
[2] Schastian Ruder. An overview of gradient descent optimization algorithms. arXiv preprint arXiv:1609.04777, 2016.
[3] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1rXiv:1412.6980 2014.
[4] Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q Weinberger. Deep networks with stochastic depth. In European conference computer vision, pages 646-661. Springer, 2016.
[5] Andrew Brock, Theodore Lim, James M Ritchie, and Nick Weston. Freezeout: Accelerate training by progressively freezing layers. arXiv preprint arXiv: /0ki:::706.0000 04983, 2017.
[6] Dami Choi, Alexandre Passos, Christopher J Shallue, and George E Dahl. Faster neural network training with data echoing, arXiv preprint 100xiv:1907.05505 2019.
[7] Sergey Toffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arxiv: 2015.
[8] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[9] NVIDIA. Nvidia data center deep learning product performance. URI.httrp::/ddevelprr... nvidia com/ Seapp---errnnnn-eeeeeeee
teeeettraiin training- inference
[10] Tom B Brown, Benjamin Mann, Nick Ryder, Mclanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005. 2020.
[11] Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in nlp. arXiv preprint arXiv: 02243, 2019.
[12] Han Cai, Chuang Gan, Tianzhe Wang. Zhekai Zhang, and Song Han. Once-for-all: Train one network and specialize it for efficient deployment, 2019.
[13] Neil C Thompson, Kristjan Greenewald, Keehcon Lee, and Gabriel F Manso. The computational limits of deep learning. arXiv preprint urXiv:2007.05558. 2020.
[14] David Ha, Andrew Dai, and Quoe V Le. Hypernetworks. arXiv preprint arXiv: 2016.
[15] Alex Krizhevsky et al. Learning multiple layers of fcatures from tiny images. 2009.
[16] Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey. Meta-learning in neural networks: A survey. arXiv preprint arXiv:2004.054:3 2020.
[17] Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew W Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando De Freitas. Learning to learn by gradient descent by gradient descent. In Advances in neural information processing systems, pages 3981-3989, 2016.
[18] Sachin Ravi and Hugo Larochelle. Optimization model for few-shot learning. 2016.
11

---

[19] Hanxiao Liu, Karen Simonyan. and Yiming Yang. Darts: Differentiable architecture search. arXiv preprint arXiv:/806.00055 2018.
[20] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 11929,, 2020.
[21] Thomas N. Kipf and Max Welling. Semi-superrissee classification with graph convolutional networks. In Sth International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017. Conference Track Proceedings. OpenReview.net, 2017. URL http://///opeeeeeeeeeeeeeeeeeeee YJD4ayYg1
[22] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In International Conference on Learning Representations, 2018. URL
https://openrreieww.nr http://eiwe..tttttooooorrrrrrrod
[23] Vijay Prakash Dwivedi, Chaitanya K Joshi, Thomas Laurent, Yoshua Bengio, and Xavier Bresson. Benchmarking graph neural networks. arXiv preprint arXiv:2003.00982, 2020.
[24] Chris Zhang. Mengye Ren, and Raquel Urtasun. Graph hypernetworks for neural architecture search. arXiv preprint arXiv: arXiv:1810.0577 1810.05749, 2018.
[25] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint IrXiv: 2014.
[26] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming Hc. Aggregated residual transforma- tions for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1492- 1500, 2017.
[27] Gao Huang. Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convo- lutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700-4708, 2017.
[28] Barret Zoph and Quoe V Lc. Neural architecture search with reinforcement learning. arXiv preprint arXiv: 1,000057, 2016.
[29] Barret Zoph. Vijay Vasudevan, Jonathon Shlens, and Quoe V Le. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8697-8710, 2018.
[30] Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy. Progressive neural architecture search. In Proceedings of the European conference on computer vision (ECCV), pages 19-34, 2018.
[31] Esteban Real, Alok Aggarwal, Yanping Huang, and Quoe V Le. Regularized evolution for image classifier architecture search. In Proceedings of the aaai conference on artificial intelligence, volume 33, pages 4780-4789,40 2019.
[32] Xin Chen, Lingxi Xie, Jun Wu, and Qi Tian. Progressive darts: Bridging the optimization gap for nas in the wild, arXiv preprini 191iv: 10952, 2019.
[33] Andrew Howard, Mark Sandler, Grace Chu, Liang-Chies Ching Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for mobilenetv3. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1314-1324, 2019.
[34] Yujia Li, Daniel Tarlow, Mare Brockschmidt, and Richard Zemel. Gated graph sequence neural networks. arXiv preprint arXiv: I5I 1.04933, 2015.
[35] Kyunghyun Cho, Bart Van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using mn encoder-decoder for statistical machine translation. arXiv preprint 140iv 1078, 2014.
[36] Andrew Brock, Soham De and Samuel L Smith. Characterizing signal propagation to close the perfor- mance gap in unnormalized resnets. arXiv preprint arXiv:2 2021.
[37] Andrew Brock, Soham De, Samuel I. Smith, and Karen Simonyan. peff-mnefoomance large-scale image recognition without normalization. arXiv preprint arXiv:2102.06171, 2021.
[38] Laurent Sifre and Stéphane Mallat. digid-motion scattering for texture classification. arXiv preprint arXiv:1403.1687. 2014.
12

---

[39] Andrew G Howard, Menglong Zhu. Bo Chen, Dmitry Kalenichenko, Weijun Wang. Tobias Weyand, Marco Andreetio, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint axX44:1704.048880 2017.
[40] Jie Hu, Li Shen, and Gang Sun. Squeeze-and-exciaan networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7132-7141, 2018
[41] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. arXiv preprint (iix:17066037622. 2017.
[42] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint
16Xiv: 607.06450, 2016.
[43] Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly, 2(1-2):83-97, 1955.
[44] Aric Hagherg, Pieter Swart, and Daniel S Chult. Exploring network structure, dynamics, and function using networkx. Technical report, Los Alamos National Lab.(IANN.), Los Alamos, NM (United States), 2008.
[45] Alain Barrat, Mare Barthelemy, Romualdo Pastor-Satorras and Alessandro Vespignani. The architecture of complex weighted networks. Proceedings of the national academy of sciences, 101(11):3747-3755 2004.
[46] Jiaxuan You, Jure Leskovec, Kaiming He, and Saining Xie. Graph structure of neural networks, 2020.
[47] Anna Golubeva, Behnam Neyshabur, and Guy Gur-Ari. Are wider nets better given the same number of parameters? arXiv preprint arXiv:2010. 14495, 2020.
[48] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint arXiv:6605.07146 2016.
[49] Quynh Nguyen and Matthias Hein. The loss surface of deep and wide neural networks. In International conference on machine learning, pages 2603-2612. PMLR, 2017.
[50] Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. Training very deep networks. arXiv preprint 1=0..00000006220 2015.
[51] Sara Hooker. The hardware lottery, 2020.
[52] Angus Galloway, Anna Golubeva. Thomas Tanay, Medhat Moussa, and Graham Taylor. Batch normalization is cause of adversarial vulnerability, arXiv preprint arXiv: 1905.02161, 2019.
[53] Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint vrXiv:1933.22266 2019.
[54] Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European conference on computer vision (ECCV), pages 3-19, 2018.
[55] Siyuan Qiao, Huiyu Wang, Chenxi Liu, Wei Shen, and Alan Yuille. Micro-batch training with batch- channel normalization and weight standardization. arXiv preprint arXiv: arXiv:1903.100 10520, 2019.
[56] Hongyi Zhang, Yann N Dauphin, and Tengyu Ma. Fixup initialization: Residual learning without normalization. arXiv preprint 1901 2019.
[57] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pages 1026-1034, 2015.
[58] Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 249-256, 2010.
[59] Oscar Chang, Lampros Flokas, and Hod Lipson. International Conference on Learning Representations, 2019.
Principled weight initialization for hypernetworss. In
[60] Uni Alou and Eran Yahav. On the bottleneck of graph neural networks and its practical implications. arXiv preprint arXiv:2006005205, 2020.
[61] Salah El Hihi and Yoshua Bengio. Hierarchical recurrent neural networks for torg-term dependencies. In Advances in neural information processing systems, pages 493-499, 1996.
13

---

[62] Meng Liu, Zhengyang Wang, and Shuiwang Ji. Non-local graph neural networks. arXiv preprint arXiv:2005. 14612, 2020.
[63] Hongbin Pei. Bingzhe Wei, Kevin When-Chuan Chang, Yu Lei, and Bo Yang. Geom gen: Geometric graph convolutional networks. arXiv preprint arXiv:2002.05287, 2020.
[64] Jiaxuan You, Rex Ying, and Jure Leskovee. Position-aware graph neural networks. In International Conference on Machine Leurning, pages 7134-7143. PMLR, 2019.
[65] Yiding Yang, Xinchao Wang, Mingli Song. Junsong Yuan, and Dacheng Tao, Spagan: Shortest path graph attention network. arXiv preprint (xXi:2211103444. 2021.
[66] Pavlo M Radiuk. Impact of training set batch size on the performance of convolutional neural networks for diverse datasets. Information Technology and Management Science, 20(1 (011:20022 200,2422222 2017.
[67] Zhaohui Yang, Yunhe Wang, Xinghao Chen, Boxin Shi, Chao Xu, Chunjing Xu, Qi Tian, and Chang Xu. Cars: Continuous evolution for efficient neural architecture search. In Proceedings of the IEEF/CVF Conference Computer Vision and Puttern Recognition, pages 1829-1838, 2020.
[68] Chaoyang He, Haishan Ye, Li Shen, and Tong Zhang. Milenas: Efficient neural architecture search via mixed-level reformulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11993 01993-12000 12002, 2020.
[69] Guohao Li, Guocheng Qian, Itzel C Delgadillo, Matthias Muller, Ali Thabet, and Bernard Ghanem. Sgas: Sequential greedy architecture search. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1620-1630, 2020.
[70] Olga Wichrowska, Niru Maheswaranathan, Matthew W Hoffman, Sergio Gomez Colmenarejo, Misha Denil, Nando Freitas, and Jascha Sohl-Dickster Learned optimizers that scale and generalize. In International Conference on Machine Learning, pages 3751-3760. PMLR, 2017.
[71] Luke Metz, Niru Maheswaranathan, C Daniel Freeman, Ben Poolc, and Jascha Gohl-Dickstein. Tasks, stability, architecture, and compute: Training more effective learned optimizers, and using them to train themselves. arXiv preprint arXiv:2009. 1243, 2020.
[72] Jingzhao Zhang, Sai Praneeth Karimireddy, Andreas Veit, Seungyeon Kim, Sashank Reddi, Sanjiv Kumar, and Suvrit Sra. Why adam beats sgd for attention models. arXiv e-prints, pages arXiv-1912, 2019.
[73] Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuus, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint arXiv:1706.0000
2017.
[74] Alex Krizhevsky, Tly Sutskever, and Geoffrey E. Hinton. Imagenct classification with deep convolutional neural networks. In Advances in neural information processing systems, pages 1097-1105, 2012.
[75] Wei Li, Shaogang Gong, and Xiatian Zhu. Neural graph embedding for neural architecture search. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 4707-4714, 2020.
[76] Wei Wen, Hanxiao Liu, Hai Li, Yiran Chen, Gabriel Bender, and Pieter-Jan Kindermans. Neural predictor for neural architecture search. arXiv preprint arXiv: 1912.099) 2019.
[77] Haifeng Jin, Qingquan Song, and Xia Hu. Auto-keras: An efficient neural architecture search system. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1946-1956, 2019.
[78] Nils M Kriege, Fredrik D Johansson, and Christopher Morris. A survey on graph kernels. Applied Network Science, 5(1):1-42, 2020.
[79] Ilya Makarov, Dmitrii Kiselev, Nikita Nikitinsky, and Lovro Subelj. Survey on graph embeddings and their applications to machine learning problems on graphs. PeerJ Computer Science, 7, 2021.
[80] Wei Wen, Hanxiao Liu, Yiran Chen, Hai Li, Gabriel Bender, and Picter-Jan Kindermans. Neural predictor for neural architecture search. In European Conference on Computer Vision, pages 660-676. Springer, 2020.
[81] Jovita Lukasik, David Friede, Heiner Stuckenschmidr, and Margret Keuper. Neural architecture perfor- mance prediction using graph neural networks. arXiv preprint arXiv:2010.1002, 2020.
14

---

[82] Bowen Baker. Otkrist Gupta, Ramesh Raskar, and Nikhil Naik. Accelerating neural architecture search using performance prediction. arXiv preprini
2017.
[83] Simon Kornblith, Jonathon Shlens, and Quoe V Le. Do better imagenet models transfer better? In Proceedings of the IEEE conference computer vision and pattern recognition, pages 2661-2671, 2019.
[84] Minyoung Huh, Pulkit Agrawal, and Alexei A Efros. What makes imagenet good for transfer learning? arXiv preprint 1600.. 08614, 2016.
[85] Behnam Neyshabur, Hanic Sedghi, and Chiyuan Zhang. What is being transferred in transfer learning? arXiv preprint arXiv:2008. 1667, 2020.
[86] Maithra Raghu, Chiyuan Zhang. Jon Kleinberg, and Samy Bengio. Transfusion: Understanding transfer learning for medical imaging. arXiv preprint arXiv:1902.072000 2019.
[87] Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, et al. A large-scale study of representation learning with the visual task adaptation benchmark. arXiv preprint arXiv:1910004867, 2019.
[88] PyTorch. Pytorch object detection finetuning tutorial. URL http:::/pptoorcc.or///uooooooore interrediaaee/torcciiiie
tutorial html
[89] Liming Wang, Jianbo Shi. Gang Song. and I-fan Shen. Object detection combining recognition and segmentation. In Asian conference on computer vision, pages 189-199. Springer, 2007.
[90] Jürgen Schmidhuher and Al Blog. Metalearning machines learn learn (1987-).
[91] Louis Kirsch and Jürgen Schmidhuber. Meta learning backpropagation and improving it. arXiv preprint 1:Xi::2012.14000 2020.
[92] Hugo Siqueira Gomes, Benjamin Léger, and Christian Gagné. Meta learning black-box population-based optimizers. arXiv preprint arXiv:2103.0355: 2021.
[93] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-lcarning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning- Volume 70. pages 1126-1135, JMLR. org, 2017.
[94] Jake Snell, Kevin Swersky, and Richard S Zemel. Prototypical networks for few-shot learning. arXiv preprint arXiv: 1703005775.5 2017.
[95] Adriana Romero, Pierre Lue Carrier, Akram Erraqabi, Tristan Sylvain, Alex Auvolat, Etienne Dejoie, Mare-André Legault, Marie-Piens Dubé, Julie G Hussin, and Yoshua Bengio. Diet networks: thin parameters for fat genomics, arXiv preprint orXiv:1611.09344 2016.
[96] James Requeima, Jonathan Gordon, John Bronskill, Sebastian Nowozin, and Richard E Turner. Fast and flexible multi-task classification using conditional neural adaptive processes. arXiv preprint 190iv:1906.0777 07697, 2019.
[97] Huaiyu Li, Weiming Dong, Xing Mci, Chongyang Ma, Feiyue Huang, and Bao-Gang Hu. I.m-net: Learning to generate matching networks for few-shot learning. In International conference machine learning, pages 3825-3834. PMLR, 2019.
[98] Luca Bertinetto, João F Henriques. Jack Valmadre, Philip HS Torr, and Andrea Vedaldi. Learning feed-forward one-shot learners. arXiv preprint /00l::666005522 2016.
[99] Dongze Lian, Yin Zheng, Yintao Xu, Yanxiong Lu, Leyu Lin, Peilin Zhao, Junzhou Huang, and Shenghua Gao. Towards fast adaptation of neural architectures with meta learning. In ICLR. JMLR. org, 2020.
[100] Thomas Elsken, Benedikt Staffler, Jan Hendrik Metzen, and Frank Hutter. Meta-lcarning of neural architectures for few-shot lcarning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pages 12365-12375, 2020.
[101] Xin Chen, Yawen Duan, Zewei Chen, Hang Xu, Zihao Chen, Xiaodan Liang, Tong Zhang, and Zhenguo Li Catch: Context- meta reinforcement learning for transferrable architecture search. In European Conference Computer Vision, pages 185-202. Springer, 2020.
[102] Jiahui Yu, Pengchong Jin, Hanxiao Liu, Gabriel Bender, Pieter-Jan Kindermans, Mingxing Tan, Thomas Huang, Xiaodan Song, Ruoming Pang, and Quoe Le. Bignas: Scaling up neural architecture search with big single-stag models, arXiv preprint 2xXvv2003..1142. 2020.
15

---

[103] Xin He. Kaiyong Zhao, and Xiaowen Chu. Automl: A survey of the state-of-the-art. Knowledge Bused Systems, 212:106622, 2021.
[104] Andrew Brock, Theodore Lim, James M Ritchie, and Nick Weston. Smash: one-shot model architecture search through hypernetworks. arXiv preprint 1708. 708404344. 2017.
[105] Gabriel Bender, Pieter-Jan Kindermans, Barret Zoph, Vijay Vasudevan, and Quoe Lc. Understanding and simplifying one-shot architecture search. In International Conference Machine Learning, pages 550-559, 2018
[106] Guixiang Ma, Nesreen K Ahmed, Theodore L Willke, and S Yu Philip. Deep graph similarity learning; A survey. Data Mining and Knowledge Discovery, pages 1-38, 2021.
[107] Yunsheng Bai, Hao Ding, Song Bian. Ting Chen, Yizhou Sun, and Wei Wang. Simgnn: A neural network approach to fast graph similarity computation. In Proceedings the Twelfth ACM International Conference Web Search and Data Mining, pages 384-392, 2019.
[108] DC Dowson and BV Landau. The fréchet distance between multivariate normal distributions. Journal of multivariate analysis, 1253):450-455, 1982.
[109] Chia-Cheng Liu, Harris Chan, Kevin Luk, and Al Borcalis. Auto-regressive graph generation modeling with improved evaluation methods. In 33rd Conference on Neural Information Processing Systems. Vancouver, Canada, 2019.
[110] Julian Zilly, Hannes Zilly, Oliver Richter, Roger Wattenhofer, Andrea Censi, and Emilio Frazzoli. The frechet distance of training and test distribution predicts the generalization gap. 2019.
[111] Rylee Thompson, Elahe Ghalebi, Terrance De and Graham W Taylor. Building lego using deep generative models of graphs. arXiv preprint arXiv:20/2. /1543, 2020.
[112] Pinar Yanardag and SVN Vishwanathan. Deep graph kernels. In Proceedings of the 2/th ACM SIGKDD international conference on knowledge discovery and data mining, pages 1365-1374, 2015.
[113] Misha Denil, Babak Shakibi, I Laurent Dinh, Marc' Aurelio Ranzato, and Nando de Freitas. Predicting parameters in deep learning. In C J C Burges, L Bottou, M Welling, Ghahramani, and K Q Weinberger, editors, Advances in Neural Information Processing Systems 26, pages 2148-2156. Curran Associates. Inc., 2013.
[114] Neale Ratzlaff and Li Fuxin. Hypergan: generative model for diverse, performant neural networks. In International Conference on Machine Learning, pages 5361-5369. PMIR, 2019.
[115] lou-Jen Liu, Jian Peng, and Alexander G Schwing. Knowledge flow: Improve upon your teachers. arXiv preprint 1rXiv:1904.000 008805878. 2019.
[116] Yu Cheng, Duo Wang, Pan Zhou, and Tao Zhang. A survey of model compression and acceleration for deep neural networks. arXiv preprint 171000220.0 2017.
[117] Yann Dauphin and Samuel S Schoenholz. Metainit: Initializing learning by learning to initialize. 2019.
[118] Chen Zhu, Renkun Ni, Zheng Xu, Kezhi Kong, W Ronny Huang. and Tom Goldstein. Gradinit: Learning initialize neural networks for stable and efficient training. arXiv preprint 102i::202200099 2021.
[119] Debasmit Das, Yash Bhalgat, and Fatih Porikli. Data-driven weight initialization with sylvester solvers. arXiv preprint 205. 10335, 2021.
[120] Yue Yu, Jie Chen, Tian Gao, and Mo Yu. Dag-gnn: Dag structure learning with graph neural networks. In International Conference on Machine Learning, pages 7154-7163. PMLR, 2019.
[121] Xiaojic Guo and Liang Zhao. A systematic survey on deep generative models for graph generation. arXiv preprint ar0i 20060660.,..66 2020.
[122] Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Designing network design spaces. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 14428-10436 2020.
[123] Jiaxuan You, Zhitao Ying, and Jure Leskovec. Design space for graph neural networks. Advances in Neural Information Processing Systems, 33, 2020.
16


Reviews:
[{"review_id": "repBJtrMeVX", "reviewer": "Reviewer_mxL9", "summary": "This paper proposes and analyzes a method for generating trained weights already optimized for a given single task for a large class of diverse architectures using Graph hypernetworks. They introduce a new DEEPNETS-1M benchmark with in-distribution and out of distribution architectures. The work also proposed three main improvements to improve performance of their GHN: normalization of predicted parameters, enhanced long-range interactions in the GHN, and meta-batching of architectures.  They demonstrate that their method is able to produce weights that do surprisingly well on unseen and diverse networks in less than 1 second on CPU or GPU for CIFAR-10 and Imagenet. They demonstrate that even for large Resnet-50s which are out of the training distribution, they could achieve 60% accuracy on CIFAR10.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper analyzes the interesting task of directly predicting trained parameters for diverse feed-forward networks. This in itself is not novel since it has been done previously in many NAS works for the purposes for creating performance predictors as well works on targeting a specific class of networks. However, there is significant novelty in the scope of diverse network architectures that their method can predict reasonable weights for and this may be the first which does a focused analysis on accuracy performance of these predicted weights. The analysis of results on in distribution and out of distribution architectures is quite novel. Unfortunately in terms of practical significance, the practical use cases of using these weights to generate good initializations for fine tuning seems to currently significantly degrade performance. That in itself is still an interesting finding to invite further analysis. \n\nThe paper provides thorough analysis of their methods against the baseline of training from scratch for a very diverse set of architectures. The work also proposed three main improvements to improve performance: normalization of predicted parameters, enhanced long-range interactions in the GHN, and meta-batching of architectures. They provide thorough experiments ablating the effect of these changes.\n\nSome of the results could benefit from further analysis. For example, it's interesting that the CIFAR-10 trained networks seem to generalize significantly better to Resnet than the Imagenet-trained networks.\n\nI believe that this paper's main weakness is that it would be very useful to be able to compare the performance of the GHN with some stronger competitors. While there may be little or no work on predicting weights for out-of-distribution architectures, there has been significant NAS work on training supernets. A strong baseline then would be to train a supernet on some subset of the architecture space using a uniform prior and examine performance when directly deriving networks from those subnetworks.\n\nIn addition, this paper would greatly benefit from significant analysis of the weights that are generated. How sparse are the weights generated? Is there a correlation between more weights or less weights and model accuracy? For deep networks for example is it generating a few layers and a bunch of identity or skip layers? It would be interesting to visualize the activations. \n\nCurrently, it is also a bit difficult to judge the usability of this work since it lacks discussion of the training cost of the graph hypernetwork.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper adequately addressed the limitations and potential negative societal impact of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "kBWZw-uneL", "reviewer": "Reviewer_YdDc", "summary": "This paper propose a way to predict/generate the parameters of an image based deep network instead of training it via SGD. It's a thought-provoking paper and the authors appears to be very detailed in their implementation. The results are surprisingly good but not better than standard optimization algorithms.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper propose a way to predict/generate the parameters of an image based deep network instead of training it via SGD. It's a thought-provoking paper and the authors appears to be very detailed in their implementation. The results are surprisingly good but not better than standard optimization algorithms.\n\nPositives of the paper:\n- The idea is novel and very thought provoking and different than standard way of approaching learning. The ability to accelerate learning is a great thing to do, given the time it takes to train large models.\n- The paper is well written and clear. The authors have also shown the limitations of this approach which provides good context and discussion.\n\nNegatives of the paper:\n- The results doesn't surpass He-initialization with finetuning, which would've be nice and strong suggestions in favor this this approach.\n- The authors suggested that it was 'natural' to be able to predict parameters from architecture. I think this isn't quite clear and it would be good for the author to outline more benefits to predicting the parameters. Given that the weights are good but not near SoTA, it's is naturally to ask why predict the parameters?\n- The proposed method predicts parameters given architecture. How does this fit in with NAS?\n- How sensitive are the changes of the parameters to network architecture (of the same parameter dimension), is it possible that the good weights are 'stored' in the GHN ?\n\nOverall I think the paper is well written and novel and interesting, even though the results are not SoTA. The authors are advised to address the comments above for the next version of the paper.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "inYtvK7T6vi", "reviewer": "Reviewer_CZfY", "summary": "This paper studied the problem of parameter prediction for deep neural networks. Especially, the authors focused on the task of predicting parameters for various network structures with a single hypernetwork. The method is based GHN and introduces several important modifications. To evaluate the performance of the proposed method, a standardized benchmark is introduced.  The improved method showed impressive performance on CIFAR-10 and ImageNet even with large neural networks. Besides, the method is also proved to be useful in architecture representation learning.\n\n ", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The task of predicting the parameter of diverse architectures with a single hyper-network is novel and important. Though iterative optimization is the mainstream for learning deep networks, it should be encouraged to explore different directions.\n\n2. The method is based on the existing work Graph-hypernetworks, while there are three key elements introduced to gain superior improvement. The modification is certainly well motivated which also provides impressive empirical performance.\n\n3. The evaluation part of this paper is strict and extensive. I appreciate the efforts in providing the benchmarks for evaluating the generalization ability of the different methods which will certainly benefit the research in this area. The ablation study on different components is detailed and clear which makes the results convincing.\n\nQuestions:\n1. One appealing property of the proposed method is the decreased computation burden. To this end, I suggest the authors provide some statistics about the learning procedure of the Graph-hypernetworks.\n\n2. The experiments is conducted mainly in the image classification settings. I wonder whether the proposed method could apply to different data type, such as language and graph, or different architectures such as transformers. \n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.\nRefer to the main reviews.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hours"}, "extra_text": {}}, {"review_id": "Aeq4qJ0cmv4", "reviewer": "Reviewer_KeMh", "summary": "In this paper, the authors propose the task of parameter prediction with a new DeepNets-1M dataset. By using a improved Graph HyperNetwork, they can achieve reasonable performance for unseen and diverse networks without iterative optimization. The learned neural architecture representation also performs better than previous works.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- The three components proposed by this paper significantly improve the original GHN performance. The authors also provide rich ablation studies to show the effectiveness of their modifications.\n- The authors extend the network design space to include a lot of different ops. The proposed GHN-2 shows good generalization ability on OOD architectures.\n- The neural architecture representation learned by GHN seems to be more powerful than previous works.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "- Although the authors claim this parameters prediction task is more general than NAS, I think the most suitable application scenario of this work is still in NAS (given the performance of predicted parameters still have a huge gap with iterative optimized parameters). It would be interesting to see some results in NAS. I believe the network representation learned by GHN-2 should be powerful for NAS.\n- Including some OOD architectures in the training set would be an interesting direction. It's great to see whether this framework can work for architectures with mixed ops (for example an architecture with both convolutions and multi-head attentions).\n- It would be interesting to see whether this work can be applied to other tasks beyond image classification (e.g. object detection).", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a technique for prediction of parameters in deep architectures. After the initial reviews, the reviewers were concerned about 1) performance in the low-data regime 2) training cost of GHNs 3) relevance/utility for NAS application 4) improvement via finetuning.
The authors clarified 1) and 4) well. 2) is high (as one would expect) but still reasonable, considering that predictions for new architectures can be quickly generated after initial training and that practitioners often adopt even more expensive approaches, such as OFA. The advantage of the OFA framework is that each architecture can be trained in parallel on a cluster, but then again, only a finite set of architectures can be trained this way, and the cost increases linearly with the number of architectures. 3) was briefly discussed in the paper and is relatively strong for the DARTS search space on CIFAR-10, although a comparison to standard/broader NAS benchmarks would have made a stronger point here. 
Decision: Accept (Poster)