Principled Weight Initialization for Hypernetworks
Authors: Oscar Chang
Lampros Flokas
Hod Lipson
Venue:  (2020)
Keywords: hypernetworks
initialization
optimization
meta-learning

Abstract: Hypernetworks are meta neural networks that generate weights for a main neural network in an end-to-end differentiable manner. Despite extensive applications ranging from multi-task learning to Bayesian deep learning, the problem of optimizing hypernetworks has not been studied to date. We observe that classical weight initialization methods like Glorot & Bengio (2010) and He et al. (2015), when applied directly on a hypernet, fail to produce weights for the mainnet in the correct scale. We develop principled techniques for weight initialization in hypernets, and show that they lead to more stable mainnet weights, lower training loss, and faster convergence.

Published as a conference paper at ICLR 2020
PRINCIPLED WEIGHT INITIALIZATION FOR HYPERNETWORKS
Oscar Chang, Lampros Flokas, Hod Lipson
Columbia University
New York, NY 10027
{oscar. chang, 1f2540, hod. hod.
edu
ABSTRACT
Hypernetworks are meta neural networks that generate weights for a main neural network in an end-to-end differentiable manner. Despite extensive applications ranging from multi-task learning to Bayesian deep learning, the problem of op- timizing hypernetworks has not been studied to date. We observe that classical weight initialization methods like Glorot & Bengio (2010) and He et al. (2015). when applied directly on hypernet, fail to produce weights for the mainnet the correct scale. We develop principled techniques for weight initialization in hypernets, and show that they lead to more stable mainnet weights, lower training loss, and faster convergence.
INTRODUCTION
Meta-learning describes a broad family of techniques in machine learning that deals with the prob- lem of learning to learn. An emerging branch of meta-learning involves the use of hypernetworks, which are meta neural networks that generate the weights of main neural network solve given task in an end-to-end differentiable manner. Hypernetworks were originally introduced by Ha et al. (2016) as way induce weight-sharing and achieve model compression by training the same meta network to learn the weights belonging to different layers in the main network. Since then, hyper- networks have found numerous applications including but limited to: weight pruning (Liu al.) 2019). neural architecture search (Brock et al. 2017 /Zhang et al.. 2018)) Bayesian neural networks (Krueger et al. 2017 Ukai et al. [018]] Paallowki al. Henningg et al.) 2018 Deutsch al. 2019). multi-task learning (Pan et al. 2018: Shen et [0][201]]] et 2019 Serrà et al. 2019
Meyerson Mükkuulainen 2019). continual learning (von Oswald et al. 2019). generative models (Suarez 2017; Ratzlaff & Fuxin 2019), ensemble learning (Kristiadi & Fischer 2019), hyperpa-
rameter optimization (Lorraine & Duvenaud 2018). and adversarial defense (Sun et al. 2017).
Despite the intensified study of applications of hypernetorkss, the problem of optimizing them to this day remains significantly understudied. In fact, even the problem of initializing hypernetworks has not been studied. Given the lack of principled approaches, prior work in the is mostly limited to ad-hoc approaches based on trial and error (c.f. Section (3.. For example, it is common to initialize the weights of hypernttwrk by sampling a "small" random number. Nonetheless, these ad-hoc methods do lead to successful hypernetwork training primarily due to the use of the Adam optimizer (Kingma & Bal 2014), which has the desirable property of being invariant the scale of the gradients. However, even Adam will not work if the loss diverges (i.e. overflow) at initialization, which will happen sufficiently big models. The normalization of badly scaled gradients also results in noisy training dynamics where the loss function suffers from bigger fluctuations during training compared vanilla stochastic gradient descent (SGD). Willson et al. (2017); Reddi et al.) (0118) showed that while adaptive optimizers like Adam may exhibit lower training error, they fail to generalize as well to the test set as non-adaptive gradient methods. Moreover, Adam incurs computational overhead and requires 3X the amount of memory for the gradients compared to vanilla SGD.
Small random number sampling is reminiscent of early neural network research (Rumelhart al. 1986) before the advent of classical weight initialization methods like Xavier init (Glorot & Bengio 2010 and Kaiming init (He al. 2015). Since then, a big lesson learned by the neural network

---

Published as a conference paper at ICLR 2020
optimization community is that architecture specific initialization schemes are important to the bust training of deep networks, as shown recently in the case of residual networks (Zhang et al.) recognized outstanding open prob-
2019). In fact, weight initialization for hypernetworks lem by prior work (Deutsch et al. 2019) that had questioned the suitability of classical initialization methods for hypernetworks.
Our results We show that when classical methods are used to initialize the weights of hypernet- works, they fail produce mainnet weights in the correct scale, leading exploding activations and losses. This is because classical network weights transform one layer's actívations into another, while hypernet weights have the added function of transforming the hypemnet' 's activations into the mainnet's 's weights. Our solution is to develop principled techniques for weight initialization in hy- pernetworks based on variance analysis. The hypernet case poses unique challenges. For example, in contrast to variance analysis for classical networks, the case for hypernetworks can be asymmetri- cal between the forward and backward pass. The asymmetry arises when the gradient flow from the mainnet into the hypernet is affected by the biases, whereas in general, this does not occur for gra- dient flow in the mainnet. This underscores again why architecture specific initialization schemes essential. We show both theoretically and experimentally that methods produce hypernet weights in the correct scale. Proper initialization mitigates exploding activations and gradients or the need depend Adam. Our experiments reveal that it leads to more stable mainnet weights, lower training loss, and faster convergence.
Section briefly covers the relevant technical preliminaries, and Section 3 reviews problems with the ad-hoc methods currently deployed by hypernetwork practitioners. We derive novel weight initialization formulae for hypernttworks in Section 4 empirically evaluate our proposed methods in Section and finally conclude in Section 6
PRELIMINARIES
Definition. A hypernetwork a meta neural network H with its own parameters that generates the weights of main network 0 from some embedding e in differentiable manner: f = He(e). Unlike classical network, in hypernetwork, the weights of the main network are not model param- eters. Thus the gradients 20 have to be further backpropagated to the weights of the hypernetwork AQ. which is then trained via gradient descent d1+1 = 6r - >DØ
This fundamental difference suggests that conventional knowledge about neural networks may not apply directly to hypernetworks and novel ways of thinking about weight initialization, optimization dynamics and architecture design for hypernetworks are sorely needed.
2.1 RICCI CALCULUS
We propose the use of Ricci calculus, opposed to the more commonly used matrix calculus, suitable mathematical language for thinking about hypernetworks. Ricei calculus useful because it allows us to reason about the derivatives of higher-order tensors with notational ease. For readers not familiar with the index-based notation of Ricci calculus, please refer to Laue et al. (2018) for a good introduction to the topic written from machine learning perspective.
For a general nth-order tensor T " we use di to refer to the dimension the index set that d is drawn from. We include explicit summations where the relevant expressions might be ambiguous, and use Einstein summation convention otherwise. We use square brackets to denote different layers for added elarity, so for example W[[] denotes the I-th weight layer.
2.2 XAVIER INITIALIZATION
Glorot & Bengio 2010) derived weight initialization formulae for a feedforward neural network by conducting a variance analysis over activations and gradients. For a linear layer y = Wj + 1i, suppose we make the following Xavier Assumptions at initialization: (1) The W x², and b' are
all independent of each other. (2) Vi,j B[W;) = 0. (3) Vj RB-' O. (4) Vi D' 0.
2

---

Published as a conference paper at ICLR 2020
Then, Ely'l = 0 and Var(y') = d, Var(W))aar(z>)) To keep the variance of the output and input activations the same, i.c. Var(y') Var(x"), we have sample W3 from distribution whose variance is equal to the reciprocal of the fan-in: Var(W)) -
If analogous assumptions hold for the backward pass, then to keep the variance of the output and input gradients the same, we have sample Wj from a distribution whose variance is equal the reciprocal the fan-out: Var(W)) = d.
1
Thus, the forward pass and backward pass result in symmetrical formulae. Glorot & Bengio (2010) proposed initialization based on their harmonic mean: Var(W))
dy+d;
In general, a feedforward network is non-linear, so these assumptions are strictly invalid. But odd activation functions with unit derivative at O results in a roughly linear regime at initialization.
2.3 KAIMING INITIALIZATION
He et al. 2015) extended Glorot & Bengio 2010)'s 's analysis by looking at the case of ReLU activa- tion functions, i.e. y' W ReLU( x3 b.. We can write 2 ReLUU(>) (x') to get
1 Var(y') Σ B[(=") = Σ ((aa))) 2 d, Var(WW))
This results in an extra factor of in the variance formula. Wj have to be symmetric around 0 to enforce Xavier Assumption 3 as the activations and gradients propagate through the layers. HHe al. (2013) argued that both the forward or backward version of the formula can be adopted, since the activations gradients will only be scaled by a depth-independdnt factor. For convolutional layers, we have to further divide the variance by the size of the receptive field.
"avvier init' and 'Kamming init' terms that sometimes used interchangeably. Where there might be confusion, we will refer to the forward version as fan-in init, the backward version as fan-out init, and the harmonic mean version as harmonic init.
3 REVIEW OF CURRENT METHODS
In the seminal Ha et al. 2016) paper, the authors identified two distinct classes of hypernetworks: dynamic (for recurrent networks) and static (for convolutional networks). They proposed Orthogonal init (Saxe et al. 2013) for the dynamic class, but omitted discussion of initialization for the static class. The static class has since proven to be the dominant variant, covering all kinds of non-recurrent networks (not just convolutional), and thus will be the central object of our investigation.
Through an extensive literature and code review, we found that hypernet practitioners mostly depend on the Adam optimizer, which is invariant to and normalizes the scale of gradients, for training and resort to one of four weight initialization methods:
MI Xavier or Kaiming init (as found in Pawlowski et al. (017): Balazevic et al. 2018): Serrà et 2019); von Oswald al. (201))).
M2 Small random values (as found in Krueger al. (2017): Lorraine & Duvenaud 2018
M3 Kaiming init, but with the output layer scaled by I
To (as found Ukai et al. (2018))
M4 Kaiming init, but with the hypernet embedding set to be a suitably scaled constant (as found in Meyerson & Miikkulainen (019)).
MI uses classical neural network initialization methods to initialize hypernetworks. This fails to produce weights for the main network in the correct scale. Consider the following illustrative exam- ple one-layer linear hypernet generating a linear mainnet with T + 1 layers, given embeddings sampled from standard normal distribution and weights sampled entry-wise from zero-mean
3

---

Published as a conference paper at ICLR 2020
distribution. We leave the biases out for now, and assume the input data z/1] is standardized.
z|t 1| W[t 1 x[1]"
W
H[ e
1sts..
T
(I)
Var(x[]'+ 1) = Var(*[1]** II de (ar(W[])?+)) = Var(=[1]*) II t=1
d, de, Var ((
In this case, if the variance of the weights in the hypernet (t is equal to the reciprocal of the fan-in d then the variance of the activations Var(æ[[T + 1] I d; explodes. If it is equal to the reciprocal of the fan-out d, di.,. then the activation variance Var(æ[T + 1]
dee is likely to vanish, since the size of the embedding vector is typically small relatively to T d., 1 the width of the mainnet weight layer being generated.
Where the fan-in is of a different scale than the fan-out, the harmonic mean has a scale close to that of the smaller number. Therefore, the fan-in, fan-out, and harmonic variants of Xavier and Kaiming init will all result in activations and gradients that scale exponentially with the depth of the mainnet.
M2 and M3 introduce additional hyperparameters into the model, and the ad-hoc manner in which they work is reminiscent of pre deep learning neural network research, before the introduction of classical initialization methods like Xavier and Kaiming init. This ad-hoc manner not only inele- gant and consumes more compute, but will likely fail for deeper and more complex hypernetworks.
M4 proposes set the embeddings et suitable constant (d.," 1/2 in this case), such that both W and H fiikk can seem to be initialized with the same variance as Kaiming init. This ensures that the variance of the activations in the mainnet are preserved through the layers, the restrictions on the embeddings might not be desirable in many applications.
Luckily, the fix appears simple set Var (+
1. This results in the variance of the d.,
generated weights in the mainnet Var(W(())))) di resembling conventional neural networks initialized with fan-in init. This suggests general hypernet weight initialization strategy: initialize the weights of the hypernet such that the mainnet weights approximate classical neural network initialization. We elaborate on and generalize this intuition in Section
1
4 HYPERFAN INITIALIZATION
Most hypemetwork architectures use linear output layer so that gradients can pass from the main- net into the hypernet directly without any non lineeanaisiiss. We make use of this fact in developing methods called hyperfan-in init and hyperfan-out init for hypernetwork weight initialization based on the principle of variance analysis.
4.1 HYPERFAN-IN
Suppose a hypernetwork comprises a linear oulput layer. Then, the variance between the imput and output activations of a linear layer in the mainnet y² WW + b' can be preserved using fan-in in init in the hypernetwork with appropriately scaled output layers.
Proposition.
Case 1. The hypernet generates the weights but not the biases of the mainnet. The bias in the mainnet is initialized to zero. We can write the weight generation in the form W = he)h(e)) +33 where computes all but the last layer of the hypernet and (H, 3) form the output layer. We make the following Hyperfan Assumptions at initialization: (1) Xavier assumptions hold for all the layers in the hypernet.
(2) The H h(e)) < B3.. ni, and by are all independent of each other. (3)
Vi, J. k: E 0. (4) B[a²] = 0. (5) Vi =.
Use fan-in init initialize the weights for h. Then, Var(h(e)**> Var(e"). If we initialize H with the formula Var(H;;) 4,d. and B with zeros, we arrive at Var(W;) = d, which is the formula for fan-in init in the mainnet. The Hyperfan assumptions imply the Xavier assumptions
4

---

Published as a conference paper at ICLR 2020
hold in the mainnet, thus preserving the imput and output activations.
Var(>*) Var(y') = Σ Var Var(a)) = ΣΣ
Var(h(e) Var Var(he)) Var(h(e)))a((x>)))
&
(2)
ΣΣ
dedg Var(e!) Var Var(e") Var(>*)
Var(x").
Case 2. The hypernet generates both the weights and biases of the mainnet. We can write the weight and bias generation the form Wj (((]() + 33 and & G]g(e[2)) + y respectively, where and g compute all but the last layer of the hypernet, and (H, 3) and (G,)) form the output layers. We modify Hyperfan Assumption 2 so it includes Gj, (e(2)), and y and further assume Var(>²) 1, which holds at initialization with the common practice of data standardization.
Use fan-in init to initialize the weights for h and Then, Var(h(e(11)**) = Var(e[1]")) and Var(g(e(2)) Var(e[2]"). If we initialize H with the formula Var(H', jkb 2d,d, Var(e]1/=) with the formula Var(G)) 24, varl[[[[)) and B, with zeros, then the input and output activations in the mainnet can be preserved.
Varly') )- Σ Var(W)))a((a?)) Var(b")
Σ Var( (ar(HH))))(((((())))))
+ Σ
Σ 2d,ds dg var
(aree(]"")Va((a))
Σ
1
Var(e[2]n)
2d, Var(e(2]m)
1 Var(x") ) - Var(>>).
If we initialize Gj to zeros, then its contribution to the variance will increase during training, causing exploding activations in the mainnet. Hence, we prefer to introduce a factor of 1/2 to divide the variance between the weight and bias generation, where the variance of each component allowed to either decrease increase during training. This becomes problem if the variance of the activations in the mainnet deviates too far away from 1, but we found that works well in practice.
4.2 HYPERFAN-OUT
Case 1. The hypernet generates the weights but not the biases of the mainnet. A similar deriva- tion can be done for the backward pass using analogous assumptions on gradients flowing
ØL
OL
in the mainnet:
Orlt + 1| W
(x(t]
ØL
ØL
OL
0L
through mainnet weights:
x1t]', (h(t
H
aW[!)
Oxis + 1]
aw
aL
0L
OL ØL
and through mainnet biases:
G
ab[t]i++1
Oxlt + 1l Dg(t](e)
abltjit+1
(4)
If we initialize the output layer H with the analogous hyperfan-out formula Var(H(1))
dry de, t) and the rest of the hypernet with fan-in init, then we can preserve input and out- put gradients on the mainnet: (ar(
AL aL = Var 1efe+1 However, note that the gradients will
de Var( aL shrink when flowing from the mainnet to the hypernet: Var (n[ de, varlets) and scaled by ceepth-independent factor due to the of fan-in rather than fan-out init.
aw
Case 2. The hypernet generates both the weights and biases of the mainnet. In the classical case, the forward version (fan-in init) and the backward version (fan-out init) are symmetrical. This
5

---

Published as a conference paper at ICLR 2020
remains true for hypernets if they only generated the weights of the mainnet. However, if they were to also generate the biases, then the symmetry no longer holds, since the biases do not affect the
gradient flow in the mainnet but they do for the hypemeet (c.f. Equation 4 Nevertheless, can initialize G so that it helps hyperfan-out init preserve activation variance on the forward pass as much possible (keeping the assumption that Var(æ>)
before):
Var(y")
> [Var(W).")) + Var(b')
ddddæ Var(e[1]"") Var hyperfan- Var(=") ) + di Var()) d,de Var(e[[]"")) Var( (H [hyperfan-in]ja Var(<>)
(5)
Plugging in the formulae for Hyperfan-in and Hyperfan-out from above, we get
(1 d
Var(Gi)
d, Var(e(2]mm
We summarize the variance formulac for hyperfan-in and hyperfan-out init Table I It is not uncommon to re-use the same hypernet to generate different parts of the mainnet, as was originally done in Ha al. (2016). We discuss this case in more detail in Appendix Section
+ Table 1: Hyperfan-in and Hyperfan-out Variance Formulae for Wj- = ( of ReLU(W3a2 +b)), then IROLU 1, else if y' Wja² +6', then ARLLU = 0. If = + then HBisS 1, else if b O, then LhBiS 0. We initialize and g with fan-in init, and Sj.n* = 0. For convolutional layers, we have to further divide Var H by the size of the receptive field. Uniform init: ~ ((-v()v (3(-3//arXX), /3Vr((X)) Normal init: ~ NO, Var(X)).
Initialization
Variance Formula
Initialization
Variance Formula
Hyperfan-in
Var Hz++) 2 Varte [1
Hyperfan-out
Var (Hija)
TRVL d,d. [
Hyperfan-in
Var(G])
RaU 20, [)
Hyperfan-out
Var Var(G]) max de Vare 0)
5 EXPERIMENTS
We evaluated our proposed methods on four sets of experiments involving different use cases of hypernetworks: feedforward networks, continual learning, convolutional networks, and Bayesian neural networks. In all cases, we optimize with vanilla SGD and sample from the uniform distribu- tion according to the variance formula given by the init method. More experimental details can be found in Appendix Section B
5.1 FEEDFORWARD NETWORKS ON MNIST
As an illustrative first experiment, we train a feedforward network with five hidden layers (500 hidden units), hyperbolic tangent activation function, and a softmax output layer, on MNIST across four different settings: (1) a classical network with Xavier init, (2) hypernet with Xavier init that generates the weights of the mainnet, (3) hypernet with hyperfan-in init that generates the weights of the mainnet, (4) and hypernet with hyperfan-our init that generates the weights of the mainnet.
The use of hyperfan init methods on a hypernetwork reproduces mainnet weights similar to those that have been trained from Xavier init on classical network, while the use of Xavier init on hypernetwork causes exploding activations right at the beginning of training (see Fi. Observe Figure that when the hypernetwork is initialized the proper scale, the magnitude of generated weights stabilizes quickly. This in turn leads more stable training regime, as seen in Figure
More visualizations of the activations and gradients of both the mainnet and hypernet can be viewed in Appendix Section B.I Qualitatively similar observations were made when replaced the activation function with ReLU and Xavier with Kaiming init, with Kaiming init leading to even bigger activations at initialization.

---

Published as a conference paper at ICLR 2020
Suppose now the hypernet generates both the weights and biases of the mainnet instead of just the weights. We found that this architectural change leads the hyperfan init methods to take more time (but still less than Xavier init), generate stable mainnet weights (c.f. Figure 25 in the Appendix).
Sayottic
40.2 8:0
69
-06
+0.4
U2
+6.0 -800 -C.L V.D 3.0
-108 Hi20 -iL B.A 535 5.0
Figure 1: Mainnet Activations before the Start of Training on MNIST.
1e-4
Dyyper
1--1
Hypertann-n
1e-1
Hyperfa--out
layer laver four
threa byee four
tree
liver
1.0
Mein wettrrtttooon
6.2
0.5
0.2
-03
-62
0
25
50
75
105 iitettionn
125
150
175
25
50
100
125
150
175
25
50
35
300
125
350
175
1x Pierattoons
nerstion
Figure 2: Evolution of Hypernet Output Layer Activations during Training on MNIST. Xavier init results in unstable mainnet weights throughout training, while hyperfan-in and hyperfan-out init result in mainnet weights that stabilize quickly.
Training
Loss
Accuracy
585
0.01
Ppperten
"yporfon in
0.02
our
67.5
97.0
0.02
0.10
66.0
Xaster Kawier bnypert Hpporfan-n
Hppertan-sst
3
5
100
15 Eperte
23
25
10
5
10
1.5
70
25
10
15 Epechs
26
Eports
Figure 3: Loss and Test Accuracy Plots on MNIST.
5.2 CONTINUAL LEARNING ON REGRESSION TASKS
Continual learning solves the problem of learning tasks in sequence without forgetting prior tasks. von Oswald et al. (2019) used hypernetwork to learn embeddings for each task as way ef- ficiently regularize the training process to prevent catastrophic forgetting. We compare different initialization schemes on their hypernetwork implementation, which generates the weights and bi- ases of a ReLU mainnet with two hidden layers to solve a sequence of three regression tasks.
In Figure 4 ing the standard error. We observe that the hyperfan methods produce smaller training losses at initialization and during training, eventually converging smaller loss for each task.
we plot the training loss averaged over 15 different runs, with the shaded area show-
7

---

Published as a conference paper at ICLR 2020
Task
Task
Task3
122
Karning Nan
kaining (hn-sut)
2.0
Karning Bun ni Kiining (ha-au
1.0
Ko Dunini
ta (hever)
3.0
Paperiar-nn on
1.4
Popaa lavi
0.0
Heperianin
2.8
0.6
0.0
0.4
24
9.4
6.2
2.
0.0
Iteradora
Iaeratore
Figure 4: Continual Learning Loss a Sequence of Regression Tasks.
5.3 CONVOLUTIONAL NETWORKS ON CIFAR-10 -10
Ha et al. (2016) applied a hypernetwork on a convolutional network for image classification on CIFAR- IO. We note that our initialization methods do not handle residual connections, which were in their chosen mainnet architecture and are important topics for future study. Instead, we imple- mented their hypernetwork architecture on a mainnet with the All Convolutional Net architecture Springenberg et al. 2014) that is composed of convolutional layers and ReLU activation functions.
After searching through a dense grid of learning rates, we failed to enable the fan-in version of Kaiming init to train even with very small learning rates. The fan-out version managed to begin delayed training, starting from around epoch 270 (see Figure 5). By contrast, both hyperfan-in and hyperfan-out outt init led to successful training immediately. This shows good init can make it possible to successfully train models that would have otherwise been unamenable to training on a bad init.
Training Loss
Test Lows
Test Accuracy
0.0003
70
0.0175
1.75
0.052
0.012
Hyperfan-co our
Hypertann-ly
125
1.50
Hyperfaniout
Hypertan-
40
Hypertan Hyperfon
0.0055
10
202
300
100
400
502
302
Figure 5: Loss and Test Accuracy Plots on CIFAR-10.
5.4 BAYESIAN NEURAL NETWORKS ON IMAGENET
Bayesian neural networks improve model calibration and provide uncertainty estimation, which guard against the pitfalls of overconfident networks. [Kai al. (2018) developed Bayesian neural network by using hypernetwork to simulate an expressive prior distribution. We trained a similar hypernetwork by applying Ukai et al. (2018)ss 's methods on ImageNet, but differed in our choice of MobileNet (Howard et al. 2017) as a mainnet architecture that does not have residual connections.
In the work of Ukai et al. (2018). was noticed that even with the use of batch normalization in the mainnet, classical initialization approaches still led to diverging losses (due to exploding activations, c.f. Section 3) We observe similar results in our experiment (see Figure ( the fan-in version of Kaiming init, which is the default initialization in popular deep learning libraries like PyTorch and Chainer, resulted in substantially higher initial losses and led to slower training than the hyperfan methods. We found that the observation still stands even when the last layer of the mainnet is not generated by the hypernet. This shows that while batch normalization helps, it is not the solution for a bad init that causes exploding activations. Our approach solves this problem in principled way, and is preferable to the trial-and -nd--orr based heuristics that (Kai et al. (2018) had to resort to in order train their model.
8

---

Published as a conference paper at ICLR 2020
Surprisingly, the fan-out version of Kaiming init led to similar results the hyperfan methods, sug- gesting that batch normalization might be sufficient to correct the bad initializations that result in vanishing activations. That being said, hypernet practitioners should not expect batch normaliza- tion to be the panacea for problems caused by bad initialization, especially scenarios. In Bayesian neural network application (especially in hypernet architectures without relaxed
memory-constraines
the blowup in the number of parameters limits the use of big batch sizes, which is essential to the performance of batch normalization (Wu & He 2018). For example, in this experiment, our hypermet model requires 32 times as many parameters as a classical MobileNet.
To the best of our knowledge, the interaction between batch normalization and initialization is not well-understood, even in the classical case, and thus, our findings prompt an interesting direction for future research.
Training Loss
Top Test Acourcy
39
my
Ppechs
Figure 6: Loss and Test Accuracy Plots on ImageNet.
In all our experiments, hyperfan-in and hyperfan-out both led to successful hypernetwork training with SGD, We did not find a good reason to prefer one over the other (similar to He al. (2015)ss observation the classical case for fan-in and fan-out init).
CONCLUSION
For a long time, the promise of deep nets to learn rich representations of the world was left unfulfilled due to the inability train these models. The discovery of greedy layer-wise pre-training (Hinton ct 2006 | et al., (207)) and later, Xavier and Kaiming init, as weight initialization strategies enable such training
pivotal achievement that kickstarted the deep learning revolution. This underscores the importance of model initialization as a fundamental step in learning complex representations.
In this work, we developed the first principled weight initialization methods for hypernetworks, rapidly growing branch of meta-learning. We hope our work will spur momentum towards the development of principled techniques for building and training hypernetworkss, and eventually lead significant progress in learning meta representations. Other neeehyeeretwork methods of neural network generation (Stanley et al.) 2009, Koutnik et al. 2010) can also be improved by considering whether their generated weights result exploding activations and how to avoid that so.
ACKNOWLEDGEMENTS
This research was supported in part by the US Defense Advanced Research Project Agency (DARPA) Lifelong Learning Machines Program, grant ERR0011-18---01 We thank Dan Mar- tin and Yawei Li for helpful discussions, and the ICLR reviewers for their constructive feedback.

---

Published as a conference paper at ICLR 2020
REFERENCES
Ivana Balazevic, Carl Allen, and Timothy M Hospedales. Hypernetwork knowledge graph embed- dings, arXiv preprint arXiv: 07018, 2018.
Yoshua Bengio, Pascal Lamblin, Dan Popovici, and Hugo Larochelle. Greedy layer-wise training of deep networks. In Advances in neural information processing systems, pp. 153-160, 2007.
Andrew Brock, Theodore Lim, James M Ritchie, and Nick Weston. Smash: one-shot model archi- tecture search through hypernetworks. arXiv preprint arXiv: Sativ:1700.0534400 2017.
Lior Deutsch, Erik Nijkamp, and Yu Yang. A generative model for sampling high- performance and diverse weights for neural networks. arXiv preprint FrXiv::905.02850 2019.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 249-256, 2010.
David Ha, Andrew Dai, and Quoe V Le. Hypernetworss. arXiv preprint 169iv:100.009100 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp.
2015.
Christian Henning, Johannes von Oswald, João Sacramento, Simone Carlo Surace, Jean- lean-Pascal Pfis- ter, and Benjamin F Grewe. Approximating the predictive distribution via adversarially-trainn hypemnetwrrs.. In Bayesian Deep Learning Workshop. NeurlPS (Spotlight), volume 2018, 2018.
Geoffrey E Hinton, Simon Osindero, and Yee-Whye Teh. A fast learning algorithm for deep belief nets. Neural computation, 18(7):1527-1544 2006.
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.0486,, 2017.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Sylwester Klocek, Lukasz Maziarka, Maciej Wolczyk, Jacek Tabor, Marek Smieja, and Jakub Nowak. Hypernetwork functional image representation. arXiv preprint 19XXiv::002.0000 2019.
Jan Koutnik, Faustino Gomez, and Jürgen Schmidhuber. Evolving neural networks in compressed weight space. In Proceedings of the 12th annual conference on Genetic and evolutionary compu- tation, pp. 119-626. ACM, 2010.
Agustinus Kristiadi and Asja Fischer. Predictive uncertainty quantification with compound density networks. arXiv preprint arXiv: (0t.::/902.000000 2019.
David Krueger, Chin-Wer Wei Huang, Riashat Islam, Ryan Turner, Alexandre Lacoste, and Aaron Courville. Bayesian hypernetworks. arXiv preprint 1rX.v:1710.04759, 2017.
Sören Laue, Matthias Mitterreiter, and Joachim Giesen. Computing higher order derivatives of matrix and tensor expressions, In Advances in Neural Information Processing Systems, 2755- 2764, 2018
Zechun Liu, Haoyuan Mu, Xiangyu Zhang, Zichao Guo, Xin Yang, Tim Kwang-Ting Cheng, and Jian Sun. Metapruning: Meta learning for automatic neural network channel pruning. arXiv preprint arXiv: 10258, 2019.
Jonathan Lorraine and David Duvenaud. Stochastic hyperparameter optimization through hypernet- works. arXiv preprint arXiv: 1802.09419, 2018
Elliot Meyerson and Risto Mükkkulaienn. Modular universal reparameterizatton: Deep multi-task learning across diverse domains. arXiv preprint arXiv: 000.19960009990 2019.
10

---

Published as a conference paper at ICLR 2020
Zheyi Pan, Yuxuan Liang, Junbo Zhang, Xiuwen Yi, Yong Yu, and Yu Zheng. Hyperst-net: Hyper- networks for pratio-tempordl forecasting. arXiv preprint arXiv: 10889, 2018.
Nick Pawlowski, Andrew Brock, Matthew CH Lee, Martin Rajchl, and Ben Glocker. Implicit weight uncertainty in neural networks. arXiv preprint arXiv: 2017.
Neale Ratzlaff and Li Fuxin, Hypergan: A generative model for diverse, performant neural networks. arXiv preprint arXiv: 1058, 2019.
Sashank J. Reddi, Satyen Kale, and Sanjiv Kumar. On the convergence of adam and beyond. In International Conference on Learning Representations, 2018. URL /tL/tttpss//opeeeevviee
net/Eorum?ida
RZ
David E Rumelhart, Geoffrey E Hintont, and Ronald Williams. Learning representations by back- propagating errors. NATURE, 323:9, 1986.
Andrew M Saxe, James L McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynam- ics learning in deep linear neural networks. arXiv preprint 132.v:1312.61200, 2013.
Joan Serrà, Santiago Pascual, and Carlos Segura. Blow: single-scale hyperconditioned flow for non-parallel raw-audio voice conversion. arXiv preprint arXiv:1966.00 1906.00794 2019.
Falong Shen, Shuicheng Yan, and Gang Zeng. Meta networks for neural style transfer. arXiv preprint arXiv:1709.04111, 2017.
Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. Striving for simplicity: The all convolutional net. arXiv preprint arXiv: /4/2.6806 2014.
Kenneth O Stanley, David D' Ambrosio, and Jason Gauci. A hypercube- based encoding for evolv- ing large-scale neural networks. Artificial life. 15(2):185-21) 2009.
Joseph Suarez. Language modeling with recurrent highway hypernetworks. In Advances in neural information processing systems, pp. 3267-3276, 2017.
Zhun Sun, Mete Ozay, and Takayuki Okatani. Hypernetworks with statistical filtering for defending adversarial examples. arXiv preprint arXiv:171L.01791, 2017.
Kenya Ukai, Takashi Matsubara, and Kuniaki Uehara. Hypemnetwrrk-basee based implicit posterior es- timation and model averaging of cnn. In Asian Conference on Machine Learning, pp. 176-191, 2018.
Johannes von Oswald, Christian Henning, João Sacramento, and Benjamin F Grewe. Continual learning with hypernetworks. arXiv preprint arXiv: (ee::990600999 2019.
Ashia C Wilson, Rebecca Roelofs, Mitchell Stern, Nati Srebro, and Benjamin Recht. The marginal value of adaptive gradient methods in machine learning. In Advances in Neural Information Processing Systems, pp. 41588, 2017.
Yuxin Wu and Kaiming He, Group normalization. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 3-19, 2018.
Chris Zhang, Mengye Ren, and Raquel Urtasun. Graph hypernetworks for neural architecture search. arXiv preprint arXiv:1810.05749, 2018.
Hongyi Zhang, Yann N Dauphin, and Tengyu Ma. Fixup initialization: Residual learning without normalization. arxiv preprint arXiv: arXiv:/901. 1wt./1911.....0 2019.
11

---

Published as a conference paper at ICLR 2020
APPENDIX
A RE-USING HYPERNET WEIGHTS
A.1 FOR MAINNET WEIGHTS THE SAME SIZE
For model compression or weight-sharing purposes, different parts of the mainnet might be gen- crated by the same hypernet function. This will cause some assumptions independence in our analysis to be invalid. Consider the example of the same hypernet being used to generate multiple different mainnet weight layers of the same size, i.e. [ H(t + 1)fet2 d d -, Then, x(t 1J [| =[(])* JL WIt + 1] H Mifteeeet ellt + 1Jket1.
The relaxation some of these independence assumptions does not always prove to be a big prob- lem in practice. because the correlations introduced by repeated use of H can be minimized with the use of flat distributions like the uniform distribution. can even be helpful, since the re-use of the same hypernet for different layers causes the gradient flowing through the hypernet output layer OL Ee
to be the sum of the gradients from the weights of these layers: ah(e)*
OL
H thus
combating the shrinking effect.
A.2 FOR MAINNET WEIGHTS OF DIFFERENT SIZES
Similar reasoning applies if the same hypernet was used to generate differently sized subsets of weights in the mainnet. However, encourage avoiding this kind of hypernet architecture design not otherwise essential, since will complicate the initialization formulae listed in Table [
Consider Ha et al. 2016)'s hypernetwork architecture. Their two-layer hypernet generated weight chunks of size (K, n. n) for a main convolutional network where K 16 was found to be the highest common factor among the size of mainnet layers, and n² 9 the size of the receptive field. We simplify the presentation by writing for ét. j for jt. k for ki.m. and for It.-
W[t] = H illod K)
aatili + L d + Ge(mod K) 5 K)s(med K) [He K) aft[[e + Id,, gitmod k0]
if i is divisible by K
if j is divisible by K
a[t[[m;] G[tl[mo]e [fe]|mm] + [+|
(6)
Because the output layer (H, in the hypernet was re-used to generate mainnet weight matrices of different sizes (i.c. in general, i, a i441, # ft+1), effectively becomes the output layer that we want to be considering for hyperfan-in and hyperfan-out initialization.
Hence, to achieve fan-in in the mainnet Var(W[!))) d we have to use fan-in init for H (i.e.
Var H (mod K)
*
a,d, Vad. and hyperfan-in init for G (i.e. Var(G'[[[mm)ft)
Varc 15.
Analogously, to achieve fan-out in the mainnet Var(W((d)) = +.- we have to use fan-in init for H 1 * a.d. v.d.var(e()..)))>> and hyperfan-out init for G (i.e. Var(G]t][mm]1)) =
(i.e. Var( Var(Hr (Hodd K)) = de
d,d. Var(e [[m.)))
12

---

Published as a conference paper at ICLR 2020
B MORE EXPERIMENTAL DETAILS
B.1 FEEDFORWARD NETWORKS ON MNIST
The networks were trained on MNIST for 30 epochs with batch size 10 using a learning rate of 0.0005 for the hypernets and 0.01 for the classical network. The hypernets had one linear layer with embeddings of size and different hidden layers in the mainnet were all generated by the same hypernet output layer with a different embedding, which was randomly sampled from ((-v3, v3) and fixed. We use the mean cross entropy loss for training, but the summed cross entropy loss for testing.
We show activation and gradient plots for two cases: (i) the hypernet generates only the weights of the mainnet, and (ii) the hypernet generates both the weights and biases of the mainnet. covers
Figures 3
9 10 and
14 15
and 16 (ii) covers Figures 17
18
19
20
21
22
23
24 25
The activations and gradients in our plots were calculated by averaging across a fixed held-out set of 300 examples drawn randomly from the test set.
In Figures α
16
26 and 29 the axis shows the number
of activations/gradiintt, while the x axis shows the value the activations//radients. The value of activations/gradiints from the hypernet output layer correspond to the value of mainnet weights.
In Figures 2 tions/gradients while each increment the axis corresponds measurement that was taken every 1000 training batches, with the bars denoting one standard deviation away from the mean.
25 and 28 the y axis shows the mean value of the activa-
13

---

Published as a conference paper at ICLR 2020
B.1.1 HYPERNET GENERATES ONLY THE MAINNET WEIGHTS
-14
120
272
120
272
373
Figure 7: Evolution of Mainnet Activations during Training on MNIST.
24
AM
8.2
6.5
28 JUSS -2X 7o0 K3%
635
-224 K25
3.5 875
-0.71 4.34 2.50
Figure 8: Mainnet Activations at the End of Training on MNIST.
Cae
Figure 9: Mainnet Gradients before the Start of Training on MNIST.
14

---

Published as a conference paper at ICLR 2020
Pasder 2014
Mawler Hipper
Hyperkmon
Hypertan-st
Figure 10: Evolution of Mainnet Gradients during Training on MNIST.
Myper
Ibpartnnct
10
Figure 11: Mainnet Gradients at the End of Training on MNIST.
102
Xaxier Otyper
3e3
Hyperfan-in
245
Hyperfan-out
layer three byyer four
tree
Uroz
Wye fove
Numbor Actioat or
byerr four
6
Nomber
-5005-6300-6005 5,000 5,05 5,50 6:355 Acciation Valuz
-115
-0.10
-0.05
0.00
0.05
0.00
0.35
-0.15
-0.10
-8.05
0.00
6,55
620
035
Adtration
Figure 12: Hypernet Output Layer Activations before the Start of Training on MNIST.
1.45
Xavier (Hyper)
1+3
Hyporfan-in
1+3
Hyperfan-out
2.5
shres
three oyer four
Mkmber inttaattee
bayer faur
byver four
MATOOE Of ATT 20
2.5
-33
.02
-0.3
0.5
6.1
5.2
63
20 6.15 -110
-6.05 0.00 0.5 015
635 0.28
-615 0010 -.05 0.05 6.5 030 015 323 4425
Artiaation Valle
Actinttor Valae
Actsatior Valua
Figure 13: Hypernet Output Layer Activations at the End of Training on MNIST.
15

---

Published as a conference paper at ICLR 2020
Xavier (Hyper
Hyportan-n
Hyperton out
Pgeer the
9/er one
bayer one
lppr uau
laysr lnc
aayer Thrie
ayer
layer
Wirber Gradert
Mamber Graderta
Wimber Gradtrit
2.5
layer
2.0
1x-2
Gndeettvalee
2e-3
Figure 14: Hypernet Output Layer Gradients before the Start of Training on MNIST.
1x-66
(Hypar)
Hyperfan-in
1c-1
out
layer
layer
Coutert
Yiar
3.5
275
122
172
75
122
250
Lx
Figure 15: Evolution of Hypernet Output Layer Gradients during Training on MNIST.
(Hyperi
Hyperfan
Hyperfon
3.0
layer lsyer thre
Payer thene la four
syer tiee sysr four
Grsdente
Nanbar Gradett
Namber Grudente
laysr four
1.00
0.75
5.50
0:25
2.00
0.25
030
0.75
1.00
-5
-2
Figure 16: Hypernet Output Layer Gradients at the End of Training on MNIST.
16

---

Published as a conference paper at ICLR 2020
B.1.2 HYPERNET GENERATES BOTH MAINNET WEIGHTS AND BIASES
Training Loss
Test Loss
Test Accuracy
2ay er litper)
0.250
Rowler (Mpee])
typerton
68
6:225
0,200
6:279
0.02
0.229
0.03
6,100
Hypertanin Pspertaniott
0.073
Esscht
Figure 17: Loss and Test Accuracy Plots MNIST.
Xavier Dhypart
Hyperfan-in
Hyperfan-ozz
aedrore
layck
23
three
laper
Dayer Tile
buyer flve
-100 -2.75
-930
-0.25
0.00
0.0
0.75
0..00 -.70--775
-3.50-0225
0.32
0.73
2.02
-0. -26 -5A
-3.2
022
6.2
0.4
0.
0.8
Acciaaton Valus
Activak on Value
Acdivation Value
Figure 18: Mainnet Activations before the Start of Training on MNIST.
(lyper)
Hyperfan
Hypertan-out
0.4
layer Unree
terez layer four byyer
0.6
layer four
By
Mear Aetvadte
-02
lgys
Sgyar theer Rayer four
-0.5
25
50
75
202
125
250
775
50
200
125
250
175
23
50
75
200
125
250
175
Earattoos
Figure 19: Evolution of Mainnet Activations during Training on MNIST.
17

---

Published as a conference paper at ICLR 2020
Xavier [Hyperi
Hypertaniin
Hypertan
80
byae four
layer Uriee
byye four
loyor
layer fou"
layer fire
layer fiee
40
laysr lwee
Number
20
10
-6.75
-150
-2.25
0.00
325
5.50
0.75
-,00 -0.7 -0.50 0.5---25
5.0 3.25 0.50
0.75
1.00
-.000 -0.73 -250
-2.25
2.03
0.25
2.50
0.75
1.00
Actiration
Actical or Value
Activation Value
Figure 20: Mainnet Activations at the End of Training on MNIST.
Xavier [Hyper]
Hyper farwin
Hyperfan-out
byerrnn one
byyr one
bayer one
layot Ueo
kayo bwy byer thris
lay und
byar layer
layer three
Gonderte
layer
layer
20
20
10
Graskenc
Se-5
Gradient
1e-5
GradentVvakz
Figure 21: Mainnet Gradients before the Start of Training on MNIST.
1e-4
Xavier (Hypar)
Ie-5
Hyperfan-in
Le-4
Hyper fam-out
laper layer four
kpptr oric
koyo ond
Vwan Gra VoL
layer five
liver five
-0.3
-1.0
100
250
375
50
120
150
22
120 Ic Scharaaions
125
250
1k. Bar stoma
14 Schrratooa
Figure 22: Evolution of Mainnet Gradients during Training on MNIST.
Xavier (fyper)
Myperfan-in
Iyperfesout
Tayer layer byee-Tu Your lxyer
laysr 514 lays
lgsr ors
loyer
Diyer Sea
byer Sur
Diyer Tun
byee fuv
20
Gradentraaie
32-3
Gradlnt
Gradieen
Figure 23: Mainnet Gradients at the End of Training on MNIST.
18

---

Published as a conference paper at ICLR 2020
Xacier Otyper
Hyperfan-in
Hyperfan-out
layer byee tao layer Utree byyer
Wyer ten
pyer twn sye Unoc
kayor Uiree
Kumber Schoodrre
byyer
M Number Acbratorr
ayer
-
-2.00
-2,02 3.00
0.00
-1.3
0.00
-6.15
6.35 0.00
Figure 24: Hypernet Output Layer Activations before the Start of Training on MNIST.
Xavier (Hyper)
Hyperfan-in
Hyperfar-out
layer ane layer tno
syer two
byer
1.0
byer thres
MAAY Actiaater Valce
layer
0.5
byer one
laper
275
202
300
130
Figure 25: Evolution of Hypernet Output Layer Activations during Training on MNIST.
Xavier [Hyper)
Hyperfan-in
Hyperfan-out
layel layer tao layer theee layer
layor Byer two layer Uh ez layer
1.0
Syer two
kyyer Uvee
2.2
0.6
kyye
0.6
0.9
Parter
0.4
82
0.2
0.0
-3.00
0.00
-22
-0.1
0.2
Figure 26: Hypernet Output Layer Activations at the End of Training on MNIST.
Xavier (Hyper)
Hyperfan-in
Hyperfan-out
byeer one layor owo byerr loyer
layer one layor tnc
pyer one
kye two
layer time
ayer trree
Contiets
Grasoors
layer
Number Crazlapa
2n-3
Figure 27: Hypernet Output Layer Gradients before the Start of Training on MNIST.
19

---

Published as a conference paper at ICLR 2020
12-3
Xavier (Hyper)
Hyperfansin
Hyperfattout
layer
layer
Meen
Menn
Usez
syer far
125
25
300
152
335
125
Te Ineration
18 Irni one
Figure 28: Evolution of Hypernet Output Layer Gradients during Training on MNIST.
(Hyper)
Lec
Hyperfan-in
Hyperfan-out
laper
fur
Pumber of 22
-6
Grad A0 Valus
GraclentVolil
Grudient Vilue
1=-3
Figure 29: Hypernet Output Layer Gradients at the End of Training on MNIST.
20

---

Published as a conference paper at ICLR 2020
B.1.3 REMARK ON THE COMBINATION FAN-IN AND FAN-OUT INIT
Glorot & Bengio 2010) proposed to use the harmonic mean of the two different initialization for- mulae derived from the forward and backward pass. He et al. (2013) commented that either version suffices for convergence, and that it does not really matter given that the difference between the two will be a depth- depth-indeponn independent factor.
We experimented with the harmonic, geometric, and arithmetic means of the two different formulae in both the classical and the hypernet case. There was no indication of any significant benefit from taking any of the three different means in both cases. Thus, we confirm and concur with He et al. 2015 's original observation that either the fan-in or the fan-out version suffices.
B.2 CONTINUAL LEARNING ON REGRESSION TASKS
The mainnet is a feedforward network with two hidden layers (10 hidden units) and the ReLU activation function. The weights and biases of the mainnet are generated from hypernet with two hidden layers (10 hidden units) and trainable embeddings of size 2 sampled from (3, v3). We keep the same continual learning hyperparameter Boutpus value of 0.005 and pick the best learning rate for each initialization method from {10 2, 10 -0- 10 10 $3. Notably, Kaiming (fan-in) could only be trained from learning rate 10-5, with losses diverging soon after initialization using the other learning rates. Each task was trained for 6000 training iterations using batch size 32, with Figure plotted from losses measured at every 100 iterations.
B.3 CONVOLUTIONAL NETWORKS ON CIFAR-10
The networks were trained on CIFAR-10 10 for 500 epochs starting with an initial learning rate of 0.0005 using batch size 100, and decaying with 7 0.1 at epochs 350 and 450. The hypernet is composed of two layers (50 hidden units) with separate embeddings and separate input layers but shared output layers. The weight generation happens in blocks of (96, 3, 3) where
96 is the highest common factor between the different sizes of the convolutional layers in the mainnet and n 3 is the size of the convolutional filters (see Appendix Section A.2 for a more detailed explanation on the hypernet architecture). The embeddings are size 50 and fixed after random sampling from u(-v3, v3). We use the mean cross entropy loss for training, but the summed cross entropy loss for testing.
B.4 BAYESIAN NEURAL NETWORK ON IMAGENET
Ukai et (2018) showed that Bayesian neural network can be developed by using hypemet- work to express prior distribution without substantial changes to the vanilla hypernetwork setting. Their methods simply require putting Ze-rglularizttion on the model parameters and sampling from stochastic embeddings. We trained linear hypernet generate the weights of MobileNet main- net architecture (excluding the batch normalization layers), using the block-wise sampling strategy described in [Kai et al. (2018), with factor of 0.0005 for the Le-reuularization. We initialize fixed embeddings of size 32 sampled from (d-- /3, /3), and sample additive stochastic noise coming from 1(, 0.1) at the beginning of every mini-batch training. The training was done on Ima- geNet with batch size 256 and learning rate 0.1 for 25 epochs, or equivalently, 125125 iterations. The testing was done with 10 Monte Carlo samples. We omit the test loss plots due to the computa- tional expense of doing 10 forward passes after every mini-bateh instead of every epoch.
21


Reviews:
[{"review_id": "rkxlxvd2KB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents an extension of Glorot/He weight variance initiazation formula for the hypernetworks. Hypernetworks are the class of neural networks where one (hyper) model generates the weights for the another (main) network, introduced by Ha et.al in 2016.\nAuthors argue and show via experiments that standard weight init formulas do not work for hypernetworks, resulting in vanishing or exploding activations and re-derive the formula for convolutional/fully-connected networks + ReLU. \nThey show that proposed method allows  hypernet training when the standard ways don`t. \n\nThe technical contribution seems as logical and straightforward yet necessary step for hypernetwork-related research.\n\nQuestions:\n \n - In standard NNs, initialization issues are mostly solved after introduction of BatchNorm. Wouldn`t it be the case for hypernetwork as well: to just add BN layers between main net layers?\n \n - Figure 2. What are is the y axis of the figure? Norm? Variance? Mean? The same question for the most of Figures in appendix \n \n- It would be nice to see how proposed method stands vs mentioned heuristics like M3 and M4. \n \nOverall I like the paper. \n\nMinor comments:\n\n \n > \"This fundamental difference suggests that conventional knowledge about neural networks may not\napply directly to hypernetworks and radically new ways of thinking about weight initialization,\noptimization dynamics and architecture design for hypernetworks are sorely needed.\"\n\nI don`t see anything \"radically new\" in re-derivation of Xavier formula to the new type of network.\n\n\n======\nRevision:\n\nRevised version addressed my concerns and the batchnorm-related experiment is indeed surprising.\nOverall, I like the paper and increase evaluation to strong accept. \n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJlpawuhFB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Principled Weight Initialization for Hypernetworks\n\nSummary:\n\nThis paper investigates initialization schemes for hypernets, models where the weights are generated by another neural network, which takes as input either a learned embedding or the activations of the model itself. Standard initialization schemes for deep networks (Glorot, He, Fixup) are based on variance analyses that aim to keep the scale of activations/gradients well-behaved through forward-backward propagation, but using this approach is ill-founded for hypernets where the output is a set of weights rather than e.g. a softmax’d classification output. This paper extends the standard variance analysis to consider the hypernet case by investigating what the choice of hypernet initialization should be if one still wishes to maintain the well-behaved activations/gradients in the main model. The authors present results showing the evolution of hypernet outputs with their scheme are better-behaved, demonstrate that their modification leads to improved stability for hypernet-based convnets on CIFAR  (over a model which, for the standard choice of He init, basically does not train) and improved performance on a continual learning task.\n\nMy take:\n\nThis is an \"aha!\" or an “obvious in retrospect” paper: a simple idea based on noticing something being done wrong in practice with a fairly straightforward fix, coupled with a decent empirical evaluation and analysis. The paper is well-written and reasonably easy to follow (although I am not familiar with Ricci calculus I did not feel that I was flummoxed at any point during the maths), and the potential impact is decent: any future work which employs a hypernetwork would likely do well to consider the methods in this paper. While I would like to see the empirical evaluations taken a bit further, I think this paper is a solid accept (7/10) and would make a good addition to ICLR2020.\n\nNotes:\n\nWhile this is a good paper, I think the impact of the paper could be magnified if the authors were a bit more ambitious with their empirical evaluations. This method seems to enable training hypernets in settings which would previously have been unstable; it would be good to more thoroughly characterize robustness to hyperparameters or otherwise demonstrate additional practical value. Showing results on ImageNet would also be helpful (I’m well aware this is not always in the realm of compute possibility) or just showing progress on a more challenging task outside of CIFAR or a small continual learning problem would, I think, greatly increase the chances that this paper catches on. As this is a “unleash your potential” note, I have not taken this sentiment into account in my review (as should hopefully be evident from my accept score).\n\nMinor:\n\n-Caption for figure 2 should indicate what kind of magnitudes are represented—are these the average weight norms in each layer vs epochs? The axes should be labeled.\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "BJlfwPbitH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Review of “Principled Weight Initialization for Hypernetworks”\n\nThere has been a lot of existing work on neural network initialization, and much of this work has made large impact in making deep learning models easier to train in practice. There has also been a line of work on indirect encoding of neural works (i.e. HyperNEAT work of Stanley, and more recent Hypernetworks proposed by Ha et al) which showed promising results of training very large networks (in the case of Stanley), or have network weights that can adapt to the training data (in the case of Hypernetworks), and these approaches have been shown to be useful in applications such as meta-learning or few-shot learning (i.e. [1]). However, as far as I know, there hasn't been any work that looks at a principled way of initializing the weights of a weight-generating network, which this work tries to explore.\n\nMaking the observation (and claim) that traditional init methods don't init hypernetworks properly, they propose a few techniques to initialize hypernetworks (\"Hyperfan\"-family), which are justified in a similar way as original classical init techniques (i.e. preserving variance like in Xavier init), and they demonstrate that their method works well for feed forward networks on MNIST, CIFAR-10 tasks compared to traditional classical init methods, as well for a continual learning task.\n\nI liked the paper as they identified a problem that hasn't been studied, and proposed a reasonable method to solve it. Their method may be able to make Hypernetworks accessible to many more researchers and practitioners, the way classifical init techniques have made neural net training more accessible.\n\nThere are a few things that could improve the paper (and get an improvement score from me). The authors don't have to do all of these, but just a few suggestions:\n\n1) The experiments, to my understanding, are all feed forward networks. How about RNNs or LSTMs?\n\n2) Are there any (interesting) tasks that use Hypernetworks that are not trainable with existing methods, but made trainable using this proposed scheme?\n\n3) Would this method also work with HyperNEAT [2] or Compressed Network Search [3]? (probably should cite that line of work too). In [3], a research group at IDSIA used DCT compression to compress millions of weights into a few dozen parameters, so would be interesting if the approach will work on similar \"learn-from-pixels\" RL experiments.\n\nI'm assigning a score of 6 (it's currently like a \"really good\" workshop paper, but a normal conf paper IMO), but I like this paper and would like to see the authors make an attempt to improve it, so I can improve the score to see it get accepted with a higher certainty.\n\nGood luck!\n\n[1] i.e. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6519722/ https://arxiv.org/pdf/1710.03641.pdf https://arxiv.org/pdf/1703.03400.pdf\n\n[2] http://eplex.cs.ucf.edu/hyperNEATpage/\n\n[3] http://people.idsia.ch/~juergen/compressednetworksearch.html\n\n*** Revised Score ***\n\nNov20: Upon reading the other reviews, and looking at the changes to the paper with the extra citations, I'm improving the score to 8. (For the record, if this was a 1-10 scale, I would have liked my score to be a 7).", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: All the reviewers agreed that this was a sensible application of mostly existing ideas from standard neural net initialization to the setting of hypernetworks.  The main criticism was that this method was used to improve existing applications of hypernets, instead of extending their limits of applicability.
Decision: Accept (Talk)