Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: Principled Weight Initialization for Hypernetworks

Summary:

This paper investigates initialization schemes for hypernets, models where the weights are generated by another neural network, which takes as input either a learned embedding or the activations of the model itself. Standard initialization schemes for deep networks (Glorot, He, Fixup) are based on variance analyses that aim to keep the scale of activations/gradients well-behaved through forward-backward propagation, but using this approach is ill-founded for hypernets where the output is a set of weights rather than e.g. a softmax’d classification output. This paper extends the standard variance analysis to consider the hypernet case by investigating what the choice of hypernet initialization should be if one still wishes to maintain the well-behaved activations/gradients in the main model. The authors present results showing the evolution of hypernet outputs with their scheme are better-behaved, demonstrate that their modification leads to improved stability for hypernet-based convnets on CIFAR  (over a model which, for the standard choice of He init, basically does not train) and improved performance on a continual learning task.

My take:

This is an "aha!" or an “obvious in retrospect” paper: a simple idea based on noticing something being done wrong in practice with a fairly straightforward fix, coupled with a decent empirical evaluation and analysis. The paper is well-written and reasonably easy to follow (although I am not familiar with Ricci calculus I did not feel that I was flummoxed at any point during the maths), and the potential impact is decent: any future work which employs a hypernetwork would likely do well to consider the methods in this paper. While I would like to see the empirical evaluations taken a bit further, I think this paper is a solid accept (7/10) and would make a good addition to ICLR2020.

Notes:

While this is a good paper, I think the impact of the paper could be magnified if the authors were a bit more ambitious with their empirical evaluations. This method seems to enable training hypernets in settings which would previously have been unstable; it would be good to more thoroughly characterize robustness to hyperparameters or otherwise demonstrate additional practical value. Showing results on ImageNet would also be helpful (I’m well aware this is not always in the realm of compute possibility) or just showing progress on a more challenging task outside of CIFAR or a small continual learning problem would, I think, greatly increase the chances that this paper catches on. As this is a “unleash your potential” note, I have not taken this sentiment into account in my review (as should hopefully be evident from my accept score).

Minor:

-Caption for figure 2 should indicate what kind of magnitudes are represented—are these the average weight norms in each layer vs epochs? The axes should be labeled.


review_title: Official Blind Review #1

final_score: rating: 8: Accept
review_content: The paper presents an extension of Glorot/He weight variance initiazation formula for the hypernetworks. Hypernetworks are the class of neural networks where one (hyper) model generates the weights for the another (main) network, introduced by Ha et.al in 2016.
Authors argue and show via experiments that standard weight init formulas do not work for hypernetworks, resulting in vanishing or exploding activations and re-derive the formula for convolutional/fully-connected networks + ReLU. 
They show that proposed method allows  hypernet training when the standard ways don`t. 

The technical contribution seems as logical and straightforward yet necessary step for hypernetwork-related research.

Questions:
 
 - In standard NNs, initialization issues are mostly solved after introduction of BatchNorm. Wouldn`t it be the case for hypernetwork as well: to just add BN layers between main net layers?
 
 - Figure 2. What are is the y axis of the figure? Norm? Variance? Mean? The same question for the most of Figures in appendix 
 
- It would be nice to see how proposed method stands vs mentioned heuristics like M3 and M4. 
 
Overall I like the paper. 

Minor comments:

 
 > "This fundamental difference suggests that conventional knowledge about neural networks may not
apply directly to hypernetworks and radically new ways of thinking about weight initialization,
optimization dynamics and architecture design for hypernetworks are sorely needed."

I don`t see anything "radically new" in re-derivation of Xavier formula to the new type of network.


======
Revision:

Revised version addressed my concerns and the batchnorm-related experiment is indeed surprising.
Overall, I like the paper and increase evaluation to strong accept. 


review_title: Official Blind Review #3

final_score: rating: 8: Accept
review_content: Review of “Principled Weight Initialization for Hypernetworks”

There has been a lot of existing work on neural network initialization, and much of this work has made large impact in making deep learning models easier to train in practice. There has also been a line of work on indirect encoding of neural works (i.e. HyperNEAT work of Stanley, and more recent Hypernetworks proposed by Ha et al) which showed promising results of training very large networks (in the case of Stanley), or have network weights that can adapt to the training data (in the case of Hypernetworks), and these approaches have been shown to be useful in applications such as meta-learning or few-shot learning (i.e. [1]). However, as far as I know, there hasn't been any work that looks at a principled way of initializing the weights of a weight-generating network, which this work tries to explore.

Making the observation (and claim) that traditional init methods don't init hypernetworks properly, they propose a few techniques to initialize hypernetworks ("Hyperfan"-family), which are justified in a similar way as original classical init techniques (i.e. preserving variance like in Xavier init), and they demonstrate that their method works well for feed forward networks on MNIST, CIFAR-10 tasks compared to traditional classical init methods, as well for a continual learning task.

I liked the paper as they identified a problem that hasn't been studied, and proposed a reasonable method to solve it. Their method may be able to make Hypernetworks accessible to many more researchers and practitioners, the way classifical init techniques have made neural net training more accessible.

There are a few things that could improve the paper (and get an improvement score from me). The authors don't have to do all of these, but just a few suggestions:

1) The experiments, to my understanding, are all feed forward networks. How about RNNs or LSTMs?

2) Are there any (interesting) tasks that use Hypernetworks that are not trainable with existing methods, but made trainable using this proposed scheme?

3) Would this method also work with HyperNEAT [2] or Compressed Network Search [3]? (probably should cite that line of work too). In [3], a research group at IDSIA used DCT compression to compress millions of weights into a few dozen parameters, so would be interesting if the approach will work on similar "learn-from-pixels" RL experiments.

I'm assigning a score of 6 (it's currently like a "really good" workshop paper, but a normal conf paper IMO), but I like this paper and would like to see the authors make an attempt to improve it, so I can improve the score to see it get accepted with a higher certainty.

Good luck!

[1] i.e. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6519722/ https://arxiv.org/pdf/1710.03641.pdf https://arxiv.org/pdf/1703.03400.pdf

[2] http://eplex.cs.ucf.edu/hyperNEATpage/

[3] http://people.idsia.ch/~juergen/compressednetworksearch.html

*** Revised Score ***

Nov20: Upon reading the other reviews, and looking at the changes to the paper with the extra citations, I'm improving the score to 8. (For the record, if this was a 1-10 scale, I would have liked my score to be a 7).
review_title: Official Blind Review #2

Meta-review: All the reviewers agreed that this was a sensible application of mostly existing ideas from standard neural net initialization to the setting of hypernetworks.  The main criticism was that this method was used to improve existing applications of hypernets, instead of extending their limits of applicability.
Decision: Accept (Talk)