Title: HyperNetworks
Authors: David Ha
Andrew M. Dai
Quoc V. Le
Venue: ICLR 2017
Keywords: Natural language processing
Deep learning
Supervised Learning

Abstract: This work explores hypernetworks: an approach of using one network, also known as a hypernetwork, to generate the weights for another network.  We apply hypernetworks to generate adaptive weights for recurrent networks. In this case, hypernetworks can be viewed as a relaxed form of weight-sharing across layers. In our implementation, hypernetworks are are trained jointly with the main network in an end-to-end fashion.  Our main result is that hypernetworks can generate non-shared weights for LSTM and achieve state-of-the-art results on a variety of sequence modelling tasks including character-level language modelling, handwriting generation and neural machine translation, challenging the weight-sharing paradigm for recurrent networks.

TL;DR: We train a small RNN to generate weights for a larger RNN, and train the system end-to-end.  We obtain state-of-the-art results on a variety of sequence modelling tasks.

Review discussion:
Can you explain how do you obtain the layer embedding vector like z^j in eq(1)? Do you map the layer index j represented by a one-hot vector to z^j like word index to a word vector in language modeling?
Is It correct that the run-time complexity of deep CNNs generated by the hyperNetwork is the same as the conventional main network, as the hypernetwork will be discarded, while for RNNs with hypernetworks, the run-time complexity is larger than conventional RNNs, as the hypernetwork should also operate for each time step?
For the static hypernetworks described in Section 3, and in eq(1), the embedding vectors of each layer (z^j) are also learned parameters, and they are learned end-to-end during training of the image classification task along with the rest of the model.
So as you described, we could instead choose to represent each z^j by a one-hot, say determined by the layer index, and map z^j to the embedding space with an embedding matrix and learn that matrix end-to-end during training.  This would be equivalent to learning the embedding vectors directly.
For the run-time complexity, in the CNN case described in section 3, the extra run-time required will be in the order of the feeding the set of embedding vectors (z^j) through the hypernetwork to produce the set of kernels K^j.  During inference, the set of kernels K^j can be computed just once by the hypernetwork, and stored to be used as many times as required by the main network, so in that sense the run-time complexity is largely dominated by the main network as you mentioned.  During training, however, the hypernetwork will still be utilized at each forward and backward pass of every mini-batch, but in practice, we notice that the training time per mini-batch is ~ only 10% slower than the normal convnet case for MNIST and CIFAR10 experiments, as most of the complexity still resides in the main network.
In the RNN case, you are correct that the run-time complexity is larger than conventional RNNs, as the hypernetwork is operated at each timestep.
Please let us know if you have any more questions or concerns, thanks.
To reduce the num. of parameters: in a linear element, low-rank factorization is often used, aka. linear bottleneck, often with mean or batch normalization.
Have the authors tried such technique? E.g. low-rank factorized representation of K^j.
For RNNs, Gated Recurrent Convolutional Units have also been tried in the literature.
In section 3.1:
the authors argue against the one-layer hyper network.
However using embedding vector z which depends on j (depth) and i (num of input, 1..N_{in}) would be also an option.
In eq 2, W_i*z^j + b_i := z_i,j would exactly result in this, and would not necessarily increase the num. of parameters -> no need for two layers.
Could the authors outperform the best result (or achieve the same) in Table 2 with less parameters, e.g. <=3.77%, <13.3M ?
Hi, thanks for the useful pointers.  For the RNN case, we have made an ongoing effort to try to improve the weight augmentation approach with low-rank matrix approximation, specifically using the HyperRNN to generate an additive (or multiplicative) low-rank 'fudge' to existing weight matrix of the main RNN.  I found that the current row-multiplicative method in eq 7-8 still offers the best performance while keeping computational performance relatively efficient, compared to low-rank factorized versions.  It may be possible to combine both approaches to squeeze out more performance at the expense of computational time / additional memory.
For section 3.1, your description is an elegant way to formulate the approach, by having the embedding vector z which depends on both j and i.
We have not tried to beat the SOTA results on CIFAR-10, as currently I found it more challenging to train much larger versions of a hypernetwork for CNN.  I'm actually trying out an approach of using hypernetworks for model distillation (https://arxiv.org/abs/1503.02531), which may help cope with some of the challenges.
For example, taking a pre-trained ensemble of ResNets, and recording the logits produced for the training set, and then training the hypernetwork version of the same ResNet to learn the logits.  I think learning soft-labels will help with both the training stability and performance.  This is still an ongoing effort for the future work.
*** Paper Summary ***
The paper proposes to a new neural network architecture. The layer weights of a classical network are computed as a function of a latent representation associated with the layer. Two instances are presented (i) a CNN where each layer weight is computed from a lower dimensional layer embedding vector; (ii) an RNN where each layer weight is computed from a secondary RNN state.
*** Review Summary ***
Pros:
- I like the idea of bringing multiplicative RNNs and their predecessors back into the spotlight.
- LM and MT results are excellent.
Cons:
- The paper could be better written. It is too long for the conference format and need refocussing.
- On related work, the relation with multiplicative RNN and their generic tensor product predecessor (Order 2 networks, wrt C. Lee Giles definition) should be mentioned in the related work section and the differences with earlier research need to be explained and motivated (by the way it is better to say that something is revisiting an old idea or training it at modern scale/on modern tasks than ommitting it).
- on focus, it is not clear if your goal is to achieve better performance or more compact networks. In the RNN section you lean toward the former, in the CNN section you seem to lean toward the latter.
I would suggest to make the paper shorter and clearer possibly leaving the CNN results for latter publication. The relation with multiplicative/order 2 networks and eventual differences need to be explained.
*** Detailed Review ***
Multiplicative networks are an extremely powerfull architecture and bringing them back into the spotlight is excellent. This paper has excellent results but suffer poor presentation, lack of a clear focus. It spends time on details and ommit important points. In its current form, it is much too long to long and his not self contained without the appendices.
Spending more time on multiplicative RNNs, order 2 networks at the begining of the paper would be excellent. This will let you highlight the difference between this paper and earlier work. It would also be necessary to spend a little time on why multiplicative RNN were less used than gated RNN: it seems that the optimization problem their training involve is tricker and it would be helpful to explain whether you had a harder time tweaking optimization parameters or whether you needed longer training sessions compared to LSTMs, regular CNN. On name, I am not sure that "hypernetwork" help the reader understand better what the proposed architecture compared to multiplicative interactions.
In section 3.2, you seem to imply that there are different settings of hypernetworks that allow to vary from an RNN to a CNN, this is not clear to me, maybe you could show how this would work on a simple temporal problem with equations.
The work on CNN and RNN are rather disconnected to me: for CNN, you seem to be interested in a low rank structure of the weights, showing that similar performance can be achieved with less weights. It is not clear to me why to pursue that goal. Do you expect speedups? less memory for embedded applications? In that case you should compare with alternative strategies, e.g. model compression (Caruana et al 2006, aka Dark Knowledge, Hinton et al 2014) or hashed networks (Chen et al 2015).
For RNN, you seem to target better perplexity/BLEU and model compactness is not a priority.
Instead of making the weights have a simpler structure, you make them richer, i.e. dependent over time. It seems in that case models might be bigger and take longer to train. You might want to comment on training time, inference time, memory requirement in that case, as you highlight it might be an important goal in the CNN section.
Overall, I am not sure it helps to have this mixed message. I would rather see the paper fit in the conference format with the RNN results alone and a clearer explanation and defers the publications of the CNN results when a proper comparison with memory concerned methods is performed.
Some of the discussions are not clear to me, I am not sure what message the reader should get from Figure 2 or from the discussion on saturation statistics (p10, Figure 5).
Similarly, I am not sure if Figure 4 is showing anything: everything should change more drastically at word boundaries even in a regular LSTM (states, gates units should look very different before/after a space); without such a comparison it is hard to see if this is unique to your network.
The results on handwriting generation are harder to compare for me. Log-loss are hard to understand, I have no sense whether the difference between models is significant (what would be the variance in this metric under boostrap sampling of the training set?). I am not sold either on qualitative metric were human can assess quality but human cannot evaluate if the network is repeating the training set. Did you thing at precision/recall metric for ink, possibly with some spatial tolerance ? (e.g. evaluation of segmentation tasks in vision).
The MT experiments are insufficiently discussed in the main text.
Overall, I would suggest to make the paper shorter and clearer possibly leaving the CNN results for latter publication. You need to properly discuss the relation to multiplicative/order 2 networks and highlight the differences. Unclear discussion can be eliminated to make the experimental setup and the results presentation clearer in the main text.
*** References ***
M.W. Goudreau, C.L. Giles, S.T. Chakradhar, D. Chen, "First-Order Vs.
Second-Order Single Layer Recurrent Neural Networks,"IEEE Trans. on Neural Networks, 5 (3), p. 511, 1994.
Cristian Bucila, Rich Caruana, and Alexandru Niculescu-Mizil, "Model Compression," The Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD-2006), August 2006, pp. 535-541.
Dark knowledge, G Hinton, O Vinyals, J Dean 2014
W. Chen, J. Wilson, S. Tyree, K. Weinberger and Y. Chen, Compressing Neural Networks with the Hashing Trick, Proc. International Conference on Machine Learning (ICML-15)
Hi, AnonReviewer3,
Thanks for the review.  I appreciate the effort and detail you have put into writing the review and personally found it educational.  We took your suggestion to refocus the paper on the RNN section and have rewritten it with that in mind, leaving out the CNN section, and expanded on the Machine Translation section which was a bit lacking as you mentioned.
We have also followed your advice and added a related approaches section to describe the relation and differences with multiplicative RNNs and the other related works with the HyperRNN, in addition to adding the Second Order RNN work to the related works section.
In addition, we have improved style of the writing to make it more self contained and less dependent on the appendices, as you have outlined in the review.  Would appreciate it greatly if you can help us review the revised version of our paper, and provide us feedback on the improvements.
This paper proposes an interesting new method for training neural networks, i.e., a hypernetwork is used to generate the model parameters of the main network. The authors demonstrated that the total number of model parameters could be smaller while achieving competitive results on the image classification task.
In particular, the hyperLSTM with non-shared weights can achieve excellent results compared to conventional LSTM and its variants on a couple of LM talks, which is very inspiring.
--pros
This work demonstrates that it is possible to generate the neural network model parameters using another network that can achieve competitive results by a few relative large scale experiments. The idea itself is very inspiring, and the experiments are very solid.
--cons
The paper would be much stronger if it was more focused.
In particular, it is unclear what is the key advantage of this hypernetwork approach. It is argued that in the paper that can achieve competitive results using smaller number of trainable model parameters.
However, in the running time, the computational complexity is the same as the standard main network for static networks, such as ConvNet, and the computational cost is even larger for dynamic networks such as LSTMs. The improvements of hyperLSTMs over conventional LSTM and its variants seem mainly come from increasing the number of model parameters.
--minor question,
The ConvNet and LSTM used in the experiments do not have a large softmax layer. For most of the word-level tasks for either LM or MT, the softmax layer could be more than 100K. Is it going to be challenging for the hyperNetwork generate large number of weights for that case, and is it going to slowing the training down significantly?
Hi, AnonReviewer2,
Thanks for the review!  We agree that the style of the paper suffers from a lack of focus.  We have rewritten the paper significantly to focus on the RNN part. We’re happy to change the title of the paper to more RNN-focused, if that’s something the reviewer feels the right thing to do.  Please take a look and let us know your feedback.
As for the point you mentioned about how HyperLSTM’s improvements over LSTM come mainly from increasing the number of parameters, we tried to demonstrate in the Character PTB experiment, and also in the Handwriting Generation experiment, that the HyperLSTM can outperform conventional LSTMs with a lower parameter count.
For example, in the Char PTB experiment, we found that HyperLSTM with 1000 units can outperform an LSTM with 1250 units, and for the similarly in the handwriting generation experiment, the HyperLSTM with 900 hidden units outperform significantly the vanilla LSTM with 1000 units.  We hope that with the rewritten version that focuses on the RNN, it is easier to follow and digest the experimental results.
For your question regarding the softmax layer size:  For the MT experiment, the model does use a considerably large softmax layer size (32K for the GNMT architecture), although less than the 100K you mentioned.  We did not find it challenging for the HyperLSTM Cell to generate weights for the main LSTM.  The HyperLSTM Cell only had to generate a weight scaling vector of size 1000 (number of units of the main LSTM), which is not directly related to the size of the softmax layer.  We have expanded the MT section in the revised version of the paper, to try to emphasize the applicability of HyperLSTM to large-scale architectures such as GNMT.
Although the trainable parameters might be reduced significantly, unfortunately the training and recognition speech cannot be reduced in this way.
Unfortunately, as the results show, the authors could not get better results with less parameters.
However, the proposed structure with even more number of parameters shows significant gain e.g. in LM.
The paper should be reorganized, and shortened. It is sometimes difficult to follow and sometimes inconsistent.
E.g.: the weights of the feedforward network depend only on an embedding vector (see also my previous comments on linear bottlenecks), whereas in recurrent network the generated weights also depend on the input observation or its hidden representation.
Could the authors provide the num. of trainable parameters for Table 6?
Probably presenting less results could also improve the readability.
Only marginal accept due to the writing style.
Hi, AnonReviewer4,
Thanks for your review.  We agree with your points about the writing style of the paper.  Based on your feedback, and also the feedback from the other reviewers, we have significantly rewritten the paper, shortened it and presented only the RNN part of the paper, so that it is more focused, self-contained and consistent. We’re happy to change the title of the paper to more RNN-focused, if that’s something the reviewer feels the right thing to do.  The parameter count is also included in the machine translation section as requested. Please take a look and let us know your thoughts!
Hi, AnonReviewer4,
Thanks for revising your rating from 6 -> 8, based off the revised paper we have submitted on Jan 2, 2017.
The review process have helped us strengthen the presentation of our results in the revision of the paper.
Regards,
David

Meta-review:
role: PC
data: comment: The paper contains an interesting idea, and after the revision of 3rd Jan, the presentation is clear enough as well. (Although I find it now contains an odd repetition where related work is presented first in section 2, and then later in section 3.2).

Decision: Accept (Poster)