*** Paper Summary ***  The paper proposes to a new neural network architecture. The layer weights of a classical network are computed as a function of a latent representation associated with the layer. Two instances are presented (i) a CNN where each layer weight is computed from a lower dimensional layer embedding vector; (ii) an RNN where each layer weight is computed from a secondary RNN state.  *** Review Summary ***  Pros:  - I like the idea of bringing multiplicative RNNs and their predecessors back into the spotlight.  - LM and MT results are excellent.  Cons:   - The paper could be better written. It is too long for the conference format and need refocussing.  - On related work, the relation with multiplicative RNN and their generic tensor product predecessor (Order 2 networks, wrt C. Lee Giles definition) should be mentioned in the related work section and the differences with earlier research need to be explained and motivated (by the way it is better to say that something is revisiting an old idea or training it at modern scale/on modern tasks than ommitting it). - on focus, it is not clear if your goal is to achieve better performance or more compact networks. In the RNN section you lean toward the former, in the CNN section you seem to lean toward the latter.  I would suggest to make the paper shorter and clearer possibly leaving the CNN results for latter publication. The relation with multiplicative/order 2 networks and eventual differences need to be explained.  *** Detailed Review ***  Multiplicative networks are an extremely powerfull architecture and bringing them back into the spotlight is excellent. This paper has excellent results but suffer poor presentation, lack of a clear focus. It spends time on details and ommit important points. In its current form, it is much too long to long and his not self contained without the appendices.  Spending more time on multiplicative RNNs, order 2 networks at the begining of the paper would be excellent. This will let you highlight the difference between this paper and earlier work. It would also be necessary to spend a little time on why multiplicative RNN were less used than gated RNN: it seems that the optimization problem their training involve is tricker and it would be helpful to explain whether you had a harder time tweaking optimization parameters or whether you needed longer training sessions compared to LSTMs, regular CNN. On name, I am not sure that "hypernetwork" help the reader understand better what the proposed architecture compared to multiplicative interactions.  In section 3.2, you seem to imply that there are different settings of hypernetworks that allow to vary from an RNN to a CNN, this is not clear to me, maybe you could show how this would work on a simple temporal problem with equations.   The work on CNN and RNN are rather disconnected to me: for CNN, you seem to be interested in a low rank structure of the weights, showing that similar performance can be achieved with less weights. It is not clear to me why to pursue that goal. Do you expect speedups? less memory for embedded applications? In that case you should compare with alternative strategies, e.g. model compression (Caruana et al 2006, aka Dark Knowledge, Hinton et al 2014) or hashed networks (Chen et al 2015).   For RNN, you seem to target better perplexity/BLEU and model compactness is not a priority. Instead of making the weights have a simpler structure, you make them richer, i.e. dependent over time. It seems in that case models might be bigger and take longer to train. You might want to comment on training time, inference time, memory requirement in that case, as you highlight it might be an important goal in the CNN section. Overall, I am not sure it helps to have this mixed message. I would rather see the paper fit in the conference format with the RNN results alone and a clearer explanation and defers the publications of the CNN results when a proper comparison with memory concerned methods is performed.  Some of the discussions are not clear to me, I am not sure what message the reader should get from Figure 2 or from the discussion on saturation statistics (p10, Figure 5). Similarly, I am not sure if Figure 4 is showing anything: everything should change more drastically at word boundaries even in a regular LSTM (states, gates units should look very different before/after a space); without such a comparison it is hard to see if this is unique to your network.  The results on handwriting generation are harder to compare for me. Log-loss are hard to understand, I have no sense whether the difference between models is significant (what would be the variance in this metric under boostrap sampling of the training set?). I am not sold either on qualitative metric were human can assess quality but human cannot evaluate if the network is repeating the training set. Did you thing at precision/recall metric for ink, possibly with some spatial tolerance ? (e.g. evaluation of segmentation tasks in vision).  The MT experiments are insufficiently discussed in the main text.  Overall, I would suggest to make the paper shorter and clearer possibly leaving the CNN results for latter publication. You need to properly discuss the relation to multiplicative/order 2 networks and highlight the differences. Unclear discussion can be eliminated to make the experimental setup and the results presentation clearer in the main text.  *** References ***  M.W. Goudreau, C.L. Giles, S.T. Chakradhar, D. Chen, "First-Order Vs. Second-Order Single Layer Recurrent Neural Networks,"IEEE Trans. on Neural Networks, 5 (3), p. 511, 1994.  Cristian Bucila, Rich Caruana, and Alexandru Niculescu-Mizil, "Model Compression," The Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD-2006), August 2006, pp. 535-541.  Dark knowledge, G Hinton, O Vinyals, J Dean 2014  W. Chen, J. Wilson, S. Tyree, K. Weinberger and Y. Chen, Compressing Neural Networks with the Hashing Trick, Proc. International Conference on Machine Learning (ICML-15)
Output: en