Venue: ICLR 2017 conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: The authors investigate a variety of existing and two new RNN architectures to obtain more insight about the effectiveness at which these models can store task information in their parameters and activations.

The experimental setups look sound. To generalize comparisons between different architectures it’s necessary to consider multiple tasks and control for the effect of the hyperparameters. This work uses multiple tasks of varying complexities, principled hyperparameter tuning methodology and a number of tuning iterations that can currently only be achieved by the computational resources of some of the larger industrial research groups. 

The descriptions of the models and the objective where very clear to me. The descriptions of the experiments and presentation of the results were not always clear to me at times, even with the additional details in the appendix available. Most of these issues can easily be resolved by editing the text. For example, in the memory task the scaling of the inputs (and hence also outputs) is not provided so it’s hard to interpret the squared error scores in Figure 2c. It’s not clear to me what the term ‘unrollings’ refers to in Figure 2b. Is this a time lag with additional hidden state updates between the presentation of the input sequence and the generation of the output? Since the perceptron capacity task is somewhat central to the paper, I think a slightly more precise description of how and when the predictions are computed would be helpful. Due to the large number of graphs, it can be somewhat hard to find the most relevant results. Perhaps some of the more obvious findings (like Figure 1(b-d) given Figure 1a) could move to the appendix to make space for more detailed task descriptions.

Novelty is not really the aim of this paper since it mostly investigates existing architectures. To use the mutual information to obtain bits per parameter scores in highly non-linear parameterized functions is new to me. The paper also proposed to new architectures that seem to have practical value. The paper adds to the currently still somewhat neglected research effort to employ the larger computational resources we currently have towards a better understanding of architectures which were designed when such resources were not yet available. I’d argue that the paper is original enough for that reason alone.

The paper provides some interesting new insights into the properties of RNNs. While observed before, it is interesting to see the importance of gated units for maintaining trainability of networks with many layers. It is also interesting to see a potential new use for vanilla RNNs for simpler tasks where a high capacity per parameter may be required due to hardware constraints. The proposed +RNN may turn out to have practical value as well and the hyperparameter robustness results shed some light on the popularity of certain architectures when limited time for HP tuning is available. The large body of results and hyperparameter analysis should be useful to many researchers who want to use RNNs in the future. All in all, I think this paper would make a valuable addition to the ICLR conference but would benefit from some improvements to the text.

Pros:
* Thorough analysis.
* Seemingly proper experiments.
* The way of quantifying capacity in neural networks adds to the novelty of the paper.
* The results have some practical value and suggest similar analysis of other architectures.
* The results provide useful insights into the relative merits of different RNN architectures.

Cons:
* It’s hard to isolate the most important findings (some plots seem redundant).
* Some relevant experimental details are missing.
review_title: Review: Capacity and trainability in RNNs

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper performs a very important service: exploring in a clear and systematic way the performance and trainability characteristics of a set of neural network architectures -- in particular, the basic RNN motifs that have recently been popular.   

Pros:

* This paper addresses an important question I and many others would have liked to know the answer to but didn't have the computational resources to thoroughly attack it.   This is a nice use of Google's resources to help the community. 

* The work appears to have been done carefully so that the results can be believed.

* The basic answer arrived at (that, in the "typical training environment" LSTMs are reliable but basically GRUs are the answer) seems fairly decisive and practically useful.   Of course the real answer is more complicated than my little summary here, but the subtleties are discussed nicely in the paper.

* The insistence on a strong distinction between capacity and trainability helps nicely clear up a misconception about the reasons why gated architectures work.  In sum, they're much more easily trainable but somewhat lower capacity than vanilla RNNs, and in hard tasks, the benefits of better trainability far outweigh the costs of mildly lower capacity.

* The point about the near-equivalence of capacity at equal numbers of parameters is very useful.   

* The paper makes it clear the importance of HP tuning, something that has sometimes gotten lost in the vast flow of papers about new architectures.

* The idea of quantifying the fraction of infeasible parameters (e.g. those that diverge) is nice, because it's a practical problem that everyone working with these networks has but often isn't addressed. 

* The paper text is very clearly written.

Cons:

* The work on the UGRNNs and the +RNNs seems a bit preliminary.  I don't think that the authors have clearly shown that the +RNN should be "recommended" with the same generality as the GRU.   I'd at the least want some better statistics on the significance of differences between +RNN and GRU performances quantifying the results in Figure 4 (8-layer panel).   In a way the high standards for declaring an architecture useful that are set in the paper make the UGRNNs and +RNN contributions seem less important.   I don't really mind having them in the paper though.   I guess the point of this paper is not really to be novel in the first place -- which is totally fine with me, though I don't know what the ICLR area chairs will think. 

* The paper gives short shrift to the details of the HP algorithm itself.  They do say: 

     "Our setting of the tuner’s internal parameters was such that it uses Batched GP Bandits with an expected improvement acquisition function and a   
     Matern 5/2 Kernel with feature scaling and automatic relevance determination performed by optimizing over kernel HPs"  

and give some good references, but I expect that actually trying to replicate this involves a lot of missing details.   

* I found some of the figures a bit hard to read at first, esp. Fig 4, mostly due to the panels being small, having a lot of details, and bad choices for visual cleanliness.   

* The neuroscience reference ("4.7 bits per synapse") seems a little bit of a throw-away to me, because the connection between these results and the experimental neuroscience is very tenuous, or at any rate, not well explained.  I guess it's just in the discussion, but it seems gratuitous.   Maybe it should couched in slightly less strong terms (nothing is really strongly shown to be "in agreement" here between computational architectures and neuroscience, but perhaps they could say something like -- "We wonder if it is anything other than coincidence that our 5 bits result is numerically similar to the 4.7 bits measurement from neuroscience.")




review_title: review of "CAPACITY AND TRAINABILITY IN RECURRENT NEURAL NETWORKS"

final_score: rating: null
review_content: The text mentions that the number of samples b is treated as an HP as well and optimized by the HP tuner. Currently I have no idea of the order of magnitude of this HP and I'm also somewhat curious how the optimal value related to the number of trainable parameters. It's not necessary to know this to understand the paper but I think it would still be valuable information.
review_title: From wich range was the number of samples chosen in the capacity experiments?

final_score: rating: 7: Good paper, accept
review_content: CONTRIBUTIONS
Large-scale experiments are used to measure the capacity and trainability of different RNN architectures. Capacity experiments suggest that across all architectures, RNNs can store between three and six bits of information per parameter, with ungated RNNs having the highest per-parameter capacity. All architectures are able to store approximately one floating point number per hidden unit. Trainability experiments show that ungated architectures (RNN, IRNN) are much harder to train than gated architectures (GRU, LSTM, UGRNN, +RNN). The paper also proposes two novel RNN architectures (UGRNN and +RNN); experiments suggest that the UGRNN has similar per-parameter capacity as the ungated RNN but is much easier to train, and that deep (8-layer) +RNN models are easier to train than existing architectures.

CLARITY
The paper is well-written and easy to follow.

NOVELTY
This paper is the first to my knowledge to empirically measure the number of bits of information that can be stored per learnable parameter. The idea of measuring network capacity by finding the dataset size and other hyperparameters that maximizes mutual information is a particularly novel experimental setup.

The proposed UGRNN is similar but not identical to the minimal gated unit proposed by Zhou et al, “Minimal Gated Unit for Recurrent Neural Networks”, International Journal of Automation and Computing, 2016.

SIGNIFICANCE
I have mixed feelings about the significance of this paper. I found the experiments interesting, but I don’t feel that they reveal anything particularly surprising or unexpected about recurrent networks; it is hard to see how any of the experimental results will change the way either that I think about RNNs, or the way that I will use them in my own future work. On the other hand it is valuable to see intuitive results about RNNs confirmed by rigorous experiments, especially since few researchers have the computational resources to perform such large-scale experiments.

The capacity experiments (both per-parameter capacity and per-unit capacity) essentially force the network to model random data. For most applications of RNNs, however, we do not expect them to work with random data; instead when applied in machine translation or language modeling or image captioning or any number of real-world tasks, we hope that RNNs can learn to model data that is anything but random. It is not clear to me that an architecture’s ability to model random data should be beneficial in modeling real-world data; indeed, the experiments in Section 2.1 show that architectures vary in their capacity to model random data, but the text8 experiments in Section 3 show that these same architectures do not significantly vary in their capacity to model real-world data.

I do not think that the experimental results in the paper are sufficient to prove the significance of the proposed UGRNN and +RNN architectures. It is interesting that the UGRNN can achieve comparable bits per parameter as the ungated RNN and that the deep +RNNs are more easily trainable than other architectures, but the only experiments on a real-world task (language modeling on text8) do not show these architectures to be significantly better than GRU or LSTM.

SUMMARY
I wish that the experiments had revealed more surprising insights about RNNs, though there is certainly value in experimentally verifying intuitive results. The proposed UGRNN and +RNN architectures show some promising results on synthetic tasks, but I wish that they showed more convincing performance on real-world tasks. Overall I think that the good outweighs the bad, and that the ideas of this paper are of value to the community.

PROS
- The paper is the first of my knowledge to explicitly measure the bits per parameter that RNNs can store
- The paper experimentally confirms several intuitive ideas about RNNs:
    - RNNs of any architecture can store about one number per hidden unit from the input
    - Different RNN architectures should be compared by their parameter count, not their hidden unit count
    - With very careful hyperparameter tuning, all RNN architectures perform about the same on text8 language modeling
    - Gated architectures are easier to train than non-gated RNNs

CONS
- Experiments do not reveal anything particularly surprising or unexpected
- The UGRNN and +RNN architectures do not feel well-motivated
- The utility of the UGRNN and +RNN architectures is not well-established
review_title: Review: Capacity and Trainability in Recurrent Neural Networks

final_score: rating: null
review_content: Based on your large-scale usage of automated hyperparameter optimization, can your results be used to offer any concrete advice about choosing hyperparameters for recurrent neural networks? How much did optimal hyperparameters vary across task and network architecture? I realize that this is orthogonal to the proposed set of experiments but it could be a nice addition to the paper given that few have the resources to perform such extensive hyperparameter search.
review_title: Insight from experiments about hyperparameters?

Meta-review: The reviewers all agreed that this paper should appear at the conference. The experiments seem to confirm interesting intuition about the capacity of recurrent nets and how difficult they are to train, and the reviewers appreciated the experimental rigor. This is certainly of interest and useful to the ICLR community and will lead to fruitful discussion. The reviewers did request more fine details related to the experiments for reproducibility (thank you for adding more detail to the appendix). The authors are recommended to steer clear of making any strong but unsubstantiated references to neuroscience.
Decision: Accept (Poster)