Title: Capacity and Trainability in Recurrent Neural Networks
Authors: Jasmine Collins
Jascha Sohl-Dickstein
David Sussillo
Venue: ICLR 2017
Keywords: Deep learning

Abstract: Two potential bottlenecks on the expressiveness of recurrent neural networks (RNNs) are their ability to store information about the task in their parameters, and to store information about the input history in their units. We show experimentally that all common RNN architectures achieve nearly the same per-task and per-unit capacity bounds with careful training, for a variety of tasks and stacking depths. They can store an amount of task information which is linear in the number of parameters, and is approximately 5 bits per parameter. They can additionally store approximately one real number from their input history per hidden unit. We further find that for several tasks it is the per-task parameter capacity bound that determines performance. These results suggest that many previous results comparing RNN architectures are driven primarily by differences in training effectiveness, rather than differences in capacity. Supporting this observation, we compare training difficulty for several architectures, and show that vanilla RNNs are far more difficult to train, yet have slightly higher capacity. Finally, we propose two novel RNN architectures, one of which is easier to train than the LSTM or GRU for deeply stacked architectures.


Review discussion:
The text mentions that the number of samples b is treated as an HP as well and optimized by the HP tuner. Currently I have no idea of the order of magnitude of this HP and I'm also somewhat curious how the optimal value related to the number of trainable parameters. It's not necessary to know this to understand the paper but I think it would still be valuable information.
Thank you for your question!
The search range for the dataset size hyperparameter b was chosen to be between 0.1x and 10x the number of parameters. We will add this range description to Appendix A -- thank you for catching its absence.
In practice, the optimal dataset size found by the HP tuner was only slightly larger than the mutual information reported in Fig 1a-1d. Each sample can contribute at most 1 bit to the mutual information, and when the number of samples is chosen to maximize mutual information, this 1-bit per sample bound is nearly saturated. To illustrate this, we will add a scatter plot to the Appendix of mutual information vs dataset size for all of our training runs.
Based on your large-scale usage of automated hyperparameter optimization, can your results be used to offer any concrete advice about choosing hyperparameters for recurrent neural networks? How much did optimal hyperparameters vary across task and network architecture? I realize that this is orthogonal to the proposed set of experiments but it could be a nice addition to the paper given that few have the resources to perform such extensive hyperparameter search.
While writing the paper, we wanted to be particularly careful about making strong claims about optimal hyperparameters. This is for three reasons:
- Even for identical hyperparameters, performance can vary depending on the random seed used in optimization (see Table App.1 in the Appendix).
- The best performing hyperparameters varied significantly, and in a difficult to characterize way, across task and architecture.
- Hyperparameter search was driven by a Gaussian process.
While this accelerates the discovery of good configurations, it also introduces a difficult to quantify bias into the configurations that are explored for each task.
However, we are in the process of releasing a dataset of all hyperparameter configurations, and the corresponding training, validation, and test losses, for all of our tasks. We will publish this dataset with our final paper. Researchers will thus be able to look up our best performing hyperparameters for each task, and will have the option to do more complex analysis of how training depends on hyperparameters.
The authors investigate a variety of existing and two new RNN architectures to obtain more insight about the effectiveness at which these models can store task information in their parameters and activations.
The experimental setups look sound. To generalize comparisons between different architectures it’s necessary to consider multiple tasks and control for the effect of the hyperparameters. This work uses multiple tasks of varying complexities, principled hyperparameter tuning methodology and a number of tuning iterations that can currently only be achieved by the computational resources of some of the larger industrial research groups.
The descriptions of the models and the objective where very clear to me. The descriptions of the experiments and presentation of the results were not always clear to me at times, even with the additional details in the appendix available. Most of these issues can easily be resolved by editing the text.
For example, in the memory task the scaling of the inputs (and hence also outputs) is not provided so it’s hard to interpret the squared error scores in Figure 2c. It’s not clear to me what the term ‘unrollings’ refers to in Figure 2b. Is this a time lag with additional hidden state updates between the presentation of the input sequence and the generation of the output? Since the perceptron capacity task is somewhat central to the paper, I think a slightly more precise description of how and when the predictions are computed would be helpful. Due to the large number of graphs, it can be somewhat hard to find the most relevant results. Perhaps some of the more obvious findings (like Figure 1(b-d) given Figure 1a) could move to the appendix to make space for more detailed task descriptions.
Novelty is not really the aim of this paper since it mostly investigates existing architectures. To use the mutual information to obtain bits per parameter scores in highly non-linear parameterized functions is new to me. The paper also proposed to new architectures that seem to have practical value. The paper adds to the currently still somewhat neglected research effort to employ the larger computational resources we currently have towards a better understanding of architectures which were designed when such resources were not yet available. I’d argue that the paper is original enough for that reason alone.
The paper provides some interesting new insights into the properties of RNNs.
While observed before, it is interesting to see the importance of gated units for maintaining trainability of networks with many layers. It is also interesting to see a potential new use for vanilla RNNs for simpler tasks where a high capacity per parameter may be required due to hardware constraints. The proposed +RNN may turn out to have practical value as well and the hyperparameter robustness results shed some light on the popularity of certain architectures when limited time for HP tuning is available. The large body of results and hyperparameter analysis should be useful to many researchers who want to use RNNs in the future.
All in all, I think this paper would make a valuable addition to the ICLR conference but would benefit from some improvements to the text.
Pros:
* Thorough analysis.
* Seemingly proper experiments.
* The way of quantifying capacity in neural networks adds to the novelty of the paper.
* The results have some practical value and suggest similar analysis of other architectures.
* The results provide useful insights into the relative merits of different RNN architectures.
Cons:
* It’s hard to isolate the most important findings (some plots seem redundant).
* Some relevant experimental details are missing.
Thank you for your review!
You are correct, by “Number steps unrolled” in fig 2b, we are referring to the number of hidden state updates. We have updated the text to better define this term.
For the memory task, inputs are drawn from the uniform distribution with range -sqrt(3) to sqrt(3). In the perceptron capacity task, predictions are computed 5 time steps after the initial presentation of input, except in Figure 2b, which explores the effect of varying the number of time steps. Thank you for noticing these important missing details! We have updated the text in order to better explain the experiments.
CONTRIBUTIONS
Large-scale experiments are used to measure the capacity and trainability of different RNN architectures. Capacity experiments suggest that across all architectures, RNNs can store between three and six bits of information per parameter, with ungated RNNs having the highest per-parameter capacity. All architectures are able to store approximately one floating point number per hidden unit. Trainability experiments show that ungated architectures (RNN, IRNN) are much harder to train than gated architectures (GRU, LSTM, UGRNN, +RNN). The paper also proposes two novel RNN architectures (UGRNN and +RNN); experiments suggest that the UGRNN has similar per-parameter capacity as the ungated RNN but is much easier to train, and that deep (8-layer) +RNN models are easier to train than existing architectures.
CLARITY
The paper is well-written and easy to follow.
NOVELTY
This paper is the first to my knowledge to empirically measure the number of bits of information that can be stored per learnable parameter. The idea of measuring network capacity by finding the dataset size and other hyperparameters that maximizes mutual information is a particularly novel experimental setup.
The proposed UGRNN is similar but not identical to the minimal gated unit proposed by Zhou et al, “Minimal Gated Unit for Recurrent Neural Networks”, International Journal of Automation and Computing, 2016.
SIGNIFICANCE
I have mixed feelings about the significance of this paper. I found the experiments interesting, but I don’t feel that they reveal anything particularly surprising or unexpected about recurrent networks; it is hard to see how any of the experimental results will change the way either that I think about RNNs, or the way that I will use them in my own future work.
On the other hand it is valuable to see intuitive results about RNNs confirmed by rigorous experiments, especially since few researchers have the computational resources to perform such large-scale experiments.
The capacity experiments (both per-parameter capacity and per-unit capacity) essentially force the network to model random data. For most applications of RNNs, however, we do not expect them to work with random data;
instead when applied in machine translation or language modeling or image captioning or any number of real-world tasks, we hope that RNNs can learn to model data that is anything but random. It is not clear to me that an architecture’s ability to model random data should be beneficial in modeling real-world data;
indeed, the experiments in Section 2.1 show that architectures vary in their capacity to model random data, but the text8 experiments in Section 3 show that these same architectures do not significantly vary in their capacity to model real-world data.
I do not think that the experimental results in the paper are sufficient to prove the significance of the proposed UGRNN and +RNN architectures. It is interesting that the UGRNN can achieve comparable bits per parameter as the ungated RNN and that the deep +RNNs are more easily trainable than other architectures, but the only experiments on a real-world task (language modeling on text8) do not show these architectures to be significantly better than GRU or LSTM.
SUMMARY
I wish that the experiments had revealed more surprising insights about RNNs, though there is certainly value in experimentally verifying intuitive results. The proposed UGRNN and +RNN architectures show some promising results on synthetic tasks, but I wish that they showed more convincing performance on real-world tasks.
Overall I think that the good outweighs the bad, and that the ideas of this paper are of value to the community.
PROS
- The paper is the first of my knowledge to explicitly measure the bits per parameter that RNNs can store
- The paper experimentally confirms several intuitive ideas about RNNs:
- RNNs of any architecture can store about one number per hidden unit from the input
- Different RNN architectures should be compared by their parameter count, not their hidden unit count
- With very careful hyperparameter tuning, all RNN architectures perform about the same on text8 language modeling
- Gated architectures are easier to train than non-gated RNNs
CONS
- Experiments do not reveal anything particularly surprising or unexpected
- The UGRNN and +RNN architectures do not feel well-motivated
- The utility of the UGRNN and +RNN architectures is not well-established
Thank you for your review!
As you mentioned, the Minimally Gated Unit (MGU) by Zhou et al. is similar but not identical to the UGRNN. The MGU is a simplification of the GRU while the UGRNN is an extension of the vanilla RNN. They are somewhat similar in form and both motivated by the desire to create an RNN cell that is simpler than existing architectures. Thank you for bringing the MGU to our attention - we have updated our Related Work section accordingly.
While some of our results confirm preexisting intuitions about RNNs, we believe that other results are both non-intuitive and interesting.
For example, the quantification of a near constant 5 bits per parameter capacity across architectures and scales is a novel finding (Fig 1). We also show that ReLU nonlinearities decrease capacity (Fig 2a).
Another finding that is not immediately obvious is that capacity improves and then saturates quickly as a function of the number of times the recurrent map is iterated (Fig 2b).
Finally, we innovate two novel architectures, which perform very well in our studies.
We agree that more extensive testing of the +RNN and UGRNN is needed, especially on real world tasks, and so have softened our language in the discussion to explicitly mention, “Of course further experiments will be required to fully vet the UGRNN and +RNN. All things considered, in an uncertain training environment, we would recommend using the GRU or +RNN”.
This paper performs a very important service: exploring in a clear and systematic way the performance and trainability characteristics of a set of neural network architectures -- in particular, the basic RNN motifs that have recently been popular.
Pros:
* This paper addresses an important question I and many others would have liked to know the answer to but didn't have the computational resources to thoroughly attack it.   This is a nice use of Google's resources to help the community.
* The work appears to have been done carefully so that the results can be believed.
* The basic answer arrived at (that, in the "typical training environment" LSTMs are reliable but basically GRUs are the answer) seems fairly decisive and practically useful.
Of course the real answer is more complicated than my little summary here, but the subtleties are discussed nicely in the paper.
* The insistence on a strong distinction between capacity and trainability helps nicely clear up a misconception about the reasons why gated architectures work.
In sum, they're much more easily trainable but somewhat lower capacity than vanilla RNNs, and in hard tasks, the benefits of better trainability far outweigh the costs of mildly lower capacity.
* The point about the near-equivalence of capacity at equal numbers of parameters is very useful.
* The paper makes it clear the importance of HP tuning, something that has sometimes gotten lost in the vast flow of papers about new architectures.
* The idea of quantifying the fraction of infeasible parameters (e.g. those that diverge) is nice, because it's a practical problem that everyone working with these networks has but often isn't addressed.
* The paper text is very clearly written.
Cons:
* The work on the UGRNNs and the +RNNs seems a bit preliminary.  I don't think that the authors have clearly shown that the +RNN should be "recommended" with the same generality as the GRU.   I'd at the least want some better statistics on the significance of differences between +RNN and GRU performances quantifying the results in Figure 4 (8-layer panel).   In a way the high standards for declaring an architecture useful that are set in the paper make the UGRNNs and +RNN contributions seem less important.   I don't really mind having them in the paper though.   I guess the point of this paper is not really to be novel in the first place -- which is totally fine with me, though I don't know what the ICLR area chairs will think.
* The paper gives short shrift to the details of the HP algorithm itself.  They do say:
"Our setting of the tuner’s internal parameters was such that it uses Batched GP Bandits with an expected improvement acquisition function and a
Matern 5/2 Kernel with feature scaling and automatic relevance determination performed by optimizing over kernel HPs"
and give some good references, but I expect that actually trying to replicate this involves a lot of missing details.
* I found some of the figures a bit hard to read at first, esp. Fig 4, mostly due to the panels being small, having a lot of details, and bad choices for visual cleanliness.
* The neuroscience reference ("4.7 bits per synapse") seems a little bit of a throw-away to me, because the connection between these results and the experimental neuroscience is very tenuous, or at any rate, not well explained.  I guess it's just in the discussion, but it seems gratuitous.   Maybe it should couched in slightly less strong terms (nothing is really strongly shown to be "in agreement" here between computational architectures and neuroscience, but perhaps they could say something like -- "We wonder if it is anything other than coincidence that our 5 bits result is numerically similar to the 4.7 bits measurement from neuroscience.")
Thank you for your review!
Our experiments have demonstrated that the UGRNN performs competitively with gated architectures in terms of trainability (Fig 4) while maintaining capacity comparable to the vanilla RNN. In the deepest trainability scenario we tested (8 layer architectures), we show that the +RNN outperforms all other architectures in both tasks, on both of our metrics.
While we do not report additional statistics on the significance of the differences between architectures on these trainability tasks (this is difficult as we believe that the use of an HP tuner leads to samples that are not i.i.d.), we measured how robust our experiments are against random batching and weight initializations by running the same HP sets multiple times and looking at the final losses (Table 1 of appendix). We found that the losses don’t deviate very much across runs, leading us to believe our results generally, and our findings regarding the UGRNN and +RNN.  Obviously, the GRU and LSTM are better vetted through extensive study by the entire deep learning community.  We have softened our language in the discussion regarding our recommendation of the +RNN, to “Of course further experiments will be required to fully vet the UGRNN and +RNN. All things considered, in an uncertain training environment, we would recommend using the GRU or +RNN”.
As you suggested, we have also now done statistical tests on the experiment described in Fig. 5, which shows evaluation losses for randomly selected HPs for different architectures trained on the parentheses task. We ran a Welch’s t-test and found that for the 1 layer case, the differences between all loss distributions are statistically significant. In the 8 layer case, we found that the differences were statistically significant for most architecture pairs, except for the differences between the GRU and UGRNN, LSTM and RNN, and IRNN and RNN.  These findings have been summarized in the caption for Fig. 5., and exact values are reported in the Appendix.
We have just uploaded the newest version of our paper. Based on the reviewers' helpful comments, the changes include:
- Minimal Gated Unit for Recurrent Neural Networks citation
- Improved descriptions of training tasks in Appendix (added missing details, such as time steps unrolled before computing loss)
- Added range of perceptron capacity HP b and plot showing that optimal dataset size selected by the HP tuner is only slightly more than mutual information calculated (Figure 1 of Appendix)
- Small text changes to emphasize that our results indicate the promise of the UGRNN and +RNN, but of course much more extensive testing is needed to declare that they are comparable to well-known, tried and true architectures
- Welch’s t-test statistics to test the significance of differences between architecture losses as a result of randomly selected hyperparameters (the experiment shown in Figure 5 and Table 1, and test statistics are reported in Table 2 of Appendix)
- Softening of our recommendation regarding the +RNN in the discussion.
We are also in the process of releasing a dataset of the HPs and associated losses for all experiments described in the paper.
While we can not be positive how long the release process will take, we plan to have the HP dataset publicly available by February 2017!

Meta-review:
role: PC
data: comment: The reviewers all agreed that this paper should appear at the conference. The experiments seem to confirm interesting intuition about the capacity of recurrent nets and how difficult they are to train, and the reviewers appreciated the experimental rigor. This is certainly of interest and useful to the ICLR community and will lead to fruitful discussion. The reviewers did request more fine details related to the experiments for reproducibility (thank you for adding more detail to the appendix). The authors are recommended to steer clear of making any strong but unsubstantiated references to neuroscience.

Decision: Accept (Poster)