Title: Recurrent Batch Normalization
Authors: Tim Cooijmans
Nicolas Ballas
César Laurent
Çağlar Gülçehre
Aaron Courville
Venue: ICLR 2017
Keywords: Deep learning
Optimization

Abstract: We propose a reparameterization of LSTM that brings the benefits of batch normalization to recurrent neural networks. Whereas previous works only apply batch normalization to the input-to-hidden transformation of RNNs, we demonstrate that it is both possible and beneficial to batch-normalize the hidden-to-hidden transition, thereby reducing internal covariate shift between time steps.

We evaluate our proposal on various sequential problems such as sequence classification, language modeling and question answering. Our empirical results show that our batch-normalized LSTM consistently leads to faster convergence and improved generalization.

TL;DR: Make batch normalization work in recurrent neural networks

Review discussion:
Ioffe and Szegedy (2015) report that deep convolutional networks with batch normalization can often be trained with much higher learning rates than equivalent networks without batch normalization. For your MNIST, Penn Treebank, and text8 experiments you use the same learning rate for models with and without recurrent batch normalization; can you comment on how these learning rates were chosen, and more generally whether you have found recurrent batch normalization to allow for higher learning rates?
Thank you for your question. We evaluated a variety of learning rates for both models on the tasks you mention, however the ones reported in the paper worked best. We do find that batch normalization makes these models more robust to high learning rates, however recurrent neural networks are more sensitive to high learning rates in the first place due to strong curvature in the loss.
Thanks for the paper - I've enjoyed reading it :)
The gradient flow insight, specifically the impact of unit variance on tanh derivatives, seems something that should be more broadly known and may have impacts in other areas. Showing it not just for batch normalization but the "toy task" (Figure 1b) was hugely useful.
Demonstrating the advantages of batch normalization on both sequential and permuted MNIST was a great idea and shows it works well for many standard dependencies and highly convoluted long term dependencies.
I may have missed it, but I'd be curious as to the impact of generalizing beyond T_max. You note that the sequences tend to a steady state distribution - does that imply you think there's minimal impact to performance? Is that the figure A in the appendix? Longer sequences than seen in training are already a difficult enough proposition to most models, I'm just curious as to what the impact of batch normalization may be upon them.
Thanks for your response!
We do show generalization beyond T_max in Figure 4b. There the model was trained on subsequences of length 100 and subsequently evaluated on subsequences of lengths up to 1000. The figure confirms that we can exploit the stationarity of statistics by repeating the last population statistic.
The figure also appears to show that performance *improves* with sequence length, however we think this improvement is due to having fewer hidden state resets as evaluation proceeds along the test sequence.
Do you have any experiments for vanilla RNNs with e.g. tanh transfer functions? It would be interesting to know how your method performs in that case.
We performed preliminary experiments on tanh RNNs and observed similar effects -- faster convergence and better generalization.
However, we quickly moved on to the more practically relevant LSTMs and so we do not have a formal set of results for vanilla RNNs.
Contributions
The paper presents an adaptation of batch normalization for RNNs in the case of LSTMs, along the horizontal depth.
Contrary to previous work from (Laurent 2015; Amodei 2016), the work demonstrates that batch-normalizing the hidden states of RNNs can improve optimization, and argues with quantitative experiments that the key factor to making this work is proper initialization of parameters, in particular gamma. Experiments show some gain in performance over vanilla LSTMs on Sequential MNIST, PTB, Text8 and CNN Question-Answering.
Novelty+Significance
Batch normalization has been key for training deeper and deeper networks (e.g. ResNets) and it seems natural that we would want to extend it to RNNs.  The paper shows that it is possible to do so with proper initialization of parameters, contrary to previous work from (Laurent 2015; Amodei 2016). Novelty comes from where to batch norm (i.e. not in the cell update) and in the per-time step statistics.
Adding batch normalization to LSTMs incurs additional computational cost and bookkeeping; for training speed comparisons (e.g. Figure 2) the paper only compares LSTM and BN-LSTM by iteration count; given the additional complexity of the BN-LSTM I would have also liked to see a wall-clock comparison.
As RNNs are used across many tasks, this work is of interest to many.
However, the results gains are generally minor and require several tricks to work in practice.
Also, this work doesn’t address a question about batch normalization that it seems natural that it helps with faster training, but why would it also improve generalization?
Clarity
The paper is overall very clear and well-motivated. The model is well described and easy to understand, and the plots illustrate the points clearly.
Summary
Interesting though relatively incremental adaptation, but shows batch normalization to work for RNNs where previous works have not succeeded. Comprehensive set of experiments though it is questionable if the empirical gains are significant enough to justify the increased model complexity as well as computational overhead.
Pros
- Shows batch normalization to work for RNNs where previous works have not succeeded
- Good empirical analysis of hyper-parameter choices and of the activations
- Experiments on multiple tasks
- Clarity
Cons
- Relatively incremental
- Several ‘hacks’ for the method (per-time step statistics, adding noise for exploding variance, sequence-wise normalization)
- No mention of computational overhead
- Only character or pixel-level tasks, what about word-level?
Thank you for the detailed feedback. We address some of your concerns below.
“... why would it also improve generalization?”
This is one of the effects of batch normalization in feed-forward networks. We believe that  generalization improvement is due to the use of batch statistics. As the estimated means and variances through the batch are not perfect, they introduce some noise in the training process that have a regularizing effect.
We experimentally verified this effect by varying the minibatch size.
When the minibatch size is increased, noise is reduced in the estimated means and variances.
Consequently, we observed faster optimization, but it also tends to increase the model overfitting. By reducing the batch size, we observe a gain in generalization, but a relative slowdown in the optimization.
“...Several ‘hacks’ for the method (per-time step statistics, adding noise for exploding variance, sequence-wise normalization)”
We would like to clarify the motivations behind the use of per-time step statistics and sequence-normalization.
When BN is applied to an LSTM, one needs to consider the normalization of the state activation and the inputs.
While LSTM activations do converge to a stationary distribution,  we observe that their statistics during the initial transient differ significantly (see Figure 5 in Appendix A). It is therefore necessary to leverage per time step statistics for estimating the mean and variance of the *state activation*, to take into account this initial transient. Note that we always use per-time step statistics for the state activations in our experiments.
On the other hand, inputs means and variances  are not necessarily time-dependent. You can therefore share the statistics over the different time-step leading to the sequence-wise normalization for the *input*.
Finally, adding noise is a way to artificially increase the variance, when the estimated variance  is exactly 0. This problem occurs in the in-order MNIST task due to the first hundred or so pixels being black. Increasing the $\epsilon$ in (eq. 5) would be an alternative way to deal with 0 variances, however, we empirically observe that increasing $\epsilon$ also tends to degrade the optimization.
“...  computational overhead”
Introducing BN indeed implies a computational overhead as you need to compute the mean and variance statistics and normalize the activation.  One SGD update of our BN-LSTM implementation is usually about 50% slower than a vanilla LSTM.
However, since BN eases optimization, it typically requires fewer epochs than the LSTM baseline to converge.
In addition, we observed that introducing BN made it possible to train models with lower capacity, which are faster to compute. We were not able to train models with similar reduced capacity using a baseline LSTM.
In particular, on the CNN task, it took roughly three weeks to train the LSTM baseline with a capacity of 240 units. We tried to reduce the capacity of the baseline, but we were not able to train the baseline models with reduced capacity. By introducing BN, we were able to train a model with a capacity of 120 units on this task, reducing the total training time to about a week.
“... Only character or pixel-level tasks, what about word-level?”
We also report on the CNN dataset which is a word-level question answering task.
This paper extends batch normalization successfully to RNNs where batch normalization has previously failed or done poorly. The experiments and datasets tackled show definitively the improvement that batch norm LSTMs provide over standard LSTMs. They also cover a variety of examples, including character level (PTB and Text8), word level (CNN question-answering task), and pixel level (MNIST and pMNIST). The supplied training curves also quite clearly show the potential improvements in training time which is an important metric for consideration.
The experiment on pMNIST also solidly shows the advantage of batch norm in the recurrent setting for establishing long term dependencies. I additionally also appreciated the gradient flow insight, specifically the impact of unit variance on tanh derivatives. Showing it not just for batch normalization but additionally the "toy task" (Figure 1b) was hugely useful.
Overall I find this paper a useful additional contribution to the usage of batch normalization and would be necessary information for successfully employing it in a recurrent setting.
Thank you for your review and positive feedback,
The paper shows that BN, which does not work out of the box for RNNs, can be used with LSTM when the operator is applied to the hidden-to-hidden and the input-to-hidden contribution separately. Experiments are conducted to show that it leads to improved generalisation error and faster convergence.
The paper is well written and the idea well presented.
i) The data sets and consequently the statistical assumptions used are limited (e.g. no continuous data, only autoregressive generative modelling).
ii) The hyper parameters are nearly constant over the experiments. It is ruled out that they have not been picked in favor of one of the methods. E.g. just judging from the text, a different learning rate could have lead to equally fast convergence for vanilla LSTM.
Concluding, the experiments are flawed and do not sufficiently support the claim. An exhaustive search of the hyper parameter space could rule that out.
Thank you for your feedback,
1)  We evaluate our BN-LSTM proposal on 5 different datasets, two sequential classification tasks, two language modelling tasks and one large-scale question answering task. The different datasets also cover a variety of inputs, including characters (PTB and Text8), words (CNN question-answering), and pixels (MNIST and pMNIST).
In addition,  all the proposed tasks have been extensively used in the RNN literature and can be considered as standard benchmarks for such models.
We think that the proposed set of tasks is fairly representative of the usual problems on which RNN model are applied.
2) Hyperparameters have not been picked in favor of one of the models. We performed a grid search for both the vanilla LSTM and BN-LSTM, on the learning rate for all the tasks, on model capacity for text8 and CNN and on type of initialization methods for MNIST/pMNIST.  We will clarify this in the paper.
To the best of our knowledge, our hyperparameters exploration allowed us to obtain the best performance with a straight-up vanilla LSTM model on the MNIST/pMNIST, Penntreebank and Text8 tasks, showing that we consider strong and competitive baselines.
Also, we would like to emphasize that using a different learning rate does not lead to an “equally fast convergence for vanilla LSTM”. More specifically on PennTreebank, increasing the learning rate in the vanilla LSTM baseline led to unstable training that diverged after a given number of epoch. Decreasing the learning rate on the other hand, made the overall optimization slower. We observed similar behavior on the other tasks as well.
We believe that our experimental evaluation is fair and accurately demonstrates the advantage of BN-LSTM compared to a vanilla LSTM.
Can you point me to where I can find a description of the hyper parameter search in the text?
Thanks.
Hi,
We updated the paper and all the information related to the hyperparameters exploration are in the appendix D, at the end of the paper.
Let us know if you need more information.
Thanks!

Meta-review:
role: PC
data: comment: The reviewers believe this paper is of significant interest to the ICLR community, as it demonstrates how to get the popular batch normalization method to work in the recurrent setting. The fact that it has already been cited a variety of times also speaks to its interest within the community. The extensive experiments are convincing that the method works. One common criticism is that the authors don't address enough the added computational cost of the method in the text or empirically. Plots showing loss as a function of wall-clock time instead of training iteration would be more informative to readers deciding whether to use batch norm. 
 
 Pros: 
 - Gets batch normalization to work on recurrent networks (which had been elusive to many)
 - The experiments are thorough and demonstrate the method reduces training time (as a function of training iterations)
 - The paper is well written and accessible
 
 Cons
 - The contribution is relatively incremental (several tweaks to an existing method)
 - The major disadvantage to the approach is the added computational cost, but this is conspicuously not addressed.

Decision: Accept (Poster)