Title: Recurrent Normalization Propagation
Authors: César Laurent
Nicolas Ballas
Pascal Vincent
Venue: ICLR 2017
Keywords: Deep learning
Optimization

Abstract: We propose a LSTM parametrization  that preserves the means and variances of the hidden states and memory cells across time. While having training benefits similar to Recurrent Batch Normalization and Layer Normalization, it does not need to estimate statistics at each time step,  therefore, requiring fewer computations overall. We also investigate the parametrization impact on the gradient flows and  present a way of initializing the weights accordingly.

We evaluate our proposal on language modelling and image generative modelling tasks. We empirically show that it performs similarly or better than other recurrent normalization approaches, while being faster to execute.

TL;DR: Extension of Normalization Propagation to the LSTM.

Review discussion:
Hi,
Sorry for the delay in posting this. One argument you make is that due to this new parametrization, the new method might be cheaper (computationally) then recurrent batch normalization.
But you have to compute the norms of each column of the weight.. can you give a comparison about how much you save?
Also isn't there a clean connection between your normalization and projected SGD ? I think except the cell update (which I agree doesn't fit the bill), the use of the normalized columns, you get an identical effect if you just use projected SGD (whereby you project your weights after each update such that each column has unit norm). Can you compare this things?
Thanks for your questions!
Regarding the computation time:
For Recurrent Batch Normalization, you have to compute 3 means and 3 variances **at each time step**. For Recurrent Normalization Propagation, you compute the matrix W'= W/||W_i|| **only once** before the recurrence. This is why Recurrent Normalization Propagation is cheaper computationally. Table 1 p. 6 gives the measured computation time per epochs for both variants.
Regarding projected SGD:
If you have a layer of the form y_i = W/||W_i||*x, and the norm of the lines of W is 1, then the forward pass and the gradient of y_i with respect to x will be identical to projected SGD.
However the update equation is different (see equation 18 p.4), than the one you would have with projected SGD. This difference is important, because we perform a reparametrization of the network, so the gradient computations take into account the fact that we performed the rescaling during the forward pass, which is not the case with projected SGD. This is similar to batch normalization: to make it work properly, you need to backpropagate through the computation of the means and variances to account for the reparametrization.
In section 5.1 it is claimed that the proposed method compares favorably compared to other normalization techniques. To me, the results don't seem large. E.g. the improvements over batch normalization and weight normalization are about 1%.
Given that only a single set of hyper parameters was tried, I am unconvinced that a rigorous hyper parameter search and an evaluation featuring confidence intervals will lead to significant improvements;
additionally, the proposed method is arguably more complex.
A skeptical reader might attribute the results to cherry-picking. I think a more extensive experimental evaluation, including hyper parameter search and corresponding confidence intervals as well as learning curves would rule out those concerns.
For Batch Normalization and Layer Normalization, we used the same initialization that their respective authors suggest. For Weight Normalization, we did a grid search for the value of gamma, since the authors propose the value of 1.0 kind of by default.
The main point of this paper is not about getting SOTA results, but more to propose a reparametrization similar to the BN-LSTM, but that doesn't suffer from the disadvantages of BN (computation of the population statistics, no clear generalization to variable length sequences, batch size big enough to have good statistical estimates,...) and is actually cheaper computationally (cf table 1).
The text suggests that the gamma parameters are treated as hyperparameters, and are kept fixed during training. Is that right? (For batch norm and weight norm these parameters are usually learned as well)
Thanks for your question,
Gamma parameters are learned as it is the case in batch norm and weight norm, we will clarify this in the paper next update,
The paper proposes an extension of weight normalization / normalization propagation to recurrent neural networks. Simple experiments suggest it works well.
The contribution is potentially useful to a lot of people, as LSTMs are one of the basic building blocks in our field.
The contribution is not extremely novel: the change with respect to weight normalization is minor. The experiments are also not very convincing: Layer normalization is reported to have higher test error as it overfits on their example, but in terms of optimization it seems to work better.
Also the authors don't seem to use the data dependent parameter init for weight normalization as proposed in that paper.
Thanks for your comments!
About data dependent initialization: As stated in the Weight Normalization paper, data dependent initialization can’t be used for RNNs, because of the recurrent nature of RNNs.
Instead, they simply propose to initialize gamma to 1, and we therefore use the same setup as in the Weight Normalization paper.
About the novelty: We extended the Normalization Propagation framework to the LSTM, in order to be able to better preserve the variance through the recurrence (which is not the case in Weight Normalization). We empirically showed that preserving the variance seems to help with the optimization of the Norm-LSTM compared to the WN-LSTM, leading to slightly better results on both experiments.
I think this build upon previous works, in the attempt of doing something similar to batch norm specific for RNNs. To me the experiments are not yet very convincing, I think is not clear this works better than e.g. Layer Norm or not significantly so. I'm not convinced on how significant the speed up is either, I can appreciate is faster, but it doesn't feel like order of magnitude faster. The theoretical analysis also doesn't provide any new insights.
All in all I think is good incremental work, but maybe is not yet significant enough for ICLR.
Thanks for your comments.
About the experiments: We do agree that the experimental setup could be improved.
About the speed up: Since we are still computing an LSTM, we can’t expect order of magnitude of speed up with such reparametrization, and can only wish for computation time close to the vanilla LSTM.
From a computational point of view, the main difference between the BN-LSTM (or LN-LSTM) and the Norm-LSTM is that the BN-LSTM adds extra computations (means and variances) at each time step, where the Norm-LSTM only adds an overhead before the recurrence (the normalization of the weight matrices).
Thus for long sequences, such as in speech recognition, the Norm-LSTM has a serious advantage over the BN-LSTM: Their performances are similar, while the Norm-LSTM is almost as cheap computationally as the vanilla LSTM.
About the theoretical analysis: In our opinion, the theoretical analysis provides good insights on the role of gamma_x with respect to gamma_h, showing how we can play with their initialization to bias the network into a more short or long term memory.
The authors show how the hidden states of an LSTM can be normalised in order to preserve means and variances. The method’s gradient behaviour is analysed. Experimental results seem to indicate that the method compares well with similar approaches.
Points
1) The writing is sloppy in parts. See at the end of the review for a non-exhaustive list.
2) The experimental results show marginal improvements, of which the the statistical significance is impossible to asses. (Not completely the author’s fault for PTB, as they partially rely on results published by others.) Weight normalisation seems to be a viable alternative in the: the performance and runtime are similar. The implementation complexity of weight norm is, however, arguably much lower. More effort could have been put in by the authors to clear that up. In the current state, practitioners as well as researchers will have to put in more effort to judge whether the proposed method is really worth it for them to replicate.
3) Section 4 is nice, and I applaud the authors for doing such an analysis.
List of typos etc.
- maintain -> maintain
- requisits -> requisites
- a LSTM -> an LSTM
- "The gradients of ot and ft are equivalent to equation 25.” Gradients cannot be equivalent to an equation.
- “beacause"-> because
- One of the γx > γh at the end of page 5 is wrong.
Thanks for your feedback.
1) We updated the paper with better English.
2) The main difference between the WN-LSTM and Norm-LSTM is the preservation of the variance through the recurrence. In both experiments that we conducted, we observed way better optimization (see for example figure 2 for the training performances on the PTB experiment), and slightly better generalization, at an almost identical computational cost.
Moreover, having normalized hidden states is also nice when dealing with stacks of LSTMs. WN doesn’t provide normalized hidden states, unless you perform a data dependent initialization (which is arguably more complex to implement). We also plan to release the code soon.
3) Thank you!

Meta-review:
role: PC
data: comment: Paper proposes a modification of batch normalization. After the revisions the paper is a much better read. However it still needs more diverse experiments to show the success of the method.
 
 Pros:
 - interesting idea with interesting analysis of the gradient norms
 - claims to need less computation
 
 Cons:
 - Experiments are not very convincing and only focus on only a small set of lm tasks.
 - The argument for computation gain is not convincing and no real experimental evidence is presented. The case is made that in speech domain, with long sequences this should help, but it is not supported.
 
 With more experimental evidence the paper should be a nice contribution.

Decision: Invite to Workshop Track