Venue: ICLR 2017 conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: The authors show how the hidden states of an LSTM can be normalised in order to preserve means and variances. The method’s gradient behaviour is analysed. Experimental results seem to indicate that the method compares well with similar approaches.

Points

1) The writing is sloppy in parts. See at the end of the review for a non-exhaustive list.

2) The experimental results show marginal improvements, of which the the statistical significance is impossible to asses. (Not completely the author’s fault for PTB, as they partially rely on results published by others.) Weight normalisation seems to be a viable alternative in the: the performance and runtime are similar. The implementation complexity of weight norm is, however, arguably much lower. More effort could have been put in by the authors to clear that up. In the current state, practitioners as well as researchers will have to put in more effort to judge whether the proposed method is really worth it for them to replicate.

3) Section 4 is nice, and I applaud the authors for doing such an analysis.


List of typos etc.

- maintain -> maintain
- requisits -> requisites
- a LSTM -> an LSTM
- "The gradients of ot and ft are equivalent to equation 25.” Gradients cannot be equivalent to an equation.
- “beacause"-> because
- One of the γx > γh at the end of page 5 is wrong.




review_title: Sloppy writing, unsufficient experimental validation

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper proposes an extension of weight normalization / normalization propagation to recurrent neural networks. Simple experiments suggest it works well.

The contribution is potentially useful to a lot of people, as LSTMs are one of the basic building blocks in our field.

The contribution is not extremely novel: the change with respect to weight normalization is minor. The experiments are also not very convincing: Layer normalization is reported to have higher test error as it overfits on their example, but in terms of optimization it seems to work better. Also the authors don't seem to use the data dependent parameter init for weight normalization as proposed in that paper.
review_title: null

final_score: rating: null
review_content: Hi, 

 Sorry for the delay in posting this. One argument you make is that due to this new parametrization, the new method might be cheaper (computationally) then recurrent batch normalization. But you have to compute the norms of each column of the weight.. can you give a comparison about how much you save?

Also isn't there a clean connection between your normalization and projected SGD ? I think except the cell update (which I agree doesn't fit the bill), the use of the normalized columns, you get an identical effect if you just use projected SGD (whereby you project your weights after each update such that each column has unit norm). Can you compare this things? 
review_title: Pre-review question

final_score: rating: 6: Marginally above acceptance threshold
review_content: I think this build upon previous works, in the attempt of doing something similar to batch norm specific for RNNs. To me the experiments are not yet very convincing, I think is not clear this works better than e.g. Layer Norm or not significantly so. I'm not convinced on how significant the speed up is either, I can appreciate is faster, but it doesn't feel like order of magnitude faster. The theoretical analysis also doesn't provide any new insights. 

All in all I think is good incremental work, but maybe is not yet significant enough for ICLR.
review_title: incremental

final_score: rating: null
review_content: The text suggests that the gamma parameters are treated as hyperparameters, and are kept fixed during training. Is that right? (For batch norm and weight norm these parameters are usually learned as well)
review_title: is gamma optimized during training or kept fixed?

final_score: rating: null
review_content: In section 5.1 it is claimed that the proposed method compares favorably compared to other normalization techniques. To me, the results don't seem large. E.g. the improvements over batch normalization and weight normalization are about 1%. 

Given that only a single set of hyper parameters was tried, I am unconvinced that a rigorous hyper parameter search and an evaluation featuring confidence intervals will lead to significant improvements; additionally, the proposed method is arguably more complex.

A skeptical reader might attribute the results to cherry-picking. I think a more extensive experimental evaluation, including hyper parameter search and corresponding confidence intervals as well as learning curves would rule out those concerns.
review_title: Significance of performance

Meta-review: Paper proposes a modification of batch normalization. After the revisions the paper is a much better read. However it still needs more diverse experiments to show the success of the method.
 
 Pros:
 - interesting idea with interesting analysis of the gradient norms
 - claims to need less computation
 
 Cons:
 - Experiments are not very convincing and only focus on only a small set of lm tasks.
 - The argument for computation gain is not convincing and no real experimental evidence is presented. The case is made that in speech domain, with long sequences this should help, but it is not supported.
 
 With more experimental evidence the paper should be a nice contribution.
Decision: Invite to Workshop Track