Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: 1. Specific problem tackled by the paper:

Moving the LayerNorm layer to be inside the residual connection in a stack of transformers can remove the need for learning rate warm up. This paper provides a theoretical motivation for doing this.

2. Motivation of the paper:

The authors motivate the problem clearly, performing experiments to demonstrate the problem. Their experiments provide good context for the problem they are solving and acts as a solid reference point for their (and other peoples' future) work. To be more convincing  the authors should have performed multiple runs and shown the standard deviations across runs.

3. Claims of the paper:

The authors claim that layer norm should be placed inside the residual connection (Pre-LN) rather than outside (Post-LN) it. The authors show, theoretically, that in a Post-LN transformer the magnitude of the gradients in a transformer scale with the number of layers and that the magnitude of the hidden states scale linearly with the layers that that they are output by. While in a Pre-LN transformer the magnitudes of the gradients and states are independent of the number of layers.

i.e. During training a Post-LN transformer is likely to have weaker gradients in the lower (closer to input layers) than at the output layers.

This theory is backed by an empirical study. It is good that these experiments are repeated ten times however the authors should show the standard deviations too.

The authors show machine translation results, demonstrating that using Pre-LN rather than Post-LN leads to faster convergence, however the models converge to the same result. However, there is benefit in not having to design a training schedule. Wang et al. have also shown the benefits of Pre-LN rather than Post-LN transformers for machine translation.

4. Decision (accept or reject) with one or two key reasons for this choice and reasons for the decision.

Weak accept.

Pros:
(1) The authors provide theory that supports the use of Pre-LN rather than Post-LN transformers. Using Pre-LN rather than Post-LN transformers may save a lot of time by avoiding hyper-parameter tuning, without loss in performance and this looks very easy to implement.
(2) The paper is well organised and the problem is very well motivated.
(3) The experiments are sufficient. They could be improved with repeats and by showing the standard deviations (as mentioned above).

Cons:
(1) This work is incremental. Wang et al. have already shown that Pre-LN transformers are better that Post-LN transformers when the network has many transformer layers, they explain theoretically why this is the case. 
(2) While the organisation of the paper is good, the paper is not well written. The gramma is poor.
(3) The paper is very long for an incremental improvement.

5. Additional feedback with the aim to improve the paper. 

[a] Ideally, it would be good to see repeats for Figure 2 and the standard deviations for Figure 3.
[b] Without reading the appendix it is not clear where the assumption that W^Q and W^K are zero is used. Making some connection with how this assumption relates to the lemmas would be useful. Additionally, you should explain what this means qualitatively, because this makes the assumption more acceptable. I am assuming that it means that the attention is uniform.
[c] In Lemma two you are comparing the magnitudes for the input in the Pre-LN and the output in the Post-LN transformer according to how x_{l,i}^post and x_{l, i}^pre are defined in Table 1.
[d] In Figure 3(b) the gradients are clearly decreasing with the number of layers, are there any comments on this? In the limit this could cause vanishing gradients?
[e] On the surface Figure 2 and 4 appear to contradict. Is the difference a result of using RAdam? If so, this should be made very clear. If not, why are the results contradictory?

More minor comments:
[1] Many gramma errors.
[2] Figure 1 is referenced before explaining what an FFN is. Figure 1 could also be enhanced by labelling Post-LN as previous work and Pre-LN as current work.
[3] MultiHeadAtt is not well defined. Multi-Head( ), Attention( ) and Head( ) each take three arguments, while MutliHeadAtt( ) takes two. It would be worth connecting these.
[4]"sub-layer --> ..." here sub-layer is not defined. Should this say self-attention sub-layer?
[5] Where does equation (4) come from? There should be a citation and/or explanation.
[6] The BLEU score is not defined (this could be with a footnote).
[7] Top of page 7: "As most of the parameters are initialized by Gaussian distributions" --> Using the word "most" is very vague. The authors should be specific about which parameters they are referring to.
[8] There is a good balance of equations in the main text with most of the proofs in the appendix.
[9] Inconsistent use of LN and LayerNorm, both are used.
review_title: Official Blind Review #4

final_score: rating: 6: Weak Accept
review_content: 
=========Update========
I appreciate the authors' response and additional appendix sections connecting theory and practice, verifying the claims about the final layer gradients scaling with L. I have also read the other reviews. I'm still not completely convinced that the multi-layer analysis well explains the reality, but because of the other contributions I am updating my score, which would be a 5/10 on the old ICLR scale.
=======================

This paper investigates the placement of Layer Normalization in the transformer architecture. The authors show that the Pre-LN placement leads to better behaved gradients as the network gets larger. This in turn allows them to remove the warmup stage of the learning rate schedule, leading to a faster and simpler training procedure. They examine a simplification of the attention layer analytically, and show a scaling with the number of layers that occurs with the Pre-LN placement but not the Post-LN placement. Finally, they demonstrate the effectiveness of the transformer changes both for machine translation and in BERT pretraining.

Even though the novelty here is limited, Pre-LN placement has been used in prior work, the potential for accelerating future research is large. In general I think the impact of this kind of research, exploring improvements to commonly used methods that add no additional complexity or even simplify, is underappreciated in the reviewing process. Still, I have some concerns about the relation between the analytic investigation of the gradient norms and the empirical results that are presented, and I am concerned that the analytical results are used to imply something stronger than they actually show.

At some level, it seems like the theoretical results have come along for the ride but do not clearly demonstrate that there is a problem with Post-LN and that this problem is fixed by switching to Pre-LN, or at least the relationship is not clearly explained. In the empirical study and central to the paper’s narrative is the fact that Post-LN normalization leads to a vanishing gradient problem where gradient in later layers is substantially larger in magnitude than in early layers at initialization, whereas the Pre-LN placement does not suffer from this problem. The point mentioned in the main text (theorem 1) is that the final FFN layer has gradient magnitude that scales at most like 1/sqrt(L) in the Pre-LN case and independent of L in the Post-LN case. This alone does not imply any of the observed empirical behavior because if all of the layers were scaled by this factor 1/sqrt(L) then the same problems would persist for the Pre-LN network. More relevant to the findings is how the scaling of the gradient changes throughout the layers which is examined in appendix section F, where the gradient norm of the Post-LN network can be upper bounded by an expression in which one of the terms scales approximately as (2/3)^(L-l)/2, whereas in the Pre-LN network the scaling is explained to be independent of layer index l. The connection between the expression that scales in the upper bound and the actual gradient norm is tenuous and there multiple places where the argument could break down (upper bound not being tight, scaling of other terms canceling out, validity of the simplifying assumptions being used). It would be useful to verify this sqrt(2/3) scaling on the data from the empirical study that is shown, is the decay shown in figure 3 geometric with this factor. Also the fact that the expectation near the bottom of page 20 is approximately 2/3 needs to be explained as it’s not obvious where that comes from.

review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: Summary: The paper investigates the myth about layer normalization and learning rate warmup for the Transformer architecture. It shows, both theoretically and empirically, that putting the layer normalization in the residual blocks rather than between the residual blocks, could make a big difference to the scale of the gradient at the initialization stage.

Pros:
  + A well-written paper with a good organization; notations are clear.
  + The proof I checked seem correct (but I didn't check all of them).
  + Good experimental design that compares the pre-LN and post-LN Transformers in different settings/tasks.
  + Investigating the purpose and the theory behind using LN and learning rate warmup is a very interesting topic to me, as these modules are rarely used when training other kinds of deep nets (e.g., ConvNets, etc.).

I will detail the cons below, along with my other questions/concerns/issues.

----------------------------------------

Questions/concerns:

1. One of the two major concerns I have is the novelty of this paper in terms of its methodology and empirical value to the community. The Pre-LN setting of Transformers has already been widely used. For instance, Baevski et al. [1] and Child et al. [2] are both well-known works that have applied the pre-LN setting and achieved SOTA results on various very challenging benchmarks. These papers also reported that using layer normalization before self-attention brought "more effective training", which is one of the major empirical remarks that this paper made as well.

2. The second major concern I have is the connection the authors established between its theoretical findings and the empirical findings. While the post-LN Transformers may have larger gradient at higher-levels (as in, close-to-the-output levels), actually some (famous) prior works on Transformers have applied gradient clipping to their architecture, such as BERT [3] (https://github.com/google-research/bert/blob/master/optimization.py#L74), sparse transformer [2] and Transformer-XL [4]. But even when the gradient clipping is applied, learning rate warm-up still seems very helpful (and sometimes necessary), as was used in all of these works. Therefore, I think to further verify the theoretical hypotheses of the paper, the authors should at least also study whether (and to what degree) the very simple "gradient clipping" (or other gradient normalization techniques) solves the problem (which is a common solution to exploding gradients).

3. While Figure 3 is interesting to see, I don't think it verifies Theorem 1 exactly. What it verifies is the "extending to other layers/parameters" paragraph (i.e., the gradient scale decreases with layers). Did you try training post-LN Transformers and pre-LN Transformer with different # of layers from scratch (i.e., different L)? According to Thm. 1, I think we should expect to see a plot where post-LN gradient expectation remains at the same level for all L, and pre-LN gradually decreases with L.

4. The proofs are based on the core assumption that we are "at initialization" (e.g., you assumed W_V entries are sampled from N(0, 1/d), that W_Q=W_K=0, and that the input data are normally distributed x ~ N(0, \sigma^2 I_d)). How will the conclusion/derivation to change when these conditions are no longer met (e.g., after a few steps of warmup)? What do you expect to be the relationship between "the number of warmup steps" and solving the "gradient scale problem", which you proved on these assumptions?

======================================

Some minor issues that didn't impact the score:

1. You used \delta = e^{-d \epsilon^2 / 8} when discussing the tail bound, and later 3e^{-4}, 1e^{-4} for the learning rate. Just for notational consistency, maybe use 10^{-4} for the learning rate instead.

2. Referred to the wrong equation numbers (mentioned in another comment from me). 

3. Appendix C: radius d -> radius \sqrt{d}

4. Appendix C: "Similarly, we have \| x_{l,i}^{post, 3} \|_2^2 = d" should have an expected value.

======================================

Despite the potential lack of novelty on method, I do think investigating these myths and instabilities of training Transformers is a very interesting direction to pursue. I think this paper can be improved with more experimental settings to verify the claim it proposes (i.e., on the gradient, which brings a lot of different things to analyze/study/fix here). I put the paper slightly above the acceptance borderline.



[1] https://arxiv.org/abs/1809.10853
[2] https://arxiv.org/abs/1904.10509
[3] https://arxiv.org/abs/1810.04805
[4] https://arxiv.org/abs/1901.02860
review_title: Official Blind Review #2

final_score: rating: null
review_content: Hi,

Some minor questions while reading the paper (I may be missing some assumptions, so please correct me if possible).

1) In your Eq. (6) -> (7), how did you eliminate the third term from the equation? If the covariance matrix of x_{l,i} and x_{l,j} is not diagonal, will the product still have an expected value of 0?

2) Right above Eq. (17), you said "similar to (16)-(21)". Do you mean (8)-(13) instead?

3) Did you clip your gradient in the experiments?
review_title: Some questions

Meta-review: This paper investigates layer normalization and learning rate warmup in transformers, demonstrating that placing layer norm inside the residual connection (pre-LN) leads to better behaved gradients than post-LN placement. Doing so allows the learning rate warm-up stage to be removed, leading to faster training.

Reviewers were mildly positive about the submission, commenting on the interesting insight provided about transformers, as well as the clear, focused motivation and contribution.

However they also stated that it seem rather incremental of a contribution, as pre-LN placement has been introduced before, and found it confusingly written at times.

R2 clearly read it very closely, and had many detailed comments and discussions with authors and other reviewers. They had concerns about the relationship of this work with gradient clipping. The authors deserve credit for quickly investigating this in further experiments. Interestingly, the found that even with gradient clipping, post-LN models still needed the learning rate warm-up stage, although this issue went away with smaller clipped values or much lower learning rates. Overall, R2 appears to find the paper’s motivation very compelling, but the insights incomplete and not fully satisfactory, while all reviewers find the novelty rather limited.

I think a future submission that forges closer connections between the empirical findings and the theoretical interpretations would be of a great interest to the community, but in its current form is probably unsuitable for publication at ICLR 2020.

Decision: Reject