Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper explores potential explanations for the lack of effectiveness of batch norm compared to LayerNorm. They find a phenomenon that correlates with this (TID) and propose a method to tackle this (RBN).

---

After response: Thank you for your response, I appreciate the additional clarifications and experiments!

---
strengths_and_weaknesses: Pros:
- Solid finding about the correlation between TID and BN
- Good presentation

Cons:
- Not much discussion on **why** this happens...which I feel would be a greater contribution for future work to build upon.
- Furthermore, it is unclear whether RBN is superior to LayerNorm. One claim related to its use in practice is that RBN tends to converge faster, however this is not supported directly. It would be nice to have 
questions: ### Questions:
- Does this finding invalidate the PowerNorm findings? Or is simply an added observation. 
- TID seems to be clearly correlated with performance, however do you have any intuition on why this discrepancy occurs?
- Furthermore, why is this discrepancy smaller on some tasks versus others? Is this something to do with distributional shift on the dataset or the nature of the task?

### Missing Citations:
[Understanding and Improving Layer Normalization](https://arxiv.org/pdf/1911.07013)
limitations: They address the main limitation in their paper, I appreciate this.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper attempts to explain the failure of Batch Normalization (BN) in NLP tasks. The authors find that the inconsistency between training and inference leads to the failure. They define Training Inference Discrepancy (TID) to quantitatively measure this inconsistency and show that BN can obtain better performance when TID keeps small during training. Based on this observation, the authors propose Regularized BN that adds a regularization term of TID during training. Experiences show that RBN can improve the performance of BN and outperforms or is on par with layer normalization. The authors also conduct some analyses to show the advantages of their method.
strengths_and_weaknesses: Strengths
1.	This paper focuses on a well-known learning problem in NLP field.
2.	This paper finds an interesting observation from extensive experiments across different CV and NLP tasks.
3.	Based on their observation, the authors proposed a simple yet effective method.
4.	This paper is well-written and easy to follow

Weaknesses
1.	Lack of theoretical explanation and guarantee. 
2.	Without compared with other related normalization methods, for example, Powernorm.



Novelty:
This paper is novel. The failure of Batch Normalization in NLP model is a well-known learning problem in NLP field. This paper is the first one to find that the inconsistency between training and inference and the performance gap between batch normalization and layer normalization are highly correlated. Based on this observation, the authors proposed a novel and simple method that can consistently improve the performance of batch normalization.

Soundness:
The proposed method is empirically validated on different tasks and dataset. But there is no theocratical guarantee about the soundness of their method.

Significance:
The proposed RBN method can consistently improve the performance of Batch Normalization to on be par with Layer Normalization on different tasks and dataset.
However, the author didn’t compare their method with other batch normalization methods, for example, Powernorm. They only discuss the difference in Section2.

Presentation:
This paper is well-organized. Starting from an observation, the authors raise a hypothesis and conduct extensive experiments to validate their hypothesis. Based on that, the authors propose a method that can significantly improve the performance. Finally, the authors conduct analyses on the model trained with their method to validate their understanding.

questions: This paper can be further improved if the author could provide more explanation about their observation. Here are some questions that I hope the authors can answer:
1.	Can you explain why the performance of BN on language modeling tasks is already better than LN?
2.	Can you explain why Transformer_BN has a much bigger mean and variance deviation than ResNet18?
3.	Can you compare your RBN with Powernorm?

limitations: The authors mention that they don’t have a theoretical guarantee about their method in the Limitation section. I agree with it. But I think this work contains sufficient empirical findings that can benefit the community. 

There is no potential negative social impact of this work.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The authors begin by relating the failure of BN variance in NLP models. They define Training Inference Discrepancy (TID) to quantitatively measure this inconsistency and show that TID can serve as an indicator of BN’s performance. 
In particular, BN reaches much better test performance than LN when TID keeps small through training (e.g., in the CV case). 
They propose Regularized BN (RBN) that adds a regularization term in BN to penalize and reduce the TID when the TID of BN is large.
In experimetns, RBN can outperform BN in a lot of settings, and outperform LN in some settings.
strengths_and_weaknesses: Strength:
The proposed approach is novel and the point of view is convincing.
It's good to address the BN mystery in NLP.

Weakness:
No theory advancement.
Empirical improvements are not large.
questions: I  don't have any question at this point.
limitations: Yes.
review_title: null

Meta-review: metareview: The paper studies the reason why Batch Normalization is not effective in NLP tasks. The authors find that the inconsistency between training and inference leads to the failure. They define Training Inference Discrepancy (TID) to measure the inconsistency and show that BN can obtain better performance when TID is small. The authors propose Regularized BN with an additional regularization term. Experiments show RBN is better than plain BN and comparable to Layer Normalization. 
Authors may want incorporate the additional analysis in the feedback. 
confidence: Less certain
Decision: Accept