Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposed to forward an input to neural networks twice with two different dropout masks, and apply a consistent loss as regularization, named R-Drop, between the two outputs to penalize the discrepancy. KL divergence of the two output probabilities is used as the consistent loss. Experiments on different NLP tasks demonstrate the effectiveness of R-Drop.
main_review: Strengths:

The proposed R-Drop is simple and effective, achieving impressive improvements on several tasks.

Weakness:

The main issue of this paper is that the authors completely ignore the existing research on regularizing the consistency of dropout. The research line can be back to the work of Ma et al., (2017), which analyzed the gap of dropout between training and inference time, and also proposed to regularized dropout neural networks by adding a loss to penalize the gap. 
Subsequently, Zolna et al., (2018) proposed Fraternal Dropout, which also penalize the discrepancy of two dropout masks. The only difference between Fraternal Dropout and R-Drop is that Fraternal Dropout used L2 distance between the hidden representations of the final layer as loss while R-Drop used KL-divergence.
Recently, Gao et al., (2021) applied the same framework in contrastive learning by formulating the outputs with two dropout masks as positive pairs. It is the authors' responsibility to comprehensively discuss the relations and differences between R-Drop and the existing methods.

==============================Update after Rebuttal==============================================
Thanks for the authors' efforts on the response. It is highly appreciated that the authors provided detailed discussions of the missed related work and promised to revise the paper. This has addressed my concerns and I upgraded my score to 6.

References:

Ma et al., Dropout with Expectation-Linear Regularization. 2017.

Zolna et al., Fraternal Dropout. 2018.

Gao et al., SimCSE: Simple Contrastive Learning of Sentence Embeddings. 2021.
limitations_and_societal_impact: NA
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The paper proposes the dropout-based regularization method R-Drop. The proposed R-Drop further regularizes the output predictions of sub-models from dropout.  The theoretical analysis was performed to show that the proposed method reduces the freedom of model parameters. The empirical results on multiple datasets were performed to show that the proposed method improves the performance (accuracy) for four NLP tasks and one CV task. 
main_review: The method needs clarity. For example,

Re: "Concretely, given the input data xi at each training step, we feed xi to go through the forward pass  of the network twice"

Does each forward pass change parameters involved in both distributions (P1w (yi |xi ) and P2w (yi|xi )) learning? 

Concretely, it will be useful to elaborate following steps of the algorithm: 
4:  forward the data one time and obtain the output distribution P1w (yi |xi ), 
5:  forward the data one more time and obtain the output distribution P2w (yi |xi ),

Interestingly, at the implementation, there were not actually two passes, rather input concatenation was done resulting in larger batch size to reduce the time complexity (as per lines 100-101). Why not keep just one type of formulation of the method (i.e. two passes or larger batch size)?  

The model parameters are increased in the proposed methods, and potentially converge times increases to learn extra parameters. I am not sure the performance improvement is due to the better regularization or larger model with longer training. 


limitations_and_societal_impact: The limitation of the current approach was mentioned in the paper (in line 103). Is there any way this limitation can be mitigated?
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a simple yet effective regularization method for deep learning models. The model performs two forward passes with different dropout masks for the same input and minimizes the difference between the two output distributions by the bidirectional KL divergence. The proposed method has been sufficiently validated on both CV and NLP benchmarks. 
main_review: ### Update after Response:
Thank you for the very detailed response, I would like to increase my score to 7 since some of my concerns have been alleviated.

### Strengths: 

1.	The proposed method is simple and might be widely used in the community.
2.	The empirical results are strong. I ran the experiment on the IWSLT German-English translation task and it can indeed achieve the BLEU scores shown in the paper.
3.	The code is released and the documentation is well-organized. 
4.	The experiments are conducted on both NLP and CV tasks, showing the strong generality of the proposed method.

### Weaknesses: 
1. (**Main**) The proposed method can be seen as a combination of [cutoff](https://arxiv.org/abs/2009.13818) (very similar) and [simCSE](https://arxiv.org/abs/2104.08821), but this paper does not cite and discuss them. Is the proposed method inspired by them? What is the relationship between these methods? 

2.	The authors should report sacreBLEU scores for machine translation experiments to improve the reproducibility, especially for Table 2 where I find the results are not comparable since the models use different tokenizations. Could you provide the scores in the rebuttal? **after response**: I don't agree with the authors that the results are comparable. [sacreBLEU paper](https://aclanthology.org/W18-6319/) has clearly shown that BLEU scores produced by multi-bleu.perl are hard to compare and this is the reason why they proposed sacreBLEU score for a fair comparison. Furthermore, as the scareBLEU score of R-Drop is much lower than Transformer Admin, I suggest the authors remove the claim of SOTA results on the En-De task.

3.	I find the proposed method is of low training efficiency (the small IWSLT task needs to run over 200 epochs). Could you discuss it more and provide the wall time of training in the rebuttal and the future paper? 

4.	The paper does not provide any qualitative analyses. It is not clear that how the proposed method improves the models? For example, does the proposed method improve the translation of rare words in machine translation experiments? It would be nice if the authors could provide some qualitative analyses, which can provide more insights for future research. 

5.	How does the $m$-time ($m$>2) drop deal with the bidirectional KL divergence? Could you give the formulations? Could you provide the 3-time result on the WMT English-German task? The claim in this part is somewhat arbitrary. 

6.	I think the authors should not update the code after the supplementary material submission deadline. This might hurt the fairness of the community since the other submissions have no chance to update.


limitations_and_societal_impact: Please refer to the weaknesses.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper introduces a variant of dropout called R-drop. First, the data goes through the forward pass once, and we obtain the output distribution. Next, the data goes through the forward pass (with the same dropout rate but different dropout neurons) another time, and we obtain the output distribution. The goal is to minimize the loss, which is the standard negative log-likelihood (adding two NLL losses from two different passes together) together with KL divergence (KL(p1 || p2) + KL(p2 || p1)). There is large performance gain on machine translation, GLUE tasks, as well as language modeling. 
main_review: Strengths

- Elegant method and easy implementation for practitioners. 

- Evaluated on many tasks. Hyperparameter information is comprehensive in the main text and the appendix. 

- Theoretical analysis in the appendix. 


Comments / points for improvements

My only major issue about the paper is as follows. R-drop somehow regularizes the model parameters so that they are not too extreme. (1) Parameter averaging and (2) deep ensembling also effectively try to make sure that the parameters / output distributions are not too extreme. 

For parameter averaging, one can average the parameters of two trained networks (and perhaps finetune it a little). For deep ensembling, we simply predict two output distributions (using two trained networks) and then take the average of the distributions. Do the authors think that these are relevant baselines? 

How would the authors think R-drop would perform compared to those baselines? What are the pros and cons of R-drop compared to deep ensembling for example?


Another question: The alpha value (weight for KL divergence term) seems to be a sensitive hyperparameter. How sensitive is alpha for GLUE tasks / language modeling? 

limitations_and_societal_impact: I don't see potential negative societal impacts.
review_title: null

Meta-review: The paper presents a method for self-supervising models by ensuring that different forward passes through dropout result in the same prediction distributions. Results show the utility of the method. The reviewers raised really interesting points and brought up prior work but the authors were able to place their work within this context, and show their novelty.  Points raised during the discussion will be added to the paper for publication.
Decision: Accept (Poster)