R-Drop: Regularized Dropout for Neural Networks
Year: 2022

Abstract:
Dropout is a powerful and widely used technique to regularize the training of deep neural networks. Though effective and performing well, the randomness introduced by dropout causes unnegligible inconsistency between training and inference. In this paper, we introduce a simple consistency training strategy to regularize dropout, namely R-Drop, which forces the output distributions of different sub models generated by dropout to be consistent with each other. Specifically, for each training sample, R-Drop minimizes the bidirectional KL-divergence between the output distributions of two sub models sampled by dropout. Theoretical analysis reveals that R-Drop reduces the above inconsistency. Experiments on 5 widely used deep learning tasks (18 datasets in total), including neural machine translation, abstractive summarization, language understanding, language modeling, and image classification, show that R-Drop is universally effective. In particular, it yields substantial improvements when applied to fine-tune large-scale pre-trained models, e.g., ViT, RoBERTa-large, and BART, and achieves state-of-the-art (SOTA) performances with the vanilla Transformer model on WMT14 English→German translation (30.91 BLEU) and WMT14 English→French translation (43.95 BLEU), even surpassing models trained with extra large-scale data and expert-designed advanced variants of Transformer models. Our code is available at GitHub2.

heading: 1 Introduction
text: In recent years, deep learning has achieved remarkable success in various areas, e.g., natural language processing, computer vision, speech/audio processing, etc. When training a deep neural network, regularization techniques [57, 60, 27, 3, 67, 58, 23, 71] are indispensable to prevent over-fitting and improve the generalization ability of deep models. Among them, the dropout technique [24], the most widely used one, aims to prevent co-adaptation and performs implicit ensemble by simply dropping a certain proportion of hidden units from the neural network during training. Existing literature [40, 77] has revealed the possible side effect of dropout that there is an unnegligible inconsistency between training and inference stage of dropout models, i.e., the randomly sampled sub model (caused by dropout) during training is inconsistent with the full model (without dropout) during inference. Through imposing L2 regularization on the inconsistent hidden states [40, 77], current methods can mitigate the inconsistency problem to some extent but are far from being widely used.
In this paper, we introduce a simple yet more effective alternative to regularize the training inconsistency induced by dropout, named as R-Drop. Concretely, in each mini-batch training, each data sample goes through the forward pass twice, and each pass is processed by a different sub model by randomly dropping out some hidden units. R-Drop forces the two distributions for the same
∗Equal contribution (listed in alphabetical order). 2https://github.com/dropreg/R-Drop
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
data sample outputted by the two sub models to be consistent with each other, through minimizing the bidirectional Kullback-Leibler (KL) divergence between the two distributions. That is, R-Drop regularizes the outputs of two sub models randomly sampled from dropout for each data sample in training. In this way, the inconsistency between the training and inference stage can be alleviated. Compared with the dropout strategy in conventional neural network training, R-Drop only adds a KL-divergence loss without any structural modifications.
From the perspective of deep neural network regularization, our proposed R-Drop can be treated as a new variation of dropout. Different from most of the previous methods that merely work on the hidden units of each layer (e.g., the standard dropout [24]) or model parameters (e.g., dropconnect [60]), R-Drop works on both the hidden units and the output of sub models sampled by dropout, which is much more effective. We theoretically analyze the regularization effect of R-Drop, where the result shows that R-Drop can reduce the inconsistency existed in the training and inference.
Though R-Drop regularization is simple, we find it is surprisingly effective through extensive experiments on 5 tasks with 18 datasets, spanning from natural language processing, including language modeling, neural machine translation, abstractive summarization, and language understanding, to computer vision, i.e., image classification. It creates new records on multiple datasets, such as 30.91 BLEU score on WMT14 English→German and 43.95 on WMT14 English→French translation tasks while only be simply applied to the training of the vanilla Transformer, and also achieves SOTA results on the CNN/DailyMail summarization dataset. These universal improvements clearly demonstrate the effectiveness of R-Drop.
Our main contributions are summarized as follows:
• We propose R-Drop, a simple yet effective regularization method built upon dropout, which can be universally applied to train different kinds of deep models.
• We theoretically show that our R-Drop can reduce the inconsistency between training and inference of the dropout based models.
• Through extensive experiments on 4 NLP and 1 CV tasks with a total of 18 datasets, we show that R-Drop achieves extremely strong performances, including multiple SOTA results.

heading: 2 Approach
text: The overall framework of our R-Drop regularization method is shown in Figure 1. Before elaborating on the details, we first present some necessary notations. Given the training datasetD = {(xi, yi)}ni=1, the goal of the training is to learn a model Pw(y|x), where n is the number of the training samples, (xi, yi) is the labeled data pair. xi is input data and yi is the label. For example, in NLP, xi can be the source language sentence in machine translation, and yi is the corresponding target language sentence. In CV, xi can be one image, and yi is the categorical class label. The probability distribution of the
mapping function is also denoted as Pw(y|x), and the Kullback-Leibler (KL) divergence between two distributions P1 and P2 is represented by DKL(P1||P2). In the following, we will explain our proposed R-Drop, training algorithm, and theoretical analysis, respectively.

heading: 2.1 R-Drop Regularization
text: We introduce our simple regularization method in this part. Given the training dataD = {(xi, yi)}ni=1, the main learning objective for a deep learning model is to minimize the negative log-likelihood loss function, which is as follow:
Lnll = 1
n n∑ i=1 − logPw(yi|xi). (1)
Since the deep neural networks are prone to over-fitting, regularization methods such as dropout [57] are usually adopted during training to reduce the generalization error of the model. Specifically, dropout randomly drops part of units in each layer of the neural network to avoid co-adapting and over-fitting. Besides, dropout also approximately performs to combine exponentially many different neural network architectures efficiently [57], while model combination can always improve the model performance. Though simple and effective, there is a huge inconsistency between training and inference that hinders the model performance. That is, the training stage takes the sub model with randomly dropped units, while the inference phase adopts the full model without dropout. Also, the sub models caused by randomly sampled dropout units are also different without any constraints. Based on above observations and the randomness of the structure brought by dropout, we propose our R-Drop to regularize the output predictions of sub models from dropout.
Concretely, given the input data xi at each training step, we feed xi to go through the forward pass of the network twice. Therefore, we can obtain two distributions of the model predictions, denoted as Pw1 (yi|xi) and Pw2 (yi|xi). As discussed above, since the dropout operator randomly drops units in a model, the two forward passes are indeed based on two different sub models (though in the same model). As shown in the right part of Figure 1, the dropped units in each layer of the left path for the output prediction Pw1 (yi|xi) are different from that of the right path for output distribution Pw2 (yi|xi). Thus the distributions of Pw1 (yi|xi) and Pw2 (yi|xi) are different for the same input data pair (xi, yi). Then, at this training step, our R-Drop method tries to regularize on the model predictions by minimizing the bidirectional Kullback-Leibler (KL) divergence between these two output distributions for the same sample, which is:
LiKL = 1
2 (DKL(Pw1 (yi|xi)||Pw2 (yi|xi)) +DKL(Pw2 (yi|xi)||Pw1 (yi|xi))). (2)
With the basic negative log-likelihood learning objective LiNLL of the two forward passes:
LiNLL = − logPw1 (yi|xi)− logPw2 (yi|xi), (3)
the final training objective is to minimize Li for data (xi, yi):
Li = LiNLL + α · LiKL =− logPw1 (yi|xi)− logPw2 (yi|xi) (4)
+ α
2 [DKL(Pw1 (yi|xi)||Pw2 (yi|xi)) +DKL(Pw2 (yi|xi)||Pw1 (yi|xi))],
where α is the coefficient weight to control LiKL. In this way, our R-Drop further regularizes the model space beyond dropout and improves the generalization ability of a model. Compared Equation (1) with Equation (4), our R-Drop only adds a KL-divergence loss LiKL based on two forward passes in training. Note that our regularization methodology can be universally applied on different model structures if there exists randomness in a model (e.g., dropout) that can produce different sub models or outputs. We leave further explorations as future work.

heading: 2.2 Training Algorithm
text: The overall training algorithm based on our R-Drop is presented in Algorithm 1. As introduced before, at each training step, Line 3-4 show that we go forward the model and obtain output distributions Pw1 (y|x) and Pw2 (y|x), then Line 5-6 calculate the negative log-likelihood and the KL-divergence
Algorithm 1 R-Drop Training Algorithm Input: Training data D = {(xi, yi)}ni=1. Output: model parameter w.
1: Initialize model with parameters w. 2: while not converged do 3: randomly sample data pair (xi, yi) ∼ D, 4: repeat input data twice as [xi;xi] and obtain the output distribution [Pw1 (yi|xi),Pw2 (yi|xi)], 5: calculate the negative log-likelihood loss LiNLL by Equation (3), 6: calculate the KL-divergence loss LiKL by Equation (2), 7: update the model parameters by minimizing loss Li of Equation (4). 8: end while
between the two distributions. It is worth nothing that we do not forward the input data twice, instead, we repeat the input data x and concatenate them ([x;x]) in batch-size dimension, which can make forward procedure happen in the same mini-batch to save the training cost. Finally, the model parameters are updated (Line 7) according to the loss of Equation (4). The training will continue over the data epochs till convergence. Compared to the conventional training, our implementation is similar to enlarge the batch size to be double, and one potential limitation is that the computational cost of R-Drop increases at each step. As we show in Section 4.1, similar to other regularization methods (e.g., training w/ or w/o dropout), though R-Drop needs more training to converge, the final optimum is much better with a superior performance. We also show another study of baseline with doubled batch size in Appendix C.1.

heading: 2.3 Theoretical Analysis
text: We analyze the regularization effect of R-Drop in this subsection. Let hl(x) ∈ Rd denote the output of the l-th layer of a neural network with input vector x, and let ξl ∈ Rd denote a random vector, each dimension of which is independently sampled from a Bernoulli distribution B(p):
ξli = { 1, with probability p, 0, with probability 1-p.
Then the dropout operation on hl(x) can be represented by hlξl(x) = 1 p ξl hl(x), where denotes the element-wised product. Hence, the output distribution of the neural network with parameter w after applying dropout is Pwξ (y|x) := softmax(linear(hLξL(· · · (h 1 ξ1(xξ0))))), where ξ = (ξ
L, · · · , ξ0). The objective for R-Drop enhanced training can be formulated as solving the following constrained optimization problem:
min w
1
n n∑ i=1 Eξ[− logPwξ (yi|xi)], (5)
s.t. 1
n n∑ i=1 Eξ(1),ξ(2) [DKL(Pwξ(1)(yi|xi)||P w ξ(2)(yi|xi)))] ≤ . (6)
More precisely, R-Drop optimizes the constrained optimization problem in Equation (5) and Equation (6) in a stochastic manner, i.e., it samples two random vectors ξ(1) and ξ(2) (corresponding to two dropout instantiations) from Bernoulli distribution and one training instance (xi, yi), and updates the parameter w according to the stochastic gradient∇wLi from Equation (4). As we presented, one problem for dropout is the inconsistency between the training and inference models. Specifically, the training objective for dropout is the average loss of the sub models, i.e., minw 1 n ∑n i=1 Eξ[− logPwξ (yi|xi)], while the full model (denoted as P̃w(y|x)) is used for inference. Our proposed R-Drop enhanced training reduces this inconsistency by forcing the sub structures to be similar. The following proposition uses a linear model to demonstrate that with the constraint in Equation (6), the inconsistency gap between the average loss of sub structures and the loss of the full model can be bounded (detailed proof can be found in Appendix B). Proposition 2.1. For a linear model Pw(y|x) = softmax(Norm(wTx)) where Norm(·) denotes the normalization layer and x ∈ Rd, with the constraint in Equation (6) in the main paper, we have
|Lnll(w)− Eξ[Lnll(w, ξ)]| ≤ c √ , where Lnll(w),Lnll(w, ξ) are the empirical loss calculated by the full model and a random sub model respectively, c is a constant related to the Liptschtz constant of the softmax operator.

heading: 2.4 Discussion
text: The most related works with our R-Drop are ELD [40] and FD [77], which also study the consistency training with dropout. However, R-Drop has key differences with them. (1) The gap control is from different views. ELD works on directly reducing the gap between the sub model with dropout (train) and the expected full model without dropout (inference), while R-Drop and FD are both working on penalizing the discrepancy between the sub models, the superiority of regularizing the sub models has been proved in FD. (2) The regularization efficiency is different. ELD only back-propagates the gradients through sub model without the full model, which is less efficient than R-Drop that updates both sub models. (3) The regularization effect is different. Both ELD and FD use the L2 distance on hidden states as the regularization loss function. However, this is far away from the main training objective that minimizes the negative log-likelihood over model output distribution. The distance of hidden states is not in the same space as the probability distribution since log-softmax hugely affects the optimization. In comparison, R-Drop utilizes the KL-divergence between the output probability distributions as the consistency regularization, which is in the same space as the training objective. More analysis and experimental comparisons are shown in Appendix C.4.

heading: 3 Experiments
text: To evaluate our approach and show its universal impact, we conduct experiments on 5 different tasks, including 4 natural language processing (NLP) and 1 computer vision (CV) tasks, which are neural machine translation (NMT) (6 datasets), abstractive summarization (1 dataset), language understanding (8 datasets), language modeling (1 dataset), and image classification (2 datasets). For convenience, we utilize ‘RD’ to represent R-Drop in the tables of experimental results hereinafter. More details of experimental settings for each dataset can be found in Appendix A.

heading: 3.1 Application to Neural Machine Translation
text: We first evaluate the NMT tasks, which is very important in NLP. To best show the effectiveness of our method, experiments are conducted on both low-resource and rich-resource translation tasks.
Datasets The datasets of low-resource scenario are from IWSLT competitions, which include IWSLT14 English↔German (En↔De), English↔Spanish (En↔Es), and IWSLT17 English↔French (En↔Fr), English↔Chinese (En↔Zh) translations. The rich-resource datasets come from the widely acknowledged WMT translation tasks, and we take the WMT14 English→German and English→French tasks. The IWSLT datasets contain about 170k training sentence pairs, 7k valid pairs, and 7k test pairs. The WMT data sizes are 4.5M , 36M for En→De and En→Fr respectively, valid and test data are from the corresponding newstest data.
Model & Training We take the most popular Transformer [59] network as our model structure. The transformer_iwslt_de_en and transformer_vaswani_wmt_en_de_big are the configurations for IWSLT and WMT translations respectively. The weight α is set as 5 for all translation tasks. Implementation is developed on Fairseq [48].
Results We calculate the BLEU scores on these tasks for evaluation, following [76]. The IWSLT performances are shown in Table 1 and the rich-resource WMT results are in Table 2. First, we can see that our R-Drop achieves more than 2.0 BLEU score improvements on 8 IWSLT translation tasks, which clearly shows the effectiveness of our method. The results on WMT translations are more impressive. After applying our simple method on the basic Transformer network, we achieve the state-of-the-art (SOTA) BLEU score on WMT14 En→De (30.91) and En→Fr (43.95) translation tasks, which surpass current SOTA models, such as the BERT-fused NMT [76] model that leverages large-scale monolingual data, and the Data Diversification [47] method trained with many translation models. Note that R-Drop is complementary to the above methods, and we believe stronger results can be achieved if we apply R-Drop on their methods and better backbone models beyond Transformer.

heading: 3.2 Application to Language Understanding
text: Dataset We further evaluate our proposed approach on the language understanding tasks by finetuning the pre-trained models3, which are the standard development sets of GLUE [61] benchmark. The GLUE benchmark includes 8 different text classification or regression tasks, which are MNLI, MRPC, QNLI, QQP, RTE, SST-2, STS-B (regression), CoLA. The detailed statistics are in Appendix.
Model & Training We take the BERT-base [9] and strong RoBERTa-large [38] pre-trained models as our backbones to perform fine-tuning, which are publicly available. For each task, different random seeds and parameter settings are required, thus we dynamically adjust the coefficient α among {0.1, 0.5, 1.0} for each setting. Other configurations are following the previous works [9, 38]. For the regression task STS-B, we use MSE instead of KL-divergence to regularize the outputs (see Appendix for MSE regularization details).
Results The evaluation metrics for above 8 tasks are as follows: The result for STS-B is the Pearson correlation; Matthew’s correlation is used for CoLA; Other tasks are measured by Accuracy. The results are presented in Table 3. We can see that R-Drop achieves 1.21 points and 0.80 points (on average) improvement over the two baselines BERT-base and RoBERTa-large, respectively, which clearly demonstrate the effectiveness of R-Drop. Specifically, our RoBERTa-large + RD also surpasses the other two strong models: XLNet-large [68] and ELECTRA-large [7], which are specially designed with different model architecture and pre-training task.

heading: 3.3 Application to Summarization
text: Dataset Abstractive summarization task is to summarize the long sentence/document into a short sequence/sentence (through generation) with the main content remained. For this generation task, we use the CNN/Daily Mail dataset originally introduced by Hermann et al. [22] to evaluate our method. This dataset contains news documents (source), and their corresponding highlights (target) crawled from CNN and Daily Mail website. It contains 287,226 documents for training, 13,368 documents for validation and 11,490 documents for test. We follow [34] to preprocess the dataset.
Model & Training To mostly show the effectiveness, we take the super strong pre-trained sequenceto-sequence BART [34] model as our backbone and fine-tune it using our method. In this task, the coefficient weight α is set as 0.7 to control the KL-divergence. For other hyper-parameters, we follow the setting of the original paper [34] without modification.
3We apply our R-Drop on the fine-tuning stage only in this work. R-Drop can also be applied during pre-training. Due to the computational cost, we leave this as future work.
Results The performance is evaluated by ROUGE F1 score [36]. Specifically, we report the unigram ROUGE1 (RG-1) and bigram ROUGE-2 (RG-2) overlap to assess the informativeness, and the longest common subsequence ROUGE-L (RG-L) score to assess the fluency. The results are shown in Table 4. We can see that R-Drop based training outperforms the fine-tuned BART model by 0.3 points on RG-1 and RG-2 score and achieves the SOTA performance. Specifically, our result also surpasses the PEGASUS method [70], which brings a novel self-supervised paradigm carefully designed for summarization, and the previous best work BART+R3F [1], which introduces a parametric noise
sampled from normal or uniform distributions. Instead, our R-Drop does not introduce any extra parameters or model structure changes during training.

heading: 3.4 Application to Language Modeling
text: Dataset We also evaluate our approach on another widely acknowledged NLP task: language modeling. The dataset we choose for this task is the commonly adopted Wikitext-103 dataset [41], which is the largest available word-level language modeling benchmark with long-term dependency. WikiText-103 contains about 103M training tokens from 28K articles on Wikipedia, and the average length of tokens per article is about 3.6K. The data is preprocessed by following [48].
Model & Training We take two models to conduct the language modeling task. One is the basic Transformer decoder [59], another is the more advanced one: Adaptive Input Transformer [5], which introduces adaptive input embeddings into the Transformer model. We use the open-source Fairseq [48] toolkit, and the corresponding model configurations are transformer_lm_gpt and transformer_lm_wiki103 for Transformer and Adaptive Input Transformer. We simply set the weight α to be 1.0 without tuning during training. Other configurations are same as [48] and [5].
Results The evaluation metric for language modeling is perplexity, which can well measure the probability of a sentence. Same as [5], we report the perplexity on both valid and test sets. The results are shown in Table 5. From the table, we can see that our R-Drop based training improves the perplexity on both two different model structures, e.g., 0.80 perplexity improvement on test set over Adaptive Input Transformer. Besides, more improvement can be achieved when the baseline model is not so strong, e.g., 1.79 perplexity gain on valid set and 1.68 on test set above the Transformer baseline.

heading: 3.5 Application to Image Classification
text: Dataset For image classification, we conduct experiments on two widely acknowledged benchmark datasets, i.e., CIFAR-100 [31] and the ILSVRC-2012 ImageNet dataset [8] (denoted as ImageNet for short). CIFAR-100 dataset consists of 60k images of 100 classes, and there are 600 images per class with 500 for training and 100 for testing.
The ImageNet dataset consists of 1.3M image samples of 1, 000 categorical classes. We utilize the same data preprocessing strategies with [11], where the details are given in [29].
Model & Training We choose the recent strong and popular Vision Transformer (ViT) [11] model as our backbone. More specifically, we take the two publicly released pre-trained models, ViT-B/16 and ViT-L/16, with 86M and 307M parameters respectively, and we
conduct model fine-tuning on the CIFAR-100 and ImageNet datasets. During fine-tuning, the weight α is set as 0.6 for both models, and we set other hyper-parameters/training details to be same as [11].
Results The classification performance is measured by Accuracy, and the results are presented in Table 6. For CIFAR-100, we achieve about 0.65 accuracy improvement over ViT-B/16 baseline, and 0.41 points over ViT-L/16 model. Similarly, on the large-scale ImageNet dataset, consistent improvements are also obtained. These observations demonstrate that our R-Drop can still benefit the model performance even the baseline is powerful. In a word, through the above NLP tasks and this image classification task, we clearly show R-Drop is effective and can be universally applied.

heading: 4 Study
text: Beyond the superior experimental results, in this section, we conduct extensive studies on different perspectives to better understand our R-Drop method. The analysis experiments are performed on the IWSLT14 De→En translation task. More studies can be found in Appendix C.

heading: 4.1 Regularization and Cost Analysis
text: We first show the regularization effect of our R-Drop and study the potential limitation of training cost (as discussed in Section 2.2). Hence, we plot the curves of training/valid loss and valid BLEU along the training update number for Transformer and Transformer + RD models. Besides, we also plot the corresponding curves along the training time (minutes). The curves are shown in Figure 2. We can observe: 1) Along with the training, Transformer quickly becomes over-fitting, and the gap between train and valid loss of Transformer is large, while R-Drop has a lower valid loss. This well proves that R-Drop can provide persistent regularization during training. 2) At the early training stage, Transformer improves the BLEU score quickly but converges to bad local optima soon. In comparison, R-Drop gradually improves the BLEU score and achieves a much superior performance. Though it needs more training to converge, the final optimum is better. This is same as other regularization methods (e.g., training w/ or w/o dropout). R-Drop indeed increases the training cost at each step since it requires repeating input x for another computation in a mini-batch. Note that this is similar to batch size doubled training without KL-divergence. In Appendix C.1, we conduct this training and show that R-Drop increases negligible cost but with a much stronger performance.
0k 50k 100k 150k 200k 250k 300k 0
2
4
6
8
Lo ss
Transformer (dev) + RD (dev) Transformer (train) + RD (train)
0 100 200 300 400 500 0
2
4
6
8
Lo ss
0k 50k 100k 150k 200k 250k 300k Update Number
30
32
34
36
38
BL EU
0 100 200 300 400 500 Minutes
30
32
34
36
38
BL EU
Figure 2: Loss/BLEU curves along with model training.
0k 50k 100k 150k 200k 250k 300k Update Number
33
34
35
36
37
38
BL EU
step=1 Step=2 Step=5 Step=10
0 100 200 300 400 500 Minutes
33
34
35
36
37
38
BL EU
Figure 3: R-Drop with different step.
4.2 k-step R-Drop
The above study shows that R-Drop can achieve much stronger performance, but with a lower convergence, thus we study another training strategy that is to perform R-Drop every k steps to improve the training efficiency, instead of applying at each step. We vary k in {1, 2, 5, 10} to see the difference, where k = 1 is the current training strategy. The valid BLEU curves along with training update number and training time are presented in Figure 3. From the curves, we can conclude that though the convergence is faster with larger k, the training fails to fall into good optima, which quickly over-fits, and the BLEU scores become worse and worse when we increase k. This proves that our R-Drop at each step can well regularize the training and obtain superior performances.
4.3 m-time R-Drop
Our method regularizes the model output between two distributions Pw1 (y|x) and Pw2 (y|x), and it is also interesting to see whether more improvements can be achieved if we regularize m distributions for the same input data, where m = 2 is the current setting. Therefore, we extend our R-Drop to be: LKL = αm∗(m−1) ∑i 6=j i,j∈1,··· ,mDKL(P w i (y|x)||Pwj (y|x)), and we take m = 3 for a feasible implementation. The BLEU score for IWSLT14 De→En test set is 37.30 when m = 3, which is similar to that when m = 2 (37.25 BLEU score). This reflects that R-Drop already has a strong regularization effect between two distributions, without the necessity of stronger regularization.
4.4 Two Dropout Rates
Besides the above studies, we investigate RDrop from another perspective, i.e., the dropout values. In current training, the two distributions are based on the same dropout value (e.g., 0.3 for IWSLT translations). In this study, we utilize two different dropout values for the two output distributions during training (e.g., 0.1 for Pw1 (y|x), 0.3 for Pw2 (y|x)) to see the difference. We choose the two dropout rates from {0.1, 0.2, 0.3, 0.4, 0.5} with total 15 (C25 for two different rates + C15 for two same rates) combinations. The results are shown in Figure 4. Among these different results, we can see that: 1) Dropout rates with the same value (0.3, 0.3) is the best choice (current setting), 2) R-Drop
can stably achieve strong results when the two dropout rates are in a reasonable range (0.3 ∼ 0.5) without a big performance difference. One interesting point is that even the two dropout values are both 0.5, which means half of the units are expected to be dropped, R-Drop can still obtain a satisfied result (36.48 BLEU) compared with the baseline Transformer (34.64 BLEU). These results all confirm the advantage of our R-Drop, and we are interested in studying more in the future.
4.5 Effect of Weight α
Further, we investigate the impact of the KL-divergence loss weight α. As mentioned in Section 3.1, we set α = 5 for NMT experiments. Here we vary the α in {1, 3, 5, 7, 10} and conduct experiments. As shown in Table 7, small α (e.g., 1) can not perform as good as large α (e.g., 5), which means we should pay more attention to the KLdivergence regularization. However, too much regularization (α = 10) is also not good, and the best balanced choice is α = 5. Note that the choice of α is distinct for different tasks (e.g., NMT, language understanding), which depends on how easy the over-fitting happens caused by the specific data size and model size of each task.

heading: 5 Related Work
text: Regularization Methods. Bigger models always tend to have better performance, especially for various large-scale pre-trained models, e.g., Vision Transformer [11], Swin Transformer [39], GPT families [52, 53, 6], BERT [9], BART [34], Switch Transformers [14], etc. With millions and even billions of parameters, these deep models are prone to over-fitting, thus requiring regularization strategies to improve their generalization ability [33]. To tackle with over-fitting, many regularization techniques have been proposed, e.g., weight decay [32, 30, 28, 63], dropout [24, 60, 4, 62, 57], normalization [27, 54, 3, 26, 67], adding noise [25, 50], layer-wise pre-training and initialization [12, 21], label-smoothing [58], and so on. Among which, dropout and its variants are most popular owing to its effectiveness and moderate cost as well as good compatibility with other regularization methods [45], which has been successfully applied to regularize a wide range of neural network
architectures [49], e.g., convolutional neural network layers [64, 10], recurrent neural networks [17, 55, 42], Transformer [69, 75, 65]. The success of dropout methods can be interpreted by preventing co-adaptation of neurons and performing an implicit ensemble of sub models from dropout. Owing to the effect in promoting sparsity of weights and stochastic nature, dropout methods are also adapted to other applications, e.g., contrastive learning for sentence representation learning [18], neural network compression [44, 46] and model uncertainty estimation [16].
Unlike previous researches of designing specific dropout variants or adapting dropout to different applications, we consider to further regularize the model on the success of dropout. Specifically, any two sub models sampled from dropout are encouraged to produce consistent model prediction for an input data by utilizing KL-divergence in the training stage. That is, we conduct regularization on the model output level. In doing so, the sub model outputs produced by the randomness of dropout are regularized to reduce the parameter freedom, which will enhance generalization in inference.
Consistency Training. Besides regularization methods, our work also relates to a few works of consistency training on dropout models or data augmentation. Among them, the most representative methods are ELD [40], FD [77], and Cutoff [56]. As discussed in Section 2.4, ELD only focuses on the inconsistency between the sub model with dropout (train) and the expected full-model without dropout (inference), while FD works between the sub models only (consistence between two sub models). Both ELD and FD utilize L2 to regularize the hidden space. Instead, our R-Drop performs consistency training on dropout from the output space with a more effective bidirectional KL loss. Unlike the above consistency training method on sub models, Cutoff resembles launching consistency training from a data perspective by regularizing the inconsistency between the original data the augmented samples with part of the information within an input sentence being erased.
Self-distillation. Minimizing the KL-divergence between the output distributions of two different models correlates with knowledge distillation [23, 15, 2, 35, 13, 74], where the two models refer to teacher and student, respectively. In our setting, the teacher and student are the dropout instantiations of the same model, and thus it resembles self-knowledge distillation [43] scenario. Different from existing method that exploits dark knowledge from the model itself [20, 19] or distills knowledge between different layers [71], our strategy can be regarded as an instance-wise self-knowledge distillation, i.e., each pair of sampled sub models perform distillation between each other for the same input, which also relates to mutual learning [72] but ours is much more efficient without extra parameters.

heading: 6 Conclusions and Future Work
text: In this paper, we proposed a simple yet very effective consistency training method built upon dropout, namely R-Drop, which minimizes the bidirectional KL-divergence of the output distributions of any pair of sub models sampled from dropout in model training. Experimental results on 18 popular deep learning datasets show that not only can our R-Drop effectively enhance strong models, e.g., ViT, BART, Roberta-large, but also work well on large-scale datasets and even achieve SOTA performances when combined with vanilla Transformer on WMT14 English→German and English→French translations. Due to the limitation of computational resources, for pre-training related tasks, we only tested R-Drop on downstream task fine-tuning in this work. We will test it on pre-training in the future. In this work, we focused on Transformer based models. We will apply R-Drop to other network architectures such as convolutional neural networks.
Acknowledgments and Disclosure of Funding
We would like to thank the reviewers for their constructive comments. Juntao Li is the corresponding author. This work was supported by the National Science Foundation of China (NSFC No. 62036004).

Review summaries:
This paper proposes a simple yet effective regularization method for deep learning models. The model performs two forward passes with different dropout masks for the same input and minimizes the difference between the two output distributions by the bidirectional KL divergence. The proposed method has been sufficiently validated on both CV and NLP benchmarks. 

This paper proposed to forward an input to neural networks twice with two different dropout masks, and apply a consistent loss as regularization, named R-Drop, between the two outputs to penalize the discrepancy. KL divergence of the two output probabilities is used as the consistent loss. Experiments on different NLP tasks demonstrate the effectiveness of R-Drop.

The paper proposes the dropout-based regularization method R-Drop. The proposed R-Drop further regularizes the output predictions of sub-models from dropout.  The theoretical analysis was performed to show that the proposed method reduces the freedom of model parameters. The empirical results on multiple datasets were performed to show that the proposed method improves the performance (accuracy) for four NLP tasks and one CV task. 

The paper introduces a variant of dropout called R-drop. First, the data goes through the forward pass once, and we obtain the output distribution. Next, the data goes through the forward pass (with the same dropout rate but different dropout neurons) another time, and we obtain the output distribution. The goal is to minimize the loss, which is the standard negative log-likelihood (adding two NLL losses from two different passes together) together with KL divergence (KL(p1 || p2) + KL(p2 || p1)). There is large performance gain on machine translation, GLUE tasks, as well as language modeling.