SlowGuess/NAACL_2021_OCR huggingface dataset:
Structure dropout regularizes all elements in Hessian matrix  $D_h^2\mathcal{L}$ . For Hessian matrix of loss function, we have  $D_h^2\mathcal{L} = J_{F,h}^T (D_F^2\mathcal{L})J_{F,h} + \sum_k(D_{F_k}\mathcal{L})(D_h^2 F_k(h))$ . Thus, regularizing all elements in  $D_h^2\mathcal{L}$  means regularizing both  $J_{F,h}$  and  $D_h^2 F_k(h)$ . As shown in Eq.5, regularizing this two terms can make  $|F_{k}(h + \epsilon) - F_{k}(h)|$  smaller. Therefore, the robustness of the model is improved and the generalization ability of the model can also be improved (Hoffman et al., 2019; Jakubovitz and Giryes, 2018).
Feature dropout regularizes diagonal element of  $D_h^2\mathcal{L}$ . Using the approximation  $D_h^2\mathcal{L} \approx J_{F,h}^T(D_F^2\mathcal{L})J_{F,h}$  (Wei et al., 2020), regularizing diagonal elements  $D_h^2\mathcal{L}$  equals to regularizing norm of Jacobian, i.e.,  $\|J_{F,h}\|_2$  if  $D_F^2\mathcal{L}$  is roughly a diagonal matrix. For cross-entropy loss,  $D_F^2\mathcal{L} = \text{diag}(z) - zz^T$ , where  $z$  is the probability vector predicted by the model encoding the distribution over output class labels, the matrix  $D_F^2\mathcal{L}$  can be approximated by a diagonal matrix. Thus, feature dropout mainly regularizes the first-order coefficient  $J_{F_k,h}$  in Taylor expansion in Eq.5, which is different from structure dropout. Since Jacobian is an essential quantity for the generalization (Wei et al., 2020; Hoffman et al., 2019), emphasising this term is necessary for generalization although structure dropout can also regularize it.
Similar analysis can be applied to data dropout and we only need to replace hidden output  $h$  to the input  $x$ .
A.2 Statistics of Datasets
Table 7 and Table 8 respectively show the statistics of machine translation and text classification bench
Datasets	Train	Dev	Test
En←De	160k	7k	7k
En←Ro	180k	4.7k	1.1k
En←Nl	170k	4.5k	1.1k
En←Pt-br	175k	4.5k	1.2k
Table 7: Statistics for machine translation datasets.
Datasets	Classes	Train	Dev
MNLI	3	393k	20k
QNLI	2	105k	5.5k
SST-2	2	67k	0.9k
MRPC	2	3.7k	0.4k
Datasets	Classes	Train	Test
IMDB	2	25k	25k
Yelp	5	650k	50k
AG's News	4	120k	76k
TREC	6	5.4k	0.5k
Table 8: Statistics for text classification datasets.
mark datasets we used to evaluate the UniDrop for Transformer.
For machine translation tasks, the four language pairs all contain around  $170\mathrm{k}\sim 190\mathrm{k}$  training pairs.
Text classification experiments are conducted in GLUE tasks (Wang et al., 2019a) and typical text classification benchmarks datasets (Voorhees and Tice, 1999; Maas et al., 2011; Zhang et al., 2015). For GLUE tasks, we adopt the four datasets MNLI, QNLI, SST-2 and MRPC. They are used to evaluate the ability of models on language inference, sentiment classification and paraphrase detection. In typical text classification datasets, IMDB is binary film review classification task (Maas et al., 2011). Yelp and AG's News datasets are built by (Zhang et al., 2015), respectively for sentiment classification and topic classification. TREC is a question classification dataset consisting of 6 question types (Voorhees and Tice, 1999).
A.3 Dropout Attempts
Besides the different dropout methods introduced in Section 3, we also tried some other dropouts. We first introduce their settings. The 'QKV Projekt' applies dropout to query, key, and value after linear projection. In contrast, FD-3 is to add dropout to query, key, and value before projection. Similarly, 'LogitsDrop' means that we use dropout after obtaining output logits from output projection layer. Compared to LogitsDrop, FD-4 directly applies dropout before the output projection layer.
BLEU
Transformer	34.84
+FD-1, FD-2	35.46
+FD-1, FD-2, FD-3	36.10
+FD-1, FD-2, QKV proj	35.75
+FD-1, FD-2, FD-4	36.15
+FD-1, FD-2, LogitsDrop	36.00
+FD-1, FD-2, FD-3, LogitsDrop	36.06
+FD-1, FD-2, FD-3, FD-4	36.48
+FD-1, FD-2, Encoder LayerDrop	35.24
+FD-1, FD-2, Decoder LayerDrop	35.99
+FD-1, FD-2, Encoder&Decoder LayerDrop	35.74
+FD-1, FD-2, EncoderDrop	35.64
+FD-1, FD-2, DD	36.09
+FD-1, FD-2, FD-3, FD-4, Decoder LayerDrop	36.61
+UniDrop	36.88
'EncoderDrop' means that we randomly drop the whole information of Transformer encoder with a probability and only use previous outputs to generate the next token during training. Obviously, it is a language modeling task when dropping the encoder. 'Encoder LayerDrop' is that we apply LayerDrop only on the Transformer encoder. Table 9 shows the BLEU scores of different models on IWSLT-2014 De  $\rightarrow$  En translation task. All dropout rates are tuned within [0.1, 0.2, 0.3, 0.4] according to the performance of the dev set.
FD-1 and FD-2 are two existing feature dropouts for Transformer. We first use them and achieve better BLUE scores than the standard Transformer, which demonstrates the existence of serious overfitting in Transformer model. On this basis, we try to add further feature dropout to prevent Transformer from overfitting. However, we can see that QKK_Proj achieves fewer improvements compared with FD-3. Similarly, LogitsDrop also underperforms FD-4. Therefore, we finally use FD-3 and FD-4 as our feature dropout components together with FD-1 and FD-2.
Among all structure dropout models, decoder LayerDrop outperforms all compared methods. In contrast, EncoderDrop only brings small improvements. Surprisingly, we can see that here the encoder LayerDrop actually has a negative effect on Transformer. Thus we integrate the promising decoder LayerDrop as structured dropout component into UniDrop.
Table 9: The results of different dropouts on IWSLT14 De→En translation task.
MNLI	QNLI	SST-2	MRPC
RoBERTaBASE	87.1	92.7	94.7	89.0
+UniDrop	87.8	93.2	95.5	90.4
w/o FD	87.3	92.9	94.8	90.1
w/o SD	87.5	93.1	95.1	89.5
w/o DD	87.7	93.1	95.0	89.5
RoBERTaLAEGE	89.8	94.3	96.3	90.4
+UniDrop	90.2	94.8	96.6	91.4
w/o FD	89.9	94.6	96.2	90.4
w/o SD	90.0	94.6	96.3	90.7
w/o DD	90.2	94.7	95.2	90.7
Table 10: Ablation Study on GLUE tasks (dev set). The "w/o FD", "w/o SD", "w/o DD" indicate respectively removing feature dropout, structure dropout, and data dropout from RoBERTa<sub>BASE</sub>+UniDrop or RoBERTa<sub>LARGE</sub>+UniDrop.
A.4 Loss Curves
Figure 5 shows the loss curves of different models during training. Overall, we can see that our Transformer+UniDrop obtains the minimal gap of training loss and dev loss compared with other dropout models and the standard Transformer. This observation shows the better capability of UniDrop to prevent Transformer from overfitting. Benefitting from the advantage, Transformer+UniDrop achieves the best generalization and dev loss on IWSLT14 De→En translation task.
A.5 Ablation Study on Text Classification
Table 10 show the accuracy of standard RoBERTa<sub>BASE</sub> and RoBERTa<sub>LARGE</sub>, the models with UniDrop and corresponding ablated models on GLUE tasks. Compared the base
Figure 5: The training and dev loss of different models on IWSLT14 De→En translation task.
models RoBERTa<sub>BASE</sub> and RoBERTa<sub>LARGE</sub>, we can observe that UniDrop further improves their performance on text classification tasks. After removing FD, SD, or DD from UniDrop, the corresponding accuracy has decreased more or less. The consistent declines again demonstrate the necessity of the feature dropout, structure dropout and data dropout for UniDrop.