SlowGuess/NAACL_2021_OCR huggingface dataset:
Zhen Wu $^{1*}$  Lijun Wu $^{2}$  Qi Meng $^{2}$  Yingce Xia $^{2}$  Shufang Xie $^{2}$
Tao Qin² Xinyu Dai¹ Tie-Yan Liu²
$^{1}$ National Key Laboratory for Novel Software Technology, Nanjing University
wuz@smail.nju.edu.cn, daixinyu@nju.edu.cn
{Lijun.Wu,meq,yingce.xia,shufxi,taoqin,tyliu}@microsoft.com
Transformer architecture achieves great success in abundant natural language processing tasks. The over-parameterization of the Transformer model has motivated plenty of works to alleviate its overfitting for superior performances. With some explorations, we find simple techniques such as dropout, can greatly boost model performance with a careful design. Therefore, in this paper, we integrate different dropout techniques into the training of Transformer models. Specifically, we propose an approach named UniDrop to unite three different dropout techniques from fine-grain to coarse-grain, i.e., feature dropout, structure dropout, and data dropout. Theoretically, we demonstrate that these three dropouts play different roles from regularization perspectives. Empirically, we conduct experiments on both neural machine translation and text classification benchmark datasets. Extensive results indicate that Transformer with UniDrop can achieve around 1.5 BLEU improvement on IWSLT14 translation tasks, and better accuracy for the classification even using strong pre-trained RoBERTa as backbone.
In recent years, Transformer (Vaswani et al., 2017) has been the dominant structure in natural language processing (NLP), such as neural machine translation (Vaswani et al., 2017), language modeling (Dai et al., 2019) and text classification (Devlin et al., 2019; Liu et al., 2019). To further improve the model performance, there has been much effort in designing better architectures or introducing external knowledge into Transformer models (Wu et al., 2019; Lu et al., 2019; Kitaev et al., 2020; Ahmed et al., 2017; Hashemi et al., 2020), which increases computational costs or requires extra resources.
Despite the effectiveness of above strategies, the over-parameterization and overfitting is still
a crucial problem for Transformer. Regularization methods such as weight decay (Krogh and Hertz, 1992), data augmentation (Sennrich et al., 2016a), dropout (Srivastava et al., 2014), parameter sharing (Dehghani et al., 2018; Xia et al., 2019) are all widely adopted to address overfitting. Among these regularization approaches, dropout (Srivastava et al., 2014), which randomly drops out some hidden units during training, is the most popular one and various dropout techniques have been proposed for Transformer. For example, Fan et al. (2020a) propose LayerDrop, a random structured dropout, to drop certain layers of Transformer during training. Zhou et al. (2020) alternatively propose DropHead as a structured dropout method for regularizing the multi-head attention mechanism. Both of them achieved promising performances. One great advantage of dropout is that it is free of additional computational costs and resource requirements. Hence we ask one question: can we achieve stronger or even state-of-the-art (SOTA) results only relying on various dropout techniques instead of extra model architecture design or knowledge enhancement?
To this end, in this paper, we propose UniDrop to integrate three different-level dropout techniques from fine-grain to coarse-grain, feature dropout, structure dropout, and data dropout, into Transformer models. Feature dropout is the conventional dropout (Srivastava et al., 2014) that we introduced before, which is widely applied on hidden representations of networks. Structure dropout is a coarse-grained control and aims to randomly drop some entire substructures or components from the whole model. In this work, we adopt the aforementioned LayerDrop (Fan et al., 2020a) as our structure dropout. Different from the previous two dropout methods, data dropout (Iyyer et al., 2015) is performed on the input data level, which serves as a data augmentation method by randomly dropping out some tokens in an input sequence.
* This work was done when Zhen Wu was a research intern at Microsoft Research Asia.
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3865-3878 June 6-11, 2021. ©2021 Association for Computational Linguistics
(a) Transformer architecture.
(b) Structure and overview of feature dropout.
Figure 1: Transformer structure and feature dropout applied in different Transformer components.
We first theoretically analyze different regularization roles played by the three dropout techniques, and we show they can improve the generalization ability from different aspects. Then, we provide empirical evaluations of the UniDrop approach. We conduct experiments on neural machine translation with 8 translation datasets, and text classification task with 8 benchmark datasets. On both sequence generation and classification tasks, experimental results show that the three dropouts in UniDrop can jointly improve the performance of Transformer.
- We introduce UniDrop, which unites three different dropout techniques into a robust one for Transformer, to jointly improve the performance of Transformer without additional computational cost and prior knowledge.
- We theoretically demonstrate that the three dropouts, i.e., feature dropout, structure dropout, and data dropout play different roles in preventing Transformer from overfitting and improving the robustness of the model.
- Extensive results indicate that Transformer models with UniDrop can achieve strong or even SOTA performances on sequence generation and classification tasks. Specifically, around 1.5 BLEU improvement on IWSLT14 translation tasks, and better accuracy for classification even using strong pre-trained model RoBERTa as backbone.
Feature dropout (FD) and structure dropout (SD) are highly coupled with model architecture. Therefore, we briefly recap Transformer and refer the readers to Vaswani et al. (2017) for details.
As shown in Figure 1a, Transformer is stacked by several identical blocks, and each block contains two sub-layers, which are multi-head self-attention layer and position-wise fully connected feed-forward layer. Each sub-layer is followed by an AddNorm operation that is a residual connection Add (He et al., 2016) and a layer normalization LN (Ba et al., 2016).
Multi-head Attention sub-layer consists of multiple parallel attention heads, and each head maps the query  $\mathbf{Q}$  and a set of key-value pairs  $\mathbf{K}, \mathbf{V}$  to an output through a scale dot-product attention:
$$
\operatorname {A t t n} (\mathbf {Q}, \mathbf {K}, \mathbf {V}) = \operatorname {s o f t m a x} \left(\frac {\mathbf {Q} \mathbf {K} ^ {\top}}{\sqrt {d _ {k}}}\right) \mathbf {V}, \tag {1}
$$
where  $d_{k}$  is the dimension of query and key, and  $\frac{1}{\sqrt{d_k}}$  is a scaling factor. The outputs of these heads are then concatenated and projected again to result in the final values.
Position-wise Feed-Forward sub-layer applies two linear transformations with an inner ReLU (Nair and Hinton, 2010) activation:
$$
\operatorname {F F N} (\mathbf {x}) = \max  (0, \mathbf {x} \mathbf {W} _ {1} + \mathbf {b} _ {1}) \mathbf {W} _ {2} + \mathbf {b} _ {2}, \tag {2}
$$
where  $\mathbf{W}$  and  $\mathbf{b}$  are parameters.
The output of each sub-layer is then followed with AddNorm: AddNorm(x) = LN(Add(x)).
3 UniDrop
In this section, we first introduce the details of the three different levels of dropout techniques we study, feature dropout, structure dropout and data dropout. Then we provide the theoretical analysis of these dropout methods on the regularization perspectives. Finally, we present our proposed UniDrop approach for training Transformer.
3.1 Feature Dropout
The feature dropout (FD), as a well-known regularization method, is proposed by Srivastava et al. (2014), which is to randomly suppress neurons of neural networks during training by setting them to 0 with a pre-defined probability  $p$ .
In practice, dropout is applied to the output of each sub-layer by default. Besides, Transformer also contains two specific feature dropouts for multi-head attention and activation layer of feedforward network. In this work, we also explore their effects on the performance of Transformer.
- FD-1 (attention dropout): according to Equation (1), we can obtain attention weight matrix  $\mathbf{A} = \mathbf{Q}\mathbf{K}^{\top}$  towards value sequence  $\mathbf{V}$ . Our FD-1 is applied to the attention weight  $\mathbf{A}$ .
- FD-2 (activation dropout): FD-2 is employed after the activation function between the two linear transformations of FFN sub-layer.
In addition to the above FDs for Transformer, we still find the risk of overfitting in pre-experiments. Therefore, we further introduce another two feature dropouts into the model architecture:
- FD-3 (query, key, value dropout): FD-1 is used to improve generalization of multi-head attention. However, it is directly applied to the attention weights  $\mathbf{A}$ , where drop value  $\mathbf{A}(i,j)$  means ignore the relation between token  $i$  and token  $j$ , thus a larger FD-1 means a larger risk of losing some critical information from sequence positions. To alleviate this potential risk, we add dropout to query, key, and value before the calculation of attention.
- FD-4 (output dropout): we also apply dropout to the output features before linear transformation for softmax classification. Specifically, when dealing with sequence-to-sequence tasks such as machine translation, we add FD-4 to the output features of the last layer in the
Transformer decoder, otherwise the last layer of the Transformer encoder.
The positions of each feature dropout applied in Transformer<sup>1</sup> are shown in Figure 1b.
3.2 Structure Dropout
There are three structure dropouts, respectively LayerDrop (Fan et al., 2020a), DropHead (Zhou et al., 2020) and HeadMask (Sun et al., 2020), which are specifically designed for Transformer.
Some recent studies (Voita et al., 2019; Michel et al., 2019) show multi-head attention mechanism is dominated by a small portion of attention heads. To prevent domination and excessive coadaptation between different attention heads, Zhou et al. (2020) and Sun et al. (2020) respectively propose structured DropHead and HeadMask that drop certain entire heads during training. In contrast, LayerDrop (Fan et al., 2020a) is a higher-level and coarser-grained structure dropout. It drops some entire layers at training time and directly reduces the Transformer model size.
In this work, we adopt LayerDrop as the structure dropout to incorporate it into our UniDrop.
3.3 Data Dropout
Data dropout aims to randomly remove some words in the sentence with a pre-defined probability. It is often used as a data augmentation technique (Wei and Zou, 2019; Xie et al., 2020). However, directly applying vanilla data dropout is hard to keep the original sequence for training, which leads to the risk of losing high-quality training samples. To address this issue, we propose a two-stage data dropout strategy. Specifically, given a sequence, with probability  $p_k$  (a hyperparameter lies in  $(0,1)$ ), we keep the original sequence and do not apply data dropout. If data dropout is applied, for each token, with another probability  $p$  (another hyperparameter lies in  $(0,1)$ ), we will drop the token.
In this section, we provide theoretical analysis for feature dropout, structure dropout and data dropout, to show their different regularization effects. We first re-formulate the three dropout methods. For some probability  $p$  and layer representation  $h \in \mathbb{R}^d$  (i.e.,  $h$  is the vector of outputs of some layer), we
<sup>1</sup>We also explored other positions for feature dropout, but their performances are not so good (see Appendix A.3).
randomly sample a scaling vector  $\xi \in \mathbb{R}^d$  with each independent coordinate as follows:
$$
\xi_ {i} = \left\{ \begin{array}{c l} - 1 & \text {w i t h p r o b a b i l i t y} p \\ \frac {p}{1 - p} & \text {w i t h p r o b a b i l i t y} 1 - p. \end{array} \right. \tag {3}
$$
Here,  $i$  indexes a coordinate of  $\xi$ ,  $i \in [1, \dots, d]$ . Then feature dropout can be applied by computing
$$
h _ {f d} = (\mathbf {1} + \xi) \odot h,
$$
where  $\odot$  denotes element-wised product and  $1 = (1, 1, \dots, 1)'$ .
We use  $F(h_{fd}(x))$  to denote the output of a model after dropping feature from a hidden layer and  $\mathcal{L}$  to denote the loss function. Similar to Wei et al. (2020), we apply Taylor expansion to  $\mathcal{L}$  and take expectation to  $\xi$ :
$$
\begin{array}{l} \mathbb {E} _ {\xi} \mathcal {L} (F (h _ {f d} (x))) = \mathbb {E} _ {\xi} \mathcal {L} (F ((\mathbf {1} + \xi) \odot h (x))) \\ \approx \mathcal {L} (F (h (x)) + \frac {1}{2} \mathbb {E} _ {\xi} (\xi \odot h (x)) ^ {T} D _ {h} ^ {2} \mathcal {L} (x) (\xi \odot h (x)) \\ = \mathcal {L} (F (h (x)) + \frac {p}{2 (1 - p)} \sum_ {j = 1} ^ {d} D _ {h _ {j}, h _ {j}} ^ {2} \mathcal {L} (x) \cdot h _ {j} (x) ^ {2}, \tag {4} \\ \end{array}
$$
where  $D_h^2\mathcal{L}$  is the Hessian matrix of loss with respect to hidden output  $h$  and  $D_{h_j,h_j}^2\mathcal{L}(x)$  is the  $j$ -th diagonal element of  $D_h^2\mathcal{L}$ . Expect the original loss  $\mathcal{L}(F(h(x)))$ , the above formula shows that feature dropout implicitly regularize the term  $\sum_{j = 1}^{d}D_{h_j,h_j}^2\mathcal{L}(x)\cdot h_j(x)^2$ , which relates to the trace of the Hessian.
For structure dropout, we use a 1-dim random scalar  $\eta \in \mathbb{R}$  whose distribution is:  $\eta = -1$  with probability  $p$ , and  $\eta = 0$  with probability  $1 - p$ . The structure dropout is similarly applied by computing  $h_{sd} = (1 + \eta)\cdot h$ .
For input data  $x \in \mathbb{R}^m$ , here  $x$  is a sequence of tokens and  $m$  is the sequence length, we sample a random scaling vector  $\beta \in \mathbb{R}^m$  with independent random coordinates where each coordinate is identically distributed as  $\eta$ . The input data after drop data becomes  $x_{dd} = (1 + \beta) \odot x$ .
Similar to feature dropout, we can obtain that data dropout implicitly optimizes the regularized loss as follows:  $\mathcal{L}(F(h(x))) - p\cdot x^T\nabla_x\mathcal{L}(x) + p\cdot \sum_{j = 1}^{m}D_{x_j,x_j}^2\mathcal{L}(x)\cdot x_j^2$  , and structure dropout implicitly optimizes the regularized loss:  $\mathcal{L}(F(h(x)))-$ $p\cdot h(x)^T\nabla_h\mathcal{L}(x) + p\cdot \sum_{i,j = 1}^{m}D_{h_i,h_j}^2\mathcal{L}(x)\cdot h_i(x)h_j(x)$  where  $D_{h_i,h_j}^2\mathcal{L}(x)$  is the  $(i,j)$  -th element in Hessian matrix  $D_h^2\mathcal{L}$
Interpretation From the above analysis, we can conclude that feature dropout, structure dropout and data dropout regularize different terms of the
Figure 2: Different dropout components in UniDrop. The gray positions denote applying the corresponding dropout.
model, and they can not be replaced by each other. (1) Because the hidden output will be normalized by layer normalization, the term  $h(x)^T \nabla_h \mathcal{L}(x)$  equals to zero according to Lemma 2.4 in Arora et al. (2019). Therefore, structure dropout implicitly regularizes the term  $\sum_{i,j=1}^{m} D_{h_i, h_j}^2 \mathcal{L}(x)$ . Hence, structure dropout can regularize the whole elements of Hessian of the model with respect to hidden output, while feature dropout only regularizes the diagonal elements of the Hessian. Thus, integrating structure dropout and feature dropout can regularize every component of Hessian with emphasizing the diagonal elements of the Hessian. (2) Since  $x$  is also normalized, the term  $x^T \nabla_x \mathcal{L}(x)$  equals to zero according to Lemma 2.4 in Arora et al. (2019). Different from feature dropout and structure dropout, data dropout regularizes Hessian of loss with respect to input data.
Regularizing Hessian matrix with respect to both input and hidden output can improve model robustness and hence the generalization ability. We put more details in Appendix A.1.
3.5 UniDrop Integration
From the above theoretical analysis, the three dropout techniques are performed in different ways to regularize the training of Transformer, each with unique property to improve the model generalization. Therefore, we introduce UniDrop to take the most of each dropout into Transformer. The overview of UniDrop is presented in Figure 2.
To better view each dropout in a model forward pass, we only show a three layers of architecture in Figure 2, and each layer with one specific dropout technique. The data dropout is applied in the input layer by dropping out some word embeddings (e.g., embedding of word  $t_i$  is dropped). In the middle layer, the feature dropout randomly drops several neurons in each word representations (e.g., the third neurons of word  $t_{i-1}$  is dropped). The last layer is
directly dropped out through layer dropout2.
We conduct experiments on both sequence generation and classification tasks, specifically, neural machine translation and text classification, to validate the effectiveness of UniDrop for Transformer.
In this section, we introduce the detailed settings for the neural machine translation tasks and report the experimental results.
We adopt the widely acknowledged IWSLT14 datasets<sup>3</sup> with multiple language pairs, including English↔German (En↔De), English↔Romanian (En↔Ro), English↔Dutch (En↔Nl), and English↔Portuguese-Brazil (En↔Pt-br), a total number of 8 translation tasks. Each dataset contains about  $170\mathrm{k}\sim 190\mathrm{k}$  translation data pairs. The datasets are processed by Moses toolkit<sup>4</sup> and byte-pair-encoding (BPE) (Sennrich et al., 2016b) is applied to obtain subword units. The detailed statistics of datasets are shown in Appendix A.2.
4.1.2 Model
We use the transformer_iwslt_de_en configuration for all Transformer models. Specifically, the encoder and decoder both consist of 6 blocks. The source and target word embeddings are shared for each language pair. The dimensions of embedding and feed-forward sub-layer are respectively set to 512 and 1024, the number of attention heads is 4. The default dropout (not our four feature dropout) rate is 0.3 and weight decay is 0.0001. All models are optimized with Adam (Kingma and Ba, 2015) and the learning rate schedule is same as in Vaswani et al. (2017). The weight of label smoothing (Pereyra et al., 2017) is set to 0.1.
For the Transformer models with our UniDrop, we set all feature dropout rates to 0.1. The structure dropout LayerDrop is only applied to the decoder with rate 0.1. For the data dropout, the sequence
keep rate  $p_k$  and token dropout rate  $p$  are respectively 0.5 and 0.2. The other settings are the same as the configuration of the baseline Transformer.
To evaluate the model performance, we use beam search (Sutskever et al., 2014) algorithm to generate the translation results. The beam width is 5 and the length penalty is 1.0. The evaluation metric is the tokenized BLEU (Papineni et al., 2002) score with multi-bleu.perl script<sup>6</sup>. We repeat each experiment three times with different seeds and report the average BLEU.
Table 1 shows the BLEU results of the Transformer baselines and models with different dropouts. Compared with baselines, we can see that the dropouts FD, SD, or DD all bring some improvements<sup>7</sup>. This observation verifies the existence of overfitting in the Transformer. In contrast, our model Transformer+UniDrop achieves the most improvements across all translation tasks, which demonstrates the effectiveness of UniDrop for the Transformer architecture. To further explore the effects of the three different grained dropouts in UniDrop, we conduct ablation studies and respectively remove the FD, SD, and DD from Transformer+UniDrop. The results in Table 1 show that three ablated models obtain lower BLEU scores compared to the full model. This observation validates the necessity of them for UniDrop. Among all ablation versions, the Transformer-UniDrop w/o FD obtains the least improvements. It is reasonable because FD actually contains four feature dropouts on different positions, which can effectively prevent Transformer from overfitting.
To show the superiority of UniDrop, we also compare the Transformer+UniDrop with several existing works on the widely acknowledged benchmark IWSLT14 De  $\rightarrow$  En translation. These works improve machine translation from different aspects, such as the training algorithm design (Wang et al., 2019b), model architecture design (Lu et al., 2019; Wu et al., 2019) and data augmentation (Gao et al., 2019). The detailed results are shown in Table 2. We can see that the Transformer model with our UniDrop outperforms all previous works and achieve state-of-the-art performance, with 36.88
2Except the data dropout is only applied in the input layer, feature/structure dropout can be applied in each layer.
3https://wit3.fbk.eu/mt.php?release=2014-01
4https://github.com/moses-smt/ mosesdecoder/tree/master/scripts
5https://github.com/pytorch/fairseq
https://github.com/moses-smt/ mosesdecoder/blob/master/scripts/ generic/multi-bleu.perl
The dropout rates of model Transformer+FD, Transformer+SD, Transformer+DD are tuned with IWSLT14 De→En dev set and respectively set to 0.2, 0.2, 0.3.
En→De	De→En	En→Ro	Ro→En	En→NI	NI→En	Nn→Pt-br	Pt-br→En	Avg.	△
Transformer	28.67	34.84	24.74	32.14	29.64	33.28	39.08	43.63	33.25	-
+FD	29.61	36.08	25.45	33.12	30.37	34.50	40.10	44.74	34.24	+0.99
+SD	29.03	35.09	25.03	32.69	29.97	33.94	39.78	44.02	33.69	+0.44
+DD	28.83	35.26	24.98	32.76	29.72	34.00	39.50	43.71	33.59	+0.34
+UniDrop	29.99	36.88	25.77	33.49	31.01	34.80	40.62	45.62	34.77	+1.52
w/o FD	29.24	35.68	25.18	33.17	30.16	33.90	39.97	44.81	34.01	+0.76
w/o SD	29.92	36.70	25.59	33.26	30.55	34.75	40.45	45.60	34.60	+1.35
w/o DD	29.76	36.38	25.44	33.26	30.86	34.55	40.37	45.27	34.49	+1.24
Table 1: Machine translation results of the standard Transformer and our models on various IWSLT14 translation datasets. The “+FD”, “+SD”, “+DD”, and “+UniDrop” denotes applying the feature dropout, structure dropout, data dropout, or UniDrop to the standard Transformer. The “w/o FD”, “w/o SD” and “w/o DD” respectively indicate the removal of the feature dropout, structure dropout, or data dropout from the model Transformer+UniDrop. Avg. and  $\triangle$  denote the average results of the 8 translation tasks and improvements compared with the standard Transformer. Best results are in bold.
Approaches	BLEU
Adversarial MLE (Wang et al., 2019b)	35.18
DynamicConv (Wu et al., 2019)	35.20
Macaron (Lu et al., 2019)	35.40
IOT (Zhu et al., 2021)	35.62
Soft Contextual Data Aug (Gao et al., 2019)	35.78
BERT-fused NMT (Zhu et al., 2020)	36.11
MAT (Fan et al., 2020b)	36.22
MixReps+co-teaching (Wu et al., 2020)	36.41
Transformer	34.84
+UniDrop	36.88
Table 2: Comparison with existing works on IWSLT-2014 De→En translation task.
Approaches	En→De	Ro→En	NI→En
MAT (Fan et al., 2020b)	29.90	-	-
MixReps+co-teaching (Wu et al., 2020)	29.93	33.12	34.45
Transformer	28.67	32.14	33.38
+UniDrop	29.99	33.49	34.80
Table 3: Comparison with existing works on IWSLT-2014 En→De, Ro→En, and Nl→En translation tasks.
BLEU score. Especially, it surpasses the BERT-fused NMT model (Zhu et al., 2020), which incorporates the pre-trained language model BERT, by a non-trivial margin. We also show some comparisons on IWSLT14 En→De, Ro→En, and Nl→En translations, the results are shown in Table 3.
According to the above results, UniDrop successfully unites the FD, SD, and DD, and finally improves the performance of Transformer on neural machine translation tasks, without any additional computation costs and resource requirements.
4.2 Text Classification
We also conduct experiments on text classification tasks to further demonstrate the effectiveness of UniDrop for the Transformer models.