SlowGuess/NAACL_2021_OCR huggingface dataset:
We evaluate different methods on the text classification task based on 8 widely-studied datasets, which can be divided into two groups. The first group is from GLUE tasks (Wang et al., 2019a), and they are usually used to evaluate the performance of the large-scale pre-trained language models after fine-tuning. The second group is some typical text classification datasets that are widely used in previous works (Voorhees and Tice, 1999; Maas et al., 2011; Zhang et al., 2015). The statistics of all datasets are shown in Appendix A.2.
4.2.2 Model
We employ RoBERTa<sub>BASE</sub> (Liu et al., 2019) as the strong baseline and fine-tune it on the text classification datasets. Different from BERT<sub>BASE</sub> (Devlin et al., 2019), RoBERTa<sub>BASE</sub> is pre-trained with dynamic masking, full-sentences without NSP loss and a larger mini-batches. It has 12 blocks, and the dimensions of embedding and FFN are 768 and 3072, the number of attention heads is 12. When fine-tuning, we set the batch size to 32 and the max epoch to 30. Adam is applied to optimize the models with a learning rate of 1e-5 and a warm-up step ratio of 0.1. We employ the polynomial decay strategy to adjust the learning rate. The default dropout and weight decay are both set to 0.1.
When adding UniDrop to RoBERTa<sub>BASE</sub>, we empirically set feature dropout rate and LayerDrop rate to 0.1. For data dropout, the sequence keep rate  $p_k$  and token dropout rate  $p$  are respectively 0.5 and 0.1. The other settings are the same as in the baseline RoBERTa<sub>BASE</sub>. We use the standard accuracy to evaluate different methods on text classification tasks.
MNLI	QNLI	SST-2	MRPC
BiLSTM+Attn, CoVe	67.9	72.5	89.2	72.8
BiLSTM+Attn, ELMo	72.4	75.2	91.5	71.1
BERTBASE	84.4	88.4	92.9	86.7
BERTLARGE	86.6	92.3	93.2	88.0
RoBERTaBASE	87.1	92.7	94.7	89.0
+UniDrop	87.8	93.2	95.5	90.4
Table 4: Accuracy on GLUE tasks (dev set). The models BiLSTM+Attn, CoVe and BiLSTM+Attn, ELMo are from Wang et al. (2019a). Best results are in bold.
IMDB	Yelp	AG	TREC
Char-level CNN	-	62.05	90.49	-
VDCNN	-	64.72	91.33	-
DPCNN	-	69.42	93.13	-
ULMFiT	95.40	-	94.99	96.40
BERTBASE	94.60	69.94	94.75	97.20
RoBERTaBASE	95.7	70.9	95.1	97.6
+UniDrop	96.0	71.4	95.5	98.0
Table 5: Accuracy on the typical text classification datasets. Char-level CNN and VDCNN are from Zhang et al. (2015) and Conneau et al. (2017), DPCNN and ULMFiT are from Johnson and Zhang (2017) and Howard and Ruder (2018). Best results are in bold.
Table 4 and Table 5 respectively show the accuracy of different models on GLUE tasks and typical text classification datasets.
Compared with the conventional BiLSTM and CNN based models, we can observe the pre-trained models, including ULMFiT, BERT, RoBERTa, achieve obvious improvements on most datasets. Benefiting from better training strategy, RoBERTaBASE outperforms  $\mathrm{BERT}_{\mathrm{BASE}}$  and even  $\mathrm{BERT}_{\mathrm{LARGE}}$  on GLUE tasks.
We can see our proposed UniDrop further improve the performance RoBERTaBASE on both small-scale and large-scale datasets. Specifically, UniDrop brings about 0.4 improvements of accuracy on the typical text classification datasets from Table 5. In contrast, RoBERTaBASE+UniDrop achieves more improvements on GLUE tasks. The experimental results on the 8 text classification benchmark datasets consistently demonstrate the facilitation of UniDrop for Transformer. We show more results and ablation study on text classification task in Appendix A.5.
In this section, we use IWSLT14 De  $\rightarrow$  En translation as the analysis task to investigate the capability of UniDrop to avoid overfitting, as well as the effects of different dropout components and dropout
Figure 3: The dev loss of different models on IWSLT14 De  $\rightarrow$  En translation task.
rates on UniDrop.
5.1 Overfitting
To show the superiority of UniDrop to prevent Transformer from overfitting, we compare the dev loss during training of Transformer, Transformer with each dropout technique, Transformer+UniDrop, and ablated models of Transformer+UniDrop. Figure 3 shows loss curves of different models.
We can observe that the standard Transformer is quickly overfitted during training, though it is equipped with a default dropout. In contrast, the feature dropout, structure dropout, and data dropout, as well as the combinations of any two dropouts (i.e., ablated models), greatly reduce the risk of overfitting to some extent. Among all compared models, our Transformer+UniDrop achieves the lowest dev loss and shows great advantage to prevent Transformer from overfitting. Besides, we also find that the dev loss of Transformer+UniDrop continuously falls until the end of the training. We stop it to keep training epochs of all models same for a fair comparison.
In Appendix A.4, we also plot the curves of training loss for the above models, together with the dev loss, to make a better understanding of the regularization effects from these dropout techniques.
In Table 1, we have presented some important ablation studies by removing FD, SD, or DD from UniDrop. The consistent decline of BLEU scores demonstrates their effectiveness. Besides, we further investigate the effects of the two existing feature dropouts FD-1, FD-2, two new feature dropouts FD-3, FD-4, and our proposed two-stage
(a). Varying FD rate.
(b). Varying SD rate.
Figure 4: The BLEU scores of Transformer+UniDrop on IWSLT14 De→En translation dev set and test test, with varying the rates of FD, SD and DD respectively.
(c). Varying DD rate.
De→En	En→De	Ro→En
Transformer	34.84	28.67	32.14
+UniDrop	36.88	29.99	33.49
w/o FD-1	36.72	29.84	33.33
w/o FD-2	36.57	29.76	33.28
w/o FD-3	36.59	29.83	33.31
w/o FD-4	36.65	29.59	33.24
w/o 2-stage DD	36.61	29.78	33.12
Table 6: Ablation study of data dropout and different feature dropouts on IWSLT14 De→En, En→De, and Ro→En translation tasks.
data dropout strategy on Transformer models. The experimental results are shown in Table 6.
From Table 6, we can see the four ablation models removing FDs underperform the full model Transformer+UniDrop, which means they can work together to prevent Transformer from overfitting. In multi-head attention module, FD-3 brings more BLUE improvement than FD-1. This comparison shows the insufficiency of only applying FD-1 for the Transformer architecture. The Transformer+UniDrop w/o 2-stage DD means we directly apply conventional data dropout to the sequence instead of our proposed 2-stage strategy. Compared with the full model, its performance also decreases. This shows the necessity of keeping the original sequence for data dropout.
5.3 Effects of Different Dropout Rates
To investigate the effects of FD, SD, and DD dropout rates on the UniDrop, we respectively vary them based on the setting  $(\mathrm{FD} = 0.1, \mathrm{SD} = 0.1, \mathrm{DD} = 0.2)$ . When varying one dropout component, we keep other dropout rates unchanged. Figure 4 shows the corresponding results.
We can observe that the performance of each dropout for Transformer+UniDrop first increases then decreases when varying the dropout rates from small to large. Especially, varying the rate for FD
dropout makes a more significant impact on the model performance since FD contains four feature dropout positions. In contrast, the DD is least sensitive to the dropout rate change, but it still plays a role in the model regularization.
6.1 Dropout
Dropout is a popular regularization method for neural networks by randomly dropping some neurons during training (Srivastava et al., 2014). Following the idea, there are abundant subsequent works designing specific dropout for specific architecture, such as StochasticDepth (Huang et al., 2016), DropPath (Larsson et al., 2017), DropBlock (Ghiasi et al., 2018) for convolutional neural networks, Variational Dropout (Gal and Ghahramani, 2016), ZoneOut (Krueger et al., 2017), and Word Embedding Dropout (Gal and Ghahramani, 2016) for recurrent neural networks. Recently, the Transformer architecture achieves great success in a variety of tasks. To improve generalization of Transformer, some recent works propose LayerDrop (Fan et al., 2020a), DropHead (Zhou et al., 2020) and HeadMask (Sun et al., 2020) as structured regularizations, and obtain better performance than standard Transformer. Instead of designing a specific dropout for Transformer, in this work, we focus on integrating the existing dropouts into one UniDrop to further improve generalization of Transformer without any additional cost.
6.2 Data Augmentation
Data augmentation aims at creating realistic-looking training data by applying a transformation to a sample, without changing its label (Xie et al., 2020). In NLP tasks, data augmentation often refers to back-translation (Sennrich et al., 2016a), word replacing/inserting/swapping/dropout (Wei
and Zou, 2019; Xie et al., 2020), etc. In this work, we adopt simple but effective word dropout as data level dropout in our UniDrop. We, additionally, design a two-stage data dropout strategy.
In this paper, we present an integrated dropout approach, UniDrop, to specifically regularize the Transformer architecture. The proposed UniDrop unites three different level dropout techniques from fine-grain to coarse-grain, feature dropout, structure dropout, and data dropout respectively. We provide a theoretical justification that the three dropouts play different roles in regularizing Transformer. Extensive results on neural machine translation and text classification datasets show that our Transformer+UniDrop outperforms the standard Transformer and various ablation versions. Further analysis also validates the effectiveness of different dropout components and our two-stage data dropout strategy. In conclusion, the UniDrop improves the performance and generalization of the Transformer without additional computational cost and resource requirement.
The authors would like to thank the anonymous reviewers for their valuable comments. Xinyu Dai and Lijun Wu are the corresponding authors. This work was partially supported by the NSFC (No. 61976114,61936012) and National Key R&D Program of China (No. 2018YFB1005102).
Karim Ahmed, Nitish Shirish Keskar, and Richard Socher. 2017. Weighted transformer network for machine translation. arXiv preprint arXiv:1711.02132.
Sanjeev Arora, Zhiyuan Li, and Kaifeng Lyu. 2019. Theoretical analysis of auto rate-tuning by batch normalization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. Layer normalization. arXiv preprint arXiv:1607.06450.
Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. In Proceedings of
the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017, pages 670-680.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G Carbonell, Quoc Le, and Ruslan Salakhutdinov. 2019. Transformer-xl: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2978-2988.
Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. 2018. Universal transformers. arXiv preprint arXiv:1807.03819.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186.
Angela Fan, Edouard Grave, and Armand Joulin. 2020a. Reducing transformer depth on demand with structured dropout. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020.
Yang Fan, Shufang Xie, Yingce Xia, Lijun Wu, Tao Qin, Xiang-Yang Li, and Tie-Yan Liu. 2020b. Multi-branch attentive transformer. CoRR, abs/2006.10270.
Yarin Gal and Zoubin Ghahramani. 2016. A theoretically grounded application of dropout in recurrent neural networks. In Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pages 1019-1027.
Fei Gao, Jinhua Zhu, Lijun Wu, Yingce Xia, Tao Qin, Xueqi Cheng, Wengang Zhou, and Tie-Yan Liu. 2019. Soft contextual data augmentation for neural machine translation. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28-August 2, 2019, Volume 1: Long Papers, pages 5539-5544.
Golnaz Ghiasi, Tsung-Yi Lin, and Quoc V. Le. 2018. **Dropblock: A regularization method for convolutional networks.** In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, 3-8 December 2018, Montréal, Canada, pages 10750-10760.
Helia Hashemi, Hamed Zamani, and W Bruce Croft. 2020. Guided transformer: Leveraging multiple external sources for representation learning in conversational search. In Proceedings of the 43rd International ACM SIGIR Conference on Research and
Development in Information Retrieval, pages 1131-1140.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778.
Judy Hoffman, Daniel A Roberts, and Sho Yaida. 2019. Robust learning with jacobian regularization. arXiv preprint arXiv:1908.02729.
Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers, pages 328-339.
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q. Weinberger. 2016. Deep networks with stochastic depth. In Computer Vision - ECCV 2016 - 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV, volume 9908 of Lecture Notes in Computer Science, pages 646-661. Springer.
Mohit Iyyer, Varun Manjunatha, Jordan Boyd-Graber, and Hal Daumé III. 2015. Deep unordered composition rivals syntactic methods for text classification. In Proceedings of the 53rd annual meeting of the association for computational linguistics and the 7th international joint conference on natural language processing (volume 1: Long papers), pages 1681-1691.
Daniel Jakubovitz and Raja Giryes. 2018. Improving dnn robustness to adversarial attacks using jacobian regularization. In Proceedings of the European Conference on Computer Vision (ECCV), pages 514-529.
Rie Johnson and Tong Zhang. 2017. Deep pyramid convolutional neural networks for text categorization. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 - August 4, Volume 1: Long Papers, pages 562-570.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. 2020. Reformer: The efficient transformer. In International Conference on Learning Representations.
Anders Krogh and John A Hertz. 1992. A simple weight decay can improve generalization. In Advances in neural information processing systems, pages 950-957.
David Krueger, Tegan Maharaj, János Kramár, Mohammad Pezeshki, Nicolas Ballas, Nan Rosemary Ke, Anirudh Goyal, Yoshua Bengio, Aaron C. Courville, and Christopher J. Pal. 2017. *Zoneout: Regularizing mnns by randomly preserving hidden activations.* In 5th International Conference on Learning Representations, ICLR 2017, Toulouse, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net.
Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. 2017. Fractalnet: Ultra-deep neural networks without residuals. In 5th International Conference on Learning Representations, ICLR 2017, Toulouse, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Yiping Lu, Zhuohan Li, Di He, Zhiqing Sun, Bin Dong, Tao Qin, Liwei Wang, and Tie-Yan Liu. 2019. Understanding and improving transformer from a multi-particle dynamic system point of view. arXiv preprint arXiv:1906.02762.
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In The 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Conference, 19-24 June, 2011, Portland, Oregon, USA, pages 142-150. The Association for Computer Linguistics.
Paul Michel, Omer Levy, and Graham Neubig. 2019. Are sixteen heads really better than one? In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, 8-14 December 2019, Vancouver, BC, Canada, pages 14014-14024.
Vinod Nair and Geoffrey E Hinton. 2010. Rectified linear units improve restricted boltzmann machines. In ICML.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, July 6-12, 2002, Philadelphia, PA, USA, pages 311-318. ACL.
Gabriel Pereyra, George Tucker, Jan Chorowski, Lukasz Kaiser, and Geoffrey E. Hinton. 2017. Regularizing neural networks by penalizing confident output distributions. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Workshop Track Proceedings. OpenReview.net.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016a. Improving neural machine translation models with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 86-96.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016b. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers. The Association for Computer Linguistics.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-1958.
Zewei Sun, Shujian Huang, Xinyu Dai, and Jiajun Chen. 2020. Alleviating the inequality of attention heads for neural machine translation. CoRR, abs/2009.09672.
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pages 3104-3112.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008.
Elena Voita, David Talbot, Fedor Moiseev, Rico Senrich, and Ivan Titov. 2019. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28-August 2, 2019, Volume 1: Long Papers, pages 5797-5808. Association for Computational Linguistics.
Ellen M. Voorhees and Dawn M. Tice. 1999. The TREC-8 question answering track evaluation. In Proceedings of The Eighth Text RETrieval Conference, TREC 1999, Gaithersburg, Maryland, USA, November 17-19, 1999, volume 500-246 of NIST Special Publication. National Institute of Standards and Technology (NIST).
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019a. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
Dilin Wang, ChengYue Gong, and Qiang Liu. 2019b. Improving neural language modeling via adversarial
training. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 6555-6565. PMLR.
Colin Wei, Sham Kakade, and Tengyu Ma. 2020. The implicit and explicit regularization effects of dropout. arXiv preprint arXiv:2002.12915.
Jason W. Wei and Kai Zou. 2019. EDA: easy data augmentation techniques for boosting performance on text classification tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pages 6381-6387.
Felix Wu, Angela Fan, Alexei Baevski, Yann Dauphin, and Michael Auli. 2019. Pay less attention with lightweight and dynamic convolutions. In International Conference on Learning Representations.
Lijun Wu, Shufang Xie, Yingce Xia, Yang Fan, Tao Qin, Jianhuang Lai, and Tie-Yan Liu. 2020. Sequence generation with mixed representations. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event.
Yingce Xia, Tianyu He, Xu Tan, Fei Tian, Di He, and Tao Qin. 2019. Tied transformers: Neural machine translation with shared encoder and decoder. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 5466-5473.
Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoc Le. 2020. Unsupervised data augmentation for consistency training. Advances in Neural Information Processing Systems, 33.
Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, pages 649-657.
Wangchunshu Zhou, Tao Ge, Furu Wei, Ming Zhou, and Ke Xu. 2020. Scheduled drophead: A regularization method for transformer models. In Findings, EMNLP 2020, Online Event, 16-20 November 2020, pages 1971-1980.
Jinhua Zhu, Lijun Wu, Yingce Xia, Shufang Xie, Tao Qin, Wengang Zhou, Houqiang Li, and Tie-Yan Liu. 2021. {IOT}: Instance-wise layer reordering for transformer structures. In International Conference on Learning Representations.
Jinhua Zhu, Yingce Xia, Lijun Wu, Di He, Tao Qin, Wengang Zhou, Houqiang Li, and Tie-Yan Liu. 2020. Incorporating BERT into neural machine translation. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
A.1 Supplementary materials for theoretical analysis
In this section, we explain why regularizing Hessian matrix with respect to input or hidden output can improve model robustness and generalization.
We use  $D_f^\alpha \mathcal{L}$  to denote the  $\alpha$ -order derivatives of loss  $\mathcal{L}$  with respect to  $f$ . If the hidden output is perturbed by  $\epsilon$ , i.e.,  $\tilde{h} = h + \epsilon$ , the  $k$ -th output  $F_k$  shifts to
$$
\begin{array}{l} F _ {k} (h + \epsilon) = F _ {k} (h) + \epsilon^ {T} J _ {F _ {k}, h} \\ + \frac {1}{2} \epsilon^ {T} \left(D _ {h} ^ {2} F _ {k} (h)\right) \epsilon + o \left(\epsilon^ {2}\right), \tag {5} \\ \end{array}
$$
where  $J_{F_k,h}(x)$  is the Jacobian between hidden output  $h$  and final output  $F_k$ .