SlowGuess/ACL_2020_OCR huggingface dataset:
RoBERTaLARGE
RoBERTa (Liu et al., 2019c)	90.2/-	92.2/-	86.6	94.7	-/90.9	68.0	96.4	92.4/-
PGD (Zhu et al., 2020)	90.5/-	92.5/-	87.4	94.9	-/90.9	69.7	96.4	92.4/-
FreeAT (Zhu et al., 2020)	90.0/-	92.5/-	86.7	94.7	-/90.7	68.8	96.1	92.4/-
FreeLB (Zhu et al., 2020)	90.6/-	92.6/-	88.1	95.0	-/91.4	71.1	96.7	92.7/-
SMARTRoBERTa	91.1/91.3	92.4/89.8	92.0	95.6	89.2/92.1	70.6	96.9	92.8/92.6
Table 1: Main results on GLUE development set. The best result on each task produced by a single model is in bold and “-” denotes the missed result.
Model /#Train	CoLA 8.5k	SST 67k	MRPC 3.7k	STS-B 7k	QQP 364k	MNLI-m/mm 393k	QNLI 108k	RTE 2.5k	WNLI 634	AX	Score	#param
Human Performance	66.4	97.8	86.3/80.8	92.7/92.6	59.5/80.4	92.0/92.8	91.2	93.6	95.9	-	87.1	-
Ensemble Models
RoBERTa1	67.8	96.7	92.3/89.8	92.2/91.9	74.3/90.2	90.8/90.2	98.9	88.2	89.0	48.7	88.5	356M
FreeLB2	68.0	96.8	93.1/90.8	92.4/92.2	74.8/90.3	91.1/90.7	98.8	88.7	89.0	50.1	88.8	356M
ALICE3	69.2	97.1	93.6/91.5	92.7/92.3	74.4/90.7	90.7/90.2	99.2	87.3	89.7	47.8	89.0	340M
ALBERT4	69.1	97.1	93.4/91.2	92.5/92.0	74.2/90.5	91.3/91.0	99.2	89.2	91.8	50.2	89.4	235M*
MT-DNN-SMART†	69.5	97.5	93.7/91.6	92.9/92.5	73.9/90.2	91.0/90.8	99.2	89.7	94.5	50.2	89.9	356M
Single Model
BERTLarge5	60.5	94.9	89.3/85.4	87.6/86.5	72.1/89.3	86.7/85.9	92.7	70.1	65.1	39.6	80.5	335M
MT-DNN6	62.5	95.6	90.0/86.7	88.3/87.7	72.4/89.6	86.7/86.0	93.1	75.5	65.1	40.3	82.7	335M
T58	70.8	97.1	91.9/89.2	92.5/92.1	74.6/90.4	92.0/91.7	96.7	92.5	93.2	53.1	89.7	11,000M
SMARTRoBERTa	65.1	97.5	93.7/91.6	92.9/92.5	74.0/90.1	91.0/90.8	95.4	87.9	91.88	50.2	88.4	356M
Table 2: GLUE test set results scored using the GLUE evaluation server. The state-of-the-art results are in bold. All the results were obtained from https://gluebenchmark.com/leaderboard on December 5, 2019. SMART uses the classification objective on QNLI. Model references:  ${}^{1}$  Liu et al. (2019c);  ${}^{2}\mathrm{{Zhu}}$  et al. (2020);  ${}^{3}$  Wang et al. (2019);  ${}^{4}$  Lan et al. (2019);  ${}^{5}$  Devlin et al. (2019);  ${}^{6}$  Liu et al. (2019b);  ${}^{7}$  Raffel et al. (2019) and  ${}^{8}$  He et al. (2019), Kocijan et al. (2019). * ALBERT uses a model similar in size, architecture and computation cost to a 3,000M BERT (though it has dramatically fewer parameters due to parameter sharing). † Mixed results from ensemble and single of MT-DNN-SMOTE and with data augmentation.
MNLI task, the performance of SMART on the out-domain setting is better than the in-domain setting, e.g., (86.0% vs. 85.6%) by  $\mathrm{SMART}_{\mathrm{BERT}}$  and (91.3% vs. 91.1%) by  $\mathrm{SMART}_{\mathrm{RoBERTa}}$ , showing that our proposed approach alleviates the domain shifting issue. Furthermore, on the small tasks, the improvement of SMART is even larger. For example, comparing with BERT,  $\mathrm{SMART}_{\mathrm{BERT}}$  obtains  $71.2\%$  (vs.  $63.5\%$ ) on RTE and  $59.1\%$  (vs.  $54.7\%$ ) on CoLA in terms of accuracy, which are  $7.7\%$  and  $4.4\%$  absolute improvement for RTE and CoLA, respectively; similarly,  $\mathrm{SMART}_{\mathrm{RoBERTa}}$  outperforms RoBERTa  $5.4\%$  (92.0% vs.  $86.6\%$ ) on RTE and  $2.6\%$  (70.6% vs.  $68.0\%$ ) on CoLA.
We also compare SMART with a range of models which used adversarial training such as FreeLB. From the bottom rows in Table 1, SMART outperforms PGD and FreeAT across the all 8 GLUE tasks. Comparing with the current state-of-the-art adversarial training model, FreeLB, SMART outperforms it on 6 GLUE tasks out of a total of 8 tasks (MNLI, RTE, QNLI, MRPC, SST-2 and STS-B) showing the effectiveness of our model.
Table 2 summarizes the current state-of-the-art models on the GLUE leaderboard. SMART obtains a competitive result comparing with T5 (Raffel et al., 2019), which is the leading model at the GLUE leaderboard. T5 has 11 billion parameters,
while SMART only has 356 millions. Among this super large model (T5) and other ensemble models (e.g., ALBERT, ALICE), SMART, which is a single model, still sets new state-of-the-art results on SST-2, MRPC and STS-B. By combining with the Multi-task Learning framework (MT-DNN), MT-DNN-SMART obtains new state-of-the-art on GLUE, pushing the GLUE benchmark to  $89.9\%$ . More discussion will be provided in Section 5.3.
5 Experiment - Analysis and Extension
In this section, we first analyze the effectiveness of each component of the proposed method. We also study that whether the proposed method is complimentary to multi-task learning. We further extend SMART to domain adaptation and use both SNLI (Bowman et al., 2015) and SciTail (Khot et al., 2018) to evaluate the effectiveness. Finally, we verified the robustness of the proposed method on ANLI (Nie et al., 2019).
Note that due to the limitation of time and computational resources, all the experiments reported below are based on the BERT<sub>BASE</sub> model. In this section, we study the importance of each component of SMART: smoothness-inducing adversarial regularization and Bregman proximal point optimization. All models in this study used the BERT<sub>BASE</sub> as the encoder for fast training. Furthermore, we also include the BERT<sub>BASE</sub> model as an additional baseline for a fair comparison. SMART denotes the proposed model. Then we set  $\lambda_s$  to 0, which denotes as  $-\mathcal{R}_s$ . The model with  $\mu = 0$  is noted as  $-\mathcal{D}_{\mathrm{Breg}}$ .
Model	MNLI Acc	RTE Acc	QNLI Acc	SST Acc	MRPC Acc
BERT	84.5	63.5	91.1	92.9	89.0
SMART	85.6	71.2	91.7	93.0	91.3
-Rs	84.8	70.8	91.3	92.8	90.8
-DBreg	85.4	71.2	91.6	92.9	91.2
Table 3: Ablation study of SMART on 5 GLUE tasks. Note that all models used the  $\mathrm{BERT}_{\mathrm{BASE}}$  model as their encoder.
The results are reported in Table 3. It is expected that the removal of either component (smooth regularization or proximal point method) in SMART would result in a performance drop. For example, on MNLI, removing smooth regu
larization leads to a  $0.8\%$  (85.6% vs. 84.8) performance drop, while removing the Breg proximal point optimization, results in a performance drop of  $0.2\%$  (85.6% vs. 85.4%). It demonstrates that these two components complement each other. Interestingly, all three proposed models outperform the BERT baseline model demonstrating the effectiveness of each module. Moreover, we oversere that the generalization performance benefits more from SMART on small datasets (i.e., RTE and MRPC) by preventing overfitting.
To understand why SMART improves the performance, we analyze it on the ambiguous samples of MNLI dev set containing 3 classes, where each sample has 5 annotations. Based on the degree of agreement between these annotations, we divide the samples into 4 categories: 1)  $\mathbf{5 / 0 / 0}$  all five annotations are the same; 2)  $\mathbf{4 / 1 / 0}$  four annotations are the same; 3)  $\mathbf{3 / 2 / 0}$  three annotations are the same and the other two annotations are the same; 4)  $\mathbf{3 / 1 / 1}$  three annotations are the same and the other two annotations are different.
Figure 2 summarizes the results in terms of both accuracy and KL-divergence:  $-\frac{1}{n}\sum_{i=1}^{n}\sum_{j=1}^{3}p_{j}(x_{i})\log(f_{j}(x_{i}))$ . For a given sample  $x_{i}$ , the KL-Divergence evaluates the similarity between the model prediction  $\{f_{j}(x_{i})\}_{j=1}^{3}$  and the annotation distribution  $\{p_{j}(x_{i})\}_{j=1}^{3}$ . We observe that  $\mathrm{SMART}_{\mathrm{RoBERTa}}$  outperforms RoBERTa across all the settings. Further, on high degree of ambiguity (low degree of agreement),  $\mathrm{SMART}_{\mathrm{RoBERTa}}$  obtains an even larger improvement showing its robustness to ambiguity.
5.3 SMART with Multi-task Learning
It has been shown that multi-task learning (MTL, Caruana (1997); Liu et al. (2015, 2019b)) has a regularization effect via alleviating overfitting to a specific task. One question is whether MTL helps SMART as well. In this section, we are going to answer this question. Following Liu et al. (2019b), we first "pre-trained" shared embeddings using MTL with SMART, denoted as MT-DNN-SMART $^{8}$ , and then adapted the training data on each task on top of the shared embeddings. We also include a baseline which fine-tuned each task
Due to limitation of computational resources, we only trained jointly using MTL on MNLI, RTE, QNLI, SST and MRPC, while MT-DNN was trained on the whole GLUE tasks except CoLA.
Figure 2: Score breakdown by degree of agreement.
on the publicly released MT-DNN checkpoint  $^{9}$ , which is indicated as MT-DNN-SMARTv0.
Model	MNLI Acc	RTE Acc	QNLI Acc	SST Acc	MRPC F1
BERT	84.5	63.5	91.1	92.9	89.0
MT-DNN	85.3	79.1	91.5	93.6	89.2
SMART	85.6	71.2	91.6	93.0	91.3
MT-DNN-SMARTv0	85.7	80.2	92.0	93.3	91.5
MT-DNN-SMOTE	85.7	81.2	92.0	93.5	91.7
We observe that both MT-DNN and SMART consistently outperform the BERT model on all five GLUE tasks. Furthermore, SMART outperforms MT-DNN on MNLI, QNLI, and MRPC, while it obtains worse results on RTE and SST, showing that MT-DNN is a strong counterpart for SMART. By combining these two models, MT-DNN-SMART $_{v0}$  enjoys advantages of both and thus improved the final results. For example, it achieves  $85.7\%$ $(+0.1\%)$  on MNLI and  $80.2\%$ $(+1.1\%)$  on RTE comparing with the best results of MT-DNN and SMART demonstrating that these two techniques are orthogonal. Lastly we also trained SMART jointly and then finetuned on each task like Liu et al. (2019b). We observe that MT-DNN-SMART outperforms MT-DNN-SMART $_{v0}$  and MT-DNN across all 5 tasks (except MT-DNN
Table 4: Comparison between SMART and MTL.
Model	0.1%	1%	10%	100%
SNLI Dataset (Dev Accuracy%)
#Training Data	549	5,493	54,936	549,367
BERT	52.5	78.1	86.7	91.0
MT-DNN	82.1	85.2	88.4	91.5
MT-DNN-SMOTE	82.7	86.0	88.7	91.6
SciTail Dataset (Dev Accuracy%)
#Training Data	23	235	2,359	23,596
BERT	51.2	82.2	90.5	94.3
MT-DNN	81.9	88.3	91.1	95.8
MT-DNN-SMOTE	82.3	88.6	91.3	96.1
Table 5: Domain adaptation on SNLI and SciTail.
on SST) showing that SMART improves the generalization of MTL.
5.4 Domain Adaptation
In this section, we evaluate our model on the domain adaptation setting. Following Liu et al. (2019b), we start with the default training/dev/test set of SNLI and SciTail. Then, we randomly sample  $0.1\%$ ,  $1\%$ ,  $10\%$  and  $100\%$  of its training data, which is used to train a model.
The results are reported in Table 5. We observe that both MT-DNN and MT-DNN-SMART significantly outperform the BERT baseline. Comparing with MT-DNN, MT-DNN-SMART also achieves some improvements indicating the robustness of SMART. Furthermore, MT-DNN-SMART outperforms current state-of-the-art on the SNLI/SciTail test.
5.5 Results on SNLI and SciTail
In Table 7, we compare our methods, using all in-domain training data, against several state-of-the-art models. We observe that SMART obtains the same improvement on SNLI in the BERT setting. Combining SMART with MT-DNN achieves a significant improvement, e.g., our BASE model even outperforms the  $\mathrm{BERT}_{\mathrm{LARGE}}$  model. Similar observation is found on SciTail and in the  $\mathrm{BERT}_{\mathrm{LARGE}}$  model setting. We see that incorporating SMART into MT-DNN achieves new state-of-the-art results on both SNLI and SciTail, pushing benchmarks to  $91.7\%$  on SNLI and  $95.2\%$  on SciTail.
5.6 Robustness
One important property of the machine learning model is its robustness to adversarial attack. We
<sup>9</sup>It is from: https://github.com/namisan/mt-dnn. Note that we did not use the complicated answer module, e.g., SAN (Liu et al., 2018).
Method	Dev	Test
R1	R2	R3	All	R1	R2	R3	All
MNLI + SNLI + ANLI + FEVER
BERTLARGE (Nie et al., 2019)	57.4	48.3	43.5	49.3	-	-	-	44.2
XLNetLARGE (Nie et al., 2019)	67.6	50.7	48.3	55.1	-	-	-	52.0
RoBERTaLARGE (Nie et al., 2019)	73.8	48.9	44.4	53.7	-	-	-	49.7
SMARTRoBERTa-LARGE	74.5	50.9	47.6	57.1	72.4	49.8	50.3	57.1
ANLI
RoBERTaLARGE (Nie et al., 2019)	71.3	43.3	43.0	51.9	-	-	-	-
SMARTRoBERTa-LARGE	74.2	49.5	49.2	57.1	72.4	50.3	49.5	56.9
Table 6: Experiment Result for Each Round of ANLI.
Model	Dev	Test
SNLI Dataset (Accuracy%)
BERTBASE	91.0	90.8
BERTBASE+SRL(Zhang et al., 2018)	-	90.3
MT-DNNBASE	91.4	91.1
SMARTBERT-BASE	91.4	91.1
MT-DNN-SMARTBASEv0	91.7	91.4
MT-DNN-SMARTBASE	91.7	91.5
BERTLARGE+SRL(Zhang et al., 2018)	-	91.3
BERTLARGE	91.7	91.0
MT-DNNLARGE	92.2	91.6
MT-DNN-SMARTLARGEv0	92.6	91.7
SciTail Dataset (Accuracy%)
GPT (Radford et al., 2018)	-	88.3
BERTBASE	94.3	92.0
MT-DNNBASE	95.8	94.1
SMARTBERT-BASE	94.8	93.2
MT-DNN-SMARTBASEv0	96.0	94.0
MT-DNN-SMARTBASE	96.1	94.2
BERTLARGE	95.7	94.4
MT-DNNLARGE	96.3	95.0
SMARTBERT-LARGE	96.2	94.7
MT-DNN-SMARTLARGEv0	96.6	95.2
Table 7: Results on the SNLI and SciTail dataset.
test our model on an adversarial natural language inference (ANLI) dataset (Nie et al., 2019).
We evaluate the performance of SMART on each subset (i.e., R1,R2,R3) of ANLI dev and test set. The results are presented in Table 6. Table 6 shows the results of training on combined NLI data (ANLI (Nie et al., 2019) + MNLI (Williams et al., 2018) + SNLI (Bowman et al., 2015) + FEVER (Thorne et al., 2018)) and training on only ANLI data. In the combined data setting, we obverse that  $\mathrm{SMART}_{\mathrm{ROBERTa-LARGE}}$  obtains the best
performance compared with all the strong baselines, pushing benchmarks to  $57.1\%$ . In case of the RoBERTaLarge baseline,  $\mathrm{SMART}_{\mathrm{RoBERTa-LARGE}}$  outperforms  $3.4\%$  absolute improvement on dev and  $7.4\%$  absolute improvement on test, indicating the robustness of SMART. We obverse that in the ANLI-only setting,  $\mathrm{SMART}_{\mathrm{RoBERTa-LARGE}}$  outperforms the strong RoBERTaLarge baseline with a large margin,  $+5.2\%$  (57.1% vs. 51.9%)
We propose a robust and efficient computation framework, SMART, for fine-tuning large scale pre-trained natural language models in a principled manner. The framework effectively alleviates the overfitting and aggressive updating issues in the fine-tuning stage. SMART includes two important ingredients: 1) smooth-inducing adversarial regularization; 2) Bregman proximal point optimization. Our empirical results suggest that SMART improves the performance on many NLP benchmarks (e.g., GLUE, SNLI, SciTail, ANLI) with the state-of-the-art pre-trained models (e.g., BERT, MT-DNN, RoBERTa). We also demonstrate that the proposed framework is applicable to domain adaptation and results in a significant performance improvement. Our proposed fine-tuning framework can be generalized to solve other transfer learning problems. We will explore this direction as future work.
We thank Jade Huang, Niao He, Chris Meek, Liyuan Liu, Yangfeng Ji, Pengchuan Zhang, Oleksandr Polozov, Chenguang Zhu and Keivn Duh for valuable discussions and comments, and Microsoft Research Technology Engineering team for setting up GPU machines. We also thank the anonymous reviewers for valuable discussions.
Roy Bar-Haim, Ido Dagan, Bill Dolan, Lisa Ferro, and Danilo Giampiccolo. 2006. The second PASCAL recognising textual entailment challenge. In Proceedings of the Second PASCAL Challenges Workshop on Recognising Textual Entailment.
Luisa Bentivogli, Ido Dagan, Hoa Trang Dang, Danilo Giampiccolo, and Bernardo Magnini. 2009. The fifth pascal recognizing textual entailment challenge. In In Proc Text Analysis Conference (TAC09).
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics.
Rich Caruana. 1997. Multitask learning. Machine learning, 28(1):41-75.
Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. 2017. Semeval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 1-14.
Andrew R Conn, Nicholas IM Gould, and Ph L Toint. 2000. Trust region methods, volume 1. Siam.
Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The pascal recognising textual entailment challenge. In Proceedings of the First International Conference on Machine Learning Challenges: Evaluating Predictive Uncertainty Visual Object Classification, and Recognizing Textual Entailment, MLCW'05, pages 177-190, Berlin, Heidelberg. Springer-Verlag.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186.
William B Dolan and Chris Brockett. 2005. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005).
Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019. Unified language model pre-training for natural language understanding and generation. pages 13042-13054.
Jonathan Eckstein. 1993. Nonlinear proximal point algorithms using bregman functions, with applications to convex programming. Mathematics of Operations Research, 18(1):202-226.
Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. 2007. The third PASCAL recognizing textual entailment challenge. In Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing, pages 1-9, Prague. Association for Computational Linguistics.
Osman Güler. 1991. On the convergence of the proximal point algorithm for convex minimization. SIAM Journal on Control and Optimization, 29(2):403-419.
Osman Güler. 1992. New proximal point algorithms for convex minimization. SIAM Journal on Optimization, 2(4):649-664.
Frank R Hampel. 1974. The influence curve and its role in robust estimation. Journal of the american statistical association, 69(346):383-393.
Pengcheng He, Xiaodong Liu, Weizhu Chen, and Jianfeng Gao. 2019. A hybrid neural network model for commonsense reasoning. In Proceedings of the First Workshop on Commonsense Inference in Natural Language Processing, pages 13-21.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning, pages 2790-2799.
Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 328-339.
Peter J Huber. 2011. *Robust statistics*. Springer.
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S Weld, Luke Zettlemoyer, and Omer Levy. 2020. Spanbert: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics, 8:64-77.
Tushar Khot, Ashish Sabharwal, and Peter Clark. 2018. SciTail: A textual entailment dataset from science question answering. In AAAI.
Diederik Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
Vid Kocijan, Ana-Maria Cretu, Oana-Maria Camburu, Yordan Yordanov, and Thomas Lukasiewicz. 2019. A surprisingly robust trick for the winograd schema challenge. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4837-4842.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942.
Hector Levesque, Ernest Davis, and Leora Morgenstern. 2012. The winograd schema challenge. In Thirteenth International Conference on the Principles of Knowledge Representation and Reasoning.
Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. 2020a. On the variance of the adaptive learning rate and beyond. In Proceedings of the Eighth International Conference on Learning Representations (ICLR 2020).
Xiaodong Liu, Kevin Duh, and Jianfeng Gao. 2018. Stochastic answer networks for natural language inference. arXiv preprint arXiv:1804.07888.
Xiaodong Liu, Jianfeng Gao, Xiaodong He, Li Deng, Kevin Duh, and Ye-Yi Wang. 2015. Representation learning using multi-task deep neural networks for semantic classification and information retrieval. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 912-921.
Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. 2019a. Improving multi-task deep neural networks via knowledge distillation for natural language understanding. arXiv preprint arXiv:1904.09482.
Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. 2019b. Multi-task deep neural networks for natural language understanding. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4487-4496, Florence, Italy. Association for Computational Linguistics.
Xiaodong Liu, Yu Wang, Jianshu Ji, Hao Cheng, Xueyun Zhu, Emmanuel Awa, Pengcheng He, Weizhu Chen, Hoifung Poon, Guihong Cao, and Jianfeng Gao. 2020b. The microsoft toolkit of multi-task deep neural networks for natural language understanding. arXiv preprint arXiv:2002.07972.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019c. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. 2018. Virtual adversarial training: a regularization method for supervised and semi-supervised learning. IEEE transactions on pattern analysis and machine intelligence, 41(8):1979-1993.
Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. 2018. Foundations of machine learning. MIT press.
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2019. Adversarial nli: A new benchmark for natural language understanding. arXiv preprint arXiv:1910.14599.