SlowGuess/ACL_2020_OCR huggingface dataset:
Pengcheng He, Weizhu Chen
Microsoft Dynamics 365 AI
{penhe, wzchen}@microsoft.com
Xiaodong Liu, Jianfeng Gao
{xiaodl,jfgao}@microsoft.com
Transfer learning has fundamentally changed the landscape of natural language processing (NLP). Many state-of-the-art models are first pre-trained on a large text corpus and then fine-tuned on downstream tasks. However, due to limited data resources from downstream tasks and the extremely high complexity of pre-trained models, aggressive fine-tuning often causes the fine-tuned model to overfit the training data of downstream tasks and fail to generalize to unseen data. To address such an issue in a principled manner, we propose a new learning framework for robust and efficient fine-tuning for pre-trained models to attain better generalization performance. The proposed framework contains two important ingredients: 1. Smoothness-inducing regularization, which effectively manages the complexity of the model; 2. Bregman proximal point optimization, which is an instance of trust-region methods and can prevent aggressive updating. Our experiments show that the proposed framework achieves new state-of-the-art performance on a number of NLP tasks including GLUE, SNLI, SciTail and ANLI. Moreover, it also outperforms the state-of-the-art T5 model, which is the largest pre-trained model containing 11 billion parameters, on GLUE.
The success of natural language processing (NLP) techniques relies on huge amounts of labeled data in many applications. However, large amounts of labeled data are usually prohibitive or expensive to obtain. To address this issue, researchers have resorted to transfer learning.
Transfer learning considers the scenario, where we have limited labeled data from the target domain for a certain task, but we have relevant tasks
with a large amount of data from different domains (also known as out-of-domain data). The goal is to transfer the knowledge from the high-resource domains to the low-resource target domain. Here we are particularly interested in the popular two-stage transfer learning framework (Pan and Yang, 2009). The first stage is pre-training, where a high-capacity model is trained for the out-of-domain high-resource relevant tasks. The second stage is fine-tuning, where the high-capacity model is adapted to the low-resource task in the target domain.
For many applications in NLP, most popular transfer learning methods choose to pre-train a large language model, e.g., ELMo (Peters et al., 2018), GPT (Radford et al., 2019) and BERT (Devlin et al., 2019). Such a language model can capture general semantic and syntactic information that can be further used in downstream NLP tasks. The language model is particularly attractive, because it can be trained in a completely unsupervised manner with huge amount of unlabeled data, which are extremely cheap to fetch from internet nowadays. The resulting extremely large multidomain text corpus allows us to train huge language models. To the best of our knowledge, by far the largest language model, T5, has an enormous size of about 11 billion parameters (Raffel et al., 2019).
For the second fine-tuning stage, researchers adapt the pre-trained language model to the target task/domain. They usually replace the top layer of the language model by a task/domain-specific sub-network, and then continue to train the new model with the limited data of the target task/domain. Such a fine-tuning approach accounts for the low-resource issue in the target task/domain, and has achieved state-of-the-art performance in many popular NLP benchmarks (Devlin et al., 2019; Liu et al., 2019c; Yang et al.,
* Work was done during an internship at Microsoft Dynamics 365 AI.
$^{1}$ https://github.com/namisan(mt-dnn
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2177-2190 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
2019; Lan et al., 2019; Dong et al., 2019; Raffel et al., 2019).
Due to the limited data from the target task/domain and the extremely high complexity of the pre-trained model, aggressive fine-tuning often makes the adapted model overfit the training data of the target task/domain and therefore does not generalize well to unseen data. To mitigate this issue, the fine-tuning methods often rely on hyper-parameter tuning heuristics. For example, Howard and Ruder (2018) use a heuristic learning rate schedule and gradually unfreeze the layers of the language model to improve the fine-tune performance; Peters et al. (2019) give a different suggestion that they only adapt certain layers and freeze the others; (Houlsby et al., 2019; Stickland and Murray, 2019) propose to add additional layers to the pre-trained model and fine-tune both of them or only the additional layers. However, these methods require significant tuning efforts.
To fully harness the power of fine-tuning in a more principled manner, we propose a new learning framework for robust and efficient fine-tuning on the pre-trained language models through regularized optimization techniques. Specifically, our framework consists of two important ingredients for preventing overfitting:
(I) To effectively control the extremely high complexity of the model, we propose a Smoothness-inducing Adversarial Regularization technique. Our proposed regularization is motivated by local shift sensitivity in existing literature on robust statistics. Such regularization encourages the output of the model not to change much, when injecting a small perturbation to the input. Therefore, it enforces the smoothness of the model, and effectively controls its capacity (Mohri et al., 2018).
(II) To prevent aggressive updating, we propose a class of Bregman Proximal Point Optimization methods. Our proposed optimization methods introduce a trust-region-type regularization (Conn et al., 2000) at each iteration, and then update the model only within a small neighborhood of the previous iterate. Therefore, they can effectively prevent aggressive updating and stabilize the finetuning process.
We compare our proposed method with several state-of-the-art competitors proposed in (Zhu et al., 2020; Liu et al., 2019b,c; Lan et al., 2019; Raffel et al., 2019) and show that our proposed method significantly improves the training sta
bility and generalization, and achieves comparable or better performance on multiple NLP tasks. We highlight that our single model with 356M parameters (without any ensemble) can achieve three state-of-the-art results on GLUE, even compared with all existing ensemble models and the T5 model (Raffel et al., 2019), which contains 11 billion parameters. Furthermore, we also demonstrate that the proposed framework complements with SOTA fine-tuning methods (Liu et al., 2019b) and outperforms the T5 model.
We summarize our contribution as follows: 1. We introduce the smoothness-inducing adversarial regularization and proximal point optimization into large scale language model fine-tuning; 2. We achieve state-of-the-art results on several popular NLP benchmarks (e.g., GLUE, SNLI, SciTail, and ANLI).
Notation: We use  $f(x; \theta)$  to denote a mapping  $f$  associated with the parameter  $\theta$  from input sentences  $x$  to an output space, where the output is a multi-dimensional probability simplex for classification tasks and a scalar for regression tasks.  $\Pi_{\mathcal{A}}$  denotes the projection operator to the set  $\mathcal{A}$ .  $\mathcal{D}_{KL}(P||Q) = \sum_k p_k \log (p_k / q_k)$  denotes the KL-divergence of two discrete distributions  $P$  and  $Q$  with the associated parameters of  $p_k$  and  $q_k$ , respectively.
The transformer models were originally proposed in Vaswani et al. (2017) for neural machine translation. Their superior performance motivated Devlin et al. (2019) to propose a bidirectional transformer-based language model named BERT. Specifically, Devlin et al. (2019) pre-trained the BERT model using a large corpus without any human annotation through unsupervised learning tasks. BERT motivated many follow-up works to further improve the pre-training by introducing new unsupervised learning tasks (Yang et al., 2019; Dong et al., 2019; Joshi et al., 2020), enlarging model size (Lan et al., 2019; Raffel et al., 2019), enlarging training corpora (Liu et al., 2019c; Yang et al., 2019; Raffel et al., 2019) and multi-tasking (Liu et al., 2019a,b).
The pre-trained language model is then adapted to downstream tasks and further fine-tuned. Specifically, the top layer of the language model can be replaced by a task-specific layer and then continue to train on downstream tasks. To prevent overfitting, existing heuristics include choosing a
small learning rate or a triangular learning rate schedule, and a small number of iterations, and other fine-tuning tricks mentioned in (Howard and Ruder, 2018; Peters et al., 2019; Houlsby et al., 2019; Stickland and Murray, 2019).
Our proposed regularization technique is related to several existing works (Miyato et al., 2018; Zhang et al., 2019; Shu et al., 2018). These works consider similar regularization techniques, but target at other applications with different motivations, e.g., semi-supervised learning, unsupervised domain adaptation and harnessing adversarial examples in image classification.
Our proposed optimization technique covers a large class of Bregman proximal point methods in existing literature on optimization, including vanilla proximal point method proposed in Rockafellar (1976), generalized proximal point method (Teboulle, 1997; Eckstein, 1993), accelerated proximal point method, and other variants (Güler, 1991, 1992; Parikh et al., 2014).
There is a related fine-tuning method - FreeLB Zhu et al. (2020), which adapted a robust adversarial training method. However, our framework focuses on the local smoothness, leading to a significant performance improvement. More discussion and comparison are provided in Section 4.
We describe the proposed learning framework - SMART for robust and efficient fine-tuning of pre-trained language models. Our framework consists of two important ingredients: SMoothness-inducing Adversarial Regularization and BRegman pRoximal pointOpTimization $^2$ .
3.1 Smoothness-Inducing Adversarial Regularization
We propose to impose an explicit regularization to effectively control the model complexity at the fine-tuning stage. Specifically, given the model  $f(\cdot ;\theta)$  and  $n$  data points of the target task denoted by  $\{(x_i,y_i)\}_{i = 1}^n$ , where  $x_{i}$ 's denote the embedding of the input sentences obtained from the first embedding layer of the language model and  $y_{i}$ 's are the associated labels, our method essentially solves the following optimization for fine-tuning:
$$
\min  _ {\theta} \mathcal {F} (\theta) = \mathcal {L} (\theta) + \lambda_ {\mathrm {s}} \mathcal {R} _ {\mathrm {s}} (\theta), \tag {1}
$$
where  $\mathcal{L}(\theta)$  is the loss function defined as
$$
\mathcal {L} (\theta) = \frac {1}{n} \sum_ {i = 1} ^ {n} \ell (f (x _ {i}; \theta), y _ {i}),
$$
and  $\ell (\cdot ,\cdot)$  is the loss function depending on the target task,  $\lambda_{\mathrm{s}} > 0$  is a tuning parameter, and  $\mathcal{R}_{\mathrm{s}}(\theta)$  is the smoothness-inducing adversarial regularizer. Here we define  $\mathcal{R}_{\mathrm{s}}(\theta)$  as
$$
\mathcal {R} _ {\mathrm {s}} (\theta) = \frac {1}{n} \sum_ {i = 1} ^ {n} \max  _ {\| \widetilde {x} _ {i} - x _ {i} \| _ {p} \leq \epsilon} \ell_ {\mathrm {s}} (f (\widetilde {x} _ {i}; \theta), f (x _ {i}; \theta)),
$$
where  $\epsilon > 0$  is a tuning parameter. Note that for classification tasks,  $f(\cdot; \theta)$  outputs a probability simplex and  $\ell_{\mathrm{s}}$  is chosen as the symmetrized KL-divergence, i.e.,
$$
\ell_ {\mathrm {s}} (P, Q) = \mathcal {D} _ {\mathrm {K L}} (P | | Q) + \mathcal {D} _ {\mathrm {K L}} (Q | | P);
$$
For regression tasks,  $f(\cdot ;\theta)$  outputs a scalar and  $\ell_{s}$  is chosen as the squared loss, i.e.,  $\ell_s(p,q) = (p - q)^2$ . Note that the computation of  $\mathcal{R}_{\mathrm{s}}(\theta)$  involves a maximization problem and can be solved efficiently by projected gradient ascent.
We remark that the proposed smoothness-inducing adversarial regularizer was first used in Miyato et al. (2018) for semi-supervised learning with  $p = 2$ , and then in Shu et al. (2018) for unsupervised domain adaptation with  $p = 2$ , and more recently in Zhang et al. (2019) for harnessing the adversarial examples in image classification with  $p = \infty$ . To the best of our knowledge, we are the first applying such a regularizer to fine-tuning of pre-trained language models.
The smoothness-inducing adversarial regularizer is essentially measuring the local Lipschitz continuity of  $f$  under the metric  $\ell_s$ . More precisely speaking, the output of  $f$  does not change much if we inject a small perturbation  $(\ell_p$  norm bounded by  $\epsilon$ ) to  $x_i$ . Therefore, by minimizing the objective in (1), we can encourage  $f$  to be smooth within the neighborhoods of all  $x_i$ 's. Such a smoothness-inducing property is particularly helpful to prevent overfitting and improve generalization on a low resource target domain for a certain task. An illustration is provided in Figure 1.
Note that the idea of measuring the local Lipschitz continuity is similar to the local shift sensitivity criterion in existing literature on robust statistics, which dates back to 1960's (Hampel, 1974; Huber, 2011). This criterion has been used to characterize the dependence of an estimator on the value of one of the sample points.
3.2 Bregman Proximal Point Optimization
We propose to develop a class of Bregman proximal point optimization methods to solve (1). Such optimization methods impose a strong penalty at
The complete name of our proposed method is  $\mathbf{SMAR}^3\mathbf{T}^2$ , but we use SMART for notational simplicity.
(a)
Figure 1: Decision boundaries learned without (a) and with (b) smoothness-inducing adversarial regularization, respectively. The red dotted line in (b) represents the decision boundary in (a). As can be seen, the output  $f$  in (b) does not change much within the neighborhood of training data points.
each iteration to prevent the model from aggressive update. Specifically, we use a pre-trained model as the initialization denoted by  $f(\cdot ;\theta_0)$ . At the  $(t + 1)$ -th iteration, the vanilla Bregman proximal point (VBPP) method takes
$$
\theta_ {t + 1} = \operatorname {a r g m i n} _ {\theta} \mathcal {F} (\theta) + \mu \mathcal {D} _ {\mathrm {B r e g}} (\theta , \theta_ {t}), \quad (2)
$$
where  $\mu > 0$  is a tuning parameter, and  $\mathcal{D}_{\mathrm{Breg}}(\cdot, \cdot)$  is the Bregman divergence defined as
$$
\mathcal {D} _ {\mathrm {B r e g}} (\theta , \theta_ {t}) = \frac {1}{n} \sum_ {i = 1} ^ {n} \ell_ {\mathrm {s}} (f (x _ {i}; \theta), f (x _ {i}; \theta_ {t})),
$$
where  $\ell_s$  is defined in Section 3.1. As can be seen, when  $\mu$  is large, the Bregman divergence at each iteration of the VBPP method essentially serves as a strong regularizer and prevents  $\theta_{t + 1}$  from deviating too much from the previous iterate  $\theta_t$ . This is also known as the trust-region type iteration in existing optimization literature (Conn et al., 2000). Consequently, the Bregman proximal point method can effectively retain the knowledge of the out-of-domain data in the pre-trained model  $f(\cdot ;\theta_0)$ . Since each subproblem (2) of VBPP does not admit a closed-form solution, we need to solve it using SGD-type algorithms such as ADAM. Note that we do not need to solve each subproblem until convergence. A small number of iterations are sufficient to output a reliable initial solution for solving the next subproblem.
Moreover, the Bregman proximal point method is capable of adapting to the information geometry (See more details in Raskutti and Mukherjee (2015)) of machine learning models and achieving better computational performance than the standard proximal point method (i.e.,  $\mathcal{D}_{\mathrm{Breg}}(\theta ,\theta_t) = \|\theta -\theta_t\|_2^2)$  in many applications.
Acceleration by Momentum. Similar to other optimization methods in existing literature, we can accelerate the Bregman proximal point method
Algorithm 1 SMART: We use the smoothness-inducing adversarial regularizer with  $p = \infty$  and the momentum Bregman proximal point method.
Notation: For simplicity, we denote  $g_{i}(\widetilde{x}_{i},\bar{\theta}_{s}) = \frac{1}{|\mathcal{B}|}\sum_{x_{i}\in \mathcal{B}}\nabla_{\widetilde{x}}\ell_{\mathrm{s}}(f(x_{i};\bar{\theta}_{s}),f(\widetilde{x}_{i};\bar{\theta}_{s}))$  and AdamUpdate denotes the ADAM update rule for optimizing (3) using the mini-batch  $\mathcal{B}$ ;  $\Pi_{\mathcal{A}}$  denotes the projection to  $\mathcal{A}$ .
Input:  $T$ : the total number of iterations,  $\mathcal{X}$ : the dataset,  $\theta_0$ : the parameter of the pre-trained model,  $S$ : the total number of iteration for solving (2),  $\sigma^2$ : the variance of the random initialization for  $\widetilde{x}_i$ 's,  $T_{\widetilde{x}}$ : the number of iterations for updating  $\widetilde{x}_i$ 's,  $\eta$ : the learning rate for updating  $\widetilde{x}_i$ 's,  $\beta$ : momentum parameter.
1:  $\theta_{1}\gets \theta_{0}$
2: for  $t = 1,..,T$  do
3:  $\bar{\theta}_{1}\gets \theta_{t - 1}$
4: for  $s = 1,..,S$  do
5: Sample a mini-batch  $\mathcal{B}$  from  $\mathcal{X}$
6: For all  $x_{i}\in \mathcal{B}$  , initialize  $\widetilde{x}_i\gets x_i + \nu_i$  with  $\nu_{i}\sim \mathcal{N}(0,\sigma^{2}I)$
7: for  $m = 1,..,T_{\bar{x}}$  do
8:  $\widetilde{g}_i\gets \frac{g_i(\widetilde{x}_i,\theta_s)}{\left\|g_i(\widetilde{x}_i,\bar{\theta}_s)\right\|_\infty}$
9:  $\widetilde{x}_i\gets \Pi_{\| \widetilde{x}_i - x\|_\infty \leq \epsilon}(\widetilde{x}_i + \eta \widetilde{g}_i)$
11:  $\bar{\theta}_{s + 1}\gets \mathrm{AdamUpdate}_{\mathcal{B}}(\bar{\theta}_s)$
13:  $\theta_t\gets \bar{\theta}_S$
14:  $\bar{\theta}_{t + 1}\gets (1 - \beta)\bar{\theta}_S + \beta \bar{\theta}_t$
Output:  $\theta_T$
by introducing an additional momentum to the update. Specifically, at the  $(t + 1)$ -th iteration, the momentum Bregman proximal point (MBPP) method takes
$$
\theta_ {t + 1} = \operatorname {a r g m i n} _ {\theta} \mathcal {F} (\theta) + \mu \mathcal {D} _ {\mathrm {B r e g}} (\theta , \widetilde {\theta} _ {t}), \tag {3}
$$
where  $\widetilde{\theta}_t = (1 - \beta)\theta_t + \beta \widetilde{\theta}_{t-1}$  is the exponential moving average and  $\beta \in (0,1)$  is the momentum parameter. The MBPP method is also called the "Mean Teacher" method in existing literature (Tarvainen and Valpola, 2017) and has been shown to achieve state-of-the-art performance in popular semi-supervised learning benchmarks. For convenience, we summarize the MBPP method in Algorithm 1.
4 Experiment - Main Results
We demonstrate the effectiveness of SMART for fine-tuning large language models using GLUE (Wang et al., 2018) by comparing with existing state-of-the-art methods. Dataset details can be found in Appendix A.
Our implementation of SMART is based on BERT $^3$  (Wolf et al., 2019), RoBERTa $^4$  (Liu et al., 2019c), MT-DNN $^5$  (Liu et al., 2020b) and HNN $^6$ . We used ADAM (Kingma and Ba, 2014) and RADAM (Liu et al., 2020a) as our optimizers with a learning rate in the range  $\in \{1 \times 10^{-5}, 2 \times 10^{-5}, 3 \times 10^{-5}, 5 \times 10^{-5}\}$  and a batch size  $\in \{16, 32, 64\}$ . The maximum number of epochs was set to 6. A linear learning rate decay schedule with warm-up of 0.1 was used, unless stated otherwise. We also set the dropout rate of all the task specific layers as 0.1, except 0.3 for MNLI and 0.05 for CoLA. To avoid gradient exploding, we clipped the gradient norm within 1. All the texts were tokenized using wordpieces and were chopped to spans no longer than 512 tokens. For SMART, we set the perturbation size  $\epsilon = 10^{-5}$  and  $\sigma = 10^{-5}$ . We set  $\mu = 1$  and  $\lambda_s \in \{1, 3, 5\}$ . The learning rate  $\eta$  in Algorithm 1 is set to  $10^{-3}$ . We set  $\beta = 0.99$  for the first  $10\%$  of the updates ( $t \leq 0.1T$ ) and  $\beta = 0.999$  for the rest of the updates ( $t > 0.1T$ ) following (Tarvainen and Valpola, 2017). Lastly, we simply set  $S = 1$ ,  $T_{\widetilde{x}} = 1$  in Algorithm 1.
4.2 GLUE Main Results
We compare SMART with a range of strong baselines including large pre-trained models and approaches with adversarial training, and a list of state-of-the-art models that have been submitted to the GLUE leaderboard. SMART is a generic framework, we evaluate our framework on two pre-trained models, the BERT<sub>BASE</sub> model (Devlin et al., 2019) and the RoBERTa<sub>LARGE</sub> model (Liu et al., 2019c), which are available publicly. Most of our analyses are done with the BERT<sub>BASE</sub> to make our results comparable to other work, since BERT<sub>BASE</sub> has been widely used as a baseline. To make our result comparable to other state-of-the-art models, we also evaluate the framework on the
RoBERTa<sub>LOUD</sub> model.
- BERT (Devlin et al., 2019): This is the  $\mathrm{BERT}_{\mathrm{BASE}}$  model released by the authors. In Devlin et al. (2019), authors only reported the development results on a few tasks, thus we reproduced the baseline results, which are denoted by  $\mathrm{BERT}_{\mathrm{ReImp}}$ .
- RoBERTa (Liu et al., 2019c): This is the RoBERTaLarge released by authors, and we present the reported results on the GLUE dev.
- PGD, FreeAT, FreeLB (Zhu et al., 2020): They are three adversarial training approaches built on top of the RoBERTaLARGE.
- SMART: our proposed method as described in section 3. We use both the  $\mathrm{BERT}_{\mathrm{BASE}}$  model (SMART_BERT) and the RoBERTa LARGE model (SMART_RoBERTa) as the pretrained model to evaluate the effectiveness of SMART.
The main results are reported in Table 1. This table can be clustered into two groups based on different pretrained models: the BERT<sub>BASE</sub> model (the first group) and the RoBERTa<sub>LARGE</sub> model (the second group). The detailed discussions are as follows.
For a fair comparison, we reproduced the BERT baseline  $\mathrm{(BERT_{ReImp})}$ , since several results on the GLUE development set were missed. Our implemented BERT baseline is even stronger than the originally reported results in Devlin et al. (2019). For instance, the implemented model obtains  $84.5\%$  vs.  $84.4\%$  on MNLI in-domain development in terms of accuracy. On SST-2,  $\mathrm{BERT_{ReImp}}$  outperforms BERT by  $0.2\%$  (92.9% vs. 92.7%) accuracy. All these results demonstrate the fairness of our baselines.
Comparing with two strong baselines BERT and RoBERTa  $^{7}$ , SMART, including  $\mathrm{SMART}_{\mathrm{BERT}}$  and  $\mathrm{SMART}_{\mathrm{RoBERTa}}$ , consistently outperforms them across all 8 GLUE tasks by a big margin. Comparing with BERT,  $\mathrm{SMART}_{\mathrm{BERT}}$  obtained  $85.6\%$  (vs.  $84.5\%$ ) and  $86.0\%$  (vs.  $84.4\%$ ) in terms of accuracy, which is  $1.1\%$  and  $1.6\%$  absolute improvement, on the MNLI in-domain and out-domain settings. Even comparing with the state-of-the-art model RoBERTa,  $\mathrm{SMART}_{\mathrm{RoBERTa}}$  improves  $0.8\%$  (91.1% vs.  $90.2\%$ ) on MNLI in-domain development set. Interestingly, on the
In our experiments, we use BERT referring the  $\mathrm{BERT}_{\mathrm{BASE}}$  model, which has 110 million parameters, and RoBERTa referring the RoBERTaLarge model, which has 356 million parameters, unless stated otherwise.
<sup>4</sup>https://github.com/pytorch/fairseq
$^{5}$ https://github.com/namisan/mt-dnn
$^{6}$ https://github.com/namisan(mt-dnn/tree/master/hnn
Model	MNLI-m/mm Acc	QQP Acc/F1	RTE Acc	QNLI Acc	MRPC Acc/F1	CoLA Mcc	SST Acc	STS-B P/S Corr
BERTBASE
BERT (Devlin et al., 2019)	84.4/-	-	-	88.4	-/86.7	-	92.7	-
BERTReImp	84.5/84.4	90.9/88.3	63.5	91.1	84.1/89.0	54.7	92.9	89.2/88.8
SMARTBERT	85.6/86.0	91.5/88.5	71.2	91.7	87.7/91.3	59.1	93.0	90.0/89.4