SlowGuess/NAACL_2021_OCR huggingface dataset:
Constantinos Karouzos<sup>1</sup>, Georgios Paraskevopoulos<sup>1,4</sup>, Alexandros Potamianos<sup>1,2,3</sup>
<sup>1</sup> School of ECE, National Technical University of Athens, Athens, Greece
$^{2}$  Signal Analysis and Interpretation Laboratory (SAIL), USC, Los Angeles, CA, USA
$^{3}$  Behavioral Signal Technologies, Los Angeles, CA, USA
<sup>4</sup> Institute for Language and Speech Processing, Athena Research Center, Athens, Greece  
ckarouzos@gmail.com, geopar@central.ntua.gr, potam@central.ntua.gr
In this work we explore Unsupervised Domain Adaptation (UDA) of pretrained language models for downstream tasks. We introduce UDALM, a fine-tuning procedure, using a mixed classification and Masked Language Model loss, that can adapt to the target domain distribution in a robust and sample efficient manner. Our experiments show that performance of models trained with the mixed loss scales with the amount of available target data and the mixed loss can be effectively used as a stopping criterion during UDA training. Furthermore, we discuss the relationship between A-distance and the target error and explore some limitations of the Domain Adversarial Training approach. Our method is evaluated on twelve domain pairs of the Amazon Reviews Sentiment dataset, yielding  $91.74\%$  accuracy, which is an  $1.11\%$  absolute improvement over the state-of-the-art.
Deep architectures have achieved state-of-the-art results in a variety of machine learning tasks. However, real world deployments of machine learning systems often operate under domain shift, which leads to performance degradation. This introduces the need for adaptation techniques, where a model is trained with data from a specific domain, and then can be optimized for use in new settings. Efficient techniques for model re-usability can lead to faster and cheaper development of machine learning applications and facilitate their wider adoption. Especially techniques for Unsupervised Domain Adaptation (UDA) can have high real world impact, because they do not rely on expensive and time-consuming annotation processes to collect labeled data for domain-specific supervised training, further streamlining the process.
UDA approaches in the literature can be grouped in three major categories, namely pseudo-labeling techniques (e.g. Yarowsky, 1995; Zhou and Li,
2005), domain adversarial training (e.g. Ganin et al., 2016) and pivot-based approaches (e.g. Blitzer et al., 2006; Pan et al., 2010). Pseudolabeling approaches use a model trained on the source labeled data to produce pseudo-labels for unlabeled target data and then train a model for the target domain in a supervised manner. Domain adversarial training aims to learn a domain-independent mapping for input samples by adding an adversarial cost during model training, that minimizes the distance between the source and target domain distributions. Pivot-based approaches aim to select domain-invariant features (pivots) and use them as a basis for cross-domain mapping. This work does not fall under any of these categories, rather we aim to optimize the fine-tuning procedure of pretrained language models (LMs) for learning under domain-shift.
Transfer learning from language models pretrained in massive corpora (Howard and Ruder, 2018; Devlin et al., 2019; Yang et al., 2019; Liu et al., 2019; Brown et al., 2020) has yielded significant improvements across a wide variety of NLP tasks, even when small amounts of data are used for fine-tuning. Fine-tuning a pretrained model is a straightforward framework for adaptation to target tasks and new domains, when labeled data are available. However, optimizing the fine-tuning process in UDA scenarios, where only labeled out-of-domain and unlabeled in-domain data are available is challenging.
In this work, we propose UDALM, a fine-tuning method for BERT (Devlin et al., 2019) in order to address the UDA problem. Our method is based on simultaneously learning the task from labeled data in the source distribution, while adapting to the language in the target distribution using multi-task learning. The key idea of our method is that by simultaneously minimizing a task-specific loss on the source data and a language modeling loss on the target data during fine-tuning, the model
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2579-2590 June 6-11, 2021. ©2021 Association for Computational Linguistics
will be able to adapt to the language of the target domain, while learning the supervised task from the available labeled data.
Our key contributions are: (a) We introduce UDALM, a novel, simple and robust unsupervised domain adaptation procedure for downstream BERT models based on multitask learning, (b) we achieve state-of-the-art results for the Amazon reviews benchmark dataset, surpassing more complicated approaches and (c) we explore how A-distance and the target error are related and conclude with some remarks on domain adversarial training, based on theoretical concepts and our empirical observations. Our code and models are publicly available<sup>1</sup>.
Traditionally, UDA has been performed using pseudo-labeling approaches. Pseudo-labeling techniques are semi-supervised algorithms that either use the same model (self-training) (Yarowsky, 1995; McClosky et al., 2006; Abney, 2007) or multiple ensembles of models (tri-training) (Zhou and Li, 2005; Søgaard, 2010) in order to produce pseudo-labels for the target unlabeled data. Saito et al. (2017) proposed an asymmetric tri-training approach. Ruder and Plank (2018) introduced a multi-task tri-training method. Rotman and Reichart (2019) and Lim et al. (2020) study pseudolabeling with contextualized word representations. Ye et al. (2020) combine self-training with XLMR (Conneau et al., 2020) to reduce the produced label noise and propose CFd, class aware feature self-distillation.
Another line of UDA research includes pivot-based methods, focusing on extracting cross-domain features. Structural Correspondence Learning (SCL) (Blitzer et al., 2006) and Spectral Feature Alignment (Pan et al., 2010) aim to find domain-invariant features (pivots) to learn a mapping between two domain distributions. Ziser and Reichart (2017, 2018, 2019) combine SCL with neural network architectures and language modeling. Miller (2019) propose to jointly learn the task and pivots. Li et al. (2018b) learn pivots with hierarchical attention networks. Pivot-based methods have also been used in conjunction with BERT (Ben-David et al., 2020).
Domain adversarial training is a dominant approach for UDA (Ramponi and Plank, 2020), in
spired by the theory for learning from different domains introduced in Ben-David et al. (2007, 2010). Ganin et al. (2016); Ganin and Lempitsky (2015) propose to learn a task while not being able to distinguish if samples come from the source or the target distribution, through use of an adversarial cost. This approach has been adopted for a diverse set of problems, e.g. sentiment analysis, tweet classification and universal dependency parsing (Li et al., 2018a; Alam et al., 2018; Sato et al., 2017). Du et al. (2020) pose domain adversarial training in the context of BERT models. Zhao et al. (2018) propose multi-source domain adversarial networks. Guo et al. (2018) propose a mixture-of-experts approach for multi-source UDA. Guo et al. (2020) explore distance measures as additional losses and use them to construct dynamic multi-armed bandit controller for the source domains. Shen et al. (2018) learn domain invariant features via Wasserstein distance. Bousmalis et al. (2016) introduce domain seperation networks with private and shared encoders.
Unsupervised pretraining on domain-specific corpora can be an effective adaptation process. For example BioBERT (Lee et al., 2020) and SciBERT (Beltagy et al., 2019) are specialized BERT variants, where pretraining is extended on large amounts of biomedical and scientific corpora respectively. Sun et al. (2019) propose continuing the pretraining of BERT with target domain data and multitask learning using relevant tasks for BERT fine-tuning. Xu et al. (2019) introduce a review reading comprehension task and a post-training approach for BERT with an auxiliary loss on a question-answering task. Continuing pretraining on multiple phases, from general to domain specific (DAPT) and task specific data (TAPT), further improves performance of pretrained language models, as shown by Gururangan et al. (2020). Han and Eisenstein (2019) propose AdaptaBERT, which includes a second phase of unsupervised pretraining, in order to use BERT in a unsupervised domain adaptation context.
Recent works have highlighted the merits of using Language Modeling as an auxiliary task during fine-tuning. Chronopoulou et al. (2019) use an auxiliary LM loss to avoid catastrophic forgetting in transfer learning and Jia et al. (2019) adopt this approach for cross-domain named-entity recognition. We draw inspiration from these approaches and utilize auxiliary Language Modeling for UDA.
$^{1}$ https://github.com/ckarouzos/slp_daptmlm
Figure 1: (a) BERT (Devlin et al., 2019) is pretrained on English Wikipedia and BookCorpus with the Masked Language Modeling (MLM) and the Next Sentence Prediction (NSP) tasks. (b) We continue the pretraining of BERT on unlabeled target domain data using the MLM task. (c) We train a task classifier with source domain labeled data, while we keep the MLM objective on unlabeled target domain data.
Let  $X$  be the input space and  $Y$  the set of labels. For binary classification tasks  $Y = \{0,1\}$ . In domain adaptation there are two different distributions over  $X \times Y$ , called the source domain  $D_{S}$  and the target domain  $D_{T}$ . In the unsupervised setting labels are provided for samples drawn from  $D_{S}$ , while samples drawn from  $D_{T}$  are unlabeled. The goal is to train a model that performs well on samples drawn from the target distribution  $D_{T}$ . This is summarized in Eq. 1.
$$
S = \left(x _ {i}, y _ {i}\right) _ {i = 1} ^ {n} \sim \left(D _ {S}\right) ^ {n} \tag {1}
$$
$$
T = (x _ {i}) _ {i = n + 1} ^ {n + m} \sim (D _ {T} ^ {X}) ^ {m}
$$
where  $D_T^X$  is the marginal distribution of  $D_T$  over  $X$ ,  $n$  is the number of samples from the source domain and  $m$  is the number of samples from the target domain.
Fig. 1 gives an overview of the proposed Unsupervised Domain Adaptation through Language Modeling (UDALM). Starting from a model that is pretrained in general corpora (Fig. 1a), we keep pretraining it on target domain data using the masked language modeling task (Fig. 1b). On the final fine-tuning step (Fig. 1c) we update the model weights using both a classification loss on the labeled source data and Masked Language Modeling loss on the unlabeled target data.
In Fig. 1a we see the BERT general pretraining phase. BERT (Devlin et al., 2019) is based on the Transformer architecture (Vaswani et al., 2017). During BERT pretraining, input tokens are randomly selected to be masked. BERT is trained using the Masked Language Modeling (MLM) objective, which consists of predicting the most probable tokens for the masked positions. Additionally it uses a Next Sentence Prediction (NSP) loss, which classifies whether the pair of input sentences are continuous or not. If a labeled dataset is available, a pretrained BERT model can be fine-tuned for the downstream task in a supervised manner with the addition of an output layer.
In Fig. 1b we initialize a model using the weights of a generally pretrained BERT and continue pretraining on an unsupervised set of in-domain data, in order to adapt to the target domain. This step does not require use of supervised data, since we use the MLM objective.
For the final fine-tuning step, shown in Fig. 1c we perform supervised fine-tuning on the source data, while we keep the MLM objective on the target data as an auxiliary task. Following standard practice, we use the [CLS] token representation for classification. The classifier consists of a single feed-forward layer.
During this procedure the model learns the task through the classification objective using the labeled source domain samples, and simultaneously
it adapts to the target domain data through the MLM objective. The model is trained on the source domain labeled data for the classification task and target domain unlabeled data for the masked language modeling task. We mask only the target domain data. During training we interleave source and target data and feed them to the BERT encoder. Features extracted from the source data are then used for classification, while target features are used for Masked Language Modeling.
The mixed loss used for the fine-tuning step, is the sum of the classification loss  $L_{CLF}$  and the auxiliary MLM loss  $L_{MLM}$ .  $L_{CLF}$  is a cross-entropy loss, calculated on labeled examples from source domain, while  $L_{MLM}$  is used to predict masked tokens for unlabeled examples from target domain. We train the model over mixed batches, that include both source and target data, used for the respective tasks. The mixed loss is presented in Eq. 2:
$$
L (\mathbf {s}, \mathbf {t}) = \lambda L _ {C L F} (\mathbf {s}) + (1 - \lambda) L _ {M L M} (\mathbf {t}) \tag {2}
$$
We process  $n$  labeled source samples  $\mathbf{s} \sim D_S$  and  $m$  unlabeled target samples  $\mathbf{t} \sim D_T$  on a batch. The weighting factor  $\lambda$  is selected as the ratio of labeled source data over the sum of labeled source and unlabeled target data, as stated in Eq. 3:
$$
\lambda = \frac {n}{n + m} \tag {3}
$$
We evaluate UDALM on the Amazon reviews multi-domain sentiment dataset (Blitzer et al., 2007), a standard benchmark dataset for domain adaptation. Reviews with one or two stars are labeled as negative, while reviews with four or five stars are labeled as positive. The dataset contains reviews on four product domains: Books (B), DVDs (D), Electronics (E) and Kitchen appliances (K), yielding 12 adaptation scenarios of source-target domain pairs. Balanced sets of 2000 labeled reviews are available for each domain. We use 20000 (randomly selected) unlabeled reviews for (B), (D) and (E). For (K) 17805 unlabeled reviews are available. For each of the 12 adaptation scenarios we use  $20\%$  of both labeled source and unlabeled target data for validation, while labeled target data are used for testing exclusively and are not seen during training or validation.
We use  $BERT_{BASE}$  (uncased) as the Language Model on which we apply domain pretraining. The  $BERT_{BASE}$  original English model is a 12-layer, 768-hidden, 12-heads, 110M parameter transformer architecture, trained on the BookCorpus with 800M words and a version of the English Wikipedia with 2500M words. We convert source and target sentences to WordPieces (Wu et al., 2016). For target sentences we randomly mask  $15\%$  of WordPiece tokens, as in (Devlin et al., 2019). If a token in a specific position is selected to be masked  $80\%$  of the time is replaced with a [MASK] token,  $10\%$  of the time with a random token and  $10\%$  of the time remains unchanged.
The maximum sequence length is set to 512 by truncation of inputs. During domain pretraining we train with batch size of 8 for 3 epochs (2 hours on two GTX-1080Ti cards). During the final finetuning step of UDALM we train with batch size 36, consisting of  $n = 1$  source sub-batch of 4 samples and  $m = 8$  target sub-batches of 4 samples each. We update parameters after every 5 accumulated sub-batches. We train for 10 epochs with early stopping on the mixed loss in Eq. 2. For all experiments we use AdamW optimizer (Loshchilov and Hutter, 2018) with learning rate  $10^{-5}$ . Each adaptation scenario requires one hour on one GTX-1080Ti. For the domain adversarial experiments we set  $\lambda_d = 0.01$  in Eq.  $4^2$  and train for 10 epochs. Models are developed with PyTorch (Paszke et al., 2019) and HuggingFace Transformers (Wolf et al., 2019).
5.3 Baselines - Compared methods
We select three state-of-the-art methods for comparison. Each of the selected methods represents a different line of UDA research, namely domain adversarial training BERT-DAAT (Du et al., 2020), self-training XLM-R based  $\mathbf{p} + \mathbf{C}\mathbf{f}\mathbf{d}$  (Ye et al., 2020) and pivot-based R-PERL (Ben-David et al., 2020). We report results for the following settings with BERT models:
Source only (SO): We fine-tune BERT on source domain labeled data, without using target data.
Domain Pretraining (DPT): We use the target domain unlabeled data in order to continue pretraining of BERT with MLM loss (as in Fig. 1b) and then
2We also manually experimented with  $\lambda_{d} = 1$  and lambda  $d = 0.1$ , and a sigmoid schedule for  $\lambda_{d}$ . We report best results.
R-PERL	DAAT	p+CFd	SO BERT	DAT BERT	DPT BERT	UDALM
B→D	87.8	90.9	87.7	89.51 ± 0.76	87.31 ± 2.14	90.49 ± 0.38	90.97 ± 0.22
B→E	87.2	88.9	91.3	90.51 ± 0.51	86.91 ± 2.71	90.38 ± 1.59	91.69 ± 0.31
B→K	90.2	88.0	92.5	91.75 ± 0.28	90.59 ± 1.17	92.66 ± 0.43	93.21 ± 0.22
D→B	85.6	89.7	91.5	90.26 ± 0.64	86.30 ± 3.10	91.02 ± 0.75	91.00 ± 0.42
D→E	89.3	90.1	91.6	88.71 ± 1.48	87.85 ± 1.24	91.03 ± 0.82	92.30 ± 0.47
D→K	90.4	88.8	92.5	91.22 ± 0.69	89.95 ± 1.53	92.30 ± 0.42	93.66 ± 0.37
E→B	90.2	89.6	88.7	87.96 ± 0.89	85.65 ± 1.91	88.52 ± 0.55	90.61 ± 0.30
E→D	84.8	89.3	88.2	87.37 ± 0.64	83.99 ± 1.31	87.85 ± 0.47	88.83 ± 0.61
E→K	91.2	91.7	93.6	93.30 ± 0.50	92.45 ± 1.35	94.39 ± 0.72	94.43 ± 0.24
K→B	83.0	90.8	89.8	88.15 ± 0.64	85.07 ± 1.03	88.83 ± 0.81	90.29 ± 0.51
K→D	85.6	90.5	87.8	87.23 ± 0.49	84.11 ± 0.62	88.52 ± 0.69	89.54 ± 0.59
K→E	91.2	93.2	92.6	93.23 ± 0.34	92.07 ± 0.24	93.42 ± 0.40	94.34 ± 0.26
Average	87.50	90.12	90.63	89.93 ± 0.65	87.68 ± 1.53	90.78 ± 0.67	91.74 ± 0.38
Table 1: Accuracy of unsupervised domain adaptation on twelve domain pairs of Amazon Reviews Multi Domain Sentiment Dataset.
fine-tune the resulting model on source domain labeled data.
Domain Adversarial (DAT): Domain Adversarial Training with BERT. Starting from the domain pretrained BERT (as in Fig. 1b), we then fine-tune the model with domain adversarial training as in Ganin et al. (2016). For a BERT model with parameters  $\theta$ , with  $L_{CLF}$  being a cross-entropy loss for supervised task prediction,  $L_{ADV}$  being a cross-entropy loss for domain prediction and  $\lambda_d$  being a weighting factor, domain adversarial training consists of the minimization criterion described in Eq. 4.
$$
\min  _ {\theta} L _ {C L F} (\theta ; D _ {S}) - \lambda_ {d} L _ {A D V} (\theta ; D _ {S}, D _ {T}) \tag {4}
$$
UDALM: The proposed method, where we finetune the model created in the domain pretraining step using the mixed loss in Eq. 2.
6.1 Comparison to state-of-the-art
We present results for all 12 domain adaptation settings in Table 1. Results for SO BERT, DAT BERT, DPT BERT and UDALM are averaged over five runs and we include standard deviations The last line of Table 1 contains the macro-averaged accuracy and deviations over all domain pairs. UDALM surpasses all other techniques, yielding an absolute improvement of  $1.81\%$  over the SO BERT baseline. For fair comparison, we compare only with methods based on pretrained models, mostly BERT. We observe that BERT fine-tuned only with the source domain labeled data, without any knowledge of the target domain, is a competitive baseline. This source-only model even surpasses state-of-the-art methods developed for UDA, e.g. R-PERL (Ben-David et al., 2020).
We reproduce the domain adversarial training procedure and present results in the DAT BERT column of Table 1. Adversarial training proved to be unstable in our experiments, even after careful tuning of the adversarial loss weighting factor  $\lambda_{d}$ . This is evidenced by the high standard deviations in the DAT BERT experiments. We observe that adversarial training does not manage to outperform the source-only baseline.
Domain pretraining increases the average accuracy with an absolute improvement of  $0.85\%$  over the source-only baseline. Continuing MLM pretraining on the target domain data leads to better model adaptation, and therefore improved performance, on the target domain. This is consistent with previous works on supervised (Gururangan et al., 2020; Xu et al., 2019; Sun et al., 2019) and unsupervised settings (Han and Eisenstein, 2019; Du et al., 2020).
UDALM yields an additional  $0.96\%$  absolute improvement of average accuracy over domain pretraining. Keeping the MLM loss during fine-tuning therefore, leads to better adaptation and acts as a regularizer that prevents the model from overfitting on the source domain. We also observe smaller standard deviations when using UDALM, which indicates that including the MLM loss during fine-tuning can result in more robust training.
6.2 Sample efficiency
UDALM surpasses in terms of macro-average accuracy all other approaches for unsupervised domain adaptation on the Amazon reviews multi-domain sentiment dataset. Specifically, our method improves on the state-of-the-art pseudo-labeling
Note that we did not have to perform extensive tuning for the other methods, including UDALM.
Figure 2: Average accuracy for different amount of target domain unlabeled samples of: (1) DPT BERT (2) DAT BERT and (3) UDALM.
$(p + \mathrm{CFdYe}$  et al., 2020), domain adversarial (DAAT Du et al., 2020) and pivot-based (R-PERL BenDavid et al., 2020) approaches by  $1.11\%$ $1.62\%$  and  $4.24\%$  respectively.
We further investigate the impact of using different amount of target domain unlabeled data on model performance, to study the sample efficiency of UDALM. We experiment with settings of 500, 2000, 6000, 10000 and 14000 samples, by randomly limiting the number of unlabeled target domain data. For each setting we conduct three experiments with BERT models: (1) DPT, (2) DAT and (3) UDALM. When no target data are available, all methods are equivalent to a source only fine-tuned BERT. Again, we do not tune the hyper-parameters for DPT or UDALM. Fig. 2 shows the average accuracy on the twelve adaptation scenarios of the studied dataset. We see that UDALM produces robust performance improvement when we limit the amount of target data, indicating that it can be used in low-resource settings. However, training BERT in a domain adversarial manner shows instabilities. This is further discussed in Section 7.
6.3 On the stopping criteria for UDA training
A common problem when performing UDA is the lack of target labeled data that can be used for hyperparameter validation. For example, Ruder and Plank (2018) use a small set of labeled target data for validation, putting the problem in a semi-supervised setting. When training under a domain shift, optimization of model performance on the source data may not result in optimal performance for the target data.
To illustrate this, we examine if the minimization of the mixed loss can be used as a stopping criterion for UDA training. We compare five stopping criteria: (1) fixed training for 1 epoch, (2) fixed training for 3 epochs, (3) fixed training for 10 epochs, (4) stop when the minimum classification loss is reached for the source data and (5) stop when the minimum mixed loss ( Eq. 2) is reached. For (4) and (5) we train for 10 epochs with patience 3. We report average accuracy of the five stopping criteria over the twelve adaptation scenarios of Amazon Reviews dataset on Table 2. Training for a fixed number of 10 epochs and stopping when the minimum mixed loss perform best, yielding comparable accuracies of  $91.75\%$  and  $91.73\%$  respectively. Note that stopping when the minimum source loss stops the fine-tuning process too soon and does not allow the model to learn the target domain effectively. Overall, we observe that the mixed loss can be effectively used for early stopping, regularizing the model and alleviating the need for extensive search for the optimal number of training steps. This is an indication that the mixed loss could be used for model validation.
Stopping Criterion	Epochs	Av. Acc.
Fixed	1	90.98
Fixed	3	91.65
Fixed	10	91.75
Min source loss	10, patience 3	91.30
Min mixed loss	10, patience 3	91.74
Table 2: Comparison of average accuracy for various validation settings.