SlowGuess/ACL_2020_OCR huggingface dataset:
$^{12}$ Chunning Du,  $^{12}$ Haifeng Sun,  $^{12}$ Jingyu Wang, $^{*12}$ Qi Qi,  $^{12}$ Jianxin Liao
<sup>1</sup>State Key Laboratory of Networking and Switching Technology,
Beijing University of Posts and Telecommunications, Beijing 100876, China
$^{2}$ EBUPT Information Technology Co., Ltd., Beijing 100191, China
{duchunning,sunhaifeng_1,wangjingyu,qiqi}@ebupt.com
Cross-domain sentiment classification aims to address the lack of massive amounts of labeled data. It demands to predict sentiment polarity on a target domain utilizing a classifier learned from a source domain. In this paper, we investigate how to efficiently apply the pre-training language model BERT on the unsupervised domain adaptation. Due to the pre-training task and corpus, BERT is task-agnostic, which lacks domain awareness and can not distinguish the characteristic of source and target domain when transferring knowledge. To tackle these problems, we design a post-training procedure, which contains the target domain masked language model task and a novel domain-distinguish pre-training task. The post-training procedure will encourage BERT to be domain-aware and distill the domain-specific features in a self-supervised way. Based on this, we could then conduct the adversarial training to derive the enhanced domain-invariant features. Extensive experiments on Amazon dataset show that our model outperforms state-of-the-art methods by a large margin. The ablation study demonstrates that the remarkable improvement is not only from BERT but also from our method.
Sentiment analysis aims to automatically identify the sentiment polarity of the textual data. It is an essential task in natural language processing with widespread applications, such as movie reviews and product recommendations. Recently, deep networks have significantly improved the state-of-the-art in sentiment analysis. However, training deep networks is highly depended on a large amount of labeled training data which is time-consuming and requires expensive manual labeling. Thus, there is a strong need to leverage or reuse rich labeled
data from a different but related source domain. Cross-domain sentiment analysis, which transfers the knowledge learned from source domain to a new target domain, becomes a promising direction.
The main challenge of cross-domain sentiment analysis is domain discrepancy due to different expression of the user's emotion across domains. To address the problem, a wide-used approach is designed to extract domain invariant features, which means that the distributions of features from the source domain and target domain are similar (Zellinger et al., 2017; Persello and Bruzzone, 2016; Ganin et al., 2016; Yu and Jiang, 2016a). One effective way to obtain domain-invariant features is adversarial training(Ganin et al., 2016; Li et al., 2017; Zheng et al., 2019). Specifically, A domain discriminator is learned by minimizing the classification error of distinguishing the source from the target domains, while a deep classification model learns transferable representations that are indistinguishable by the domain discriminator.
Very recently, pre-trained language models have shown to be effective for improving many language tasks (Peters et al., 2018). Bidirectional Encoder Representations from Transformers (BERT) realized a breakthrough, which pre-trains its encoder using language modeling and by discriminating surrounding sentences in a document from random ones (Devlin et al., 2019). Pre-training in this manner can construct bidirectional contextual representation, and the large-scale pre-training enables BERT powerful ability in language understanding. We only need to add one output layer and fine-tune BERT to get the state-of-the-art results in sentiment analysis. Theoretically, BERT can enhance the performance in cross-domain sentiment analysis. However, some important problems remain when directly fine-tuning BERT in the task of cross-domain sentiment analysis:
Firstly, there is no labeled data in the target do
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4019-4028 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
main which brings many difficulties to the fine-tune procedure. If we fine-tune BERT only by the source labeled data, the shift between training and test distributions will degrade the BERT's performance. Secondly, BERT is task-agnostic and has almost no understanding of opinion text. BERT is pretrained by the universal language Wikipedia, leaving the domain challenges unresolved (Xu et al., 2019). For example, in the pre-training procedure, BERT may learn to guess the [MASK] in "The [MASK] is bright" as "sun". But in a laptop sentiment analysis, it is more likely to be "screen". Especially, in the cross-domain sentiment analysis scenario, the labeled data is limited, which is insufficient to fine-tune BERT to ensure full domain-awareness. Thirdly, cross-domain sentiment analysis also arises a new challenge for BERT to distinguish the characteristic of source and target domain to keep the transferable features and abandon domain-specific information.
To address above problems, we design a novel pre-training task to make BERT domain-aware and then improve the BERT's fine-tuning procedure by adversarial training. Specifically, a novel post-training procedure is implemented that adapts BERT with unlabeled data from different domains to enhance the domain-awareness. Apart from the target domain masked language model task, we introduce the domain-distinguish pre-training task. BERT will be pre-trained to distinguish whether the two sentences come from the same domain. The domain-distinguish pre-training task will encourage BERT to distill syntactic and semantic domain-specific features, so as to be domain-aware. The proposed post-training procedure gives us a new way to fully utilize language knowledge from the target domain and link the source and target in a self-supervised way. Based on this, we could then conduct the adversarial training to derive the enhanced domain-invariant features.
Experiments on Amazon reviews benchmark dataset show that our model gets the average result  $90.12\%$  in accuracy,  $4.22\%$  absolute improvement compared with state-of-the-art methods. The ablation study shows that the remarkable improvement is not only from BERT but also from our method. The contributions of this paper can be summarized as follows:
- We apply BERT to cross-domain sentiment analysis task and leverage the post-training method to inject the target domain knowledge
to BERT.
- A novel domain-distinguish pre-training task is proposed for the BERT post-training. This design encourages BERT to be domain-aware and distill the domain-specific features in a self-supervised way.
2.1 Cross-Domain Sentiment Analysis
Cross-domain sentiment analysis aims to generalize a classifier that is trained on a source domain, for which typically plenty of labeled data is available, to a target domain, for which labeled data is scarce. There are many pivot-based methods (Blitzer et al., 2007a; Yu and Jiang, 2016b; Ziser and Reichart, 2018; Peng et al., 2018), which focus on inducing a low-dimensional feature representation shared across domains based on the cooccurrence between pivots and non-pivots. However, selecting pivot words is very tedious, and the pivot words are manually selected, which may not be accurate. Recently, some adversarial learning methods (Ganin et al., 2016; Li et al., 2017; Zheng et al., 2019) propose to train the feature generator to minimize the classification loss and simultaneously deceive the discriminator, which is end-to-end without manually selecting pivots.
2.2 Language Model Pre-training
Pre-trained language representations with self-supervised objectives have become standard in a wide range of NLP tasks. Previous work can be divided into two main categories: feature-based approaches and fine-tuning approaches.
The recent proposed feature-based approaches mainly focus on learning contextualized word representations such as CoVe (McCann et al., 2017) and ELMo (Peters et al., 2018). As with traditional word embeddings, these learned representations are also typically used as features in a downstream model. On the other hand, fine-tuning approaches mainly pre-train a language model on a large corpus with an unsupervised objective and then fine-tune the model with in-domain labeled data for downstream applications. The advantage of these approaches is that few parameters need to be learned from scratch. Specifically, Howard and Ruder (2018) propose ULMFiT, which uses a different learning rate for each layer with learning warmup and gradual unfreezing. GPT (Radford et al., 2018) uses a transformer encoder (Vaswani
et al., 2017) instead of an LSTM and jointly finetunes with the language modeling objective. Moreover, BERT (Devlin et al., 2019) is a large-scale language model consisting of multiple layers of transformer, which further incorporates bidirectional representations. BERT is the state-of-art pre-training language model. However, in the cross-domain sentiment analysis scenario, BERT is task-agnostic and can not distinguish the characteristic of source and target domain.
In this section, we introduce the proposed model for cross-domain sentiment analysis in detail. We begin by giving the problem definition and notations. Then BERT and post-training method are formally presented in the second subsection. Finally, the adversarial training process is introduced. We also give a theoretical analysis of our model.
3.1 Problem Definition and Notations
In the task of cross-domain sentiment analysis, we are given two domains  $D_{s}$  and  $D_{t}$  which denote a source domain and a target domain, respectively. In the source domain,  $D_{s}^{l} = \{x_{s}^{i},y_{s}^{i}\}_{i = 1}^{N_{s}^{l}}$  are  $N_{s}^{l}$  labeled source domain examples, where  $x_{s}^{i}$  means a sentence and  $y_{s}^{i}$  is the corresponding polarity label. There are also  $N_{s}^{u}$  unlabeled data  $D_{s}^{u} = \{x_{s}^{i}\}_{i = 1 + N_{s}^{l}}^{N_{s}^{l} + N_{s}^{u}}$  in the source domain. In the target domain, there is a set of unlabeled data  $D_{t} = \{x_{t}^{i}\}_{i = 1}^{N_{t}}$ , where  $N_{t}$  is the number of unlabeled data. Cross-domain sentiment analysis demands us to learn a robust classifier trained on labeled source domain data to predict the polarity of unlabeled sentences from the target domain.
3.2 Background of BERT
BERT (Devlin et al., 2019) builds on the Transformer networks (Vaswani et al., 2017), which relies purely on attention mechanism and allows modeling of dependencies without regard to their distance in the input sequences. BERT is pre-trained by predicting randomly masked words in the input (MLM task) and classifying whether the sentences are continuous or not (NSP task). The MLM task allows the word representation to fuse the left and the right context, and the NSP task enables BERT to infer the sentences' relationship. The pre-trained BERT can be easily fine-tuned by one softmax output layer for classification task.
3.3 BERT Post-training
Despite the success, BERT suffers from the domain challenge. BERT is pre-trained by Wikipedia, leading to task-agnostic and little understanding of opinion text. Especially in the cross-domain sentiment analysis scenario, the lack of abundant labeled data limits the fine-tune procedure, which degrades BERT due to the domain shift. This task also demands BERT to distinguish the characteristic of source and target domain for better knowledge transfer. Therefore, we propose BERT posttraining which takes BERT's pre-trained weights as the initialization for basic language understanding and adapts BERT by novel self-supervised pretrained tasks: domain-distinguish task and target domain masked language model.
3.3.1 Domain-distinguish Task
The next sentence prediction (NSP) task encourages BERT to model the relationship between sentences beyond word-level, which benefits the task of Question Answering and Natural Language Inference. However, cross-domain sentiment analysis operates on a single text sentence and does not require the inference ability. Instead, the ability to distinguish domains plays an important role. Therefore, during the post-training procedure, we replace the NSP task by domain-distinguish task (DDT). Specifically, we construct the sentence-pair input: [CLS] sentence A [SEP] sentence B [SEP], where [CLS] and [SEP] are special embeddings for classification and separating sentences.  $50\%$  of time sentence A and sentence B are all randomly sampled from target domain reviews, we label it TargetDomain. And  $50\%$  of time sentence A and sentence B come from target domain and another domain, whose label is MixDomain. We do not fix the collocation, in another word, we only ensure that the two sentences come from different domains but the order is random. For example:
$$
\begin{array}{l} \begin{array}{l} \text {I n p u t} = [ \text {C L S} ] \text {T h e m o u s e i s s m o o t h a n g r e t} \\ [ \text {S E P} ] \text {T h e s c r e e n i s p l a i n} [ \text {S E P} ] \end{array} \\ \text {L a b e l} = \text {T a r g e t D o m a i n} \\ \text {I n p u t} = [ \mathrm {C L S} ] \text {T h i s b o o k i s b o r i n g [ S E P ] T h e} \\ \text {L a b e l} = \text {M i x D o m a i n} \\ \end{array}
$$
The domain-distinguish pre-training is a classifi-
cation task. We add one output layer on the pooled representation and maximize the likelihood of the right label. The domain-distinguish pre-training enables BERT to distill the specific features for different domains, which enhances the downstream adversarial training and benefits the cross-domain sentiment analysis.
3.3.2 Target Domain MLM
To inject the target domain knowledge, we leverage the masked language model (MLM) (Devlin et al., 2019). It requires to predict the randomly masked words in the sentence, which encourages BERT to construct a deep bidirectional representation. In the cross-domain sentiment analysis, there are no labeled data but plenty of unlabeled data in the target domain to post-train BERT by MLM. Specifically, we replace  $15\%$  of tokens by [MASK] at random. The final hidden vectors corresponding to the mask tokens are fed into an output softmax over the vocabulary. We maximize the likelihood of the masked token id.
Post-training by unlabeled review data in target domain will effectively alleviate the shift of domain knowledge. For example, if the masked word is an opinion word in "This movie is [MASK]", this objective challenges BERT to learn the representation for fine-grained opinion words in movie review domain, such as "touchable" or "disturbing".
One problem is that the DDT task mixes sentences from other domains in the sentence pair. The sentence from other domains will be the noise which brings domain bias. Therefore, we only mask the tokens in target domain sentences if the domain-distinguish task label is MixDomain.
The total loss of the post-training procedure is the sum of losses of target domain MLM and domain-distinguish task. The adaptation takes about 5 hours to complete on one single NVIDIA P100 GPU.
3.4 Adversarial Training
The post-training procedure injects target domain knowledge and brings domain-awareness to BERT. Based on the post-trained BERT, we now could utilize the adversarial training to abandon the distilled domain-specific features to derive the domain-invariant features. Specifically, a sentiment classifier and a domain discriminator is designed operating on the hidden state  $h_{[CLS]}$  of the special classification embedding [CLS].
3.4.1 Sentiment Classifier
The sentiment classifier is simply a fully-connected layer and outputs the probabilities through a softmax layer:
$$
y _ {s} = \operatorname {s o f t m a x} \left(W _ {s} h _ {[ C L S ]} + b _ {s}\right). \tag {1}
$$
The classifier is trained by the labeled data in the source domain and the loss function is cross-entropy:
$$
L _ {s e n} = - \frac {1}{N _ {s} ^ {l}} \sum_ {i = 1} ^ {N _ {s} ^ {l}} \sum_ {j = 1} ^ {K} \hat {y} _ {s} ^ {i} (j) \log y _ {s} ^ {i} (j), \tag {2}
$$
where  $\hat{y}_s^i\in \{0,1\}$  is the ground truth label in the source domain, and  $K$  denotes the number of different polarities.
3.4.2 Domain Discriminator
The domain discriminator aims to predict domain labels of samples, i.e., coming from the source or target domain. The parameters of BERT are optimized to maximize the loss of the domain discriminator. This target will encourage BERT to fool the domain discriminator to generate domain-invariant features.
Specifically, before feeding  $h_{[CLS]}$  to the domain discriminator, the hidden state of classification embedding [CLS]  $h_{[CLS]}$  goes through the gradient reversal layer (GRL) (Ganin et al., 2016). During the forward propagation, the GRL acts as an identity function but during the backpropagation, the GRL reverses the gradient by multiplying it by a negative scalar  $\lambda$ . GRL can be formulated as a 'pseudo-function'  $Q_{\lambda}(x)$  by two equations below in order to describe its forward- and backward-behaviors:
$$
Q _ {\lambda} (x) = x, \tag {3}
$$
$$
\frac {\partial Q _ {\lambda} (x)}{\partial x} = - \lambda I. \tag {4}
$$
We denote the hidden state  $h_{[CLS]}$  through the GRL as  $Q_{\lambda}(h_{[CLS]}) = \hat{h}_{[CLS]}$  and then feed it to the domain discriminator as:
$$
d = \operatorname {s o f t m a x} \left(W _ {d} \hat {h} _ {[ C L S ]} + b _ {d}\right). \tag {5}
$$
The target is to minimize the cross-entropy for all data from the source and target domains:
$$
L _ {d o m} = - \frac {1}{N _ {s} + N _ {t}} \sum_ {i} ^ {N _ {s} + N _ {t}} \sum_ {j} ^ {K} \hat {d} ^ {i} (j) \log d ^ {i} (j), \tag {6}
$$
4022
where  $\hat{d}^i\in \{0,1\}$  is the ground truth domain label. Due to the GRL, the parameters for domain discriminator  $\theta_{dd}$  are optimized to increase the ability to predict domain labels, however, the parameters for BERT  $\theta_{BERT}$  are optimized to fool the domain discriminator, leading to domain-invariant features.
3.4.3 Joint Learning
The sentiment classifier and the domain discriminator are jointly trained, and the total loss is:
$$
L _ {t o t a l} = L _ {s e n} + L _ {d o m}. \tag {7}
$$
The post-training procedure and our proposed domain-distinguish pre-training task will enhance the adversarial training to obtain lower classification error in the target domain, we will analyze it in Sec 3.5.
3.5 Theoretical Analysis
In this section, we provide a theoretical analysis of our approach. First, we provide an insight into existing theory, then we introduce an expansion of the theory related to our method and explain how the post-training and adversarial training cooperate to obtain a remarkably better result than state-of-the-art methods.
For each domain, there is a labeling function on inputs  $X$ , defined as  $f: X \to [0,1]$ . The ideal label functions for source and target domain are denoted as:  $f_s$  and  $f_t$ , respectively. We define a hypothesis label function  $h: X \to [0,1]$  and a disagreement function:
$$
\epsilon \left(h _ {1}, h _ {2}\right) = E \left[ \left| h _ {1} (x) - h _ {2} (x) \right| \right]. \tag {8}
$$
Then the expected error on the source samples of  $h$  is defined as:  $\epsilon_s(h) = \epsilon_s(h, f_s)$ . For the target domain, we have:  $\epsilon_t(h) = \epsilon_t(h, f_t)$ .
The divergence between source and target domain could thus be measured by  $\mathcal{H}\Delta \mathcal{H}$ -distance, which is defined as follows:
$$
d _ {\mathcal {H} \Delta \mathcal {H}} \left(D _ {s}, D _ {t}\right) = 2 \sup  _ {h, h ^ {\prime} \in \mathcal {H}} \left| \epsilon_ {s} \left(h, h ^ {\prime}\right) - \epsilon_ {t} \left(h, h ^ {\prime}\right) \right| \tag {9}
$$
This distance is firstly proposed in (Ben-David et al., 2010) and frequently used to measure the adaptability between different domains (Shen et al., 2018; Chen et al., 2019).
3.5.1 Theorem 1.
Let  $\mathrm{H}$  be the hypothesis class. Given two different domains  $D_{s},D_{t}$ , we have:
$$
\forall h \in H, \epsilon_ {t} (h) \leq \epsilon_ {s} (h) + \frac {1}{2} d _ {\mathcal {H} \Delta \mathcal {H}} \left(D _ {s}, D _ {t}\right) + C \tag {10}
$$
This theorem means that the expected error on the target domain is upper bounded by three terms: (1) the expected error on the source domain; (2) the divergence between the distributions  $D_{s}$  and  $D_{t}$ ; (3) the error of the ideal joint hypothesis. Normally,  $C$  is disregarded because it is considered to be negligibly small. Therefore, the first and second terms are important quantitatively in computing the target error.
For the first term, the error rate of source domain  $\epsilon_{s}$ , it is easy to minimize with source labeled training data. Moreover, we adopt BERT, which brings powerful contextual representation for lower error rate. The second item in Eq. 10 demands us to generate similar features among different domains. Our proposed domain-distinguish pre-training task and post-training for BERT enable the model to identify the specific features for different domains. This ability will enhance the domain discriminator, which will help to find more complicated domain specific features and get abandoned by adversarial training. Therefore, we further decrease the divergence between the domains, which is quantitatively measured by  $\mathcal{A}$ -distance in Sec 4.6.
In this section, we empirically evaluate the performance of our proposed methods.
4.1 Datasets and Experimental Setting
We conduct the experiments on the widely-used Amazon reviews benchmark datasets collected by (Blitzer et al., 2007b). It contains reviews from four different domains: Books (B), DVDs (D), Electronics (E) and Kitchen appliances (K). For each domain, there are 2,000 labeled reviews and approximately 4000 unlabeled reviews. Following the convention of previous works (Ziser and Reichart, 2018; Ganin et al., 2016; Qu et al., 2019), we construct 12 cross-domain sentiment analysis tasks. For each task, we employ a 5-fold cross-validation protocol, that is, in each fold, 1600 balanced samples are randomly selected from the labeled data for training and the rest 400 for validation.
S→T	Previous Models	BERT
DANN	PBLM	HATN	ACAN	IATN	BERT	HATN-BERT	BERT-AT	BERT-DA	BERT-DAAT
D→B	81.70	82.50	86.30	82.35	87.00	89.40	89.81	89.55	90.40	90.86
E→B	78.55	71.40	81.00	79.75	81.80	86.50	87.10	87.15	88.31	88.91
K→B	79.25	74.20	83.30	80.80	84.70	87.55	87.88	87.65	87.90	87.98
B→D	82.30	84.20	86.10	83.45	86.80	88.96	89.36	89.70	89.75	89.70
E→D	79.70	75.00	84.00	81.75	84.10	87.95	88.81	88.20	89.03	90.13
K→D	80.45	79.80	84.50	82.10	84.10	87.30	87.89	87.72	88.35	88.81
B→E	77.60	77.60	85.70	81.20	86.50	86.15	87.21	87.30	88.11	89.57
D→E	79.70	79.60	85.60	82.80	86.90	86.55	86.99	86.05	88.15	89.30
K→E	86.65	87.10	87.00	86.60	87.60	90.45	90.31	90.25	90.59	91.72
B→K	76.10	82.50	85.20	83.05	85.90	89.05	89.41	89.55	90.65	90.75
D→K	77.35	83.20	86.20	78.60	85.80	87.53	87.59	87.69	88.55	90.50
E→K	83.95	87.80	87.90	83.35	88.70	91.60	92.01	91.91	92.75	93.18
Average	80.29	80.40	85.10	82.15	85.90	88.25	88.69	88.56	89.37	90.12
Table 1: Accuracy of domain adaptation on Amazon benchmark.
We adopt  $\mathrm{BERT}_{\mathrm{base}}$  (uncased) as the basis for all experiments. When generating the post-training data, each sentence in the target domain gets duplicated 10 times with different masks and sentences pair. We limit the maximum sequence length is 256. During the post-training, we train with batch size of 16 for 10000 steps. The optimizer is Adam with learning rate  $2\mathrm{e} - 5$ ,  $\beta_{1} = 0.9$ ,  $\beta_{2} = 0.999$ , L2 weight decay of 0.01. During the adversarial training, The weights in sentiment classifier and domain discriminator are initialized from a truncated normal distribution with mean 0.0 and stddev 0.02. In the gradient reversal layer (GRL), we define the training progress as  $p = \frac{t}{T}$ , where  $t$  and  $T$  are current training step and the maximum training step, respectively, and the adaptation rate  $\lambda$  is increased as  $\lambda = \frac{2}{1 + \exp(-10p)} - 1$ .
4.3 Compared Methods