SlowGuess/ACL_2020_OCR huggingface dataset:
We compare our method with 5 state-of-the-art methods: DANN (Ganin et al., 2016), PBLM (Ziser and Reichart, 2018), HATN (Li et al., 2018), ACAN (Qu et al., 2019), IATN (Zhang et al., 2019). We also design several variants of BERT as baselines:
- BERT: Fine-tuning vanilla BERT by the source domain labeled data.
- HATN-BERT: HATN (Li et al., 2018) model based on BERT.
- BERT-AT: This method conducts the adversarial training operating on vanilla BERT.
- BERT-DA: Fine-tuning domain-aware BERT by the source domain labeled data. The domain-aware BERT is obtained by post-training.
- BERT-DAAT: Our proposed method introduced in Sec 3.
Table 2 shows the classification accuracy of different methods. We can observe that the proposed BERT-DAAT outperforms all other methods.
For the previous models, they mostly base on the word2vec (Mikolov et al., 2013) or glove (Pennington et al., 2014). Compared to BERT's contextual word representation, they can not model complex characteristics of word use and how these uses vary across linguistic contexts, resulting in relatively worse overall performance. We can see that the vanilla BERT, which is fine-tuned only by the source domain labeled data without utilizing target domain data, can still outperform all the previous methods. For fair comparison, we reproduce the experiment of HATN model (Li et al., 2018) that incorporates BERT as the base model. As shown in Table 2, HATN-BERT achieves a comparable result with BERT-AT.
For the BERT variants, we did not see a remarkable improvement in the results of BERT-AT, which conducts adversarial training on BERT. It demonstrates that, in the task of cross-domain sentiment analysis, the bottleneck of BERT is the lack of domain-awareness and can not be tackled purely
Figure 1: The effect of post-training and adversarial training on the distribution of the extracted features. The figure shows t-SNE visualization of the BERT's hidden state for the  $\mathrm{B}\rightarrow \mathrm{E}$  task. The red, blue, green and black points denote the source negative, source positive, target negative and target positive examples, respectively.
by adversarial training. On the contrary, the posttraining procedure could improve the result by  $1.12\%$  on average. It verifies the effectiveness of our proposed post-training methods that could inject the domain knowledge to BERT. As expected, BERT-DAAT performs best among the variants of BERT,  $0.75\%$  absolute improvement to BERT-DA and  $1.87\%$  absolute improvement to BERT, showing that the post-training procedure could further enhance the adversarial training.
4.5 Visualization of Features
To intuitively assess the effects of the post-training and adversarial training on BERT, we further perform a visualization of the feature representations of the variants of BERT for the training data in the source domain and the testing data in the target domain for the  $\mathrm{B}\rightarrow \mathrm{E}$  task. As shown in Figure 1, the graphs are obtained by applying t-SNE on the set of all representation of source and target data points. Every sample is mapped into a 768-dimensional feature space through BERT and projected back into a two-dimensional plane by the t-SNE.
In the vanilla BERT representation (first subgraph in Figure 1), we could observe that data points of different polarities in source domain are well separated. While for the target domain, some data points are mixed together. It shows that only utilizing source domain labeled data is not enough for the target domain classification. For the post-trained BERT (subgraph for BERT-DA), data points belong to four clusters, indicating that domains and sentiment polarities are both well classified. It verifies that our post-training strategy brings domain-awareness to BERT. Moreover, compared to the first subgraph, the boundary for sentiment polarity classification is more clear, showing that injecting domain knowledge by post-training is beneficial to sentiment classification.
The latter two subgraphs in Figure 1 are the feature distributions obtained by adversarial training. One common characteristic is that data samples from different domains are very close to each other through adversarial training. However, the boundary for sentiment polarity classification is not very clear in BERT-AT's feature representation, resulting in degraded performance. For our proposed BERT-DAAT, the post-training enables the domain-awareness and help to distill more complicated domain specific features. The adversarial training is thus enhanced to get more domain-invariant features. We can find that target points are homogeneously spread out among source points, which decreases the divergence between the domains. According to Theorem 10, it can lower the upper boundary of the target error.
4.6  $\mathcal{A}$ -distance
Theorem 10 shows that the divergence between domains  $d_{\mathcal{H}\Delta \mathcal{H}}(D_s,D_t)$  plays an important role. To quantitatively measure it, we compare the  $\mathcal{A}$ -distance, which is usually used to measure domain discrepancy (Ben-David et al., 2010). The definition of  $\mathcal{A}$ -distance is:  $d_{\mathcal{A}} = 2(1 - 2\epsilon)$ , where  $\epsilon$  is the generalization error of a classifier trained with the binary classification task of discriminating the source domain and target domain. More precisely, to obtain  $\mathcal{A}$ -distance, we firstly split source and target domain data into two subsets of equal size and get the feature representation. We then train a linear SVM on the first subset to predict which domain the sample comes from. The error rate  $\epsilon$  could be calculated on the second subset through the trained SVM, and  $\mathcal{A}$ -distance is obtained by  $d_{\mathcal{A}} = 2(1 - 2\epsilon)$ .
We compare the  $\mathcal{A}$ -distance of BERT, BERT-AT, and BERT-DAAT. Results are shown in Figure 2. For each cross-domain sentiment analysis task,
4025
Figure 2: Comparison of  $\mathcal{A}$ -distance of different models.
the  $\mathcal{A}$ -distance of BERT is highest. It is easy to conclude that applying adversarial training can effectively decrease the  $\mathcal{A}$ -distance. Overall, the  $\mathcal{A}$ -distance of BERT-DAAT is lower than BERT-AT, verifying that the post-training could enhance the adversarial training to decrease the domain discrepancy.
To analyze the effect of different components including post-training steps and post-training tasks, we conduct the ablation experiments.
4.7.1 Effects of Post-Training Steps
In this subsection, we study the effect of post-training steps. Figure 3 presents the accuracy on the task of  $\mathrm{E} \rightarrow \mathrm{K}$  based on the checkpoint that has been post-trained for  $k$  steps. The results for BERT-DA are obtained by fine-tuning source domain labeled data, BERT-DAAT is adversarial training by source labeled data and target unlabeled data.
We find that, with limited post-training steps (fewer than 5000 steps), BERT-DA and BERT-DAAT perform similarly with BERT and BERT-AT, respectively. However, given post-training steps more than 5000, both the results of BERT-DA and BERT-DAAT see an increase. Especially, after post-training more than 5000 steps, BERT-DAAT shows remarkable strengths compared to BERT-DA. This shows that plenty of post-training steps is necessary to inject domain knowledge and domain-awareness.
4.7.2 Effects of Post-training Tasks
The post-training tasks in our work include target domain masked language model (MLM) and our proposed domain-distinguish task (DDT). We design two models which ablate MLM and DDT
Figure 3: Ablation study on the number of post-training steps. The x-axis is the value of post-training steps  $k$ . The y-axis is the accuracy on the task of  $\mathrm{E} \rightarrow \mathrm{K}$ .
Model	D→B	E→B	K→B
BERT	89.40	86.50	87.55
BERT-DAAT	90.86	88.91	87.98
-w/o MLM	89.91	87.39	87.80
-w/o DDT	90.02	88.01	87.63
Table 2: Ablation study over post-training tasks. w/o means without.
separately and compare them with BERT-DAAT on the tasks of  $\mathrm{D}\rightarrow \mathrm{B}$ $\mathrm{E}\rightarrow \mathrm{B}$  ,and  $\mathrm{K}\rightarrow \mathrm{B}$  .Results in Table 2 indicate that: the target domain masked language model task (MLM) and domain-distinguish taskDDT) are both beneficial to cross-domain sentiment analysis.
In this paper, we propose the BERT-DAAT model for cross-domain sentiment analysis. Our purpose is to inject the target domain knowledge to BERT and encourage BERT to be domain-aware. Specifically, we conduct post-training and adversarial training. A novel domain-distinguish pre-training task is designed to distill the domain-specific features in a self-supervised. Experimental results on Amazon dataset demonstrate the effectiveness of our model, which remarkably outperforms state-of-the-art methods.
The proposed post-training procedure could also be applied to other domain adaptation scenarios such as named entity recognition, question answering, and reading comprehension. In the future, we would like to investigate the application of our theory in these domain adaptation tasks.
This work was supported in part by the National Key R&D Program of China 2018YFB1800502, in part by the National Natural Science Foundation of China under Grants 61671079 and 61771068, in part by the Beijing Municipal Natural Science Foundation under Grant 4182041, and in part by the Ministry of Education and China Mobile Joint Fund MCM20180101. This work was also supported by BUPT Excellent Ph.D. Students Foundation CX2020206.
Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. 2010. A theory of learning from different domains. Machine Learning.
John Blitzer, Mark Dredze, and Fernando Pereira. 2007a. Biographies, bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification. In ACL, 2007.
John Blitzer, Mark Dredze, and Fernando Pereira. 2007b. Biographies, bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification. In ACL 2007.
Xinyang Chen, Sinan Wang, Mingsheng Long, and Jianmin Wang. 2019. Transferability vs. discriminability: Batch spectral penalization for adversarial domain adaptation. In ICML 2019.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor S. Lempitsky. 2016. Domain-adversarial training of neural networks. J. Mach. Learn. Res.
Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In ACL, 2018.
Zheng Li, Ying Wei, Yu Zhang, and Qiang Yang. 2018. Hierarchical attention transfer network for cross-domain sentiment classification. In AAAI, 2018.
Zheng Li, Yu Zhang, Ying Wei, Yuxiang Wu, and Qiang Yang. 2017. End-to-end adversarial memory network for cross-domain sentiment classification. In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI 2017.
Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher. 2017. Learned in translation: Contextualized word vectors. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013.
Minlong Peng, Qi Zhang, Yu-Gang Jiang, and Xuanjing Huang. 2018. Cross-domain sentiment classification with target domain specific information. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In EMNLP 2014.
Claudio Persello and Lorenzo Bruzzone. 2016. Kernel-based domain-invariant feature selection in hyperspectral images for transfer learning. IEEE Trans. Geoscience and Remote Sensing.
Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018.
Xiaoye Qu, Zhikang Zou, Yu Cheng, Yang Yang, and Pan Zhou. 2019. Adversarial category alignment network for cross-domain sentiment classification. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding with unsupervised learning. In Technical report, OpenAI.
Jian Shen, Yanru Qu, Weinan Zhang, and Yong Yu. 2018. Wasserstein distance guided representation learning for domain adaptation. In AAAI, 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017..
Hu Xu, Bing Liu, Lei Shu, and Philip S. Yu. 2019. BERT post-training for review reading comprehension and aspect-based sentiment analysis. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019.
Jianfei Yu and Jing Jiang. 2016a. Learning sentence embeddings with auxiliary tasks for cross-domain sentiment classification. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016, pages 236-246.
Jianfei Yu and Jing Jiang. 2016b. Learning sentence embeddings with auxiliary tasks for cross-domain sentiment classification. In EMNLP, 2016.
Werner Zellinger, Thomas Grubinger, Edwin Lughofer, Thomas Natschlager, and Susanne Saminger-Platz. 2017. Central moment discrepancy (CMD) for domain-invariant representation learning. In ICLR, 2017.
Kai Zhang, Hefu Zhang, Qi Liu, Hongke Zhao, Hengshu Zhu, and Enhong Chen. 2019. Interactive attention transfer network for cross-domain sentiment classification. In AAAI 2019.
Li Zheng, Li Xin, Wei Ying, Bing Lidong, Zhang Yu, and Yang Qiang. 2019. Transferable end-to-end aspect-based sentiment analysis with selective adversarial learning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).
Yftah Ziser and Roi Reichart. 2018. Pivot based language modeling for improved neural domain adaptation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018.