SlowGuess/NAACL_2022_OCR huggingface dataset:
BERT base: Fine-tune BERT on the source, without using target unlabeled data, then directly evaluate on the target labeled data. KL: Use symmetric KL-divergence loss of embedded instances between the source and target domains (Zhuang et al., 2015). MMD: Maximum Mean Discrepancy loss (Gretton et al., 2012) measures the distance based on the notion of embedding probabilities in a reproducing kernel Hilbert space. We implement a gaussian kernel which is a common choice. DANN: The adaptation rate is  $\lambda = \frac{2}{1 + \exp(-\gamma p)} - 1$ ,  $p = \frac{t}{T}$ , where  $t$  and  $T$  are the number of current training steps and total steps.  $\gamma$  requires careful tuning within [0.05, 0.1, 0.15, 0.2]. backtrans+CL: To investigate the effectiveness of domain puzzles, we implement Back translation (Sennrich et al., 2016; Edunov et al., 2018), which is one of the widely used data augmentation techniques. We utilize en-de translation model pretrained on WMT19 and released in fairseq (Ott et al., 2019). The model is trained with contrastive loss on the source and target domain respectively. mask+CL: We mask domain specific tokens to make the augmentation become domain-agnostic. Since information extraction is not our focus, we identify domain specific tokens via a simple frequency-ratio method (Li et al., 2018; Wu et al., 2019a):  $s(u,d) = \frac{\text{count}(u,\mathcal{D}^d) + \lambda}{\sum_{d'\in\mathcal{D},d'\neq d}\text{count}(u,\mathcal{D}^{d'})+\lambda}$ , where count(u,  $\mathcal{D}^d$ ) represents the number of times a token  $u$  appears in domain  $d$ . Smoothing  $\lambda$  is set to 1. When  $s(u,d)$  is larger than 5, we mark a token  $u$  as a domain specific token. Through counting, the number of masked tokens accounted for 0.06 of the total length. mask: To investigate the effectiveness of contrastive loss, after masking domain-specific tokens, we further let the model train with augmented data without contrastive loss. R-PERL (Ben-David et al., 2020) is a pivot-based method, and DAAT (Du et al., 2020) combines DANN and post-training. All the methods in Tabel 1 and Table 2 are implemented based on BERT model for fair comparisons.
<sup>3</sup>https://github.com/ruidan/DAS
4The original crawled reviews contain star ratings (1 to 5 stars). Comparing with Amazon Benchmark which discards the neutral class, this dataset labels them with rating  $< 3, > 3$ ,  $= 3$  as negative, positive, and neutral respectively.
Method	E→BK	BT→BK	M→BK	BK→E	BT→E	M→E	BK→BT	E→BT	M→BT	BK→M	E→M	BT→M	Ave.
BERT base	64.34*1.3	65.87*1.9	64.12*1.7	52.25*1.8	66.01*1.0	59.49*1.1	55.01*1.7	66.02*1.3	56.50*2.1	55.66*1.5	61.34*0.8	62.34*1.2	60.74*1.5
KL	64.77*0.8	67.03*0.7	65.13*1.0	56.96*1.2	65.43*1.5	60.30*0.7	58.47*0.9	66.22*0.5	58.80*1.3	57.53*0.7	60.02*1.4	64.41*1.1	62.09*1.0
MMD	65.41*0.7	68.54*1.1	64.77*1.3	58.84*1.6	66.39*1.5	59.29*1.9	60.75*1.5	66.50*0.9	57.42*1.3	59.37*1.4	59.47*1.0	65.03*1.6	62.65*1.3
DANN	66.43*2.4	67.74*2.9	65.64*3.7	54.31*4.1	65.93*3.0	59.91*1.9	58.11*3.5	67.19*0.5	56.00*2.7	54.02*2.9	60.36*3.1	63.09*2.7	61.56*2.8
back-trans+CL	67.16*1.4	68.31*0.8	67.78*1.8	59.55*1.1	66.51*0.3	60.13*1.4	60.92*0.7	68.54*0.8	60.44*0.9	60.21*1.1	61.73*1.3	64.29*0.7	63.79*1.0
mask	65.17*1.1	66.91*1.4	65.59*0.9	52.74*1.5	67.14*1.9	60.48*0.7	54.76*1.6	66.50*1.4	56.61*1.7	55.64*1.6	62.17*0.9	63.09*1.0	61.40*1.3
mask+CL	68.80*1.2	69.93*0.7	70.65*0.9	60.50*1.3	68.02*0.7	60.80*1.2	62.17*0.8	69.17*0.5	62.06*1.5	60.56*1.4	63.94*0.7	65.40*0.6	65.17*0.9
DCCL	70.33*0.3	70.92*0.6	71.11*0.7	62.36*0.7	68.41*0.2	62.11*0.6	64.13*1.4	69.33*0.4	65.40*0.8	64.67*1.7	64.67*1.0	66.70*1.4	66.68*0.8
Table 1: Accuracy  $(\%)$  results of Amazon review dataset. For example,  $\mathrm{E}\rightarrow \mathrm{BK}$  denotes training on Electronics (E) and adapting to Book (BK). All the results are reported with average and standard deviation in 5 runs. * indicates the DCCL improvements are significant with  $p < 0.05$ . There are four domains available in the dataset, therefore we have 12 adaptation groups of tasks.
Method	D→BK	K→BK	E→BK	BK→D	K→D	E→D	BK→K	D→K	E→K	BK→E	D→E	K→E	Ave.
R-PERL	85.6	83.0	83.9	87.8	85.6	84.8	90.2	90.4	91.2	87.2	89.3	91.2	87.5
DAAT	90.86	87.98	88.91	89.70	88.81	90.13	90.75	90.50	93.18	89.57	89.30	91.72	90.12
BERT base	89.76*0.3	88.31*0.4	88.09*0.9	89.27*0.5	87.68*0.8	88.41*1.2	88.27*0.4	87.67*0.5	92.160.7	87.18*0.4	86.91*0.7	91.220.8	88.740.6
KL	89.23*0.5	87.61*0.5	88.37*0.7	89.49*0.6	88.03*0.5	88.56*0.7	88.77*0.5	87.89*0.7	91.211.3	88.62*0.4	87.03*0.9	90.34*1.5	88.760.7
MMD	88.68*0.7	88.61*0.3	88.27*1.0	88.38*1.0	89.090.5	89.31*0.6	87.74*0.8	89.04*0.8	91.201.0	89.49*0.3	88.70*0.5	89.79*1.2	89.030.6
DANN	88.64*1.3	86.09*1.7	87.91*1.5	89.56*1.0	89.110.9	89.19*1.1	89.68*1.3	88.73*1.8	91.41*1.3	87.30*1.6	88.53*0.6	90.81*1.1	88.91*1.3
mask+CL	90.740.3	88.280.7	89.44*0.4	90.310.2	88.74*0.5	89.72*0.8	90.40*0.4	89.57*0.4	92.490.4	89.91*0.3	89.650.4	91.190.6	90.080.5
DCCL	91.170.3	88.530.4	89.700.5	90.030.4	89.530.3	90.490.5	91.050.5	90.780.4	92.540.5	90.420.3	89.550.4	91.930.5	90.480.4
Table 2: Accuracy (%) results of Amazon Benchmark. Results of R-PERL and DAAT are taken from Ben-David et al. (2020) and Du et al. (2020) respectively.
Will contrastive learning necessarily help the Unsupervised Domain Adaptation?
As discussed earlier, when performing contrastive learning on source labeled examples and target unlabeled examples respectively, it learns a uniformly distribute representation and helps promote better discriminative features for both domains. However, for some adaptation tasks in Table 1, for example,  $\mathrm{BT} \rightarrow \mathrm{E}$ ,  $\mathrm{M} \rightarrow \mathrm{E}$ , and  $\mathrm{E} \rightarrow \mathrm{M}$ , back-trans+CL shows that contrastive learning only gains marginal benefit. When masking domain-specific tokens and pulling original sentence representations to those domain puzzles, the effect of contrastive learning becomes more apparent (mask+CL with average score 65.17 compared with back-trans+CL 63.79 and mask 61.40 in Table.1). This finding helps to explain that choices of positive examples are critical and domain confused augmentations will further benefit adaptation.
DCCL outperforms baselines.
From Table 1 we can observe that the proposed DCCL outperforms all other methods with a large margin and  $p < 0.05$  using paired t-test, and  $5.94\%$  improvement over BERT base. From Table 2, we can also observe  $1.74\%$  improvement over BERT base, and DCCL also surpasses state-of-the-art
methods R-PERL (Ben-David et al., 2020) and DAAT (Du et al., 2020). We note that Amazon Benchmark dataset is quite easy, since it discards neutral instances, BERT base model has already achieved high scores on this dataset. Besides, we observe that the effect of distribution matching methods (KL and MMD) is limited on two datasets. The reason might be that pre-trained language models trained with massive and heterogeneous corpora already have strong generalization ability. Learning such cross-domain and instance-based matching will bring perplexity to language models and sacrifice task discrimination (Saito et al., 2017; Lee et al., 2019). On the contrary, the proposed DCCL retains such information in a self-supervised way. Furthermore, we notice that DANN is very unstable, besides adaptation rate  $\lambda$ , the model is also sensitive to other hyperparameters such as learning rate and training epochs because the performance on the target domain will keep decreasing when training with longer steps. Hence, it is difficult for the model to achieve the lowest error rates on both the source and target domains simultaneously. Compared to DANN, DCCL is much more stable and has lower standard deviations on most adaptation tasks.
Contrastive learning designs.
We explore different hyperparameter values for the
Figure 5: Hyperparameters for contrastive learning
proposed DCCL in  $\mathrm{E} \rightarrow \mathrm{BK}$ , as is shown in Fig. 5. We find that a temperature of 0.5 combined with the batch size 32 can achieve the best performance. We also notice that setting the temperature too high or too low can significantly affect adaptation performances, while a larger batch size usually brings a relatively smaller improvement.
We conduct an ablation study on each component in Eq. 10 to inspect the contribution of contrastive learning, as is shown in Table 3. We can see that every single component can render a better adaptation performance. In particular, the effectiveness of  $\mathcal{L}_{\mathrm{contrast}}$  is observable with  $3\sim 5$  performance gain compared to baselines (row 4 vs. row 2, and row 5 vs. row 3). When training curated domain puzzles as simple augmentations without contrastive loss, we can observe only a slight improvement (row 2 vs. row 1). This result demonstrates that the performance gain brought by DCCL does not come from data augmentation, and learning robustness against adversarial attacks will not largely help adaptation.
Ldomain	Lconsist	Lcontrast	E→BK	M→BT
64.65	55.85
✓	67.23	58.9
✓	✓	67.85	59.1
✓	✓	70.12	64.73
✓	✓	✓	70.21	64.87
Table 3: Ablation studies of DCCL on each component
5.1 Visualization
We perform visualizations for trained representations as illustrated in Fig. 6. When training with the source domain and then adapting to the target domain (BERT-base), we can observe a considerable
domain shift for BERT encoder on this amazon review dataset. Moreover, as mentioned before, continuing training DANN with larger epochs will substantially drop the score (from the highest point (DANN-best) to the lowest point (DANN-worst)). However, we can also see that DCCL mitigates domain shift but remains good sentiment discrimination on the target domain.
BERT-base acc 64.7
DANN-best acc 68.25
DANN-worst acc 33.75
Figure 6: t-SNE visualization for  $\mathrm{E}\rightarrow \mathrm{BK}$  task
$\bullet$  source negative
$\bullet$  target negative
$\bullet$  target neutral  $\bullet$  target positive
DCCL acc 70.15
$\odot$  source neutral  $\odot$  source positive
$\mathcal{A}$ -distance measures domain discrepancies (Ben-David et al., 2006; Ben-David et al., 2010), with the definition as  $d_{\mathcal{A}} = 2(1 - 2\epsilon)$ , where  $\epsilon$  is the domain classification error. To fairly compare with  $\mathcal{A}$ -distance of the baselines, we use linear SVM to calculate  $\epsilon$  following previous work (Saito et al., 2017; Du et al., 2020). We randomly select 2,000 instances for both source and target domain and split them with 1:1 for train and test for the SVM model. From Fig. 7, we can observe that DCCL can learn a good balance between sentiment classification and domain discrepancy, compared to DANN-best and DANN-worst.
Unsupervised Domain Adaptation
UDA in NLP has the following approaches: (1) Pivot-based methods use unlabeled data from both domains, trying to discover characteristics that are similar (Pan et al., 2010). Some recent works
Figure 7:  $\mathcal{A}$ -distance for  $\mathrm{E} \rightarrow \mathrm{BK}$  and  $\mathrm{M} \rightarrow \mathrm{BT}$  tasks
extend pivots with autoencoders and contextualized embeddings (Ziser and Reichart, 2017; Miller, 2019; Ben-David et al., 2020). (2) Pseudo-labeling leverages a trained classifier to predict labels on unlabeled examples, which are subsequently considered as gold labels (Yarowsky, 1995; Zhou and Li, 2005; McClosky et al., 2006). Recent works also combine this technique with pre-trained language models (Lim et al., 2020; Ye et al., 2020). (3) Data selection methods adopt domain similarity metrics to find the best match for each data and use curriculum learning for large pre-trained models (Ma et al., 2019; Aharoni and Goldberg, 2020). A recent method adopt distance-bandit (Guo et al., 2020) for similarity metric. (4) Domain Adversarial Neural Networks (Ganin et al., 2016). Some approaches leverage Wasserstein distance to stabilize adversarial training (Shen et al., 2018; Shah et al., 2018), and combine it with post-training which can produce better adversarial results (Du et al., 2020). (5) Adaptive pre-training is a more straightforward but effective method (Gururangan et al., 2020; Han and Eisenstein, 2019; Karouzos et al., 2021) by leveraging the objective of masked language model (MLM). A wide range of pre-training methods for domain adaptation (multi-phase, multi-task) are put forward (Han and Eisenstein, 2019; Karouzos et al., 2021).
Contrastive learning
CL has recently gained popularity as a reliable approach for unsupervised representation learning. It is generally acknowledged that a good representation should distinguish itself from other instances while identifying similar instances. For Computer Vision, there are approaches obtaining augmented images using transformations including cropping, rotation, etc. (Chen et al., 2020a; Khosla et al., 2020; He et al., 2020; Chen et al., 2020b). As for Natural Language Processing, many works study different label-preserving augmentations, such as back-translations, synonyms, adversaries, dropout,
and their combinations (Qu et al., 2021; Gao et al., 2021). In addition, many pre-trained language models trained with contrastive loss are also released. DeCLUTR (Giorgi et al., 2021) and CLEAR (Wu et al., 2020) jointly train the model with a contrastive objective and a masked language model setting. ConSERT (Yan et al., 2021) overcomes the collapse problem of BERT-derived sentence representations and makes them more suitable for downstream applications by using unlabeled texts.
In this work, we put forward a new concept, domain puzzles, which can be crafted through domain-specific token mask and domain-confused adversarial attacks. And we provide a more stable and effective solution to learn domain invariance for unsupervised domain adaptation. The proposed method DCCL surpasses baselines with a large margin by mitigating domain shift without losing discriminative power on the target domain. Moreover, the proposed framework can also be extended to other NLP tasks demanding adaptations, and we leave this for future work.
This work is supported by the 2020 Microsoft Research Asia collaborative research grant.
Roeaharoni and Yoav Goldberg. 2020. Unsupervised domain clusters in pretrained language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7747-7763, Online. Association for Computational Linguistics.
Eyal Ben-David, Carmel Rabinovitz, and Roi Reichart. 2020. PERL: Pivot-based domain adaptation for pre-trained deep contextualized embedding models. Transactions of the Association for Computational Linguistics, 8:504-521.
Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. 2010. A theory of learning from different domains. Machine learning, 79(1):151-175.
Shai Ben-David, John Blitzer, Koby Crammer, and Fernando Pereira. 2006. Analysis of representations for domain adaptation. In Advances in Neural Information Processing Systems 19, Proceedings of the Twentieth Annual Conference on Neural Information Processing Systems, Vancouver, British Columbia, Canada, December 4-7, 2006, pages 137-144. MIT Press.
John Blitzer, Mark Dredze, and Fernando Pereira. 2007. Biographies, Bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 440-447, Prague, Czech Republic. Association for Computational Linguistics.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. 2020a. A simple framework for contrastive learning of visual representations. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 1597-1607. PMLR.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. 2020b. Improved baselines with momentum contrastive learning. ArXiv preprint, abs/2003.04297.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Chunning Du, Haifeng Sun, Jingyu Wang, Qi Qi, and Jianxin Liao. 2020. Adversarial and domain-aware BERT for cross-domain sentiment analysis. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4019-4028, Online. Association for Computational Linguistics.
Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018. Understanding back-translation at scale. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 489-500, Brussels, Belgium. Association for Computational Linguistics.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Lavolette, Mario Marchand, and Victor Lempitsky. 2016. Domain-adversarial training of neural networks. The journal of machine learning research, 17(1):2096-2030.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. SimCSE: Simple contrastive learning of sentence embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6894-6910, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
John Giorgi, Osvald Nitski, Bo Wang, and Gary Bader. 2021. DeCLUTR: Deep contrastive learning for unsupervised textual representations. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International
Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 879-895, Online. Association for Computational Linguistics.
Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. 2015. Explaining and harnessing adversarial examples. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Scholkopf, and Alexander Smola. 2012. A kernel two-sample test. The Journal of Machine Learning Research, 13(1):723-773.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Ávila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. 2020. Bootstrap your own latent - A new approach to self-supervised learning. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
Han Guo, Ramakanth Pasunuru, and Mohit Bansal. 2020. Multi-source domain adaptation for text classification via distancenet-bandits. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 7830-7838. AAAI Press.
Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don't stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8342-8360, Online. Association for Computational Linguistics.
Xiaochuang Han and Jacob Eisenstein. 2019. Unsupervised domain adaptation of contextualized embeddings for sequence labeling. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4238-4248, Hong Kong, China. Association for Computational Linguistics.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. 2020. Momentum contrast for unsupervised visual representation learning. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 9726-9735. IEEE.
Ruidan He, Wee Sun Lee, Hwee Tou Ng, and Daniel Dahlmeier. 2018. Adaptive semi-supervised learning
for cross-domain sentiment classification. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3467-3476, Brussels, Belgium. Association for Computational Linguistics.
Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Tuo Zhao. 2020. SMART: Robust and efficient fine-tuning for pretrained natural language models through principled regularized optimization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2177-2190, Online. Association for Computational Linguistics.
Constantinos Karouzos, Georgios Paraskevopoulos, and Alexandros Potamianos. 2021. UDALM: Unsupervised domain adaptation through language modeling. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2579-2590, Online. Association for Computational Linguistics.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. 2020. Supervised contrastive learning. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
David Krueger, Ethan Caballero, Jorn-Henrik Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Le Priol, and Aaron C. Courville. 2021. Out-of-distribution generalization via risk extrapolation (rex). In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 5815-5826. PMLR.
Seungmin Lee, Dongwan Kim, Namil Kim, and SeongGyun Jeong. 2019. Drop to adapt: Learning discriminative features for unsupervised domain adaptation. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019, pages 91-100. IEEE.
Juncen Li, Robin Jia, He He, and Percy Liang. 2018. Delete, retrieve, generate: a simple approach to sentiment and style transfer. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1865-1874, New Orleans, Louisiana. Association for Computational Linguistics.
KyungTae Lim, Jay Yoon Lee, Jaime G. Carbonell, and Thierry Poibeau. 2020. Semi-supervised learning