SlowGuess/NAACL_2021_OCR huggingface dataset:
The review shows that most works extend the DSN framework to learn domain invariant representations in different scenarios (cf. Table 1, in the appendix). The original work from (Bousmalis et al., 2016) includes MMD divergence besides PAD, which is not adopted in subsequent works, possibly due to the reported poor performance. Most works require careful balancing between multiple objective functions (Han and Eisenstein, 2019), which can affect the stability of training. The stability of training can be improved by selecting appropriate divergence measures like CMD (Zellinger et al., 2017) and Wasserstein Distance (Arjovsky et al., 2017). We believe additional future works will adopt such measures.
3.3 Decisions in the Wild
Models can perform poorly when they are deployed in the real world. The performance degrades due to the difference in distribution between training and test data. Such performance degradation can be alleviated by large-scale annotation in the new domain. However, annotation is expensive, and — given thousands of domains — quickly becomes infeasible. Predicting the performance in a new domain, where there is no labelled data, is thus important. Much recent work provides theory (Rosenfeld et al., 2020; Chuang et al., 2020; Steinhardt and Liang, 2016). As models are put into production in the real world, this application becomes practically important as well. Empirically, NLP considers the divergence between the source and the target domain to predict performance drops.
Simple measures based on word level features have been used to predict the performance of a
machine learning model in new domains. Information theoretic measures like Renyi-Div and KL-Div has been used for predicting performance drops in POS (Van Asch and Daelemans, 2010) and Cross-Entropy based measure has been used for dependency parsing (Ravi et al., 2008). Prediction of performance can also be useful for machine translation where obtaining parallel data is hard. Based on distance between languages, (Xia et al., 2020) predict performance of the model on new languages for MT, among other tasks. Such performance prediction models have also been done in the past for SMT (Birch et al., 2008; Specia et al., 2013). However, Ponomareva and Thelwall (2012) argue that predicting drops in performance is more appropriate compared to raw performance. They find that JS-Div effective for predicting performance drop of Sentiment Analysis systems.
Only recently, predicting model failures in practical deployments from an empirical viewpoint has regained attention. Elsahar and Galle (2019) find the efficacy of higher-order measures to predict the drop in performance for POS and SA and do not rely on hand crafted measures as in previous works. However, analysing performance drops using CWR is still lacking. We tackle this in the next section.
A practical use case of domain divergences is to predict the performance drop of a model applied to a new domain. We ask how relevant are traditional measures over word distributions compared to higher-order measures like CMD and MMD over contextual word representations like BERT, Elmo, DistilBERT (Devlin et al., 2019; Peters et al., 2018; Sanh et al., 2019)? We perform an empirical study to assess their suitability to predict performance drops for three important NLP tasks: POS, NER, and SA leaving machine translation to future work.
Performance difference between the source and the target domain depends on the divergence between their feature distributions (Ben-David et al., 2010). We assume a co-variate shift, as in (Ganin et al., 2016), where the marginal distribution over features change, but the conditional label distributions does not - i.e.,  $P_{\mathcal{D}_s}(y|x) = P_{\mathcal{D}_T}(y|x)$ $\mathcal{P}_{\mathcal{D}_s}(x) \neq P_{\mathcal{D}_T}(x)$ . Although difference in conditional label distribution can increase the  $\mathcal{H}$ -Divergence measure (Wisniewski and Yvon, 2019), it requires labels in the target domain for assessment. In this work, we assume no labelled data in
the target domain, to best mimic realistic settings.
Datasets: For POS, we select 5 different corpora from the English Word Tree Bank of Universal Dependency corpus (Nivre et al., 2016) $^{1}$  and also include the GUM, Lines, and ParTUT datasets. We follow Elsahar and Gallé (2019) and consider these as 8 domains. For NER, we consider CONLL 2003 (Tjong Kim Sang and De Meulder, 2003), Emerging and Rare Entity Recognition Twitter (Derczynski et al., 2017) and all 6 categories in OntoNotes v5 (Hovy et al., 2006) $^{2}$ , resulting in 8 domains. For SA, we follow Guo et al. (2020), selecting the same 5 categories $^{3}$  for experiments (Liu et al., 2017).
Divergence Measures: We consider 12 divergences. For Cos, we follow the instance based calculation (Ruder et al., 2017). For MMD, Wasserstein and CORAL, we randomly sample 1000 sentences and average the results over 3 runs. For MMD, we experiment with different kernels (cf. Appendix A) and use default values of  $\sigma$  from the GeomLoss package (Feydy et al., 2019). For TVO, KL-div, JS-div, Renyi-div, based on word frequency distribution we remove stop-words and consider the top 10k frequent words across domains to build our vocabulary (Ruder et al., 2017; Gururangan et al., 2020). We use  $\alpha = 0.99$  for Renyi as found effective by Plank and van Noord (2011). We do not choose CE as it is mainly used in MT and ineffective for classification and structured prediction (Ruder et al., 2017).
Model Architecture: For all our experiments, unless otherwise mentioned, we use the pre-trained DistilBERT (Sanh et al., 2019) model. It has competitive performance to BERT, but has faster inference times and lower resource requirements. For every text segment, we obtain the activations from the final layer and average-pool the representations. We train the models on the source domain training split and test the best model — picked from validation set grid search — on the test dataset of the same and other domains (cf. Appendix C).
For POS and NER, we follow the original BERT model where a linear layer is added and a prediction is made for every token. If the token is split into
Yahoo! Answers, Email, NewsGroups, Reviews and Weblogs.
2 Broadcast News (BN), Broadcast Conversation (BC), Magazine (MZ), Telephone Conversation (TC) and Web (WB).
3 Apparel, Baby, Books, Camera and MR.
(a) (POS)-MMD-Gaussian
(b) (NER)-MMD-RQ
Figure 2: t-SNE plots for select measures. The complete set of diagrams are available in Appendix D.
(c) (SA)-JS-Div
multiple tokens due to Byte Pair Encoding, the label for the first token is predicted. For SA and domain discriminators, we pool the representation from the last layer of DistilBERT and add a linear layer for prediction (Appendix B).
4.2 Are traditional measures still relevant?
For POS, the PAD measure has the best correlation with performance drop (cf. Table 2). Information-theoretic measures over word frequency distributions, such as JS-div, KL-div, and TVO, which have been prevalent for data selection and performance drop use cases (cf. Table 1) are comparable to PAD. Plank et al. (2014) claim that the errors in POS are dictated by out of vocabulary words. Our findings validate their claim, as we find strong correlation between POS performance drop and word probability distribution measures For NER, MMD-RQ provides the best correlation of 0.495. CORAL — a higher-order measure — and JS-div are comparable. For SA, Renyi-div and other information-theoretic measures provide considerably better correlation compared to higher-order measures. Cos is a widely-used measure across applications, however it did not provide significant correlation for either task. TVO is used for selecting pretraining data for NER (Dai et al., 2019) and as a measure to gauge the benefits of fine-tuning pre-trained LMs on domain-specific data (Gururangan et al., 2020). Although TVO does not capture the nuances of domain divergences, it has strong, reliable correlations for performance drops. PAD has been suggested for data selection in SA by Ruder and Plank (2017) and for predicting drop in performance by Elsahar and Galle (2019). Our analysis confirms that PAD provides good correlations across POS, NER, and SA.
We find no single measure to be superior across all tasks. However, information theoretic measures consistently provide good correlations. Currently,
when contextual word representations dictate results in NLP, simple measures based on frequency distributions are strong baselines for predicting performance drop. Although higher-order measures do not always provide the best correlation, they are differentiable, thus suited for end-to-end training of domain-invariant representations.
Why are some divergence measures better at predicting drops in performance? The one-dataset-one-domain is a key assumption in such works. However, many works have questioned this assumption (Plank and van Noord, 2011). Multiple domains may exist within the same domain (Webber, 2009) and two different datasets may not necessarily be considered different domains (Irvine et al., 2013). Recently Aharoni and Goldberg (2020) show that BERT representations reveal their underlying domains. They qualitatively show that a few text segments from a dataset actually belong to another domain. However the degree to which the samples belong to different domains is unclear.
We first test the assumption that different datasets are different domains using Silhouette scores (Rousseeuw, 1987) which quantify the separability of clusters. We initially assume that a dataset is in its own domain. A positive score shows that datasets can be considered as well-separated domains; a negative score shows that most of the points within a dataset can be assigned to a nearby domain; and 0 signifies overlapping domains. We calculate Silhouette scores and t-SNE plots (Maaten and Hinton, 2008) for different divergence measures. Refer to the plots (Figures 3a to 3c) and calculation details in Appendix D.
Almost all the measures across different tasks have negative values close to 0 (Table 2, (r)).
- For POS, CORAL, Wasserstein and Cos strongly indicate that text within a dataset belongs to other
Measure	Correlations	Silhouette Coefficients
-	POS	NER	SA	POS	NER	SA
Cos	0.018	0.223	-0.012	-1.78 × 10-1	-2.49 × 10-1	-2.01 × 10-1
KL-Div	0.394	0.384	0.715	-	-	-
JS-Div	0.407	0.484	0.709	-8.50 × 10-2	-6.40 × 10-2	+2.04 × 10-2
Renyi-Div	0.392	0.382	0.716	-	-	-
PAD	0.477	0.426	0.538	-	-	-
Wasserstein	0.378	0.463	0.448	-2.11 × 10-1	-2.36 × 10-1	-1.70 × 10-1
MMD-RQ	0.248	0.495	0.614	-4.11 × 10-2	-3.04 × 10-2	-1.70 × 10-2
MMD-Gaussian	0.402	0.221	0.543	+4.25 × 10-5	+2.37 × 10-3	-8.42 × 10-5
MMD-Energy	0.244	0.447	0.521	-9.84 × 10-2	-1.14 × 10-1	-8.48 × 10-2
MMD-Laplacian	0.389	0.273	0.623	-1.67 × 10-3	+4.26 × 10-4	-1.08 × 10-3
CORAL	0.349	0.484	0.267	-2.34 × 10-1	-2.78 × 10-1	-1.41 × 10-1
TVO	-0.437	-0.457	-0.568	-	-	-
Table 2: (l): Correlation of performance drops with divergence measures. Measures with higher correlations are better indicators of performance drops. (r): Silhouette coefficients considering different divergence measures. We randomly sample 200 points for calculation and average the results over 5 runs. Only certain divergences which are metrics are allowed. The colours are from the taxonomy of divergence measures in Figure 1.
domains. However, for MMD-Gaussian the domains overlap (Figure 2a).
- For NER, MMD-Gaussian and MMD-Laplacian indicate that the clusters overlap while all other metrics have negative values.
- For SA, JS-Div has positive values compared to other measures, and as seen in Figure 2c, we can see a better notion of distinct clusters.
The Silhouette scores along with the t-SNE plots show that datasets are, in fact, not distinct domains. Considering data-driven methods for defining domains is needed (Aharoni and Goldberg, 2020).
If there are indeed separate domains, does it explain why some measures are better than the others? We see better notions of clusters for NER and sentiment analysis (cf. Figures 2b and 2c). We can expect the drop in performance to be indicative of these domain separations. Comparing the best correlations from Table 2, correlations for NER and sentiment analysis are higher compared with POS. For POS, there are no indicative domain clusters and the correlation between domain divergence and performance may be less; whereas for SA, both the t-SNE plot and the Silhouette scores for JS-Div (cf. Figure 2c) corroborate comparatively better separation. If datasets are indeed different domains, these divergence measures are reliable indicators of performance drops. If they are not, there might be other confounding factors (such as differences in label distribution) and one has to be cautious in using them.
Domain overlap also has consequences for data selection strategies. For example, Moore and Lewis (2010) select pseudo in-domain data from source corpora (cf Section 3.1). As the Silhouette coefficients are negative and close to 0, many data points
in a dataset belong to nearby domains. Data selection strategies thus may be effective. If the Silhouette coefficients are more negative and if more points in the source aptly belong to the target domain, we should expect increased sampling from such source domains to yield additional performance benefits in the target domain.
We survey domain adaptation works, focusing on divergence measures and their usage for data selection, learning domain-invariant representations, and making decisions in the wild. We synthesised the divergence measures into a taxonomy of information theoretic, geometric and higher-order measures. While traditional measures are common for data selection and making decisions in the wild, higher-order measures are prevalent in learning representations. Based on our correlation experiments, silhouette scores, and t-SNE plots, we make the following recommendations:
- PAD is a reliable indicator of performance drop. It is best used when there are sufficient examples to train a domain discriminator.
- JS-Div is symmetric and a formal metric. It is related to PAD, easy to compute, and serves as a strong baseline.
- While Cosine is popular, it is an unreliable indicator of performance drop.
- One-dataset-is-not-one-domain. Instead, cluster representations and define appropriate domains.
We would also like to acknowledge the support of the NExT research grant funds, supported by the National Research Foundation, Prime Ministers Office, Singapore under its IRC@ SG Funding Initiative, and to gratefully acknowledge the support of NVIDIA Corporation with the donation of the GeForce GTX Titan XGPU used in this research.
Charu C. Aggarwal, Alexander Hinneburg, and Daniel A. Keim. 2001. On the surprising behavior of distance metrics in high dimensional spaces. In Proceedings of the 8th International Conference on Database Theory, ICDT '01, page 420-434, Berlin, Heidelberg. Springer-Verlag.
Roee Aharoni and Yoav Goldberg. 2020. Unsupervised domain clusters in pretrained language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7747-7763, Online. Association for Computational Linguistics.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. 2017. Wasserstein generative adversarial networks. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 214-223, International Convention Centre, Sydney, Australia. PMLR.
Amittai Axelrod, Xiaodong He, and Jianfeng Gao. 2011. Domain adaptation via pseudo in-domain data selection. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 355-362, Edinburgh, Scotland, UK. Association for Computational Linguistics.
Eyal Ben-David, Carmel Rabinovitz, and Roi Reichart. 2020. Perl: Pivot-based domain adaptation for pretrained deep contextualized embedding models.
Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. 2010. A theory of learning from different domains. *Machine learning*, 79(1-2):151-175.
Shai Ben-David, John Blitzer, Koby Crammer, and Fernando Pereira. 2007. Analysis of representations for domain adaptation. In B. Schölkopf, J. C. Platt, and T. Hoffman, editors, Advances in Neural Information Processing Systems 19, pages 137-144. MIT Press.
Douglas Biber and Susan Conrad. 2009. Register, Genre, and Style. Cambridge Textbooks in Linguistics. Cambridge University Press.
Alexandra Birch, Miles Osborne, and Philipp Koehn. 2008. Predicting success in machine translation.
In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 745-754, Honolulu, Hawaii. Association for Computational Linguistics.
John Blitzer, Ryan McDonald, and Fernando Pereira. 2006. Domain adaptation with structural correspondence learning. In Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing, pages 120-128, Sydney, Australia. Association for Computational Linguistics.
Konstantinos Bousmalis, George Trigeorgis, Nathan Silberman, Dilip Krishnan, and Dumitru Erhan. 2016. Domain separation networks. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems 29, pages 343-351. Curran Associates, Inc.
Denny Britz, Quoc Le, and Reid Pryzant. 2017. *Effective domain mixing for neural machine translation.* In Proceedings of the Second Conference on Machine Translation, pages 118-126, Copenhagen, Denmark. Association for Computational Linguistics.
Pengfei Cao, Yubo Chen, Kang Liu, Jun Zhao, and Shengping Liu. 2018. Adversarial transfer learning for Chinese named entity recognition with self-attention mechanism. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 182–192, Brussels, Belgium. Association for Computational Linguistics.
Marine Carpuat, Yogarshi Vyas, and Xing Niu. 2017. Detecting cross-lingual semantic divergence for neural machine translation. In Proceedings of the First Workshop on Neural Machine Translation, pages 69–79, Vancouver. Association for Computational Linguistics.
Rich Caruana. 1997. Multitask learning. Mach. Learn., 28(1):41-75.
Boxing Chen, Colin Cherry, George Foster, and Samuel Larkin. 2017. Cost weighting for neural machine translation domain adaptation. In Proceedings of the First Workshop on Neural Machine Translation, pages 40-46, Vancouver. Association for Computational Linguistics.
Boxing Chen and Fei Huang. 2016. Semi-supervised convolutional networks for translation adaptation with tiny amount of in-domain data. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pages 314-323, Berlin, Germany. Association for Computational Linguistics.
Xilun Chen and Claire Cardie. 2018. Multinomial adversarial networks for multi-domain text classification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1226-1240,
New Orleans, Louisiana. Association for Computational Linguistics.
Xilun Chen, Yu Sun, Ben Athiwaratkun, Claire Cardie, and Kilian Weinberger. 2018. Adversarial deep averaging networks for cross-lingual sentiment classification. Transactions of the Association for Computational Linguistics, 6:557-570.
Chenhui Chu and Rui Wang. 2018. A survey of domain adaptation for neural machine translation. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1304-1319, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Ching-Yao Chuang, Antonio Torralba, and Stefanie Jegelka. 2020. Estimating generalization under distribution shifts via domain-invariant representations. CoRR, abs/2007.03511.
I. Csiszár. 1972. A class of measures of informativity of observation channels. Periodica Mathematica Hungarica, 2(1):191-213.
Xiang Dai, Sarvaz Karimi, Ben Hachey, and Cecile Paris. 2019. Using similarity measures to select pretraining data for NER. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1460-1470, Minneapolis, Minnesota. Association for Computational Linguistics.
Leon Derczynski, Eric Nichols, Marieke van Erp, and Nut Limsopatham. 2017. Results of the WNUT2017 shared task on novel and emerging entity recognition. In Proceedings of the 3rd Workshop on Noisy User-generated Text, pages 140-147, Copenhagen, Denmark. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
X. Ding, Q. Shi, B. Cai, T. Liu, Y. Zhao, and Q. Ye. 2019. Learning multi-domain adversarial neural networks for text classification. IEEE Access, 7:40323-40332.
Kevin Duh, Graham Neubig, Katsuhito Sudoh, and Hajime Tsukada. 2013. Adaptation data selection using neural language models: Experiments in machine translation. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 678-683, Sofia, Bulgaria. Association for Computational Linguistics.
Hady Elsahar and Matthias Galle. 2019. To annotate or not? predicting performance drop under domain shift. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2163-2173, Hong Kong, China. Association for Computational Linguistics.
Jean Feydy, Thibault Séjourné, François-Xavier Vialard, Shun-ichi Amari, Alain Trouve, and Gabriel Peyré. 2019. Interpolating between optimal transport and mmd using sinkhorn divergences. In *The 22nd International Conference on Artificial Intelligence and Statistics*, pages 2681–2690.
Yaroslav Ganin and Victor Lempitsky. 2015. Unsupervised domain adaptation by backpropagation. In Proceedings of the 32nd International Conference on International Conference on Machine Learning - Volume 37, ICML'15, page 1180-1189. JMLR.org.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario March, and Victor Lempitsky. 2016. Domain-adversarial training of neural networks. Journal of Machine Learning Research, 17(59):1-35.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Lavi-olette, Mario Marchand, and Victor S. Lempitsky. 2015. Domain-adversarial training of neural networks. Journal of Machine Learning Research, 17:59:1-59:35.
Deepanway Ghosal, Devamanyu Hazarika, Abhinaba Roy, Navonil Majumder, Rada Mihalcea, and Soujanya Poria. 2020. Kingdom: Knowledge-guided domain adaptation for sentiment analysis. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 3198-3210. Association for Computational Linguistics.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 27, pages 2672-2680. Curran Associates, Inc.
Arthur Gretton, Karsten Borgwardt, Malte Rasch, Bernhard Scholkopf, and Alex J. Smola. 2007. A kernel method for the two-sample-problem. In B. Scholkopf, J. C. Platt, and T. Hoffman, editors, Advances in Neural Information Processing Systems 19, pages 513-520. MIT Press.
Shuhao Gu, Yang Feng, and Qun Liu. 2019. Improving domain adaptation translation with domain invariant and specific information. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human