SlowGuess/NAACL_2021_OCR huggingface dataset:
Abhinav Ramesh Kashyap†, Devamanyu Hazarika†, Min-Yen Kan†, Roger Zimmermann†
†National University of Singapore, Singapore
{abhinav, hazarika, kanmy, rogerz}@comp.nus.edu.sg
Domain divergence plays a significant role in estimating the performance of a model in new domains. While there is a significant literature on divergence measures, researchers find it hard to choose an appropriate divergence for a given NLP application. We address this shortcoming by both surveying the literature and through an empirical study. We develop a taxonomy of divergence measures consisting of three classes — Information-theoretic, Geometric, and Higher-order measures and identify the relationships between them. Further, to understand the common use-cases of these measures, we recognise three novel applications – 1) Data Selection, 2) Learning Representation, and 3) Decisions in the Wild – and use it to organise our literature. From this, we identify that Information-theoretic measures are prevalent for 1) and 3), and Higher-order measures are more common for 2). To further help researchers choose appropriate measures to predict drop in performance – an important aspect of Decisions in the Wild, we perform correlation analysis spanning 130 domain adaptation scenarios, 3 varied NLP tasks and 12 divergence measures identified from our survey. To calculate these divergences, we consider the current contextual word representations (CWR) and contrast with the older distributed representations. We find that traditional measures over word distributions still serve as strong baselines, while higher-order measures with CWR are effective.
Standard machine learning models do not perform well when tested on data from a different target domain. The performance in a target domain largely depends on the domain divergence (Ben-David et al., 2010) – a notion of distance between the two domains. Thus, efficiently measuring and reducing divergence is crucial for adapting models to the new domain — the topic of domain adaptation. Divergence also has practical applications in predicting
the performance drop of a model when adapted to new domains (Van Asch and Daelemans, 2010), and in choosing among alternate models (Xia et al., 2020).
Given its importance, researchers have invested much effort to define and measure domain divergence. Linguists use register variation to capture varieties in text – the difference between distributions of the prevalent features in two registers (Biber and Conrad, 2009). Other measures include probabilistic measures like  $\mathcal{H}$ -divergence (Ben-David et al., 2010), information theoretic measures like Jenssen-Shannon and Kullback-Leibler divergence (Plank and van Noord, 2011; Van Asch and Daelemans, 2010) and measures using higher-order moments of random variables like Maximum Mean Discrepancy (MMD) and Central Moment Discrepancy (CMD) (Gretton et al., 2007; Zellinger et al., 2017). The proliferation of divergence measures challenges researchers in choosing an appropriate measure for a given application.
To help guide best practices, we first comprehensively review the NLP literature on domain divergences. Unlike previous surveys, which focus on domain adaptation for specific tasks such as machine translation (Chu and Wang, 2018) and statistical (non-neural network) models (Jiang, 2007; Margolis, 2011), our work takes a different perspective. We study domain adaptation through the vehicle of domain divergence measures. First, we develop a taxonomy of divergence measures consisting of three groups: Information-Theoretic, Geometric, and Higher-Order measures. Further, to find the most common group used in NLP, we recognise three novel application areas of these divergences — Data Selection, Learning Representations, and Decisions in the Wild and organise the literature under them. We find that Information-Theoretic measures over word distributions are popular for Data Selection and Decisions in the wild, while Higher-order measures over continuous features
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1830-1849 June 6-11, 2021. ©2021 Association for Computational Linguistics
are frequent for Learning representations.
Domain divergence is a major predictor of performance in the target domain. A better domain divergence metric ideally predicts the corresponding performance drop of a model when applied to a target domain – a practical and important component of Decisions in the Wild. We further help researchers identify appropriate measures for predicting performance drops, through a correlation analysis over 130 domain adaptation scenarios and three standard NLP tasks: Part of Speech Tagging (POS), Named Entity Recognition (NER), and Sentiment Analysis and 12 divergence metrics from our literature review. While information-theoretic measures over traditional word distributions are popular in the literature, are higher-order measures calculated over modern contextual word representations better indicators of performance drop? We indeed find that higher-order measures are superior, but traditional measures are still reliable indicators of performance drop. The closest to our work is (Elsahar and Galle, 2019) who perform a correlation analysis. However, they do not compare against different divergence measures from the literature. Comparatively, we consider more tasks and divergence measures.
- We review the literature from the perspective of domain divergences and their use-cases in NLP.
- We aid researchers to select appropriate divergence measure that indicate performance-drops, an important application of divergence measures.
2 A Taxonomy of Divergence Measures
We devise a taxonomy for domain divergence measures, shown in Figure 1. It contains three main classes. Individual measures belong to a single class, where relationships can exist between measures from different classes. We provide detailed description of individual measures in Appendix A.
Geometric measures calculate the distance between two vectors in a metric space. As a divergence measure, they calculate the distance between features (tfidf, continuous representations, etc.) extracted from instances of different domains. The P-norm is a generic form of the distance between two vectors, where Manhattan  $(p = 1)$  and Euclidean distance  $(p = 2)$  are common. Cosine (Cos) uses the cosine of the angle between two vectors to measure similarity and 1-Cos measures distance. Geometric measures are easy to calculate, but are ineffective
in a high dimensional space as all distances appear the same (Aggarwal et al., 2001).
Information-theoretic measures captures the distance between probability distributions. For example, cross entropy over n-gram word distributions are extensively used in domain adaptation for machine translation.  $f$ -divergence (Csiszár, 1972) is a general family of divergences where  $f$  is a convex function. Different formulations of the  $f$  function lead to KL and JS divergence. Chen and Cardie (2018) show that reducing  $f$ -divergence measure is equivalent to reducing the PAD measures (see next section). Another special case of  $f$ -divergence is the family of  $\alpha$  divergences, where KL-Div is a special case of  $\alpha$  divergence. Renyi Divergence is a member of the  $\alpha$ -divergences and tends towards KL-Div as  $\alpha \rightarrow 1$  (Edge A); Often applied to optimal transport problems, Wasserstein distance measures the amount of work needed to convert one probability distribution to the other as distance and is used extensively for domain adaptation. KL-Div is also related to Cross Entropy (CE). In this paper, CE refers to measures based on entropy.
Higher-Order measures consider matching higher order moments of random variables or divergence in a projected space. Their properties are amenable to end-to-end learning based domain adaptation and recently have been extensively adopted. Maximum Mean Discrepancy (MMD) is one such measure which considers matching first order moments of variables in a Reproducible Kernel Hilbert Space. On the other hand, CORAL (Sun et al., 2017) considers second order moments and CMD (Zellinger et al., 2017) considers higher order moments. CORAL and CMD are desirable because they avoid computationally expensive kernel matrix computations. KL-Div can also be considered as matching the first-order moment (Zellinger et al., 2017); Edge  $\mathbb{B}$ . Proxy-A-Distance (PAD) measures the distance between source and target distributions via the error of a classifier in target domain samples as source domain samples (BenDavid et al., 2007).
A few other measures do not have ample support in the literature. These include information-theoretic measures such as Bhattacharya coefficient, higher-order measures like PAD* (Elsahar and Galle, 2019), Word Vector Variance (WVV), and Term Vocabulary Overlap (TVO) (Dai et al., 2019). Our taxonomy synthesises the diversity and
Figure 1: Taxonomy for divergence measures. i) Geometric measures the distance between vectors in a metric space ii) Information-theoretic measures the distance between probability distributions and iii) Higher-order measures the distance between distributions considering higher moments or the distance between representations or their projections in a nonlinear space. Edge  $\mathbb{A}$  indicates that Renyi divergence tends towards KL divergence as  $\alpha \to 1$  and Edge  $\mathbb{B}$  indicates KL-Div can be considered as matching first-order moment.
the prevalence of the divergence measures in NLP.
3 Applications of Divergence Measures
Our key observation of the literature is that there are three primary families of applications of divergences (cf. Table 1 in the appendix): (i) Data Selection: selects a subset of text from a source domain that shares similar characteristics as target domain. The selected subset is then used to learn a target domain model. (ii) Learning Representations: aligns source and target domain distributions and learn domain-invariant representations. (iii) Decisions in the Wild: helps practitioners predict the performance or drops in performance of a model in a new target domain.
We limit the scope of our survey to works that focus on divergence measures. We only consider unsupervised domain adaptation (UDA) – where there is no annotated data available in the target domain. It is more practical yet more challenging. For a complete treatment of neural networks and UDA in NLP, refer to (Ramponi and Plank, 2020). Also, we do not treat multilingual work. While cross-lingual transfer can be regarded as an extreme form of domain adaptation, measuring the distance between languages requires different divergence measures, outside our purview.
Divergence measures are used to select a subset of text from the source domain that shares similar characteristics to the target domain. Since the source domain has labelled data, the selected data serves as supervised data to train models in the target domain. We note that the literature pays closer attention to data selection for machine translation compared to other tasks. This can be attributed to its popularity in real-world applications and the difficulty of obtaining parallel sentences for every
pair of language.
Simple word-level and surface-level text features like word and n-gram frequency distributions and  $tf.idf$  weighted distributions have sufficient power to distinguish between text varieties and help in data selection. Geometric measures like cosine, used with word frequency distributions, are effective for selecting data in parsing and POS tagging (Plank and van Noord, 2011). Instead of considering distributions as (sparse) vectors, one can get a better sense of the distance between distributions using information-theoretic measures. Remus (2012) find JS-Div effective for sentiment analysis. While word-level features are useful to select supervised data for an end-task, they also can be used to select data to pre-train language-models subsequently used for NER. Dai et al. (2019) use Term Vocabulary Overlap for selecting data for pretraining language models. Geometric and Information-theoretic measures with word level distributions are inexpensive to calculate. However, the distributions are sparse and continuous word distributions help in learning denser representations.
Continuous or distributed representations of words, such as CBOw, Skip-gram (Mikolov et al., 2013) and GloVe (Pennington et al., 2014), address shortcomings of representing text as sparse, frequency-based probability distributions by transforming them into dense vectors learned from freeform text. A geometric measure (e.g., Word Vector Variance used with static word embeddings) is useful to select pre-training data for NER (Dai et al., 2019). Such selected data is found to be similar in tenor (the participants in a discourse, the relationships between them, etc.) to the source data. But static embeddings do not change according to the context of use. In contrast, contextual word representations (CWR) — mostly derived from neural networks (Devlin et al., 2019; Peters et al., 2018)
Paper	Task(s)	Information-Theoretic	Geometric	Higher-Order	Others
KL	JS	Renyi	CE	Wass.	Cos	P-Norm	PAD	CMD	MMD
DATA SELECTION
(Plank and van Noord, 2011)	Par, POS	✓	✓	✓	✓
(Dai et al., 2019)	NER	✓
(Ruder and Plank, 2017)	SA, NER, Par	✓	✓	✓	✓	✓	✓
(Ruder et al., 2017)	SA	✓	✓	✓	✓
(Remus, 2012)	SA	✓
(Li et al., 2007)	SMT	✓
(Zhao et al., 2004)	SMT	✓
(Yasuda et al., 2008)	SMT	✓
(Moore and Lewis, 2010)	SMT	✓
(Axelrod et al., 2011)	SMT	✓
(Duh et al., 2013)	SMT	✓
(Liu et al., 2014)	SMT	✓
(van der Wees et al., 2017)	NMT	✓
(Silva et al., 2018)	NMT	✓
(Aharoni and Goldberg, 2020)	NMT	✓
(Wang et al., 2017)	NMT	✓
(Carpuat et al., 2017)	NMT	✓
(Vyas et al., 2018)	NMT	✓
(Chen and Huang, 2016)	SMT	✓
(Chen et al., 2017)	NMT	✓
LEARNING REPRESENTATIONS
(Ganin et al., 2015)	SA	✓
(Kim et al., 2017)	Intent-clf	✓
(Liu et al., 2017)	SA	✓
(Li et al., 2018)	Lang-ID	✓
(Chen and Cardie, 2018)	SA	✓
(Zellinger et al., 2017)	SA	✓
(Peng et al., 2018)	SA	✓
(Wu and Guo, 2020)	SA	✓
(Ding et al., 2019)	Intent-Clf	✓
(Shah et al., 2018)	Question sim	✓	✓
(Zhu et al., 2019)	Emo-Regress	✓
(Gui et al., 2017)	POS	✓
(Zhou et al., 2019)	NER	✓
(Cao et al., 2018)	NER	✓
(Wang et al., 2018)	NER	✓
(Gu et al., 2019)	NMT	✓
(Britz et al., 2017)	NMT	✓
(Zeng et al., 2018)	NMT	✓
(Wang et al., 2019)	NMT	✓
DECISIONS IN THE WILD
(Ravi et al., 2008)	Parsing	✓	✓
(Elsahar and Gallé, 2019)	SA, POS	✓	✓	✓
(Ponomareva and Thelwall, 2012)	SA	✓	✓	✓	✓
(Van Asch and Daelemans, 2010)	POS	✓	✓	✓	✓
Table 1: Prior works using divergence measures for Data Selection, Learning Representations and Decisions in the Wild. Tasks can be Par: dependency parsing, POS: Parts of Speech tagging, NER: Named Entity Recognition, SA: Sentiment Analysis, SMT: Statistical and NMT: Neural Machine Translation, Intent-Clf: Intent classification, Lang-ID: Language identification, Emo-Regress: Emotional regression. Wass. denotes Wasserstein.
- capture contextual similarities between words in two domains. That is, the same word used in two domains in different contexts will have different embeddings. CWRs can be obtained from hidden representations (NMT) models. (McCann et al., 2017) have found such representations along with P-norm ef
fective for data selection in MT (Wang et al., 2017). Compared to representations from shallow NMT models, hidden representations of deep neural network language models (LM) like BERT have further improved data selection for NMT (Aharoni and Goldberg, 2020).
Divergences can be measured by comparing the
probabilities of a language model, in contrast to directly using its hidden representations. If a LM trained on the target domain assigns high probability to a sentence from the source domain, then the sentence should have similar characteristics to the target domain. Cross Entropy (CE) between probability distributions from LMs capture this notion of similarity between two domains. They have been extensively used for data selection in statistical machine translation (SMT) (Yasuda et al., 2008; Moore and Lewis, 2010; Axelrod et al., 2011; Duh et al., 2013; Liu et al., 2014). However, CE based methods for data selection are less effective for neural machine translation (van der Wees et al., 2017; Silva et al., 2018). Instead, van der Wees et al. (2017) come up with a dynamic subset selection where new subset is chosen every epoch during training. We note again the common refrain that sufficient amount of data should be available; here, to train good language models in the target domain.
Similar to language models, probabilistic scores from classifiers — which distinguish between samples from two domains — can aid data selection. The probabilities assigned by such classifiers in construing source domain text as target domain has been used as a divergence measures in machine translation (Chen and Huang, 2016). However, the classifiers require supervised target domain data which is not always available. As an alternative, Chen et al. (2017) train a classifier and selector in an alternating optimisation manner.
From this literature review, we find that distinct measures are effective for different NLP tasks. Ruder and Plank (2017) argue that owing to their varying task characteristics, different measures should apply. They show that learning a linear combination of measures is useful for NER, parsing and sentiment analysis. However, this is not always possible, especially in unsupervised domain adaptation where there is no supervised data in target domain. We observe that information theoretic measures and geometric measures based on frequency distributions and continuous representations are common for text and structured prediction tasks (cf. Table 1 in the appendix). The effectiveness of higher order measures for these tasks are yet to be ascertained.
Further, we find that for SMT data selection, variants of Cross Entropy (CE) measures are used extensively. However, the conclusions of van der Wees et al. (2017) are more measured regarding
the benefits of CE and related measures for NMT. Contextual word representations with cosine similarity has found some initial exploration for neural machine translation (NMT), with higher order measures yet to be explored for data selection in NMT.
3.2 Learning Representations
One way to achieve domain adaptation is to learn representations that are domain-invariant which are sufficiently powerful to perform well on an end task (Ganin et al., 2015; Ganin and Lempitsky, 2015). The theory of domain divergence (Ben-David et al., 2010) shows that the target domain error is bounded by the source domain error and domain divergence  $(\mathcal{H}$ -divergence) and reducing the domain divergence results in domain-invariant representation. The theory also proposes a practical alternative to measure  $\mathcal{H}$ -divergence called PAD. The idea is to learn a representations that confuses a domain discriminator sufficiently to make samples from two domains indistinguishable.
Ganin et al. (2015) operationalise PAD in a neural network named Domain Adversarial Neural Networks (DANN). The network employs a min-max game — between the representation learner and the domain discriminator — inspired by Generative Adversarial Networks (Goodfellow et al., 2014). The representation learner is not only trained to minimise a task loss on source domain, but also maximise a discriminator's loss, by reversing the gradients calculated for the discriminator. Note that this does not require any supervised data for target domain. In later work, Bousmalis et al. (2016) argue that domain-specific peculiarities are lost in a DANN, and propose Domain Separation Networks (DSN) to address this shortcoming. In DSN, both domain-specific and -invariant representations are captured in a shared-private network. DSN is flexible in its choice of divergence measures and they find PAD performs better than MMD. Here, we limit our review to works utilising divergence measures. We exclude feature-based UDA methods such as Structural Corresponding Learning (SCL) (Blitzer et al., 2006), Autoencoder-SCL and pivot based language models (Ziser and Reichart, 2017, 2018, 2019; Ben-David et al., 2020).
Obtaining domain invariant representations is desirable for many different NLP tasks, especially for sequence labelling where annotating large amounts of data is hard. They are typically used when there is a single source domain and a single target do
main — for sentiment analysis (Ganin et al., 2016), NER (Zhou et al., 2019), stance detection (Xu et al., 2019), machine translation (Britz et al., 2017; Zeng et al., 2018). The application of DANN and DSN to a variety of tasks are testament of their generality.
DANN and DSN are applied in other innovative situations. Text from two different periods of time can be considered as two different domains for intent classification (Kim et al., 2017). Gui et al. (2017) consider clean formal newswire data as source domain and noisy, colloquial, unlabeled Twitter data as the target domain and use adversarial learning to learn robust representations for POS. Commonsense knowledge graphs can help in learning domain-invariant representations as well. Ghosal et al. (2020) condition DANN with an external commonsense knowledge graph using graph convolutional neural networks for sentiment analysis. In contrast, Wang et al. (2018) use MMD outside the adversarial learning framework. They use MMD to learn to reduce the discrepancy between neural network representations belonging to two domains. Such concepts have been explored in computer vision (Tzeng et al., 2014).
While single source and target domains are common, complementary information available in multiple domains can help to improve performance in a target domain. This is especially helpful when there is no large-scale labelled data in any one domain, but where smaller amounts are available in several domains. DANN and DSN have been extended to such multi-source domain adaptation: for intent classification (Ding et al., 2019), sentiment analysis (Chen and Cardie, 2018; Li et al., 2018; Guo et al., 2018; Wright and Augenstein, 2020) and machine translation (Gu et al., 2019; Wang et al., 2019).
DANN and DSN can also help in multitask learning which considers two complementary tasks (Caruana, 1997). A key to multitask learning is to learn a shared representation that captures the common features of two tasks. However, such representations might still contain task-specific information. The shared-private model of DSN helps in disentangling such representations and has been used for sentiment analysis (Liu et al., 2017), Chinese NER and word segmentation (Cao et al., 2018). Also, although beyond the scope of our discussion here, DANN and DSN have been used to learn language-agnostic representations for text classification and structured prediction in multilingual
learning (Chen et al., 2018; Zou et al., 2018; Yasunaga et al., 2018).
Most works that adopt DANN and DSN framework reduce either the PAD or MMD divergence. However, reducing the divergences, combined with other auxiliary task specific loss functions, can result in training instabilities and vanishing gradients when the domain discriminator becomes increasingly accurate (Shen et al., 2018). Using other higher order measures can result in more stable learning. In this vein, CMD has been used for sentiment analysis (Zellinger et al., 2017; Peng et al., 2018), and Wasserstein distance has been used for duplicate question detection (Shah et al., 2018) and to learn domain-invariant attention distributions for emotional regression (Zhu et al., 2019).