Active Learning over Multiple Domains in Natural Language Tasks

1 INTRODUCTION . New natural language problems , outside the watershed of core NLP , are often strictly limited by a dearth of labeled data . While unlabeled data is frequently available , it is not always from the same source as the target distribution . This is particularly prevalent for tasks characterized by ( i ) significant distribution shift over time , ( ii ) personalization for user subgroups , or ( iii ) different collection mediums ( see examples in Section A ) . A widely-used solution to this problem is to bootstrap a larger training set using active learning ( AL ) : a method to decide which unlabeled training examples should be labeled on a fixed annotation budget ( Cohn et al. , 1996 ; Settles , 2012 ) . However , most active learning literature in NLP assumes the unlabeled source data is drawn from the same distribution as the target data ( Dor et al. , 2020 ) . This simplifying assumption avoids the frequent challenges faced by practitioners in multi-domain active learning . In this realistic setting , there are multiple sources of data ( i.e . domains ) to consider . In this case , it ’ s unclear whether to optimize for homogeneity or heterogeneity of selected examples . Secondly , is it more effective to allocate an example budget per domain , or treat examples as a single unlabeled pool ? Where active learning baselines traditionally select examples the model is least confident on ( Settles , 2009 ) , in this setting it could lead to distracting examples from very dissimilar distributions . In this work we empirically examine four separate families of methods ( uncertainty-based , HDivergence , reverse classification accuracy , and semantic similarity detection ) over several question answering and sentiment analysis datasets , following ( Lowell et al. , 2019 ; Elsahar & Gallé , 2019b ) , to provide actionable insights to practitioners facing this challenging variant of active learning for natural language . We address the following questions : 1 . What family of methods are effective for multi-domain active learning ? 2 . What properties of the example and domain selection yield strong results ? While previous work has investigated similar settings ( Saha et al. , 2011 ; Liu et al. , 2015 ; Zhao et al. , 2021 ; Kirsch et al. , 2021 ) we contribute , to our knowledge , the first rigorous formalization and broad survey of methods within NLP . We find that many families of techniques for active learning and domain shift detection fail to reliably beat random baselines in this challenging variant of active learning , but certain H-Divergence methods are consistently strong . Our analysis identifies stark dissimilarities of these methods ’ example selection , and suggests domain diversity is an important factor in achieving strong results . These results may serve as a guide to practitioners facing this problem , suggesting particular methods that are generally effective and properties of strategies that increase performance . 2 RELATED WORK . Active Learning in NLP Lowell et al . ( 2019 ) shows how inconsistent active learning methods are in NLP , even under regular conditions . However , Dor et al . ( 2020 ) ; Siddhant & Lipton ( 2018 ) survey active learning methods in NLP and find notable gains over random baselines . Kouw & Loog ( 2019 ) survey domain adaptation without target labels , similar to our setting , but for non-language tasks . We reference more active learning techniques in Section 4 . Domain Shift Detection Elsahar & Gallé ( 2019b ) attempt to predict accuracy drops due to domain shifts and Rabanser et al . ( 2018 ) surveys different domain shift detection methods . Arora et al . ( 2021 ) examine calibration and density estimation for textual OOD detection . Active Learning under Distribution Shift A few previous works investigated active learning under distribution shifts , though mainly in image classification , with single source and target domains . Kirsch et al . ( 2021 ) finds that BALD , which is often considered the state of the art for unshifted domain settings , can get stuck on irrelevant source domain or junk data . Zhao et al . ( 2021 ) investigates label shift , proposing a combination of predicted class balanced subsampling and importance weighting . Saha et al . ( 2011 ) , whose approach corrects joint distribution shift , relies on the covariate shift assumption . However , in practical settings , there may be general distributional shifts where neither the covariate shift nor label shift assumptions hold . Transfer Learning from Multiple Domains Attempts to better understand how to handle shifted domains for better generalization or target performance has motivated work in question answering ( Talmor & Berant , 2019 ; Fisch et al. , 2019 ; Longpre et al. , 2019 ; Kamath et al. , 2020 ) and classification tasks ( Ruder & Plank , 2018 ; Sheoran et al. , 2020 ) . Ruder & Plank ( 2017 ) show the benefits of both data similarity and diversity in transfer learning . Rücklé et al . ( 2020 ) find that sampling from a widevariety of source domains ( data scale ) outperforms sampling similar domains in question answering . He et al . ( 2021 ) investigate a version of multi-domain active learning where models are trained and evaluated on examples from all domains , focusing on robustness across domains . 3 MULTI-DOMAIN ACTIVE LEARNING . Suppose we have multiple domains D1 , D2 , ... , Dk.1 Let one of the k domains be the target set DT , and let the other k − 1 domains comprise the source set DS = ⋃ i 6=T Di . Given : . • Target : Small samples of labeled data points ( x , y ) from the target domain . DtrainT , D dev T , D test T ∼ DT .2 • Source : A large sample of unlabeled points ( x ) from the source domains . DS = ⋃ i6=T Di Task : . 1 . Choose n samples from DS to label . DchosenS ⊂ DS , |DchosenS | = n , selected by argmaxx∈DS Af ( x ) where Af is an acquisition function : a policy to select unlabeled examples from DS for labeling . 2 . Train a model M on Dfinal−train , validating on DdevT . Dfinal−train = DtrainT ∪DchosenS 3 . Evaluate M on DtestT , giving score s. 1We define a domain as a dataset collected independently of the others . 2|DtrainT | = 2000 to simulate a small but reasonable quantity of labeled , in-domain training data for active learning scenarios . For Step 1 , the practitioner chooses n samples with the highest scores according to their acquisition function Af . M is fine-tuned on these n samples , then evaluated on DtestT to demonstrate Af ’ s ability to choose relevant out-of-distribution training examples . 4 METHODS . We identify four families of methods relevant to active learning over multiple shifted domains . Uncertainty methods are common in standard active learning for measuring example uncertainty or familiarity to a model ; H-Divergence techniques train classifiers for domain shift detection ; Semantic Similarity Detection finds data points similar to points in the target domain ; and Reverse Classification Accuracy approximates the benefit of training on a dataset . A limitation of our work is we do not cover all method families , such as domain adaptation , just those we consider most applicable . We derive ∼18 active learning variants , comprising the most prevalent and effective from prior work , and novel extensions/variants of existing paradigms for the multi-domain active learning setting ( see KNN , R̃CA and DAL-E ) . Furthermore , we split the families into two acquisition strategies : Single Pool Strategy and Domain Budget Allocation . Single Pool Strategy , comprising the first three families of methods , treats all examples as coming from one single unlabeled pool . Domain Budget Allocation , consisting of Reverse Classification Accuracy methods , simply allocate an example budget for each domain . We enumerate acquisition methods Af below . Each method produces a full ranking of examples in the source set DS . To rank examples , most acquisition methods train an acquisition model , MA , using the same model architecture as M . MA is trained on all samples from DtrainT , except for DAL and KNN , which split DtrainT into two equal segments , one for training MA and one for an internal model . Some methods have both ascending and descending orders of these rankings ( denoted by ↑ and ↓ respectively , in the method abbreviations ) , to test whether similar or distant examples are preferred in a multi-domain setting . Certain methods use vector representations of candidate examples . We benchmark with both taskagnostic and task-specific encoders . The task-agnostic embeddings are taken from the last layer ’ s CLS token in Reimers & Gurevych ( 2019 ) ’ s sentence encoder ( Appendix for details ) . The task-specific embeddings are taken from the last layer ’ s CLS token in the trained model MA . The motivation of the task-specific variant is that each example ’ s representation will capture taskrelevant differences between examples while ignoring irrelevant differences.3 The versions of DAL and KNN methods that use task-specific vectors are denoted with “ ∗ ” in their abbreviation . Otherwise , they use task-agnostic vectors . 4.1 UNCERTAINTY METHODS . These methods measure the uncertainty of a trained model on a new example . Uncertainty can reflect either aleatoric uncertainty , due to ambiguity inherent in the example , or epistemic uncertainty , due to limitations of the model ( Kendall & Gal , 2017 ) . For the following methods , let Y be the set of all possible labels produced from the model M ( x ) and ly be the logit value for y ∈ Y . Confidence ( CONF ) A model ’ s confidence P ( y|x ) in its prediction y estimates the difficulty or unfamiliarity of an example ( Guo et al. , 2017 ; Elsahar & Gallé , 2019a ) . Entropy ( ENTR ) Entropy applies Shannon entropy ( Shannon , 1948 ) to the full distribution of class probabilities for each example , formalized as AENTR . ACONF ( x , MA ) = −max ( P ( y|x ) ) AENTR ( x , MA ) = − |Y |∑ i=1 P ( yi|x ) · logP ( yi|x ) Energy-based Out-of-Distribution Detection ( ENG ) Liu et al . ( 2020 ) use an energy-based score to distinguish between in- and out-distribution examples . They demonstrate this method is less susceptible to overconfidence issues of softmax approaches . 3For instance , consider in one domain every example is prefixed with “ Text : ” while the other is not — telling the difference is trivial , but the examples could be near-identical with respect to the task . Bayesian Active Learning by Disagreement ( BALD ) Gal & Ghahramani ( 2016 ) introduces estimating uncertainty by measuring prediction disagreement over multiple inference passes , each with a distinct dropout mask . BALD isolates epistemic uncertainty , as the model would theoretically produce stable predictions over inference passes given sufficient capacity . We conduct T = 20 forward passes on x. ŷt = argmaxiP ( yi|x ) t , representing the predicted class on the t-th model pass on x . Following ( Lowell et al. , 2019 ) , ties are broken by taking the mean label entropy over all T runs . AENG ( x , MA ) = − log ∑ y∈Y ely ABALD ( x , MA ) = 1− count ( modet∈T ( ŷt ) ) T 4.2 H-DIVERGENCE METHODS Ben-David et al . ( 2006 ; 2010 ) formalize the divergence between two domains as theH-Divergence , which they approximate as the difficulty for a discriminator to differentiate between the two.4 Discriminative Active Learning ( DAL ) applies this concept to the active learning setting ( Gissin & Shalev-Shwartz , 2019 ) . We explore variants of DAL , using an XGBoost decision tree ( Chen & Guestrin , 2016 ) as the discriminator model g.5 For the following methods , let Dtrain−BT be the 1k examples from D train T that were not used to train MA . Let E be an encoder function , which can be task-specific or agnostic as described above . We use samples both fromDtrain−BT andDS to train the discriminator . We assign samples origin labels l , which depend on the DAL variant . Samples from DS with discriminator predictions closest to 1 are selected for labeling . The acquisition scoring function for each DAL method and training set definition , respectively , are : ADAL ( x , g , E ) = g ( E ( x ) ) { ( E ( x ) , l ) | x ∈ Dtrain−BT ∪DS } Discriminative Active Learning — Target ( DAL-T ) DAL-T trains a discriminator g to distinguish between target examples in Dtrain−BT and out-of-distribution examples from DS . For DAL-T , l = 1Dtrain−BT ( x ) . Discriminative Active Learning — Error ( DAL-E ) DAL-E is a novel variant of DAL . DALE ’ s approach is to find examples that are similar to those in the target domain that MA misclassified . We partition Dtrain−BT further into erroneous samples D err T and correct samples D corr T , where Dtrain−BT = D err T ∪DcorrT . For DAL-E , l = 1DerrT ( x ) .

Summary: This paper surveys a broad range of techniques for active learning in the multi-domain setting applied to text - specifically, given a small labeled dataset from a target domain and a large amount of unlabeled data from a collection of source domains, how should examples be picked from the source domains to for labeling to maximise performance on the target domain. The main findings of the work are that selecting examples based on H-divergence measures perform much better than most active learning approaches. In-particular, they propose a method termed DAL-E where examples are selected from the source domain based on similarity to *misclassified* target domain examples, and show that it outperforms standard “disciminative active learning” on most of their experiments. From analysis, it is revealed that selecting from diverse domains helps, and their proposed DAL-E can help with avoiding bad examples to be selected.