Year: 2021

Abstract:
Studies of active learning traditionally assume the target and source data stem from a single domain. However, in realistic applications, practitioners often require active learning with multiple sources of out-of-distribution data, where it is unclear a priori which data sources will help or hurt the target domain. We survey a wide variety of techniques in active learning (AL), domain shift detection (DS), and multi-domain sampling to examine this challenging setting for question answering and sentiment analysis. We ask (1) what family of methods are effective for this task? And, (2) what properties of selected examples and domains achieve strong results? Among 18 acquisition functions from 4 families of methods, we findHDivergence methods, and particularly our proposed variant DAL-E, yield effective results, averaging 2-3% improvements over the random baseline. We also show the importance of a diverse allocation of domains, as well as room-for-improvement of existing methods on both domain and example selection. Our findings yield the first comprehensive analysis of both existing and novel methods for practitioners faced with multi-domain active learning for natural language tasks.

heading: 1 INTRODUCTION
text: New natural language problems, outside the watershed of core NLP, are often strictly limited by a dearth of labeled data. While unlabeled data is frequently available, it is not always from the same source as the target distribution. This is particularly prevalent for tasks characterized by (i) significant distribution shift over time, (ii) personalization for user subgroups, or (iii) different collection mediums (see examples in Section A).
A widely-used solution to this problem is to bootstrap a larger training set using active learning (AL): a method to decide which unlabeled training examples should be labeled on a fixed annotation budget (Cohn et al., 1996; Settles, 2012). However, most active learning literature in NLP assumes the unlabeled source data is drawn from the same distribution as the target data (Dor et al., 2020). This simplifying assumption avoids the frequent challenges faced by practitioners in multi-domain active learning. In this realistic setting, there are multiple sources of data (i.e. domains) to consider. In this case, it’s unclear whether to optimize for homogeneity or heterogeneity of selected examples. Secondly, is it more effective to allocate an example budget per domain, or treat examples as a single unlabeled pool? Where active learning baselines traditionally select examples the model is least confident on (Settles, 2009), in this setting it could lead to distracting examples from very dissimilar distributions.
In this work we empirically examine four separate families of methods (uncertainty-based, HDivergence, reverse classification accuracy, and semantic similarity detection) over several question answering and sentiment analysis datasets, following (Lowell et al., 2019; Elsahar & Gallé, 2019b), to provide actionable insights to practitioners facing this challenging variant of active learning for natural language. We address the following questions:
1. What family of methods are effective for multi-domain active learning? 2. What properties of the example and domain selection yield strong results?
While previous work has investigated similar settings (Saha et al., 2011; Liu et al., 2015; Zhao et al., 2021; Kirsch et al., 2021) we contribute, to our knowledge, the first rigorous formalization and broad survey of methods within NLP. We find that many families of techniques for active learning and
domain shift detection fail to reliably beat random baselines in this challenging variant of active learning, but certain H-Divergence methods are consistently strong. Our analysis identifies stark dissimilarities of these methods’ example selection, and suggests domain diversity is an important factor in achieving strong results. These results may serve as a guide to practitioners facing this problem, suggesting particular methods that are generally effective and properties of strategies that increase performance.

heading: 2 RELATED WORK
text: Active Learning in NLP Lowell et al. (2019) shows how inconsistent active learning methods are in NLP, even under regular conditions. However, Dor et al. (2020); Siddhant & Lipton (2018) survey active learning methods in NLP and find notable gains over random baselines. Kouw & Loog (2019) survey domain adaptation without target labels, similar to our setting, but for non-language tasks. We reference more active learning techniques in Section 4.
Domain Shift Detection Elsahar & Gallé (2019b) attempt to predict accuracy drops due to domain shifts and Rabanser et al. (2018) surveys different domain shift detection methods. Arora et al. (2021) examine calibration and density estimation for textual OOD detection.
Active Learning under Distribution Shift A few previous works investigated active learning under distribution shifts, though mainly in image classification, with single source and target domains. Kirsch et al. (2021) finds that BALD, which is often considered the state of the art for unshifted domain settings, can get stuck on irrelevant source domain or junk data. Zhao et al. (2021) investigates label shift, proposing a combination of predicted class balanced subsampling and importance weighting. Saha et al. (2011), whose approach corrects joint distribution shift, relies on the covariate shift assumption. However, in practical settings, there may be general distributional shifts where neither the covariate shift nor label shift assumptions hold.
Transfer Learning from Multiple Domains Attempts to better understand how to handle shifted domains for better generalization or target performance has motivated work in question answering (Talmor & Berant, 2019; Fisch et al., 2019; Longpre et al., 2019; Kamath et al., 2020) and classification tasks (Ruder & Plank, 2018; Sheoran et al., 2020). Ruder & Plank (2017) show the benefits of both data similarity and diversity in transfer learning. Rücklé et al. (2020) find that sampling from a widevariety of source domains (data scale) outperforms sampling similar domains in question answering. He et al. (2021) investigate a version of multi-domain active learning where models are trained and evaluated on examples from all domains, focusing on robustness across domains.

heading: 3 MULTI-DOMAIN ACTIVE LEARNING
text: Suppose we have multiple domains D1, D2, ..., Dk.1 Let one of the k domains be the target set DT , and let the other k − 1 domains comprise the source set DS =
⋃ i 6=T Di.

heading: Given:
text: • Target: Small samples of labeled data points (x, y) from the target domain. DtrainT , D dev T , D test T ∼ DT .2 • Source: A large sample of unlabeled points (x) from the source domains. DS =
⋃ i6=T Di

heading: Task:
text: 1. Choose n samples from DS to label. DchosenS ⊂ DS , |DchosenS | = n, selected by argmaxx∈DS Af (x) where Af is an acquisition function: a policy to select unlabeled examples from DS for labeling. 2. Train a model M on Dfinal−train, validating on DdevT . Dfinal−train = DtrainT ∪DchosenS
3. Evaluate M on DtestT , giving score s. 1We define a domain as a dataset collected independently of the others. 2|DtrainT | = 2000 to simulate a small but reasonable quantity of labeled, in-domain training data for active
learning scenarios.
For Step 1, the practitioner chooses n samples with the highest scores according to their acquisition function Af . M is fine-tuned on these n samples, then evaluated on DtestT to demonstrate Af ’s ability to choose relevant out-of-distribution training examples.

heading: 4 METHODS
text: We identify four families of methods relevant to active learning over multiple shifted domains. Uncertainty methods are common in standard active learning for measuring example uncertainty or familiarity to a model; H-Divergence techniques train classifiers for domain shift detection; Semantic Similarity Detection finds data points similar to points in the target domain; and Reverse Classification Accuracy approximates the benefit of training on a dataset. A limitation of our work is we do not cover all method families, such as domain adaptation, just those we consider most applicable. We derive ∼18 active learning variants, comprising the most prevalent and effective from prior work, and novel extensions/variants of existing paradigms for the multi-domain active learning setting (see KNN, R̃CA and DAL-E).
Furthermore, we split the families into two acquisition strategies: Single Pool Strategy and Domain Budget Allocation. Single Pool Strategy, comprising the first three families of methods, treats all examples as coming from one single unlabeled pool. Domain Budget Allocation, consisting of Reverse Classification Accuracy methods, simply allocate an example budget for each domain.
We enumerate acquisition methods Af below. Each method produces a full ranking of examples in the source set DS . To rank examples, most acquisition methods train an acquisition model, MA, using the same model architecture as M . MA is trained on all samples from DtrainT , except for DAL and KNN, which split DtrainT into two equal segments, one for training MA and one for an internal model. Some methods have both ascending and descending orders of these rankings (denoted by ↑ and ↓ respectively, in the method abbreviations), to test whether similar or distant examples are preferred in a multi-domain setting.
Certain methods use vector representations of candidate examples. We benchmark with both taskagnostic and task-specific encoders. The task-agnostic embeddings are taken from the last layer’s CLS token in Reimers & Gurevych (2019)’s sentence encoder (Appendix for details). The task-specific embeddings are taken from the last layer’s CLS token in the trained model MA.
The motivation of the task-specific variant is that each example’s representation will capture taskrelevant differences between examples while ignoring irrelevant differences.3 The versions of DAL and KNN methods that use task-specific vectors are denoted with “∗” in their abbreviation. Otherwise, they use task-agnostic vectors.

heading: 4.1 UNCERTAINTY METHODS
text: These methods measure the uncertainty of a trained model on a new example. Uncertainty can reflect either aleatoric uncertainty, due to ambiguity inherent in the example, or epistemic uncertainty, due to limitations of the model (Kendall & Gal, 2017). For the following methods, let Y be the set of all possible labels produced from the model M(x) and ly be the logit value for y ∈ Y .
Confidence (CONF) A model’s confidence P (y|x) in its prediction y estimates the difficulty or unfamiliarity of an example (Guo et al., 2017; Elsahar & Gallé, 2019a).
Entropy (ENTR) Entropy applies Shannon entropy (Shannon, 1948) to the full distribution of class probabilities for each example, formalized as AENTR.
ACONF(x,MA) = −max(P (y|x)) AENTR(x,MA) = − |Y |∑ i=1 P (yi|x) · logP (yi|x)
Energy-based Out-of-Distribution Detection (ENG) Liu et al. (2020) use an energy-based score to distinguish between in- and out-distribution examples. They demonstrate this method is less susceptible to overconfidence issues of softmax approaches.
3For instance, consider in one domain every example is prefixed with “Text:” while the other is not — telling the difference is trivial, but the examples could be near-identical with respect to the task.
Bayesian Active Learning by Disagreement (BALD) Gal & Ghahramani (2016) introduces estimating uncertainty by measuring prediction disagreement over multiple inference passes, each with a distinct dropout mask. BALD isolates epistemic uncertainty, as the model would theoretically produce stable predictions over inference passes given sufficient capacity. We conduct T = 20 forward passes on x. ŷt = argmaxiP (yi|x)t, representing the predicted class on the t-th model pass on x. Following (Lowell et al., 2019), ties are broken by taking the mean label entropy over all T runs.
AENG(x,MA) = − log ∑ y∈Y ely ABALD(x,MA) = 1− count(modet∈T (ŷt)) T
4.2 H-DIVERGENCE METHODS
Ben-David et al. (2006; 2010) formalize the divergence between two domains as theH-Divergence, which they approximate as the difficulty for a discriminator to differentiate between the two.4 Discriminative Active Learning (DAL) applies this concept to the active learning setting (Gissin & Shalev-Shwartz, 2019).
We explore variants of DAL, using an XGBoost decision tree (Chen & Guestrin, 2016) as the discriminator model g.5 For the following methods, let Dtrain−BT be the 1k examples from D train T that were not used to train MA. Let E be an encoder function, which can be task-specific or agnostic as described above. We use samples both fromDtrain−BT andDS to train the discriminator. We assign samples origin labels l, which depend on the DAL variant. Samples from DS with discriminator predictions closest to 1 are selected for labeling. The acquisition scoring function for each DAL method and training set definition, respectively, are:
ADAL(x, g, E) = g(E(x)) {(E(x), l) | x ∈ Dtrain−BT ∪DS}
Discriminative Active Learning — Target (DAL-T) DAL-T trains a discriminator g to distinguish between target examples in Dtrain−BT and out-of-distribution examples from DS . For DAL-T, l = 1Dtrain−BT (x).
Discriminative Active Learning — Error (DAL-E) DAL-E is a novel variant of DAL. DALE’s approach is to find examples that are similar to those in the target domain that MA misclassified. We partition Dtrain−BT further into erroneous samples D err T and correct samples D corr T , where Dtrain−BT = D err T ∪DcorrT . For DAL-E, l = 1DerrT (x).

heading: 4.3 REVERSE CLASSIFICATION ACCURACY
text: RCA Reverse Classification Accuracy (RCA) estimates how effective source set Di,i∈S is as a training data for target test set DT (Fan & Davidson, 2006; Elsahar & Gallé, 2019b). Without gold labels for Di we compute soft labels instead, using the BERT-Base MA trained on the small labeled set DtrainT . We then train a child model Mi on Di using these soft labels, and evaluate the child model on DdevT . RCA chooses examples randomly from whichever domain i produced the highest score si.
ARCA = 1D(arg maxi∈S si)(x) R̃CA : τi = si
sT − si , |Dchoseni | = τi∑ j sj
RCA-Smoothed (R̃CA) Standard RCA only selects examples from one domain Di. We develop a novel variant which samples from multiple domains, proportional to their relative performance on the target domain DdevT . RCA-smoothed (R̃CA) selects |Dchoseni | examples from source domain i, based on the relative difference between the performance si (of child model Mi trained on domain i with pseudo-labels from MA) on the target domain, and the performance sT of a model trained directly on the target domain DdevT . Since these strategies directly estimates model performance on the target domain resulting from training on each source domain, RCA and R̃CA are strong Domain Budget Allocation candidates.

heading: 4.4 NEAREST NEIGHBOUR / SEMANTIC SIMILARITY DETECTION (KNN)
text: Nearest neighbour methods (KNN) are used to find examples that are semantically similar. Using sentence encoders we can search the source set DS to select the top k nearest examples by cosine similarity to the target set. We represent the target set as the mean embedding of DtrainT . For question answering, where an example contains two sentences (the query and context), we refer to KNN-Q where we only encode the query text, KNN-C where we only encode the context text, or KNN-QC where we encode both concatenated together. The acquisition scoring function per example, uses either a task-specific or task-agnostic encoder E:
AKNN(x,E) = CosSim(E(x),Mean(E(DtrainT ))

heading: 5 EXPERIMENTS
text: Experiments are conducted on two common NLP tasks: question answering (QA) and sentiment analysis (SA), each with several available domains.
Question Answering We employ 6 diverse QA datasets from the MRQA 2019 workshop (Fisch et al., 2019), shown in Table 1 (left).6 We sample 60k examples from each dataset for training, 5k for validation, and 5k for testing. Questions and contexts are collected with varying procedures and sources, representing a wide diversity of datasets.
Sentiment Analysis For the sentiment analysis classification task, we follow (Blitzer et al., 2007) and (Ruder & Plank, 2018) by randomly selecting 6 Amazon multi-domain review datasets, as well as Yelp reviews (Asghar, 2016) and IMDB movie reviews datasets (Maas et al., 2011). 7 Altogether, these datasets exhibit wide diversity based on review length and topic (see Table 1). We normalize all datasets to have 5 sentiment classes: very negative, negative, neutral, positive, and very positive. We sample 50k examples for training, 5k for validation, and 5k for testing.
Experimental Setup To evaluate methods for the multi-domain active learning task, we conduct the experiment described in Section 3 for each acquisition method, rotating each domain as the target set. Model M , a BERT-Base model (Devlin et al., 2019), is chosen via hyperparameter grid search over learning rate, number of epochs, and gradient accumulation. The large volume of experiments entailed by this search space limits our capacity to benchmark performance variability due to isolated factors (the acquisition method, the target domain, or fine-tuning final models). However, our hyperparameter search closely mimics the process of an ML practitioner looking to select a best method and model, so we believe our experiment design captures a fair comparison among methods. See Algorithm 1 in Appendix Section B for full details.
4The approximation is also referred to as Proxy A-Distance (PAD) from (Elsahar & Gallé, 2019b) 5Hyperparameter choices and training procedures are detailed in the Appendix. 6The workshop pre-processed all datasets into a similar format, for fully answerable, span-extraction QA: https://github.com/mrqa/MRQA-Shared-Task-2019. 7https://jmcauley.ucsd.edu/data/amazon/, https://www.yelp.com/dataset, https://ai.stanford.edu/˜amaas/data/sentiment/.

heading: 6.1 COMPARING ACQUISITION METHODS
text: Results in Figure 1 show the experiments described in Section 5: benchmarking each acquisition method for multi-domain active learning. We observe for both question answering (QA) and sentiment analysis (SA), most methods manage to outperform the no-extra-labelled data baseline (0% at the y-axis) and very narrowly outperform the random selection baseline (red line). Consistent with prior work (Lowell et al., 2019), active learning strategies in NLP have brittle and inconsistent improvements over random selection. Our main empirical findings, described in this section, include:
• H-Divergence, and particularly DAL-E variants, consistently outperform baselines and other families of methods.
• The ordering of examples in Uncertainty methods depend significantly on the diversity in source domains. BALD variants perform best among available options.
• Task-agnostic representations, used in KNN or DAL variants, provide consistently strong results on average, but task-specific representations significantly benefit certain target sets.
• Different families of methods rely on orthogonal notions of relevance in producing their example rankings.
H-Divergence methods categorically achieved the highest and most reliable scores, both as a family and individual methods, represented in the top 3 individual methods 11 / 18 times for QA, and 20 / 24 times for SA. For QA, BALD↑ and DAL-E∗ had the best mean and median scores respectively, and for SA DAL-E achieved both the best mean and median scores. Among these methods, our proposed DAL-E variants routinely outperform DAL-T variants by a small margin on average, with equivalent training and tuning procedures. We believe this is because DAL-E captures both notions of domain similarity and uncertainty. By design it prioritizes examples that are similar to in-domain samples, but also avoids those which are uninformative, because the model already performs well on them.
Among Uncertainty methods, for SA methods which select for higher uncertainty vastly outperformed those which selected for low uncertainty. The opposite is true for QA. This suggests the
diversity of QA datasets contain more extreme (harmful) domain shift than the (mostly Amazon-based) SA datasets.8 In both settings, the right ordering of examples with BALD (epistemic uncertainty) achieves the best results in this family of methods, over the others, which rely on total uncertainty.
Among Reverse Classification Accuracy methods, our R̃CA variant also noticeably outperforms standard RCA and most other methods, aside from DAL and BALD. Combining R̃CA with an example ranking method is a promising direction for future work, given the performance it achieves selecting examples randomly as a Domain Budget Allocation strategy.
Lastly, the Semantic Similarity Detection set of methods only rarely or narrowly exceed random selection. Intuitively, task-agnostic representations (KNN) outperform KNN∗, given the task-agnostic sentence encoder was optimized for cosine similarity.
Embedding Ablations To see the effects of embedding space on KNN and DAL, we used both a task-specific and task-agnostic embedding space. While a task-specific embedding space reduces the examples to features relevant for the task, a task-agnostic embedding space produces generic notions of similarity, unbiased by the task model.
According to Figure 1, KNN outperforms KNN∗. In the QA setting, KNN∗’s median is below the random baseline’s. In both plots, KNN∗’s whiskers extend below 0, indicating that in some cases the method actually chooses source examples that are harmful to target domain performance.
For DAL methods, task-agnostic and task-specific embeddings demonstrated mostly similar median performances. Notably, the boxes and whiskers are typically longer for task-specific methods than task-agnostic methods. This variability indicates certain target datasets may benefit significantly from task-specific embeddings, though task-agnostic embeddings achieve more consistent results.
Comparing Example Rankings For each setting, we quantify how similar acquisition methods rank examples from DS . In Figure 2, for each pair of methods, we calculate the Kendall’s Tau coefficient between the source example rankings chosen for a target domain, then average this coefficient over the target domains. Kendall’s Tau gives a scores [−1, 1], with -1 meaning perfect anti-correlation, 0 meaning no correlation, and 1 meaning perfect correlation between the rankings. Methods from different families show close to no relationship, even if they achieve similar performances, suggesting each family relies on orthogonal notions of similarity to rank example relevance. This suggests there is potential for combining methods from different families for this task in future work.
In Sentiment tasks, all uncertainty methods had highly correlated examples. In QA, ENTR had little correlation with any method. This is likely due to the significantly larger output space for QA models. Compared to only 5 label classes in SA, question answering models distribute their start and end confidences over sequences of up to 512, where there can be multiple valid answer candidates. Embedding space also largely influences the examples that methods chose. DAL methods had higher correlations with each other when they share the same embedding space; i.e. DAL-E’s ranking has a higher correlation with DAL-T than with DAL-E∗.

heading: 6.2 PROPERTIES OF OPTIMAL EXAMPLE SELECTION
text: We examine three properties of optimally selected examples: (i) whether selecting from many diverse or one single domain leads to better performance, (ii) whether the selection of a domain or the individual examples matters more to performance, and (iii) whether selection strategies can benefit from source domain information rather than treating samples as drawn from a single pool? Our findings regarding properties of optimal selection, as described in this section, include:
• Selecting a diversity of domains usually outperforms selecting examples from a single domain.
• Acquisition functions such as DAL-E∗ do rely on example selection, mainly to avoid the possibility of large negative outcomes.
• Domain Budget Allocation during selection may improve performance. Surprisingly, even random selection from an “optimal” balance of domains beats our best performing acquisition methods most of the time.
8Accordingly, we attempt to derive a relationship between domain distance and method performance in Appendix F, but find intuitive calculations of domain distance uninterpretable.
Are Many Diverse or One Single Domain Preferable? To answer this question we conduct a full search over all combinations of source datasets. For each target set, we fix 2k in-domain data points and sample all combinations of other source sets in 2k increments, such that altogether there are 10k training data points. For each combination of source sets, we conduct a simple grid search, randomly sampling the source set examples each time, and select the best model, mimicking standard practice among practitioners.
The result is a comprehensive search of all combinations of source sets (in 2k increments) up to 10k training points, so we can rank all combinations of domains per target, by performance. Tables 2a and 2b show the optimal selections, even as discrete as 2k increments, typically select at least two or more domains to achieve the best performance. However, 1 of 6 targets for QA, or 2 of 8 for the SA tasks achieve better results selecting all examples from a single domain, suggesting this is a strong baseline, if the right source domain is isolated. We also report the mean score of all permutations to demonstrate the importance of selecting the right set of domains over a random combination.
Domains or Examples? Which is more important, to select the right domains or the right examples within some domain? From the above optimal search experiment we see selecting the right combination of domains regularly leads to strong improvements over a random combination of domains. Whether example selection is more important than domain selection may vary depending on the example variety within each domain. We narrow our focus to how much example selection plays a role for one of the stronger acquisition functions: DAL-E∗. We fix the effect of domain selection (the number of examples from each domain) but vary which examples are specifically selected. Using DAL-E∗’s distribution of domains, we compare the mean performance of models trained on it’s highest ranked examples against a random set of examples sampled from those same domains. We find a +0.46 ± 0.25% improvement for QA, and +0.12 ± 0.19% for SA. We also compare model performances trained on random selection against the lowest ranked examples by DAL-E∗. Interestingly, we see a -1.64 ± 0.37% performance decrease for QA, and -1.46 ± 0.56% decrease for sentiment tasks. These results suggest that example selection is an important factor beyond domain selection, especially for avoiding bad example selections.
Single Pool or Domain Budget Allocation Does using information about examples’ domains during selection lead to better results than treating all examples as coming from a single unlabeled pool? Originally, we hypothesized Single Pool Strategy methods would perform better on smaller budget sizes as they add the most informative data points regardless of domain. On the other hand, we thought that if the budget size is large, Domain Budget Allocation would perform best, as they choose source domains closest to the target domain. Based on Tables 1b and 1a, we were not able to draw conclusions about this hypothesis, as each sample size n = {8000, 18000, 28000} produced roughly similar winning methods. Future work should include a wider range of budget sizes with larger changes in method performance between sizes.
In our main set of experiments, the RCA acquisition functions follow the Domain Budget Allocation strategy, while all other acquisition functions follow the Single Pool strategies. Based on median performance, R̃CA outperformed all other methods (we’re including BALD here due to inconsistency in performance between QA and SA) except for those in theH-Divergence family. This suggests that using domain information during selection can lead to performance gains.
The Optimal Domain Search experiments, shown in Tables 2a and 2b, further suggest that allocating a budget from each domain can improve performance. For 8 out of our 14 experiments, selecting random samples according to the optimal domain distribution outperform any active learning strategy. While the optimal domain distributions were not computed a priori in our experiments, this result shows the potential for Domain Budget Allocation strategies. Future work could reasonably improve our results by developing an acquisition function that better predicts the optimal domain distributions than R̃CA, or to even have greater performance gains by budgeting each domain, then applying an active learning strategy (e.g. DAL-E) within each budget.

heading: 7 CONCLUSION
text: We examine a challenging variant of active learning where target data is scarce, and multiple shifted domains operate as the source set of unlabeled data. For practitioners facing multi-domain active learning, we benchmark 18 acquisition functions, demonstrating theH-Divergence family of methods and our proposed variant DAL-E achieve the best results. Our analysis shows the importance of example selection in existing methods, and also the surprising potential of domain budget allocation strategies. Combining families of methods, or trying domain adaptation techniques on top of selected example sets, offer promising directions for future work.

heading: A MULTI-DOMAIN ACTIVE LEARNING TASK
text: In this section, we would enumerate real-world settings in which a practitioner would be interested in multi-domain active learning methods. We expect this active learning variant to be applicable to cold starts, rare classes, personalization, and settings where the modelers are constrained by privacy considerations, or a lack of labelers with domain expertise.
• In the cold start scenario, for a new NLP problem, there is often little to no target data available yet (labeled or unlabelled), but there are related sources of unlabelled data to try. Perhaps an engineer has collected small amounts of training data from an internal population. Because the data size is small, the engineer is considering out-of-domain samples, collected from user studies, repurposed from other projects, scraped from the web, etc..
• In the rare class scenario, take an example of a new platform/forum/social media company classifying hate speech against a certain minority group. Perhaps the prevalence of positive, in-domain samples on the social media platform is small, so an engineer uses out-domain samples from books, other social media platforms, or from combing the internet.
• In a personalization setting, like spam filtering or auto-completion on a keyboard, each user may only have a couple hundred of their own samples, but out-domain samples from other users may be available in greater quantities.
• In the privacy constrained setting, a company may collect data from internal users, user studies, and beta testers; however, a commitment to user privacy may incentivize the company to keep the amount of labeled data from the target user population low.
• Lastly, labeling in-domain data may require certain domain knowledge, which would lead to increased expenses and difficulty in finding annotators. As an example, take a text classification problem in a rare language. It may be easy to produce out-domain samples by labeling English text and machine translating it to the rare language, whereas generating in-domain labeled data would require annotators who are fluent in the rare language.
In each of these settings, target distribution data may not be amply available, but semi-similar unlabelled domains often are. This rules out many domain adaptation methods that rely heavily on unlabelled target data.
We were able to simulate the base conditions of this problem with sentiment analysis and question answering datasets, since they are rich in domain diversity. We believe these datasets are reasonable proxies to represent the base problem, and yield general-enough insights for a practitioner starting on this problem.

heading: B REPRODUCIBILITY

heading: B.1 DATASETS AND MODEL TRAINING
text: We choose question answering and sentiment analysis tasks as they are core NLP tasks, somewhat representative of many classification and information-seeking problems. Multi-domain active learning is not limited to any subset of NLP tasks, so we believe these datasets are a reasonable proxie for the problem.
For question answering, the MRQA shared task (Fisch et al., 2019) includes SQuAD (Rajpurkar et al., 2016), NewsQA (Trischler et al., 2016), TriviaQA (Joshi et al., 2017), SearchQA (Dunn et al., 2017), HotpotQA (Yang et al., 2018), and Natural Questions (Kwiatkowski et al., 2019).
For the sentiment analysis classification task, we use Amazon datasets following (Blitzer et al., 2007) and (Ruder & Plank, 2018), as well as Yelp reviews (Asghar, 2016) and IMDB movie reviews datasets (Maas et al., 2011). 9 Both question answering and sentiment analysis datasets are described in Table 1.
For reproducibility, we share our hyper-parameter selection in Table 3. Hyper-parameters are taken from Longpre et al. (2019) for training all Question Answering (QA) models since their parameters
9https://jmcauley.ucsd.edu/data/amazon/, https://www.yelp.com/dataset, https://ai.stanford.edu/˜amaas/data/sentiment/.
are tuned for the same datasets in the MRQA Shared Task. We found these choices to provide stable and strong results across all datasets. For sentiment analysis, we initially experimented on a small portion of the datasets to arrive at a strong set of base hyper-parameters to tune from.
Our BERT question answering modules build upon the standard PyTorch (Paszke et al., 2019) implementations from HuggingFace, and are trained on one NVIDIA Tesla V100 GPU.10.

heading: B.2 EXPERIMENTAL DESIGN
text: For more detail regarding the experimental design we include Algorith 1, using notation described in the multi-domain active learning task definition.
Algorithm 1 EXPERIMENTAL DESIGN 1: for each Acquisition Function Af do 2: for each Target set DT ∼ D do 3: DtrainT , D dev T , D test T ∼ DT
4: DS := {x ∈ D | x /∈ DT } 5: MA ← TRAIN(DtrainT , DdevT ) 6: Dchosen ← [Rankx∈DSAf (x,MA)][: n] 7: Dfinal−train = DtrainT ∪Dchosen 8: M ← GRIDSEARCH(Dfinal−train, DdevT ) 9: (Af , DT ) = s Af T ←M(D test T )
10: end for 11: end for 12: return Scores Dictionary (Af , DT )→ s Af T

heading: C ACQUISITION FUNCTIONS

heading: C.1 TASK AGNOSTIC EMBEDDINGS
text: To compute the semantic similarity between two examples, we computed the example embeddings using the pre-trained model from a sentence-transformer (Reimers & Gurevych, 2019). We used the
10https://github.com/huggingface/transformers
RoBERTa large model, which has 24 layers, 1024 hidden layers, 16 heads, 355M parameters, and fine tuning on the SNLI (Bowman et al., 2015), MultiNLI (Williams et al., 2018), and STSBenchmark (Cer et al., 2017) datasets. Its training procedure is documented in https://www.sbert.net/ examples/training/sts/README.html.

heading: C.2 BAYESIAN ACTIVE LEARNING BY DISAGREEMENT (BALD)
text: We note that Siddant and Lipton’s presentation of BALD is more closely related to the Variation Ratios acquisition function described in Gal et al. (2017) than the description of dropout as a Bayesian approximation given in Gal & Ghahramani (2016). In particular, Gal et al. (2017) found that Variation Ratios performed on par or better than Houlsby’s BALD on MNIST but was less suitable for ISIC2016.

heading: C.3 DISCRIMINATIVE ACTIVE LEARNING MODEL (DAL) TRAINING
text: DAL’s training set is created using the methods detailed in Section 4.2. The training set is then partitioned into five equally sized folds. In order to predict on data that is not used to train the discriminator, we use 5-fold cross validation. The model is trained on four folds, balancing the positive and negative classes using sample weights. The classifier then predicts on the single heldout fold. This process is repeated five times so that each example is in the held out fold exactly once. Custom model parameters are shown in Table 4; model parameters not shown in the table are the default XGBClassifier parameters in xgboost 1.0.2. The motivations for choice in model and architecture are the small amount of target domain examples requiring a simple model to prevent overfitting and the ability of decision trees to capture collective interactions between features.

heading: D FULL METHOD PERFORMANCES
text: We provide a full breakdown of final method performances in Tables 5 and 6.

heading: E KENDALL’S TAU

heading: E.1 DEFINITION
text: Kendall’s Tau is a statistic that measures the rank correlation between two quantities. Let X and Y be random variables with (x1, y1), (x2, y2), ..., (xn, yn) as observations drawn from the joint distribution. Given a pair (xi, yi) and (xj , yj), where i 6= j, we have: yj−yi xj−xi > 0 : pair is concordant
yj−yi xj−xi < 0 : pair is discordant
yj−yi xj−xi = 0 : pair is a tie
Let nc be the number of concordant pairs and nd the number of discordant pairs. Let ties add 0.5 to the concordant and discordant pair counts each. Then, Kendall’s Tau is computed as:11
τ = nc−ndnc+nd
E.2 INTER-FAMILY COMPARISON
Here, we extend on our comparison of example rankings by presenting plots of Kendall Tau scores normalized by intra-family scores in 3. For the sentiment setting, the ranges of intra-family Kendall Tau coefficients are smaller than the MRQA setting. Methods in the uncertainty family have especially strong correlations with each other and much weaker with methods outside of the family. For Hdivergence based methods, intra-family correlations are not’t as strong as for the uncertainty family;
11https://www.itl.nist.gov/div898/software/dataplot/refman2/auxillar/ kendell.htm
in fact, the Kendall Taus between DAL-E/KNN and DAL-T/KNN appear to be slightly within the H-divergence intra-family range.
Furthermore, intra-family ranges are quite large for all families in the MRQA setting. For each method, there is at least one other method from a different family with which it had a higher Kendall Tau coefficient than the least similar methods of its own family.

heading: F RELATING DOMAIN DISTANCES TO PERFORMANCE
text: We investigated why certain methods work better than others. One hypothesis is that there exists a relationship between between target-source domain distances and method performance. We estimated the distance between two domains by computing the Wasserstein distance between random samples of 3k example embeddings from each domain. We experimented with two kinds of example embeddings: 1. A task agnostic embedding computed by the sentence transformer used in the KNN method, and 2. A task specific embedding computed by a model trained with the source domain used in the DAL∗ method. Given that there are k − 1 source domains for each target domain, we tried aggregating domain distances over its mean, minimum, maximum, and variance to see if Wasserstein domain distances could be indicative of relative performance across all methods.
Figure 4, Figure 5, Figure 6, and Figure 7 each show, for a subset of methods, the relationship between each domain distance aggregation and the final performance gap between the best performing method. Unfortunately, we found no consistent relationship for both MRQA and the sentiment classification tasks. We believe that this result arose either because our estimated domain distances were not reliable measures of domain relevance, or because the aggregated domain distances are not independently sufficient to discern relative performance differences across methods.
Figure 5: Minimum Wasserstein domain distance vs method performance.
Figure 6: Maximum Wasserstein domain distance vs method performance.

Review summaries:
This paper makes comparison with techniques used in active learning (AL), domain shift detection (DS), and multi-domain sampling to combine data from multiple sources. The experiments are conducted on datasets from questions answering and sentiment analysis.   The paper is well organized and easy to follow. However, the contribution of this paper is not clear. Specifically, I would expect authors provide more detailed recommendation for AL, DS, and multi-domain sampling in terms of sampling techniques, and population of different sources for certain application.  My another is concern is that the motivation of the experimental design is not clear. Why authors consider questions answering and sentiment analysis as the applications? It requires more analysis about experimental results, such as Figure 1 and tables in Section D.   

This paper surveys a broad range of techniques for active learning in the multi-domain setting applied to text - specifically, given a small labeled dataset from a target domain and a large amount of unlabeled data from a collection of source domains, how should examples be picked from the source domains to for labeling to maximise performance on the target domain. The main findings of the work are that selecting examples based on H-divergence measures perform much better than most active learning approaches. In-particular, they propose a method termed DAL-E where examples are selected from the source domain based on similarity to *misclassified* target domain examples, and show that it outperforms standard “disciminative active learning” on most of their experiments. From analysis, it is revealed that selecting from diverse domains helps, and their proposed DAL-E can help with avoiding bad examples to be selected.  

The authors investigate the efficacy of several active learning-related techniques in classification under domain shift with multiple source domains. They include: uncertainty methods, H-divergence methods, reverse-classification methods, and nearest neighbor methods.  They construct 18 methods with various combinations and settings and carry out experiments in two text classification datasets: Question answering and sentiment analysis.  They report several findings, among them, the most interesting one to me is that uncertainty in various models manifests itself in various ways and candidate data points are not necessarily the same across models.