Active Learning over Multiple Domains in Natural Language Tasks
Authors: Shayne Longpre
Julia Rachel Reisler
Edward Greg Huang
Yi Lu
Andrew Frank
Nikhil Ramesh
Chris DuBois
Venue: ICLR 2022 Submitted (2022)
Keywords: natural language processing
active learning
domain shift detection

Abstract: Studies of active learning traditionally assume the target and source data stem from a single domain. However, in realistic applications, practitioners often require active learning with multiple sources of out-of-distribution data, where it is unclear a priori which data sources will help or hurt the target domain. We survey a wide variety of techniques in active learning (AL), domain shift detection (DS), and multi-domain sampling to examine this challenging setting for question answering and sentiment analysis. We ask (1) what family of methods are effective for this task? And, (2) what properties of selected examples and domains achieve strong results? Among 18 acquisition functions from 4 families of methods, we find H- Divergence methods, and particularly our proposed variant DAL-E, yield effective results, averaging 2-3% improvements over the random baseline. We also show the importance of a diverse allocation of domains, as well as room-for-improvement of existing methods on both domain and example selection. Our findings yield the first comprehensive analysis of both existing and novel methods for practitioners faced with multi-domain active learning for natural language tasks.

Under review as a conference paper at ICLR 2022
ACTIVE LEARNING OVER MULTIPLE DOMAINS IN NATURAL LANGUAGE TASKS
Anonymous authors Paper under double-blind review
ABSTRACT
Studies of active learning traditionally assume the target and source data stem from single domain. However, in realistic applications, practitioners often require active learning with multiple sources of out-of-distribution data, where it is unclear priori which data sources will help hurt the target domain. We survey a wide variety of techniques in active learning (AL), domain shift detection (DS), and multi-domain sampling to examine this challenging setting for question answering and sentiment analysis. We ask (1) what family of methods are effective for this task? And, (2) what properties of selected examples and domains achieve strong results? Among 18 acquisition functions from 4 families of methods, we find H-- Divergence methods, and particularly our proposed variant DAL-E, yield effective results, averaging 2-3% improvements over the random baseline. We also show the importance of diverse allocation of domains, well as room-for-img iorooo-ooeeeeeennm of existing methods on both domain and example selection. Our findings yield the first comprehensive analysis of both existing and novel methods for practitioners faced with multi-domain active learning for natural language tasks.
INTRODUCTION
New natural language problems, outside the watershed of core NLP, are often strictly limited by dearth of labeled data. While unlabeled data is frequently available, not always from the same source as the targer distribution. This is particularly prevalent for tasks characterized by (i) significant distribution shift over time, (ii) personalization for user subgroups, or (iii) different collection mediums (see examples in Section
widely-used solution to this problem is to bootstrap larger training set using active learning (AL): method to decide which unlabeled training examples should be labeled on fixed annotation budget (Cohn et al. 1996; Settles 2012). However, most active learning literature NLP assumes the unlabeled source data is drawn from the same distribution the target data (Dor et al. 2020). This simplifying assumption avoids the frequent challenges faced by practitioners in multi domain active learning. In this realistic setting, there are multiple sources of data (i.e. domains) consider. In this case, it's unclear whether to optimize for homogeneity or heterogeneity of selected examples. Secondly, is more effective allocate example budget per domain, or treat examples unlabeled pool? Where active learning baselines traditionally select examples the model is least confident on (Settles 2009), in this setting could lead distracting examples from very dissimilar distributions.
single
In this work we empirically examine four separate families of methods (uncertainty- Divergence, reverse classification accuracy, and semantic similarity detection) over several question answering and sentiment analysis datasets, following (Lowell al. 2019. Elsahar & Gallé 2019h), to provide actionable insights to practitioners facing this challenging variant of active learning for natural language. We address the following questions:
H-
1. What family of methods are effective for multi-domain active learning?
2. What properties of the example and domain selection yield strong results?
While previous work has investigated similar settings (Saha al. 2011 Liu et al. 2015 Zhao et al. 2021 Kirsch et al. 202T) we contribute, to our knowledge, the first rigorous formalization and broad survey of methods within NL.P. We find that many families of techniques for active learning and

---

Under review as a conference paper at ICLR 2022
domain shift detection fail to reliably beat random baselines in this challenging variant of active learning, but certain H-Divergence methods are consistently strong. Our analysis identifies stark dissimilarities of these methods' example selection, and suggests domain diversity is important factor in achieving strong results. These results may serve as guide to practitioners facing this problem, suggesting particular methods that are generally effective and properties of strategies that increase performance.
RELATED WORK
Active Learning in NLP Lowell al. (2019) shows how inconsistent active learning methods are in NLP, even under regular conditions, However, Dor et al. (2020); Siddhant & Lipton (2018) survey active learning methods in NLP and find notable gains over random baselines, Kouw Loog (019)
survey domain adaptation without target labels, similar to our setting, but for non-language tasks. We reference more active learning techniques in Section
Domain Shift Detection Elsahar & Gallé (2019b) attempt to predict accuracy drops due to domain shifts and Rabanser et al. (2018) surveys different domain shift detection methods, |Arora et al. (2021)
examine calibration and density estimation for textual OOD detection.
Active Learning under Distribution Shift A few previous works investigated active learning under distribution shifts, though mainly in image classification, with single source and target domains. Kirsch al. (202T) finds that BALD, which is often considered the state of the art for unshifted domain settings, can get stuck on irrelevant source domain or junk data. Zhao et al. (2021) investi- gates label shift, proposing a combination of predicted class balanced subsampling and importance weighting. [Saha et al. 2011)) whose approach corrects joint distribution shift, relies on the covariate shift assumption. However, in practical settings, there may be general distributional shifts where neither the covariate shift nor label shift assumptions hold.
Transfer I Learning from Multiple Domains
Attempts to better understand how to handle shifted domains for better generalization or target performance has motivated work in question answering (Tlmor & Berant 2019] Fisch et al. [2019] Longpre et al. 2019: Kamath et al. (020) and classification tasks (Ruder & Plank| 2018: [hhorann al.] (0200. Ruder Plank (2017) show the benefits of both data similarity and diversity in transfer learning. Rücklé et al. (2020) find that sampling from a wide- variety of source domains (data scale) outperforms sampling similar domains in question answering.
He et al. (202T) investigate a version of multi-domain active learning where models are trained and evaluated on examples from all domains, focusing on robustness across domains.
3 MULTI-DOMAIN ACTIVE LEARNING
Suppose we have multiple domains D1, D2. .... Du Let one of the k domains be the target set Dr. and let the other i domains comprise the source set Ds U D,.
Let
T
Given:
Target: Small samples of labeled data points (Ir, v) from the target domain. Dtrrin Ddc=, Diast ~ Dr
Source: A large sample of unlabeled points (z) from the source domains. Ds UD; D;
iT
Task:
1. Choose n. samples from Ds to label. Dekosen C Ds. Dehosen n., selected by arg max A5(x) where A, is an acquisi- tion function: a policy to select unlabeled examples from Ds for labeling.
2. Train model M -ninalttrain validating on Ddev. Diinal -tmain Dirran U Dehasen
3. Evaluate M on Diest giving
We define a domain as a dataset collected independently the others.
? D 2000 to simulate small but reasonable quantity of labeled, in-domain training data for active learning scenarios.
2

---

Under review as a conference paper at ICLR 2022
For Step the practitioner chooses samples with the highest scores according their acquisition function Ay M is fine-tuned on these n samples, then evaluated on Dieat to demonstrate A ability to choose relevant out-of-distribution training examples.
METHODS
We identify four families of methods relevant active learning over multiple shifted domains. Uncertainty methods are common in standard active learning for measuring example uncertainty or familiarity to a model; H-Divergence techniques train classifiers for domain shift detection; Semantic Similarity Detection finds data points similar to points in the target domain; and Reverse Classification Accuracy approximates the benefit of training on a dataset. A limitation of our work is we do not cover all method families, such domain adaptation, just those we consider most applicable. We derive ~8 active learning variants, comprising the most prevalent and effective from prior work, and novel extensions/variants of existing paradigms for the multi-domain active learning setting (see KNN, RCA and DAL-E).
Furthermore, split the families into two acquisition strategies: Single Pool Strategy and Domain Budget Allocation. Single Pool Strategy, comprising the first three families methods, treats all examples coming from one single unlabeled pool. Domain Budget Allocation, consisting of Reverse Classification Accuracy methods, simply allocate an example budget for each domain.
We enumerate acquisition methods A below. Each method produces full ranking of examples in the source set Ds- To rank examples, most acquisition methods train an acquisition model, MA. using the same model architecture as M. MA is trained on all samples from Diroin, except for DAL and KNN, which split Derein into two equal segments, one for training MA and one for an internal model. Some methods have both ascending and descending orders of these rankings (denoted by and respectively, in the method abbreviations). to test whether similar distant examples are preferred in a multi-domain setting.
Certain methods use vector representations of candidate examples. We benchmark with both task- agnostic and task-specific encoders. The aask-agnostic embeddings are taken from the last layer's CLS token in Reimers & Gurevych| (201))'s sentence encoder (Appendix for details). The task-specific embeddings are taken from the last layer's CLS token in the trained model MA.
The motivation of the task-specific variant is that each example's representation will capture task- relevant differences between examples while ignoring irrelevant differences [ The versions of DAL and kNN methods that use tasc-spec vectors are denoted with "*" in their abbreviation. Otherwise, they use task-agnostic vectors.
4.1 UNCERTAINTY METHODS
These methods measure the uncertainty of trained model on new example. Uncertainty can reflect either aleatoric uncertainty, due to ambiguity inherent in the example, or epistemic uncertainty, due to limitations of the model (Kendall Gal 2017). For the following methods, let Y be the set of all possible labels produced from the model M (x) and by be the logit value for y E Y.
Confidence (CONF) A model's confidence P(yla: in its prediction estimates the difficulty or unfamillarity of an example (Guo al. 2017 Elsahar & Gallé 2019a).
Entropy (ENTR) Entropy applies Shannon entropy (Shannon 1948 to the full distribution of class probabilities for each example, formalized as AgNIR-
Y
AcONF(I, MA) = max.mamylaaaa (P()
Σ 1=1
P(n(=) log P(ysax))
Energy-based Distr---Dsttibuutiin Detection (ENG) Liu et al. (2020) use an energy-based score
to distinguish between in- and out-distribution examples. They demonstrate this method is less susceptible to overconfidence issues of softmax approaches.
For instance, consider in one domain every example is prefixed with "Text:" while the other is not- telling the difference is trivial, but the examples could he near-identical with respect the task.
3

---

Under review as a conference paper at ICLR 2022
Bayesian Active Learning by Disagreement (BALD) Gal & Ghahramani (016) introduces
estimating uncertainty by measuring prediction disagreement over multiple inference passes, each with distinct dropout mask. BALD isolates epistemic uncertainty, the model would theoretically produce stable predictions over inference passes given sufficient capacity. We conduct T 20 forward passes Ut argmax, P(y xt), representing the predicted class the 1-th model pass on æ. Following (Lowell et al. 2019), ties are broken by taking the mean label entropy over all T runs.
ABALD(E, MA) =1 count (ū.))
A MA) log Σ ev
yEY
4.2 H-DIVERGENCE METHODS
Ben-David al. (2066 2010) formalize the divergence between two domains the H-Divergence, which they approximate as the difficulty for a discriminator to differentiate between the two Discriminative Active Learning (DAL) applies this concept the active learning setting Gissin &
Shalev-Shwartz 2019).
We explore variants of DAL, using an XGBoost decision tree (Chen & Guestrin, 2016 as the discriminator model f For the following methods, let Df be the 1K examples from Derain that were not used to train MA. Let E be an encoder function, which can be task-specific or agnostic as described above. We use samples both from D and Dg to train the discriminator, We assign samples origin labels 1, which depend the DAL variant. Samples from Dg with discriminator predictions closest to I are selected for labeling. The acquisition scoring function for each DAL method and training set definition, respectively, are:
ADAL g. E) = ((((x))
{(E(r), 1) Dirain-B U UD}
Discriminative Active Learning - Target (DAL.-T) DAL-T trains a discriminator g to distin- guish between target examples in D B and out-of-distribution examples from Ds. For DAL-T, D
I = Dirain (I).
Discriminative Active Learning - Error (DAL-E) DAL-E is a novel variant of DAL. DAL- E's approach is to find examples that are similar to those in the target domain that MA misclassified. We partition Derain-2 - further into erroneous samples Der and correct samples Deorr, where D Dirain-B D U Doore. For DAL-E, 1 1 D (z:).
4.3 REVERSE CLASSIFICATION ACCURACY
RCA Reverse Classification Accuracy (RCA) estimates how effective source set Di.iss is as training data for target
Dr (Fan Davidson, 2006 Elsahar & Gallé 2019b). Without gold labels for Dj we compute soft labels instead, using the BERT-Base A trained on the small labeled Dtrain We then train a child model M, on D; using these soft labels, and evaluate the child model on Ddrr RCA chooses examples randomly from whichever domain i produced the highest
score
ARCA = 1 Dtarx masies e
(x)
RCA:
8;
chossen
ST
Σ 8J
RCA-Smoothed (RCA) Standard RCA only selects examples from one domain D.. We develop novel variant which samples from multiple domains, proportional to their relative performance
the target domain D RCA-smoothed (RC'A) selects [Dchosen] examples from source domain i, based the relative difference between the performance 8; (of child model M, trained domain i with pseudo-labels from MA) on the target domain, and the performance 87 of model trained directly the target domain Deev. Since these strategies directly estimates model performance
the target domain resulting from training on each source domain, RCA and RC'A are strong Domain Budget Allocation candidates.
4

---

Under review as a conference paper at ICLR 2022
MRQA Datasets
Sentiment Datasets
Dataset
C
Q
Q
T C
Dataset
R
N
SQuAD
Crowd
Wiki
11
X
Amzn-Booke
144
12.1
8.8
79.1
NewsQA
Crowd
News
8
V
Amzn-Health
80
9.3
7.0
83.7
TriviaQA
Trivia
Web
16
Amzn-Music
132
36.2
9.1
54.7
SearchQA
Jeopardy
Web
17
V
Amzn-Software
126
14.2
8.1
77.6
HotpotQA
Crowd
Wiki
22
Amzn-Sports
49.9
0.0
50.1
Natural-QS
Search
Wiki
9
V
Amzn-Tools
89
15.3
7.9
76.8
Imdb
230
16.4
7.5
76.1
Yelp
109
24.3
10.7
65.0
Table I: Datasets: The question answering (left) and sentiment analysis (right) datasets our
experiments. Left: Query source (Q), Context source (C), mean query length (IQ)), and whether the query was written independently from the context (Q1C). Right: mean review length ([R)) and the percent representation of negative (-), neutral (N) and positive (+) labels.
4.4 NEAREST NEIGHBOUR SEMANTIC SIMILARITY DETECTION (KNN)
Nearest neighbour methods (KNN) are used find examples that are semantically similar. Using sentence encoders we can search the source set Dg to select the top k nearest examples by cosine similarity to the target set. We represent the target set as the mean embedding of Dfrain For question answering, where example contains two sentences (the query and context), we refer to KNN-Q where we only encode the query text, KNN-C where we only encode the context text, or KNN-QC where encode both concatenated together. The acquisition scoring function per example, uses either a task-specific or task-agnostic encoder E:
AKNN (z, E) CosSim (z)), Mean (CcanEE(D)) E(D(rain)
EXPERIMENTS
Experiments are conducted on two common NLP tasks: question answering (QA) and sentiment analysis (SA), cach with several available domains.
Question Answering We employ 6 diverse QA datasets from the MRQA 2019 workshop Fiish al., 2019). shown Table (left) 6 We sample 60k examples from each dataset for training, 5k for validation, and 5k for testing. Questions and contexts are collected with varying procedures and sources, representing a wide diversity of datasets.
Sentiment Analysis For the sentiment analysis classification task, we follow (Blitzer al. 2007 and (Ruder & Plank 2018) 8) by randomly selecting Amazon multi-domain review datasets, as well as Yelp reviews (Asghar 2016) and IMDB movie reviews datasets (Maas al. 2011) Altogether, these datasets exhibit wide diversity based on review length and topic (see Table l. We normalize all datasets have 5 sentiment classes: very negative, negative, neutral, positive, and very positive. sample 50k examples for training, 5k for validation, and 5k for testing.
Experimental Setup
To evaluate methods for the multi-domain active learning task, we conduct the experiment described in Section 3 for each acquisition method, rotating each domain the target set. Model M, a BERT-Base model (Devlin et al. 2019), is chosen via hyperparameter grid search over learning rate, number of epochs, and gradient accumulation. The large volume experiments entailed by this search space limits our capacity to benchmark performance variability due to isolated factors (the acquisition method, the target domain, or fine-tuning final models). However, our hyper- parameter search closely mimics the process of an MI. practitioner looking to select a best method and model, so we believe our experiment design captures a fair comparison among methods. See Algorithm in Appendix Section B for full details.
"The approximation is also referred as Proxy A-Distance (PAD) from Elsahar & Gallé 2019b 'Hyperaarameter choices and training procedures are detailed the Appendix.
'The workshop pre- processed all datasets into similar format, fully answerable, span-extraction QA:
https://githule com http://gtthuuub.ccoo////////////
ucad edw.sdddddddeeeeee http:://mmaaaaae....cc...c/a/aa/
https: htops://www.veer wwwppppppppppp....oommmooemomooa
https: http:://aaa.staaorraa...
iment/
5

---

Under review as a conference paper at ICLR 2022
Family
H-Covereente
Absolute Performanc (%)
RCA
Uncertaiity
conry
cotey
Enrry
6mm
BAL
BALB)
BM gtr
pat-1
Dit-2
DM
Kley
(a)
Sentiment Analysis performance improvement (Accuracy %) by acquisition method.
Family
h-Slvergence
Absolute Performance (%)
RCA
Semantic Siilaity
Unowtbiiity
coupy
comp
ENTr
ENTRI
BALDY
BAIDI
DeL-g"
Del-Tr
DML-8
RDr
konc
<om Grc
(b) Question Answering performance improvement (FI %) by acquisition method.
The improvement of each acquisition method over the model given no extra labelled data. Boxplot and whiskers denote the median, quartiles and min/max scores aggregated across each target domain and sample sizes (n = {8000, 18000, 28000)). The red line represents the median performance of baseline that randomly selects examples to annotate. RESULTS
Figure I: Performance by Method:
6.1 COMPARING ACQUISITION METHODS
Results in Figure T show the experiments described in Section benchmarking each acquisition method for multi-domain active learning. We observe for both question answering (QA) and sentiment analysis (SA), most methods manage to outperform the no-extra-labelled data baseline (0% at y-axis) and very narrowly outperform the random selection baseline (red line). Consistent with prior work (Lowell et al. 2019). active learning strategies in NLP have brittle and inconsistent improvements over random selection. Our main empirical findings, described in this section, include:
H-Divergence, and particularly DAL-E variants, consistently outperform baselines and other families of methods.
The ordering of examples in Uncertainty methods depend significantly on the diversity in source domains. BALD variants perform best among available options.
Task-agnostic representations, used in KNN or DAL variants, provide consistently strong results average, but task-specific representations significantly benefit certain target sets. Different families of methods rely on orthogonal notions of relevance in producing their example rankings.
H-Divergence methods categorically achieved the highest and most reliable scores, both as family and individual methods, represented in the top 3 individual methods 11 / 18 times for QA, and 20 / 24 times for SA. For QA, BALD1 and DAI.-E* had the best mean and median scores respectively, and for SA DAL-E achieved both the best mean and median scores. Among these methods, our proposed DAL-E variants routinely outperform DAL-T variants by a small margin on average, with equivalent training and tuning procedures. We believe this is because DAL-E captures both notions of domain similarity and uncertainty. By design prioritizes examples that are similar to in-domain samples, but also avoids those which uninformative, because the model already performs well them.
Among Uncertainty methods, for SA methods which select for higher uncertainty vastly outper- formed those which selected for low uncertainty. The opposite is true for QA. This suggests the

---

Under review as a conference paper at ICLR 2022
diversity of QA datasets contain more extreme (harmful) domain shift than the (mostly Amazon-basee SA datasets & In both settings, the right ordering of examples with BALD (epistemic uncertainty) achieves the best results in this family of methods, over the others, which rely on total uncertainty.
Among Reverse Classification Accuracy methods, our RCA variant also noticeably outperforms standard RCA and most other methods, aside from DAL and BALD. Combining RCA with an example ranking method is a promising direction for future work, given the performance it achieves selecting examples randomly as a Domain Budget Allocation strategy.
Lastly, the Semantic Similarity Detection set of methods only rarely or narrowly exceed random selection. Intuitively, task-agnostic representations (KNN) outperform KNN*, given the task-agnostic sentence encoder was optimized for cosine similarity.
To see the effects of embedding space on KNN and DAL., we used both task-specific and task-agnostic embedding space. While a task-specific embedding space reduces the examples to features relevant for the task, task-agnostic embedding space produces generie notions of similarity, unbiased by the task model.
Embedding Ablations
According to Figure D KNN outperforms KNN.. In the QA setting, KNN''s 's median is below the random baseline's s. In both plots, KNN*'s whiskers extend below 0, indicating that in some cases the method actually chooses source examples that are harmful to target domain performance.
For DAL methods, task-agnostic and task-speeific embeddings demonstrated mostly similar median performances. Notably, the boxes and whiskers are typically longer for task-specific methods than task-agnostic methods. This variability indicates certain target datasets may benefit significantly from task-specific embeddings, though task-agnostic embeddings achieve more consistent results.
Comparing Example Rankings For each setting, we quantify how similar acquisition methods rank examples from Ds. In Figure 2 for each pair of methods, we calculate the Kendall's Tau coeffi- cient between the source example rankings chosen for target domain, then average this coefficient over the target domains. Kendall's Tau gives scores [-1, 1], with meaning perfect anti-correlation, O meaning no correlation, and meaning perfect correlation between the rankings. Methods from different families show close to no relationship, even if they achieve similar performances, suggesting each family relies on orthogonal notions of similarity to rank example relevance. This suggests there is potential for combining methods from different families for this task in future work.
In Sentiment tasks, all uncertainty methods had highly correlated examples. In QA, ENTR had little correlation with any method. This is likely due to the significantly larger output space for QA models. Compared to only 5 label classes in SA, question answering models distribute their start and end confidences over sequences of up to 512, where there can be multiple valid answer candidates. Embedding space also largely influences the examples that methods chose. DAL methods had higher correlations with each other when they share the same embedding space: i.e. DAL-E's ranking has a higher correlation with DAL-T than with DAL-E+.
6.2 PROPERTIES OF OPTIMAL EXAMPLE SELECTION
We examine three properties of optimally selected examples: (i) whether selecting from many diverse or one single domain leads to better performance, (ii) whether the selection of a domain or the individual examples matters more to performance, and (iii) whether selection strategies can benefit from source domain information rather than treating samples as drawn from a single pool? Our findings regarding properties of optimal selection, as described this section, include:
Selecting a diversity of domains usually outperforms selecting examples from a single domain.
Acquisition functions such as DAL-E+ do rely on example selection, mainly to avoid the possibility of large negative outcomes.
Domain Budget Allocation during selection may improve performance. Surprisingly, even random selection from an "optimal" balance of domains beats our best performing acquisition methods most of the time.
"Accordingly, we attempt to derive relationship between domain distance and method performance in Appendix F but find intuitive calculations of domain distance uninterpretable.
7

---

Under review as a conference paper at ICLR 2022
CONFE
1.00
CONF
0.75
ENTRY
ENTRI
E R
ENERI
BALD
0.25
BALDI
DALE?
0.00
DAL
DAL-Y"
-0225
DAL-T
-0.50
IsN (C)
MNN-GC
CONF)
CONF)
ENTRY
ENTR)
ENERI
ENER
BALD:
BALD:
DAL-E"
DAL-E
DAL-T
DAL-T
KAN (C)
MAN--
KNN OC
-1.00
Figure 2: Similarities of Example Rankings Mcasured by Kendall's Tau Coefficients, for QA (above diagonal) and SA (below diagonal). Kendall's Tau coefficient is computed between the example rankings of each pair of methods. The heatmap contains these coefficients averaged over each target dataset (some cells are crossed out for SA since SA's KNN methods don't 't have C/Q/QC variants). indicates perfect relationship between the rankings, 0 means relationship, and - means an inverse relationship.
Are Many Diverse or One Single Domain Preferable? To answer this question we conduct full search over all combinations of source datasets. For each target set, fix 2k in-domain data points and sample all combinations of other source sets in 2k increments, such that altogether there are 10k training data points. For each combination source sets, we conduct a simple grid search, randomly sampling the source set examples each time, and select the best model, mimicking standard practice among practitioners.
The result is comprehensive search of all combinations of source sets (in 2k increments) up to 10k training points, so we can rank all combinations domains per target, by performance. Tables 2a and[2b show the optimal selections, even discrete as 2k increments, typically select at least two or more domains to achieve the best performance. However, of targets for QA, or tasks achieve better results selecting all examples from single domain, suggesting this is strong baseline, the right source domain is isolated. We also report the mean score of all permutations to demonstrate the importance of selecting the right set of domains
for the SA
random combination.
Domains or Examples? Which is more important, select the right domains the right examples within some domain? From the above optimal search experiment we see selecting the right combi- nation of domains regularly leads to strong improvements Whether example selection is more important than domain selection may vary depending on the example variety within each domain. We narrow our focus how much example selection plays role for one of the stronger acquisition functions: DAL-Es.
a random combination of domains.
We fix the effect of domain selection (the number of examples from each domain) but vary which examples are specifically selected. Using DAL-E*'s 's distribution of domains, we compare the mean performance of models trained on it's highest ranked examples against random set of examples sampled from those same domains. We find a +0.46 + 0.25% improvement for QA, and +0.12 12 + 0.19% for SA. We also compare model performances trained on random selection against the lowest ranked examples by DAL-Es. Interestingly, we see a 1.64 = 0.37% performance decrease for QA, and -1.46 + 0.56% decrease for sentiment tasks. These results suggest that example selection an important factor beyond domain selection, especially for avoiding bad example selections.
Single Pool or Domain Budget Allocation Does using information about examples' domains during selection lead better results than treating all examples as coming from single unlabeled pool? Originally, we hypothesized Single Pool Strategy methods would perform better smaller budget sizes as they add the most informative data points regardless of domain. On the other hand, we thought that if the budget size is large, Domain Budget Allocation would perform best, as they choose source domains closest to the target domain. Based on Tables lb and Ta we were not able to draw conclusions about this hypothesis, as each sample size (8000, 1 18000. 28000) produced roughly similar winning methods. Future work should include a wider range budget sizes with larger changes in method performance between sizes.
8

---

Under review as a conference paper at ICLR 2022
Optimal Sample
Fl Score
Single Domain
SQ
NE
TR
SE
HT
NQ
Optimal
Mcan
Best AF
SQUAD
2k
8k
0
0
0
0
78.0
74.0
78.0
77.0
NEWSQA
2k
2k
0
0
56.1
52.0
55.2
55.9
TRIVIAQA
2k
4k
2k
0
0
2k
62.9
58.8
61.8
61.9
SEARCHQA
4k
0
0
2k
2k
2k
64.4
61.2
63.5
65.1
HOTPOTQA
2k
2k
67.1
63.6
66.4
66.0
NATURALQ
2k
4k
0
0
2k
2k
63.7
59.8
63.0
64.6
MEAN
65.4
61.5
64.6
65.1
(a) Optimal domain search for Question Answering (QA).
Optimal Sample
Accuracy Score
Single Domain
A-B
A-So
A-Sp
A-T
IM
Optimal
Mean
Best AF
AMZN-B
2k
2k
4k
69.0
66.5
67.6
AMZN-H
0
2k
69.8
67.8
70.0
AMZN-M
0
0
2k
1)
2k
6k
70.8
69.0
70.1
70.4
AMZN-SO SO
2k
2k
64.7
62.6
AMZN-SP
2k
2k
67.5
65.3
67.5
68.1
AMZN-T
0
8k
0
2K
0
68.4
65.7
68.4
68.3
IMDB
4k
2k
2k
2k
60.2
57.8
59.9
60.5
67.0
64.9
MEAN
67.2
64.9
(b) Optimal domain search for Sentiment Analysis (SA).
Table 2: Optimal Domain Search: The optimal distribution examples is shown per target domain, in 2k increments. The underlined value indicates the "Single source Domain" (2k in in-domain, 8k source domain) that gave best results. On the right we show the F1 score for this optimal distribution, the mean score across all distribution combinations, the best Single source Domain, and the Best Acquisition Function (from Figure [ Typically allocating optimal domain budgets and the best acquisition functions both performed strongly.
In our main set of experiments, the RCA acquisition functions follow the Domain Budget Allocation strategy, while all other acquisition functions follow the Single Pool strategies. Based on median performance, RCA.A outperformed all other methods (we' 're including BALD here due to inconsistency in performance between QA and SA) except for those in the H-Divergence family. This suggests that using domain information during selection can lead to performance gains.
The Optimal Domain Search experiments, shown in Tables 2a and 2b further suggest that allocating a budget from each domain can improve performance. For 8 out our 14 experiments, selecting random samples according to the optimal domain distribution outperform any active learning strategy. While the optimal domain distributions were not computed priori in our experiments, this result shows the potential for Domain Budget Allocation strategies. Future work could reasonably improve our results by developing an acquisition function that better predicts the optimal domain distributions than RCA, or even have greater performance gains by budgeting each domain, then applying an active learning strategy (c.g. DAL-E) within each budget.
7 CONCLUSION
We examine challenging variant of active learning where target data is scarce, and multiple shifted domains operate as the source set of unlabeled data. For practitioners facing multi-domain active learning, we benchmark 18 acquisition functions, demonstrating the H-Divergence family of methods and our proposed variant DAL-E achieve the best results. Our analysis shows the importance of example selection in existing methods, and also the surprising potential of domain budget allocation strategies. Combining families methods, or trying domain adaptation techniques on top of selected example sets, offer promising directions for future work.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Udit Arora, William Huang, and He He. Types of out-of-distribution texts and how to detect them, 2021.
Nabiha Asghar. Yelp dataset challenge: Review rating prediction. arXiv preprint arXiv: 1605:056. 2016.
Shai Ben- John Blitzer, Koby Crammer, and Fernando Pereira. Analysis of representations for domain adaptation. In Proceedings of the 19th International Conference on Neural Information Processing Systems, pp. 137-144, 2006.
Shai Ben-David. John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. A theory of learning from different domains. Machine learning, 79(1):151-175. 2010.
John Blitzer, Mark Dredze, and Fernando Pereira. Biographies, Bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguisties, pp. 440 447, Prague, Czech Republic, June 2007. Association for Computational Linguistics. URL https: www. aclweb. org/ anthology/
P07-1056
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguisties, 2015.
Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez- and Lucia Specia. Semeval-2011 task and cross-lingual focused evaluation. arXiv preprint
I: Semantic textual arXiv:
00055000055 2017.
Tianqi Chen and Carlos Guestrin. XGBoost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD 16, pp. 785-794, New York, NY, USA, 2016. ACM. ISBN 978 42522-4232-2 doi: 10.1145/2939672.29999979
URL / /ooi acm wttp///ww.........////.....
David Cohn, Zoubin Ghahramani, and Michael Jordan. Active learning with statistical models. Journal of artificial intelligence research, 4:129-145, 1996.
Jacob Devlin, Ming- Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pp. 4171-41866 2019.
Liat Ein Dor, Alon Halfon, Ariel Gera, Eyal Shnarch, Lena Dankin, Leshem Choshen, Marina Danilevsky, Ranit Aharonov, Yoav Katz, and Noam Slonim. Active learning for bert: An empirical study. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 7949-7962, 2020.
Matthew Dunn, Levent Sagun, Mike Higgins, Ugur Guney, Volkan Cirik, and Kyunghyun Cho. Searchqa: A new q&a dataset augmented with context from search engine. arXiv preprint 17X 2017.
Hady Elsahar and Matthias Gallé. To annotate or not? predicting performance drop under domain shift. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Language Processing (EMNLP-IJCNLP), pp. 2163 2173, Hong Kong, China, November 2019a. Association for Computational Linguistics. doi: 10. www./65///////1.//1 1222. URL wtttps::///ettt wwwwwb.ww..cc..... org/ wttp.//rrr.aaooooo//i/... 1222
Hady Elsahar and Matthias Gallé. To annotate or not? predicting performance drop under domain shift. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing FEMNLP-IJCNLP.
pp. 2163-2173, 2019b.
10

---

Under review as a conference paper at ICLR 2022
Wei Fan and Ian Davidson. Reverse testing: an efficient framework to select amongst classifiers under sample selection bias. In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 147-156, 2006.
Adam Fisch, Alon Talmor, Robin Jia, Minjoon Seo, Eunsol Choi, and Danqi Chen. Mrqa 2019 shared task: Evaluating generalization in reading comprehension. EMNLP 2019 MRQA Workshop, pp. 1, 2019.
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty deep learning- In Proceedings of the 33rd International Conference on Machine Learning, New York, NY, USA, 2016.
Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep bayesian active learning with image data. In Proceedings of the 34th International Conference on Machine Learning (ICML), Sydney, Australia, 2017. URL http://proeeee...e.ee mir. pttps//tt//aa.//aaa/aa////////
Daniel Gissin and Shai SallrrSh Discriminative active learning. arXiv preprint arXiv:1907.06347, 2019.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning, Sydney, Australia, 2017. URL SRRL/ttt://ppoceedinne http://pw..ocoooooooooor///////a
ml pptcs////00/guoooa//uuoooo.. pdd
Rui He, Shan He, and Ke Tang. Multi-domain active learning: A comparative study. arXiv preprint arXiv:2106. 13516, 2021.
Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint urXiv:1705.03555 2017.
Amita Kamath, Robin Jia, and Perey Liang. Selective question answering under domain shift. In
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5684-5696, 2020.
Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision? In Proceedings of the 1 International Conference on Neural Information Processing Systems, pp- 5580-5590, 2017.
Andreas Kirsch, Tom Rainforth, and Yarin Gal. Active learning under pool set distribution shift and noisy data, 2021.
Wouter Marco Kouw and Marco Loog. A review of domain adaptation without target labels. JEEE fransactions on pattern analysis and machine intelligence, 2019.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453-4666 2019.
Angi Liu, Lev Reyzin, and Brian Ziebart. Shift-pessimistic active learning using robust bias-aware prediction. Proceedings of the AAAI Conference on Artificial Intelligence, 29(1). Feb. 2015. URL
httpss://jjs.ddt aaai. wwwp/rigeeedeee..oooooooeeoee.ee wwtp//nxxxppppp/aaee pttp://wat/aattcl////...s///////
Weitang Liu, Xiaoyun Wang. John D. Owens, and Yixuan Li. Energy- based tion. In 34th Conference on Neural Information Processing Systems (NeurIPS 2020), Vancouver, Canada, 2020. URL URL btttps://erxi.....
detec-
23w.://1901.0009...0 pdf
Shayne Longpre, Yi Lu, Zhucheng Tu, and Chris DuBois. An exploration of data augmentation and sampling techniques for domain-agnostie question answering. In Proceedings of the 2nd Workshop on Machine Reading for Question Answering, pp. 220-227, 2019.
David Lowell, Zachary C Lipton, and Byron C Wallace. Practical obstacles to deploying active learning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP.
IJCNLP), pp. 21-30, Hong Kong, China, 2019. URL https:: stpps:/weee //www/www./.. aclwea www..oogggoooooooo.. anthology/ D1 9 003.pdff.
11

---

Under review as a conference paper at ICLR 2022
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng. and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pp. 142-150. Portland, Oregon, USA, June 2011. Association for Computational Linguistics. URL http: //www. www.aaclwe.ccggaaaaee wttpthhhoooy/P111111
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vetito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 8024-8035, Curran Associates, Inc., 2019. URL http:/htp://// oag.groooooot.taaaee-eeeeeeeeeaa
1015-pytorch: an an- wwwpiamtieeeeeeeeeeeeeeeeeeeeeee pdf
Stephan Rabanser, Stephan Günnemann, and Zachary C Lipton. Failing loudly: An empirical study of methods for detecting dataset shift, arXiv preprint arXiv: 1810.. 1153,, 2018.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2383-2392, 2016.
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2019. URL https:: /
10084
Andreas Rücklé Jonas Pfeiffer, and Iryna Gurevych. Multicqa: Zero-shot transfer of self-supervised text matching models on massive scale. In Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing (EMNLP), pp. 2471 2471-2486, 2020.
Sebastian Ruder and Barbara Plank. Learning to select data for transfer learning with bayesian optimization. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp- 372-382, 2017.
Sebastian Ruder and Barbara Plank. Strong baselines for neural semi-supervised learning under domain shift. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1044-1054, 2018.
Avishek Saha, Piyush Rai, Hal Daumé, Suresh Venkatasubramanian, and Scott L. Dulal.. Active supervised domain adaptation. In Dimitrios Gunopulos, Thomas Hofmann, Donato Malerba, and Michalis Vazirgiannis (eds.), Machine Learning and Knowledge Discovery in Databases, pp. 97-112, Berlin, Heidelherg, 2011. Springer Berlin Heidelberg. ISBN 978 948-3--22-2222
Burr Settles. Active learning literature survey. 2009.
Burr Settles. Active learning. Synthesis lectures on artificial intelligence and machine learning, 6(1): 1-114, 2012.
Claude E. Shannon. A mathematical theory of communication. In Bell System Technical Journal, volume 27(3), pp. 379-423, 1948.
Akash Sheoran, Diptesh Kanojia, Aditya Joshi, and Pushpak Bhattacharyya. Recommendation chart of domains for cross-domaie sentiment analysis: Findings of 20 domain study. In Proceedings of The 12th Language Resources and Evaluation Conference, pp. 4982-4990, 2020.
Aditya Siddhant and Zachary C. Lipton. Deep Bayesian active learning for natural language pro- cessing:
Results of a large-scale empirical study. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2904-2909, Brussels, Belgium, October- November 2018. Association for Computational Linguistics. doi: 18.186 10/65///////8//// 1/D18-13111 1318. URL
https: ant http://acoonttholoo.oooor///////
12

---

Under review as a conference paper at ICLR 2022
Alon Talmor and Jonathan Berant. Multiqa: An empirical investigation of generalization and transfer in reading comprehension. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 491 19211 2019.
Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. Newsqa: A machine comprehension dataset. arXiv preprint arXiv: 161 1.1..1111100 1.09830, 2016.
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolo- gies, Volume 1 (Long Papers), pp. 1112-1122. Association for Computational Linguistics, 2018. URL htttp://acwwbb org/ http://////aaattooooo////--1--0-
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. arXiv preprint arXiv: 10.Xiv:800..00 889009900, 2018.
Eric Zhao, Anqi Liu, Animashree Anandkumar, and Yisong Yue. Active learning under label shift. In Arindam Banerjee and Kenji Fukumizu (eds.), Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, volume 130 of Proceedings of Machine Learning Research, URL https: pttpss/pproceeingss.oo mlr. www.ppresssvv... vv10
pp. 3412-3420 PMLR, 13-15 Apr 2021.
Ehao2lb I1mm1
13

---

Under review as a conference paper at ICLR 2022
A MULTI-DOMAIN ACTIVE LEARNING TASK
In this section, we would enumerate real-world settings in which a practitioner would be interested in domti- active learning methods. We expect this active learning variant to be applicable to cold starts, rare classes, personalization, and settings where the modelers are constrained by privacy considerations, or a lack of labelers with domain expertise.
In the cold start scenario, for a new NLP problem, there is often little to no target data available yet (labeled or unlabelled), but there are related sources of unlabelled data try. Perhaps an engineer has collected small amounts of training data from an internal population. Because the data size is small, the engineer considering out-of-domain samples, collected from user studies, repurposed from other projects, scraped from the web, etc..
In the rare class scenario, take an example of a new platform/forum/sociaa media company classifying hate speech against a certain minority group. Perhaps the prevalence of positive, in-domain samples on the social media platform small, so an engineer uses out-domain samples from books, other social media platforms, or from combing the internet.
In personalization setting, like spam filtering or auto-completion user may only have a couple hundred of their own samples, but out-domain samples from other users may be available in greater quantities.
keyboard, each
In the privacy constrained setting, a company may collect data from internal users, user studies, and beta testers; however, commitment to user privacy may incentivize the company to keep the amount of labeled data from the target user population low.
Lastly, labeling in-domain data may require certain domain knowledge, which would lead increased expenses and difficulty in finding annotators. As an example, take text classification problem in rare language. It may be easy to produce out-domain samples by labeling English text and machine translating the rare language, whereas generating in-domain labeled data would require annotators who are fluent in the rare language.
In each of these settings, target distribution data may not be amply available, but semi-similar unlabelled domains often are. This rules out many domain adaptation methods that rely heavily on unlabelled target data.
We were able to simulate the base conditions of this problem with sentiment analysis and question answering datasets, since they are rich in domain diversity. We believe these datasets are reasonable proxies to represent the base problem, and yield general-enough insights for a practitioner starting on this problem.
B REPRODUCIBILITY
B. DATASETS AND MODEL TRAINING
We choose question answering and sentiment analysis tasks as they are core NLP tasks, somewhat representative of many classification and information-seeking problems. Multi-domain active learning not limited to any subset of NLP tasks, so we believe these datasets are a reasonable proxie for the problem.
For question answering. the MRQA shared task (Fisch et al. 2019) includes SQuAD (Rajpurkar et al. 2016). NewsQA (Trischler et al., 2016). TriviaQA (Joshi et al. (017). SearchQA (Dunn et al. 2017. HotpotQA (Yang et al. 2018), and Natural Questions (K (Kiiatkowski et al.) 2019).
For the sentiment analysis classification task, we use Amazon datasets following (Blitzer et al. 2007 and (Ruder Plank 2018), as well as Yelp reviews (Asghar (016) and IMDB movie reviews datasets Both question answering and sentiment analysis datasets are described in
(Maas et al. 2011
For reproducibility, we share our hyper-paramette selection in Table 3 flyper-parameter are taken from Longpre et al. (2019) for training all Question Answering (QA) models since their parameters
https: http://ccaaaaaaa ucsd. wwt..dd..//aaaaaaaaaaaae/e/
htttp::/www.yeet wwwpw//ewelp.oomaaaeaettttt
https: http:/aa.sttafoooooo wdw.deedu/eoo. wdw/- htt.://arrrrroeoeaaaaaaaa//////a sent iment/
14

---

Under review as a conference paper at ICLR 2022
tuned for the same datasets in the MRQA Shared Task. We found these choices provide stable and strong results across all datasets. For sentiment analysis, we initially experimented on a small portion of the datasets to arrive strong set of base syper-parameters to tune from.
Our BERT question answering modules build upon the standard PyTorch (Paszke et al. 2019) implementations from HuggingFace, and are trained one NVIDIA Tesla V100 GPU
Model Parameter
Value
Base Pre-trained Model
BERT-base
Model Size (# params)
108.3M
Learning Rate
5e 5
Optimizer
Adam
Gradient Accumulation
Dropout
0.1
Lower Case
False
Question Answering model
Avg. Train Time
2h 20m
Batch Size
25
Num Epochs
Max Query Length
64
Max Sequence Length
512
Sentiment Classifcation model
Avg. Train Time
43m
Batch Size
20
Num Epochs
Max Sequence Length
128
Table 3: Hyperparameter selection for task models.
B.2 EXPERIMENTAL DESIGN
For more detail regarding the experimental design we include Algorith | the multi-domain active learning task definition.
using notation described in
Algorithm 1 EXPERIMENTAL DESIGN
1:
for each Acquisition Function A, do for each Target set Dy
2:
D do Dirain D Deest Dr
3:
4:
Ds = fo E
x $ Dr
5:
MA TRAIN (arann D
6:
Dchosen & Rankze As (x, Ma]][: n] Dinnat D'inat-train train Dirasi Dehnsen
7:
8:
M 4- GRIDSEARCH(D! (fintt finat train D
As (A,, Dr) = ST + M(D
9:
10:
end for
11:
end for
return Scores Dictionary (A,, Dr) + o7
C ACQUISITION FUNCTIONS
C.1 TASK AGNOSTIC EMBEDDINGS
To compute the semantic similarity between two examples, we computed the example embeddings
using the pre-trained model from semience-traansformmr (Reimers & Gurevych 2019). We used the
httpp:://// http//gttttttutt thi...orr/wuggiiggfaaerrrrrooon
15

---

Under review as a conference paper at ICLR 2022
Model Parameter
Value
DAL Discriminator
Model Type
XGBoost
Model Size (# trees)
10
Model Size (maximum depth)
2
Learning Rate
0.1
Objective
binary:logistic
Booster
gbtree
Tree Method
gpu_hist
Gamma
5
Min Child Weight
5
Max Delta Step
0
Subsample
1
Colsample Bytree
Colsample Bynode
Reg Alpha
0
Reg Lambda
5
Scale Pos Weight
Table 4: Hyperparameter selection for DAL discriminators.
RoBERTa large model, which has 24 layers, 1024 hidden layers, 16 heads, 355M parameters, and fine tuning on the SNLI (Bowman et al. 2015). MultiNLI (Williams et al.) 2018), and STSBenchmark
(Cer et al. 2017) datasets. Its training procedure is documented in https: /www. sbert. hkamples/traine
st sts////// READDE. html
C.2 BAYESIAN ACTIVE LEARNING BY DISAGREEMENT (BALD)
We note that Siddant and Lipton's 's presentation of BALD is more closely related to the Variation Ratios acquisition function described in Gal al. (211)) than the description of dropout Bayesian approximation given in Gal & Ghahramani (2016). In particular, Gal et al. (2017) found that Variation Ratios performed on par or better than Houlsby's 's BALD on MNIST but less suitable for ISIC2016.
C.3 DISCRIMINATIVE ACTIVE LEARNING MODEL (DAL) TRAINING
DAL's training set is created using the methods detailed in Section 4.2 The training set is then partitioned into five equally sized folds. In order predict data that is not used to train the diseriminator, we use 5-fold cross validation. The model is trained on four folds, balancing the positive and negative classes using sample weights. The classifier then predicts out fold. This process is repeated five times so that each example is in the held out fold exactly once. Custom model parameters are shown in Table 4 model parameters the default XGBClassifier parameters in xgboost 1.0.2. The motivations for choice in model and architecture are the small amount of target domain examples requiring a simple model prevent overfitting and the ability of decision trees to capture collective interactions between features.
the single held-
shown the table
D FULL METHOD PERFORMANCES
provide full breakdown of final method performances Tables 5 and 6
16

---

Under review as a conference paper at ICLR 2022
Morwonga
60:13
11330
62.4
16.13
10.0
12.54
0415
18:40
13.04
10:35
01.
StaRcaGA
$3.3
28-46
0334
03:0
64.33
62.13
03.22
61.3
60:2
62.
13:99
03
63:31
10:22
6340
11:4
63.84
63:27
60.99
Phrtagg
01.45
18.19
0513
$6.55
18:44
914
10:00
61:05
01:24
61.00
0.10
6381
24.66
06.05
40.2
MecrontA
06.29
0413
06.3
05.15
03.03
65.88
03.80
61.11
03.15
60.00
01:33
04
04.33
00.81
15.9
11.05
64.14
04.00
06.30
12.55
6530
62.56
61.45
63.31
68.3
6250
03.16
02.05
13.30
20.55
64.00
63.32
34.m
07:45
08:33
61.09
01.00
66.33
63.99
401
4050
63.5
03
605
18
01.00
07.14
61
19.66
6..0
25.05
04.9
03.27
54.11
60.32
64.90
08.02
3003
50.08
6372
3423
56.9
56.8
48.35
361
565
1148
4
58.3
26.1
513
25.44
4613
24.00
58.34
54.6
S0LAB
77335
78:
3531
7.55
73:3
77.6%
793
7321
7911
16.9
161H
73.77
716
71.3
Table MQRA F1 scores from each active learning method over every training set size and target domain. The best performances are bolded and underlined.
53.04
65.99
63:36
13.35
18J6
16.55
08.02
66-65
06.3
57.16
86.18
06.25
07.83
57.34
06.3
65.05
07.26
AsGos-M
08.35
13.2
07.4
71.
76A2
05.4
20,55
91.88
09144
09.55
Asers km
63:84
61:98
64:05
62-43
641
61.36
63:08
6::33
62.6
64.13
41.13
65.0
07.85
63.94
63:54
03:26
62.56
6654
67.3
68.46
15.48
07.85
08.34
56.58
03.6
66.35
07.54
09:75
04.94
64.66
63.44
65.85
16.00
65.94
06.33
06.34
05.32
06.55
Aseru-33
14:41
(69)
T5-12
18.33
1818
5136
67.65
AMGos-M
05.16
70.85
06.2
61264
5510
16.15
563
10.55
11.88
65.36
7054
70.3
71.32
65.6
06.75
21.35
21000
Asoms 8.
07:13
65.00
63.76
1TTE
6566
663
63.0
63.36
18.20
18.7%
68
08.33
83.54
65:3
66.55
15.75
0173
681
18.6
58.58
59.56
16726
13.96
54.88
62.50
YELP
05.79
06.32
02.52
64.44
15.54
6334
66.15
6606
66.2
67.
D67
612
06.35
55.95
68.30
65-46
67
Asons-11
67.0
71.1
6452
6463
6490
6675
6336
16.6
1336
87
26.34
136
31.3
762
66.45
3131
20000
AMC2S-30
Asoru-9
52.54
07.83
06.55
6341
6316
0111
61.06
63.8
03.0
60.2
1338
20
04.38
08.55
655
53.44
06.1
06.35
0.33
68.22
Tone
59.03
19.0
19.11
674
96
1946
64.
96.6
16.76
16.33
61.8
63.
606
60.33
191
00.99
63.3
56.75
06.94
Table 6: Sentiment accuracy scores from each active learning method over every training set size and target domain. The best performances are bolded and underlined.
E KENDALL'S TAU
E.1 DEFINITION
Kendall's Tau is statistic that measures the rank correlation between two quantities. Let X and Y be random variables with (I1, 31); (222, 12),, (I', yn) as observations drawn from the joint distribution. Given pair (zi, vi) and (zz, Uj), where o j, we have:
pair is concordant
8i pair is discordant
25-
pair is tie
2, -.
Let n. be the number of concordant pairs and ny the number of discordant pairs. Let ties add 0.5 to the concordnnt and discordant pair counts each. Then, Kendall's Tau is computed as
E.2 WNTER-FAMILY COMPARISON
Here, we extend on our comparison of example rankings by presenting plots of Kendall Tau scores normalized by intra-family scores in 3 For the sentiment setting, the ranges of intra-family Kendall Tau coefficients are smaller than the MRQA setting. Methods in the uncertainty family have especially strong correlations with each other and much weaker with methods outside of the family. For H- divergence based methods, intra- family correlations are not't t as strong as for the uncertainty family;
htttts:: h/wp:::::/www... it ni nist. at. v http:///w.......e.eeaaaaa/////// kende11 hht
17

---

Under review as a conference paper at ICLR 2022
127
SALLA
(a) MRQA
(b) Sentiment
Figure 3: Kendall Tau scores normalized by intra-family scores according the family of the method on the y-axis (with uncertainty uncertainty-ascending and -eeeeendd--ddseddiing as distinct families). If the cell'ss corresponding Kendall Tau score within the satraffamily range, it's value will be in [0, 1]. Below the range is negative, and above the range is greater than 1.
in fact, the Kendall Taus between DAL-E/KNN and DAL-T/KNN appear to be slightly within the H-divergence fntra-family range.
Furthermore, intra-family ranges are quite large for all families in the MRQA setting. For each method, there is at least other method from different family with which it had higher Kendall Tau coefficient than the least similar methods of its own family.
F RELATING DOMAIN DISTANCES TO PERFORMANCE
We investigated why certain methods work better than others. One hypothesis is that there exists relationship between between target-source domain distances and method performance. We estimated the distance between two domains by computing the Wasserstein distance between random samples of 3k example embeddings from each domain. We experimented with kinds of example embeddings: 1. A task agnostic embedding computed by the sentence transformer used in the KNN method, and 2. task specific embedding computed by model trained with the source domain used in the DAL* method. Given that there are 1 source domains for each target domain, we tried aggregating domain distances over its mean, minimum, maximum, and variance to see if Wasserstein domain distances could be indicative of relative performance across all methods.
Figure 4 Figure 5 Figure 6 and Figure 7 each show, for subset methods, the relationship between each domain distance aggregation and the final performance gap between the best performing method. Unfortunately, we found no consistent relationship for both MRQA and the sentiment classification tasks. We believe that this result arose either because estimated domain distances measures of domain relevance, or because the aggregated domain distances are not independently sufficient discern relative performance differences across methods.
not reliable
18

---

Under review as a conference paper at ICLR 2022
The average domain distance is calculated by finding the distance between 3k examples from Dr and the combined set made from choosing 3k examples from each domain Ds. Since the Wasserstein metric is symmetric, this yields & points for comparison.
Figures 3-6:
maga avg
Daight
target
Conkage
-3.55
Figure 4: Average Wasserstein domain distance vs performance.
mroa min. dist ait_bestt.orf
sentiment min, dist vs ditt, best.perf
dit Look
1.5
Figure 5: Minimum Wasserstein domain distance vs method performance.
miga mac, ciatve best per!
sentineet max, diss vo vil. pest
3.0
DK.
-107
1106 J01
Figure Maximum Wasserstein domain distance vs method performance.
19

---

Under review as a conference paper at ICLR 2022
chat bast
Targe
Dece
0.40
1.50
220
7700
2.00
Figure 7: Wasserstein Domain distance variance performance.
20


Reviews:
[{"review_id": "p2c4AQ7fNVP", "reviewer": "Reviewer_cB6j", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, I like this work since it serves as a guide for practitioners faced with similar settings, and gives good guidelines on what works in these challenging domain shift settings. The experiments are  extensive and cover many of the domain shift settings in NLP. However, I think some of the analysis could be improved.\n\nWeaknesses: \n- I like the analysis on measuring how different methods rank examples. However, I would highly recommend performing some kind of normalization - e.g. to compare method a1 from family A and method b1 from family B, kendall’s tau score should be normalized w.r.t the score between methods from the same family. \n- I would like to challenge some of the claims made in Section 6.2 of the paper. First, while it does seem like a diversity of examples helps, the boost is *very minimal* and it is certainly not true in general. Indeed, one can construct a domain distribution where getting data from several domains could hurt performance, if the mapping function (or the underlying P(y | x)) changes between domains.\n\nBased on these weaknesses, I think the paper is currently borderline. However, if these are addressed i’m happy to adjust my scores. ", "summary_of_the_paper": "This paper surveys a broad range of techniques for active learning in the multi-domain setting applied to text - specifically, given a small labeled dataset from a target domain and a large amount of unlabeled data from a collection of source domains, how should examples be picked from the source domains to for labeling to maximise performance on the target domain. The main findings of the work are that selecting examples based on H-divergence measures perform much better than most active learning approaches. In-particular, they propose a method termed DAL-E where examples are selected from the source domain based on similarity to *misclassified* target domain examples, and show that it outperforms standard “disciminative active learning” on most of their experiments. From analysis, it is revealed that selecting from diverse domains helps, and their proposed DAL-E can help with avoiding bad examples to be selected. \n", "summary_of_the_review": "The paper is well written and the experiments are useful for practitioners looking to apply active learning, however I think some of the analysis could be improved. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "m1H1fmwFja2", "reviewer": "Reviewer_EJMS", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Reasons to accept:\n\n- The study has a very ambitious goal. In the framework that the problem is defined, the authors, in my opinion, do a good job of defining sub-problems and validating the claims.\n\n- The paper is framed and written well. The study inherently has many parts, consequently this makes following up the content a bit difficult. But it appears that the authors have recognized this challenge and structured the paper to minimize this issue for the reader, e.g., summarizing the findings at the beginning of each section.\n\n- The paper is very detailed. Namely, the experiments are thorough, the discussions are extensive, and the references are comprehensive.\n\nReasons to reject:\n\n- My first concern is about the topic itself. As a person who has experience in text classification. Has done research in Active Learning and also in Domain Adaptation. I am not clear in what real world situations one might need to do active learning with multiple-source domain adaptation. The authors cite some papers as the studies in this area. Some I already knew, and some I specifically checked. But to my knowledge none of them actually did research in this exact topic. Please note that the problems of label shift or dataset shift are different from what the authors are proposing here. At the beginning of the introduction section the authors provide a very short description. But I am still not clear about this. On the other hand, if this problem applies to only some very specific settings, then why not specifically write the paper for those settings. Because, currently the paper claims that the proposed problem is a general and common challenge.\\\nAnother issue that further adds to the confusion is the data split used in the experiments. The authors assume each target domain already has several thousand labeled documents. So I can again ask, under what text classification scenarios there are several thousand labeled documents available but there is no unlabeled document available in that domain, so that we may be forced to use unlabeled documents from several other sources.\n\n- My second concern is regarding some of the methods used in the experiments. I understand that preparing such a paper is difficult, but still to me it was not initially easy to relate the KNN methods and active learning. Or H-divergence methods and active learning. Of course, KNN relates to representativeness metric and H-divergence relates to data sampling. But the way that the authors frame these methods implies that these are commonly used in active learning methods. I disagree with the authors, the connection is not initially obvious at all. Another method used in the paper, which is still unclear to me, is to rank data points in an ascending order of uncertainty. It is still not clear to me why one should select the points with most confident label predictions. this contradicts the active learning purpose.\n\n- My third concern is regarding the experimental setting: 1) The correct way of comparing active learning methods is to compare the learning curves. The authors only compare the final F1 measure. 2) None of the methods used in the paper are actual domain adaptation models. The authors simply aggregate the source and target data points to create a training set. This is not Domain Adaptation. 3) In H-divergence methods, authors use a decision tree-based classifier as discriminator. On the other hand the vectors are distributed representations, and the features are not independent. Tree-based classifiers assume feature independence.\n\n\nOther less severe issues:\n\n- Several questions that authors pose to empirically answer remained open and without answers, e.g., the questions in Section 6.2. There is nothing wrong with reporting failed expectations and failed methods. These are still informative. But when the number of these go up, it indicates that some of the choices were initially incorrect. This further confirms my concern about the lack of relation between some of the methods used in the paper and active learning.\n\n- When the authors describe the method BALD, they cite a paper and state that the paper used dropout to quantify uncertainty. The paper that actually used dropout for this purpose is a different one [1].\n\n- The two tasks used to evaluate the AL methods are very similar in nature.\n\n- Not sure why the authors have used bold faced fonts so frequently. They also have used red font a few times, which is not necessary.\n\n\n[1] Gal and Ghahramani. Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. ICML 2016.", "summary_of_the_paper": "The authors investigate the efficacy of several active learning-related techniques in classification under domain shift with multiple source domains. They include: uncertainty methods, H-divergence methods, reverse-classification methods, and nearest neighbor methods.\n\nThey construct 18 methods with various combinations and settings and carry out experiments in two text classification datasets: Question answering and sentiment analysis.\n\nThey report several findings, among them, the most interesting one to me is that uncertainty in various models manifests itself in various ways and candidate data points are not necessarily the same across models.", "summary_of_the_review": "The paper is an exploratory study. It is well written and covers a wide range of experiments. But I question the application of the task, also in my opinion the connection between the used methods and active learning should be further discussed. Additionally, there are a few problems with the experimental setup.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "5vj6M5jcn0T", "reviewer": "Reviewer_aTPg", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper makes comparison with techniques used in active learning (AL), domain shift detection (DS), and multi-domain sampling to combine data from multiple sources. The experiments are conducted on datasets from questions answering and sentiment analysis. \n\nThe paper is well organized and easy to follow. However, the contribution of this paper is not clear. Specifically, I would expect authors provide more detailed recommendation for AL, DS, and multi-domain sampling in terms of sampling techniques, and population of different sources for certain application.\n\nMy another is concern is that the motivation of the experimental design is not clear. Why authors consider questions answering and sentiment analysis as the applications? It requires more analysis about experimental results, such as Figure 1 and tables in Section D. \n\n", "summary_of_the_paper": "This paper makes comparison with techniques used in active learning (AL), domain shift detection (DS), and multi-domain sampling to combine data from multiple sources. The experiments are conducted on datasets from questions answering and sentiment analysis. \n\nThe paper is well organized and easy to follow. However, the contribution of this paper is not clear. Specifically, I would expect authors provide more detailed recommendation for AL, DS, and multi-domain sampling in terms of sampling techniques, and population of different sources for certain application.\n\nMy another is concern is that the motivation of the experimental design is not clear. Why authors consider questions answering and sentiment analysis as the applications? It requires more analysis about experimental results, such as Figure 1 and tables in Section D. \n\n", "summary_of_the_review": "This paper makes comparison with techniques used in active learning (AL), domain shift detection (DS), and multi-domain sampling to combine data from multiple sources. The experiments are conducted on datasets from questions answering and sentiment analysis. \n\nThe paper is well organized and easy to follow. However, the contribution of this paper is not clear. Specifically, I would expect authors provide more detailed recommendation for AL, DS, and multi-domain sampling in terms of sampling techniques, and population of different sources for certain application.\n\nMy another is concern is that the motivation of the experimental design is not clear. Why authors consider questions answering and sentiment analysis as the applications? It requires more analysis about experimental results, such as Figure 1 and tables in Section D. \n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "3BFF41AI5jc", "reviewer": "Reviewer_mKxw", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Unlike most of previous work on active learning (AL) focusing on a single domain, this paper targets the multi-domain AL setting, a more realistic setting. The paper presents an extremely thorough study of how different active learning methods (18 acquisition functions from 4 different families of methods) and rigorous experiments & analysis (correlation analysis among rankings of examples, searching for the best combination of source domains) provides insights to several key questions regarding multi-domain AL. \n\nOverall, the paper is very well written and I quite enjoyed reading it. However, with it being a great experimental report, my main concern is regarding the limited technical novelty of the work besides the two proposed simple variants (DAL-E derived from DAL-T, and RCA-smoothed derived from RCA, and all other are existing AL methods). Plus, part of the analysis on AL is also covered by previous work (e.g., Lowell et al., 2019). \n\nAlso in terms of providing a empirical study to multi-domain AL, it appears [1] also studies a similar problems and offers similar analysis. How much does the proposed work connect and differ from it? This is not discussed in the paper.\n\n[1] Multi-Domain Active Learning: A Comparative Study. He et al. https://arxiv.org/pdf/2106.13516v1.pdf (June 2021)", "summary_of_the_paper": "This paper studies active learning with multiple domains to select examples for two natural language tasks, sentiment analysis and question answering. By benchmarking 18 acquisition strategies from 4 families, the paper shows that overall the H-Divergence methods achieves the best improvements overall (including a proposed variant DAL-E, which considers both domain selection and example selection). ", "summary_of_the_review": "+ Good summary and thorough analysis of experiments studying 18 AL methods (2 of which are proposed variants) on 2 NLP tasks\n+ Paper is well written\n\n- Limited technical novelty (the two variants are simple modifications from existing methods)\n- Similar analysis exists but not discussed", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper considers the problem of active learning (AL) with data drawn from multiple domains. This framing motivates integrating work on domain shift detection and adaptation into standard AL approaches. 

The reviewers agreed that the work reports a robust set of experiments, which is a clear strength. However, they also raised key concerns, namely: (i) The heterogeneous setting considered is not particularly well motivated; (ii) The technical contributions of this work are limited. The latter would not be a major issue if the empirical evaluation addressed a clear open question (since this would constitute a useful contribution in and of itself), but the empirical contribution is somewhat limited given the unique setting considered and the relevant prior work (some of which seems to have been overlooked by the authors).
Decision: Reject