Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper makes comparison with techniques used in active learning (AL), domain shift detection (DS), and multi-domain sampling to combine data from multiple sources. The experiments are conducted on datasets from questions answering and sentiment analysis. 

The paper is well organized and easy to follow. However, the contribution of this paper is not clear. Specifically, I would expect authors provide more detailed recommendation for AL, DS, and multi-domain sampling in terms of sampling techniques, and population of different sources for certain application.

My another is concern is that the motivation of the experimental design is not clear. Why authors consider questions answering and sentiment analysis as the applications? It requires more analysis about experimental results, such as Figure 1 and tables in Section D. 


main_review: This paper makes comparison with techniques used in active learning (AL), domain shift detection (DS), and multi-domain sampling to combine data from multiple sources. The experiments are conducted on datasets from questions answering and sentiment analysis. 

The paper is well organized and easy to follow. However, the contribution of this paper is not clear. Specifically, I would expect authors provide more detailed recommendation for AL, DS, and multi-domain sampling in terms of sampling techniques, and population of different sources for certain application.

My another is concern is that the motivation of the experimental design is not clear. Why authors consider questions answering and sentiment analysis as the applications? It requires more analysis about experimental results, such as Figure 1 and tables in Section D. 


summary_of_the_review: This paper makes comparison with techniques used in active learning (AL), domain shift detection (DS), and multi-domain sampling to combine data from multiple sources. The experiments are conducted on datasets from questions answering and sentiment analysis. 

The paper is well organized and easy to follow. However, the contribution of this paper is not clear. Specifically, I would expect authors provide more detailed recommendation for AL, DS, and multi-domain sampling in terms of sampling techniques, and population of different sources for certain application.

My another is concern is that the motivation of the experimental design is not clear. Why authors consider questions answering and sentiment analysis as the applications? It requires more analysis about experimental results, such as Figure 1 and tables in Section D.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies active learning with multiple domains to select examples for two natural language tasks, sentiment analysis and question answering. By benchmarking 18 acquisition strategies from 4 families, the paper shows that overall the H-Divergence methods achieves the best improvements overall (including a proposed variant DAL-E, which considers both domain selection and example selection). 
main_review: Unlike most of previous work on active learning (AL) focusing on a single domain, this paper targets the multi-domain AL setting, a more realistic setting. The paper presents an extremely thorough study of how different active learning methods (18 acquisition functions from 4 different families of methods) and rigorous experiments & analysis (correlation analysis among rankings of examples, searching for the best combination of source domains) provides insights to several key questions regarding multi-domain AL. 

Overall, the paper is very well written and I quite enjoyed reading it. However, with it being a great experimental report, my main concern is regarding the limited technical novelty of the work besides the two proposed simple variants (DAL-E derived from DAL-T, and RCA-smoothed derived from RCA, and all other are existing AL methods). Plus, part of the analysis on AL is also covered by previous work (e.g., Lowell et al., 2019). 

Also in terms of providing a empirical study to multi-domain AL, it appears [1] also studies a similar problems and offers similar analysis. How much does the proposed work connect and differ from it? This is not discussed in the paper.

[1] Multi-Domain Active Learning: A Comparative Study. He et al. https://arxiv.org/pdf/2106.13516v1.pdf (June 2021)
summary_of_the_review: + Good summary and thorough analysis of experiments studying 18 AL methods (2 of which are proposed variants) on 2 NLP tasks
+ Paper is well written

- Limited technical novelty (the two variants are simple modifications from existing methods)
- Similar analysis exists but not discussed
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper surveys a broad range of techniques for active learning in the multi-domain setting applied to text - specifically, given a small labeled dataset from a target domain and a large amount of unlabeled data from a collection of source domains, how should examples be picked from the source domains to for labeling to maximise performance on the target domain. The main findings of the work are that selecting examples based on H-divergence measures perform much better than most active learning approaches. In-particular, they propose a method termed DAL-E where examples are selected from the source domain based on similarity to *misclassified* target domain examples, and show that it outperforms standard “disciminative active learning” on most of their experiments. From analysis, it is revealed that selecting from diverse domains helps, and their proposed DAL-E can help with avoiding bad examples to be selected. 

main_review: Overall, I like this work since it serves as a guide for practitioners faced with similar settings, and gives good guidelines on what works in these challenging domain shift settings. The experiments are  extensive and cover many of the domain shift settings in NLP. However, I think some of the analysis could be improved.

Weaknesses: 
- I like the analysis on measuring how different methods rank examples. However, I would highly recommend performing some kind of normalization - e.g. to compare method a1 from family A and method b1 from family B, kendall’s tau score should be normalized w.r.t the score between methods from the same family. 
- I would like to challenge some of the claims made in Section 6.2 of the paper. First, while it does seem like a diversity of examples helps, the boost is *very minimal* and it is certainly not true in general. Indeed, one can construct a domain distribution where getting data from several domains could hurt performance, if the mapping function (or the underlying P(y | x)) changes between domains.

Based on these weaknesses, I think the paper is currently borderline. However, if these are addressed i’m happy to adjust my scores. 
summary_of_the_review: The paper is well written and the experiments are useful for practitioners looking to apply active learning, however I think some of the analysis could be improved.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors investigate the efficacy of several active learning-related techniques in classification under domain shift with multiple source domains. They include: uncertainty methods, H-divergence methods, reverse-classification methods, and nearest neighbor methods.

They construct 18 methods with various combinations and settings and carry out experiments in two text classification datasets: Question answering and sentiment analysis.

They report several findings, among them, the most interesting one to me is that uncertainty in various models manifests itself in various ways and candidate data points are not necessarily the same across models.
main_review: Reasons to accept:

- The study has a very ambitious goal. In the framework that the problem is defined, the authors, in my opinion, do a good job of defining sub-problems and validating the claims.

- The paper is framed and written well. The study inherently has many parts, consequently this makes following up the content a bit difficult. But it appears that the authors have recognized this challenge and structured the paper to minimize this issue for the reader, e.g., summarizing the findings at the beginning of each section.

- The paper is very detailed. Namely, the experiments are thorough, the discussions are extensive, and the references are comprehensive.

Reasons to reject:

- My first concern is about the topic itself. As a person who has experience in text classification. Has done research in Active Learning and also in Domain Adaptation. I am not clear in what real world situations one might need to do active learning with multiple-source domain adaptation. The authors cite some papers as the studies in this area. Some I already knew, and some I specifically checked. But to my knowledge none of them actually did research in this exact topic. Please note that the problems of label shift or dataset shift are different from what the authors are proposing here. At the beginning of the introduction section the authors provide a very short description. But I am still not clear about this. On the other hand, if this problem applies to only some very specific settings, then why not specifically write the paper for those settings. Because, currently the paper claims that the proposed problem is a general and common challenge.\
Another issue that further adds to the confusion is the data split used in the experiments. The authors assume each target domain already has several thousand labeled documents. So I can again ask, under what text classification scenarios there are several thousand labeled documents available but there is no unlabeled document available in that domain, so that we may be forced to use unlabeled documents from several other sources.

- My second concern is regarding some of the methods used in the experiments. I understand that preparing such a paper is difficult, but still to me it was not initially easy to relate the KNN methods and active learning. Or H-divergence methods and active learning. Of course, KNN relates to representativeness metric and H-divergence relates to data sampling. But the way that the authors frame these methods implies that these are commonly used in active learning methods. I disagree with the authors, the connection is not initially obvious at all. Another method used in the paper, which is still unclear to me, is to rank data points in an ascending order of uncertainty. It is still not clear to me why one should select the points with most confident label predictions. this contradicts the active learning purpose.

- My third concern is regarding the experimental setting: 1) The correct way of comparing active learning methods is to compare the learning curves. The authors only compare the final F1 measure. 2) None of the methods used in the paper are actual domain adaptation models. The authors simply aggregate the source and target data points to create a training set. This is not Domain Adaptation. 3) In H-divergence methods, authors use a decision tree-based classifier as discriminator. On the other hand the vectors are distributed representations, and the features are not independent. Tree-based classifiers assume feature independence.


Other less severe issues:

- Several questions that authors pose to empirically answer remained open and without answers, e.g., the questions in Section 6.2. There is nothing wrong with reporting failed expectations and failed methods. These are still informative. But when the number of these go up, it indicates that some of the choices were initially incorrect. This further confirms my concern about the lack of relation between some of the methods used in the paper and active learning.

- When the authors describe the method BALD, they cite a paper and state that the paper used dropout to quantify uncertainty. The paper that actually used dropout for this purpose is a different one [1].

- The two tasks used to evaluate the AL methods are very similar in nature.

- Not sure why the authors have used bold faced fonts so frequently. They also have used red font a few times, which is not necessary.


[1] Gal and Ghahramani. Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. ICML 2016.
summary_of_the_review: The paper is an exploratory study. It is well written and covers a wide range of experiments. But I question the application of the task, also in my opinion the connection between the used methods and active learning should be further discussed. Additionally, there are a few problems with the experimental setup.
review_title: null

Meta-review: This paper considers the problem of active learning (AL) with data drawn from multiple domains. This framing motivates integrating work on domain shift detection and adaptation into standard AL approaches. 

The reviewers agreed that the work reports a robust set of experiments, which is a clear strength. However, they also raised key concerns, namely: (i) The heterogeneous setting considered is not particularly well motivated; (ii) The technical contributions of this work are limited. The latter would not be a major issue if the empirical evaluation addressed a clear open question (since this would constitute a useful contribution in and of itself), but the empirical contribution is somewhat limited given the unique setting considered and the relevant prior work (some of which seems to have been overlooked by the authors).
Decision: Reject