Active Learning for Multiple Target Models
Authors: Ying-Peng Tang
Sheng-Jun Huang
Venue: NeurIPS 2022 Accept (2022)
Keywords: active learning
machine learning

Abstract: We describe and explore a novel setting of active learning (AL), where there are multiple target models to be learned simultaneously. In many real applications, the machine learning system is required to be deployed on diverse devices with varying computational resources (e.g., workstation, mobile phone, edge devices, etc.), which leads to the demand of training multiple target models on the same labeled dataset. However, it is generally believed that AL is model-dependent and untransferable, i.e., the data queried by one model may be less effective for training another model. This phenomenon naturally raises a question "Does there exist an AL method that is effective for multiple target models?" In this paper, we answer this question by theoretically analyzing the label complexity of active and passive learning under the setting with multiple target models, and conclude that AL does have potential to achieve better label complexity under this novel setting. Based on this insight, we further propose an agnostic AL sampling strategy to select the examples located in the joint disagreement regions of different target models. The experimental results on the OCR benchmarks show that the proposed method can significantly surpass the traditional active and passive learning methods under this challenging setting.

Active Learning for Multiple Target Models
Ying-Peng Tang and Sheng-Jun Huang
College Computer Science and Technology, Nanjing University of Aeronautics and Astronautics Collaborative Innovation Center of Novel Software Technology and Industrialization MIIT Key Laboratory of Pattern Analysis and Machine Intelligence, Nanjing 21 106, China {tangyp, huangsj /nnuaa enuaa edu.
Abstract
We describe and explore novel setting of active learning (AL), where there are multiple target models to be learned simultaneously. machine learning system is required to be deployed on diverse devices with varying computational resources (e.g., workstation, mobile phone, edge devices), which leads to the demand of training multiple target models on the same labeled dataset. However, it is generally believed that AL is model-dependent and untransferable, i..., the data queried by model may be less effective for training another model. This phenomenon naturally raises a question "Does there exist an AL method that is effective for multiple target models?". In this paper, we answer this question by theoretically analyzing the label complexity active and passive learning under the setting with multiple target models, and conclude that AL does have potential to achieve better label complexity under this novel setting. Based on this insight, we further propose agnostic AL sampling strategy to select the examples located in the joint disagreement regions of different target models. The experimental results the OCR benchmarks show that the proposed method significantly surpass the traditional active and passive learning methods under this challenging setting.
many real applications, the
Introduction
Data labeling is expensive due to the involving of human annotator. Active learning (AL.) is one of the main approaches to reduce the labeling cost (28). evaluates the utility of the unlabeled data based on the target model, and actively queries the labels from the oracle for the examples that is the most beneficial to the performance improvement of the target model.
Existing active learning methods assume that there is only one specific target model, and try to fit it with least queries. However, many real applications, the machine learning system required to be deployed on multiple types of devices with different resource constraints [6]. For example, speech recognition software needs to support diverse machines with varying hardware efficiency, ranging from high-performance workstation to the mobile-phone. Due to the different computational
resources, the applicable model architectures vary a lot, e.g., deep model which well-performed on the cloud server may not be deployed on the edge device. It thus raises the demand of training multiple models with different complexity to accommodate these devices.
Given multiple target models, how to effectively improve them with least labeled data becomes practical and challenging problem. It is generally believed that AL is usually model-dependent and untransferable [22 24 38
i.e., the best query strategy for different target models varies a lot [39]. In other words, the data queried by one model may be less effective for training another model [22] These observations imply that the existing active query strategies can hardly benefit all target models simultaneously, and the design of AL algorithm for multi-models can be rather difficult. natural
"Corresponding author: Sheng-Jun Huang <huangsj @ edu.en...
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

question might be asked: "Does there exist an active learning method which queries a set of labeled data, such that all the target models can be effectively trained with them?"
In this paper, we formally define the active learning for multiple target models problem, and reveal the potential improvement of AL under this novel setting. Based on this insight, we further propose an agnostic setggreement-baseee selection criterion. Specifically, we first define and analyze the label complexity for both active and passive learning under the setting with multiple target models. This label complexity characterizes the number of labeled examples sufficient to train an é-good classifier 8 for each target model. Moreover, we find that the label complexity
with probability at least of single model has a close relation to that of multiple models under the realizable case, e.g., the former provides an upper bound of the label complexity for multiple models, which also implies the potential improvement of AL under this setting. To further explore the agnostic case, we propose an active selection method DIAM (i.e., Disagreement-basee select the best examples that are beneficial to all target models. It prefers the data located in the joint disagreement regions of different models, which expected to have higher potential to reduce the soft version space (i.e., the set of hypotheses with lower errors). Experiments are conducted on the OCR benchmarks to validate the necessity of designing active query method under this practical setting and the effectiveness of the proposed approach. The results show that the DIAM method can significantly reduce the number of queries to achieve a higher mean accuracy for multiple models compared to the traditional active and passive learning methods.
AI. for Multi-models) to effectively
The rest of the paper is organized as follows. related work is first reviewed in the following section, then we formally define the AL for multiple target models problem and provide a general result to bridge the label complexity between single and multiple models. Then, we reveal the potential improvement of AL under this novel setting. After that, an agnostic active selection criterion is proposed and analyzed, followed by the empirical studies, And at last, we conclude this work.
2 Related work
Active learning has received much attention in recent years due to the greatly increasing demands of labeling data effectively train more complex models (c.g., deep models) [25]. One of the cores of AL is how to evaluate the potential contribution to the performance improvement of the target model for each candidate query. Most of existing criteria for active learning can be categorized into informativeness and representativeness. The informativeness-base data which is near the decision boundary, and the representativeness- based methods [2 30 21
methods [I3] 36 17 prefer the
impose the constraints to regularize the queried data to be dissimilar with each other or conform to the latent data distribution. Many works also try to combine both criteria to obtain better performances Beyond these hand-crafted selection criteria, several meta-active-learning methods are proposed to learn generalizable query strategy across tasks. Most of the existing active learning query strategies target improving one specific target model. They are less applicable to the multiple target models setting.
11
37
31
18
23
35
From the thcoretical view, active learning theory has also been widely studied under certain conditions (e.g., binary classification, finite VC dimension) [16). One of the interested properties of an active learning algorithm is the label complexity, which characterizes the number of queries needed to obtain an E-good classifier with probability at least
[4. To bound this value, disagreement coefficient [5] 4l and Shattering 7 are two commonly used techniques. While most works deal with the single model setting, Balcan el al. 3 study the label complexity of the hypothesis space and its subclasses, which sheds light on this work. However, they mainly focus on how to construct subclasses to achieve a certain label complexity, while we aim to find an effective active learning algorithm the given target models.
Recently, some AL studies tackle related problem that the target model prior cannot be obtained. In this setting, they will not only search the appropriate target model for the current task, but also avoid noneffective querying. To this end, ALMS ① either randomly labels data to calculate the unbiased validation error for model selection, or queries by the expected error reduction to improve the models. Active-iNAS [2 considers the deep learning setting, the authors on one hand perform Neural Architecture Search (NAS) to find the appropriate model architecture, on the other hand query the examples by the searched network. Recently, Tang and Huang (2) propose a unified framework DUAL to solve this problem. They query the data that is beneficial to not only the winner model, but

---

also the model search to identify the high potential model with least queries. All these methods try to search effective model configurations, but not improve multiple target models, which are different from our work.
3 Label Complexity of Single Model and Multiple Models
3.1 Notations and Definitions
Suppose the data is sampled from an unknown distribution Dxy over the feature space X' and label space V. Denote by Dx the marginal data distribution, and Dy the marginal label distribution. Given a dataset with n instances, which includes a small labeled set C = { 3r) ha) with instances, and large unlabeled set U (æ; >est with n. instances, where & nu and n = ny + fuy At each iteration, the active learning method will select a batch of examples Q from for querying-
In the single model setting, we are given a hypothesis space C before querying. While in the multiple ..
target models setting, there are k hypothesis spaces T = {Cali = ..,. with T = =U C.. our goal is to actively query set of examples to output a well-performed hypothesis h2 from each C, Vi = 1..... k. We define the true enror of a hypothesis as er(h) = IP Px (h(x) o h (x)), where h is the target concept. h(x) is the model prediction on the data æ. The empirical error of on L is defined by er ch) = [ Emee L[h(xz) * h- (z)), where 1[-] is the indicator function. Let mingee er(h), and i minsec, (r()), Log(a) max (In(), 13, Va 0.
Here we introduce the definition of the pseudo- metric between hypotheses, which is frequently used in the subsequent proof.
Definition Pseudo-metric between Hypotheses: Given Dx the probability of disagreement between two classifiers hn and liz is defined as ((hn,ha)) = P Px (1)(2r) + h2(x))
Then we introduce the label complexity of AL for single target model [6].
Definition 2. Label Complexity of for Single Target Model: For any active learning algorithu A. we say A achieves label complexity A on the hypothesis space C if. for every E, E (0, 1), every distribution D DxY over X x y, and every integer t 2 A (c, 6, Dxy), if h4,6 s is the classifier produced by rumning A with budget t, then with probability at least 1 5, ch(,s))) - v S
Now we formally define the label complexity of active learning for multiple target models. It is defined on multiple hypothesis spaces, and the goal is to output an e-good classifier for each target model. Specifically, the label complexity for the AL with multiple target models is defined as
Definition 3. Label Complexity of for Multiple Target Models: Given a set of target models For any active learning algorithm A, we say A achieves label complexity
T= =Calli = 1,2.....k) ,k). À for multiple target models if. for every e, ò E (0, every distribution Dxy over X y,, and every integer t 2 Å (e, 6, Dxy, TT), if (h:6 E C,li 1..... kb} is the classifiers produced by running A with budget t, then with probability at least 1 - 5, er t.b v S = = L,..,, k.
In the following, learning, and use the notations A ALL, A and AA P to distinguish the label complexity of them, respectively. We hide the superseript when the context is clear.
take the passive learning (PL), i.c., random sampling, as trivial case of active
3.2 Translating the Label Complexity of Single Model Multiple Models
Denote by A, the label complexity of the i-th target model C. Trivially, the AL label complexity for multiple models has AL < Σ, A," (applying the AL algorithm A on each of the target model i to get the result). For the passive learning, however, its label complexity for multiple models has a much tighter upper bound Á PT. < max, APT. Because the data randomly sampled, if t ' max, APL (e, 6, >xy) examples are queried, according to the definition of label complexity, passive learning will output an E-good classifier with probability at least 1 - for each target model. Such result implicitly indicates that the AL can hardly outperform PL under this setting.
To break this curse, the following theorem is provided to show that, we can expect much better Ä AL for AL under the realizable case (i.c., h" is in the combined hypothesis space T). It generally says
3

---

that, given arbitrary set of target models T = [ = 1, 1.2.....k) if a learning method has label complexity A AAL on the combined hypothesis space T, it also has the ability to output good classifiers for each C, i.e. after querying at most t examples to output s-good classifiers with probability at least 1 for each C.,
Theorem 1. Considering binary classification tasks and realizable case, given target models T = {C,li = 1.2...., k), assume that active learning algorithm A achieves label complexity A on T. Then, there exists an active learning algorithm. A' which achieves the label complexity À such that (c, D T) = A (E/2, 6, )
Proof. Define an algorithm A'' that can output the required classifier hús E First, run the algorithm A on (T, Dxy) to query 2 A(s/2, 6, Dxy) labels and output a classifier hA- According to the definition, d(h h²) bounded by 2/2 with probability least 5. Then, for any C, output the classifier hú E C, such that h2 = arg minn,ec, d(h,, hA). Next, we prove that er(hu) vs S & holds with probability at least
& as follows.
To bound er er(h.). it is equivalent to bound dih;, h²) by Definition | Let h = arg mins, ec, er(h;). is easy to verify that, d(-) satisfies triangle inequality in binary classification problems, i.e.,
d(h, hi") S d(hi, hA) + d(h,,h")
For the d(hi, hA), know that h, arg min d(bs; ha), which means
d(h, hA) S d(h,,
Again, by the triangle inequality, we have
d(h?, ha) S d(h;, h²) + d(h", hA).
(3)
Combining Eq. 2 we have
d(h;, h²) d(h;, h²) + 2d(ha,h*)
(4)
Since d(h,,. h²) is bounded by e/2 with probability least 1 8, get er(h)) vs S holds with probability at least 5.
Theorem says that if we can find an active learning method to obtain a classifier h E ? such that el(ha) < e/2 with probability 1 - 8, then we can obtain s-good classifier h, with probability at least 1 - 5 for C,, Vi = 1, k, where h, arg minA,ec; d(h;, ha). This result provides general guarantee that if an algorithm can achieve a label complexity on the combined hypothesis space of different models, it also can achieve bounded label complexity these models (i.e., the label setting.
complexity for multiple models). It can be served as a baseline AL under
Potential Improvements of Active over Passive
Although Theorem I bridges the traditional label complexity that of multiple models setting, it does not reveal the improvement of active over passive learning. Next, we will show the potential of AL under this setting in the realizable case.
According to the theoretical analysis of the passive learning algorithm empirical risk minimization (ERM) [16] for single hypothesis space C with VC dimension d (34), we know that
Lemma 1. Considering the binary classification, given the hypothesis space C with VC dimension The passive learning algorithm ERM achieves label complexity A such that, for any Dxy v in the realizable case, Ve, E (0, 1),
( (dTog(((e)) + Log(1/8))
P (e, 8, DxD)
(5)
For the agnostic case, ERM achieves label complexity A PI. such that,
A PL t,,
(
(
Log(((( - E)) + Log Log(118)) (1/8)),
(6)
52

---

where (()) is the disagreement coefficient which is formally defined as Definition 4. Disagreement Coefficient: For any ro 2 O and classifier h, define the disagreement coefficient of h with respect to C on Dxy as
((Dis(B((brr)) VI.
0% (ro) sup rpro
r
Where V is the max operator For set of hypotheses H. DIS(H) = {x E X 3h, h' E H. s.I. h(x)) + 1 (.)), and (b) = t0 H ((h,g) rr
This value roughly characterizes the behavior of the size of disagreement region DIS(.) function of the hypotheses within radius around the classifier h. As aforementioned, passive learning has the label complexity for multiple models A I max; A!". We note that the target concept h will usually not be included by every hypothesis space C, thus its label complexity A will usually be the agnostic form in Lemma under the realizable case.
To show the potential of AL. under this setting, we take the CAL method 9 as an example, which is representative and well-analyzed approach in the active learning literature [16]. CAL queries the examples from the disagreement region of a set of consistent hypotheses, i... DIS(V) { E X1 3h, h' V s.f. h(x) A h h.))), where V { E C C1((2)
h- (æ)), Ve E L}. It achieves the label complexity O(0(s) log(1/s) (s) log(1/s) (1/))) for the realizable case. According to Theorem T it will have the following label complexity for the multiple target models
Corollary 1. Given target models T = {Cali = 1,2,....,AA k.). Suppose T has VC dimension d < co. CAL achieves a label complexity AL for multiple target models such that, for D Dx in the realizable case, for any VE,8 8 E (0,1).
AL (x T) s (12 (1/2)) + Log (Log(2/E) ))
(Lor) 8
Proof. By combining the label complexity of CAL for single model from [6 and Theorem we can get the result.
To reveal the potential improvement, note that the label complexity for passive learning heavily depends on the property of the worst hypothesis space, i.e., the value of max; minnec, er(h). Assume that max; minkec, er > E. Then according to Lemma 1 and Corollary [ the label complexity of passive learning for multiple target models P (e. 6, Dxy, T) ((2/e). On the other side, CAL has label complexity ((Log(2/e)) which implies the potential of the improvement of active learning under this setting. We leave the guarantee of strict improvement of AL under this setting an interesting future work. Next we further study the agnostic case (i.c., h $ 7).
5 An Agnostic Disagreement-based AL method for Multiple Models
Define the set V; for each C, as (h E C. h(())= = (x)) Vx: E L). we propose to query the examples located in the joint disagreement regions for all C., Vi 1, 1,2...., k. i.e., DIS(Vi) n DIS(V)) ...DIS(V)). ). Intuitively, we know that V. must be subset of V. if such data exists, we can expect it has higher potential to reduce V. This statement can be simply implied by the Bayesian formula.
Proposition 1. Considering binary classification problem. Given hypothesis space C. Let V, (x) {h E V h((x) = +1}, V_(x) = {h E V [((x)===>>>))))(<)))) = -1}, where V = {h E C h(xx) = (x). Vx e (z) where | is the number of elements of set. The ideal case is to query the x which has A(x) =1 Given any sequences subset V1, Vz...., V& randomly sampled from
L}. Denote X(x) V4 (x)]
V. define the event E,z that data x falls into DIS(V)) n DIS(V)) DIS(V&). According to the
Bayesian formula, we have
((((x)) 1|Ex) P(Ez X(x) = 1)P({(<)) 1)
P(Em)
(8)
2 p(((T) = =1).
5

---

Algorithm The DIAM-online Algorithm
Algorithm The DIAM-pool Algorithm
Initialize: hyperparameter
constants
Initialize: labeled set L, unlabeled set U, hyper- parameters o;, V; t- C, vi = 1....,k. k. ,.....,.. k}.
m < 0, V2 + C, Vi = =1, ....,.. Output: Any E V, Vi = I.
Output: {h:le
1: while Labeling budget is not run out do
1: while Labeling budget is not run out do
2: m < m7 +1
2:
x² = arg max Σ, I[ E DIS(
3: Request an unlabeled data &m
aEll
4: if Σ I[mm E DIS(V)) 2 q then
3: Query x" from the oralce: + CU
((x h² (x))))}
5: Query: C CU {I h (2m)))
6: end if
4:
U +- (x*)
for i = -l...., do
7:
if logz m
then
h min seR erc(g)
8: V
f {h E (h)erc(h)
min gEV: R. crc(g) oz), vi = k.
+ (h V.l(erc(h) - h.) {
9:
end if
8: end for
10: end while
9: end while
Proof. Since each V, is randomly sampled from V. ((Ex)) will reach its maximum value when ((x) = thus we have P(E (x(=)= = (E,) 2 1, which leads to the conclusion.
Following this principle, we would like to query the examples located in the joint disagreement regions of C, vi 1,2..... .,, k. However, since we have multiple target models, the target concept h' might not be included by every C, in practice, which turns the learning problem to the agnostic setting. Inspired by the RobustCAL method [2, which is disagroement-basg based AL algorithm for the agnostic setting, we propose DIAM (i.e., Disagreement- based AL for Multi-models query strategy for the multiple target models problem. Note that we define a new form of Vú2 as V. to tackle the noisy setting, i.c., V, {h E C, erc(h) mingec, (g((g) S J. where o; is constant. To simplify the theoretical analysis, we first propose an online version of DIAM, then we define the DIAM method for the pool-based setting and empirically validate its effectiveness. They are summarized at Algorithm
T and respectively. The hyperparameter & controls the conservativeness of the algorithm. With a larger q, it will reject more less informative unlabeled data in the online setting.
Now let us analyze the DIAM method. Since we are considering the agnostic setting, is necessary to model the noise. Here we employ the commonly used Tsybakov noise condition [3
Condition [33 Tsybakov noise] For some oo) and a E [0, 1]. assume that f achieves
inf (h)),, ),,, for every h E C,
P (xx: : Mxx) # f*)))) S a (cr(h) (f**))
We assume that there exists pair of a; and a; for each target model C, Considering conservative situation that the hyperparameter g = 1, by further taking the constants a; in DIAM-online algorithm as the same form in the RobustCAL method [16), which relates to the properties of the noise, hypothesis space, and disagreement coefficient, we can have the following result. The proof is deferred to the appendix.
Theorem Considering agnostic setting and binary classification tasks. Given set of target models
T = ic,li = 2,2,..., h, in which each C, has VC dimensions d; < со and meet Condition Let h = arg mins.ec, cr(h,). For any E, E (0, 1), if = 1, DIAM-online algorithm achieves label complexity (e, Dxy, () for multiple target models such that, for ag and as as Condition for any Dxv, A (e, Dxx,T T) is no larger than
(Log(a/EE)
)))) Lo
Σ 2706 ( e 2a,-2 [oLogg Log (0h; (are + Log
(9)
and no larger than,
Log (1)) de Log(0%; (ur E)) + Log
( (oo((//)))
Σ c (vi +
(10)
5

---

Theorem 2 provides an upper bound of the label complexity of the DIAM-online method when 1. considers general situation with arbitrary target models and data distributions, even the unlabeled data will never fall into the joint disagreement regions. However, one may be more interested in the situation that if can always query the x such that >. 1/ E DIS(V;) = k. Next, we prove that if such ideal situation exists, DIAM-online will achieve a better label complexity than applying CAL on the multiple target models setting even under the realizable setting.
Theorem 3. Considering binary classification tasks and realizable case. Given set of target models T= = [C,li- ..,...., k). in which each C, has VC dimensions d: < oo and meet Condition | and T with VC dimension d < co. Assume that, if a data point falls into the disagreement region of any C. it also falls into the disagreement regions of the others {C₂ * i,j = 1, ...,,., k}. Assume the m-th target model achieves the highest label complexity. Let h = arg mill) EC.. er er(hm and Um = ee(hmm) For any E (0, 1), & E (0, (0,1/e), h" E If Pm < 2 DIAM-online achieves a better upper bound of Å than that of applying method on T.
The key of the proof is comparing the disagreement coefficients defined on different functions and hypothesis spaces, i.e., ff and of. We defer the proof to the appendix. Although Theorem holds with somewhat strict conditions, we note that Theorem oonyy works in the realizable case, while DIAM does not require this condition. Next, we discuss how to implement DIAM in the real applications for deep models.
generally believed that finding the disagreed pair of classifiers from a set of hypotheses for a given æ is non-trivial. Most existing methods randomly sample functions from the hypothesis space for validation, or turn to select the data close to the decision boundary (e.g., uncertainty); which can be expensive or inaccurate. This problem becomes more prohibitive to the deep models.
To efficiently estimate the disagreement regions for the neural networks, we propose exploit the predictions of the unlabeled data during the later epochs in the training phase, typically after the network converging. Recall the definition of disagreement region DIS(V)), we should firstly find the hypotheses that are basically consistent with the labeled data, then validate whether there exists a pair of hypotheses that disagree the given unlabeled data. For the first characteristic, the models on the later epochs, i.e., has smaller training errors, can represent the well-learned hypotheses. For the second characteristic, if there exists models i,j from the later epochs such that the model trained at epoch has inconsistent prediction with the model trained at epoch j on the unlabeled data æ, we can say that the example æ falls into (V,
More concretely, according to the training loss curve, we empirically take the models in the latter half of the training epochs as the well-performe with them. Since there may be multiple examples have the same value of Σ; I| E DIS(V.)), we further calculate the vote entropy [00] of the well-performed hypotheses on the specific data, and take as the secondary sort key in the data selection phase. We also note that the query batch size in deep learning is usually large. (T avoid overmuch information redundancy, top-rated unlabeled data with 5 times the size of the batch size, and randomly sample 20% from it for querying. We leave the more advancee diversity measurement future work.
hypotheses set, and estimate the disagreement region
heuristically keep the
By applying the above heuristics to the deep models, DIAM method quite efficient. evaluates the unlabeled data with the models trained with later epochs, which roughly takes the size of the well- performed hypotheses set times of that of the entropy method to make the data selection. However, we also note that the DIAM method has some limitations. It only considers the informativeness of the unlabeled data, which may be less effective for the batch-mode selection. For the potential negative social impact, DIAM may reduce the cost of training multiple malicious machine learning models. Nevertheless, we believe the positive contribution is more significant.
6 Experiment
6.1 Empirical Settings
To construct the multiple target models scenario, we introduce the results of a recent NAS method OFA (6. which tries to efficiently search model architectures for different devices by training only one super-net. They report the searched effective architectures that meet the hardware constraints
7

---

95
98
90
mean accuracy
mean accuracy cy
85
96
80
DIAM
DIAM
94
CAL
75
CAL
Entropy
Entropy
70
92
Least conf.
Least conf.
Coreset
65
Coreset
Random
Ran dom
90
60
3000
6000
9000
12000
15000
3000
9000
15000
number of queries
number of querles
(a) MNIST
(b) Kuzushij-MMNIST
Figure 1: The learning curves with the mean accuracy of the target models of the compared methods. The error bars indicate the standard deviation of the performances of target models.
of various machines on the GitHub D which is well suited to our problem setting. Specifically, we take 12 specialized model architectures with different prediction accuracies and speeds that target on Samsung S7 Edge, Samsung Note8 and Samsung Note 10 as target models. They are pruned from a MobileNetV3 (which the super-net), but have very different prediction time and accuracies. Their Multiply-Accumulat Operations (MACs) range from 66M to 237M which denote the diversity of the architectures. The model specifications are listed in the appendix.
We compare the following query strategies in our experiments.
DIAM: The proposed method this paper, which queries the data located in the joint disagreement regions of multiple target models.
CAL [9:
Query the data falls into the disagreement region of any target models. It has bounded label complexity for the multiple target models setting according to Theorem
Query the data with the highest prediction entropies. We take the mean entropies calculated by all target models to support the novel problem setting.
Entropy 20
Least Confidence 29 Query the data with the least prediction confidence. We take the mean values calculated by all target models to support the novel problem setting.
Query the data with the minimum prediction margin. We take the mean margin values calculated by all target models to support the novel problem setting.
Margin 26
Query the most representative data. The distance is calculated by the features extracted by pretrained MobileNetV3, which is the super-net OFA [
Coreset [7]
Random: Query data randomly. Note that this is highly competitive baseline.
Since Optical Character Recognition (OCR) is of the representative machine learning systems that are required to be deployed on diverse devices, two commonly used hand-writing characters classification benchmarks
employed in experiments, i.e., the MNIST [99] and Kuzushiji- MNIST [ datasets. They are under the CC BY-SA 3,0 and CC BY-SA 4.0 licenses, respectively. Here we consider the prevalent pool-based active learning setting. Specifically, we randomly take 3, 000 training data as our initially labeled data, and the rest as the unlabeled pool. At each iteration, the compared sampling methods will select 500 unlabeled examples for querying, then re-train the models. The mean and standard deviation of the accuracies of multiple target models are reported. Note that more results can be found in the appendix.
For the model training, We mainly follow the training configs of OFA. Specifically, the hyperparam- the default values in the project. For example, the learning rate is set by 7.5e batch size is 128, SGD optimizer is employed with momentum 0.9. Since the initially labeled data is limited, a small number of training epochs taken avoid over-fitting. Specifically, we employ the
httpp: bitsps://githhul comm/iit.mommmmmmmm/// han- http://tttttaa--laaaaeo----tt-o
It is under the MIT license.

---

Table 1: The mean of the learning curves and the mean of standard deviation values with different numbers of target models on the OCR benchmarks achieved by the compared methods (mean accuracy mean standard deviation). The best performance is highlighted boldface.
Number of Target Models
Methods
4
6
8
12
MNIST
DIAM CAL
0.13
97.29
0.99
97.55 0.85
97.34 1.09
97.34 1.04
0.14
97.24 0.89 0.92
96.95 1.07
96.98 = 1.10
Entropy
97.83
9.09
97.15
96.92
96.98 = 1.00
Margin
97.79
9.02
0.96
9.19
97.00 = 1.02
Least conf.
07.8400 0.11
96.89 1.02
97.23 0.92
96.88
96.96 1.07
Coreset
97.64 + 0.13
96.69 1.07
97.03 + 0.97
96.36 + 1.82
96.56 - 1.40
Random
97.81 + 0.12
96.93 + 0.97
97.21 + 0.94
96.83 + 1.12
97.03 + 0.99
Kuzushiji
hIJUMSIIIMNNIST
DIAM
90.38 + 0.21
85.76 4.69
86.91 + 4.38
86.23 + 4.68
86.85 4.25
CAL
87.06 + 0.34
83.61 4.29
84.70
83.40 4.32
$3.31 4.53
Entropy
87.09 + 0.34
$3.22
84.39
4.28 83.28 + 4.28
83.33 4.33
Margin
86.91 = 0.35
83.20
4.003
83.11
83.16 4.29
Least conf.
86.71 = 0.26
83.38 #4225
8.77
83.20
83.04 4.33
Coreset
87.49 = 0.36
82.97 5.16
84.80
83.00 = 4.93
82.91 = 5.03
Random
87.34 + 0.31
82.97 + 4.38
84.22 + 3.98
83.02 + 4.19
83.26 + 4.36
pretrained weights on the image-net dataset for initialization, then finetune 20 epochs on the labeled data.
6.2 Results
We report the trend of mean accuracy of multiple target models with the number queries increasing in Fig. [T] The error bars indicate the standard deviation of the performances of multiple target models. First of all, the high deviation of the performances of the initial point shows the diversity of the target models, which symbolizes the practicability and difficulty of the experimental settings. It is conceivable that different target models will have various preferences of training data due to the diverse architectures. Under this challenging setting, it can be observed from the figure that method can significantly outperform the traditional active and passive learning methods. shows great potential of improvements over the random sampling, which is very competitive baseline in this novel setting. This result sufficiently reveals the effectiveness of DIAM and the necessity of designing active query method under this practical setting. The uncertainty-baat entropy, least confidence and margin, achieve comparable performances with random sampling. These results meet our expectation. Because traditional AL methods are usually model-dependent, i... the data queried by one model may be less effective for training another model. By taking the mean uncertainty scores of diverse target models, the data selection may tend be non-informative. The coreset method is less stable than random. We note that coreset is still a model- selection method in deep learning. Because the features of the data will be optimized along with the training procedures. Thus it may also suffer from the model dependence problem.
methods, i.e.,
6.3 Study on Different Numbers of Target Models
We further explore the influence of the number of target models to the data selection methods. Due to the space limitation, we report the mean of the learning curves and the mean of standard deviations in Table and defer the whole learning curves to the appendix. The results show that our method can consistently outperform the other compared methods, which demonstrate its robustness to the number of models. This property also denotes that the DIAM method has the potential tackle more challenging situations, i.e., improving sufficient numbers of target models simultaneously. It is essential to the machine learning systems which have a wide range of applications. The performances of the other compared methods have similar trends with more target models setting. It again verifies that the traditional AL methods are usually model-depend,ett, and emphasizes the necessity of designing novel selection approaches under this practical setting.

---

7 Conclusion
In this paper, we propose to study active learning novel setting, where the task is select and label the most useful examples that are beneficial to multiple target models. We firstly analyze the label complexity of active and passive learning to reveal the potential improvement of AL under this novel setting. Based on this insight, we further propose active selection criterion DIAM that prefers the data located the joint disagreement regions of different target models. Empirical studies on the OCR benchmarks, which is one of the representative applications that required to accommodate different devices, show the effectiveness of the proposed method. In the future, we will tackle more complex and important learning tasks (e.g., face recognition, object detection), and design effective query strategies which incorporate both informativeness and representativenes: under the multiple target models setting.
Acknowledgments and Disclosure of Funding
This research supported by the National Key R&D Program of China (200AAAA0107000,, NSFC (62222605, 62076128), and Natural Science Foundation of Jiangsu Province China (BK20211517, BK2022050029).
References
[1] Alnur Ali, Rich Caruana, and Ashish Kapoor. Active learning with model selection. In AAAI Conference Artificial Intelligence, pages 1673-1679, 2014.
[2] Maria-Florina Balcan, Alina Beygelzimer, and John Langford. Agnostic active learning. Journal of Computer and System Sciences, 75(1 897 2009.
[3] Maria-Florina Balcan, Steve Hanneke, and Jennifer Wortman Vaughan. The true sample complexity of active learning. Machine Learning. 80(2):111-139, 2010.
[4] Alina Beygelzimer, Daniel J. Hsu, John Langford, and Chicheng Zhang. Search improves label for active learning. In Advances in Neural Information Processing Systems, pages 3342-3350, 2016.
[5] Alina Beygelzimer, Daniel J. Hsu, John Langford, and Tong Zhang. Agnostic active learning without constraints. In Advances Neural Information Processing Systems, pages 199-207, 2010.
[6] Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, and Song Han. Once-for-all: Train one network and specialize it for efficient deployment. In International Conference on Learning Representations, 2020.
[7] Xiaofeng Cao and Ivor W Tsang, Shattering distribution for active learning. IEEE Transactions on Neural Networks and Learning Systems, 33(1):215-228, 2022.
[8] Tarin Clanuwat, Mikel Bober- Asanobu Kitamoto, Alex Lamb, Kazuaki Yamamoto, and David Ha. Deep learning for classical japanese literature, 2018.
[9] David Cohn, Les Atlas, and Richard I Ladner. Improving generalization with active learning. Machine Learning, 15(2):201-22 1994.
[10] Ido Dagan and Sean P. Engelson. Committee-based sampling for training probabilistic classifiers. In International Conference on Machine Learning, pages 150-157, 1995.
[11] Bo Du, Zengmao Wang, Lefei Zhang, Liangpei Zhang, Wei Liu, Jialie Shen, and Dacheng Tao. Exploring representativeness and informativeness for active learning. IEFE Transactions on Cybernetics, 47(1):14-26, 2015.
[12] Yonatan Geifman and Ran El- Yania.. Deep active learning with neural architecture search. In Advances in Neural Information Processing Systems, pages 5976-5986. 2019.
10

---

[13] Ran Gilad-Bachrach Amir Navot, and Naftali Tishby. Query by committee made real. In Advances in Neural Information Processing Systems, pages 443-450, 2005.
[14] Steve Hanneke. A bound on the label complexity of agnostic active learning. In International Conference Machine Learning, pages 353-360, 2007.
[15] Steve Hanneke. Activized learning: Transforming passive to active with improved label complexity. The Journal of Machine Learning Research, 13(1):1469-155 1587, 2012.
[16] Steve Hanneke. Theory of active learning. Foundations and Trends in Machine Learning, 7(2-3),
2014.
[17] Andreas Kirsch, Joost van Amersfoort, and Yarin Gal. Batchbald: Efficient and diverse batch acquisition for deep bayesian active learning. In Advances in Neural Information Processing Systems, pages 7026-7037, 2019.
[18] Ksenia Konyushkova, Raphael Sznitman, and Pascal Fua. Learning active learning from data. In Advances in Neural Information Processing Systems, pages 4225-4233 2017.
[19] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 11:2278-22222 1998.
[20] David D Lew is and Jason Catlett. Heterogeneous uncertainty sampling for supervised learning. In Machine Learning: Proceedings of the 11th International Conference, pages 148-15c 156. Elsevier, 1994.
[21] Changsheng Li, Handong Ma, Zhao Kang, Ye Yuan, Xiao Yu Zhang, and Guoren Wang. On deep unsupervised active learning. In International Joint Conferences on Artificial Intelligence, pages
2020.
[22] David Lowell, Zachary C. Lipton, and Byron C. Wallace. Practical obstacles to deploying active learning. In EMNLP-IJCNE pages 21-30, 2019.
[23] Kunkun Pang, Mingzhi Dong, Yang Wu, and Timothy Hospedales. Meta-learning transferable active learning policies by deep reinforcement learning. arXiv preprint 1rX1v:1800.04498, 2018
[24] Davi Pereira-Santos, Ricardo Bastos Cavalcante Prudêncio, and André CPLF de Carvalho. Empirical investigation of active learning strategies. Neurocomputing, 32-:15-27, 2019.
[25] Pengzhen Ren, Yun Xiao, Xiaojun Chang, Yao Huang. Zhihui Li, Xiaojiang Chen, and Xin Wang. A survey of deep active learning. arXiv preprint arXiv:2009.00336 2020.
[26] Tobias Scheffer, Christian Decomain, and Stefan Wrobel. Active hidden markov models for information extraction. In International Symposium on Intelligent Data Analysis, pages 309-318. Springer, 2001.
[27] Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. In International Conference on Learning Representations, 2018.
[28] Burr Settles. Active learning literature survey. Technical report, University of Wisconsin- Madison, 2009.
[29] Burr Settles and Mark Craven. An analysis of active learning strategies for sequence labeling tasks. In Conference on Empirical Methods in Natural Language Processing, pages 1070-1079, 2008.
[30] Samarth Sinha. Sayna Ebrahimi, and Trevor Darrell. Variational adversarial active learning, In International Conference on Computer Vision, pages 5972-5981, 2019.
[31] Ying-Peng Tang and Sheng-Jun Huang. Self-paced active learning: Query the right thing at the right time. In AAAI Conference on Artificial Intelligence, volume 33, pages 51 17-5124, 2019.
[32] Ying-Peng Tang and She Huang. Dual active learning for both model and data selection. In International Joint Conference on Artificial Intelligence, pages 3052-3058, 2021.
11

---

[33] Alexander B Tsybakov. Optimal aggregation of classifiers in statistical learning, The Annals of Statistics, 32(1):135-166, 2004.
[34] VN Vapnik and A Ya Chervonenkis. On the uniform convergence of relative frequencies of events to their probabilities. Theory of Probability and its Applications, 16(2):264, 1971.
[35] Thuy-Trang Vu, Ming Liu, Dinh Phung, and Gholamreza Haffari. Learning how active learn by dreaming. In Annual Meeting of the Association for Computational Linguistics, pages 4091-4101, 2019.
[36] Yifan Yan and Sheng-Jun Huang. Cost-effective active learning for hierarchical multi-label classification. In International Joint Conferences on Artificial Intelligence, pages 2962-2968, 2018.
[37] Yi Yang. Zhigang Ma, Feiping Nie, Xiaojun Chang, and Alexander G Hauptmann. Multi-class active learning by uncertainty sampling with diversity maximization. International Journal of Computer Vision, 113(2):13-1227) 2015.
[38] Xueying Zhan, Huan Liu, Qing Li, and Antoni B. Chan. A comparative survey: Benchmarking for pool-based active learning. In International Joint Conferences Artificial Intelligence, pages 4679-4686, 2021.
[39] Yilun Zhou, Adithya Renduchintala, Xian Li, Sida Wang, Yashar Mehdad, and Asish Ghoshal. Towards understanding the behaviors of optimal deep active learning algorithms. In International Conference on Artificial Intelligence and Statistics, pages 1486-1494. PMLR, 2021.
12


Reviews:
[{"review_id": "yen6I419NJX", "reviewer": "Reviewer_FvK3", "summary": "This paper proposes a novel problem setting for active learning, where the aim is to query samples to improve multiple target models' performance simultaneously. This paper theoretically analyzes the label complexity of active learning under this setting and shows that active learning has the potential to achieve better label complexity. In addition, based on this analysis, this paper proposes an active learning strategy to query samples in the region of the joint disagreement of different target models, and experimentally shows the effectiveness of this strategy with two OCR datasets.", "questions": "* it is better to explicitly define the terms \"realizable and agnostic cases\" for improving clarity and readability.\n* In Theorem 1, each classifier $\\hat{h} _i$ is obtained by ${\\rm min} _{C_i} \\ d(h_i, h_A)$. However, can such $( \\hat{h} _i ) _i$ be learned from the same data? This point is unclear to me. \n* In Line 230, it would be better to explain the hyperparameter $q$.", "limitations": "The paper addressed the limitations in page 6 but does not mention any potential negative social impact.", "rating": 6, "confidence": 1, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n* The problem setting tackled in this paper seems to be interesting and practically useful although it is difficult to judge novelty correctly for me because I'm not an expert in this field.\n* There is a theoretical analysis. It seems intuitive and reasonable to select samples in the disagreement area of multiple target models. However, since I'm not an expert in the field, I was not able to follow all the proof and statements.\n* Experimental results show the effectiveness of this method.\n\n### Weaknesses\n* Since the proposed method seems to be a general algorithm, it might be better to include evaluation other than OCR benchmark datasets.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "symrIuZYaKc", "reviewer": "Reviewer_rdsY", "summary": "It is generally believed active learning is model-dependent. This paper studies the problem of active learning for multiple target models. The authors first analyze the label complexity for active learning under the setting with multiple target models, then find the label complexity of single model is an upper bound of that for multiple models under the realizable case. They also propose an agnostic disagreement-based selection criterion for the agnostic case. Experiments verify the effectiveness of the proposed approach.", "questions": "Theorem 3 is based on hyperparameter q=1. Is there a tighter result when q>1?", "limitations": "None", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper explores a new setting of active learning where there are multiple target models to be learned. How to design effective active learning algorithms is practical and important to the machine learning community. It first analyzes the label complexity of passive learning and active learning for multiple target models under realizable case, and advocates active learning has potential improvement under this setting. For the agnostic setting, the authors propose disagreement-based selection criterion to query examples located in the joint disagreement regions. Theoretical analysis shows the proposed method achieves better label complexity than that of CAL (a representative approach) under some ideal situations. The writing is good and clear. The theoretical analysis and empirical experiments are solid.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ePfJKVYItT3", "reviewer": "Reviewer_mjax", "summary": "This paper studies an important setting of active learning, motivated by the need to efficiently acquire a single dataset that is useful to simultaneously train various models/architectures. Contrary to common observations against a possible improvement of AL over PL under this setting, the authors theoretically prove a label complexity bound of active learning for multiple models which combined with a bound on passive learning implies active learning may also be efficient under this setting. The authors then modify a disagreement-based algorithm used for single-model AL to that for multiple models by taking their average disagreement and prove its label complexity in the agnostic setting. Lastly, this paper proposes a heuristic to circumvent the computational complexity when employing neural networks as the hypothesis class, and empirically demonstrates that this method outperforms well-known active learning algorithms designed for single-model AL, extended using the same average over multiple models, under this multiple target setting.", "questions": "Could the authors elaborate on the remark after Theorem 2? The fact that the target concept h* may not be included in every hypothesis space is not unique to PL, and is problematic for AL as well, no? Further, why is the last sentence in Section 4.1 included here? It seems that this would better fit under Section 4.2.\n\nPlease correct me if there are any misunderstandings in my comments above.\n", "limitations": "This draft does not clearly state apparent limitations, but does mention future work. Please incorporate my comments on the algorithmic and computational limitations and add others as appropriate.", "rating": 5, "confidence": 3, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The main (and only) major theoretical contribution in this paper seems to be Theorem 1: It is an improved analysis method over a straightforward proof technique that translates the sample complexity of AL applied to a single hypothesis class indexed by i to that for multiple hypothesis classes as the sum of each complexity, as the authors mention (line 125). For PL, one can easily obtain a stronger translation since by definition of passive learning (random sampling), the complexity of PL for multiple hypothesis classes is the maximum over each sample complexity. I would think that for PL, this is the only guarantee possible without additional assumptions on how the hypothesis classes are related. Theorem 1 is a tool that is used to get a stronger translation from AL bounds for a single model to that for multiple models, and the authors use this to show that AL can potentially achieve an exponential improvement over PL in the multiple-model setting.\n\nThis is an interesting conclusion, but the reason I say this is the only theoretical contribution is that these bounds and corresponding comparisons between AL and PL are well-known for the single-model case, which this work uses extensively. In proving the theoretical statements in the paper, the only non-trivial modification to proofs by Henneke under this multiple-model setting is to use Theorem 1 in place of the sum of sample complexities for single-model AL.  This isn’t necessarily bad, but I would appreciate if the authors mention whether it’s possible that a stronger translation bound in place of Theorem 1 is likely. Additionally, I would appreciate if the authors remark on whether the bound translation for PL (as I mentioned above) is perhaps the tightest possible, and if not, it would be nice to see an effort improving the PL bound for multiple models being max_i \\Lambda_i to another method analogous to Theorem 1 for AL.\n\nDespite these concerns on theoretical statements, it is nice to see Theorem 4 which shows that DIAM-online is better than CAL applied to multiple target models. However, its presentation could be improved by instead writing that the sample complexity of DIAM-online (e.g. by writing \\Lambda_m (DIAM) < \\Lambda_m (CAL)), as the terms are redundant.\n\nHere are a few major comments\n* Presentation could be improved significantly. Many statements are defined in-text much after they are used, and some notations are never defined formally. Word choices are also poor, making it hard to infer what the authors attempt to say. There are some redundant statements that make the reader confused whether the statements are regarding different situations, and assumed settings go back and forth between agnostic and realizable. How the disagreement regions are to be computed in practice is described briefly in the last paragraph, whereas this the algorithm used throughout experiments. \n* I like how despite this paper being mostly theoretical, the experiments are done using somewhat large models and resembles the motivation mentioned in the introduction. However, I don’t think the tasks being classification on MNIST is convincing and I would like to see experiments on at least one larger dataset (even CIFAR-10 would be fine, or others used in the base code by Cai et al. 2020).\n* The proposed problem setting is interesting and well-motivated. This work is the first to theoretically analyze the sample complexity of active learning for multiple target models, when the hypothesis classes are fixed (due to different target deployments). Section 2 compares with related work that studies similar but different problems and helps understand this problem setting’s position within literature.\n* Algorithmically, the downside is that DIAM probably doesn’t work in the batch-setting, i.e. the algorithm would collect many similar samples when present. This is of large interest in the single-model AL setting, and I think the authors should mention this in the paper as a limitation.\n* Also, the computational complexity should also be mentioned. As the authors note, computing the disagreements is computationally prohibitive for deep models. It would be nice to see how the heuristic overcomes this difficulty and what the resulting runtime is for practically-sized neural networks.\n\nMinor comments\n* Many grammatical errors should be fixed, e.g. remove “with” in line 83, must has -> is in line 125 and move the parenthesis before the period or even better, remove the parenthesis brackets and fix accordingly.\n* Notations could be improved, but this might require much effort. For example, \\Lambda_i is not defined other than the description in parenthesis in line 125-126, although I understand it to be the single-model AL complexity for class C_i (explained later in the draft). The fact that \\Lambda_m is the notation for multiple-model AL where m is a different meaning than the “i” in \\Lambda_i makes it confusing at first read.\n* I don’t think the statement in Theorem 2 should be stated as a “Theorem”. I recommend changing this to a proposition or writing it as an in-text sentence after line 125, since the two statements are related in how to obtain a multiple-model sample complexity for AL (line 125) and PL (this statement). \n\nIn summary, two major concerns I have is originality and presentation. For originality, I would like to hear the authors' remarks on whether Theorem 1 is tight and how a bound on single-model PL may or may not be improved. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "4ouN-UijRv7", "reviewer": "Reviewer_xpsR", "summary": "This work investigates an interesting problem, i.e., how to actively select the informative data to improve multiple target models simultaneously. First of all, the authors formally define the active learning (AL) for multiple target models problem and bridge the label complexity between AL for single and multiple models. After revealing the potential improvement of AL for this setting, a novel query strategy DIAM for multiple target models is further proposed, which prefers the data located in the joint disagreement regions. An efficient implementation for deep models is also provided. Both theoretical and empirical studies are conducted to validate the proposed approach.", "questions": "1. Theorem 1 also works for passive learning. How to guarantee the superiority of active learning?\n\n2. Intuitively, the target models should share some specifications to make the active selection feasible. However, the proposed method works fine with very different target models according to the results in supplementary materials. The authors may explain more about this phenomenon.\n", "limitations": "Yes", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This work is of high quality in my view due to the following reasons:\n\n1.\tThe problem explored by this study is very important and meaningful. In many machine learning tasks, the devices to be deployed usually have varying computational resources, and there is a heavy burden of data labeling to train multiple models. These challenges become more tough with the flourishing applications of deep learning in recent years. \n2.\tFrom the technical perspective, active learning is usually believed to be model-dependent. This work presents a nice attempt towards multiple target models from both theoretical and empirical views. It broadens the learning scenarios of active learning and may inspire future work towards thinking of this problem. This contribution is significant in my opinion.\n3.\tThe paper is well-structured and easy to follow. The authors also provide enough supplementary materials and code for reproduction.\n\nWeaknesses：\n1. It seems that the specific form of \\sigma in DIAM method is not given in the paper, the authors only say that it has the same form with RobustCAL method. I suggest to add this part for a more self-contained presentation.\n2. It is better to introduce the structure of target models in the paper rather than in the supplementary materials for better understanding the results.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept