Uncertainty-aware Active Learning for Optimal Bayesian Classifier
Authors: Guang Zhao
Edward Dougherty
Byung-Jun Yoon
Francis Alexander
Xiaoning Qian
Venue: ICLR 2021 Poster (2021)
Keywords: Active learning
Bayesian classification

Abstract: For pool-based active learning, in each iteration a candidate training sample is chosen for labeling by optimizing an acquisition function. In Bayesian classification, expected Loss Reduction~(ELR) methods maximize the expected reduction in the classification error given a new labeled candidate based on a one-step-look-ahead strategy. ELR is the optimal strategy with a single query; however, since such myopic strategies cannot identify the long-term effect of a query on the classification error, ELR may get stuck before reaching the optimal classifier.  In this paper, inspired by the mean objective cost of uncertainty (MOCU), a metric quantifying the uncertainty directly affecting the classification error, we propose an acquisition function based on a weighted form of MOCU. Similar to ELR, the proposed method focuses on the reduction of the uncertainty that pertains to the classification error. But unlike any other existing scheme, it provides the critical advantage that the resulting Bayesian active learning algorithm guarantees convergence to the optimal classifier of the true model. We demonstrate its performance with both synthetic and real-world datasets.

Published as a conference paper at ICLR 2021
UNCERTAINTY-AWARE ACTIVE LEARNING FOR OPTI- MAL BAYESIAN CLASSIFIER
Guang Zhao', Edward R. Dougherty' Byung-Jun Yoon Francis J. Alexander", & Xiiaonin Qian guangzhao@tamu edu, fa falexxnnerrbre .eneeerrbnlgggg. gov, (edward, bjyoon, xqian) sammueddi
'Department of Electrical & Computer Engineering, DDpaarmeett of Computer Science & Engineering, Texas A&M University College Station, TX 77843, USA
3 "Computational Science Initiative, Brookhaven National Laborator Upton, NY 11973, USA
ABSTRACT
For pool-based active learning, in each iteration candidate training sample is chosen for labeling by optimizing an acquisition function. In Bayesian classifica- tion, expected Loss Reduction (ELR) methods maximize the expected reduction in the classification error given a new labeled candidate based on one-step-look- ahead strategy. ELR is the optimal strategy with single query; however, since such myopic strategies cannot identify the long-term effect of query on the clas- sification error, ELR may get stuck before reaching the optimal classifier. In this paper, inspired by the mean objective cost of uncertainty (MOCU), a metric quan- tifying the uncertainty directly affecting the classification error, we propose an acquisition function based on a weighted form of MOCU. Similar to ELR, the proposed method focuses on the reduction of the uncertainty that pertains the classification error. But unlike any other existing scheme, it provides the critical advantage that the resulting Bayesian active learning algorithm guarantees conver- gence to the optimal classifier of the true model. We demonstrate its performance with both synthetic and real-world datasets.
INTRODUCTION
In supervised learning, labeling data is often expensive and highly time consuming. Active learning one field research that aims to address this problem and has been demonstrated for sample- efficient learning with less required labeled data (Gal et al. 2017 Tran et al.) 2019; Sinha et al.) 2019). In this paper, we focus on pool-based Bayesian active learning for classification with 0- T loss function. Bayesian active learning starts from the prior knowledge of uncertain models. By optimizing an acquisition function, it chooses the next candidate training sample to query for labeling, and then based on the acquired data, updates the belief of uncertain models through Bayes' rule to approach the optimal classifier of the true model, which minimizes the classification error.
In active learning, maximizing the performance of the model trained on queried candidates is the ul- timate objective. However, most of the existing methods do not directly target the learning objective. For example, Maximum Entropy Sampling (MES) or Uncertainty Sampling, simply queries the can- didate with the maximum predictive entropy (Lewis & Gale 1994
[994 Sebastiaai & Wynn |000) Muss- mann & Liang. 2018); but the method fails to differentiate between the model uncertainty and the observation uncertainty. Bayesian Active Learning by Disagreement (BALD) seeks the data point that maximizes the mutual information between the observation and the model parameters (Houlsby al. 2011, Kirsch al. 2019). Besides BALD, there are also other methods reducing the model uncertainty in different forms (Golovin et al.) 2010: Cuong et al. 2013). However, not all the model uncertainty will affect the performance of the learning task of interest. Without identifying whether the uncertainty is related to the classification error or not, these methods can be inefficient in the sense that it may query candidates that do not directly help improve prediction performance.

---

Published as a conference paper at ICLR 2021
1x
1.0
0.8
2x10-1
0.6
0.2
6x100
BALD
49
Wagghted
120
Predictive probability of class
(b) Error regret comparison among methods
Figure 1: (a) Predictive probability of class 1 under uncertainty: the red lines indicate the upper and lower bounds of the predictive probability; the blue dash line is the mean of the predictive probability; the green dash line indicates that the probability is equal 0.5, (b) Active learning performance comparison.
In this paper we focus on the active learning methods directly maximizing the learning model per- formance. There exist such active learning methods by Expected Loss Reduction (ELR) that aim to maximize the expected reduction loss based on a
manner (Roy & McCallum, 2001 Zhu et al. 2003 Kapoor et al., 2007). The ELR methods can focus on only the uncertainty related the loss function achieve sample-efficient learning. fact, ELR is the optimal strategy for active learning with single query (Roy & McCallum, 2001). However, a critical shortcoming of previous ELR schemes is that none of them provide any theoretical guarantee regarding their long- term performance. In fact, since these methods are myopic and cannot identify the long-ten effect of a query on the loss functions, without special design on the loss function, they may get stuck before reaching the optimal classifier. To the best of knowledge, there is currently method that directly maximizes the model performance while simultaneously guaranteeing the convergence the optimal classifier.
Fig. Ia provides an example of binary classification with one feature where both BALD and ELR methods fail. the figure, the red lines indicate the upper and lower bounds of the prediction prob- ability of class 1, illustrating the model with higher probability uncertainty on the sides (r - 44) than that in the middle (x 0). Querying candidates on the sides will provide more information of the model parameters, and therefore is preferred in BALD. However, since the possible probabilities on the sides are always larger than or less than 0.5, querying candidates on the sides will not help reduce the classification error. On the other hand, ELR queries candidates that help reduce the clas- sification error the most, so prefers data in the middle whose optimal labels are uncertain given the prior knowledge. The performance shown in Fig. Ib agrees with our analysis. Fig. Tb shows the performance averaged over 1000 runs, with more details and discussions of the example included in Appendix C. BALD performs inefficiently the beginning by querying points both sides. On the other hand, the ELR method performs the best at the beginning, but becomes inefficient after some iterations (~ 100), indicating some of its runs get stuck before reaching the optimal classifier. In this paper, we consider the algorithm to "get stuck" when the acquisition function value is O for all the candidates in the pool and the algorithm degenerates to uniform random sampling.
In this paper, we analyze the reason why ELR methods may get stuck before reaching the optimal classifier, and propose a new strategy to solve this problem. Our contributions are in four parts: 1. We show that ELR methods may get stuck, preventing active learning from reaching the optimal classifier efficiently. We propose novel weighted-MOCU active learning method that can focus only on the uncertainty related to the loss for efficient active learning and is guaranteed to converge the optimal classifier the true model. 3. We provide the convergence proof of the weighted- MOCU method. 4. We demonstrate the sample-effciency of our Weighted-MOCU method with both synthetic and real-world datasets.
2 BACKGROUND
Optimal Bayesian classifier. Consider a classification problem with candidates x E and class labels y y {0, ., M -1] The predictive probability p(y|er, 0) is modeled with parameters
2

---

Published as a conference paper at ICLR 2021
8. Assume 0 is uncertain with distribution =(0) within the uncertainty class θ The classification problem is to find a classifier 1b : X' + V/ which assigns a predicted class label to a given candidate.
The expected 0-1 loss of the classifier z for candidate x, dependent on 0, is defined (1b x), which can be derived to be the classification error: Co( = 1 - py = (b()) 0). The op- timal classifier with 0, the is defined as the classifier minimizing the classification error: (2) arg max, p(y) jz. f). So we have: C)(V), = ming Cg(s), x) = min,, (1 p (z, 0)}. When there is model uncertainty with π (0), an Optimal Bayesian Classifier (OBC) (*())) is the classifier that has the minimum expected loss over T (0) (Dalton & Dougherty 2013):
Ex(0) (3)))())) x)]
min (Co .)] min p(y|x))
(1)
v
V
where p(yla)) Ex( p(yl,, x. 0)] is the predictive distribution. It's easily to see (x) arg , p(y]z))
Active learning. Active learning collects the training dataset D in a sequential way. For pool- based active learning, in each iteration, choose a candidate from the set of potential training samples X' to query for the class label by optimizing an acquisition function (x). Then, in the Bayesian setting, by including the observed data pair (x,y) D, we update the posterior distri- bution based on Bayes' rule. In each iteration, the acquisition function depends on the posterior distribution of model parameters T(6) D). In the following discussion, simplify notations, we omit D from the notations and use T0) and p(y]ær) to respectively denote the posterior and pre- dictive distributions conditioned on D. When new observed data point is included, the distri- 7(9)p(vlz.0) and p(g)2)
butions are updated by Bayes' rule and the total probability rule as: T (0|x,)) p(y/|z', x. y) ()) [p(y/ lar 0)].
The acquisition function of ELR methods in the Bayesian setting can be defined by the expected OBC prediction error reduction after observing the new pair (z, v) (Roy & McCallum 2001):
UELR (x) pa') (o) (Vx())))) x²)] Ep(ll)) (x(()) (Co(Vs(0)eaa) : x')I];
where p(z') is the distribution over X, independent of and D. ELR methods assume that we use OBC as the classifier, and in each iteration we should choose the query that maximize the decrease in OBC prediction error. The first term in (2) is the OBC prediction error of ()(()), and the second term is the expected prediction error of (n((f)r,s)>> the one-step-look-ahenn OBC, with respect to 2))) In the following section, we analyze why this acquisition function is sample-efficient as directly targets at classification error reduction while ignoring irrelevant uncertainty with respect to the learning task; but it may get stuck before converging to the true optimal classifier (optimal classifier of the true model).
3 MOCU-BASED ACTIVE LEARNING
3.1 MEAN OBJECTIVE COST OF UNCERTAINTY
To analyze ELR methods, we borrow the idea of the Mean Objective Cost of Uncertainty (MOCU) for active learning with respect to the corresponding posterior ा (0). MOCU is a general objective- oriented uncertainty quantification framework (Yoon et al. 2013). For active learning, MOCU can be defined as the expected loss difference between the OBC and the optimal classifier:
(() = Eptr') (o) (Co(V x') Co(Ve,
(3)
Ep(ar) (pinn) p(y' [2)) E.) [min ply'l='' |,',0)11
(4)
V
The second line is derived by the definition of the and ( The first term in is the OBC error as the loss. In the second term, the is the optimal classifier with a specific 0. For the terms inside the expectation operator, we have Co () x²) Ca(re, x²) > 0. Therefore, the second term in 3 is a lower bound of the OBC prediction error. MOCU captures the difference between the OBC error and its lower bound. When MOCU is O. the OBC converges the true optimal classifier and cannot reduce the OBC prediction error further. In that case, we say that OBC has reached the true optimal classifier.
3

---

Published as a conference paper at ICLR 2021
As in ELR methods, we can define an acquisition function by the reduction of MOCU in a ste- look-ahead manner:
MOCU (z; (0)) _M( - Ep(ll)) ((x((0zz)))))
(5)
We can show that the second term in (3). the lower bound of the OBC error, is cancelled in (5). The acquisition function (5) hence captures the expected reduction of the OBC error given new data and
is equivalent to the ELR acquisition function Expanding the second term in (5) we have:
[M z. r))] Bp(z') (Ep(ulz) (0.1))= [Co(PMn00)a.))
x²) (tre, 2 2')
(6)
Since Ey (0)))))))((0) y) (0), is assumed to be independent of B so that we have π (f)22) π (0), we can rewrite the first term in (5) as:
(( = p (Epyt (R (Co (1)(()) x²) - (V),>="))]
Combining 66 and 7 and canceling the Co(vo, x²) terms (the lower bound of the OBC error), ( can be derived as:
MOCU x (0)) Ep(zr) ( (ppp)s)) (Btte////eee/1//000000000000 (B x')
Co(P x' d'III,,
(8)
which is just the ELR acquisition function in 2). Therefore, we can conclude that MOCU-based methods are equivalent to ELR methods.
Another property we can observe from 8 is that UMOCU (2; ग (0)) > 0. By definition,
(=(())aa))) is the OBC with the minimum expected classification error over x(0),,)). Therefore,
(x(0)))))) (Ca(()x0) la.y) a²)] I E=(00 ) [Ca (Ca((( ["') and we have U M (z; (0)) 2 0, indicat- ing collecting new data will reduce MOCU.
3.2 ANALYSIS OF ELR METHODS
In the following analysis, we assume that θ contains the true model 0, and x0,,) > O. We first analyze ELR methods by the MOCU reduction to show that ELR and MOCU-based active learning ignores the uncertainty irrelevant to the OBC prediction. By that, we indicate that not all the model uncertainties directly affect the OBC prediction. Denote the contribution to the MOCU point as K(2, (()) = (n) [roo]CCo(rr)) x) - Co(wo, z)), so that M(x(0)) = En(a) [xx (())]- If K(,, n (0)) = O, then we have V0 E supp(=), (z) = (z), i.e. arg max, p( 0) = arg max, , p(y|zr) This means that for all the possible models, the optimal predictions are the same, and the OBC prediction on I will not be affected by the remaining uncertainty of p(ylæ, 0), if any. In fact, (Ix,= (0)) = 0 does not necessarily mean that there is no uncertainty associated with plylæ; 0). for which it requires that the value of p(y kr, 0) is the same v0 E supp(=), apparently stronger statement than K ((x, (0)) being O. Therefore, not all the uncertainties of p(ylæ, 8) are captured in MOCU when K(x, x (0)) = O. We consider the uncertainty in p(y|z, 0) to be "objective- irrelevant" to the OBC prediction if K(x, x x0)) O. the active learning procedure, when new observation is obtained, it reduces the uncertainty of the parameter 0; and as result, it reduces the uncertainty of p for each E X. If an observation only reduces abjective-irreleaan uncertainty, the value of MOCU will not change. For example, in Fig. |aa the uncertainty of p(y|z, 0) in the region close to x = +4
Evaluating points at x > 14 will only reduce irrelevant uncertainty and it will not be considered in either MOCU- or ELR-based methods. That explains why in the first several active learning iterations, the ELR or MOCU-based active learning can be more efficient than the methods guided by total uncertainty reduction, such as BALD
Dojective-irrelevann
Now we explain why ELR methods may get stuck before the OBC converges to the true optimal classifier. When we have Vr E X, MOCCU (r;; (0)) = O and assume the tie is broken randomly, the acquisition function will suggest any random candidate in the pool. When that happens, we say that ELR methods get stuck if the OBC has not reached the true optimal classifier; i.e., M(T(0)) is still larger than O.
Since p(y' ( [n',0] is a linear function of x (0),, the term min (1 -1 p(y x²)) in (4p is the minimum among M linear functions, and thus a concave piece-wise linear function. Within each linear piece, (zx(0) (x') arg max, P(y"lz') are the same for different π (0). The second term (o) [miny [1 p(1/ [ in 44) is linear function of r0). Subtracting it from the first term and averaging the resulting difference over p(x') maintain the concavity and the piece-wist linearity.
4

---

Published as a conference paper at ICLR 2021
0.1
0.09
#(6) 15
0.08
0.07
0.06
0.05
0.04
(0 22 0)
MOCU
WMOCUtc-1:
WMOCU(c=0.5)
WMOCU(c-0.2)
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
=(0,)
Figure 2: MOCU and weighted-MOCU functions of binary classification problem with θ 101,02].
Therefore, MOCU defined in 44 is concave piece-wise linear function of (0). Moreover, within a linear piece of MOCU, (*(0) (2²) = arg max, p(y/ p(1/|=²), i.e. the OBC prediction, the same for each of x' E X. To gain some intuition, we study a binary classification problem with the uncertainty class of two possible models ② {01, 02 and candidate pool of two training samples to query: X' (011 = - x ((2)), (,1,,2) Further details of the model setup can found in Appendix D. Since express the MOCU function univariate function of x(11) as shown in Fig. 2 It is clear that the MOCU function is a concave piece-wist wise linear function.
Ep(a (0)x, y)], from (5) the acquisition function is defined as M[Ep(ula) (0)22, si]] [M()))) (0( vi)]. Based on the concavity of MOCU M(.) and Jensen's 's inequality, we have the acquisition function MOCU (z; (0)) 2 0. The equality holds if for all E J. (0(=,, y) falls into the same linear piece of MOCU. In this case, the single query (a,, v) cannot provide enough evidence shift the OBC prediction,
Since (0))
that arg max p(y |x x, y) arg max. P(y'\1/) [x') for each of x' E X, even though p(y' |-', ,,,, g) have shown in the binary classification problem there exists such and # [z0]]21.20 /1 = 1) are within the interval of the same linear piece of the corresponding MOCU function. When MOCU is larger than if all candidates cannot provide enough evidence to change the OBC predictions, ELR and MOCU-based methods will get stuck before converging to the true optimal classifier. This due the myopic nature of the acquisition function ignoring the long-term effect of querying each candidate.
p(y'
In Fig.
that #(01 [01]01,2 V1 0)
In summary, ELR methods are efficient by ignoring objective-irrelevan irrelevant uncertainty. But if sampling one data point can only provide little information and cannot help improve OBC prediction in the current iteration, they will ignore its long-ter effect on prediction performance. As result, they may get stuck before converging to the optimal classifier. To keep the efficient sampling property by ignoring objective-irrelevan uncertainty but avoid getting stuck, we propose one-step-look-ahene acquisition function based weighted version of MOCU, which can capture the change of the predictive probability from one-step query candidates. If that change can potentially shift the OBC predictions in the long run, acquisition function will have positive value, and thereby avoid the issues that ELR methods suffer from.
3.3 WEIGHTED MOCU-BASED ACTIVE LEARNING
In this section, we propose modified MOCU-based acquisition function that has the theoretical guarantee to converge to the optimal classifier. Specifically, we propose a modified MOCU function that multiplies weight with each loss difference between the OBC (h) and the optimal classifier
5

---

Published as a conference paper at ICLR 2021
do in the original MOCU definition:
MM Ep(ze (E>()][((T(O) a', (11Co(
Co x')l},
(9)
where (0(=(0), > 0 is the weighting function. The corresponding acquisition function is:
U (x; M (=(0)) - (M (=))),,)))]
(10)
In (9). as more data are collected and the model parameter distribution (0) changes, (()(()), x', will change accordingly. The change of w (0), ,) 8) cannot affect the value of the weighted MOCU if Co(Pa(9): x²) Co(Vo. x²) = O, 40 E (upp(xx()), indicating the uncertainty at x' is objective-imelevant. This makes sure that the acquisition function based on the weighted MOCU will inherit the property of MOCU-based active learning to directly target at classification error re- duction while ignoring irrelevant uncertainty. On the other hand, by introducing the predictive prob- ability into the weighting functions, the probability change from one-step samples can be captured by the Beighted-MOCH based acquisition function such that it can have theoretical convergence guaranteed to the optimal classifier as shown below.
We would like to emphasize that there are also active learning algorithms, such as the ones based on the cyclic sampling and e-greedy policies (Hoang et al., (014), that can almost surely converge to the true model, and as a result, the OBC converges to the true optimal classifier. However, these policies focus on the total uncertainty reduction to derive the full knowledge of the true model, which is unnecessary and therefore inefficient, since we only need the knowledge of the true optimal classifier if the classification performance is the primary concern. Unlike such policies, our weighted-MOOC MOOCU based policy directly reduces the objective uncertainty affecting classification, and as a result, is much more efficient by focusing only on those queries that helpful for improving the prediction. As a result, our proposed algorithm guarantees efficiency both in the short term as well as in the longer term.
In the following, we design a weighting function to make (=((0) = 0 if and only if Ve E X, U 7 (0)) 0 and show that active learning based this weighted MOCU converges the optimal classifier. Specifically, we propose the following weighting function:
w w(T x', = 1 - c- K( (())). with
(11)
(()))((0) (0) |C& (1' (0). x') Co x')]
(12)
= min (e) 1 ply'l 0)] - E (0) [m) p(v'l='.0)
(13)
Ex [max p(y' |a', 0)] maxp(s
(14)
where 0 < C S 1 is parameter controlling the approximation of the weighted MOCU to the original MOCU, with smaller e giving a better approximation. The choice of e depends on the specific classification problem and the total query budget. Methods using a smaller e approximate the ELR methods better, hence they will perform well in the first several iterations but may converge slowly in the long run. On the other hand, when e closer to the acquisition function weighs more heavily on long-term benefits. It is clear that K(x', π (0)) 2 O by (1)). For binary classification, max,' P(y/ [x') Z 0.5. As (x) [max,' p(y' // 0)] < 1, from (4). we have K(r', n (0)) < 0.5, demonstrating that the weighting function in [1] satisfies the requirement (==()),
0.5 0.
Note that this simple weighting function does not change with respect to the model parameter values. Substituting it into the weighted MOCU expression, we have:
M (I())) = Ey(ar) ((1 cK(a', (())) K(=', (())))
(15)
which is a strictly concave function of K. We also illustrate the weighted-MOCU function in Fig. 2 for the same example in Section 3.2. As shown in the figure, smaller provides better approx- imation to the MOCU function, and all the weighted MOCU functions are strictly concave func- tions of o (01) instead of being piece-wise linear, which guarantees that the acquisition function (21( # (9)) is positive. In general, weighted MOCU is strictly concave along most of the direc- tions and only changes linearly along the directions that K(2, (()) is constant for X, which correspond to the queries that only reduce irrelevant uncertainties. Such a property can guarantee the convergence to the true optimal classifier.

---

Published as a conference paper at ICLR 2021
Before presenting the theoretical convergence guarantee of the weighted- based active learn- ing, we summarize the computation of our weighted-MOGG MOCU based acquisition function Algo- rithm which can replace ELR and MOCU-based acquisition functions in Bayesian active learning algorithms with the pseudo-code given in Appendix B. We estimate the computational complexity of Algorithm for the discrete feature and parameter spaces. Assume that the size of the discrete feature space is N₂ = |X| and the size of the uncertainty set of classifiers is Ne = |e|. We study the complexity of calculating the weighted MOCU. In the WMOCU function, the OBC error evalu- ation in line 19 is called for O() times. In ACQUISITIONFUN, WMOCU is called for constant times. Hence, the total complexity of calculating the acquisition function in weighted- MMOCU based active learning is (N), Compared with the ELR method, there is O(N₂) additional computa- tion associated with computing the weight (1 - cK) line Hence, the incurred computational complexity is of the same order as the original ELR and MOCU-based methods.
Algorithm Calculation for Weighted- MOCU based Acquisition Function
function ACQUISITIONFUNC; D.) c)
ammocu_crreet = =W (MOCUU(TT))) 3: wmocu next
2:
4: for in {0, 1 do
5: for in do
6:
Generate array p(0, y D,x) FO.D p(ylz,0) end for
7:
8:
p(y)D,x) Eo p( D.
9: TolD...v = p(B,vl D //p(yl
10: wmocu.nert = wmocu.nent + ply D,z) WMocu(Ta/D.ass.))
11: end for
12: return wmocu current. wmocu next
13:
end function
function WMOCU(TOJD.C)
15:
wHoCCu = 0 for x' in X' do bayesian_erne _rrorr = for 0 in do
17:
18:
19:
bayesian_error bayesian_erron + TTID (1 max, pls/ ///////,00 (2',0) end for for y' in {) do D/,l/D,'' = T p(y'la', (2' end for
20:
21:
22:
23:
24:
obc_error = max,' ply'l D. x!)
25:
K obc_error bayesian _error wmocu = wmocu + p(x') [[1 cK)K]
26:
end for
28:
return wmocu
29:
end function
Theoretical convergence guarantee. Now we show that if active learning for a binary classifica- tion problem is guided by the acquisition function defined by (T0) and (TT). MOCU will converge to almost surely and hence the procedure will converge to learning the optimal classifier of the true model. We assume that both X and & are discrete with finite elements; the true model parameter 0, E and the prior distribution π (0)) over satisfies 70 (0,) > O. We denote the posterior by T" (0) and predictive probability p" (ylzz) in the n-th weighted MOCU based active learning itera- tion, respectively. In the following, we give important lemmas first. All the proofs of the presented lemmas be found in Appendix A.
Lemma Given (0), M(r(O) = 0 if and only if MM = 0.
Lemma I indicates that if M (T())) = O, the OBC ()((0) converges to the optimal classifier the as explained in the first paragraph in Section 3.1.

---

Published as a conference paper at ICLR 2021
Lemma 2 Define Gi x(0)) (1 cK(('' =()) (()), 0 < 1. G(z', is concave function of (8).
is important to choose weighting scheme that renders a concave function G as it guarantees the acquisition function to be larger than or equal to 0, so that adding new observation helps to reduce weighted MOCU to effectively guide active learning.
Lemma Vx X, U (2; π (0)) 2
Lemma 4 At the n-th active learning iteration, if (1;"(";""00) O, Vo E X, M² 0.
This lemma states that if the acquisition function values of all candidates with respect to (0) are O. the weighted MOCU is O. By Lemma IT so MOCU. With these, we can conclude that the OBC with respect to π (6) has converged to the optimal classifier. This is significant when comparing with original ELR and MOCU-based methods as we have shown that this is not the case for them, which may get stuck earlier and therefore lose the long-term efficiency.
Lemma If following some policy candidate x is measured infinitely often almost surely, then lim U (a;; x" (0) almost surely.
Intuitively, if candidate has been measured many times, there is no benefit to measure it again.
With these lemmas, we can prove the convergence weighted- MOCU based active learning:
Theorem Assume that both X and & are discrete with finite elements, the true model parameter 0 E & and the prior distribution (0) over θ satisfies (0,)) > 0; then for the active learning algorithu defined by the acquisition function 10), we have lima-roo M(x" (0)) almost surely.
Proof. As the number of active learning iterations n + xc, following the acquisition func- be measured infinite times. Define XA X as the set whose candidates have been measured infinite times. Denote the measuring sequence of the can- didates following (10 as { we have: EN, s.t. Vn N, XA. Based on Lemma 5 lim U" x (()) = O.
tion (TO), some of the candidates
On the other hand, since with the weighted MOCU (Z): T7" (f))
maxzex (1; (0)), then limn-soo (Ix; x" (0) indicates that Vo E X, U (x: (x::""(( (0)) uniformly converges to O. Based on Lemma 4 limm-+000 M and we can conclude
10-
rancom
the proof with Lemmal
BALD
ELA
MOCU
4 EMPIRICAL RESULTS
100
200
300
40D
500
Beretion
bee
We benchmark our weighted-MOCU method with other active learning algorithms, including random sampling. MES (Sebastiani & Wynn 2000). BALD
Figure 3: The expected OBC error regret comparison between different active learning algorithms on binary classification.
(Houlsby et al. (011) and ELR (Roy & McCallum 2001), on both simulated and real-world classi- fication datasets. In the following experiments, we set e I for the weighted MOCU function. The code for our experiments made available at at /ggtthhb htt.chhb.cooo/c/ccoooommm /acoom/qqia anaaa// wwwaaaaaba//a....... /WMOCU_ALL
Simulated experiments. In addition to the one-dimensional simulated example introduced in Sec- tion I we test our model on a similar simulation setting as the block in the middle dataset (Houlsby et al. 2011), where noisy observations with flip emrr are simulated in block region the deci- sion boundary. We generate data based on a two-dimensional Bayesian logistic regression model: p(y 1/x, w. I+exp( w zj with æ 4/2, The block region is within [-0.5, 0.5]² with the flip error rate equal to 0.3. For the model parameter prior, w1 ~ (0.33, 0.8) is uniformly distributed and wa U( (02.. 0.25) and b (..025 0.25); W1, Wh and are independent.
We randomly sample 100 particles from the parameter prior with one of the particles as the true model parameter. The five active learning algorithms compared for 500 iterations by the OBC
8

---

Published as a conference paper at ICLR 2021
error with respect to the testing data generated from the true model. We repeat the simulations for 500 runs and plot the average performance with standard deviation bars in Fig. 3 The error regret is defined as the error difference between the OBC and the true optimal classifier. From the figure, MES simply chooses the candidates with the predic- tive probability closest to 0.5, it can sample many noisy observations from the block region. ELR per- forms well in the first several iterations but poorly after 200 samples. Our weighted MOCU performs the best.
We have also benchmarked our weighted-MOCU based method with other active learning meth- ods for a synthetic multi-class classification prob- lem. We assume that the probabilistic model
D soom
HES
BALD
Weigtted MDCJ
150
200
300
500
Reraion number
p(y|2, ot) = -fy(z, 1-2.2|3 y E (0,1.2) and fy exp(- (x (,,))/20 We set my be (0, (1,0), 10,
f(x, og with
Figure 4: The expected OBC error regret comparison between different active learning algorithms on 3 class classification.
for y 1.2 respectively; and of U(1,5) being the uncertain parameters. Same as the previous binary classification experiment, we test for 300 runs and plot the average per- formance with standard deviations in Fig. + We can observe that ELR performs poorly in the long run while our Weighted MOCU has better empirical performance par with BALD More results and discussion are in Appendix D & E.
Real-world benchmark experiments. We also present the results on the UCI User Knowledge dataset (Kahraman et al. 2013). The dataset in- cludes 403 samples assigned to 4 classes (High, Medium, Low, Very Low) with each sample having five features in [0, 1js. We have grouped the sam- ples into two classes with 224 samples in High or Medium, 179 in Low or Very Low. We consider the first and fifth features for classification and equally divide the feature space into x 4 bins. For the i-th bin, the probability of candidates belonging to High or Medium is denoted by 8,, 1 S i S 16 and 0,'s ss are independent and 0; ~ Beta 8t), with hyper- parameters ag and 3;. We present the results with the uncertainty class by setting a; = B, 10 in
0.6
0.5
Weighned
Iteoadoon number
Figure 5: Classification error rate compari-
son on UCI User Knowledge dataset
eight randomly chosen bins and for the other bins, a; = 5, 5. = 2 if the true frequency of High or Medium in the i-th bin is lower than 0.5 and a, = 2, 3, = 5 otherwise. We have randomly drawn 150 samples from each class the candidate pool and perform the five different active learning al- gorithms. We repeat the whole procedure 150 times and the average error rates are shown in Fig. 55) While ELR clearly gets stuck in this setup, our Weighted MOCU method converge the opti- mal classifier with less samples than all the competing methods. BALD performs poorly as the bins with a 3 10 have less uncertainty but have more impact on OBC prediction and BALD fails to identify that. More comprehensive results and discussion, including results on the UCI Letter Recognition dataset (Dua & Graff] 2017), can be found in Appendix F.
5 CONCLUSIONS
We have identified potential convergence problems of existing ELR methods and proposed a novel active learning strategy for classification based on weighted MOCU. Our weighted MOCU directly targets at decreasing the classification error and ignores uncertainty irrelevant to the classification performance. More critically, it can capture continuous change in objective-relevant uncertainty. Hence, our new active learning can be efficient both at the beginning and in the long run with the guarantee of converging to the optimal classifier. Empirical results have demonstrated active learn- ing guided by weighted MOCU leads to sample-efficient learning. Future work includes theoretical analysis of MOCU-gut -guided active learning for multi-class classification, as well as developing opti- mization methods for active learning in continuous space.

---

Published as a conference paper at ICLR 2021
ACKNOWLEDGMENTS
X. Qian was supported in part by the National Science Foundation (NSF) Awards 1553281, 1812641, 1835690, and 1934904. B.-J. Yoon was supported part by the NSF Award 1835690. The work of E. R. Dougherty and F. J. Alexander was supported by the U.S. Department of Energy, Office of Science, Office of Advanced Scientific Computing Research, Mathematical Multifaceted Integrated Capability Centers program under Award DE-SC0019303.
REFERENCES
Nguyen Viet Cuong, Wee Sun Lee, Nan Ye, Kian Ming A Chai, and Hai Leong Chieu. Active learning for probabilistic hypotheses using the maximum gibbs error criterion. In Advances in Neural Information Processing Systems, pp. 1457-1465, 2013.
Lori A Dalton and Edward Dougherty. Optimal classifiers with minimum expected emror within bayesian framework- part i: Discrete and gaussian models. Putlern Recognition, 46(5):1301- 1314, 2013.
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. URL http://aacciivee ics. uci. edu/ml
Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep bayesian active learning with image data. In Proceedings of the 34th International Conference on Machine Learning- Ccarning-Volume Volume 70, pp. 1183- 1192. JMLR. org, 2017.
Andrew Gelman, John B Carlin, Hal S Stem, David B Dunson, Aki Vehtari, and Donald B Rubin. Bayesian data analysis. CRC press, 2013.
Daniel Golovin, Andreas Krause, and Debajyoti Ray. Near-optimal bayesian active learning with noisy observations. In Advances in Neural Information Processing Systems, pp. 766-774, 2010.
Trong Nghia Hoang, Bryan Kian Hsiang Low, Patrick Jaillet, and Mohan Kankanhalli. Nonmyopic e-bayes-optimal active learning of gaussian processes. 2014.
Neil Houlsby, Ferene Huszár, Zoubin Ghahramani, and Máté Lengyel. Bayesian active learning for classification and preference learning. arXiv preprint arXiv: 2011.
H Tolga Kahraman, Seref Sagiroglu, and Ilhami Colak. The development of intuitive knowledge classifier and the modeling of domain dependent data. Knowledge-Bond Based Systems, 37:283-295, 2013.
Ashish Kapoor, Eric Horvitz, and Sumit Basu. Selective supervision: Guiding supervised learning with theision- active learning. In IJCAI, volume 7, pp. 877-882, 2007.
Andreas Kirsch, Joost van Amersfoort, and Yarin Gal. Batchbald: Efficient and diverse batch acqui- sition for deep bayesian active learning. In Advances in Neural Information Processing Systems, pp. 7024-7035, 2019.
David D Lewis and William A Gale. A sequential algorithm for training text classifiers. In SIGIR 94, pp. 3-12. Springer, 1994.
Stephen Mussmann and Percy Liang. On the relationship between data efficiency and error for uncertainty sampling. In Jennifer Dy and Andreas Krause (eds.). Proceedings of the 35th In- ternational Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 3674-3682, Stockholmsmässan. Stockholm Sweden, 10-15 Jul 2018. PMLR.
Roy and McCallum. Toward optimal active learning through sampling estimation of error reduction. int. conf. on machine learning, 2001.
Paola Sebastiani and Henry P Wynn. Maximum entropy sampling and optimal bayesian experimen- tal design. Journal of the Royal Statistical Society: Series (Statistical Methodology). 62(1):
145-157, 2000.
10

---

Published as a conference paper at ICLR 2021
Samarth Sinha, Sayna Ebrahimi, and Trevor Darrell. Variational adversarial active learning. In The IEEE International Conference on Computer Vision (ICCV), October 2019.
Toan Tran, Thanh-Toan Do, Ian Reid, and Gustavo Carneiro. Bayesian generative active deep learn- ing. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th Interna- tional Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Re- search, pp. 6295-6304- Long Beach, California, USA, 09-15 15 Jun 2019. PMLR.
Byung-Jun Yoon, Xiaoning Qian, and Edward R Dougherty. Quantifying the objective cost of uncertainty in complex dynamical systems. IEEE Transactions on Signal Processing, 61(9):2256- 2266, 2013.
Xiaojin Zhu, John Lafferty, and Zoubin Ghahramani. Combining active learning and semi- supervised learning using gaussian fields and harmonic functions. In ICML 2003 workshop the continuum.frow labeled to unlabeled data in machine learning and data mining, volume 3, 2003.

---

Published as a conference paper at ICLR 2021
Uncertainty-aware Active Learning for OBC: Appendix
In this appendix, we provide proofs of the lemmas, the pseudo-code of active learning algorithms, as well as more detailed descriptions of our experiments, additional results, and discussions on the weighted MOCU for multi-class classification.
A. PROOFS OF LEMMAS
Proof of Lemma 1. Based on 3 since (o())10))22)) - Co(Vo, x') 2 0, so M(((0)) = iff
Co(P+((): z') - Co(ye, x') = 0 Ve E N, v0 E supp(T). In addition, in 9 (((((),z',0) > 0, so Co(V:10), x') Co(vo.
Va E X, v9 supp(Ir) iff M (T (B) O, which concludes the
proof.
Proof of Lemma 2. In the following proof, we omit the argument x' in and K for simplicity.
Owing to the concavity of the min operator, miny [0) p(y'l( |æ', 0)] is a concave function of (f). With Exy [ming (1 - p(y'|=', |=', 8)] being linear function of x (f),, based on (13 K (x()))
equals to concave function subtracting a linear function and thus also a concave function.
As analyzed in Section 3.2, 0 < {(=(0)) < 0.5. We define T(k)
(1 ex)K, E 0.5)
a strictly increasing and strictly concave function with 0 < c S 1. G() ([K(T(0)) is composite function of T and K. So we conclude the proof with the property of the concavity for the composite functions:
[[ (0) + (1 (0)) > (T)((1(0)) + (1 (T2))
(m (1 (=2))
(16)
The first inequality is because T is increasing and K is concave; and the second inequality holds as T' is a concave function.
Proof of Lemma 3. Since T()) Ey p(ylz) (()))))(((22)))) v). by Jensen's inequality, we have ((I',T(0) 2 [G (0/xz, y))] as G is a concave function. So the weighted MOCU ac- quisition function:
U n (f))
Eg [G(z', (0)) E. (o (01x,, v)ill 2 0.
(17)
Proof of Lemma 4. We will prove the contrapositive of the lemma: assuming (T > 0, Er E X s.t. U (2; T" (0)) > 0.
Based on 15 M (T" (0)) > 0 indicating 3x E X' s.t. K(2, (0)) > O. It is sufficient to
show that if K(x, x (0) > 0, then U (x; (0) > 0. To prove that, only need to prove G(I,T"((0 > Epr(yl=) G((z, -" v))l: then by 117. U """ (x; > O.
Since G is concave function, know G(z, T² En (vl=) | x"' (0|=, v))] With ("(
Ey "y (ylx)r" (0]a, y). we can rewrite [66 as:
[[K T "e (0)) > (oEp"(o)>) [K (z,, T" (0|z, w))l (p [ x" (0|x,
The second equality holds only if Vy E [0.1}, K(x, v)) = <(x, means that to prove G(x, (0))
which
Eyja G(I, ((r,,>"00x;,, (0)x, we just need to show By
{0, 1}, K(x, (0]0,, w)) # [(x, T (0)). In the following proof, we will show if K K(x, T" (0)) > 0.
then By E (0,1), s.t. K(I, T" (0)
z0 (2,)) y)) * K(#, T (0)).
Denote arg max, p" (y/z)) By (14) we have:
K(x, π (0)) =
(0) mmxxpyyaa 0) p(ū|=, 0)]. "
(18)
8 (Esopp()r*)
Since K K(x, T" (0)) > O, the parameter set O = {0 E supp(T"): arg max, p(ylæ, 0) + /) is not empty. We only keep the nonzero terms in K:
K(x, x (())
Σ n" [mxxmax((y)z. p(0)=, OEO.
(19)
y
12

---

Published as a conference paper at ICLR 2021
arg max, p" (y]z), indicating that the predictive probability p" (((1)) 2 0.5. For 0 E Oo p(ū|z, 0) < 0.5, we have: if A E Oo. x (0)0,, 9) st (()))p(0)x.9) ("(o)))
For binary classification,
(0))..
If we observe (r in (n+1)-th iteration, the updated posterior predictive probability p"(ja, ú)) 2 (j]x)) 2 0.5 and therefore maxy p" pyly,, {ar ;)} Hence.
K(I, " (002,))) = Σ x <"(0,,, (0, D) [max p(ylor (r, 0) p(u| 0)] <
(20)
BEO.
Since K (0 0), + K ((= 2). we have G(æ, n (0) > (g)() G(ar, x v))l and U (2; n" (0)) Ep (G(z', , [0)) Ba(ar) (x) (G(z', n" (0(0,,, v))l
2 p Φ [G (0)) (ulx) (G T (0 >
(21)
This concludes our proof.
Proof Lemma 5. Adding a new data point (ar, v) to D, the posterior change is: x >= (0(s;) v)
n" (0))p(yla,
p"(y]=] Define e = {0 E θ p(ylæ, 0) plyla, jr, 0,))- Denote (n(n) as the times
of the candidate being queried at the n-th iteration. Based on the posterior consistency the- ory have Eoce, (#) - 1 (n) + Gelman et al. 2013). Since ()((x)
Eese x" (0))p(())z, 8), we have lim p" (v(2)) and p(y|x, 0z). Hence lim,-xo x x²(0fr: [0 y) (0) almost surely, which indicates lima-+0o U (x; T (0)) = 0 almost surely.
B. WEIGHTED-MOCU BASED ACTIVE LEARNING & COMPUTATIONAL COMPLEXITY
The pseudo-code of the general active learning procedure is provided in Algorithm 2. The function ACQUISITIONFUN can be acquisition functions of various methods, including weighted-MOCU, ELR, BALD, etc.
Computational complexity We study the complexity of the complete active learning procedure. As we analyzed in the main text for the computation of the weighted MOCU acquisition function, the WMOCU function is called for ((N,NN) times. In ACQUISITIONFUN, WMOCU is called for constant times. Finally, in the main procedure, in each iteration, ACQUISITIONFUN is called for each æ. Hence, the total complexity of Weighted MOCU- -bsUd active learning is O(TN2No).
0.04
0,0000
0.03
0.02
0.0 0.00003
0,00002
2.01
0,00001
0.00
0.00000
(a) Acquisition function of BALD
(b) Acquisition function of ELR
Figure S1. The acquisition functions based on the model (22)
C. DETAILS OF THE ONE- ONE-DIMENSIONAL ACTIVE LEARNING EXAMPLE IN INTRODUCTION
The example in the Introduction of the main text is binary classification problem with {0.
based on only one feature x E [-4,4] 4). The underlying discriminative model based on:
(v 1|z, b) S(x) + a(
exp (x)
S(x) 0.6 1+ exp (x) + 0.2 (xx, a, b) exx((=²² + Meexp(-(x -))) exp(-(x + 4)2)),
(22)
13

---

Published as a conference paper at ICLR 2021
Algorithm 2 General active learning procedure
1: function (
2: Set discrete candidate set X, the probability array ps, and iteration number T
3: Set the discrete parameter set @ and the corresponding probability array xa
4: Initialize the data set = 0
5: To D ne
6: for = to T do
7:
for in X' do
8:
Store ACQUISITIONFUNNG Topp) ) to the array Ux end for
9:
10:
Optimize Ux and find the maximum point x
Obtain the label y corresponds x and update D D fx" ,²} for 0 in è do
12:
Update Te x TalD 10"" 00)
14:
end for X == x/(x²}
15:
16:
end for
17:
end function
where (a, 6)TH is the uncertain parameter vector, with a and independently uniformly dis- tributed on the intervals [-0.1, 0.1] and [-0.2, respectively. The discriminative model equals sigmoid function 5(x) plus the perturbation c(x, a, b). mixed Gaussian function that changes with and b, the uncertainty class of classifiers can be constructed by such deviations on S(x). The discriminative model has higher uncertainty near x + 4, which depends on the value b, than the uncertainty near p(y = 1|x) near x + 4 and x = O, respectively. So by observing data at x 14, the uncertainty on Pv 2| 0) will not be reduced significantly.
O, which depends
Value of and b has negligible influence on the
In Figs. S1(a) and (b), we show how the acquisition functions of BALD and ELR change with respect to It is clear that the acquisition function of BALD at r #4 should have the largest value since p(ylæ=) ) has the highest uncertainty. On the other hand, since p(yl = +44 is always above or below 0.5, the specific value of p(y|æ, b) will not affect the corresponding optimal Bayesian classifier (OBC) and therefore the loss reduction in Fig. S1 at I = 44 is always O; the acquisition function of ELR around
be larger than 0, since knowing the specific value of p(y|ı 0, a. b) can reduce the classification eтror.
D. DETAILS OF THE BINARY CLASSIFICATION EXAMPLE IN SECTION 3.2
In the binary classification problem, θ (01, 62). X fai, Ø2). The probabilistic model setting for the two candidates is symmetric:
| 01) (0.6, (0.6,
(0.3,0.7)
p 22, 01) (0.7,0.3) (0.7,0.3)..P)x))z2) 02) = (0.4,
There are three intervals corresponding the linear function pieces of MOCU in Fig. 2 [0, 0.33), (0.33, 0.67] and (0.67, 1]. In the three intervals, Vi((o) (21). the OBC predictions of .i are 1, 1 and O, respectively; (o)(0) (I2). the OBC predictions of Ø2 are 1, O, and 0, respectively.
In Fig. 2 we set the prior (B1) 0.15, then based on the Bayes's rule we can obtain the pos- terior with the observations of (1). Based on the observation result of the posteriors are (0, 1x, 31 0) 0.2609 and 7 #0, [1 y1 = 1) 0.0916, both of which fall into the first linear piece of MOCU.
E. MULTI-CLASS CLASSIFICATION
Although we have shown in the main text that our WeOU can achieve good empirical performance of converging OBC with the simulated multi-class classification experiment, active learning for multi-class classification problems can be complicated. The weighting function (10) adopted in the main text may not have the same theoretical convergence guarantee the optimal
14

---

Published as a conference paper at ICLR 2021
classifier if applied to multi-class classification problems. Here we just show counter example, for which Lemma does not hold if using the same weighting function.
Assume three-class classification problem y E {0, 2). The candidate pool only has candi- date X' = {Ø} and the parameter set = { 82 03) In addition we set the probabilistic model p(y]ær, 0) and the prior :T(0) shown in Tables S1 and S2, and calculate the posterior and posterior predictive probabilities. In the tables, Yo denotes the one-step-look-ahee
observation correspond- ing x, and x is omitted for simplicity. Without loss generality, we just the weighted MOCU parameter e = 1.
ply 11)
02)
p(y
ply)
PM Vo = 0)
Ply|Do Uo =
PlU No
2)
y = 0
0.4
0.4
0.4
0.4
0.4
0.4
0.4
y = 1
0.3
0.1
0.5
0.3
0.3
0.327
0.273
y =2
0.3
0.5
0.1
0.3
0.3
0.273
0.327
Table $1. The probabilities of p(ylar, 0) and p(ul ((yla, Mo)-
(())
T0 V/o
0)
<000 1)
T0 yo 2)
0 01
0.8
0.8
0.8
0.8
0 02
0.1
0.1
0.17
0.03
0 0g
0.1
0.1
0.03
0.17
Table S2. The prior and posterior of x(f).
Here two properties in the setting are worth mentioning:
1. (01) is close to 1 and result Vy {0, 1,2), have max, p(v) max, p(ylyo) maxy (1) = 0.4;
2. p())02) and p(y)0g) are symmetric and x (02) =((s), as result Vy (0, 1,2) x01)
(0 (o) = 0.8 and therefore (8) [maxy p(y[0)] = [Ex()())) [max, ply 0) = 0.8 x 0.4 + 0.2
0.5 0.42.
Recall that the K function and weighted MOCU are:
K(=(0)) = (xx(0)
- max maxp(y), 3f
(23)
M'(TT()) = - {(T(0))) K(T(0))-
(24)
Therefore, we have Vya E {0, 1, (TT()) ((T(0l)o)) 0.02 and (T((((0))
M (T((Ol)))) > O. On the other hand,
(T(((()) = MM (x(()) - Ep(r.) (M" O.
(25)
which means that the algorithm may get stuck. Here we just give extreme where only one candidate is in the search pool, but it is straightforward to build a more practical example based on what have shown here.
We can see from the example that, unlike in the cases of binary classification problems, the weighting function 1 cK may remain unchanged for single observation multi-class problems. Because of this, the weighted-MOCU algorithm may get stuck. Since OBC prediction is the maximum of the predictive distribution p(y(a) the weight function introduced to capture the changes of p(ylz)) as that indicates the potential shift of OBC prediction in the long run. K is function of max, P(y(r), in binary case, max,, p(y(a) must change as p(y|z) 1z) changes. However, in multi- problems, the probability of the optimal label max, pzy(x) may remain unchanged, when the probability of other labels change, just like the example above where max, p(y) max, P(y Wo = 1). the next section, we propose weighting function that can capture the change of any element in p(y)z).
F. ANOTHER WEIGHTED MOCU SCHEME FOR MULTI-CLASS CLASSIFICATION
To extend the weighted MOCU scheme suit for the multi-class problem, we propose a weight func- tion that capture the change of p(y]ar). The weighting function defined the softmax of
15

---

Published as a conference paper at ICLR 2021
p(y[z):
exp(max, p(y[2))
w w (T())),',0)
(26)
Σ, exp(p((ul<))
where P(y(ar) is the posterior predictive distribution at the current active learning iteration. We compare this Weighted MOCU with other active learning algorithms empirically on the synthetic three-class classification problem and the performance comparison is shown in Fig. S2. This new Weighted MOCU (Weighted MOCU2) performs slightly better than other algorithms on this multi- class classification problem.
regret
10-3
random
MES
BALD
ELR
Weighted MOCU
Weighted_ MoCU2
10
100
200
300
400
500
Iteration number
Figure $2. The expected OBC error regret comparison between different active learning algorithms for the three-class classification problem.
G. ADDITIONAL SYNTHETIC EXPERIMENTS
We run the same synthetic experiment of Fig. with a different prior setting: Wi (0..3, 0.8), W2 U( (000.2 0.02) and b ~ U(-0.25, 0.25). and the results is shown in Figure S3. The perfor- mance shows that only our Weighted MOCU method performs better than the random benchmark.
Here benchmark different active learning strategies for OBC with another synthetic example. Assume the classification problem with two dimensional input features æ (I1, x2) E R²² and binary class labels y {0, 1). The computational model derived by decision boundary in a quadratic form: T2 = axf + be + c, i.e. p(y = 1|x a, b, c) 1 (x² > ax? + bo + c)- The parameter vector 0 (a, b) R3 uncertain and the true model is characterized by true parameter 0*. Unlike Monte Carlo sampling in the main text, here we consider a discrete grid setting for both input space and parameter space with discretization for each variable follows:
1. ranges in [-0.5, 0.5] with increment 0.05
Ø2 ranges [0, with increment 0.1.
3. ranges in [-4.3, -3.8] with increment 0.05,
ranges [-0.25, 0.25) with increment 0.05,
5. ranges in [1, 2] with increment 0.05.
For now, we simply assume that the distributions over the feature space and parameter space are all uniform to illustrate the effectiveness of MOCU-based active learning. With prior knowledge of the system of interest, knowledge-drive prior should be incorporated. Following the weighted- MOCU based active learning algorithm in Algorithm 1, sequentially query the true system and reduce the model uncertainty in a way that maximally reduces the classification enror of the corresponding OBC.
16

---

Published as a conference paper at ICLR 2021
random
MES
BALD
10-3
ELR
Weighted, MOCU
6 10-4
4 1010
100
200
300
400
500
Iteration number
Figure S3. The expected OBC error regret comparison between different active learning algorithms on binary classification.
Now we assume that when querying the system, the class label given with a heterogeneous random flipping error with the eπrоr probability being function of z1: p(y 1/2 0) p(y 0, = 0.3 (1 f + 0.1. Therefore. when æ = 0, the flipping error is 0.4; and when 21 = =0.5, the flipping error is 0.1. We have implemented the same methods in the main text with 50 iterations and 100 runs, The active learning results are illustrated in Fig. S4. As we can see, in this figure, MES does not perform well it cannot differentiate between model uncertainty and observation error. ELR performs similarly to BALD and our weighted-MOCU based method at the beginning, but then it gets stuck before finding the true boundary. BALD and our weighted MOCU perform similarly. This is because in this setting p(y Jr. 0) is either or O, so there is no irrelevant uncertainty with which p(y|2 8) is always larger or smaller than 0.5 but the value is uncertain.
In addition to the average performance comparison, we deliberately choose one of the runs in which the ELR method gets stuck to better illustrate the difference between the existing ELR methods and the proposed weighted-MOCY based method. In this run, the randomly chosen parameters are (a = -, b 0, 1.9). Fig. S5(a) shows the error regret (the OBC error minus the true opti- mal classifier error) comparison, in which ELR gets stuck and the weighted-MOCU based method reaches 0. Notice that the y-axis is in the logarithm scale, so the vertical line in the WMOCU plot implies that the value turns to O. Error regret equals to 0 indicates that the OBC classifier equals to the true optimal classifier, but in practice we don't 't know the true optimal classifier, so we need the value of MOCU to quantify the expected error difference between OBC and the optimal classifier of each 0 = (a, d, c). Fig. S5(b) shows the changes of MOCU value during the two active learn- ing procedures. Not surprisingly, the MOCU value during the iterations of the ELR method also gets stuck, while the MOCU value in the iterations of the weighted- method continues to de- crease. Fig. S5(c) shows the changes of the maximum value of acquisition function in each iteration. The acquisition function of ELR decrease to 0 after 22 iterations, and that explains why ELR gets stuck. On the other hand, the maximum acquisition function of WMOCU is always positive as the corresponding MOCU is positive, until it gets close to 10-16 which is the rounding error in floating point arithmetic. In theory, as the observation is noisy, we can not be sure of the optimal prediction. Therefore, the MOCU and the acquisition function of weighted-MOCUU should always be positive, which is demonstrated in the figures.
We have also performed experiment to show the algorithm performance change under different noise levels. We set the flipping error rate as p(y # 2|x) = c x (1 4xf) + e, 0 < e S 0.25. Therefore, when T1 0, the flipping error is 2e; and when 21 =0.5, the flipping error is We perform the same methods with 100 iterations and 100 runs on the noise level e = 0.05 and
0.25. The resulting active learning performance curves illustrated in Fig. S6. We can
17

---

Published as a conference paper at ICLR 2021
10-1
10
error
random
10-3
MES
BALD
ELR
Weighted MOCU
10-4
0
10
20
30
40
50
Iteration number
Figure S4. The expected OBC error comparison between different active learning algorithms in the setting with heterogeneous observation error.
10-1
Weighted_MOO
10'0
10
acquis
10--
EUn Weighted MOOU
10-3
10:4
20-11
10
ELR
Weighted MOCU
10
Ileratton
(a) Error regret Figure S5. Comparison of ELR and weighted MOCU on specific run
(b) MOCU value
(c) Acquisition function
random
10
MES
10-i
ELR
10-2
Weighted MOCU MOCU
eroor regret
error regret
10
10-2
random
BALD
ELR
Weighted
40
100
40
100
Iteration number
iteration number
(a) Noise level 0.05
(b) Noise level 0.25
Figure S6. Active learning algorithm performance comparison with different noise levels
from the figure that the performance of MES degrades significantly with high noise while the performance of other methods does not appear to be very sensitive to the increasing noise level.
H. REAL-WORLD BENCHMARK EXPERIMENTS.
We here present the complete results on the UCI User Knowledge dataset (Kahraman et al. 2013). In addition the uncertainty class setup in the main text, have tested two other setups of hyper-
parameter values: 1) 'uniform prior' with a; = 3; = 1. and 2) 'good prior' with a; = 3 10 in eight bins chosen randomly, for other bins a; B if the true frequency of High Medium
18

---

Published as a conference paper at ICLR 2021
rancom
MES
0.35
BALD
Weighted MOCU
random
0.25
MES
0.2
BALD
ELA
MOCU
6.1
6.15
0.0
0.05
20
25
30
20
30
50
60
Daration number
Eeratton number
(a) Uniform prior
(b) Good prior
Figure S7. Classification error rate comparison on UCI User Knowledge dataset
random
random
BALD
BALDD
Weighted MOCU
Weighted MOCU
entor rele
0.10
enarr rete
0.10
0.02
10
12
14
16
10
12
14
36
Itoration number
Steration number
(a) Performance of letter E vs. classification
(b) Performance of letter P vs. classification Figure S8. Classification error rate comparison on UCI Letter Recognition dataset
in the i-th bin is higher than 0.5 and a; 3, 5 if the frequency is lower than 0.5. We also randomly draw 150 samples from each class as the candidate pool and perform the five different active learning algorithms. We repeat the whole procedure 150 times and the average error rates are shown in Fig- S7. both Fig. S7a and Fig. S7b, ELR performs the best these two setups while our Weighted MOCU performs similarly. BALD performs reasonably in Fig. S7a but it again performs poorly Fig. S7b. This is because the bins with a = B 10 have less uncertainty but have more impact on OBC prediction and BALD fails to identify that in this setup again.
We also present the results on the UCI Letter Recognition dataset (Dua & Graff 2017). Letter Recognition is multi-class classification dataset with each sample having 16 numerical features generated from typed images of the capital letters in the English alphabet. We select two pairs of -isti--dis letters: E vs. F and D P. The total number of training samples is 1543 and 1608 for E vs. F and vs. P, respectively. Active learning algorithms are applied with Bayesian logistic regression models. We randomly take 100 data points first to construct the prior, and the rest of the data as the pool to test the five active learning algorithms. For prior construction, we train logistic regression model on the 100 data points and take the trained parameters the mean of normal distributed prior with the variance equal to 1. Then we sample 1000 particles from the prior as the uncertain parameter set. We repeat the whole procedure 100 times and the average error rates are shown in Fig. S8. Unlike the synthetic datasets, the real-world datasets have corresponding true models. We can only find the optimal models that approximate the data best. However, we can still the trends of different algorithms. Compared with random sampling, all the algorithms quickly converge to the optimal models. ELR performs the best in the first several iterations, while converges slowly in the latter iterations. Our weighted MOCU based method is again demonstrated to converge faster than other competing methods.
It is clear from all our experiments for both simulated and real-world data that, in addition to its theoretical guarantee for active learning with OBC, our weighted MOCU method has achieved con- sistently better or similar empirical performance compared to the best performing ones among the
19

---

Published as a conference paper at ICLR 2021
existing pool-based active learning methods, approaching the corresponding OBCs faster with fewer labeled samples.
20


Reviews:
[{"review_id": "jXdJ_onw86Q", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper addresses the active learning paradigm in which the learner queries an oracle to obtain the class label of some inputs. Depending on the querying strategy, the learner can improve its classification model more or less efficiently. \n\nAmong other possibilities, the authors focus on a class of methods known as Expected Loss Reduction, which is optimal in one-step-ahead risk minimization. However, the authors shed light on the fact that the long-term effect of this strategy do not necessarily allow to reach an optimal classifier. They thus introduce an alternative approach that achieves this goal by focusing on loss uncertainty.\n\nMore precisely, the authors introduce the Mean Objective Cost of Uncertainty (MOCU) that captures the expected difference between the error of Bayesian optimal classifier (BOC) and the expectation (against the parameter theta posterior) of the error of the theta-best classifier. An active learning strategy can devised by looking for a new input that (roughly speaking) will cause the largest MOCU drop.\n\nBecause the strategy consists in selecting inputs that maximizes MOCU drop, MOCU will decrease as new class labels are revealed but this does not imply that MOCU will reach its minimum (zero). To achieve long run convergence of MOCU to zero (and thus get obtain the optimal classifier), the authors propose a so-called weighted strategy that solve this issue. \n\nFinally, the authors provide fair numerical experiments on both synthetic and real datasets. The results indicates that the proposed strategy seems to provide good results in a wider range of situations as compared to SOTA.\n\n\nMajor Remarks : \n\nAlthough the authors provide some proof that the weight function can solve the long-run convergence issue, I wish they would provide intuitions as to why their particular choice can choose inputs that will be beneficial in this regard. The chosen weight function is going in the opposite direction of MOCU. Its effects are thus hard to interpret although it is instrumental to obtain a concave functions for the proofs.\n\nThe major drawback of the paper is that the scope of the proofs is very limited. Can the authors give insights as to what would remain valid in more realistic situations in which one has class imbalance, non-uniform utility (more general loss than 0-1), continuous parameter space ? The proof (unless I am mistaken) also do not account for approximation errors incurred by replacing expectation with empirical versions. \nA few comments are provided in the appendices concerning multi-class problems.\nI, however, reckon that the numerical experiments are reassuring \n\nAn algorithm is provided in the appendix but some comment on complexity as compared to prior arts in the main text would be appreciated.\n\nMinor remark :\n\nI am bewildered by this statement : \"Converging to the true model is unnecessary and inefficient for classification\". Should this be understood as the myopic strategies standpoint ?\n\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "TnMvSoAw1zJ", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nSummary:\n\nThis paper provides an interesting algorithm to address the previous Bayesian active learning query strategy in (binary) classification. By the simple modification, the algorithm can overcome the drawbacks of ELR in the convergence to the optimal classifier parameterized by $\\theta_r$. In experiments, the proposed algorithm can achieve the advantages of ELR and BALD simultaneously. \n\nReasons for score:\nOverall, I vote for the marginally above the acceptance threshold. The proposed methodology is impressive and well-motivated. However, there are some lacks in addressing the problem of ELR in a qualitative manner. The assumptions of theorems look strong. I feel that there can be room to be improved in this paper. \n\nPros:\n1. This paper was well motivated by the drawbacks of ELR and insightful comparison of BALD and ELR. In Bayesian approaches, these issues can be interesting and valuable.\n2. The proposed algorithm is simple and addresses the problem caused by only mitigating the mean difference. The proposed algorithm can diminish the mean difference and ensure that $M^w ( \\pi^*(\\theta) )$ converges to 0.   \n3. The proposed algorithm can dominate the random and ELR. Also, the prior can be used to provide better results. \n\nCons:\n1. The problem of ELR is not verified thoroughly. The stuck in the convergence of ELR can be due to the lack of considering the long term effects. However, the detail of this phenomenon is not verified in a more detailed manner. Can you show the details of what happened in the ELR active learning? At least, I want to see the values of $U$ and $M$ when the ELR is used.  \n2. The proofs assume that the supports of $\\pi(\\theta)$ and $x$ are finite, respectively, and prior is limited to the discrete-type probability measure. These assumptions can be a good starting point. However, it is better to provide any clue that we can extend this result to more general settings.  \n3. The counter-example of the proposed algorithm for multi-class in the Appendix shows this paper's prematurity in multi-class problems, and there are no details to address this problem. If you can provide some clues to address the multi-class problem, it is very helpful.  \n\nMinor Comments:\n\n1. It is not easy to follow the proofs in Section 3.1. The authors claim that the lower bound of OBC error will be canceled in the (5). The equations after (5) can imply this cancelation. However, there is no direct wording to conclude this cancelation.\n2. In the equation of $\\sum_y p^*(y | x) \\pi^*( \\theta | x, y) = \\pi^*( \\theta )$, $\\pi^* ( \\theta ) = \\pi^*(\\theta | x),$ it is better to clarify that $\\pi^*$ is not affected by $x$.  \n3. In the proofs of theorem 1,  the notation of $X_A (w)$ is not consistent with the previous notation of  $X_A$.  \n4. The infinite querying for a fixed $x$ cannot be realistic in some cases. Therefore, the proof should be extended for the case that the support of $x$ is an open subset of $\\mathbb{R}^p$ where $p$ is the dimension of $x.$ \n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "ODa07P0mvlH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the label solicitation strategy in active learning. In particular, it focuses on the expected loss reduction (ELR) strategy, analyzes its problem, and modifies the original ELR method to make sure the active learner converges to the optimal classifier along learning iterations. The paper provides theoretical guarantees on the new method’s convergence. In the experiment, the proposed method is evaluated on synthetic data and UCI data. The improvement margin over the existing method is very limited.\n \nStrong point:\n1. The paper’s finding on the existing ELR method is interesting and novel.\n2. The theoretical analysis of the convergence of the proposed method seems to be sound.\n \nWeak point:\n1. The experiment is conducted on low-dimensional data and the proposed method’s performance is not very competitive.\n2. The notation in this paper can be confusing to readers, especially the use of (*). Star usually means the “optimal”.\n3. The main paper does not have an algorithm.\n4. There is no validation in experiments for the theory. In the synthetic experiment, it should be possible to simulate a case with ground truth optimal classifier and verify whether the proposed method actually converges to the optimal.\n \nMy major concern is the practical impact of the proposed method. Therefore, I recommend a weak reject for this paper (5).\n \nAdditional questions and suggestions:\n \n1. I think the paper would be improved if there is a discussion on how the proposed method can be extended to deal with high-dimensional data and/or using deep learning models.\n\n2. It seems to me the proposed weighted method is not the only way to guarantee convergence. But I am not sure about that. It would be nice to have some discussion about that.\n\n3. The one-step-look-ahead strategy involving expectation model change or expected loss reduction usually suffers from the large class space for computing the expectation. The experiments are mostly conducted in a small class space. It would also be good to have a discussion about the complexity in terms of class space.\n\n4. MES usually suffers a lot from noise (experiments in the appendix also show that). ELR methods are usually more robust to noise. I was wondering whether different noise level has been tried and how the proposed method compare with ELR on that. A similar question is also, for certain data there is a larger gap between the proposed method and ELR (in appendix). What would be the reason?   \n\n5. The visual presentation can be improved for the paper, as well as the explanations. In figure 1’s explanation, I got very confused by the “side”. What does that mean?\n\n6. The results should be shown with error bars if experiments are conducted multiple times.\n \n================\nUpdate after rebuttal:\n\nI increased the score to 6 and appreciated the revision of the paper. The readability is improved. However, I also have different opinions with the authors in terms of how empirical evaluation of algorithms should be regarded in active learning research. So I would further encourage the authors to apply their method on high-dimensional large scale data, even it may take a lot of computing resources or require actual sample acquisition. \n\nI agree that the goal of active learning is to reduce the burden of labeling data. But it does not conflict with the requirement of dealing with high-dimensional (feature space) data. Also, I see a lot of active learning works focusing on theoretical analysis but cannot be easily put into real-world applications, which actually undermines the significance of the theory to some extent. In the real world, a lot of assumptions would be violated. As the authors also mentioned that, it is \"expected\" that different feature space and data quality affects the performance. Therefore, I think the theory does not spare us from justifying our methods in practice. \n\nLast but not least, actual sample acquisition is not unrealistic if given real-world problems. So I encourage the authors to further demonstrate the nice properties of the proposed algorithms in more realistic settings in the future.", "extra_scores": {}, "extra_text": {}}, {"review_id": "FkhoPsHrZj", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors of this paper introduced a new acquisition function of active learning for optimal Bayesian classifier. The new query strategy is based on mean objective cost of uncertainty, defined as the expected difference between losses of the optimal Bayesian classifier and the optimal classifier.\n\nI think this paper can benefit from revisions to improve clarity. Unfortunately, in the current state of writing, I found it very difficult to understand what this approach is doing exactly. The lack of clarity makes it hard to appreciate the interestingness of the proposed approach. In the following, I will list some possible improvements and my confusions.\n\n1, In the abstract, please rewrite the sentence \"To improve convergence ... classification error.\" This sentence is probably the most important summary of this work, but it's so long and dense that it's very difficult to parse. I believe people read abstract to get a general idea of what this paper is, not a dense summary of what the technical details are. \n\n2, The introduction should be called related work. Especially in 2nd-3rd paragraph, the authors tried to pack all competing algorithms in and explain why they don't work well. It is too detailed, I think. I expect more high-level descriptions of why this problem is important, where the field is now, or why the authors think this is an important problem to solve rather than, e.g., solving active learning for regression. \n\n3, The authors have a tendency of defining a symbol or an abbreviation, and expect the readers to register them in their memory. It would help the clarity significantly if the authors could just repeat in English what \\pi (or \\phi or C_\\theta , M, U etc) is again when they are mentioned. \n\n4, It's not clear to me why the entire introduction and definition of MOCU is under section 3.1 Analysis of ELR Methods. Perhaps section 3.1 needs to be segmented into more subsections.\n\n5, I'd recommend that the authors leave the most important theorem in the main paper and move all the less important lemmas and proofs to appendix. Then, the authors can have more space explaining the intuitions behind the proofs and the newly designed weighted MOCU. It is nice to see the convergence analysis, but it also makes me wonder how useful it really is. The theorem is saying, as we get infinite samples, we get the optimal classifier, under a bunch of conditions. It's nice to have, but it almost feels like every algorithm that can loop through all possible inputs can do that. What about the convergence rate that we care more about? Or how many active learning iteration is needed to achieve a certain performance.\n\n6, While it is unclear how useful the theoretical guarantees are, it is also unclear if the empirical results should enough evidence. Only toy datasets were examined, and the performance of the proposed approach is quite similar to other competitors.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposed weighted-MOCU, a novel objective-oriented data acquisition criterion for active learning. The propositions are well-motivated, and all reviewers find the analysis of the drawbacks of several popular myopic strategies (e.g. ELR tends to stuck in local optima; BALD tends to be overly explorative)) interesting and insightful. Reviews also appreciate the novelty of the proposed weighted strategy for addressing the convergence issue of MOCU-based approaches. Overall I share the same opinions and believe the paper offers useful insights for the active learning community.

In the meantime, there were shared concerns among several reviewers in the readability (structure and intuition), lack of empirical results on more realistic active learning tasks, and limited discussion on the modeling assumptions. Although the rebuttal revision does improve upon many of these points, the authors are strongly encouraged to take into account the reviews, in particular, to further strengthen the empirical analysis and discussions, when preparing a revision.

Decision: Accept (Poster)