Active Learning in Bayesian Neural Networks: Balanced Entropy Learning Principle
Authors: Jae Oh Woo
Venue: NeurIPS 2022 Submitted (2022)
Keywords: bayesian neural network
bayesian active learning
balanced entropy learning
uncertainty quantification

Abstract: Acquiring labeled data is challenging in many machine learning applications with limited budgets. Active learning gives a procedure to select the most informative data points and improve data efficiency by reducing the cost of labeling. The info-max learning principle maximizing mutual information such as BALD has been successful and widely adapted in various active learning applications. However, this pool-based specific objective inherently introduces a redundant selection. In this paper, we design and propose a new uncertainty measure, Balanced Entropy Acquisition (BalEntAcq), which captures the information balance between the uncertainty of underlying softmax probability and the label variable. To do this, we approximate each marginal distribution by Beta distribution. Beta approximation enables us to formulate BalEntAcq as a ratio between a shifted entropy and the marginalized joint entropy. The closed-form expression of BalEntAcq facilitates parallelization by estimating two parameters in each marginal Beta distribution. BalEntAcq is a purely standalone measure without requiring any relational computations with other data points. Nevertheless, BalEntAcq captures a well-diversified selection near the decision boundary with a margin, unlike other existing uncertainty measures such as BALD, Entropy, or Mean Standard Deviation (MeanSD). Finally, we demonstrate that our balanced entropy learning principle with BalEntAcq consistently outperforms well-known linearly scalable active learning methods, including a recently proposed PowerBALD, a simple but diversified version of BALD, by showing experimental results obtained from MNIST, CIFAR-100, SVHN, and TinyImageNet datasets.

Active Learning in Bayesian Neural Networks: Balanced Entropy Learning Principle
Anonymous Author(s)
Affiliation
Address
email
Abstract
Acquiring labeled data is challenging in many machine learning applications with limited budgets. Active learning gives a procedure select the most informative data points and improve data efficiency by reducing the cost of labeling. The info- max learning principle maximizing mutual information such BALD has been successful and widely adapted in various active learning applications. However, this pool- pool-based specific objective inherently introduces redundant selection. this paper, we design and propose new uncertainty measure, Balanced Entropy Acquisition (BalEntAcq), which captures the information balance between the uncertainty of underlying softmax probability and the label variable. To do this, we approximate each marginal distribution by Beta distribution. Beta approxi- mation enables to formulate BalEntAcq as a ratio between shifted entropy and the marginalized joint entropy. The closed-form expression of BalEntAcq facilitates parallelization by estimating two parameters in each marginal Beta distribution. BalEntAAq
10
11
12
14
purely standalone measure without requiring any relational computations with other data points. Nevertheless, BalEntAcq captures a well-diversified selection near the decision boundary with margin, unlike other existing uncertainty measures such BALD, Entropy, Mean Standard De- viation (MeanSD). Finally, we demonstrate that our balanced entropy learning principle with BalEntAcq consistently outperforms well-known linearly scalable active learning methods, including a recently proposed PowerBALD, a simple but diversified version of BALD, by showing experimental results obtained from MNIST, CIFAR-100, SVHN, and TinyImageNe datasets.
16
18
20
22
Introduction
23
24
Acquiring labeled data is challenging in many machine learning applications with limited budgets. As the dataset size gets bigger and bigger for training complex model, labeling data by humans
25
28 becomes more expensive. Active learning gives a procedure to select the most informative data points and improve data efficiency by reducing the cost of labeling.
27
28 The active learning problem is well-aligned with subset selection problem that can find the most efficient but minimal subset from the data pool [70] [4] [8
86 88 85). The difference is that 30 active learning is typically an iterative process where a model is trained and a collection of data
66
31 points is selected to be labeled from unlabelled data pool. Therefore, it is still theoretically very challenging but important problem.
32
It is now commonly accepted that standard deep learning models do not capture model uncertainty correctly. The simple predictive probabilities are usually erroneously described as model confidence 31. So there is a risk that a model can be misdirecting its outputs with high confidence. However, the predictive distribution generated from Bayesian deep learning models better captures the uncertainty
34
36
Submitted to 36th Conference on Neural Information Processing Systems (NeurIPS 2022). Do not distribute.

---

37 from the data 26 51 67 17.
Therefore, we focus on developing an active learning framework in 38 the Bayesian deep neural network model by leveraging the Monte-Carke Carlo (MC) dropout method as proxy of the Gaussian process [26] which may facilitate further analysis.
40
1.1
Our contributions
Our proposed active learning method is well-aligned with Bayesian experimental design {89
14 80
42
62
with an assumption that the forward active learning iterative process follows the Bayesian approach is also aligned with Bayesian uncertainty with an assumption that the working neural
prior-posterior framework. Furthermore,
44
quantification methods (40] [ 41
26
27 48 67 47
45
network model is Bayesian network
44
46
In this paper, we extend and improve recent advances in both aspects of Bayesian experimental design and Bayesian uncertainty quantification. We investigate the generalized notion of the joint entropy between model parameters and the predictive outputs by leveraging a point process entropy 664
48
16 [ By approximating the marginals using Beta distributions, we then derive an explicit formula of the marginalized joint entropy by estimating Beta parameters from Bayesian revisit the well-known entropy and mutual information measures given expected cross-entropy loss. We show that well-known acquisition measures are functions of marginal distributions through analytical formulas. We propose new uncertainty measure, Balanced Entropy Acquisition (BalEntAcq). which captures the information balance between the uncertainty of underlying softmax probability and the label variable. Finally, demonstrate that our balanced entropy learning principle with BalEntAcq consistently outperforms well-known linearly scalable active learning methods, including a recently proposed PowerBALD for mitigating the redundant selection in BALD (27), by showing experimental results obtained from MNIST, CIFAR- 100, SVHN, and Tiny ImageNet datasets.
25 73
50
deep learning models. As Bayesian experiment,
52
54
56
57
58
47
59
2
Background
60
61
2.1
Problem formulation
We write an unlabeled dataset pool and the labeled training set Praiiing C Dpoot in each active learning iteration. We denote by ((x) training it's necessary to indicate the specific n-th iteration step. Given 2 Preiiingg we train Bayesian deep neural network model & with model parameters ~ - (w).
83
64
Then for data point x given D Diraining. the Bayesian deep neural network + produces the prediction probability: & (x. w) == (Pi (X)(x.)), ...... (o w)) A where A {(...... .Pc) P1 +
67
PC = L. 2 for each i} and C/ is the number of classes. For the final class output Y, it is assumed to be multinoulli distribution (or categorical distribution):
with probability Pi (1 w)
Y(,
with probability Po(x, w).
69 For the sake of brevity, we sometimes omit x or w by writing + (w), P.(w). (ww) or L. P., Y unless
70 we need further clarifications on each data point x. Under this formulation. the oracle (active learning
71 algorithm) selects subset of data points to add to the next training set, i.e. at (n + 1) 1)-th iteration,
72 the training set is determined by D ((++1)
training D ((1)) U (Next training batch from Oracle). Once the training
73 next training batch is selected, the selected batch will be labeled. This means that the ground truth
74 label information the selected data is added in training set (2++1) training in the next round. Then the 75 goal in active learning is to minimize the number of selected data points to reach a certain level of prediction accuracy.
77
2.2
Examples of uncertainty based active learning methods
78 In this section, we list up well-known uncertainty measures suitable for Bayesian active learning.
79 Random: Rand[x] == U(w') where U.-) uniform distribution which is independent to w.
Random acquisition function assigns random uniform value on [0, 1] to each data point.

---

81
2. BALD (Bayesian active learning by disagreement) [58 35 27|
BALD|x] := 3 (w. (x, w)). where -(.)) represents a mutual information between random measures. BALD captures the mutual information between the model parameters and the predictive output of the data point. In practice, we calculate the mutual information between the predictive output and the predictive probabilities.
82
83
84
85
3. Entropy 82 Ent[x] := = E; (EP:) log (E.;)) Entropy is the Shannon entropy with respect to the expected predictive probability. Entropy can be the uncertainty of the prediction probability, Moreover, under the cross-entropy loss, we may also interpret the entropy measure as an expected loss gain since - log (EP:) is the cross-entropy loss given the ground truth label is the class i.
87
88
89
90
4. Mean standard deviation (MeanSD) 14
40 MeanSD[x] := c Ex EP? (EP:)² Mcan standard deviation captures the average of the standard deviations for each marginal distribution.
91
5.
log BALDÍx] Z, where Z is independently generated random value from Pareto distribution with the exponent a > O. We use a as default choice suggested by 447) The motivation of this randomized acquisition is to mitigate the redundant selection by diversifying selected multi-batch points. In general, we do not know which exponent will be the optimal choice.
PowerBALD 21 47
PowerBALDíx]
93
94
85
96
In multiple acquisition scenario, we simply add the above uncertainty values for each data point x,;
AcqFunc|xx, [n.]
AcqFunc|xx] [x.],
(2)
98
where AcqFunc E (Rand, BALD,, Ent,
99
2.3
Summary of other active learning approaches
100
Cohn et al. [4] provided one of the first statistical analyses in active learning, establishing how to synthesize queries that reduce the model's forward-looking error by minimizing its variance leveraging MacKay's closed-form variance approximation (60] In this fashion, there exists a line of with assumption that the forward active learning iterative process follows Bayesian prioe-ostterior framework.
102
works in Bayesian experimental design 11]
58
89
14
80
90 24 23 38
104
105
On the other hand, in active learning, accommodating both the information uncertainty and the diversification the acquired samples is essential to improve the performance under multi-batch acquisition scenarios. In theoretical perspective, the most natural way to combine the uncertainty and the diversification seems to leverage reasonable sub-modular functions, e.g. Nearest neighbor set function [92]. BatchBALD 48), Determinantal Point Process [6] and SIMILAR (50) with sub- modular information measures, and then/or apply a fast linear- time algorithm to find a diversified multi-batch with provable performance guarantee [69 (7 20 95|
106
108
110
79 36 37 57). Although fast linear-time solver available for general sub-modular functions, there still exists a gap with practical implementation, such as high memory requirements, which makes the computation unscalable for identifying multi-bated acquisition points, e.g.. BatchBALD [48]. Similar to the sub-modular function optimization, there exist many customized optimization approaches, e.g. CoreSet [8]] and more approaches 29
112
114
116
94
117
Another recent approach is to look at parameters of the neural network and to diversify points such as BADGE [4] with gradients and BAIT [3] with Fisher information. There also exist network architectural design focused approaches such as Learning loss by designing loss prediction layers (96), UncertainGCN and CoreGCN [8] with graph neural networks, VAAL [84) and TA- VAAL (2)
118
120
by applying adversarial learning methods.
Bayesian neural network model
122
123 We adopt the Bayesian neural network framework introduced in Gal et al. 226]. The core idea in the Bayesian neural network is leveraging the MC dropout feature to generate distribution of the predictive probability as an output at inference time. Under mild assumptions, it turns out that
124
125
equivalent to approximation Gaussian Process [77 68 93 26 (56).
3

---

127
3.1
Each softmax probability marginal approximately follows Beta distribution
128
We may consider a Bayesian neural network model & as a random measure, i.e., stochastic process parametrized by Dreaining over the data set poot- Given data point x E pool. ① (x,w) produces a random probability distribution in a simplex A This analogy has a close connection with the construction of random discrete distribution, originally introduced by Kingman 45]. Since then, random measure construction has been extensively developed in Bayesian nonparametrics, and it is well-known that Dirichlet probability having Beta marginals plays the central role in the construction of the random discrete distribution 466 22 [5 74 7 72 778. It is the main motivation of the Beta distribution approximation. Many kinds of literature similarly assume the Dirichlet distribution after the softmax in the Bayesian neural network.
129
130
131
132
133
134
135
137
may follow the construction of Dirichlet distribution. Follow- ing the approach by Ferguson [2 Dirichlet probability can be constructed through a collection of independent Gamma distributions. On the other hand, each marginal in Gaussian Process (approx- imated by Bayesian neural network) in the softmax output having dependent components follows a nor-nolmal distribution (before the normalization, but after the exponentiation in softmax). Then by applying the shape similarity between a log-normal distribution and Gamma distribution, the construction of random probability from log- normal distributions would produce an approximated Dirichlet distribution. Therefore we may assume that the marginal distribution would approximately follow the Beta distribution.
As illustrated by Milios al. 665
138
139
140
141
142
143
144
145
146 A lternatively, as an analytical approach, we may see that Beta approximation can be justified through Laplace approximation [6] 32 33 [7. There exists mapping between multivariate Gaussian distribution and Dirichlet distribution under a softmax basis. Then Beta distribution follows as a marginal distribution of Dirichlet distribution. Therefore we may assume that Beta approximation exists through Laplace approximation under the assumption that the Bayesian neural network produces the multivariate Gaussian distribution (as marginalized Gaussian process over finite rank covariate
147
148
149
150
151
152
function) before the softmax layer 668 93 26 [6]]
In practice, once we estimate the sample mean and sample variance for each marginal of $ (x,w), we can estimate two parameters of the Beta distribution as follows. Assume that P; ~ Beta (oz, 3:)).
153
154
If E.P = m; and Var = of, then
155
m² m)
-1)
1 m
m,
B
(3)
When P3 ~ Beta (a;, B:), E.,-- equation with respect to a₁ and Br, then the (3) follows.
m and VarP; (a.
a =0.. Solving the
t
157
3.2
Marginalized joint entropy in Bayesian neural network
159
In this section, we derive a marginalized joint entropy in the Bayesian neural network, which shall be further discussed in constructing our main results. We may formulate the Bayesian neural network cb as a well- well- encoder-decoder framework. The sender sends a message w) with a random key through the Bayesian neural network, then the receiver receives message Y (x,w).
160
161
162
163
Under this framework, controlling w is difficult, but we can control the family of the encoded messages & (x, w) in tractable manner 27 43 88). We can easily prove that the mutual information
164
165
between w and Y is the same as the mutual information between the encoded & (x,w) and the predictive output Y since Y depends only on & (x;w);
BALD[x] =- (w,, Y (x, w)) H( ()) E I1 (Y (x, [w)
(4)
=Y (x, w)) - E4 [ (Y (x, w) | (x, = (4 (x,w), Y w)),
(5)
where HY (x, (x, represents the Shannon entropy by marginalizing out the randomness of w in Y (x,w) and ((.)) represents a mutual information between two quantities.
167
168
189
The formulations of the mutual information
5 look natural, but we need to note that w
or & (x,w) is on a continuous domain, and (x.w) is on a discrete domain. This combined
170
domain implies that we cannot directly apply Shannon entropy and differential entropy notions [15]
One immediate question is what the joint entropy between & (x, and ((x,w) w) is. For this, we
172
can leverage point process entropy 664| 25 73
16 5 by generalizing the notion of the entropy
4

---

174 in this combined domain. We consider the joint entropy of Φ (x) and (xx, w), denoting by
175 5 ( () Y (xx,)) w)) through the point process entropy. We write a Janossy density function [[6] (P = i) of ($ (x,w), Y w:) on A c [C] as follows:
176
j (P.y = i) (p),
(6)
177
where p := (p . pc) and f(-) is density function of & (x, Then the joint entropy of ① ()) and Y (x, can be defined as
178
5 ( (x, Y(,, = - (p. - logj (p. dp.
179
By plugging 6 into 7p. we have the following identity.
5 ( (x) ((, =H (x,w)) + Ey [h ( (x Y ()],
(8)
where H.) represents the usual Shannon entropy, and h(-) represents the usual differential entropy.
181 By applying Jensen's 's inequality, we may derive a marginalized joint entropy as an upper bound of
182
the joint entropy:
5 ( (x,w), ((x, S EE, EP, IPg (P.f(P.))),
183
where we ambiguously write f(-) to be a density function for each P.. Assume that each P. ~ Beta 3e) by applying Beta approximation. We then define a quantity of the marginalized joint entropy from ( and we find an equivalent formulation as follows:
184
185
MJEnt[x] = Σ Ep. [P. log (P.S(P.)) = Σ (EPP) + H(Y)
(10)
entropy
posterior uncertainty
186
where p. the conjugate Beta posterior entropy of P. which follows Ps ~ Beta + +,, We
187
remark that h((,,t) can be easily calculated by the closed form entropy formula of Beta distribution.
L..
M(P,t) = log B(a; + 1, - Vaa + +1) 1) (8; ()) (a; + B, - 1)) + +3++1 + 1),
where -): is the Beta function, and (-)) is the Digamma function. We call the first term in (TO)
189
190
to be the posterior uncertainty. We may interpret the posterior uncertainty an expected posterior
191
entropy assuming that we observed a positive sample of the class toward P, for each without
192
knowing the true class label. The first term is always non-positive, and is maximized (equals to 0)
193
when each p. is Beta(1, 1), i.e., Uniform on [0, 1]. So -00 < MJEnt[x] S H(Y). The second entropy term can be decomposed into two uncertainty terms:
H(Y)= J (w, Y
E (Y
(11)
epistemic uncertainty
alcatoric uncertainty
195
The epistemic uncertainty captures the model uncertainty (as BALD). and the aleatoric uncertainty captures the data uncertainty (63). Therefore the marginalized joint entropy, MJEnt[x] is a decompo- sition of three types of uncertainty values.
196
197
198
3.3
Entropy is for maximizing an expected cross-entropy loss
199
Given a ground-truth label (Y = i), the cross-entropy loss of the neural network can be given as loss (o (x,w), Y === i) = log E.P, Therefore we can calculate the expected cross-entropy loss without knowing the truth label:
200
ExpectedLoss/x/ := Σ P = loss (4 (x,w), i)= E (EP,) log (E.,) = Ent[x].
i=1
202
Based on the re-formulation, we may interpret that entropy acquisition is for maximizing an expected cross-entropy loss in a selection of acquisition points, aligning the idea with the learning loss [96].
203
204
The natural question is, "Once we acquire data point that maximizes entropy acquisition, can we remove/or learn this expected cross-entropy amount of loss at the future stage of the active learning?".
205
206
The answer could be "No." The exhaustive loss acquisition could only happen if the neural network perfectly over-fits the training data. Therefore, there exists a gap between realistie neural network training scenario and the objective of the entropy acquisition. Our equivalent loss interpretation gives us an insight into why the entropy acquisition might not be successful practice, even in the single-poit pogltt acquisition scenario.
207
209
5

---

211
3.4 BALD is a function of marginals and is strongly aligned with maximizing an expected cross-entropy loss difference upto the next iteration
212
213
We have the mutual information between w and Y and it is the same as the mutual information between the encoded message and the channel output since Y depends only on D (x,w) 227:
214
BALD(x] :- 3 Y) = J ( ( (xx,w))
(12)
215
where J(.-)) represents mutual information between two quantities. By assuming that & (x, w)
follows a Dirichlet distribution, we can calculate the mutual information analytically [2]. Then by investigating further into the analytical mutual information formula, we see that the marginal distributions P; ss in & (x,w) are sufficient to estimate BALD. Therefore we can state BALD through Beta marginal distributions as follows. See Appendix for more details.
217
219
220
Theorem 3.1. Under Beta marginal distribution approximation, let P, ~ Betala Bi) in & (x,w).
221
Then the mutual information BALD[x] can be estimated as follows:
BetaMarginalBALDii
=
Σ (a, (a. + +.)
C
C
04 + +
03
log
a, + +
ai
a (a: 4 (a.)
ag + 3.
222
8, (a; - 1) v (a. + A + a, +
of (a. +1) - (a. + B + 1)]
a, +
As Bayesian experimental design process, we may assume that each Beta marginal distribution P; with the ground-truth label (Y i} of the next trained model would follow the Beta posterior distribution P... Without this assumption, existing choices of acquisition functions such as BALD or McanSD might not be well-justified. For example, what is the implication of maximizing mutual information through the active learning process with a Bayesian neural network? How is it different from the maximization of the entropy acquisition? To answer these questions, leveraging
224
225
227
Beta marginalization and considering the similar idea of expected information gain (24), we may consider the expected cross-entropy loss difference between the current stage model and the next stage model.
229
ExpectedEfffecciieeLoss xx
E.,
| logEP; (- log EPPt]]
Σ 1=1
ai a5 + 3,
log
a, + Bg +1 -log as + 8 )}
0i + 1
ExpectedEfficciieeLsss captures the effective amount of cross-entropy loss for the model to learn after the acquisition. By definition, we see that ExpectedEffecciiee..so aims to exclude the undesirable over- wet-fitting scenario assumption unlike Entropy acquisition.
232
234
Since Digamma function (x:) ~ log x 2e where (x) ~ g(x) implies lim f(x)//(>)) = we may expect that BetaMarginalBBADD
235
and snpectedfffectiiee.oss/// would behave similarly. Figure [ shows the Spearman's 's rank correlations among different acquisition measures upto a class dimension C' 10, 000. We observe that BetaMarginalBALD behaves equally like the original BALD and we confirm that BALD and MeanSD are strongly aligned with maximizing ExpeccedEfiectiveeooss.. Therefore, acquiring points through BALD or MeanSD could be better strategy than Entropy because BALD or MeanSD takes into account the effective loss acquisition instead of the unrealistic full amount of the loss acquisition.
236
238
240
4 Balanced entropy learning principle
243
The previous section shows that well-known acquisition measures have an objective toward the cross-entropy loss, and they are closely related to marginal distributions. According to Farquhar et al. 21), be successful in active learning, they hypothesize that it is crucial to find a good balance between active learning bias and over-fitting bias under over wer-parametrrized neural networks.
244
245
246
247
In parallel to their hypothesis, we define the balanced entropy (BalEnnt to be a ratio between the marginalized joint entropy
(9) and the shifted entropy:
MJEnt[[]
Σ, (EP.)((P)) + HYY)
BalEnt[x] x := Ent[x] + log
(13)
H(Y) log 2

---

100
105
belatay ra MLL DhacleDee
Apeccnoelleeenic (etentryy re DAL DALD
speess HOLD
Figure 1: Scatter plot at C' 10, 000 between BALD and ExpectedEffectivccle (left), Spearman's rank correlations over various class dimensions (middle), and Spearman's rank correlation matrix at C 10, 000 (right). The relationship between BetaMarginalBAALD and ExpectedEfffectieeL.ss consistently captures a high cank-correlation with > 99.6% regardless the class dimensions. BALD and ExpectedEfffectiveeoss show > 97.5% rank-correlation. We randomly generate 100 softmax applied C-dimensional Gaussian samples and repeated the process 10 times. Shaded band shows the standard deviation.
249
Recall that we call the first term in MJEnt[ to be posterior uncertainty, and it is an expected posterior entropy of underlying marginals. BalEnt captures the information balance between the posterior uncertainty from the model & and entropy of the label variable Y.
250
252
4.1
Implications of balanced entropy
To understand the implication of BalEnt[x]] we prove the following Theorem 4.1
Theorem 4.1. Let A : [2e"()) and T == {In collection of evenly divided intervals in
254
[0. 1] where In = [[n - 1)4, nA) for = 1, 1,... (A - 1) and Ia- := A, 1). Let P, be
255
discretized random variable over T of P. fromrmm +b (x,w), w). For any estimator P; of P. given the label
257
{Y= = i} we have
E, (CP)))P)++HH (1+E))
P
P. PRAPY-
BalEnl | E1) -2:
H(Y) + log
258
where E1, 62 0 are adjustment terms depending on A such that E1 + and e2 0 as A O.
259
Theorem 4.1 tries answer the following inverse problem. For the unlabeled data point, x, if we know the information of the label {Y i}. how much can we reliably estimate the underlying probability
280
261
P; from the model क As we know that log P; is the cross-entropy loss of the trained model with Y. it equivalently answers the estimation error probability of the loss prediction under unit precision
262
263
up log A level. For the precision level, we are assuming to earry - log
H (Y) natural unit of information, re-scaled amount of bits, matching the enumerator with MJEnt [x] term. It
(Y)+ + log nats
265
not clear how to determine better choice of the precision level log A. But we may understand
denominator H(Y) + log 2 is for normalizing the term BalEnt(x] [x] <
probability. Then the sign of [x] becomes very important. BalEnt [x] 2 implics that it could be impossible to perfectly predict the loss log P. given currently available information. i.c., there could exist information imbalance between the model and the label approximately starting from BalEnt[x] O. Therefore,
287
289
insight from Theorem 4.1 suggests
direction for main active learning principle. We define our primary acquisition function, namely, balanced entropy learning acquisition (BalEntAcqq.
271
follows:
[BalEn[[x] -1 if BalEnt ix] 2 0, BalEnt ix] < 0,
BalEntAcqix] :=
BalEnt[x]
273
Since the information imbalance exists at least from BalEntix] 0, we prioritize to fill the in- formation gap from BalEnt[x] 0 toward positively increasing direction. If we try to fill the information imbalance gap from the highest BalEnt(x], the information imbalance would still exist around BalEnt[x) = 0 area. Therefore, it might not improve the active learning performance much.
274
275
277
See Appendix A12.2 and A A12.3 for different prioritization and precision level results. That's the motivation why we take the reciprocal of BalEnt[x] when BalEnt[x]] > 0.
7

---

(a) BalEntAcq (ours)
(b) BALD
(c) Entropy
(d) MeanSD
(e) PowerBALD
Figure 2: Top-K K selected points are marked by red color. The first row shows the top K = 25 points.
The second row shows the top K 500 point selections among around 0.6 million grid points.
279
4.2
Toy example illustration
280
To illustrate the behavior of BalEntAcq and its relationship with other uncertainty measures, we train a simple Bayesian neural network with 3-class moon dataset in Then we calculate each acquisition measure for all fixed lattice points in the square domain by assuming that the unlabeled pool is highly regularized (or uniform). i.e., by evenly discretizing the domain, we obtain each uncertainty value for each lattice point. The total number of lattice points is around 0.6 million. Then we choose top-K high uncertainty values for each method to observe the prioritized region for each method. We use K = 25 and K = 500. Figure 2 illustrates the top- points selected by each method. The most significant phenomenon is that BalEntAcq's 's selection is highly diversified near the decision boundary showing a bifurcated margin because we are prioritizing the surface area of [aalEn[[]] 2 0}. This is well-all well-aligned with the strategy avoiding high aleatoric points. (See Appendix A.13) Then we can imagine to conduct a uniform sampling on each contour surface [BalEnt[x] = }} move the surface for each À < O. That's why we observe bifurcated but diversified and balanced selection near the decision boundary with BalEngAcq in
282
284
286
288
290
for each X 2 O,
292
Figure (a) when
293
K 25. On the other hand, there a preferred area for each method from other measures except PowerBALD. PowerBALD shows a good diversification, but it could select non-informative points.
294
5 Experimental Results
296
In this section, we demonstrate the performance of BalEntAcq from MNIST (55]
CIFAR- 100 (2],
297
SVHN (7), and Tiny ImageNet [54] datasets under various scenarios. We used single NVIDIA
298
A 100 GPU for each experiment, and details about the experiments are explained in Appendix A.12.
289
We test Random, BALD, Entropy, MeanSD, PowerBALDD ALD, and BalEntAcq measures. We add BADGE
300
for additional baseline. Note that all acquisition measures except BADGE in our experiments are stanalone quantities, so all can be easily parallelized.
301
302
Single acquisition active learning with MNIST. MNIST 55 is the most popular and elementary dataset validate the performance of image- mase--baed deep learning models initially. We use simple convolutional neural network (CNN) model applying dropouts to all layers with a single acquisition The primary purpose of this single acquisition experiment is to validate proposed balanced entropy approach by removing the contribution of diversification unlike multi-batch acquisition scenario.
304
size.
306
307
308
Fixed features with CIFAR-100 and 3x CIFAR-100. In recent years, significant efforts have been made on building an efficient framework of unsupervised or sclf-supervised feature learning such as SimCLR [2] [3]. MoCo [30], BYOL (28), SwAV [,, DINO [0,, etc. As an application in active learning, we may leverage the feature space from the unsupervised feature learning without explicitly knowing true labels but construct a good representation space. In our experiments, we adopt SimCLR for simplicity with ResNet-50 to build feature space for CIFAR-100.
310
312
12
314
With 3 CIFAR-100 dataset, we observe the effect of the redundant information treatment for each method by adding three identical points. We use the same fixed feature obtained from SimCLR with 100. We may observe how each method effectively diversifies the selection under a redundant data pool scenario by fixing the feature space.
316
8

---

318
Pre-trained backbone with SVHN and strong data augmentation with Tiny Imagent In this experiment, we follow a typical image classification scenario in practice. We use the ResNet- 18 backhone for SVHN and the ResNet-50 backbone for TinyImageNe with ImageNet pre-trained model for model architecture, and the last linear classification layer is replaced with a simple Bayesian neural network with dropouts. We apply strong data augmentations for TinyImageNe
319
320
321
322
including random crop, random flip, random color jitter, and random grayscale. Under this scenario, the feature space from the backbone is continuously evolving and keeps confused as the training and active learning process proceeds. Because of the strong data augmentation and batch normalization in ResNet-18 or 50, the decision boundary keeps confused, implying that the Bayesian experimental design
323
325
327
assumption might not hold. However, we still want to observe the general behavior of each measure and how to improve the accuracy under more dynamic feature space.
Xd viry upai for
Crirg wou Stt
whilden 24:0e kiney
une Due
wennie per
5007 the
leeel sulic
(a) MNIST
(b) CIFAR- 100
(c) 3XCIFAR- 100
(d) SVHN
(c) TinylmageNet
Figure 3: Active learning accuracy curves obtained from various scenarios. Our proposed BalEntAcq outperforms well-known acquisition measures, and we repeated the experiment 3 times.
Scanaido
Tal Orspans ChN
Fload Burara
Redasder Iragas Tlud Fazzss
Buctheea
Dar a Axgmemadio
Detauet/ugg Seeflied siee
MNINTIA9 (00
CHRAR HOVVAY 2.00
1yTTPAR SIN71 (3)
SVHNNO 50V76993
Testruphhecl S6CHIU00
Tran SootPoel See
35MMUR0
IPCNCURO
503071100
DA IL
3130073227
Raadera
T8..4 1.75
56.1 4-27
6-6---77 775
15..4 0.65
St 4-05%
21.8-8:00 8.90
G1.3 0.3
$1.8
2.05
51. 13.%
35.3 4 8-355
--
Feensyy
12:4445072 5.75
118-4 0:65
091 6-9 27g
26
8:6
(R:3 8-8 1%
92. 0
35 8-1
<1.44-.45 4%
$14-2.255
941 1.35
0.0
913
04.3
82.
9-3
542 8:6f
JU3
DWwdBALD
598.2 6.1
822
2413%
13%
2.77
2.03%
BADER dalie
750 13
465
6744-0
61
610 7-
09.8 0
(6.6 9:30
27 545
(01
Babsyny lears
018:21225
2912
Table I: Selected accuracy table. Mean and standard deviation are from 3 repeated experiments.
329
Discussion. BalEntAeq consistently outperforms other linearly scalable baselines in all datasets, as BADGE performs similarly with Entropy under a single acquisition scenario in MNIST because BADGE focuses on maximizing the loss gradient similar to Entropy, as we explained
330
shown in Table I
331
332
in Section 3.3 BADGE shows better performances at first when we fix the feature space, but our
333
BalEntAcq eventually merges with the performance of BADGE. We also note that BADGE is not a linearly scalable method. Under dynamic feature scenarios in SVHN or TinyImageNet, we observe that our BalEntAcq performs better. Considering the acquisition calculation time (see Appendix A.14),
334
335
our BalEntAAq should be better choice. Figure 3 shows the full active learning curves. For CIFAR- 100 and 3 100 cases, by fixing features, we control/remove all other effects possibly affecting the model's performance, such as data augmentation or the role of backbone in the classification.
337
339 As demonstrated in Figure? 2 BalEntAcq is very efficient in selecting diversified points along the decision boundary. Instead, PowerBALD suffers from improving accuracy because it focuses more on diversification/aandmizzaooo by missing the information near the decision boundary. For SVHN
341
342 or Imn,lm,g,Net, BalEntAcq shows better performance again. We suppose that diversification near the decision boundary in BalEntAcq also plays the data exploration because the representation space
343
keeps evolving with the backbone training.
6 Conclusion
345
346
In this paper, we designed and proposed a new uncertainty measure, Balanced Entropy Acquisition (BalEntAcq),, which captures the information balance between the underlying probability and the label variable through Beta approximation with Bayesian neural network. BalEntAcq offers a diversified selection and is unique compared to other uncertainty measures. Moreover, we expect that
347
348
349
350
our proposed balanced entropy measure does not have to be confined to active learning problems in general. EntEnt can be applied to improve the diversified selection process or accuracy estimation in different type of Bayesian neural network frameworks. Therefore, we look forward to having further follow-up up studies with broad applications beyond the active learning problems.
352

---

354
Limitations
355
As we specified in the introduction, our focus is MC-dropout-based Bayesian neural networks; our experiments have been limited to dropout-based Bayesian neural networks. However, our theoretical development does not require special architectural assumptions if we can apply Beta approximation. So one can apply our proposed method to any Bayesian classification network with Beta approximation. Moreover, considering the similarity with recent theoretically guaranteed active learning algorithm with abstention [59
357
359
361
97 (see Appendix A.13), we expect to replicate the similar out- out-performance in other types of the Bayesian networks, e.g., Gaussian process [77]. ensemble network [3 variational-dropol network 44 Laplace Redux [7. and so on.
363
References
384
[1]
Vijay Badrinarayanan Alex Kendall and Roberto Cipolla. Bayesian segnet: Model uncertainty in deep convolutional encoder-decoder architectures for scene understanding. In Proceedings of the British Machine Vision Conference (BMVC), pages 57.1-57.12. BMVA Press, September 2017.
366
367
388
369
[2] Anonymous. Analytic mutual information in bayesian neural networks. To appear in 2022 IEEE International Symposium on Information Theory (ISIT), 2022.
370
371
[3] Jordan Ash, Surbhi Goel, Akshay Krishnamurthy, and Sham Kakade. Gone fishing: Neural active learning with fisher embeddings. Advances in Neural Information Processing Systems, 34, 2021.
373
374
[4] Jordan T Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alekh Agar- wal. Deep batch active learning by diverse, uncertain gradient lower bounds. International Conference Learning Representations, 2020.
375
376
377
[5] François Baccelli and Jac Oh Woo. On the entropy and mutual information of point processes. In 2016 IEEE International Symposium on Information Theory (ISIT), pages 695-699. IEEE, 2016.
378
379
380
[6] Erdem BIyIk, Kenneth Wang, Nima Anari, and Dorsa Sadigh. Batch active learning using determinantal point processes. arXiv preprint wrXiv:1906.07955. 2019.
382
[7] Tamara Broderick, Michael Jordan, Jim Pitman, et al. Beta processes, stick-breaking and power laws. Bayesian analysis, 7(2):439 476, 2012.
384
[8] Razvan Caramalau, Binod Bhattarai, and Tae-Kyun Kim. Sequential graph convolutional network for active learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9583-9592, 2021.
385
386
387
[9] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. arXiv preprint arXiv:2006.0 09882, 2020.
389
[10] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self superviid vision transformers. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 9650-9660, 2021.
391
382
393
[11] Kathryn Chaloner and Isabella Verdinelli. Bayesian experimental design: A review. Statistical Science, pages 273-304, 1995.
394
[12] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. International conference on machine learning, pages 1597-1607. PMLR, 2020.
396
10

---

398
[13] Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E Hinton. Big self-supervised models are strong semi cemi-superiised learners. Advances neural information processing systems, 33:22243-22255, 2020.
399
[14] David A Cohn, Zoubin Ghahramani, and Michael I Jordan. Active learning with statistical models, Journal of artificial intelligence research, 4:129- -222-94,, 1996.
402
403
[15] Thomas M Cover. Elements of information theory. John Wiley Sons, 1999.
404
[16] Daryl J Daley and David Vere-Jones. An introduction to the theory of point processes: volume II: general theory and structure, volume 2. Springer Science & Business Media, 2007.
405
406
[17] Erik Daxberger, Agustinus Kristiadi, Alexander Immer, Runa Eschenhagen, Matthias Bauer, and Philipp Hennig. Laplace -eflu-efforrlesss bayesian deep learning. Advances in Neural Information Processing Systems, 34, 2021.
407
408
[18] AP Dvoredsky. Some results on convex bodies and banach spaces. 1961.
410
[19] Ehsan Elhamifar, Guillermo Sapiro, Allen Yang, and S Shankar Sasity. A convex optimization framework for active learning. In Proceedings of the IEEE International Conference on Computer Vision, pages 209-216, 2013.
411
413
[20]
Alina Ene and Huy L Nguyen. A nearly-lineat time algorithm for submodular maximization with knapsack constraint. arXiv preprint 17Xiv:1709.0990 09007, 2017.
414
415
[21]
Sebastian Farquhar, Yarin Gal, and Tom Rainforth. On statistical bias in active learning: How and when to fix it. International Conference on Learning Representations, 2021.
416
417
[22] Thomas S Ferguson. A bayesian analysis of some nonparametric problems. The annals of statistics, pages 209-230, 1973.
418
419
[23] Adam Foster, Desi R Ivanova, Ilyas Malik, and Tom Rainforth. Deep adaptive design: Amortiz- ing sequential bayesian experimental design. In International Conference on Machine Learning. pages 1384-3339.. PMLR, 2021.
420
421
422
[24] Adam Foster, Martin Jankowiak, Elias Bingham, Paul Horsfall, Yee Whye Teh, Thomas Rainforth, and Noah Goodman. Variational bayesian optimal experimental design. Advances in Neural Information Processing Systems, 32, 2019.
423
424
425
[25] Fritz. An approach to the entropy of point processes. Periodica Mathematica Hungarica, 3(1-2):73-83 1973.
426
427
[26] Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty deep learning. In international conference on machine learning, pages 1050-1059. PMLR, 2016.
428
429
430
[27]
Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep bayesian active learning with image data. In International Conference on Machine Learning, pages 1 1183-1192. PMLR, 2017.
431
432
(88] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, al. Bootstrap your own latent-a new approach supervuee learning. Advances in Neural Information Processing Systems, 33:21271-21288 2020.
433
434
436
[29] Yuhong Guo. Active instance sampling via matrix partition. Advances in Neural Information Processing Systems, 23, 2010.
437
438
[30] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages
440
2020.
441
[31] Mauhias Hein, Maksym Andriushchenko, and Julian Bitterwolf. Why relu networks yield high-confidence predictions far away from the training data and how to mitigate the problem. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages
443
41-50, 2019.
11

---

445
[32] Philipp Hennig, David Stem, Ralf Herbrich, and Thore Graepel. Kernel topic models. In Artificial intelligence and statistics, pages 511-519. PMLR, 2012.
448
[33] Marius Hobbhahn, Agustinus Kristiadi, and Philipp Hennig. Fast predictive uncertainty for classification with bayesian deep networks. arXiv preprint [tt::2003.01222)) 2020.
448
[34] Dorit S Hochbaum. Approximating covering and packing problems: set cover, vertex cover, independent set, and related problems. In Approximation algorithms for NP-hard problems, pages 94-143, 1996.
450
452
[35] Neil Houlsby, Ferenc Huszár, Zoubin Ghahramani, and Máté Lengyel. Bayesian active learning for classification and preference learning, arXiv preprint vxXiv1112.57453 2011.
454
[36] Rishabh Iyer, Ninad Khargonkar, Jeff Bilmes, and Himanshu Asnani. Generalized submod- ular information measures: Theoretical properties, examples, optimization algorithms, and applications. IEEE Transactions on Information Theory, pages 2021.
456
[37] Rishabh Iyer, Ninad A Khargonkar, Jeffrey A. Bilmes, and Himanshu Asnani. Submodular combinatorial information measures with applications in machine learning. In The 32nd International Conference on Algorithmic Learning Theory, Virtual Conference, March 2021.
458
459
480
[38] Aditi Jha, Zoe C Ashwood, and Jonathan W Pillow. Bayesian active learning for discrete latent variable models. arXiv preprint arXiv:2202.13422 2022.
461
462
[39] Ajay Joshi, Fatih Porikli, and Nikolaos Papanikolopoulos. Multi-class batch-mode active learning for image classification. In 2010 IEEE international conference on robotics and automation, pages 1873-1878. IEEE, 2010.
463
464
465
[40] Michael Kampffmeyer, Arnt-Borre Salberg, and Robert Jenssen. Semantic segmentation of small objects and modeling of uncertainty in urban remote sensing images using deep convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 1-9, 2016.
466
467
469
[41] Kirthevasan Kandasamy, Jeff Schneider, and Barnabás Póczos. Bayesian active learning for posterior estimation. In Proceedings of the 24th International Conference on Artificial Intelligence, pages 3605-3611, 2015.
471
[42] Kwanyoung Kim, Dongwon Park. Kwang In Kim, and Se Young Chun. Task-aware variational adversarial active learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8175, 2021.
473
475
[43] Diederik P Kingma and Max Welling. Auto- aato-Encoding Variational Bayes. In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014.
477
[44] Durk P Kingma, Tim Salimans, and Max Welling. Variational dropout and the local reparame- terization trick. Advances in newral information processing systems, 28, 2015.
479
[45]
John FC Kingman. Random discrete distributions. Journal of the Royal Statistical Society: Series B (Methodological) 37(1):1-15, 1975.
481
482
[46]
John FC Kingman. The population structure associated with the ewens sampling formula. Theoretical Population Biology, 1(2):274-283, 1977.
483
484
[47] Andreas Kirsch, Sebastian Farquhar, and Yarin Gal. A simple baseline for batch active learning with stochastic acquisition functions. arXiv preprint arXiv:2 20059, 2021.
486
[88] Andreas Kirsch, Joost van Amersfoort, and Yarin Gal. Batchbald: Efficient and diverse batch acquisition for deep bayesian active learning. 2019.
488
[49] Daphne Koller and Nir Friedman. Probabilistic graphical models: principles and techniques. MIT press, 2009.
12

---

490
[50]
Suraj Kothawade, Nathan Beck, Krishnateja Killamsetty, and Rishabh Iyer. Similar: Submod- ular information measures based active learning realistic scenarios. Advances in Neural Information Processing Systems, 34, 2021.
491
483
[51] Agustinus Kristiadi, Matthias Hein, and Philipp Hennig. Being bayesian, even just bit, fixes overconfidence in relu networks. In International Conference on Machine Learning, pages 1436-5446, PMLR, 2020.
494
495
486
[52]
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2012.
487
498
[53] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017.
500
[54]
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7,7):3,700 2015.
502
[55]
Yann LeCun and Corinna Cortes. MNIST handwritten digit database. 2010.
503
[56] Jaehoon Lee, Yasaman Bahri, Roman Novak, Samuel S Schoenholz, Jeffrey Pennington, and Jascha Sohl-Dickstein. Deep neural networks as gaussian processes. International Conference on Learning Representations, 2017.
504
505
[57] Wenxin Li, Moran Feldman, Ehsan Kazemi, and Amin Karbasi. Submodular maximization in clean linear time, 2020.
507
[58] Dennis V Lindley. On measure of the information provided by an experiment. The Annals of Mathematical Statistics, pages 986-1005, 1956.
509
510
[59] Andrea Locatelli, Alexandra Carpentier, and Samory Kpotufe. An adaptive strategy for active learning with smooth decision boundary. In Algorithmic Learning Theory, pages 547-571. PMLR, 2018.
511
512
513
[60] David JC MacKay. Information- objective functions for active data selection. Neural computation, 4(4):590-604, 1992.
514
[61]
David JC MacKay. Choice of basis for laplace approximation. Machine learning, 33(1):77-86, 1998.
516
[62]
Andrey Malinin and Mark Gales. Predictive uncertainty estimation via prior networks. Advances in neural information processing systems, 31, 2018
518
663 Hermann G Matthies. Quantifying uncertainty: modern computational representation of probability and applications. In Extreme man-made and natural hazards in dynamics of structures, pages 105-135. Springer, 2007.
520
522
[64] McFadden. The entropy of a point process. Journal the Mathematics, 13(4))988-99) 1965.
the Society for Industrial & Applied
523
665] Dimitrios Milios, Raffaello Camoriano, Pietro Michiardi, Lorenzo Rosasco, and Maurizio Filippone. Dirichlet-based gaussian processes for large-scale calibrated classification. In Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors. Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
525
526
527
[66] Vitali D Milman. A new proof of a. dvoretzky's theorem cross-sections of convex bodies. Funkcional. Anal. Prilozen, 5:28-37, 1971.
$29
[67] Jishnu Mukhoti, Andreas Kirsch, Joost van Amersfoort. Philip HS Torr, and Yarin Gal. De- terministic neural networks with inductive biases capture epistemic and alcatoric uncertainty. arXiv preprint arXiv:2102. 1582, 2021.
531
533
[68] Radford M Neal. Priors for infinite networks. In Bayesian Learning for Neural Networks, pages 29-53. Springer, 1996.
13

---

535
[69]
George L Nemhauser and Laurence A Wolsey. Best algorithms for approximating the maximum of a submodular set function. Mathematies of operations research. 3(3):177-188, 1978.
536
537
[70]
George L Nemhauser, Laurence A Wolsey, and Marshall L Fisher, An analysis of approximations for maximizing submodular set functions-i. Mathematical programming. 14(1):265-294, 1978.
539
540
[71] Yuval Netzer, Tao Wang. Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
[72] Alon Orlitsky, Narayana P Santhanam, and Junan Zhang. Universal compression of memoryless sources over unknown alphabets. IEEE Transactions on Information Theory, 50(7):1469-1481 2004.
543
[73] F. Papangelou. On the entropy rate of stationary point processes and its discrete approximation. Probability Theory and Related Fields, 44(3):191-211, 1978.
546
547
[74] Jim Pitman et al. Combinatorial stochastic processes. Technical report, Technical Report 621, Dept. Statistics, UC Berkeley, 2002., 2002.
548
[75] Jim Pitman and Marc Yor. The two-parameter poisson-dirichlet distribution derived from stable subordinator. The Annals of Probability, pages 855-900, 1997.
550
551
[76] Nikita Puchkin and Nikita Zhivotovskiy. Exponential savings in agnostic active learning through abstention. In Conference on Learning Theory, pages 3806-3832. PMLR, 2021.
552
553
[77]
Rasmussen and C Williams. Gaussian processes for machine learning. adaptive computation and machine learning, 2006.
554
555
[78] Narayana P Santhanam, Anand D Sarwate, and Jae Oh Woo, Redundancy of exchangeable estimators. Entropy, 16(10):5339 53519,53772000 2014.
556
557
[79] Jacob M Schreiber, Jeffrey A Bilmes, and William Stafford Noble. apricot: Submodular selection for data summarization in python. J. Mach. Learn. Res., 21:161-1, 2020
558
[80] Paola Sebastiani and Henry P Wynn. Maximum entropy sampling and optimal bayesian experimental design. Journal of the Royal Statistical Society: Series (Statistical Methodology), 62(1):145-157, 2000.
560
[81] Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. In International Conference on Learning Representations, 2018.
563
564
[82] Claude E Shannon. A mathematical theory of communication. The Bell system technical journal, 27(3):379 423, 1948.
[83] Shubhanshu Shekhar, Mohammad Ghavamzadeh, and Tara Javidi. Active learning for classifi- cation with abstention. IEEE Journal on Selected Areas in Information Theory, 2(2):705-719 2021.
567
569
[84] Samarth Sinha, Sayna Ebrahimi, and Trevor Darrell. Variational adversarial active learning. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pages 5972-5981, 2019.
570
571
572
[85] Daniel A Spielman and Nikhil Srivastava. Graph sparsification by effective resistances. SIAM Journal on Computing, 1913) 2011.
573
574
[86] Daniel A Spielman and Shang- Hua Teng. Nearly linear time algorithms for preconditioning and solving symmetric, diagonally dominant linear systems. SIAM Journal on Matrix Analysis and Applications, 35(3):835- 885, 2014.
575
576
577
[87] Daniel A Spielman and Jae Oh Woo, A note on preconditioning by low-stretch spanning trees. arXiv preprint arXiv:0903.2816, 2009.
578
579
188]
Dimitris G Tzikas, Aristidis Likas, and Nikolaos P Galatsanos. The variational approximation for bayesian inference. IEEE Signal Processing Magazine, 25(6):131-146 2008
14

---

581
[89] Isabella Verdinelli and Joseph B Kadane. Bayesian designs for maximizing information and outcome. Journal of the American Statistical Association, 87(418):510-515 1992.
582
583
[90] Benjamin T Vincent and Tom Rainforth. The darc toolbox: automated, flexible, and efficient delayed and risky choice experiments using bayesian adaptive design. PsyArXiv. October, 20, 2017.
584
585
586
[91] Zheng Wang and Jieping Ye. Querying discriminative and representative samples for batch mode active learning, ACM Transactions on Knowledge Discovery from Data (TKDD), 9(3):1-33, 2015.
588
589
[92]
Kai Wei, Rishabh Iyer, and Jeff Bilmes. Submodularity in data subset selection and active learning. In International Conference on Machine Learning, pages 1954-1963. PMLR, 2015,
[93] Christopher KI Williams. Computing with infinite networks. Advances in neural information processing systems, pages 295-301, 1997.
592
593
[94] Yi Yang. Zhigang Ma, Feiping Nie, Xiaojun Chang, and Alexander G Hauptmann. Multi-class active learning by uncertainty sampling with diversity maximization. International Journal of Computer Vision, 113(2):113-12) 2015.
695
596
[95] Grigory Yaroslavtsev, Samson Zhou, and Dmitrii Avdiukhin. "bring your own greedy"+ max: near-optimal i/2-approxsnationn for submodular knapsack. In International Conference on Artificial Intelligence and Statistics, pages 3263-3274. PMLR, 2020.
598
589
196] Donggeun Yoo and In So Kweon. Learning loss for active learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 93-102, 2019.
600
[97] Yinglun Zhu and Robert Nowak. Efficient active learning with abstention. arXiv preprini 2rX1v:2204.00443, 2022.
602
Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information how answer these questions. For each question, change default [TODO] to [Yes]. [No] or [N/A] You are strongly encouraged to include justification your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
605
606
607
608
Did you include the license to the code and datasets? [Yes] See Section ??.
609
Did you include the license to the code and datasets? [No] The code and data are proprietary.
610
Did you include the license to the code and datasets? [N/A]
Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
612
613
614
1. For all authors...
615
(a) Do main claims made the abstract and introduction accurately reflect the paper's contribu- tions and scope? [Yes]
616
617
(b) Did you describe the limitations your work? [Yes]
618
(c) Did you discuss any potential negative socictal impacts your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
620
2. If you are including thcoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
622
(b) Did you include complete proofs of all theoretical results? [Yes]
623
If you ran experiments...
624
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as URL)? [Yes]
628
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Ye]]
15

---

628
(c) Did you report enror bars (e.g.. with respect to the random seed after running experiments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes]
632
4. II you are using existing assets (e.g., code, data, models) or curating/relessnng new assets...
(a) your work existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as URL? [Yes]
638
(d) Did you discuss whether and how consent was obtained from people whose data you' re us- ing/eurating? [N/A]
638
(e) Did discuss whether the data you using/curating contains personally identifiable informa- tion or offensive content? [N/A]
840
5. If you used crowdsoureing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable?
642
[N/A]
(h) Did you descrihe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and total amount spent on participant compensation? [N/A]
646
16


Reviews:
[{"review_id": "ngmn3CYxeoH", "reviewer": "Reviewer_Swnb", "summary": "By assuming a beta distribution on the softmax probabilities of a Bayesian classification network, the authors derive the marginalized joint entropy of a datapoint x and use it in a new acquisition function (balanced entropy learning acquisition) for active learning.", "questions": "Please see above", "limitations": "The authors didn't seem to discuss the limitations of the work although they claimed to have done that in the checklist (my apologies if I missed it somewhere).", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strength**\n- The paper is well organized. The authors explained their assumptions carefully and also the method.\n- Theorem 4.1 motivates the reciprocal of BalEnt well. Note that I haven't checked the proof in detail.\n\n**Weakness and Question**\n- I wonder how the assumption about the beta distribution of the softmax probabilities holds in practice. It would be beneficial to test this, for the dropout network, and for other types of Bayesian variational approximation methods.\n\n- The authors didn't seem to discuss the limitations of the work clearly. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "k50T6lBAj_X", "reviewer": "Reviewer_fhr7", "summary": "This paper proposes a new uncertainty measure, called BalEntAcq (Balanced Entropy Acquisition), that aims to correct the potential \"bias\" of existing active learning schemes based on information-theoretic uncertainty measures, such as the popular BALD (Bayesian active learning by disagreement).\nThe paper shows that the proposed BalEntAcq captures the \"information balance\" between the uncertainty of the underlying softmax probability and the label, thereby outperforming other active learning schemes based on similar entropy-based principles.\nBased on popular benchmarks, the authors demonstrate the potential advantages of active learning based on BalEntAcq compared to other alternatives.\n\n", "questions": "1. How does the performance of BalEntAcq compare to the popular ELR strategy, which directly focuses on reducing model uncertainty via active learning so as to improve the predictive performance?\n\n2. How does the proposed active learning scheme perform, in comparison with other existing schemes, in a small-data regime, where the role of active learning would be most important?\n\n3. Does BalEntAcq provide any convergence guarantee - either theoretically or empirically? What would be its main advantages over recently proposed Bayesian active learning schemes?\n", "limitations": "The authors do not explicitly discuss the limitations of the proposed work. The authors note there are no specific \"negative societal impacts\" relevant to the proposed method.\n", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "STRENGTHS\n\n1. The proposed uncertainty measure \"BalEntAcq\", which aims to effectively capture information balance between the uncertainty of the underlying softmax probability and the label variable, has been shown to result in well-diversified data point selection near the decision boundary, consistently outperforming other information-theoretic uncertainty measures often utilized for active learning.\n\n2. The closed-form expression of the proposed BalEntAcq leads to a computationally efficient active learning scheme, which nevertheless may improve the learning outcomes compared to other existing alternatives.\n\n\nWEAKNESSES\n\n1. The overall presentation - both literary and technical - requires improvement. For example, the authors do not clearly explain nor define what type of \"bias\" this study is focusing on, although this is crucial for understanding the main motivation of the proposed scheme as well as the major contributions in this work. Furthermore, many acronyms should be clearly defined before using them.\n\n2. Literature review on existing relevant schemes is too short and shallow. For example, although the authors present a brief summary of existing active learning schemes in Sec. 2.3, the treatment and review of other schemes is not comprehensive and fairly sketchy to be informative for the readers.\n\n3. While BalEntAcq has been shown to result in improved performance over some alternatives, a more comprehensive evaluation is needed to clearly demonstrate the advantage of the proposed scheme. For example, how does it compare to various active learning schemes based on ELR (expected loss reduction) strategy, which focuses on maximally reducing the expected error, and thereby focus on acquiring data points that can maximize the uncertainty directly impacting the predictive performance?\n\n4. Current results show that the proposed BalEntAcq may have potential benefits over other alternatives in terms of efficiency (through closed-form expression of the acquisition function) and performance gain (as reflected in Figure 3 and Table 1). However, the uncertainty measure BalEntAcq is defined in a somewhat heuristic manner, and it is unclear whether it would provide meaningful advantages over recently proposed Bayesian active learning schemes - in terms of performance, robustness, and convergence properties.\n\nFor example, the following papers propose Bayesian active learning schemes based on ELR strategies, which have shown to effectively reduce model uncertainty that directly affect the predictive performance of the learned model. \n\nZhao et al. \"Uncertainty-aware Active Learning for Optimal Bayesian Classifier\", ICLR 2021.\n\nTan et al. \"Diversity Enhanced Active Learning with Strictly Proper Scoring Rules\", NeurIPS 2021.\n\nBoth papers provide theoretical convergence guarantees to the optimal model as well as consistent performance improvements compared to other existing schemes, including BALD, the main method to which BalEntAcq is compared in this study. It would be helpful if the authors could compare their scheme to the above methods or other recent Bayesian active learning schemes and discuss what would be the main benefits of the proposed scheme.\n\n5. Based on Figure 3, it is unclear whether BalEntAcq would provide meaningful advantages over other alternatives in the small-data regime, which would be of special interest when studying active learning schemes.\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Xbh7KWQrIov", "reviewer": "Reviewer_xmoA", "summary": "This paper proposes a new acquisition function BalEntAcq for Al in BNN. It captures the information balance between the uncertainty of underlying softmax probability and the label variable. It facilitates parallelization over all classes (by estimating distribution parameters in marginal distribution). It is a purely standalone measure that could reduce relational computation like BatchBALD, CoreSet, and BADGE. Nevertheless, the paper shows that BalEntAcq captures a well-diversified selection near the decision boundary. The authors also demonstrate the new principle can outperforms other well-known Bayesian active learning algorithms. ", "questions": "1) Discuss computational complexity and compare it with other AL algorithms.\n2) Can the authors explain why we can not apply non-Baysian AL algorithms to BNNs?", "limitations": "See 3 weaknesses.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1) The new acquisition function is novel. It facilitates the parallelization both over classes and batch selection (there is no relational computation within data points inside a batch). It captures both informativeness (uncertainty) and diversity.\n\nWeaknesses:\n1) Baselines are limited. (a) The paper mentioned BatchBALD. The computational cost for selecting a batch is proportional to the batch size. It is still tolerable. As shown in the paper, it is superior to BALD in many settings. This paper can compare its method with BatchBALD with a small batch size, e.g., batch size <= 40. (2) Variation-Ratio [1] is also a potential baseline that have shown competitive performance. (3) Non-Bayesian AL algorithms can be applied to BNN. For example, in order to apply CoreSet to BNN, we can take the average last-layer representations with N MC dropouts or simply one representation with MC dropout, which is also the case for any DNN with dropout layers. Similarly, we can also apply BADGE to BNN. For all the examples, we can still maintain the expressiveness of the BNNs.\n2) The authors only tried their method on MC dropout. There are many other BNNs, e.g., [2] and [3], that need to be considered since the paper focus on AL in BNNs.\n3) Datasets are limited. The datasets considered in the paper should cover different aspects of AL, e.g., batch size, data format, and dataset size. The authors can compare their method with BatchBALD and [4] on MNIST and Repeated-MNIST with small batch size, i.e., <= 10. The author can also consider some small UCI datasets and some larger batch sizes for datasets like CIFAR10 and SVHN. If the new method does not work well in these settings, the authors should discuss it in the limitation or conclusion section.\n\nReferences:\n[1] Kirsch, Andreas, Joost Van Amersfoort, and Yarin Gal. \"Batchbald: Efficient and diverse batch acquisition for deep Bayesian active learning.\" Advances in neural information processing systems 32 (2019).\n[2] Lakshminarayanan, Balaji, Alexander Pritzel, and Charles Blundell. \"Simple and scalable predictive uncertainty estimation using deep ensembles.\" Advances in neural information processing systems 30 (2017).\n[3] Kingma, Durk P., Tim Salimans, and Max Welling. \"Variational dropout and the local reparameterization trick.\" Advances in neural information processing systems 28 (2015).\n[4] Kirsch, Andreas, Tom Rainforth, and Yarin Gal. \"Test Distribution-Aware Active Learning: A Principled Approach Against Distribution Shift and Outliers.\" arXiv preprint arXiv:2106.11719 (2021).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "WvWCwrCG9J2", "reviewer": "Reviewer_ugGu", "summary": "The context is Bayesian active learning, using MC dropout-based models. The paper places Beta distributions over the class marginals (ie one for each $p(y=i|x, w)$). The paper states that BALD as it is usually computed cannot be computed as such and instead proposes a different formulation, which it upper-bounds using what the paper calls a *marginalized joint entropy (MJEnt)* based on the entropy of the Beta distribution approximations. It then introduces a balanced entropy as the ratio between the MJEnt and the regular prediction entropy. Finally, it uses either this value or the reciprocal as the acquisition score (depending on the sign). The paper tries to motivate all this as principled and reports encouraging experimental results.\n", "questions": "1. How does BalEntAcq compare to BABA?\n\n2. Appendix A.12.2 contains ablations. Could you provide an ablation for P4 = BalEnt[x]^-1.\n\nIt would seem that BalEntAcq's second case (for negative scores) does not matter as long as there are sufficient positive scorers. This raises the question: it is needed at all even for the experiments provided in the main paper?", "limitations": "--", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper presents a few interesting perspectives on BALD. The experiments show good performance on CIFAR-100 and superior performance on CIFAR-100x3 and TinyImageNet.\n\nI was aware of the papers \"BABA: Beta Approximation for Bayesian Active Learning\" and \"Analytic Mutual Information in Bayesian Neural\nNetworks\" (Anonymous, 2021 &2022), but not the authors. Comparing the text of these papers, there might seem to be a case of self-plagiarism. While the former only exists as a preprint, the latter is to be published in ISIT 2022 (as also noted in this paper). I'm flagging this for the AC to decide and take a look at. The portions in question are not restricted to the Background section of this paper.\n\nIn regards to the content of the paper:\n\n1. It is not clear why the Beta approximation makes sense. Indeed, only the marginals $p(y=i | x)$ could be motivated to follow a Beta distribution, similar to the central limit theorem, as these are marginalized over $w$. The $p(y=i|x, w)$ as r.v. of $w$ however, need not follow such a distribution. This is a well-known criticism of e.g. prior networks (Malinin et al, 2018), which fit Dirichlet distributions.\n\n2. The paper states in (4) and (5) that the common formulation of BALD is equal to the more complex formulation using $\\Phi$. It then finds an upper bound to this formulation (MJEnt). This is similar to how the predictive entropy (Ent) upper-bounds normal BALD. In (10), the paper states that MJEnt = Ent + some additional term (\"Posterior Uncertainty\") = Epistemic Uncertainty + Aleatoric Uncertainty + Posterior Uncertainty, and that it thus decomposes three different uncertainties. This is wrong: it actually confounds these three terms (as it takes a sum/is a function of them)! \n\n    The citation right before (4) and (5) is actually wrong. Gal et al, 2017, do not motivate the statement that follows.\n\n3. Section 3.3 is not novel and only restates what is known from Houlsby et al, 2011, Gal et al, 2017, and Kirsch et al, 2019: entropy acquisition is over-confident compared to BALD because it adds in aleatoric (irreducible) uncertainty (Ent =  Epistemic Uncertainty (BALD) + Aleatoric Uncertainty), which will not be learnt away.\n\n4. BalEnt falls out of nowhere as a ratio and is not actually motivated.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Reject