Active Learning in Bayesian Neural Networks with Balanced Entropy Learning Principle
Authors: Jae Oh Woo
Venue: ICLR 2023 notable top 25% (2023)
Keywords: bayesian neural network
bayesian active learning
balanced entropy learning
uncertainty quantification

Abstract: Acquiring labeled data is challenging in many machine learning applications with limited budgets. Active learning gives a procedure to select the most informative data points and improve data efficiency by reducing the cost of labeling. The info-max learning principle maximizing mutual information such as BALD has been successful and widely adapted in various active learning applications. However, this pool-based specific objective inherently introduces a redundant selection and further requires a high computational cost for batch selection. In this paper, we design and propose a new uncertainty measure, Balanced Entropy Acquisition (BalEntAcq), which captures the information balance between the uncertainty of underlying softmax probability and the label variable. To do this, we approximate each marginal distribution by Beta distribution. Beta approximation enables us to formulate BalEntAcq as a ratio between an augmented entropy and the marginalized joint entropy. The closed-form expression of BalEntAcq facilitates parallelization by estimating two parameters in each marginal Beta distribution. BalEntAcq is a purely standalone measure without requiring any relational computations with other data points. Nevertheless, BalEntAcq captures a well-diversified selection near the decision boundary with a margin, unlike other existing uncertainty measures such as BALD, Entropy, or Mean Standard Deviation (MeanSD). Finally, we demonstrate that our balanced entropy learning principle with BalEntAcq consistently outperforms well-known linearly scalable active learning methods, including a recently proposed PowerBALD, a simple but diversified version of BALD, by showing experimental results obtained from MNIST, CIFAR-100, SVHN, and TinyImageNet datasets.

Published as a conference paper at ICLR 2023
ACTIVE LEARNING IN BAYESIAN NEURAL NETWORKS WITH BALANCED ENTROPY LEARNING PRINCIPLE
Jae Oh Woo
Samsung SDS Research America San Jose, CA 95134 j jaeoh. woolaya. ya vale.c.. edu
ABSTRACT
Acquiring labeled data is challenging in many machine learning applications with limited budgets. Active learning gives a procedure to select the most informative data points and improve data efficiency by reducing the cost of labeling. The info- max learning principle maximizing mutual information such as BALD has been successful and widely adapted in various active learning applications. However, this pool-based specific objective inherently introduces a redundant selection and further requires a high computational cost for batch selection. In this paper, we design and propose a new uncertainty measure, Balanced Entropy Acquisition (BalEntAcq), which captures the information balance between the uncertainty of underlying softmax probability and the label variable. To do this, we approximate each marginal distribution by Beta distribution. Beta approximation enables us formulate BalEntAAq as ratio between an augmented entropy and the marginalized joint entropy. The closed-form expression of BalEntAcq facilitates parallelization by estimating two parameters in each marginal Beta distribution. BalEntAAq is a purely standalone measure without requiring any relational computations with selection
other data points. Nevertheless, BalEntAcq captures near the decision boundary with a margin, unlike other existing uncertainty mea- sures such BALD, Entropy, Mean Standard Deviation (MeanSD). Finally, we demonstrate that our balanced entropy learning principle with BalEntAAc consis- tently outperforms well-known linearly scalable active learning methods, including a recently proposed PowerBALD, a simple but diversified version of BALD, by showing experimental results obtained from MNIST, CIFAR-100, SVHN, and TinylmageNet datasets.
INTRODUCTION
Acquiring labeled data is challenging in many machine learning applications with limited budgets. As the dataset size gets bigger and bigger for training a complex model, labeling data by humans becomes expensive. Active learning gives procedure to select the most informative data points and improve data efficiency by reducing the cost of labeling.
The active learning problem is well-aligned with subset selection problem that can find the most
efficient but minimal subset from the data pool (Hochbaum) 1996 Nemhauser et al., 1978) Dvoretzky
1961 1961, MMilman [971 (Siillaan & Teng 20144 Spielman & Woo 2009 Batson et al.) [2009] (Spiclman & Srivastava, 2011). The difference is that active learning is typically an iterative process where
model is trained and a collection of data points is selected to labeled from an unlabelled data pool.
It is well-known that any active learning method cannot improve the label complexity better than passive learning (random acquisition) in general (Vapnik & Chervonenkis) 1974] K Käärläinen (006)] Castro & Nowak 2008). Under some conditions on labels or models, is possible to achieve
exponential savings (Balcan et al. 2007 Hanneke 2007 Dasgupta et al. 2005 Hsu 2010: Dekel
et al. 2012 |anneke| Zhang & Chaudhuri 2 Krishnamurthy al. 2017 Shekhar et al.
2021 Puchkin & Zhivotovskiy, 2021). Zhu & Nowak (002bb a) recently proposed a provably
Code available. https://gittuu..cmm/////////

---

Published as a conference paper at ICLR 2023
exponentially efficient active learning algorithm with abstention with high probability but limited in binary classification cases. On the other hand, although numerous practically successful active learning methods have been proposed, no algorithm has proven efficient enough and linearly scalable to guarantee exponential label savings in general. Therefore, it is still theoretically challenging but important to improve data efficiency significantly.
is now commonly accepted that standard deep learning models do not capture model uncertainty correctly. The simple predictive probabilities are usually erroneously described model confidence (Hein et al. 2019). So there is a risk that a model can be misdirecting its outputs with high confidence. However. the predictive distribution generated from Bayesian deep learning models better captures the uncertainty from the data (Gal & Ghahramami [016 Kristiadi et al. 20200] Mukkoti et al. 202T]
Daxberger et al., 2021 J. Therefore, we focus on developing an active learning framework in the Bayesian deep neural network model by leveraging the Monte-Carlo (MC) dropout method as proxy
of the Gaussian process (Gal Ghahramani 2016) which may facilitate further analysis.
1.1 OUR CONTRIBUTIONS
Our proposed active learning method is well-aligned with Bayesian experimental design (Verdinelli & Kadane 1992, Cohn et al. 1996, (Sebastiani & Wynn 2000, Malinin & Gales 2018 Foster et al.
2019) with an assumption that the forward active learning iterative process follows the Bayesian prior-posterior framework. Furthermore, our approach is also aligned with Bayesian uncertainty
quantification methods (Houlsby et al. [201] Kandasamy et al. 2015 Kampffmeyer et al. 2016 Gal & Ghahramani [0166] Alex Kendall & Cipolla, 2017|
|017] Gal et al. 2017 [ Kirsch et al. 2019) Mukhoti
iet al.) 20211; Kirsch et al.] 2021) with an assumption that the working neural network model is a
Bayesian network (Koller & Friedman 2009).
In this paper, we extend and improve recent advances in both aspects of Bayesian experimental design and Bayesian uncertainty quantification. We investigate the generalized notion of the joint entropy between model parameters and the predictive outputs by leveraging a point process entropy
(McFadden 1965 Fritz 1973 Papangelou 1978 Daley & Vere-Jones) 2007 Baccelli & Woo 2016). By approximating the marginals using Beta distributions, we then derive an explicit formula of the marginalized joint entropy by estimating Beta parameters from Bayesian deep learning models. As a Bayesian experiment, we revisit the well-known entropy and mutual information measures given expected cross-entropy loss. We show that well-known acquisition measures are functions of marginal distributions through analytical formulas. We propose new uncertainty measure, Balanced Entropy Acquisition (BalEntAcq), which captures the information balance between the uncertainty of underlying softmax probability and the label variable. Finally, we demonstrate that our balanced entropy learning principle with BalEntAcq consistently outperforms well-known linearly scalable active learning methods, including recently proposed PowerBALD (Kirsch et al., 2021) for mitigating the redundant selection in BALD (Gal et al. 2017), by showing experimental results obtained from MNIST, CIFAR- 100, SVHN, and Tiny ImageNet datasets.
BACKGROUND
2.1 PROBLEM FORMULATION
We write an unlabeled dataset Dpool and the labeled training set Diraining S Dpont in each active (n) learning iteration. We denote by trainng it's necessary to indicate the specific n-th iteration step. Given Prraining we train a Bayesian deep neural network model Tb with model parameters w ~ p (w).
Then for data point x given D Diraining the Bayesian deep neural network & produces the prediction probability: + w) := (Pi (Pi (x,,),.. w).... Pe w)) A where A {(p,,... -Pc) P1 + PC = 1, 2 0 for each i} and C/ is the number of classes. For the final class output Y, it is assumed to be multinoulli distribution (or categorical distribution):
with probability Pi (x,
Y(x =
(1)
c
with probability Pc(x, w).
2

---

Published as a conference paper at ICLR 2023
For the sake of brevity, we sometimes omit by writing & (w), Pi(w). (w) D, P.. Y unless we need further clarifications on each data point x. Under this formulation, the oracle (active learning algorithm) selects subset of data points to add to the next training set, i.c. at (n + 1) 1)-th iteration, the training set is determined by ((++1) training = D D(1) U [Next training batch from Oracle). Once the training
next training batch is selected, the selected batch will be labeled. This means that the ground truth label information of the selected data is added in training set training in the next round. Then the goal in active learning is to minimize the number of selected data points to reach certain level of prediction accuracy.
2.2 EXAMPLES OF UNCERTAINTY BASED ACTIVE LEARNING METHODS
In this section, we list up well-known uncertainty measures suitable for Bayesian active learning.
1. Random: Rand[x] == Uww)) where U.-) is uniform distribution which is independent to w.
Random acquisition function assigns random uniform value on [0, 1] to each data point.
2. BALD (Bayesian active learning by disagreement) (Lindley 1956 Houlsby et al. 2011 Gal et al. 2017): BALD x := 3 (w, Y (x,w), w)). where ((.-)) represents mutual information between random measures. BALD captures the mutual information between the model parameters and the predictive output of the data point. In practice, we calculate the mutual information between the predictive output and the predictive probabilities.
3. Entropy (Shannon 1948): Ent[x] == -EE, (EP.) log (EP.). Entropy is the Shannon entropy with respect to the expected predictive probability. Entropy can be the uncertainty of the prediction probability. Moreover, under the cross-entropy loss, may also interpret the entropy measure as an expected loss gain since - log (EP;) is the cross-entropy loss given the ground truth label is the class i.
4. Mean standard deviation (MeanSD) (Cohn et al. & Cipolla 2017): MeanSD[xi ? Σ: V E.P (E...²
1996, Kampfimeyer et al. 2016 Alex Kendall
Mean standard deviation captures the average of standard deviations for each marginal distribution.
5. PowerBALD (Farquhar et al. 2021 Kirsch al. 2021 PowerBALD |x] == log BALD|x] Z, where Z is an independently generated random value from Gumbel distribution with the location default choice
/ 0 and the scale / parameters, see WWkipediaa (2023). We use /
suggested by Kirsch et al. (202I). The motivation of this randomized acquisition is to mitigate the redundant selection by diversifying selected multi-batch points. In general, do not know which parameter 3 will be the optimal choice.
In multiple acquisition scenario, we simply add the above uncertainty values for each data point x,:
AcqFunc[xI.
AcqFunc|x.]]
1=1
where AcqFunc E (Rand, BALD, Ent, SeennSD,PoweBBALD.
2.3
SUMMARY OF OTHER ACTIVE LEARNING APPROACHES
Cohn al. (1996) provided one of the first statistical analyses active learning, establishing how to synthesize queries that reduce the model's ''s -wordd-ookking error by minimizing its variance
leveraging MacKay's closed-form variance approximation (MacKay 1992). In this fashion, there
exists line of works in Bayesian experimental design (Chaloner & Verdinelli 1995] Lindley 1956
Verdinelli & Kadane 1992; Cohn et al. 1996 [Seaastiani Wynn 2000; Roy McCallum 200T
Yoon al., 2013 Vincent & Rainforth 2017 77 Foster al. 2019 2021: Jha et al. 2022) with
an assumption that the forward active learning iterative process follows Bayesian prior-posterior framework.
On the other hand, in active learning, accommodating both the information uncertainty and the diversification of the acquired samples is essential to improve the performance under multi-batch acquisition scenarios. In theoretical perspective, the most natural way to combine the uncertainty and the diversification seems to leverage reasonable sub-modular functions, e.g. Nearest neighbor set
3

---

Published as a conference paper at ICLR 2023
2019). Determinantal Point Process (Bryık) (et al. 2019) and SIMILAR (Kothawade et al. 2021) 1) with sub- modular information measures, and then/or apply fast linear-time algorithm to find a diversified multi- batch with provable performance
function Wei al. 2013). BatchBALD Kirsch et al.
guarantee (Nemhauser & Wolsey 1978 Nemhauser et al. 1978 Ene & Nguyen 2017; Yaroslavtsey
et al., 2020] Schreiber al.) 2020 yyer et al. 202 Li et al. (022). Although fast linear- time solver is available for general sub-modular functions, there still exists gap with practical implementation, such as high memory requirements, which makes the computation unscalable for identifying multi-batch acquisition points, e.g., BatchBALD (Kirsch et al. 2019). Similar to the sub- modular function optimization, there exist many customized optimization approaches. e.g. CoreSet (Sener& & Savarese 2018) and more approaches (Guo) 2010; Joshi et al. 2010: Elhamifar et al. 2013
Yang et al. 2015 Wang & Ye 2013).
Another recent approach is to look at parameters of the neural network and to diversify points such as BADGE (Ash et al. 2020) with gradients and BAIT (Ash al. (00T)) with Fisher information. There also exist network architectural design focused approaches such Learning loss by designing loss
prediction layers (Yoo & Kweon 2019), UncertainGCN and CoreGCN (Caramalau et al. (02T) with
graph neural networks, VAAL (Sinha et al., 2019) and TA-VAAL (Kim et al. 202 by applying adversarial learning methods.
BAYESIAN NEURAL NETWORK MODEL
We adopt the Bayesian neural network framework introduced in Gal & Ghahramani (2016). The core idea in the Bayesian neural network is leveraging the MC dropout feature to generate a distribution of the predictive probability as an output at inference time. Under mild assumptions, it turns out that
is equivalent to an approximation to Gaussian Process (Rasmussen & WWilliass 2006 Neal 1996
Williams 1997, Gal & Ghahramani 2016 Lee et al. 2017).
3.1 SOFTMAX PROBABILITY MARGINAL APPROXIMATHLY FOLLOWS BETA DISTRIBUTION
We may consider a Bayesian neural network model o as a random measure, i.e., stochastic process parametrized by Diraining over the data set D Given a data point x E Puoot & (x, w) produces a random probability distribution in a simplex A This analogy has a close connection with the construction of random discrete distribution, originally introduced by Kingman 1975). Since then, random measure construction has been extensively developed in Bayesian nonparametries, and well-known that Dirichlet probability having Beta marginals plays the central role in the construction of the random discrete distribution (Kingman 1977, Ferguson 1973 Pitman& & Yor 1997 Pitman et al.) 2002; Broderick al. 2012) Orlitsky et al. 2004| Santhanam al.] 2014). It is the main motivation of the Beta distribution approximation. Many kinds of literature similarly assume the Dirichlet distribution after the softmax in the Bayesian neural network.
As illustrated by Milios et al. (2018). we may follow the construction of Dirichlet distribution. Following the approach by Ferguson 1973), a Dirichlet probability can be constructed through collection of independent Gamma distributions. On the other hand, each marginal in Gaussian Process (approximated by Bayesian neural network) in the softmax output having dependent components follows log-normal distribution (before the normalization, but after the exponentiation in softmax). Then by applying the shape similarity between a log-normal distribution and Gamma distribution, the construction of random probability from log- normall distributions would produce an approximated Dirichlet distribution. Therefore we may assume that the marginal distribution would approximately follow the Beta distribution.
Alternatively, as an analytical approach, we may see that Beta approximation can be justified through Laplace approximation (MacKay) 1998, Hennig et al., 2012. Hobbhahn et al., 2020 Daxberger ct al. 2021). There exists a mapping between multivariate Gaussian distribution and Dirichlet distribution under a softmax basis. Then Beta distribution follows as a marginal distribution of Dirichlet distribution. Therefore we may assume that Beta approximation exists through Laplace approximation under the assumption that the Bayesian neural network produces the multivariate Gaussian distribution (as a marginalized Gaussian process over finite rank covariate function) before
the softmax layer (Neal) 1996. Williams 1997 Gal & Ghahramani 2016
Lee et al.) 2017)
4

---

Published as a conference paper at ICLR 2023
In practice, once we estimate the sample mean and sample variance for each marginal of o (x,w), we can estimate two parameters the Beta distribution as follows. Assume that P, ~ Beta (a 5.).
EP: m. and VarP: o. then ag m (1-m)
B = -1-1 ac. When P. au+B; = m and VarP: = (or+ (lartbbt)) = o.. Solving the equation with
M&s
of
Beta (a,, B,), EP;- = c; respect to a, and 3,, then the equation follows.
0.
3.2 MARGINALIZED JOINT ENTROPY IN BAYESIAN NEURAL NETWORK
Assume that each P: ~ (eta 3x) by applying Beta approximation. We may define a quantity of the marginalized joint entropy (See Appendix A. and and we find an equivalent formulation as follows:
MJEnt(xx = Σ Ep, [P log (P:((R))) =
Σ (EP:)
Y
EEw [H (VWW)
epistemic uncertainty
aleatoric uncertainty
posterior uncertainty
where ((-) is differential entropy, 3(.-)) represents mutual information between two quantities, (.) is Shannon entropy, P.+ the conjugate Beta posterior entropy of P, which follows P.+ ~ Beta + 1, B.). We call the first term in equation 3 to be the posterior uncertainty. We may interpret the posterior uncertainty as an expected posterior entropy assuming that we observed a positive sample of the class toward P, for each i without knowing the true class label. The first term is always non-positive, and is maximized (equals to 0) when each P.+ is Beta( 1, 1), i... Uniform on [o, 1). So -00 MJEnt[x] x < HY), where we note that H (Y) = (w, Y) + E. [H (YW)).
The epistemic uncertainty captures the model uncertainty (as BALD). and the aleatoric uncertainty captures the data uncertainty (Matthies 2007). Therefore the marginalized joint entropy, MJEnt[xx is a decomposition of three types of uncertainty values.
3.3
ENTROPY FOR MAXIMIZING AN EXPECTED CROSS-ENTRON LOSS
Given a ground-truth label (Y = i). the cross-entropy loss of the neural network can be given as loss L (x, W) Y = i) log E... Therefore, inspired by the expected loss in risk management [Jorion 2000), we can calculate the expected cross-entropy loss without knowing the truth label:
Expectedl.oss(/]
Σ P = loss ① ( -)) Σ (EP.) log (EP.)) Ent(x). 1=1
Based on the re-formulation, we may interpret that entropy acquisition is for maximizing an expected cross-entropy loss in a selection of acquisition points, aligning the idea with the learning loss (Yoo) & Kweon] 2019). The natural question is, "Once we acquire a data point that maximizes entropy acquisition, can we remove/or learn this expected cross-entropy amount of loss at the future stage of the active learning?". The answer could be "No." "oo" 'The exhaustive loss acquisition could only happen the neural network perfectly over-fits the training data. Therefore, there exists a gap between a realistic neural network training scenario and the objective of the entropy acquisition. Our equivalent loss interpretation without introducing epistemic or aleatoric uncertainty confirms typical perceptions of why the entropy acquisition might not be successful in practice, even in the single-point acquisition scenario.
3.4 BALD IS A FUNCTION OF MARGINALS AND IS STRONGLY ALIGNED WITH MAXIMIZING AN EXPECTED PROSS-ENTROPY LOSS DIFFERENCE UPTO THE NEXT ITERATION
We have the mutual information between w and Y and it is the same as the mutual information between the encoded message and the channel output since Y depends only on &b (x, w) Gal et al. 2017). BALD[x] := J (w, Y) J ( (x,w). Y((x w)). By assuming that Φ (x, w) follows a Dirichlet distribution, we can calculate the mutual information analytically (Woo 2022). Then by investigating further into the analytical mutual information formula, we see that the marginal distributions P, 's in (xx w) are sufficient to estimate BALD. This marginal representation is an important phenomenon dependency of all coordinates could be removed at the predictive stage (Bobkov & Madiman
5

---

Published as a conference paper at ICLR 2023
2011 See Conjecture V.5). Therefore represent BALD through Beta marginal distributions as follows. See Appendix for more details.
Theorem 3.1. Under Beta marginal distribution approximation, let P. Betalas, 9.) in &b (x,
Then the mutual information BALD(xx can be estimated as follows:
C
c (0: (as)
BetaMarginalBALDD
(a. 1) (a (a, +
Σ
04
log
a,
04 +
Bi
as + 3:
1=1
3, (a. e
of
v (a: + 1++11+
V (1 (a: 4 131
3
+
As Bayesian experimental design process, may assume that cach Beta marginal distribution P. with the ground-truth label {Y = i} of the next trained model would follow the Beta posterior distribu- tion P... Without this assumption, existing choices of acquisition functions such as BALD or MeanSD might not be well-justified. For example, what is the implication of maximizing mutual information through the active learning process with a Bayesian neural network? How is it different from the max- imization of the entropy acquisition? To answer these questions, leveraging our Beta marginalization and considering the similar idea expected information gain (Foster et al., 2019), we may consider the expected cross-entropy loss difference between the current stage model and the next stage model.
ExpecteEffiectieelosss ix EP |- log EP. (err----uuu (Pg)
log
ai a Cu +
log
0: az P.
ExpectedFiifectiveel.os captures the effective amount of cross-entropy loss for the model to learn after the acquisition. By definition, we see that ExpectedEffectiveLoss aims to exclude the undesirable over-fitting scenario assumption unlike Entropy acquisition.
Since Digamma function (xx ~ log x 2r where (w)) ~ g(x) implies lim f<x)//(>) 1, we may expect that BetaMarginal and ExpectedEffectiveeosss
would behave similarly. Figure [ shows the Spearman's is rank correlations among different acquisition measures upto class dimension
10, 000. We observe that BetaMarginalBALDD behaves equally like the original BALD and we confirm that BALD and MeanSD are strongly aligned with maximizing ExpectedEffecciveeooss Therefore, acquiring points through BALD or MeanSD could be better strategy than Entropy because BALD or MeanSD takes into account the effective loss acquisition instead of the unrealistic full amount of the loss acquisition.
14-
341
Figure 1: Scatter plot at C' = 10, 000 between BALD and ExpectedEfffectieeloss (left), Spearman's 's rank correlations over various class dimensions (middle), and Spearman's rank correlation matrix at C = 10, 000 (right). The relationship between BetaMarginalBALJ. and ExpectedFifiectivell Loss consistently captures high rank-correlation with 99.6% regardless of the class dimensions. BALD and ExpectedEfficciveeLos show > 97.5% rank-correlation. We randomly generate 100 softmax applied C-dimensional Gaussian samples and repeated the process 10 times. Shaded band shows the standard deviation.
4 BALANCED ENTROPY LEARNING PRINCIPLE
The previous section shows that well-known acquisition measures have an objective toward cross- entropy loss and are closely related to marginal distributions. However, according to Farquhar et al.

---

Published as a conference paper at ICLR 2023
(2021), J, to be successful in active learning, they hypothesize that is crucial to find good balance between active learning bias and over-fitting bias under over-panee ver-paramettized neural networks. Although they claim that might not be possible achieve the ultimate active learning goal without having the full information, we may still define the balanced entropy (BalEnt) to be a ratio between the marginalized joint entropy equation and the augmented entropy:
MJEnt[x]
E. (EP.) ((PT))(PPPP))))))) + H(Y)
BalEnt[x] == Ent[x] + log
H(Y) log
Recall that we call the first term in MJEnt[x] to be posterior uncertainty, and is an expected posterior
entropy of underlying marginals. BalEnt captures the information balance between the posterior uncertainty from the model and entropy of the label variable Y.
4.1 IMPLICATIONS OF BALANCED ENTROPY
To understand the implication of Ballin( x). we can prove the following Theorem 4.1
Theorem 4.1. Let A-1 = (2e and T = {In}, a collection of evenly divided intervals in
[0, 1] where In = [[n - 1)4,n for n 1,... .A 1) and Ia === -. 11. Let P. be discretized random variable over T of Ps fiffrom & (x, w). For any estimator P. of P. given the label {Y- i] we have
]]
E. (B)))(())))H())))))))))))))))))+ (1+61) - E2 BalEEn X + E1) - 62,
E
P
H(Y) + log
where 51, E2 2 are adjustment terms depending on A such that E - 0 and E2 - 0 as A
Theorem 4.1 tries to answer the following inverse problem. For the unlabeled data point, if we know the information of the label {Y = i}. how much can we reliably estimate the underlying probability P; from the model Φ As we know that log P; is the cross-entropy loss of the trained model with Y. it equivalently answers the estimation error probability of the loss prediction under a unit precision up - log A level. For the precision level, we are assuming to carry - log R H (Y) ) + log nats natural unit of information, re-scaled amount of bits, matching the enumerator with MJEnt[[x term. It not clear how to determine better choice of the precision level log A. But we may understand the denominator H (YY) ) + log 2 is for normalizing the term BalEnt[x] < as probability. Then the sign of BallEn[[x] becomes very important. BalEnt [x] 2 implies that it could be impossible to perfectly predict the loss log P: given currently available information. i.e., there could exist information imbalance between the model and the label approximately starting from BalEnt[x] = O. Therefore, insight from Theorem 4.1 suggests us a new direction for main active learning principle. We define our primary acquisition function, namely, balanced entropy learning acquisition (BalEntAcq). as follows:
if BalEnt [x] 2 0, [BlEnn[[]] if BalEnt xx] < 0.
BalEnt[xx] -1
BalEntAcq[x] =
Since the information imbalance exists at least from BalEnt[x] 0, we prioritize to fill the infor- mation gap from BalEnt[x] = O toward positively increasing direction which aligns with choosing the entropy increasing contours. If try to fill the information imbalance gap from the highest [x]Ent[x], the information imbalance would still exist around BalEnt[x] = O area. Therefore, might not improve the active learning performance much. See Appendix A..1.2 and A..3. for different prioritization and precision level results. That's the motivation why we take the reciprocal of BalEnt[x] when BalEnt[x] 2 O.
4.2 TOY EXAMPLE ILLUSTRATION
To illustrate the behavior of BalEntAcq and its relationship with other uncertainty measures, train a simple Bayesian neural network with 3-class moon dataset in IR?. Then we calculate each acquisition measure for all fixed lattice points in the square domain by assuming that the unlabeled pool is highly regularized (or uniform). i... by evenly discretizing the domain, we obtain each uncertainty value for each lattice point. The total number of lattice points is around 0.6 million.

---

Published as a conference paper at ICLR 2023
(a) BalEntAcq (ours)
(b) BALD
(c) Entropy
(d) MeanSD
(e) PowerBALD
Figure 2: Top- K selected points are marked by red color. The first row shows the top K 25 points. The second row shows the top K = 500 point selections among around 0.6 million grid points.
Then we choose top-K high uncertainty values for each method to observe the prioritized region for each method. We use K = 25 and K = 500. Figure 2 iillustatee the top- points selected by each method. The most significant phenomenon that BalEntAcq's 's selection is highly diversified near the decision boundary showing bifurcated margin because are prioritizing the surface area of [BalEn[[x] x 2 0j. This is well-aligned with the strategy avoiding high aleatoric points. (See Appendix A. Then can imagine conduct a uniform sampling on each contour surface [BalEEn[x] = }} for each À 2 0, as we move to the surface for each À < 0. That's why we observe bifurcated but diversified and balanced selection near the decision boundary with BalEngAcq in Figure 2 (a) when
K = 25. On the other hand, there is preferred area for each method from other measures except PowerBALDD PowerBALD shows a good diversification, but could select non-informative points.
EXPERIMENTAL RESULTS
In this section, we demonstrate the performance of BalEntAcq from MNIST (LeCun & Cortes, 2010 CIFAR-100 (Krizhevsky et al., 2012). SVHN (Netzer et al.) (011), and TinylmageNet (Le & Yang 2013) datasets under various scenarios. We used single NVIDIA A100 GPU for each experiment, and details about the experiments are explained in Appendix A. 13 We test Random, BALD, Entropy, MeanSD, PowerBALD. and BalEntAcq measures. We add BADGE for additional baseline. Note that all acquisition measures except BADGE in our experiments are standalone quantities, so they can be easily parallelized, i.c., linearly scalable.
Single acquisition active learning with MNIST. MNIST is the most popular and elementary dataset validate the performance of image- deep learning models initially. We simple convolutional neural network (CNN) model applying dropouts to all layers with a single acquisition size. The primary purpose of this single acquisition experiment is validate our proposed balanced entropy approach by removing the contribution of diversification unlike multi-batch acquisition scenario.
Fixed features with CIFAR-100 and 3x CIFAR-100. In recent years, significant efforts have been made on building an efficient framework of unsupervised or self-supervised feature learning such as SimCLR (Chen et al. 2020a b). MoCo (He et al. 2020), BYOL (Grill et al.) 2020). SWAV (Caron al. 2020p, DINO (Caron al. 202T). , etc. As an application in active learning, we may leverage the feature space from the unsupervised feature learning without explicitly knowing true labels but construct a good representation space. In our experiments, we adopt SimCLR for simplicity with ResNet-50 to build feature space for CIFAR-100.
With 3 CIFAR-100 dataset, we observe the effect of the redundant information treatment for each method by adding three identical points. We use the same fixed feature obtained from SimCLR with CIFAR- 100. We may observe how each method effectively diversifies the selection under a redundant data data scenario by fixing the feature space.
Pre-trained backbone with SVHN and strong data augmentation with Tiny ImageNet. In this experiment, we follow a typical image classification scenario in practice. We use the ResNet- 18 backbone for SVHN and the ResNet-50 backbone for TinyImaaeNet with ImageNet pre-trained model for model architecture, and the last linear classification layer is replaced with simple Bayesian
8

---

Published as a conference paper at ICLR 2023
neural network with dropouts. In TinylmageNet iterations, we re-use the previously trained model for the next training. So the pre-trained ImageNet weight is only used at the initial iteration. We apply strong data augmentations for TinyImageNet, including random crop, random flip, random color jitter, and random grayscale. Under this scenario, the feature space from the backhone is continuously evolving and keeps confused the training and active learning process proceeds. Because of the strong data augmentation and batch normalization in ResNet-18 or ResNet-50, the decision boundary keeps confused, implying that the Bayesian experimental design assumption might not hold. However, we still want to observe the general behavior of each measure and how to improve the accuracy under a more dynamic feature space.
360 acu
uph la
hond Li0 upe
Nand naa 10
5000 the
(a) MNIST
(b) CIFAR- 100
(c) 3XCIFAR- 100
(d) SVHN
(c) TinylmageNet
Figure 3: Active learning accuracy curves obtained from various scenarios. Our proposed BalEntAcq outperforms well-known acquisition measures, and we repeated the experiment 3 times.
Table 1: Selected accuracy table. Mean and standard deviation are from repeated experiments.
Secmrt
Pal CaPpAA- CNN
Pasl Icuane
Rehante unig a Pised
Hitlwer
BaUSME 4 ARgnCHPint
Datsun/vo Sairhen-be Abe
NNSTTHO2L.0 108
CIEAR H.C-5
INCCIAA SaC-Sa411
EVIING 59326J03
TeglnupdNel ABCHIVRO
Tran Soctrel Viee
19290/06
S00V50 90)
(5 94/99)
19/00/10/00
9999911 353
357
15/00090000
Jankee
796-2 4,9%
2..7%
15.-224477 B:4%
01.02225 9.20
04-3 2.433
91.5 3.0%
10.22218.22
$2.088 1,366
06.3 0L.5
08.3 5.4.0
552 8,0%
(AU8 4-133%
9.20
04.5 5.4.63
34. 9.13%
11.250 8.TH
41.
305
Meadl))
100 1.78
10 355
8- 115
(419 8-1 E%
21
88 820
06.3 8-1 93g
913 665
9--3 m-1
31 8-40
0133 1-38-100
BADGE ladn scaitti
77.0 6.15
44
0.
61.82 0LUS
66.0 0.0.%
67.4 4.444.00 0.15
02.9 0.45
05.8 26002000 0.30
17.5 8 1/0
43.1 0.03
Discussion. BalEntAcq consistently outperforms other linearly scalable baselines in all datasets, as shown Table BADGE performs similarly to Entropy under a single acquisition scenario in MNIST because BADGE focuses on maximizing the loss gradient similar to Entropy, as we explained in Section 3.3 BADGE shows better performances at first when we fix the feature space, but our BalEntAcq eventually catches up with the performance of BADGE. We also note that BADGE is not a linearly scalable method. Under dynamic feature scenarios in SVHN or TinyImageNet, we observe that our BalEntAcq performs better. Considering the acquisition caleulation time (see Appendix 1.18) our BalEntAcq should be a better choice. Figure 3 shows the full active learning curves. For CIFAR- 100 and CIFIAR 100 cases, by fixing features, we control/remove all other effects possibly affecting the model's performance, such as data augmentation or the role of backbone in the classification. As demonstrated in Figure 2 BalEntAcq is very efficient in selecting diversified points along the decision boundary. Instead, PowerBALD suffers from improving accuracy because it focuses more on diversiiiction/zzaddmizziio by missing the information near the decision boundary. For SVHN or TinylmageNet, BalEntAcc shows better performance again. We suppose that diversification near the decision boundary BalEntAcq also plays the data exploration because the representation space keeps evolving with the backbone training.
CONCLUSION
In this paper, we designed and proposed a new uncertainty measure, Balanced Entropy Acquisition (BalEntAcq), which captures the information balance between the underlying probability and the label variable through Beta approximation with a Bayesian neural network. BalEntAcq offers a diversified selection and is unique compared to other uncertainty measures. We expect that our proposed balanced entropy measure does not have to be confined to active learning problems in general. BalEntAcq would improve the diversified selection process in many other Bayesian frameworks. Therefore, look forward to having further follow-ur up studies with broad applications beyond the active learning problems.

---

Published as a conference paper at ICLR 2023
REFERENCES
Vijay Badrinarayanan Alex Kendall and Roberto Cipolla. Bayesian segnet: Model uncertainty in deep convolutional encoder-decoder architectures for scene understanding. In Proceedings of the British Machine Vision Conference (BMVC), pp. 57.1-57.12 BMVA Press, September 2017.
Venkat Anantharam and Sergio Verdu. Bits through queues. IEEE Transactions on Information Theory, 42(1):4-18, 1996.
Jordan Ash, Surbhi Goel, Akshay Krishnamurthy, and Sham Kakade. Gone fishing: Neural active learning with fisher embeddings. Advances in Neural Information Processing Systems, 34, 2021.
Jordan T Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alekh Agarwal. Deep batch active learning by diverse, uncertain gradient lower bounds. International Conference on Learning Representations, 2020.
Clifford S Asness, Andrea Frazzini, and Lasse H Pedersen. Leverage aversion and risk parity. Financial Analysts Journal, 68(1):47-59, 2012.
François Baccelli Jae Oh Woo. On entropy and mutual information point processes. In 2016 IEEE International Symposium Information Theory (ISIT), pp. 695-699. IEEE, 2016.
Maria-Florina Balcan, Andrei Broder, and Tong Zhang. Margin based active learning. In International Conference Computational Learning Theory, pp. 35-50. Springer. 2007.
Joshua D Batson, Daniel Spielman, and Nikhil Srivastava. Twice-ramann ramaaujja sparsifiers. In Proceedings of the forty-first annual ACM symposium on Theory computing, pp. 255-262, 2009.
Erdem Biyik, Kenneth Wang, Nima Anari, and Dorsa Sadigh. Batch active learning using determi- nantal point processes. arXiv preprint arXiv:1906.0r9 07975, 2019.
Sergey Bobkov and Mokshay Madiman. The entropy per coordinate of random vector is highly constrained under convexity conditions. IEEE Transactions on Information Theory, 57(8):990-- 4954, 2011.
Tamara Broderick, Michael I Jordan, Jim Pitman, al. Beta processes, stick-breaking and power laws. Bayesian analysis, (12):439-766 2012.
Nicolas Brosse, Carlos Riquelme, Alice Martin, Sylvain Gelly, and Éric Moulines. On last-layer algorithms for classification: Decoupling representation from uncertainty estimation. arXiv preprint arXiv:2001.080499 2020.
Razvan Caramalau, Binod Bhattarai, and Tae-Kyun Kim. Sequential graph convolutional network for active learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9083-988855888 9592, 2021.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. arXiv preprint arXiv:2006. 09882, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties self-supervised vision transformers. Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 9650-9660, 2021.
Rui Castro and Robert D Nowak. Minimax bounds for active learning. IEEE Transactions on Information Theory, 54(5):2339-2333 2008
Kathryn Chaloner and Isabella Verdinelli. Bayesian experimental design: review. Statistical Science, pp. 273-304, 1995.
Denis Chaves, Jason Hsu, Feifei Li, and Omid Shakernia. Risk parity portfolio vs. other asset allocation heuristic portfolios. The Journal of Investing, 20(1):108-118 2011.
10

---

Published as a conference paper at ICLR 2023
Ting Chen, Simon Komblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp- 1597-1607. PMLR, 2020a.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E Hinton. Big self-supervised models are strong semi-supervised learners, Advances in neural information processing systems, 33:22243-22255, 2020b.
David A Cohn, Zoubin Ghahramani, and Michael Jordan. Active learning with statistical models. Journal of artificial intelligence research, 4:129-145, 1996.
Giorgio Costa and Roy H Kwon. Generalized risk parity portfolio optimization: An admm approach. Journal of Global Optimization, 78(1):207-238, 2020.
Thomas M Cover. Elements of information theory. John Wiley Sons, 1999.
Daryl Daley and David Vere-Jones. An introduction the theory of point processes: volume II: general theory and structure, volume 2. Springer Science & Business Media, 2007.
Sanjoy Dasgupta, Adam Tauman Kalai, and Claire Monteleoni. Analysis of perceptron based active learning. In International conference on computational learning theory, pp. 163. Springer, 2005.
Erik Daxberger, Agustinus Kristiadi, Alexander Immer, Runa Eschenhagen, Matthias Bauer, and Philipp Hennig. Laplace redux-effortless bayesian deep learning. Advances Neural Information Processing Systems, 34, 2021.
Ofer Dekel, Claudio Gentile, and Karthik Sridharan. Selective sampling and active learning from single and multiple teachers. The Journal of Machine Learning Research, (3()::26552269 2012.
A. Dvoretzky. Some results on convex bodies and banach spaces. 1961.
Ehsan Elhamifar, Guillermo Sapiro, Allen Yang, and Shankar Sasrty. A convex optimization framework for active learning. In Proceedings of the IEEE International Conference Computer Vision, pp. 209-216, 2013.
Alina Ene and Huy L Nguyen. A nearly-linear time algorithm for submodular maximization with knapsack constraint. arXiv preprint (rXi:: 2017.
Robert M Fano. Transmission of information: statistical theory of communications. American Journal of Physics, 29(11):73,,794 1961.
Sebastian Farquhar, Yarin Gal, and Tom Rainforth. On statistical bias in active learning: How and when fix it. International Conference on Learning Representations, 2021.
Thomas Ferguson. A bayesian analysis of some nonparametric problems. The annals of statistics, pp. 209-230, 1973.
Andrew Foong, David Burt, Yingzhen Li, and Richard Turner. On the expressiveness of approximate inference in bayesian neural networks. Advances Neural Information Processing Systems, 33: 15897-15590, 2020.
Adam Foster, Martin Jankowiak, Elias Bingham, Paul Horsfall, Yee Whye Teh, Thomas Rainforth, and Noah Goodman. Variational bayesian optimal experimental design. Advances in Neural Information Processing Systems, 32, 2019.
Adam Foster, Desi R Ivanova, Ilyas Malik, and Tom Rainforth. Deep adaptive design: Amortizing sequential bayesian experimental design. In International Conference on Machine Learning, pp. PMLR, 2021.
L.C. Freeman. Elementary Applied Statistics: For Students in Behavioral Science. For Students in Behavioral Science. Wiley, 1965. ISBN 9780471277804.
Fritz. An approach to the entropy of point processes. Periodica Mathematica Hungarica, 3(1-2): 73-83, 1973.
11

---

Published as a conference paper at ICLR 2023
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty deep learning. In international conference on machine learning, pp. 1050-1059. PMLR, 2016.
Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep bayesian active learning with image data. In International Conference on Machine Learning, pp. 1183-1199 1192. PMLR, 2017.
Jcan-Basticn Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to supervieed learning. Advances in Neural Information Processing Systems, 13:21271-21288 2020.
Yuhong Guo. Active instance sampling via matrix partition. Advances in Neural Information Processing Systems, 23, 2010.
Saeid Haghighatshour, Emmanuel Abbe, and Emre Telatar. new entropy power inequality for enteger-valued random variables. IEEE Transactions on Information Theory, 60(7):3787-3796, 2014.
Steve Hanneke. A bound on the label complexity of agnostic active learning. In Proceedings of the 24th international conference on Machine learning, pp. 353-360, 2007.
Steve Hanneke. Theory of active learning. Foundations and Trends in Machine Learning, 7(2-3), 2014.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9729-9738, 2020.
Matthias Hein, Maksym Andriushchenko, and Julian Bitterwolf. Why relu networks yield high- confidence predictions far away from the training data and how mitigate the problem. In Proceedings of the JEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 41-50, 2019.
Philipp Hennig, David Stern, Ralf Herbrich, and Thore Graepel. Kernel topic models. In Artificial intelligence and statistics, pp. 51 511-91 PMLR, 2012.
Marius Hobbhahn, Agustinus Kristiadi, and Philipp Hennig. Fast predictive uncertainty for classifica- tion with bayesian deep networks. arXiv preprint arXiv:2003.01227, 2020.
Dorit S Hochbaum. Approximating covering and packing problems: set cover, vertex cover, indepen- dent set, and related problems. In Approximation algorithms for NP-hard problems, pp. 94-143. 1996.
Neil Houlsby, Ferenc Huszár, Zoubin Ghahramani, and Máté Lengyel. Bayesian active learning for classification and preference learning. arXiv preprint arXiv: 11/.57755 2011.
Daniel Joseph Hsu. Algorithms for active learning. PhD thesis, UC San Diego, 2010.
Brian Hurst, Bryan Johnson, and Yao Hua Ooi. Understanding risk parity. AQR Capital Management, 2010.
Rishabh Iyer, Ninad Khargonkar, Jeff Bilmes, and Himanshu Asnani. Generalized submodular information measures: Theoretical properties, examples, optimization algorithms, and applications. IEEE Transactions on Information Theory, pp. 1-1, 2021a. doi: 10. 109/TTT.222.3122344.
Rishabh Iyer, Ninad A Khargonkar, Jeffrey A. Bilmes, and Himanshu Asmani. Submodular combi- natorial information measures with applications in machine learning. In The 32nd International Conference on Algorithmic Learning Theory, March 2021b
Aditi Jha, Zoe C Ashwood, and Jonathan W Pillow. Bayesian active learning for discrete latent variable models. arXiv preprint arXiv:2202. 13X4v 2022.
Varun Jog and Venkat Anantharam. The entropy power inequality and mrs, gerber's lemma for groups of order 2 IEEE transactions on information theory, 60(7)3373-3776, 2014.
12

---

Published as a conference paper at ICLR 2023
Philippe Jorion. Value risk. McGraw-Hill Professional Publishing. 2000.
Ajay J Joshi, Fatih Porikli, and Nikolaos Papanikolopoulos. Multi-class batch-mode active learning for image classification. In 2010 IEEE international conference on robotics and automation, pp- 1873-1878. IEEE, 2010.
Matti Kääriäinen. Active learning in the non-realizable case. In International Conference on Algorithmic Learning Theory, pp. 63-77. Springer, 2006.
Michael Kampffmeyer, Arnt-Borre Salberg, and Robert Jenssen. Semantic segmentation of small objects and modeling of uncertainty in urban remote sensing images using deep convolutional neural networks. Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp. 1-9, 2016.
Kirthevasan Kandasamy, Jeff Schneider, and Barnabás Póczos. Bayesian active learning for posterior estimation. In Proceedings of the 24th International Conference on Artificial Intelligence, pp. 3605-3611, 2015. 2015.
Kwanyoung Kim, Dongwon Park, Kwang In Kim, and Se Young Chun. Task-aware variational adversarial active learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2021.
Diederik P Kingma and Max Welling. Auto-Encoding Variational Bayes. 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014.
Durk Kingma, Tim Salimans, and Max Welling. Variational dropout and the local reparameteriizatin trick. Advances in neural information processing systems, 28, 2015.
John FC Kingman. Random discrete distributions. Journal of the Royal Statistical Society: Series (Methodological), 37(1):1-15, 1975.
John FC Kingman. The population structure associated with the ewens sampling formula. Theoretical Population Biology, 11(2):274-283 1977.
Andreas Kirsch, Joost van Amersfoort, and Yarin Gal. Batchbald: Efficient and diverse batch acquisition for deep bayesian active learning. 2019.
Andreas Kirsch, Sebastian Farquhar, and Yarin Gal. A simple baseline for batch active learning with stochastic acquisition functions. arXiv preprint arXiv:2106.12053 2021.
Daphne Koller and Nir Friedman. Probabilistic graphical models: principles and techniques. MIT press, 2009.
Suraj Kothawade, Nathan Beck, Krishnateja Killamsetty, and Rishabh Iyer. Similar: Submodular information measures based active learning in realistic scenarios. Advances in Neural Information Processing Systems. 34, 2021.
Akshay Krishnamurthy, Alekh Agarwal, Kuo Huang, Hal Daumé III, and John Langford. Active learning for cost-sensitive classification. In International Conference on Machine Learning, pp. 1915-1924. PMLR, 2017.
Agustinus Kristiadi, Matthias Hein, and Philipp Hennig. Being bayesian, even just bit, fixes overconfidence in relu networks. In International Conference on Machine Learning, pp. 5436- 5446. PMLR, 2020.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2012.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
Yann LeCun and Corinna Cortes. MNIST handwritten digit database. 2010. URL http:
lecun. cww http///xbbmmmmmi/eee////
13

---

Published as a conference paper at ICLR 2023
Jachoon Lee, Yasaman Bahri, Roman Novak, Samuel S Schoenholz, Jeffrey Pennington, and Jascha Sohl-Dickstein. Deep neural networks as gaussian processes. International Conference on Learning Representations, 2017.
Amnon Levy and Jing Zhang. Credit risk measurement and management. Risk Books, 2019.
Wenxin Li, Moran Feldman, Ehsan Kazemi, and Amin Karbasi. Submodular maximization in clean linear time. Advances in Neural Information Processing Systems, 2022.
Dennis V Lindley. On measure of the information provided by an experiment. The Annals of Mathematical Statistics, pp. 986-1005, 1956.
Andrea Locatelli, Alexandra Carpentier, and Samory Kpotufe. An adaptive strategy for active learning with smooth decision boundary. In Algorithmic Learning Theory, pp. 547-571. PMLR, 2018.
David JC MacKay. Information-based objective functions for active data selection. Neural computa- tion, 4(4):590-660 604, 1992.
David JC MacKay, Choice of basis for laplace approximation. Machine learning, 33(1):77-86, 1998.
Mokshay Madiman, Liyao Wang, and Jae Oh Woo. Majorization and rényi entropy inequalities via sperner theory. Discrete Mathematics, 342(10):2911-2923, 2019.
Mokshay Madiman, Liyao Wang, and Jae Oh Woo. Entropy inequalities for sums in prime cyclic groups. SIAM Journal on Discrete Mathematics, 35(3):1628-16444 2021.
Andrey Malinin and Mark Gales. Predictive uncertainty estimation via prior networks. Advances in neural information processing systems, 31, 2018.
Harry Markowitz. Portfolio selection. The Journal of Finance, 7(1):77-91, 1952. ISSN 00221082, 15406261.
Hermann G Matthies. Quantifying uncertainty: modern computational representation of probability and applications. In Extreme man-made and natural hazards in dynamics of structures, pp. 105-135. Springer, 2007.
McFadden. The entropy of point process. Journal of the Society for Industrial & Applied Mathematics, 13(4):988-994, 1965.
Dimitrios Milios, Raffaello Camoriano, Pietro Michiardi, Lorenzo Rosasco, and Maurizio Filippone.
Dirichlet-based gaussian processes for large-scale calibrated classification. 31, 2018.
Vitali Milman. A new proof of a. dvoretzky's theorem on cross-sections of convex bodies. Funkcional. Anal. i Prilozen, 5:28-37, 1971.
Jishnu Mukhoti, Andreas Kirsch, Joost van Amersfoort, Philip HS Torr, and Yarin Gal. Deterministic neural networks with inductive biases capture epistemic and aleatoric uncertainty. arXiv preprint arXiv:2/02.1 1582, 2021.
Radford M Neal. Priors for infinite networks. In Bayesian Learning for Neural Networks, pp. 29-53. Springer, 1996.
George L Nemhauser and Laurence A Wolsey, Best algorithms for approximating the maximum of submodular function. Mathematics of operations research, 3(3):177-188, 1978.
George Nemhauser, Laurence A Wolsey, and Marshall I. Fisher. An analysis of approximations for maximizing submodular set functions-i. Mathematical programming. 14(1):265-294 1978.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning. 2011.
Bernt Oksendal. Stochastic differential equations: an introduction with applications. Springer Science & Business Media, 2013.
14

---

Published as a conference paper at ICLR 2023
Alon Orlitsky, Narayana P Santhanam, and Junan Zhang. Universal compression of memoryless sources over unknown alphabets. IEEE Transactions on Information Theory, 50(7):1469-144 1481, 2004.
Papangelou. On the entropy rate of stationary point processes and its discrete approximation. Probability Theory and Related Fields, 44(3):191-211, 1978.
Jim Pitman and Mare Yor. The two-parameter poisson-dirichlet distribution derived from a stable subordinator. The Annaly of Probability, pp. 855-900, 1997.
Jim Pitman et al. Combinatorial stochastic processes. Technical report, Technical Report 621, Dept. Statistics, UC Berkeley, 2002., 2002.
Bob Prince.
Risk
parity
about balance. 2011.
URL https://www
bri
bid
Nikita Puchkin and Nikita Zhivotovskiy. Exponential savings in agnostic active learning through abstention. In Conference Learning Theory, pp. 3806-3832. PMLR, 2021.
Edward Qian. Risk parity and diversification. The Journal of Investing, 20(1): 127, 2011.
C Rasmussen and C Williams. Gaussian processes for machine learning. adaptive computation and machine learning, 2006.
Nicholas Roy and Andrew McCallum. Toward optimal active learning through sampling estimation of error reduction. 2001.
Narayana P Santhanam, Anand D Sarwate, and Jae Oh Woo. Redundancy of exchangeable estimators. Entropy, 16(10):5339-5333 2014.
Jacob M Schreiber, Jeffrey A Bilmes, and William Stafford Noble. apricot: Submodular selection for data summarization in python. J. Mach. Learn. Res., 21:161-1, 2020.
Paola Sebastiani and Henry P Wynn. Maximum entropy sampling and optimal bayesian experimental design. Journal the Royal Statistical Society: Series B (Statistical Methodology), 62(1):145-157) 2000.
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. In International Conference on Learning Representations, 2018.
Claude E Shannon. A mathematical theory of communication. The Bell system technical journal, 27 (3):379-423, 1948.
Shubhanshu Shekhar, Mohammad Ghavamzadeh, and Tara Javidi. Active learning for classification with abstention. IEEE Journal on Selected Areas in Information Theory, 2(2):705-719 2021.
Samarth Sinha, Sayna Ebrahimi, and Trevor Darrell. Variational adversarial active learning. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pp. 5972-5981, 2019.
Jasper Snoek, Oren Rippel. Kevin Swersky, Ryan Kiros, Nadathur Satish, Narayanan Sundaram, Mostofa Patwary, Mr Prabhat, and Ryan Adams. Scalable bayesian optimization using deep neural networks. In International conference on machine learning, pp. 2171-21 2180. 80. PMLR, 2015.
Daniel A Spielman and Nikhil Srivastava. Graph sparsification by effective resistances. SIAM Journal on Computing. 1916))1913-1926 2011.
Daniel A Spielman and Shang-Hur Teng. Nearly linear time algorithms for preconditioning and solving symmetric, diagonally dominant linear systems. SIAM Journal on Matrix Analysis and Applications, 35(3):835-885. 2014.
Daniel A Spielman and Jae Oh Woo. A note on preconditioning by low-stretch spanning trees. arXiv preprint arXiv:0903.28/1 2009.
15

---

Published as a conference paper at ICLR 2023
Wei Tan, Lan Du, and Wray Buntine. Diversity enhanced active learning with strictly proper scoring rules. Advances in Neural Information Processing Systems, 14:10906-10918, 2021.
Dimitris G Tzikas, Aristidis C Likas, and Nikolaos P Galatsanos. The variational approximation for bayesian inference. IEEE Signal Processing Magazine, 23(6):33 2008
Vladimir Vapnik and Alexey Chervonenkis. Theory of pattern recognition, 1974.
Sergei Vassilvitskii and David Arthur. k-means++: of the eighteenth annual ACM-SIAM symposium on Discrete algorithms, pp. 1027-1035, 2007.
The advantages of careful seeding. In Proceedings
Isabella Verdinelli and Joseph B Kadane. Bayesian designs for maximizing information and outcome. Journal of the American Statistical Association, 87(418))510-515, 1992.
Benjamin T Vincent and Tom Rainforth. The darc toolbox: automated, flexible, and efficient delayed and risky choice experiments using bayesian adaptive design. PsyArXiv. October, 20, 2017.
Liyao Wang, Jae Oh Woo, and Mokshay Madiman. A lower bound on the rényi entropy of con- volutions in the integers. In 2014 IEEE International Symposium on Information Theory, pp.
IEEE, 2014.
Zheng Wang and Jieping Ye. Querying discriminative and representative samples for batch mode active learning. ACM Transactions Knowledge Discovery from Data (TKDD), 9(3):1-23, 2015.
Kai Wei, Rishabh Iyer, and Jeff Bilmes. Submodularity in data subset selection and active learning. In International Conference on Machine Learning, pp. 1954-1963. PMLR, 2015.
Wikipedia. Gumbel distribution Wikipedia, the free encyclopedia. org/ www.wiiiiwi.... Mamb/Gumell.mmmm distriisiuuii ion 2023.
Christopher KI Williams. Computing with infinite networks. Advances in neural information processing systems, pp. 295-301, 1997.
Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric P Xing. Deep kernel learning. In Artificial intelligence and statistics, pp. 370-378. PMLR, 2016.
Jac Oh Woo. Analytic mutual information in bayesian neural networks. 2022 IEEE International Symposium on Information Theory (ISIT), 2022.
Jae Oh Woo and Mokshay Madiman. A discrete entropy power inequality for uniform distributions. In 2015 IEEE International Symposium on Information Theory (ISIT), pp. 1625-1629. IEEE, 2015.
Yi Yang, Zhigang Ma, Feiping Nie, Xiaojun Chang, and Alexander G Hauptmann. Multi-class active learning by uncertainty sampling with diversity maximization. International Journal of Computer Vision, 113(2): 13-127, 2015.
Grigory Yaroslavtsev, Samson Zhou, and Dmitrii Avdiukhin. "bring your own greedy"+ + max: near- optimal 1/2- approooxximaiions for submodular knapsack. In International Conference on Artificial Intelligence and Statistics, pp. 3263-3274. PMLR, 2020.
Donggeun Yoo and In So Kweon. Learning loss for active learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 93-102. 2019.
Byung-Jun Yoon, Xiaoning Qian, and Edward R Dougherty. Quantifying the objective cost of uncertainty in complex dynamical systems. IEEE Transactions on Signal Processing, (1): 2256-2266, 2013.
Chicheng Zhang and Kamalika Chaudhuri. Beyond disagreement-basedd agnostic active learning. Advances in Neural Information Processing Systems, 27, 2014.
Guang Zhao, Edward Dougherty, Byung-Jun Yoon, Francis Alexander, and Xiaoning Qian. Uncertainty-awart active learning for optimal bayesian classifier. In International Conference on Learning Representations (ICLR 202/), 2021.
16

---

Published as a conference paper at ICLR 2023
Yinglun Zhu and Robert Nowak. Active learning with neural networks: Insights from nonparametric statistics. Advances in Neural Information Processing Systems, 2022a.
Yinglun Zhu and Robert Nowak. Efficient active learning with abstention. Advances in Neural Information Processing Systems, 2022b.
17

---

Published as a conference paper at ICLR 2023
A APPENDIX
A.1 BAYESIAN NEURAL NETWORK AND MUTUAL INFORMATION
We may formulate the Bayesian neural network t as a well-known encoder-decoder framework. The sender sends message (x, w) with a random key through the Bayesian neural network, then the receiver receives message Y (x, w). Figure illustrates a diagram in this process.
wx)
Encoder
& (x,
Decoder
Y (x, w)
Figure 4: Bayesian neural network encoder-decoder framework
Under this framework, controlling w is difficult, but we can control the family of the encoded messages & (x, == (Pi (x,w)..... Pc (x,w)) in tractable manner (Gal et al. 2017 Kingma & Welling 2014 Tzikas et al. 2008). We can easily prove that the mutual information between w and Y is the same the mutual information between the encoded & (x,w) and the predictive output Y
since Y depends only on 4 (x,
BALD[x] : (w>, Y (x, = H( (x,w) - E. II (x, [w)] =Y (x,w) Ex [H (Y (x, w) & (x, = J ( (x,w), Y (xxw)))
(5)
(6)
where H(Y (x, w)) represents the Shannon entropy by marginalizing out the randomness of w in Y (x, w) and ((,)) represents a mutual information between two quantities.
The formulations of the mutual information equation 5 equation 6 look natural, but we need to note that w or & (x, w) is on a continuous domain, and (x. w) is on a discrete domain. This combined domain implies that we cannot directly apply Shannon entropy and differential entropy notions (Cover) 1999). One immediate question is what the joint entropy between & (x, w) and Y Y(x, w) is. For this.
can leverage point process entropy (McFadden 1965 Fritz 1973| Papangelou, 1978, Daley & Vere-Jones] 2007 Baccelli & Woo 2016) by generalizing the notion of the entropy in this combined domain. We consider the joint entropy of & (x, w) and Y (x, w) denoting by S ( (x,w), Y(x, w)
through the point process entropy. We write Janossy density function (Daley & Vere-Jones j (p. y = i) of (4 (x,w) Y(x, w)) on A [C] as follows:
2007
j (p. y i) p.f (p),
where p (p . .Pc) and f()) is density function of & (x,w). Then the joint entropy of (x, w) and ((x,w) w) can be defined as
C / j (p.y i) logj (p. dp.
5) (x,w), (xx = Σ
(8)
By plugging equation 7 into equation g we have the following identity.
S ( (x) Y Y(x, w)) =Y w)) + Ey [h (Φ (x,w) YY (x,w))),
(9)
where H()) represents the usual Shannon entropy, and I:(-) represents the usual differential entropy. By applying Jensen's inequality, we may derive a marginalized joint entropy as an upper bound of the joint entropy (See Appendix A.2):
5 (2 (x,w), Y (x,W)) Σ Ep [P, log (P.)).
(10)
where we ambiguously write f(-) to be density function for each P..
A.2 DERIVATION OF MARGINALIZED JOINT ENTROPY WITH THE POINT PROCESS ENTROPY
The Janossy density function resides in a combination of continuous and discrete domains Daley & Vere-Jones 2007). For the Janossy density of (& (x, w). (x),)) A [C), may follow the
18

---

Published as a conference paper at ICLR 2023
classical approach:
P [pi + dyrl: ... Pc E [ee + dpel.Y= =
=P (Y = = = pl... Pc = pc) (PP E [pi + dpil, ... P E [pe + dpel)
S (pi.... PC) dp dpc.
(11)
where f(.) is density function of Φ (x,w). So we may write the Janossy density of ( (x, Y (x, w)) as follows:
(PI PC. = P.f (Pi, .Pc)
(12)
Following the point process entropy (McFadden, 1965 Fritz 1973, Papangelou, 1978 Daley Vere-Jones 2007), the joint entropy of Φ (x, w) and ((x, w) can be defined as
= 5) (x,w), ((x,))
Σ
/ j (pI, PC. log (P1, ,cC, i) dp: dpc. 1=1
(13)
We note that
J&. Pif (PI... .p)) dp1 ... dpc - - / [0,1]
Pif (p:) -EP,
(14)
We may split the Jannosy density into two pieces:
j (pr," .. "
(EPP:)
Pi EP,
(p1.
(p..... ... ..))
(15)
Plugging equation 15 into equation
13 we have
( (x, ((x,.) H (Y(x,))) Ey [h (x)
(16)
On the other hand,
(6) Σ
(pr... PP.Y = i) log (p1. Pc, dp ... dpc
Σ
(E.P;)
Pt EP,
P (pl, (......, (pp......pp) .pe)
log (EP.)
Pt E..
p (pI. ... ppc) dpi ....pp
i=1 A
Σ
(EP.) log (E..)
Σ/ (EP.)
Pt EP.
(pi.
pc)
log
Pc EP.
p -.. pc) dp.
dpc.
1-1
(17)
We apply Jensen's inequality on the second term (by focusing on each summand). For each i (1....C).
-EEP.)
Pi A E.P
P (PI. PPC) log
Pi EP. (pt).....pp)))
p (pI.
.p.....))))))))))))).)))) pPc) dpp ..-dpc
-BBP) [ JACMAN) E.P (pI, ... .PC)
PPI,
log
Pi E.P.
(pI.... .P.) dp dpii
s - -BEP) I (Jan)tar) p (PI, (pp)) dp PL
log f Jaryon
pi
(pr. PC) dpri.c
dp:
EP.
EP
=- pi(
Pr E... f(p)) dp, EP, P. P.
P. f(P.)
(18)
E.P.
where dp indicates dp) ... dpc except dp.. By combining all terms together, we have
(6) <
(EP.) log (IEP))
Bp.
P. log
P. EP. ((P))) - =
ER -
log (P.)(..)).
(19)
19

---

Published as a conference paper at ICLR 2023
A.3 EQUIVALENT FORMULATION OF MARGINALIZED JOINT ENTROPY
Let us assume that P. ~ Beta(a;, B:) and Pt Beta(a; + Bi).
MJEnt[x] Σ Bp [P, log (P.f(P.))) -
- Σ/ Pff
log (p.f(pi)) dp.
Σ (EP.) /'' prf(pi)
ppf((p))
log
dp; Σ (EP.) log (BPP.
(20)
E.P
EP:
((PP)) (EP.) [h(P,t) log (EP.))
(21)
where h(P,t) is the differential entropy P++.
A.4 PROOF OF THEOREM 3.1
Let 1 (m, IC)) and n(i,++)) ++) (m,..., -7--1: 1, ;c). Let B (n)
T(me)..(mn)) and r()) is a Gamma function. Assume that & (x,w) == (PI.... Pc) ~ r(X mx)*
Dir JC).
Theorem A.I. Woo 2022 Theorem III.1) The analytical formula of the mutual information BALD is the following.
C Σ
C
C
DirichletBAl.D/s
th
C
Σ
?k
(n 1) (m:
7/c
log
Σ
k=1
1=1
km
===1
(n) 1) B (m(i, ++))
1 (m1) v (1$
B (n)
m
+
M=l
B (mii, ++))
B (m)
(m (m +1 1
Σ
m
+
where V() is Digamma function.
Given the above theorem, we can simplify the formula further: DirichletBALD|x
1k
C
Σ
M
Σ
1)
(mh)
Ti
log
T
Σ
1=1
1=1
R=
17k
k=
17k
(1 - -1)
C
Ve (1+) -
Σ T
+ 1
Σ M
izti
V (n: + +1
Σ mk
+
C
Ek=1 11
C
C
1h
C
Σ 1k
m
log
5 1= 176
1=1
7k
T (m
11
v (nd)
C
(n+
1)
1
ni
V
Σ
The
+1
Dg k= M
1=1
kk= mhk
A=
T
(n: + +1) - v
Σ m
+1]
Ek 11
Therefore DirichletBALD is function of marginals of & (x,w) with Dirich- let distribution parameters and Σ sumption, by letting " o, and Σ
Under Beta marginal distribution as-
k 7
04 B for any i since each
20

---

Published as a conference paper at ICLR 2023
marginal distribution of Dirichlet distribution
follows
Beta
distribution, we have
Be == Σ a a- 1) (a, +
Σ
ai
log
de (o: - 1) (a.) as + B.
a; + 3:
Ci + 3.
[=1
1=1
(a. 1)
C
af
v (a, + A
+
M (a, + (a, + + 1)].
+ Bi
Therefore Theorem 3.1 follows.
On the other hand, since Beta marginal distributions are sufficient to calculate the mutual information, the same idea can be applied to the aleatoric uncertainty.
Corollary 1. Under Beta marginal distribution approximation, let P. ~ Beta 3,) in &b (x,w). Then the aleatoric uncertainty can be estimated as follows:
C
as (a: -0-1 1) (o.)
BetuMarriialMeetrrricChrrraaaan =
Σ
(a, ()
v (a, +.)++
04 + +.
1=1
Bi (a:: - v (a, + 11)
C
Σ
[V v (a, + 11+1
05 + A3
04
B1
A.5 PROOF OF THEOREM 4.1
First let positive integer A-1 > 0 be given and let T = {I,), collection of evenly divided
intervals in [o, 1] where J = [nn - 1)4, nA) for n = 1,... (4 -1 - 1) and / == [1 A, 1]. Let P. be a discretized random variable over Y of P, from + (x,w). i.e., P, = (n 2) A if P. E I, such that P [P: (n- - )) A P PP, E [n]. For any estimator P: of P: given the label (Y = i). by applying Fano's 's inequality (Fano) [967] Anantharam & Verdu (996)(Cove)) 1999. Theorem 2. 10.1.))
we have (note that our log has a base e)
(P.Y = 1loo-oog log 2 log A-
H (P:) = i) - log
IP P
(22)
loog A
We note that Shannon entropy and the differential entropy have the following connection (Cover 1999. Theorem 8.3.1):
H (P.YY i) + log A h (P.lY - + + co.
(23)
where 6 is an adjustment constant depending on A such that 6; - as A - O. Note that 6 does not have be non-negative. Then we can rewrite the inequality as follows:
PP
P. + P. Y
(P:") log - -log2 2
E2 log A
(24)
log A
Taking the expectation with respect to Y, we have
E PpPE P. P. Y : Σ. (EP.) ((P)) log A - -og 2
Σ; (EP,) c;
(**).
llog A
loog A
If we let 1 (20 J. there exists a 5 2 0 such that
H(Y) + logg2 ô- log A= log 2e H()) $ HYY) + log 2.
(25)
We also note that 5 > as H(Y) + oo (or equivalently A + 0). Therefore, when A-1 ===12e"((>) have
(**) = Σ, (PP) - +
+ H(Y) + log 8
Σ, (EP:) Ci
H(Y + log 2 8
Σ. (E...) h((PP) + H()
(1+
H(Y) + log 1og2- 5) )
8
(EP.) |ca| + H(Y) +log2
(26)
(YY + log
21

---

Published as a conference paper at ICLR 2023
3, (EP.),e
Let 81 +log) 2-8 and 82
H(Y +ogg? 2 0. Since e, - 0 and 6 0 as A E1 and 62 + as > 0. Therefore Theorem 4.1 follows.
As a final remark, we note that A can be regarded as variant of the discrete entropy power Wang
et al., 2014, Woo & Madiman 2015 Madiman et al. 2019 2021 Haghighatshoar al. 2014 Jog
Anantharam 2014).
A.6 MORE BETA MARGINAL FORMULATIONS
With Beta approximation, we are able to describe Beta marginal formulation of MeanSD. Since we matching the variance of each marginal distribution, the empirical value of MeanSD should be the same as BetuMarginalMeanSD...
a, (a: + (a) 3 I))
BetaMarginalMeennDD x
MeanSD[x]
(27)
C
In Foster et al. 2019). the expected information gain has been proposed and studied. We may also formulate the expected information gain with Beta marginal distributions.
BetaMarginalElG/x/ :Y) EH (
Σ
C
ai
Σ
C
aj 8,(0)
log
aj o:(i) ag + Bg +1
log
04
04 + B,
aj + 3, +1
0g + B,
(28)
where Y+ is categorical random variable over the posterior probability given Y i, 8,1)) 1 i = j. and (5) = otherwise.
A.7 BETA MARGINAL APPROXIMATION VISUALIZATION EXAMPLES
20
15-
donaly
donsty
10
0.00
0.75
0.50
0.50
0.75
0:00 022
0:50
0.75
0.00 0.5
0:50
0.7
0.00
0.5
a5o
0/s
1.0
15-
20-
densty
10-
0.5-
0.0 n P7
0:50 8:0
0:00
P0
P9
P10
Figure 5: An example of Beta approximations (red lines) for each marginal distribution after applying softmax layer in MNIST dataset. Each Beta distribution is estimated by calculating the sample mean and sample variance of the histogram generated by the Bayesian deep learning model.
Figure 5 and Figure shows an example of Beta approximations obtained from the MNIST and CIFAR- 10 datasets. Pı. P1o show each marginal distribution of the predictive probability of each digit. We observe that the Beta approximation is a reasonable approximation.
22

---

Published as a conference paper at ICLR 2023
7.8
6.0
0--
0.0
0.50
0.00
0.5 1.00
0:00
0.26
0.00
575 1.50
0:50 0.
0:00 0. 1.00
0.00 0.
0
0.000
0.500
P9
P5
15
10.2-
consty
7.5-
10-
5.0-
0.75
0.00
0.75
0.56
025 0.50
0.50
Figure 6: An example of Beta approximations (red lines) for each marginal distribution after applying softmax layer in CIFAR- 10 dataset.
A.8 RANK CORRELATION STUDY WITH BETAMARGINALEIG
A good advantage of explicit formula is that we can study the behavior of each measure directly. For example, if C' and (x, w) Dirichlet(a, 3) such that Pi ~ Beta(a,, 3) and P3 Be)a we are able to plot the behavior of each Beta marginal measure.
(a) BetaMarginalBBLD
(b) BetaMarginalMeanSD (c) ExpectedEffectiveooss
(d) BetaMarginalEIG
Figure 7: 3D plot of each uncertainty measure when Beta marginal assumption holds.
With BetaMarginalEII, we are able to generate the same type of plot shown in Figure 1. BIG shows positive correlations with BALD and MeanSI), but the correlation is around 70% implying that EIG might show more variations.
roBAL
30-
309
Medic
634
03-4
611
405
000
250
RITHAAERROONN Fqges ww KO
014
0%1
Figure 8: Same experiments with BetaMarginalEIG described in Figure from the main article. This is another independent experiment (as validation), so the captured correlation values are slightly different.
23

---

Published as a conference paper at ICLR 2023
252
Judd Seher
251
250
240
300
2.00
0000445
1000415
8.56
8.94
899
240
2.30
230
0.0
002
240 250
300
0.0347
Figure 9: Pairwise scatter plot at C = 10, 000 companion with Figure
A.9 BALD AND BETAMARGINALBALD
Although BALD and BetaMarignalBBLD shows high rank-correlation (under softmax applied Gaussian distribution assumption), we might wonder how much different they are in the value. We plot the RMSE (rooted mean square error) between two measures. Under Dirichlet distribu- tion assumption, RMSE between BALD and BetaMarginalBALD is < 0.002 upto C' < 1, 000. However, under softmax-applied Gaussian distribution assumption, RMSE between BALD and BetaMarginalBBALD shows < 0.07 upto C' < 1, 000. This implies that Beta marginal approxi- mation still preserves a high rank-correlation, but absolute values are slightly shifted. e.g., BALD xx] BetaMarginalBBLD xx] + err for some constant err P. This study also implies that Beta marginal approximation is reasonable assumption.
24

---

Published as a conference paper at ICLR 2023
338
Figure 10: Scatter plot at C' 1, 000 between BALD and BetaMarginalBALD (left), RMSE between BALD and BetaMarginalBALD over various class dimensions (middle), and Spearman's rank correlations over various class dimensions (right). The first row the result from C-dimensional 100 random Dirichlet samples. The second row is obtained from softmax-applics 100 random Gaussian samples. Then we repeat the process 10 times. both cases, we observe > 96% rank correlations well.
A.10 TOY EXAMPLE WITH EXPECTEDEFFECTVELOSS AND BETAMARGINALEIG
As observed by high rank-correlation in Figure BALD, ExpectedEffectiieLooss, and Beta- MarginalEIG show similar selections.
(a) BALD
(b) ExpectedEfiiccivee..ss
(c) BetaMarginalFIG
Figure 11: Top-K selected points are marked by red color. The first row shows the top K 25 point selections. The second row shows the top K = 500 point selections among around 0.6 million uniform grid points. The same experiments shown in Figure in the main article.
Figure 12 shows the active learning curves for ExpectedEffectiieLoss and BetaMarginalEIG with MNIST and CIIIARR-100, This experiment also confirms that BALD and ExpectedFiffective..o. tightly aligned as show that both highly correlated. In MNIST, BetaMarginalEIGG per- forms similar to BALD and ExpectedE.ffectivel.ss However, 3 CIFAR- 100, BetaMarginalEIG performs similar BALD first, but it essentially performs better than BALD and similar the random case. Recall that the rank correlation between BALD and BetaMarginalEIGG is around 70%.
25

---

Published as a conference paper at ICLR 2023
641
BalEntAnq aquisition
BalEntAcq aquisition 500
Random aquisition size
Random aquisitian sze 500
aquisition
sirecctedEfiectivve..... oss aquisition size
Expecteddiffcciiveoos aquisition size 500
BetaMarginalE aquisition size
BetaMarginaleiG
1900
100
(a) MNIST
(b) CIFAR-100 100
Figure 12: Active learning curves for ExpectedEfffectivel..o oss and BetaMarginalEIG with MNIST and
3x CIFIRR--100,
A.11 NON-NEGATIVE BALENTACQ REGION
In this section, we study the non-negative region of BalEntAcq [x]. BalEntAcq(x] non-negative when MJEnt[x] 2 0. Under Beta marginal distribution approxima- tion, let P. Beta(as, B.) in & (x,w). Then we can fully write MJEnt[x] as follows:
MJEm([x] Σ (ER))(P')+ HTY)
log B(o + 1,8) o (0: + (8, (2) - (ar + 8 ()0 + 8 + log aa +
Then, we are able to generate plot and a contour plot of BalEnt [x] when C 2. i.c. (x ~ Dirichlet(a, 3) such that P1 ~ Beta(a, B) and P3 ~ Beta(B,
-02
100
12
Figure 13: BalEnt 3D plot (left) and Positive BalEnt contour plot (right) over parameters (a, 13). For the contour plot, starting from the outside, contours generated when BalEnt [x] (-3, -,,0,,, 0.3).
Figure 13 suggests that in Dirichlet distribution's 's parameter space, there exist (uncountably and) infinitely many parameters which produce non-negative BalEnt [x] values. Then we also plot the non-negative region (red shaded) of BalEntAAc in toy example.
Figure (a) illustrates that there exist infinitely many points which produce the same BalEntAcq [x] conducting uniform sampling each contour surface /BalEnt[x* x] = A} for each À 2 0, then moving to the surface for each À 2 0. This observation also explains how BalEnt[x] diversifies the selection the decision boundary.
values. Therefore may imagine that
26

---

Published as a conference paper at ICLR 2023
(a) Non- non-negative BalEntAcq region
(b) Acquired 25 BalEntAcq points
Figure 14: Non-negative BalEntAcq region illustration from the toy example
A.12 IMPLEMENTATION OF BALANCED ENTROPY ACTIVE LEARNING
Implementation of BalEntAcq active learning is almost the same the usual MC dropout-based uncertainty methods. The difference is the dropout samples at inference time; BalEntAcq requires an additional estimation step of Beta parameters for each marginal distribution. Algorithm[]] exxplaa the whole steps of BallnnAAq active learning. Moreover, we do not apply any early stopping criteria in each training because we understand that early stopping conflicts with dropout-based model training. In other words, if we stop too early in model training by observing validation accuracy loss, observe that model weights cannot reach to the fully mixing states of the randomness in MC dropouts, similar to the early stage of Markov Chain Monte Carlo (MCMC).
Algorithm 1: BalEntAcq active learning algorithm
Input:
training 3) the number of dropout samples M at inference time, 4) active learning budget K for each iteration, 5) total active learning budget Kooo
Unlabeled dataset D poot. initially labelled dataset D ((()
Initialize all weights of Bayesian neural network & and set
Repeat at iteration 2
Train the model & with ((n)
training
For each x D P (D(n)
training'
Generate M dropout samples
Estimate Beta parameters (a,,8)) B1)) for each marginal distribution
Calculate BalEntAcq[x]
Set (1++1) training
training U top K BalEntAcqqvalued Dpool D(n))
(n)
training }
and n + -1
Until
D
(R-1) training
reaches to Ktot
A.13 MORE EXPERIMENTAL DETAILS
Table 2 shows summary of dataset, configurations, and hyperparmeters used in our experiments.
For each experiment, we repeat 3 times to generate the full active learning accuracy curve.
Sommio
CCoaua
Irany
Optinin
Ppachs
Leamingre
Dmpurt
Pull deopors
MNIST
10
1
200
CNN
Cros ectooyy
x 25
128
Adom
150
h.0l
160
Somte
CHAAR-100
100)
300
16,000
RoNNa-10
Dow-eriney
138
Arbin
150
0.001
X0%
100)
Pie Pearaned hooMinee
TiplinageNet
200
1,300
00,000
Ro
Closs extrogy
64 64
128
Adam
100
0.0011
202
100
Table 2: Detailed configurations used in our experiments.
27

---

Published as a conference paper at ICLR 2023
In SimCLR (Chen et al. 2020a) feature training, we trained ResNet-50 with 224 x 224 image size, 192 batch size, 500 epochs, and 0.0003 learning rate with Adam optimizer for
A.13.1 SIMPLY LAST+ LAYER BAYESIAN
Dropout-basee bassed Bayesian neural network typically requires adding dropout layer with ReLU activation for each convolutional or linear layer to approximate Gaussian process (Gal & Ghahramani] 2016). But this requires a high computational cost. Therefore we adopt several additional last layer dropout architecture to build a Bayesian neural network equipped with Beta approximation. There exist several different lines of works to justify the effectiveness of this simple last layer modification (Snoek al. 2015; Wilson et al., 2016 Brosse et al. 2020; Kristiadi et al., 2020) Hobbhahn et al. 2020). More precisely, similar to Laplace approximation applied at the last layer (Kristiadi et al. 2020 See Theorem 2.4) and (Hobbhahn et al. (020), we may replace several last linear layers with dropout applied and ReLU activated linear layers.
For example, we may add two or more dropout layers after ResNet-50 fixed backbone in our CIFAR- 100 experiments to avoid any pathological cases (Foong et al. (020). In practice, we observe single dropout layer application is sufficient to achieve our Beta approximated marginals as shown below. We note that in MNIST experiment, we use 50% dropout rate and for all other our experiments, we use 20% dropout rate.
A. 13.2 CHOICES PRIORITIZATION IN BALENTACQ
In this section, we study the impact of the prioritization in BalEnt [x].
P1. P1 [x] BalEnt [x]. This is the where put higher priority when the posterior
uncertainty captures very small values. Note that this also includes epistemic uncertainty (BALD) valued case. For example, when C 2 with & (x, Dirichlet(a, 3), the posterior uncertainty goes to -00 as + O and 3 - 0. Therefore BalEnt [x] + -c0. But this case also achieves the highest epistemic uncertainty.
BallEnt [x]
if BalEnt[x] 2 0, if BalEnt[x] < 0
P2. P2[x]
This is the same case as our proposed acquisi-
BalEEnt xx]
tion measure.
P3. P3 xx] = BalEnt [x]. This is the case where we put higher priority when the posterior uncertainty captures very high values (close to zero). As discussed in Section 4.1, want to prioritize more when the information imbalance gap is higher.
Figure 15 and Table 3 show that selecting the points near BalEnt[x] Rs 0 is better way to improve the accuracy we discussed in Section 4.1. When we prioritize the small posterior uncertainty case, PI shows a very poor performance a fixed feature scenario. However, under the backbone and augmentation scenario, the performance of Pl is similar to the high posterior uncertainty case of P3. This could be because the evolution of the feature space and the batch normalization during the active learning process. i.e. previously captured uncertainty values will not be preserved under the backbone with augmentation scenario.
Table 3: Selected accuracy table depending on different prioritization. Mean and standard deviation are from 3 repeated experiments. The best performance in each column is shown bold.
Scenanoo
Redundant + Fixed feanure C*CIR 1005001100000
Hacishone Augmentation
Dotaeet/Acq SizerTest size
SizwePeol
15,0001
500000
90.0001
14/00/1000000
P]
37.6 0.77%
41.5 3. 0.4%
45.6 1.1%
48.4 0.8%
28.2 1.2%
33.8 1.00%
41.7 1 0.75
43.6 0.2%
P2 (BalEatAo Ps
56.0 12:::000 0.65%
63.5 8 4.4%
06 0.3%
6T.4 0.15
30.0 30.0.:000 0.05
388 5. 0.2%
42.8 = 0.75%
45.3 0..4%
53.6.80.119 0.1%
60.3 0.65
63.4 0..9%
64.7 16.7.000 0.256
289==0.5
35.2 0.0%%
41.0 0.9%
45.0 0.22%
A. 13.3 BEHAVIOR OF DIFFERENT PRECISION LEVELS
As shown in the proof of Theorem 4.1, may have freedom to choose the level of the precision in the P. estimation. Therefore we report the active learning behavior for other precision choices. It is not clear which precision level achieves the optimal performance, but preference
28

---

Published as a conference paper at ICLR 2023
aquisition size (BelEntAcq) aquisition size 500 aquisition size 500
cM
aquisition size (Be [CnLAcq] aquisition 1500 aquisition size 1500
(a) 3 CIFAR-100
(b) TinyImageNe
Figure 15: Active learning curves depending different prioritization.
of log A
YY) + log 2 shows reasonably superior performance in any scenario shown in Figure and Table
Case 1.
log A sJ H(Y) - log 2,
Case 2. log A HY)).
Case 3.
logg A H(Y) + log (our choice),
Case 4. log A HYY log
Case 5. log A H (Y) + log 2.
Casel aquisition size 500
Case2 aquisition size 500
Casel aquisition size 158C
Case3 (BalEntAcq) aquisition size 500
aquistion
Case4 aquisition size s00
Case3 (BalEntAca) aquisition size 1500
equisiion size
Case5 aquisition size s00
equisition
(a) 3x C1F
(b) TinyImageNet
Figure 16: Active learning curves depending on different precision levels.
Table 4: Selected accuracy table depending on different precision levels. Mean and standard deviation are from 3 repeated experiments. The best performance in each column is shown in bold.
images
Aupmentttion
Dattsee//ng Sise/Test
3xCIRAR 100.00011.0000
TinyImapeNeti S00/10.000
Train Size/Pool Suee
50000150,000
15.00/11000000
23,0001500000
50000055000000
6.000100,000
10,,000
24100/100,000
30,0001000000
(RalEmAaa)
0.2%
Cooe
0.95%
0.0%
Coc5
30.1 0.77%
42.9-035%
0.555
29

---

Published as a conference paper at ICLR 2023
A.13.4 MORE DETAILS ABOUT THE MAIN EXPERIMENT
Figure [7 shows the full active learning curves, negative log-likelihood, average epistemic uncertainty for selected samples, and average aleatoric uncertainty for selected samples. We note that our proposed method selects neither high epistemic uncertainty (=model uncertainty) nor aleatoric uncertainty (=data uncertainty) samples. Nevertheless, BelEntAcq shows a good performance improvement during the active learning iterations. Furthermore, we observe that BelEnnAAq keeps choosing low aleatoric uncertainty points but increasing epistemic uncertainty points.
SAD 1:003
Dring zo stansiie
Rsscon autor 10:3300
Meved aulive 511130
latray aquitiis oo
Artik.gge levve 118
(a) Accuracy
(b) Negative oiglliieldhod
(c) Epistemic uncertainty
(d) Aleatoric uncertainty
Figure 17: Full active learning curves obtained from different scenarios. From the top row, each row represents a result of MNIST, CIFAR-100, 3XCIFAR-100, and TinylmageNet.
30

---

Published as a conference paper at ICLR 2023
A.14 TINYIMAGENET WITHOUT PRETRAINED MODEL
In this section, we report the active learning result of TinylmageNet without a pretrained model. By doing so, we can also observe the effect of the pre-knowledge of the model.
We use the same setting as we used in the main experiment. We observe that the accuracy progression slower, it requires more samples, but the observed behavior of each method is the same as before.
BalEntAcq aquisition size 2500
0.3
Random aquisition size 2500
BALD aquisition size 2500
Entropy aquisition size 2500
MeanSD aquisition
PowerBALD aquisition size 2500
BADGE aquisition size 2500
32.00 2009
Figure 18: Active learning curves of TinylmageNet without pretrained model
Table Selected accuracy table. Mean and standard deviation are from repeated experiments.
Scenario
Fixed feature + dropouts
Dataset/Acq. Size/Test size
TinylmmageNer22550000000.00
Train Size/Pool Size
15,000/100,000
25,000/100,000
50,000/100,000
Random
26.3 +: 1.2%
32.4 0.2%
BALD
26.3 + 1.0%
32.4 0.2%
19.7%002%%
Entropy
24.3 + 0.8%
38.3 1.0%
McanSD
26.3 - 0.5%
31.5
39.2 1.3%
PowerBALD
26.2 = 0.5%
32.1
40.2 0.8%
BADGE (not-scalable)
26.7 + 0.5%
32.8 = 1.6%
40.4 + 0.5%
BalEntAcq (ours)
25.6 1.3%
33.0 = 0.9%
41.8 + 1.0%
A.15 RELATIONSHIP WITH THE EFFICIENT ACTIVE LEARNING ALGORITHM WITH ABSTENTION
This section illustrates the relationship with the recently proposed efficient Algorithm by Zhu Nowak (2022b). Note that we are not proving the equivalence between the two algorithms. As demonstrated A. we can see that our proposed BalEntAcq method shares high similarity
with active learning strategy with abstention (Locatelli al. 2018 Shekhar et al. 2021 Puchkin & Zhivotovskiy 2021 Zhu & Nowak 2022a b).
Intuitively, Algorithm 1 in Zhu & Nowak (002b)) works in the following way. Set abstention parameter > O. Train a binary classifier ((r). For unlabelled point x E X, we can calculate a
uncertainty bound, UB [x] = [leb(x), ucb(x)). If UB C
7, 1 + 7]. we abstain the point x,
ie., we do not query the point x. If E UB(x] and UB Ø 2 7, + 7]. we query the point x.
At each iteration m, add geometrically increasing 2d queried points.
The key insight this Algorithm 1 to achieve exponential label savings is to abstain from the point very close the decision boundary. Similarly, as demonstrated in A.11 BalEntAcq(xX [x] finds
31

---

Published as a conference paper at ICLR 2023
margin by focusing the positive sign of BalEntAcq(x) which corresponds to finding x outside the abstention region such that | - 2 > near the decision boundary. Then following the positive BalEntAcq[x] values, we acquire points toward the decision boundary direction, which corresponds the condition 1 E UB x]. We know that the point near the decision boundary should have high aleatoric uncertainty (so possibly noise-seeking). On the other hand, Corollary immlliess that aleatoric uncertainty increasing as a + +00. So MJEnt[xx - -00 Then BalEntAcc(x) - -o0. Therefore, our BalEntAcq[x] will acquire points near the decision boundary but will not acquire the point if it's too close to the decision boundary. This strategy in our BalEntAcq [x] exactly matches the key insight of Algorithm 1. So we may be able theoretically guarantee that our proposed acquisition function BalEntAcq[xx could be a universally working active learning algorithm by achieving exponential label savings.
A.16 MORE EXPERIMENTS WITH SMALLER ACQUISITION SIZE
In this section, conduct more experiments with 3x MNIST and 3 CIFARR-00 by adding more baselines such as VarRatio xx] == max; EP, (Freeman 1965). BatchBALD, and CoreSet. The main purpose of these experiments is test the relatively smaller acquisition size. We acquire 25 points for each active learning iteration. For x MNIST, we use CNN architectures, For 3 CIFAR-IO, we fix the feature space obtained from SimCLR (Chen al., 2020a). the same setting we used in our main experiments. Overall, the additional experimental results are well-aligned with our main results. We observe that BADGE is the best performing baseline. However, we note that BADGE is not linearly scalable, and it requires more computational costs. Figure 19 and Table 6 show full results.
120
OwEntocc squasitoon size 25
Naridum aquisiion siz 23
Sultntuu equis ton size
BALD souisitinn size 25
Random aquisition size 25
Entropy aquisition slz;
SALD aquis size 25
NeanSD equlsttion
Entropy aquis
squsition
aquistion
eculsition
BADGE squisition 25
PowerBALD aqusition
BatchBALD aqurstioon size 25
BADGE aqulsitio size 25
CoreSet aquisi on size
CoreSit aquistion sue
(a) MXISS
(b) CIFAR-
Figure 19: Active learning curves of smaller batch size with 3XMISST and 3x CIFAR-10.
Table 6: Selected accuracy table. Mean and standard deviation are from repeated experiments.
Scenario
Full dropouts CNN
Fixed feature
Dataset/Acq. Size/Test size
3xMMNIST/55/00000
3x CCIFAR--0/2//00.000
Thain Size/Pool Size
100/180,000
200/180,000
300/180,000
200/150,000
300/150,000
500/150,000
Random
83.0 + 2.5%
89.1 + 2.4%
91.9 + 1.1%
61.3 2.3%%
64.9 + 0.6%
68.1 0.7%
BALD
76.5 6.0%
83.8 2.8%
85,9 +: 5.5%
49.8 0.8%
53.7 : 1.7%
57.7 17.7.1.999 1.1%
Entropy
3.5%
81.5 2.8%
89.0 1.0%
48.5
3.7%
59.1 0.7%
McanSD
79.1
88.3
93.5 + 1.2%
55.0 0.7%
58.3 0.8%%
63.0 0.6%
Variation Ratios
80.3 1.3%
91.1 0.5%%
92.9 + 1.0%
53.0 2.7%
55.0 1.0%
61.8 + 1.6%
PowerBALD
85.6
98.2 0.5%
0.1%
+: 0.5%
0.8%
69.3 0.2%
BADGE (not-sculable
89.0 +: 0.7%
94.2 0.4%
95.8 + 0.3%
63.2 U.3%
66.8 0.8%
70.3 0.7%
BatchBALD (not- scalable)
2.44%
93.3 1.2%
95.4 0.2%
CoreSet (not (not-scalable)
20.0
0.5% 86.4
89.5 + 1.2%
57.8 - 0.9%
60.5 + 1.4%
65.2 - 0.3%
BalEntAgg (ours)
86.3 = 2.0%
92.8 : 0.5%
95.7 = 0.2%
62.7 1.6%
65.9 1.4%
70.1 0.5%
32

---

Published as a conference paper at ICLR 2023
A.17 COMPARISON WITH COREMSE
Recently, the Bayesian active learning framework considering the Expected Loss Reduction (ELR)
for the optimal Bayes classifier has been proposed (Zhao al. 2021 Tan al., 202T). Under this framework, they attempt to optimize the loss reduction in a holistical way, accounting for average loss reduction from all points. However, this non- con-arametric approach requires a very expensive computational cost. With a large dataset size, ELR (Zhao et al. 202TD, wMOCU (Zhao et al. 2021 CoreLog (Tan et al., 202T), and CoreMSE (Tan et al. (021) require vast memory size unless we apply size reductions on the data space and MC samples (Tan al. 202I). If the number of classes is large, running the algorithm in practice impossible. Therefore the naive application of the ELR-basee algorithm is not scalable. Moreover, both works have pitfalls in the convergence proof by assuming the finite data and parameter space. Both pieces of the works end up with null proof. Nevertheless, tested the performance of CoreMSE (Tan et al. (02T)) with MNIST, seemingly the best method under this framework. Figure 20 and Table 7 show the full active learning results.
335
0.5
BalEntAcq aquisition size 1
Random aquisition size 1
BALD aquisition size
CoreMSE aquisition size
Varlation Ratios aquisition size
Figure 20: Active learning curves with CoreMSE in MNIST
Table Selected accuracy table. Mean and standard deviation are from repeated experiments.
Scenario
Full dropouts CNN
Dataset/Acq. Size/Test size
MNIIT/1/110000
Train Size/Pool Size
50/60,000
100/60,000
300/60,000
Random
7
86.4 2.7%
0.76%
BALD
82.6 1.3%
90.5 0.8%
95.3
Variation Ratios
3.2%
90.0 1.2%
9.2%%
CoreMSE
86.5 + 0.8%
91.3 0.3%
96.4 0.3%
BalEntAcq (ours)
85.4 + 1.0%
91.4 = 1.3%
96.5 0.1%
A.18 ACQUISITION TIME AND SPACE COMPLEXITY
In this section, we discuss the time and space complexity the acquisition calculation for each active learning iteration. We denote by N number of unlabelled points, C number of classes, K the acquisition size. For BADGE, we use the last layer feature vector and then apply k-means+ initialization (Vassilvitski & Arthur 2007).
The main computational bottleneck of BADGE is the dependency of K in the calculation of multi- acquisition. Here, the K-iteration cannot be parallelized since the pairwise distance calculation depends on the previous selection of centers in +-means+++. On the other hand, the main computa- tional bottleneck of CoreMSE is the dependency of N in both time and space complexity. Since the
33

---

Published as a conference paper at ICLR 2023
Table 8: First two rows show the theoretical time and space complexity. Remaining rows present the
average calculation time what we observed in our experiments.
Method
BalEmtAcy
Entropy
PowerBALD
BADGE ((CKK) O (C N)
CoroME
Random O(1)
Complexity
O(CN) OCC + K;
O(CN) OCC K)
OCCM) oCC + K)
O1CN) oC K)
O1CN) acc + K)
(CC O(C"N")
Space Complexity
Case
Averae Flequed (soc)
MNIST with Acq. size
7.1
6.9
14.3 insufficient memory
0.1
CIEAR: 100 with Aug, size 500
10.2
9.4
9.5
9.8
9.8
302.5
0.1
3.CIFAR-II with
size 500
18.0
18.5
18.4
18.2
18.4
1227 4
insullicient memory
0.3
SVHN with Acq, size 2300
20.7
19.7
19.4
20.3
85.4
insufiiient
TinylmageNet Acq. size
momooy
original CoreMSE requires calculating the entire correlational effect on the loss from all data points, cannot linearly scale up this computation. Even if we try relax the memory issue by fixing the number of points of interest for correlation instead of the entire unlabelled dataset, we still experience memory deficiency.
The following Figure 21 confirms the dependency the acquisition time of our BalEntAcq and BADGE concerning the unlabelled dataset size and the acquisition size by augmenting the number of images from CIFAR-10 and CIFAR-100, Our BalEntAcq does not rely on the acquisition size, but BADGE is still linearly proportional to the acquisition size. For the unlabelled dataset size, both methods linearly increase the acquisition time, but the slope of BADGE is much steep.
aquisition
BeitntAoe equisiiin size 300 BADGE aquisiiin sibe 500
BADGE acuistion size 500
20-
Bo EntAco with untabelied led dataset site 50, 000
BolEntAce with unlabelics ed dataset size SC,000
BADGE uniabelledd
BADGE unlabelled
Accuation bar IMC
(a) Augmented CIFAR-10 (C 10)
(b) Augmented CIFAR-100 (C 100)
Figure 21: Acquisition time under different unlabelled dataset sizes (first row) and under different acquisition sizes (second row).
A.19 EXPERIMENTS UNDER HEAVILY REDUNDANT DATA SCENARIO
In this section, we observe the active learning performance behavior to various redundancy levels with Tx CIFAR-1100 dataset where (1,3,5, 7, 10, 20, 50).
34

---

Published as a conference paper at ICLR 2023
300
CIFAR-100 100 BaiEntArq aquisition size 500
0.45
3xCIFAR-100 BalEntAcq aquisition size 500
5XCIFAR-100 BalEntAcq aquisition size 500
7xCIFAR-100 BalEntAcq aquisition size 500
0.4
10xCIFAR 100 RalEntåcq aquisition size 500
10xCIIARR-10 BalEntAcq aquisition size 500
3.0
SOXCIFAR-100 BalEntAcQ aquisition size 500
10W 3090
20999 2460
3000
Figure 22: Active learning curves under various redundancy level scenarios
Active learning performance with BalEntAcq shows the best accuracy when no redundant images exist. With CIXCI 100, our BalEntAcq requires only up to 1.25% data points among 2.5 million images achieve the fully supervised accuracy (= 66%). We do not observe any significant performance deterioration compared to the increment of the pool size, even highly redundant image scenarios. It demonstrates the possibility of exponential label savings discussed in Section A.15
Refer to the following Table the specific accuracy values.
Table 9: Selected accuracy table for various redundancy levels. Mean and standard deviation are from 3 repeated experiments.
Scenario
Fixed feature + heavily redundant data scenario
Acq. Size/Test size
500010,000
Train Size
5,000
10,000
15,000
20,000
25,000
30,000
CIFAR- 100/50,000
57.2
61.5
0.2%
64.1 = 0.1%
65.9
0.4%
67.3 0.3....44%
0.2%
3XCIFAR 100/150,000
56.9 0.6%
61.0 0.5%
63.5 + 0.5%
65.4
0.4%
66.7 + 0.3%
67.4 0.14= 0.1%%
5XCIFAR CCCIFAART00/220.000
56.3 + 0.9%
61.4++0...
62.8 0.1%%
65.1 +: 0.6%
0.1%
66.8 = 0.4%
7XCIFAR 100/350,000
0.9%
61.0 0.5%
6..8
64.6 0.5%
65.9 0.35%
66.2 0.4%
10xCIFAR-100/50A 1-1000000000000
56.3 0.4%
0.2%
64.4
66.0 0.4%
0.2%
20xCIFAR-100//.0 100/1,000.000
55.7 0.8%
0.79
0..2%
64.60.88%
6
66.1 0.6%
50x 50CCIFAR-
CIFAR--IOOO D0AACIA-O0002,00000000 10002,500,00
56.5
61.41 0.3%
63.4
64.6 2: 0.2%
65.7 0.5%
66.8 0.25
A.20 DISCUSSION ABOUT BALENTACQ FORM
In this section, we discuss further the underlying motivation of the ratio form of our BalEnt[x]:
MJEnt[x] Σ. (EP.) H(Y)
BalEnt[x]
Ent[x] + log
H(Y)+ log
We design BallEn[[x] to be generalized probability a ratio between the marginalized joint entropy [JEnt[x] [x] and the augmented Shannon entropy HY) (Y)++Yme +oog 2. As shown in Appendix A.5 for the proof generalized estimation error probability
of Theorem 4.1] BalEnt[[] has natural interpretation leveraging Fano's inequality. i.c., -o < BalEnt[x]] < 1. The rigorous interpretation of the negative probability is an open problem. Still, we can intuitively understand that the negative probability implies that the model has sufficient knowledge to estimate the value of P:+ given the information of up to the specified precision level. suggests that a phase transition the model's current knowledge occurs near the zero probability point of BalEnt[x]]. It is because H HYY is the minimum amount of nats (or bits) to encode the information of Y, which is the fundamental information limit when observing labels, and require additional 2 nats given choice of the precision level. That's 's why we call the quantity BalEnt to be a balanced entropy that captures the information balance between the model and the label.
35

---

Published as a conference paper at ICLR 2023
Moreover, although empirically demonstrate the choice of the precision level H() + log to match MJEnt[x] in Appendix |..1333| MJEnt[x] itself is a result of the derivation by applying Jensen's 's inequality shown in Appendix A.2 In other words. MJEnt(xx] is the maximally achievable joint entropy between + (x, w) and ((,,), w) So we may expect that there should exist an entropy- increasing physical dynamical process, something similar to diffusion process (Oksendal 2013), from the state ( w), Y (x, w)) to another maximally marginalized state S' which has joint entropy MJEn((x). Because of our limited understanding of the point process entropy, we have yet to know the full specification of the existence of these physical dynamics.
Lastly, because our generalized probability interpretation of BalEnt[x] mainly relies on the sign of BalEnt(x), one might wonder if the denominator H (Y) + log 2 of BalEnt[x] is critical factor. So we empirically demonstrate the necessity of the denominator H(Y) log
We define an acquisition function MJEntAcq(xx from MJEnt[x] without the denominator of BalEnt[x]]
[J]En[x]] -1 if MJ]nt(( 2 0,
MJEntAcq[x]
[J]En[[] if MJEnt((] 0,
Figure 23 and Table show the active learning performances on CIFAR-100 and 3XCIFAR-100 We observe that MJEntAcq(xi performs well to reach the fully supervised accuracy. However, BalEntAcq(x] performs better than MJEEnAAc((x]. It suggests that the slight distortion of MJJnt[x] to BalEnt[x x could help improve active learning performance.
BalEntAcq MiEntAcc aquisition 500
BallEntAco aquisition
M|EntAcq aquistion
(a) CIFAR-100
(b) 3XCIFAR-100 100
Figure 23: Active learning curve comparison without denominator of BalEnt[x]
Table 10: Selected accuracy table for various redundancy levels. Mean and standard deviation are from 3 repeated experiments
Scenario
Fixed Feature different acquisition Form
Datasct/Acq. Size/Tost size
CIFAR-10W50W10.000
Train Pize/Pool Size
5,000500,00
10,0005500000
15.00050.000
20,000/50.000
25,000050.000
30.000:50,000
BalFntAcq MJEntAng
57.2
0.2%
61.5 +. 0.2%
64.1 + 0.1
65.9 0.4%
67.3 0.45
68.3 0.2%
0.2%
66.5 0.2%
67.0
Dataset/Acq, Size/Test size
3x M0LLAAR--00000010000
Train Size/Pool Size
5,000/1500000
10.000/150.000
50,000
20.000/1500000
25,000/150,000
30,000/150.000
BalEniAAy MJEntArq
56.9
0.6%
61.0 L 0.5%
63.5 0.5%
65.5 4 0.4%
66.7 0.3%
1..1%
0.39%
0..2%
0.1%
65.9 0.2%
0.2%
A.21 APPLICATION TO BAYESIAN NEURAL NETWORK WITH VARIATIONAL DROPOUTS
In this section, report our active learning experiment when train Bayesian neural network with variational dropouts (Kingma et al.) 2015) with 3x CIFAR- 10 with acquisition size 50 and 3xCIFAR-10 with acquisition size 500 under fixed feature scenario.
36

---

Published as a conference paper at ICLR 2023
We use an Adam optimizer with learning rate of 0.0003 and 500 epochs in each experiment. Compared to MC-dropout Bayesian neural network models, we observe that the convergence with variational dropouts is not stable, so it requires much longer epochs if we newly train the model at each active learning iteration for both cases. Therefore, we continue to train the model from the previously trained model at each iteration except the initial iteration that the convergence can be more stable.
Here the architecture used for our 3 CIFARR-OO experiment, and for the 3xCIFAR-10 case, can simply modify the out feature size to be 100.
(classifier): Sequential
(0) : VariationalDropour si__featuzee=2048, out jut_featture=10241
(1) : VariationalDropon (in_ fir_eeerrrsss112444 out_featuurs=1024
(2) :
out fut_features=10, bias=False,
We observe a similar result from the MC-dropout Bayesian neural networks. Our BalEntAcq consistently outperforms other linearly scalable baselines and is eventually on par with BADGE in XXCIFAR-10 10 and approaching close to BADGE in 3 CIFAR-100.
SalEntAce aquisition
Enticq
Random arpuiiiion xow
Bacdfmm equis 500
BALD squlstion dize 30
BALD seusitionn size 500
Entropy equis ton size 50
Entrasy stutstiinn size 500
MeandD aquisiion se 50
MeanSD aquis tion size 500
Powerffll apusition
PowerttAl D aesiiiiinn son
SADGE ecuisition size
300
BADGE aquisttin 500
(a) 3x CIFAR--10
(b) 3XCIRRR-100
Figure 24: Active learning curves with variational dropouts
Table 11: Selected accuracy table. Mcan and standard deviation are from 3 repeated experiments.
Scenario
Fixed feature variational dropouts
Dataset/Acq, Size/Test
3xCIFAR- CCCIAAR-1//0/10.000 10/50/10,000
Train Size/Pool Size
50,015
750/150,000
1000/150.000
50001100,000
15,000/150,000
30,000/150,000
67.0 0.5%
69.9
0.4%
71.2 0.7%
0.5%%
0.2%
60.6
BALD
62.1 1.5%
66.4 0.8
69.5 = 0.8%
52.3 + 0.1%
56.3 +: 0.2%
59.3 0.3%
Entropy
60.8 = 1.7%
1.3% 61.9
66.5 1.9%
46.0 1.6%
54.1 1.0%
58.8 0.4%
McanSD
63.5 0.6%
68.0 0.9%
70.3 0.9%
51.6 0.2%
56.0 0.3%
59.4 0.7%
PowerBALD
68.2 1 1.3%
70.7 0. 1%
72.6 0.3%
52.7 + 0.3%
58.7
0.1%
61.7 2 0..%%
BADGE (not-scalable
0.1%%
72.0
0.1%
0.5%
0.1%
63.7 0.3%
BallEntAcy (ours)
68.2 0.8%
71.9 0.6%
73.5 0.2%
55.2 0.5%
59.9 + 0.9%
63.2 0.2%
A.22 APPLICATION TO GAUSSIAN PROCESS
In this section, we report the active learning result with 3xCIFAR-10 when we train the model with the exact Gaussian process which could be of another independent interest.
We use an Adam optimizer with learning rate of 0.1 and 5, 000 iterations in each experiment. We the exact Gaussian process. We observe that the exact Gaussian process requires much more
37

---

Published as a conference paper at ICLR 2023
samples compared to the Bayesian neural network, but the observed behavior of each method is the same as before.
3.01
1.2
1.0
1.13
800000
BelEniAcq equisition
6.4
Random squisition size 500
aquisition
Entropy aquisition size 500
672
MeanSD aculsition size 500
PowerBALD aquisition size 500
4.9
Varistion Ratios aquisition size 500
BADGE aquisition size 500
340
2309
600
Figure 25: Active learning curves with exact Gaussian process
Table 12: Selected accuracy table. Mean and standard deviation are from 3 repeated experiments.
Scenario
Fixed feature + variational dropouts
Dataset/Acq. Size/Test size
3XCIFAR-10050011.000
Train Size/Pool Size
5,000/150,000
7,500/150,000
10.000/150,000
Random
79.4 + 0.2%
82.0 + 0.3%
BALD
79.4 + 0.1%
8.14
82.5 0.2%
Entropy
76.2 + 0.2%
78.1:0.2%%
+ +
MeanSD
79.6 0.3%
8.32%
$
Variation Ratio
77.1 + 0.4%
79.3
80.6 + 0.2%
PowerBALD
80.2 0.2%
81.8+0.2%%
82.9 + 0.2%
BADGE (not-scalable)
81.4 + 0.2%
83.0 + 0.2%
84.1 + 0.1%
BalEntAcq (ours)
80.4 + 0.5%
82.5 + 0.5%
83.9 0.2%
A.23 WEAK SIMILARITY WITH RISK PARITY PORTFOLIO OPTIMIZATION
This section is moderately informal and a little digression, but we think it is worth discussing. An active learning problem shares weak similarity with portfolio optimization problem (Markowitz) 1952). In portfolio optimization, we aim to minimize uncertainty but maximize profits under correlated environments. Traditional mean-variance portfolio optimization well-known, and the primary objective function is minimizing the uncertainty (=risk), but the selection needs to be well-diversified (Markowitz, 1952). In active learning, we typically aim maximize the uncertainty, but diversify the selection under correlated data points, e.g., BADGE (Ash et al. 2020
In the modern dynamic financial system, the traditional way of optimization could still be exposed to concentration risk (Levy & Zhang] 2019). To mitigate the concentration risk, equalizing the risk contribution, a.k.a. risk parity strategy, for each factor has been popularized after the financial crisis (Prince 2011 Hurst et al. 2010] Chaves et al. 2011 Qian 2011 Asness et al.) 2012 Costa & Kwon 2020). The critical idea risk parity portfolio optimization is to find a good balance between different factors. If we closely look at the risk parity optimization equation (Costa & Kwon 2020 e.g., See the equation (3)), the entropy-like constraint plays a critical role. Then we can balance the risk of each factor. Although entropy acquisition and the risk parity strategy, both goals are similar. So it would be interesting to the close relationship between our balanced entropy and risk parity strategy.
cannot clearly state tight relationship between our balanced
38

---

Published as a conference paper at ICLR 2023
A.24 MODEL ARCHITECTURES FOR OUR EXPERIMENTS
In this section, we describe model architectures what we have used in our experiments.
Toy example moon dataset
BNN
Iclassifiens Sequential
(0) Linear (inear(in_fattures;) out_features=72, bias=True)
(1): ReLu
(2) Linear(in_features==2
fut_feeaures=72, bias-True) (3) (po inplace=False)
(4) ReLJU(nnpacc==frue)
(5) Limaar(a__faature==772 out fet_ueaaue,-=7, bias=True)
(6) (po00ut(p=0.2, inplace=False)
(7): : (eplaaaiip)e
(8) Linear (in features=72, out featurees,, bias=False)
MNIST
CNNENN
(features): Sequential
CNN2D (iN2 out_chanmel=32, kernel_size=5,
stride=1, dropout_p=0.5, apply_max_pol==rrue
apply_relu=Irue)
(1): CNN2D (in_channel=32, out_channel=64, kernel_size=5, stride=l, dropout_p=O.5, apply_man pot___xxxoollruee apply_relu=Irue)
(classi : Sequential
(0): Linear (ia_feetttteuuess11022 fut_features=128, bias=True)
(1): Dropout inplaceFFasse
(2): ReLU Tru)l
(3): Linear (in features=128, fut_feetures=10, bias=False)
SVHN
RESNETENN (
(features): ResNet18 (remove_last__fulle hutpoeaeeeaaaaaaauoooooottttttte
(classifier) Sequential
(0): (inffetrrnna out_features=512, bias=True)
(1): propout inplace=Palse)
(2): ReLUiinpiace-Trus
(3): Linear((n_ out fet_feeture==10, bias=False)
CIFAR-100 and 3x 3ICARR-1000
RESNETCIASSIFFER
(classifier) Sequential
(0): (inaartin_fattuues2200 out__features=2048, bias=True)
(1): propout(p=0.2. inplace=False)
(2): Renplacne
(3): Linear (in features=2048, fut_features=100, bias-False)
39

---

Published as a conference paper at ICLR 2023
TinylmageNet
RESNETENN
(features): ResNet
(classifier): (
(0): Linear (1n_eeetrtte,,,, out_features=2048, bias=True)
(1): Dropout (p=0.2, inplace=False)
(2): ReLU (irpaceeae)))
(3): Linear fet_fffatures=2003 out_featuues-200, bias-False)
40


Reviews:
[{"review_id": "VKzXTw-Q0ut", "reviewer": "Reviewer_MCdP", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new acquisition function Balanced Entropy Acquisition (BalEntAcq) for active learning, which captures the information balance between the uncertainty of underlying softmax probability and the label distribution. The new acquisition is designed to be computed without relying on other data points, which allows parallelization in computation. With the Beta distribution approximation of the marginal distribution, the authors derive a closed-form expression of the proposed acquisition. The authors run experiments and show the efficacy of the proposed method.", "summary_of_the_review": "The proposed acquisition function is theoretically motivated and allows parallelization in computation; empirical results also confirm the efficacy of the proposed method. Overall, I am inclined to accept this paper. However, I believe this paper can be made stronger if the authors can address the weaknesses mentioned above. \n\n====after rebuttal====\n\nI would like to thank the authors for their response. I have read the response and the comments from other reviewers, and I would like to remain my scores.", "strength_and_weaknesses": "Strength:\n1. This paper proposes a theoretically-motivated (Thm 4.1) acquisition function BalEntAcq that can select a diverse set of data points near the decision boundary (at least in the toy example studied in Section 4.2). \n2. The acquisition function can be computed without relying on other data points, which allows parallelization in sampling/computation.\n3. Empirical results show the efficacy of the proposed acquisition function.\n\nWeaknesses:\n1. It seems that the calculation of BalEnt relies on the Beta distribution approximation of the marginal probability. How difficult is it to calculate the acquisition function without the Beta distribution approximation?\n2. It would be great if the authors can add further explanations for the choice of BalEnt[x]^{-1} in BalEntAcq. Currently, it seems to me that such a choice is mainly motivated by the empirical results shown in Appendix A.13.2.\n3. Since one selling point of the proposed acquisition function is that it can be computed without relying on other data points, many existing methods (e.g., BADGE) do. I think the authors should add a discussion on the computational gain of the proposed method (e.g., what is the computational complexity of your proposed method, and what is the computational complexity of BADGE).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "OAfeMmA63Ep", "reviewer": "Reviewer_gjo6", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper tackles the problem of selecting the batches from unlabelled datasets to be labelled that are the most informative for a classifier trained on the data. This problem is traditionally studied within the paradigm of active learning, where the data to be labelled is collected sequentially in batches. The batches are selected using some notion of information gain to the current classifier. The paper presents a novel acquisition function for batch selection in active learning. The authors first examine the standard Bayesian Neural network paradigm used in active learning, and study the behavior of existing acquisition functions like BALD. The authors then outline the Balanced Entropy acquisition function, which is the ratio between the marginalized joint entropy and the entropy (plus a constant). The authors then study the behavior of this acquisition function theoretically as well as through a toy experiment. Finally the authors present an empirical study of their approach on large realistic datasets. ", "summary_of_the_review": "In summary, the paper presents an novel acquisition function for active learning resulting in significant improvements on realistic large datasets. The paper is well written and accompanied with thorough analysis. The motivation for the exact form of the acquisition function is somewhat lacking and would be useful to add. I lean towards acceptance. ", "strength_and_weaknesses": "**Strengths**\n\n- The paper is very clearly written and easy to follow. The systematic analysis of Bayesian NNs and existing approaches in active learning approaches is quite insightful.\n- The proposed acquisition function is conceptually simple and can be easily adopted in practice. The toy experiments are also quite helpful in illustrating the effect of the acquisition function - selection of _diverse_ and _informative_ points.\n- The acquisition function also enjoys strong empirical performance in realistic large datasets, performing on par with or better than more sophisticated and expensive-to-evaluate acquisition functions. The experimental analysis is also quite thorough and provides useful insights. \n\n**Weaknesses**\n\n- I believe the key weakness of the approach to me is that the acquisition function is stated directly in a form without much motivation. It would be nice to understand how the particular ratio comes and the motivations behind said choice. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "FDMRkG0VapC", "reviewer": "Reviewer_VNzQ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a novel acquisition function to improve active learning performance in Bayesian neural networks.\nThe acquisition function, referred to as Balanced Entropy Acquisition (BalEntAcq), serves as an uncertainty measure that aims to capture the information balance (or lack thereof) between the uncertainty of the underlying softmax probability computed by the Bayesian neural network and the label to be predicted.\n", "summary_of_the_review": "This paper presents a new uncertainty measure, BalEntAcq, that has the potential to improve AL performance in a variety of settings while still being scalable (e.g., compared to other existing Bayesian schemes).\nThe authors support their claims based on extensive performance evaluation results using widely used benchmarks, which demonstrate the potential advantages of the proposed AL acquisition function.\n\n---------\n\nThe scores have been updated after reviewing the authors' clarifications to the review comments.\n", "strength_and_weaknesses": "Strength:\n\nThrough extensive simulations and comparison with some of the existing popular active learning (AL) schemes, the paper demonstrates the potential benefits of the proposed uncertainty measure, BalEntAcq.\nEspecially, BalEntAcq has been shown to lead to the acquisition of diversified labels for data points in the pool that are well spread along the decision boundaries, thereby resulting in more efficient model enhancement through label acquisition.\nFurthermore, the computational cost of the proposed strategy seems to be fairly low compared to many AL strategies based on Bayesian experimental design (BED), thanks to the approximation of the marginal distributions based on Beta distributions.\nThis makes the proposed AL method possibly much more scalable compared to the BED, which is an important potential merit of the proposed scheme. \nOverall, the authors have shown that AL via BalEntAcq generally leads to performance improvement over other popular schemes based on several widely used benchmarks.\n\nWeakness:\n\nWhile scalability is one of the main advantages of the proposed method, direct performance assessment results demonstrating the scalability of BalEntAcq appears to be missing in the current study.\nIt would be interesting and important to see the comparison between the proposed method compared to other schemes, in terms of computational cost and scalability, especially with methods that the paper mentions to be too costly or not scalable (e.g., BADGE, CoreMSE, and other BED based schemes).\nFurthermore, it would be helpful to provide experimental support for \"linear scalability\" and \"exponential savings\" that can be attained via AL using BalEntAcq as claimed in the current study.\n\nThe implications of the sign of BalEnt, the proximity of BalEnt to 0, and its relevance to the location of the corresponding data point with respect to the decision boundaries could (and should) be better explained, considering their importance in how BalEntAcq leads to enhancing the AL performance.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%