Black-Box Batch Active Learning for Regression
Authors: Andreas Kirsch
Venue: Accepted by TMLR ()

Abstract: Batch active learning is a popular approach for efficiently training machine learning models on large, initially unlabelled datasets by repeatedly acquiring labels for batches of data points. However, many recent batch active learning methods are white-box approaches and are often limited to differentiable parametric models: they score unlabeled points using acquisition functions based on model embeddings or first- and second-order derivatives. In this paper, we propose black-box batch active learning for regression tasks as an extension of white-box approaches. Crucially, our method only relies on model predictions. This approach is compatible with a wide range of machine learning models, including regular and Bayesian deep learning models and non-differentiable models such as random forests. It is rooted in Bayesian principles and utilizes recent kernel-based approaches. This allows us to extend a wide range of existing state-of-the-art white-box batch active learning methods (BADGE, BAIT, LCMD) to black-box models. We demonstrate the effectiveness of our approach through extensive experimental evaluations on regression datasets, achieving surprisingly strong performance compared to white-box approaches for deep learning models.

Published in Transactions on Machine Learning Research (7/2022)
Black-Box Batch Active Learning for Regression
Andreas Kirsch OATML, Department of Computer Science University of Oxford
andreas, andreas.kirschh@en wwwschhcss.or.cco
Reviewed on OpenReview: /ttp:r//opperevvieeee net/ form?? id= fuBuDDKkoo
Abstract
Batch active learning is a popular approach for efficiently training machine learning models on large, initially unlabelled datasets by repeatedly acquiring labels for batches of data points. However, many recent batch active learning methods white-boz approaches and are often limited to differentiable parametric models: they score unlabeled points using acquisition functions based on model embeddings or first- and second-order derivatives. In this paper, we propose black-bor batch active learning for regression tasks as an extension of white-box approaches. Crucially, our method only relies on model predictions. This approach compatible with a wide range machine learning models including regular and Bayesian deep learning models and non-differentiable models such as random forests. Tt is rooted in Bayesian principles and utilizes recent kernel-based approaches. This allows us to extend wide range of existing slate-of-the-ar while-box batch active learning methods (BADGE, BAIT, LCMD) to black-box models. We demonstrate the effectiveness of our approach through extensive experimental evaluations on regression datasets, achioving surprisingly strong performance compared to white-box approaches for deep learning models.
1 Introduction
By selectively acquiring labels for subset of available unlabelled data, active learning (Cohn al., 1994) is suited for situations where the acquisition of labels is costly time-consuming, such as in medical imaging natural language processing. However, in deep learning, many recent batch active learning methods have focused on white-bor approaches that rely on the model being parametric and differentiable, using first or sccond-order derivatives (c.g. model embeddings)*.
This can present a limitation in real-world scenarios where model internals or gradients might be expensive to access or might not be accessible at all. This is particularly of concern for 'foundation models' (Bommasani et al., 2021) and large language models such as GPT-3 (Brown al., 2020), for example, when accessed via a third-party API. More generally, lack of differentiability might hinder application of white-box batch active learning approaches to non-differentiable models.
To address these limitations, we examine black-bor box batch active learning (BSAL) for regression which is compatible with a wider range of machine learning models. By black-bor, we mean that our approach only relies on model predictions and does not require access to model internals or gradients. Our approach is rooted Bayesian principles and only requires model predictions from a small (bootstrapped) ensemble. Specifically, we utilize an (empirical) predictive covariance kernel based on sampled predictions. We show that the well-known gradient kernel (Kothawade et al., 2021; 2022; Ash et al., 2019; 2021) can be approximation of this predictive covariance kernel.
The proposed approach extends to non-differentiable models through Bayesian view on the hypothesis space formulation of active learning, based on the idcas behind moory-by-committe (Scung et al., 1992). This
Model embeddings can also be first-order derivatives of the score under regression in regard to the last layer. 2The code available https: //ggt//ggttt cowp/Blaccct//2220009955.

---

Published in Transactions on Machine Learning Research (7/2022)
enables us to use batch active learning methods, such BAIT (Ash et al., 2021) and BADGE (Ash et al., 2019) in a black-box setting with non-differentiable models, such as random forests or gradient-boosted trees.
By leveraging the strengths of kernel-based methods and Bayesian principles, our approach improves the labeling efficiency of a range of differentiable and non-differentiable machine-learning models with surprisingly strong performance. Specifically, we evaluate black-box batch active learning on a diverse set of regression datasets. We focus on regression because classification using the same approach would require Laplace approximations, Monte Carlo sampling, expectation propagation (Williams & Rasmussen, 2006; Hernández- Lobato et al., 2011). This would complicate a fair comparison between black-box and white-box approaches. For the same reason, we also do not consider proxy-based active learning methods (Coleman et al., 2019), which constitutes an orthogonal direction to our investigation. Unlike the above mentioned while-boz parametric active learning methods which scale in the number of (last-layer) model parameters or the embedding size, our method scales in the number of drawn predictions, and we find that we can already obtaín excellent. results with small number of them. Our results demonstrate the label efficiency of B for various machine learning models. For deep learning models, B3AL even performs better than the corresponding state-of-the-art. white-box methods in many cases.
The rest of the paper is organized as follows: in $2, we discuss related work in active learning and kernel-based methods. In $3, we describe the relevant. background and provide a detailed description of BSAL In $4, we detail the experimental setup and provide the results of our experimental evaluation. Finally, $5 concludes with a discussion and directions for future research.
2 Related Work
Active learning has a rich history, with its origins dating back to seminal works such as Cohn et al. (1994); Lindley (1956); Fedorov (1972); MacKay (1992). A comprehensive survey of early active learning methods can be found in Settles (2009), while more recent surveys of contemporary deep learning methods can be found in Ren et al. (2021) and Zhan et al. (2022). Our work focuses on pool-based batch active learning, which involves utilizing a pool set of unlabeled data and acquiring labels for batches of points rather than individual points at a time. The main challenge in pool-basel batch active learning is the choice of the acquisition function.
Differentiable Models. Many acquisition functions approximate well-known information-theorete quantities (MacKay, 1992), often by approximating Fisher information implicitly explicitly (Kirsch & Gal, 2022). This can be computationally expensive, particularly in deep learning where the number of model parameters can be large- even when using last-layer approximations or assuming generalized linear model. BADGE (Ash ct al., 2019) and BAIT (Ash et al., 2021) approximate the Fisher information using last-layer loss gradients or the Hessian, respectively, but still have a computational cost scaling with the number of last layer weights. This also applies to methods using similarity matrices (kernels) based on loss gradients of last-layer weights such as SIMILAR (Kothawade et al., 2022) and PRISM (Kothawade et al., 2022), to only name few. Importantly, all of these approaches require differentiable models.
(QbC, Seung et al. (1992)) measures the disagreement between different model instances to identify informative samples and has been applied to regression (Krogh & Vedelsby, 1994; Burbidge et al., 2007). Kee et al. (2018) extend QbC to the batch setting with a diversity term based on the distance of data points in input space. Nguyen et al. (2012) show batch active learning for random forests. They train an ensemble of random forests and evaluate the joint entropy of the predictions of the ensemble for batch active learning, which can be seen as a special case of BatchBALD in regression.
Non- Donn-ifferentablle Models. Query-by-commite
BALD. Our work most closely aligns with the BALD-family of Bayesian active learning acquisition functions (Houlsby et al., 2011), which focus on classification tasks, however. The crucial insight of BALD is applying the symmetry of mutual information to compute the expected information gain in prediction space instead of in parameter space. As a result, BALD is a black-box technique that only leverages model predictions. The extension of BALD to deep learning and multi-class classification tasks using Monto-Carlo dropout models is presented in Gal et al. (2017), which employs batch active learning by selecting the top-B scoring samples from the pool set in each acquisition round. BatchBALD (Kirsch et al., 2019) builds upon BALD
2

---

Published in Transactions on Machine Learning Research (7/2022)
by introducing a principled approach for batch active learning and consistent MC dropout to account for correlations between predictions. The estimators utilized by Gal et al. (2017) and Kirsch et al. (2019) enumerate over all classes, leading to a trade-off between combinatorial explosion and Monte-Carlo sampling, which can result in degraded quality estimates as acquisition batch sizes increase. Houlsby et al. (2011), Gal et al. (2017), and Kirsch et al. (2019) have not applied BALD to regression tasks. More recently, Jesson et al. (2021) investigate active learning for regressing causal treatment effects and adopt a sampling distribution based on undividuul-acqqisittoon scores, a heuristic proposed in Kirsch et al. (2021).
Kernel-Based Methods. Holzmillerr et al. (2022) examine the previously mentioned methods and unify them using gradient-based kernels. Specifically, they express BALD (Houlsby et al., 2011), BatchBALD (Kirsch et al., 2019), BAIT (Ash et al., 2021), BADGE (Ash et al., 2019), ACS-FW (Pinsler et al., 2019), and Core-Set (Sener & Savarese, 2017)/FF- for regression tasks. They also propose new method, LCMD (largest cluster maximum distance).
(Geifman & El- Yanva, 2017) using kernel- based methods
Our Contribution. We exlend the work of Houlsby et al. (2011) and Holzmüller et al. (2022) by combining the prediction-based approach with kernel-based formulation. This trivally enables batch active learning regression tasks using black-box predictions for wide range of existing batch active learning methods.
3 Methodology
Here, we describe the problem setting and motivate the proposed method.
Problem Setting. Our proposed method is inspired by the BAmD-family of active learning frameworks (Houlsby et al., 2011) and its extension to batch active learning (Kirsch al., 2019). In our derviation, we make use of a Bayesian model in the narrow sense that we require some stochastic parameters Z the model parameters or bootstrapped training data in the case of non-differentiable models like random forests, for example with distribution" p(w) (Bickford Smith et al., 2023):
ply.w [ x) ply xx xx,) p(w).
(1)
Bayesian model averaging (BMA) is performed by marginalizing over p(w) to obtain the predictive distribution p(y x). We can use Bayesian inference to obtain a posterior p(w | (D) for additional data D via:
p(w D) x p(D w) p(w),
(2)
where p(D w) is the likelihood of the data given the parameters Z and p(w (D) is the (new) posterior distribution over Ω. Importantly, the choice of p(w) covers ensembles (Breiman, 1996; Lakshminarayanan et al., 2017) as well as models with additional stochastic inputs (Osband et al., 2021) or randomized training data by subsampling of the training set, e.g., bagging (Breiman, 1996).
Pool-based Active Learning assumes access to a pool set of unlabeled data Dpool = (x("0)) and a small initially labeled training set Dirain = {xxtrin y, or Dirain = In the batch acquisition setting, we want to repeatedly acquire labels for a subset {x, } of the pool set of a given acquisition batch size and add them to the training set Dirain, Ideally, we want to select samples that are highly 'informative' for the model. For example, these could be samples that are likely to be misclassified or have large prediction uncertainty for models trained on the currently available training set Derain, Once we have chosen such an acquisition batch [xx"9} of unlabeled data, we acquire labels {v."9) for these samples and train a new model on the combined training set Dirain U [(x"09, y ya })} and repeat the process. Crucial to the success of active
learning is the choice of acquisition function A(fx; (x p(w)) which is a function of the acquisition batch [x[²)) } and the distribution p(w) and which we try to maximize in each acquisition round. It measures the informativeness of an acquisition batch for the current model.
Univariate Regression is common task in machine learning. We assume that the target y is real-valued (E R) with homoscedastic Gaussian noise:
Y|x, New~N((u(xxe N(x(x; xz)-
We do require prior distribution as active learning is not concerned with how we arrive at the model we want to acquire labels for. We can define prior and perform, e.g, variational inference, but we do not need to. Hence, we use p(w).
3

---

Published in Transactions on Machine Learning Research (7/2022)
Equivalently, Y xx ~ w) + with E 1ithh~~W[ NO, o.). As usual, we assume that the noise is independent for different inputs x and parameters w. Homoscedastie noise is a special case of the general heteroscedastic setting: the noise variance is simply a constant. Our approach can be extended to heteroscedastic noise by substituting a function oN (x; w) for ON. but for this work we limit ourselves to the simplest case.
3.1 Kernel-based Methods
We build on Holzmüller et al. (2022) which expresses contemporary batch active learning methods using kernel-based methods. While a full treatment is naturally beyond the scope of this paper, we briefly review some key ideas here. We refer the reader to the extensive paper for more details.
Gaussian Processes are one way to introduce kernel- based methods. A simple way to think about Gaussian Processes (Williams& & Rasmussen, 2006; Lázaro-Gredila, & AggueirasVidal,, 2010; Rudner et al., 2022) is Bayesian linear regression model with an implicit, potentially infiiitediimensional feature space (depending on the covariance kernel) that uses the kernel trick to abstract away the feature map from input space to feature space.
Multivariate Gaussian Distribution. The distinctive property of a Gaussian Process is that all predictions are jointly Gaussian distributed. We can then write the joint distribution for a univariate regression model as:
.i..... Yn Xi. x No, Cov(u(x1),... ((x))),..........)))) I)
(4)
where p(x) are the observation noise free predictions as random variables and Cov Covlu(x)).... p(xxa) is the covariance matrix of the prodictions. The covariance matrix is defined via the kernel function x(x:
[((x,, x)) n.
(5)
i,5=1
The kernel function k(x, x') can be chosen almost arbitrarily, e.g. see Williams & Rasmussen (2006, Ch. 4). The linear kernel x') = x²) and the radial basis function kernel 6(x,x') = exp(-2 [ - x'12) are common examples, as the gradient kernel, which we examine next.
Fisher Information & Linearization. When using neural networks for regression, the gradient kernel
kgrad (x;
(x:x' x' w²) A Vwj((;; w V - (V")(x;w")) Vwi(('; w )) )) logp(w")]-
(6)
(7)
is the canonical choice, where w" is mazimum likelihood or mazimum posteriori estimate (MLE, MAP) and v - log p[w")] is the Hessian of the negative log likelihood at w" Note that V=p(x;w") is a row vector. Commonly, the prior is a Gaussian distribution with an identity covariance matrix, and thus V vol-logp(w")] I.
The significance of this kernel lies in its relationship with the Fisher information matrix w' (Immer, 2020; Immer et al., 2021; Kirsch & Gal, 2022), or equivalently, with the lincarization of the loss function around w" (Holzmüller et al., 2022). This leads to Gaussian approximation, which results in a Gaussian predictive posterior distribution when combined with Gaussian likelihood. The use of the finite-dimensional gradient kernel thus results in an implicit Bayesian linear regression in the context of regression models.
Posterior Gradient Kernel. We can use the well-known properties of multivariate normal distribtions to marginalize or condition the joint distribution in (4). Following Holzmüller et al. (2022), this allows us to explicitly obtain the posterior gradient kernel given additional Xi,... Xn. as:
Egrad -post (x (x;x' Iw*)
(8)
T
V VuH((x;w*)
A Vull(xww")
- plogp(("')]
Vw)(x';W')
oN
Vult(xni w+))
4

---

Published in Transactions on Machine Learning Research (7/2022)
The factor NN originates from implicitly conditioning on Y x,, which include observation noise.
Importantly for active learning, the multivariate normal distribution is the maximum entropy distribution for a given covariance matrix, and is thus an upper-bound for the entropy of any distribution with the same covariance matrix. The entropy given by the log log-determinant of the covariance matrix:
H Yn N xn log det (et((oolu(x))......
o²1) C..
where C, A マ 1og(( e) constant that only depends on the number of samples n. Connecting kernel-basod methods to cooommttion-thoretee quantities like the expected information gain, we then know that the respective acquisition scores are upper-bounds the actual expected information gain. Please see $A in the appendix for more details information-theoretic active learning.
3.2 Black-Box Batch Active Learning
We more formally introduce the empirical covariance kernel and compare it to the gradient kernel commonly used for active learning with deep learning models in parameter-space. For non-difeerentiable models, we show how can also be derived using a Bayesian model perspective on the hypothesis space.
In addition being illustrative, this section allows us to connect prediction- based kernels the kernels used Holzmüller et al. (2022), which in turns connects them to various SotA active learning methods.
3.2.1 Predictive Covariance Kernel
To perform black-box batch active learning, directly the predictive covariance of Yi|x, and Yjlx;:
CovalYX; x,,x,] = Covalu M (
(10)
where have abbreviated μ(x;w) with 1 and used the law of total covariance and the fact that the noise is uncorrelated between samples.
We define the predictive coveriance kernel (X;, the covariance of the predicted means:
kpreal ( (x/) A Covalle MW
(11)
Compared to 13.1, we do not define the covariance via the kernel but the kernel via the covariance.
This is also simply known as the covariance kernel in the literature (Shawe-Taylor et al., 2004). We use the prefix predictive to make clear that we look at the covariance of the predictions. The resulting Grain matrix is equal the covariance matrix of the predictions and positive definite (for positive oN and otherwise positive semi-definite) and thus a valid kernel.
3.2.2 Empirical Predictive Covariance Kernel
For K sampled model parameters Wi,...,wy WK ~ p(w) for example, the members of a deep ensemble the empirical predictive covoriance kernel kproa (xi; Xj) is the empirical estimate:
K
1ー- pred (x); Cova [ 1 1
K
Px
K
Uxx
Ux
K
(12)
k=1
1=1
1
(1
(13)
K
K
with centered predictions / - - immediately follows that the empirical predictive covariance kernel is valid kernel and positive semi-definite.
_ EK
As we can write this kernel an inner product, also
5

---

Published in Transactions on Machine Learning Research (7/2022)
3.2.3 Differentiable Models
Similar to Holzmüller et al. (2022, $C.1), we show that the posterior gradient kernel is a first-order approxi- mation of the (predictive) covariance kernel. This section explicitly conditions on Dirain, The result is simple but instructive:
Proposition 3.1. The posterior gradient kernel Egrad ppastd-postttee (Diuiu) (X;;XX ww) is an approzimation of the predictive covoriance kermel (xrea(x;;;))
Proof. We use first-order Taylor expansion of the mean function μ(x; around w p(x:w) = ((x:"") + (x) (w ww)
(14)
Choose w' Eww [ [w] (BMA). Then we have Ep(ur/tten) ju(x wi] u(xi Overall this yields:
(x;
Cov Ditsin [et Xx; w | (xji w)!
(15)
~ E Awwp(u Durain) WW Aw, V W Aw)]
(16)
Vulli En Awm Diriin AwwAw | /
(17)
Vul(x;; [ D V [wil(xx;w'))
(18)
2 kgrad (Disttt (x X, w
(19)
The intermediate expectation is the model covariance Cov(Ω DDtraiu] as w is the BMA. For the last step, use the generalized Gauss-Newton (GGN) approxination (Immer el al., 2021; Kirsch & Gal, 2022) and approximate the inverse of the covariance using the Hessian of the negative log likelihood at w
Cov(S Diraing ~ V - log p(w" [Dtrain)]
(20)
-22I- log Vol-loop(DDanne (DDrrann w") (GGN) ON E, Vwu(xtrain w ";"" T VWWu(xtrain w; v2 log p(ww),
(og)(w")]
(21)
(22)
(23)
where we have first used Bayes' theorem and that (DDrrain) vanishes under differentiation it is constant in w. Socondly, the Hessian of the negative log likelihood is just the outer product of the gradients divided by the noise variance in the homoscedastic regression case. v2 logg(w")) ]] is the prior term. This matches (8).
3.2.4 Non-Differenme Diffeereitet Models
How can apply the above result to non-differemtiable models? In the following, use Bayesian view on the hypothesis space to show that we can connect the empirical predictive covariance kernel to gradient kernel in this case, too. With Ω á (wi...., w.) fixed-o.g. those could be the different parameters of the members of an ensemble- we introduce a latent y to represent the index of the 'true' hypothesis We. E Ω from this empirical hypothesis space 22, which we want to identify. This is similar to QbC (Seung et al., 1992). In essence, the latent V takes on the role of Ω from the previous section, and we are interested in learning the 'true' from additional data. We, thus, examine the kernels for V, as opposed to S2
Specifically, we model using onc-hot categorical distribution, that is a multinomial distribution from which we draw one sample: v ~ Multinomial(e, 1), with g sK-1 parameterizing the distribution, where SK-1 denotes the K - simplex in R Then, d: p(V ek), where eg denotes the k-th unit vector; and ΣK Ek=1 qs= 1. For the corresponding predictive mean pxx; V we have:
x(x v) A u(x;ww) = (u(x; -), V.
(24)

---

Published in Transactions on Machine Learning Research (7/2022)
where we wy to denote the Wa when we have j ej in slight abuse of notation, and μ(x:) :) R is column vector of the predictions ((x;wx) for x for all wi- This follows from tà being one-hot vector.
We now examine this model and its kernels. The BMA of jix;; V) matches the previous empirical mean if we choose q to have an uninformative uniform distribution over the hypotheses (qk- _ k):
1 Σ K [u(x:ws)
j(x:q) q) & Ep(e) .).g) Σ qo
(25)
What is the predictive covariance kernel of this model? And what is the posterior gradient kernel for q?
Proposition 3.2.
1. The predictive covariance kernel upred, (x,, x5) for St using uniform q is equal to the empirical predictive covariance kernel Kprea (xx;))))
2. The 'posterior' gradient kernel Egrad oppost (Dimiii (x;xx) for 52 in respect to v using uniform q is equal to the empirical predictive covariance kernel k- (x;; Xx;
pred
Proof- Like for the previous differentiable model, the BMA of the model parameters V is just q: IE VT q-
The first statement immediately follows:
Epred, y (x,; = Cov [Axx p(x;))) = Ep(w)] We
pxx
K
Σ
pxj
k pred (xxx))) (26)
For the second statement, will show that express the predictive covariance kernel as linearization around V. We can read off lincarization for Vw v)) from the inner product in Equation (25):
Veñ(x; vp) μ(x: T
(27)
This allows us to write the predictive covariance kernel as a lincarization around q:
kpred, s xx) = Cov CoVerp(M) [xxx v); (x,; })] -E++AND(s) V ( +) Agb, V V AM)
(28)
(29)
Voj((x;g) Cov L V (
(30)
Agrad io post Deman (x,
(31)
The above gradient kernel is only the posterior gradient kernel in the sense that we have sampled we from the non-differentiablc model after inference on training data. The samples themselves are drawn uniformly.
The covariance of the multinomial v is: Cov [V] diag(g) T. Thus, substituting, we can verify that the posterior gradient kernel is indeed equal to the predictive covariance kernel explicitly once more:
I kgrad, st spost Diraing (x) VeA((;;)) q) (diag(g) q9 (x;()))))
(32)
= =((x;)) diag(q) ((x;;-) (n(x;;)) 4) (q ((x;;:))
(33)
( E ((xx:ww))
K Σ (();:w)))))))) ((x;
1
(34)
K
--- (x):)).
(35)
4If we had additional information about the f2 for example, we had validation losses could use that to inform q.
7

---

Published in Transactions on Machine Learning Research (7/2022)
-0.8
Aeg Fusctios
Methed
-10
-13
BALD
Core See
BoNeHRALD
LUMD
-1.2
moar Top RMSE
-1.
-14
Acp. Fanziion
Matlad
-1.
1.7
BALD
Cano-Sal
-1.8
BachBALD
LCMD
-18
Training set size Nersis
Acquistion batch size Nust
(a) Deep Neural Networks.
-0.7
Acg,
-0.0
Uniform
ACS-FW
0.8
BALD
BatchHALD
BADGE
10030 RMOSE
-088
mecan log RMSE
0099
Acq. Fanction
Unifeen
ACS-FW
-12
BotchBALD
BADGE
236
512
1004
2045
4056
64
128
256
512
1024
2048
4096
'trininn Nown
Acquistion batch Nuscs
(b) Random Forests.
Mean logarithmác RMSE over 15 regression datasets. (a) For DNNs, we see that black- box methods work well as white-box D methods, and in most cases better, with the exception of ACS-FW and BAIT. (b) For random forests (100 estimators) with the default hyperparameters from scikit-learn (Pedregosa et al., 2011), we see that black-box methods perform better than the uniform baseline, with the exception of BALD, which uses top-k acquisition. In the appendix, see Table 3 for average performance metrics and Figure 6 and 7 for plots with additional error metrics. Averaged over at least trials.
Figure 1:
This demonstrates that a Bayesian model can be constructed on top of a non-differentiable ensemble model. Bayesian inference in this context aims to identify the most suitable member the ensemble. Given the limited number of samples and likelihood of model misspecification, is likely that none of the members accurately represents the true model. However, for active learning purposes, the main focus is solely on quantifying the degree of disagreement among the ensemble members.
A similar Bayesian model using Bayesian Model Combination (BMC) could be set up which allows for arbitrary convex mixtures of the ensemble members. This would entail using Dirichlet distribution V ~ Dirichlet(a) instead of the multinomial distribution. Assuming an uninformative prior (ax - 1/K), this leads to the same results up a constant factor of 1 + Σ. ag 2. This pleasing because does not matter whether we use a multinomial or Dirichlet distribution, that is: whether we take hypothesis space view with a 'truc' hypothesis or accept that our model is likely misspecified and we are dealing with mixture of models, the results are the same up to constant factor.
Application to DNNs, BNNs, and Other Models. The proposed approach has relevance due to its versatility, as it can be applied to a wide range of models that can be consistently queried for prediction, including deep ensembles (Lakshminarayanan et al., 2016), Bayesian neural networks (BNNs) (Blundell et al., 2015; Gal & Ghahramani, 2015), and non-differentiable models. The kernel used in this approach simple implement and scales in the number of empirical predictions per sample, rather than in the parameter space, in other methods such Ash et al. (2021).
8

---

Published in Transactions on Machine Learning Research (7/2022)
0.5
Adg Putction
-0.0
-03
BALD
Core-he
RatchBALD
(00a8 Txp RMSSE
-0.9
BATT
L.MD
-000
-10
1.0
Asp, Fanciio
-11
HALD
Curo-Sel
-13
BAFT
LCMD
-13
Training set size Nera
Acquistion batch size Nusto
(a) Random Forests (Bagging).
Unifeemn
ACS-WW
RetchBBLLD
BARGE
-0.9
moaa lop RMSSE
-1.15
L.CDD
-1.
1120
-1.
-145
-1.
Apq.
Unifeen
ACS-FW
-135
-13
BotchBALD
BADGE
-1.40
-144
236
512
1004
2045
4056
64
128
256
512
1024
2048
4095
'taining Nown
Acquiction bandh Nusts
(b) Gradient-Bousled Trees.
Figure 2: Mean logarithmic RMSE over regression datasets (cont'd) d). For random forests using bagging (a) with 10 bootstrapped training sets, and for gradient-boosted trees (Dorogush et al., 2018) (b) with virtual ensemble of 20 members, baseline: LCMD, BADGE and CoreSet. We hypothesize that the virtual ensembles and bagged ensemble of random forests do not express as much predictive disagreement which leads to worse performance for active learning. In the appendix, see Table 3 for average performance metrics and Figure and for plots with additional error metrics. Averaged over least trials.
that only few of the black-box methods perform better than the uniform
4 Results
We follow the evaluation from Holzmüllerr et al. (2022) and use their framework ease comparison. This allows to directly compare several SotA methods in regression sctting, respectively their kernel-based analogues. Specifically, compare to the following popular deep active learning methods: BALD (Houlsby
et al., 2011), BatchBALD (Kirsch et al., 2019), BAIT (Ash et al., 2021), BADGE (Ash et al., 2019), ACS-FW (Geifmaan & El-Yaniv 2017), and LCMD
(Pinsler et al., 2019), Core-Set (Sener & Savarese, (Holzmüller et al., 2022). The kernels and selection methods used are detailed in Table 1 in the appendix. We also compare to the random selection baseline ('Uniform'). We use 15 large tabular datasets from the UCT Machine Learning Repository (Dua & Graff, 2017) and the OpenML benchmark suite (Vanschoren al., 2014) for our experiments: sgemm (SGEMM GPU kernel performance); wee sydney (Wave Energy Converters); at slices (Relative location of CT slices axial axis); kegg__eggee undir (KEGG Metabolic Reaction Network Undirected); online_ pideo (Online Video Characteristics and Transcoding Time); query (Query Analytics Workloads); poker (Poker Hand); road (3D Road Network North Jutland, Demmark); mlr knn rng; fried; sarcos (SARCOS data). See
diamonds; methane; slock (BNG stock); protein (plysicochemical-prinl Table the appendix for more details.
Experimental Setup. We use the same experimental setup and hyperparameters as Holzmüller al. (2022). We report the logarithmic RMSE averaged over 5 trials for each dataset. and method. For ensembles,
9

---

Published in Transactions on Machine Learning Research (7/2022)
Dutasets
slices
kegg. undir
online vvdeo
query road
sgemm
diamonds
methune
poker
stock
fried
mir kan_rng
protein
sarcos
wec wec.sydney
Uniform (
0.5
0.0
-0.5 0.0
0.5
-0.5 0.0
0.5
0.5 0.0
0.5
-0.5 0.0
0.5
-0.5
0.0 Core-Set
0.5
-0.5 0.0
0.5
-0.5 0.0
0.5
BALD
ACS-FW
BatchBALD
BAIT
BADGE
LCMD
Il vs Uniform (1)
Figure 3: Average Logarithmic RMSE by regression datasets for DNNs: us D (us Uniform). Across acquisition functions, the performance of black-box methods is highly correlated with the performance of white-box methods, oven though black-box methods make fewer assumptions about the model. We plot the improvemennt of the white-box met over the uniform baseline on the x-axis (so for datasets with markers right of the dashed vertical lines, the white-box method performs better than uniform) and the improvement of the black-box method over the uniform bascline the y-axis (so for datasets with markers above the dashed horizontal lines, the black-box method performs better than uniform). For datasets with markers in the diagonal half, the black-box method performs better than the white-box method. The average over all datasets is marked with a star x. Surprisingly, on average over all acquisition rounds, the black-box methods perform slightly better than the white-box methods for all but ACS-FW and BAIT. In the appendix, see Figure for the final acquisition round and Table for details on the datasets. Averaged over least trials.
we compute the performance for each ensemble member separately, enabling fair comparison to the non- ensemble methods. Porformance differences can thus be attributed to the acquisition function, rather than the ensemble. We used A100 GPUs with 40GB of GPU memory.
Ensemble Size. For deep learning, we use a small ensemble of 10 models, which sufficient to achieve good performance. This ensemble size be considered 'small' in the regression sctting (Lázaro-Gredilla & Figueiras- Vidal, 2010; Zhang & Zhou, 2013), whereas in Bayesian Optimal Experimenl Design much higher sample counts are regularly used (Foster et al., 2021). In many cases, training an ensemble of regression models is still fast and could be considered cheap compared to the cost of acquiring additional labels. For non-iifferentiable models, we experiment with: a) random forests (Breiman, 2001), using the different trees as ensemble members; b) bagged random forests, each with 100 trees; and c) virtual ensemble of gradient- boosted decision trees (Prokhorenkova in scikit-learn (Pedregosa et al., 2011) with default hyperparameters, that is using 100 trees per forest. We use the predictions from each tree virtual ensemble member. We do not perform any hyperparameter tuning. We also report results for random forests with bagging, where we train real ensemble of 10 random forests. For gradient-boosted decision trees, we use a virtual ensemble of up to 20 members with early stopping using a validation S We use the implementation in CatBoost (Dorogush et al., 2018). We do not perform any hyperparameter tuning.
al., 2017). For random forests, we use the implementation provided
Black-Bos vs White-Box Deep Active Learning. In Figure la and 3, we that B3AL is competitive with white-box active learning, when using BALD, BatchBALD, BAIT, BADGE, and Core-Set. On average, A&AL outperforms the white-box methods on the 15 datasets analyzed (excluding ACS-FW and BAIT). We hypothesize that this is due to the implicit Fisher information approximation in the white-box methods (Kirsch & Gal, 2022), which is not as accurate in the low data regime as the more explicit approximation in B3AL via ensembling.
ff the virtual ensemble creation fails because there no sufficiently many due early stopping, we halve the ensemble size and retry. This was only needed for the poker dataset.
10

---

Published in Transactions on Machine Learning Research (7/2022)
-133
Aog. Fanciion
Metlad
Au. Faciss RANARAL
RADGE
Unifemn
RAIT ACS.FW
RADKIE BacABALD
LOODD
-1.4
LOMD
Coce
BALD
-1.5
00005
-1.6
-1.7
-1.8
-12
Ensemble size
Neural Network.
(b) Random Forests.
-0.8
Aop. Function
-1.05
B BADGE
Unifom
LCMD
BatchRALD
BAIT
Core-Set
-0.9
BALLD
ACS-FW
Com-Set
-1.10
ButchBALD
BAIT
-1.15
RADGE
ACS-FW
RMIN
-1.0
mncan log RMSSE
-1.20
-1.1
1.25
-130
-122
-135
-1.3
-1.40
-146
20
27
Ensemble
Emsemble
(c) Random Forests (Bagging).
(d) Gradient-Boostee Trees.
Figure 4: Ensemble Size Ablation: Mean logarithmic RMSE over 15 regression datasets. For neural networks and random forests, increasing the ensemble size generally improves the performance (with the exception of LMCD, for which performance first improves and then degrades). Uniform sampling improves slightly with increasing ensemble size due to betler model predictions; compared to the improvements of the active learning methods, these increases are small, however. For gradient- grasient-boosted trees with virtual ensembles, increasing the ensemble size (5-160) does not provide a performance boost. Averaged over at least trials.
Why Can Black-Box Methods Outperform White-Box Methods? Following $3, white-box and black- box methods are based on kernels, which can be seen as different approzimations the predictive covariance kernel. White-box methods implicitly assume that the predictive covariance kernel well approximated by the Fisher information kernel and the gradient kernel (Kirsch & Gal, 2022). However, Long (2022) demonstrated that this assumption does not always hold, particularly in low data regimes, where a Gaussian might not approximate the parameter distribution well. Instead, Long (2022) suggests using multimodal distribution. In these situations, methods that employ ensembling, such as AA, to approximate the predictive covariance kernel can be more robust. The different ensemble members can reside in different modes of the parameter distribution, allowing black-box methods to out perforrmm their white-box counterparis.
Non-Differentiable Models. In Figure 1b, 2a, and 2b, we observe that B3AL is effective for non- differentiable models, including random forests and gradient-boosted decision trees. BALD for non- differentiable models can be considered equivalent to QbC (Seung et al., 1992), while BatchBALD for non-differentiable models can be viewed as equivalent to QbC with batch acquisition (Nguyen et al., 2012). For random forests, all methods except. BALD (using top-k selection) outperform uniform acquisition. How- ever, for random forests with bagging and gradient- cradient-boosted decision trees, B3AL surpasses random acquisition only when employing LMCD and BADGE. This may be attributed to the reduced disagreement within a virtual ensemble for gradient-booste decision trees and between distinct random forests. In particular, random forests with bagging appear to support this explanation, single random forest seems to exhibit.
11

---

Published in Transactions on Machine Learning Research (7/2022)
more disagreement among its individual trees than ensemble of random forests with bagging does between different forests. This is evident in the superior overall active learning performance of the single random forest compared to the ensemble of random forests with bagging, c.f. Figure 1b and 2a.
4.1 Ablations
Here, we discuss two ablations of A3AL: the effect of the ensemble size and the effect of the acquisition batch sizes.
Acquisitinn Batch Size. The right-hand plots in Figures 1 and show that the performance generally decreases with increasing acquisition batch size for all acquisition approaches. Comparing white-box and black-box methods in Figure la, we see that only the largest batch size of 4096, the white-box methods perform well as the black-box methods for neural networks.
Ensemble Batch Size. Figure 4 shows the effect of the ensemble size the performance of B3AL for random forests. Increasing the ensemble size generally improves the performance for all acquisition approaches except LMCD, for which performance first improves and then degrades. The performance increase with uniform sampling is due to better model prodictions. This shows that the improvements stemming from more informative sample acquisition by the active learning methods due increasing ensemble sizes can be significantly larger than the improvements due to better model predictions.
Best- pest-performing Kernels and Selection Methods. Holzmüller et al. (2022) evaluate different kernels and variants of selection methods based on the SotA methods. The best-performing kernels and selection methods are shown in Table 5 in the appendix. We run an ablation comparing black-box methods with the best-performing white-box kernels from Holzmüller et al. (2022). These variants do not actually match prior art-we use the same names for the acquisition functions to denote the origin the variants. The results are shown in Appendix B.1. We observe that B3AL, also performs on par or better than the best-performing white-box kernels.
5 Conclusion and Future Work
In this paper, we have demonstrated the effectiveness of simple extension kernel-bascd methods that utilizes empirical predictions rather than gradient kernels. This modification enables black-box batch active learning with good performance. Importantly, B3AL also generalizes to non-differentiable models, an area that has received limited attention as of late.
The main limitation of our proposed approach lies in the acquisition of a sufficient. amount of empirical predictions. This could be a challenge, particularly when using deep ensembles with larger modols or non-differrentable models that cannot be parallelized efficiently. Our experiments using virtual ensembles indicate that the diversity of the ensemble members plays crucial role in determining the performance. The main limitation of the empirical comparisons is that we only consider regression tasks. Extending the results to classification is important direction for future work.
Overall, our results partially answer one of the research questions posed by Kirsch & Gal (2022): how do prediction-basee methods compare to parameter-based ones? We find that for regression the prediction-based methods are competitive with the parameter- based methods in batch active learning.
Acknowledgements
The author would like to thank the anonymous TMLR reviewers for their patience and kind feedback during the review process. Their feedback has significantly improved this work. Likewise, many thanks to David Holzmüller for their constructive and helpful feedback, as well as for making their framework and results easily available. AK is supported by the UK EPSRC CDT in Autonomous Intelligent Machines and Systems (grant reference EPP (P//1158997/) ChatGPT was used to suggest specific text and table edits.
12

---

Published in Transactions on Machine Learning Research (7/2022)
References
Christos Anagnostopoulos, Fotis Savva, and Peter Triantafillou. Scalable aggregation predictive analytics.
Applied Intelligence, 48(9):2546-2567, 2018.
Jordan T. Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alckh Agarwal. Deep batch
active learning by diverse, uncertain gradient lower bounds, 2019.
Jordan T. Ash, Surbhi Goel, Akshay Krishnamurthy, and Sham Kakade. Gone fishing: Neural active learning with fisher embeddings, 2021.
Rafael Ballester-Ripoll, Enrique G Paredes, and Renato Pajarola. Sobol tensor trains for global sensitivity analysis. Reliability Engineering & System Safety, 183:311-322 2019.
Freddie Bickford Smith, Andreas Kirsch, Sebastian Farquhar, Yarin Gal, Adam Foster, and Tom Rainforth. Prediction-Orienneed Bayesian Active Learning. AI Stats, 2023.
Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural network. In International conference on machine learning, pp. 1613 1622. PMLR, 2015.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bossclut, Emma Brunskill, et al. On the opportunities and risks of foundation models. Xrxiv preprint arXiv:2106 1221108..722 08.00258, 2021.
Leo Breiman. Bagging predictors. Machine learning, 24(2):123 140, 1996.
Leo Breiman. Random forests. Machine learning, 45:5-32, 2001.
Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranay Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Robert Burbidge, Jem Rowland, and Ross King. Active learning for regression based on query by committce. Lecture Notes in Computer Science, 4881:209-20 218, 2007.
David Cohn, Les Atlas, and Richard Ladner. Improving generalization with active learning. Machine learning, 15:201 221, 1994.
Cody Coleman, Christopher Yeh, Stephen Mussmann, Baharan Mirzasoleiman, Peter Bailis, Percy Liang, Jure Leskovec, and Matei Zaharia. Selection via proxy: Efficient data selection for deep learning, arXiv preprinl. Xrv::11966 1829, 2019.
Tewodors Deneke, Habtegebreil Haile, Sébastien Lafond, and Johan Lilius. Video transcoding time prediction for proactive load balancing. In International Conference Multimedia and Expo, 2014.
Anna Veronika Dorogush, Vasily Ershov, and Andrey Gulin. Catboost: gradient boosting with categorical fcatures support. arXiv preprint arXiv:1810. 11363, 2018.
Dheeru Dua and Cascy Graff. UCI machine learning repository, 2017. URL http://arcchive.css uci. edu/ ml.
Valerii Vadimovich Fedorov. Theory of optimal experiments. Elsevicr, 1972.
Adam Foster, Desi R Ivanova, Tyas Malik, and Tom Rainforth. Deep adaptive design: Amortizing sequential bayesian experimental design. In International Conference on Machine Learning, pp. 3384 3395. PMLR, 2021.
Jerome H Friedman. Multivariate adaptive regression splines. The Annols of Statistics, pp. 1-67, 1991.
Yarin Gal and Ghahramani. Dropout as a bayesian approximation: representing model uncertainty in deep learning. arxiv. arxiv preprinl 1KK::500.001.43 2015.
13

---

Published in Transactions on Machine Learning Research (7/2022)
Yarin Gal, Riashat Islam, and Zoubin Ghahramani. Deep bayesian active learning with image data. In International Conference on Machine Learning, pp- 1183-1192. PMLR, 2017.
Yonatan Geifman and Ran El-Yaniv. Deep active learning over the long tail. arXiv preprinl Ar::::1711.0000 2017.
Franz Graf, Hans-Peter Kriegel, Matthias Schubert, Sebastian Pölsterl, and Alexander Cavallaro. 2D image registration in ct images using radial image descriptors. In International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer, 2011.
Daniel Hernández-Lobato Jose Hernández-lobtto, and Pierre Dupont. Robust multi-class ganssian process classification. Advances in neural information processing systems, 24, 2011.
David Holzmüller, Viktor Zaverkin, Johannes Kästner, and Ingo Steinwart, A framework and benchmark for deep batch active learning for regression. arviv preprint arXiv:2203.00, 094 2022.
Neil Houlsby, Ferenc Huszár, Zoubin Ghahramani, and Máté Lengyel. Bayesian active learning for classification and preference learning. arXiv preprint arXiv:1112.5745, 2011.
Alexander Immer Discntangling the gauss-newton method and approximate inference for neural networks. arXiv preprinl arXiv:2007.1199) 1190/ 2020.
Alexander Immer, Macicj Korzepa, and Matthias Bauer. Improving predictions of bayesian neural nets via local linearization. In International Conference on Arlificial Intelligence and Slatislics, pp. 703-711. PMLR, 2021.
Andrew Jesson, Panagiotis Tigas, Joost Amersfoort, Andreas Kirsch, Uri Shalit, and Yarin Gal. Causal- bald: Deep bayesian active learning of outcomes to infer treatment-effects from observational data. Advances in Neural Information Processing Systems, 34:30465-30478, 2021.
Scho Kee, Enrique Del Castillo, and George Runger. Query-by-coomittoe improvement with diversity and density in batch active learning. Information Sciences, 454:401 418, 2018
Andreas Kirsch and Yarin Gal. Unifying approaches in active learning and active sampling via fisher information and information-theoreti. quantities. Thansactions on Machine Learning Research, 2022.
Andreas Kirsch, Joost Van Amersfoort, and Yarin Gal. Batchbald: Efficient and diverse batch acquisition for deep bayesian active learning. Advances in neural information processing systems, 32, 2019.
Andreas Kirsch, Scbastian Farquhar, Parmida Atighchchian, Andrew Jesson, Froderic Granchaud-Charron and Yarin Gal. Stochastic batch acquisition for deep active learning. CoRR, 2106:4, 2021.
Suraj Kothawade, Nathan Beck, Krishnateja Killamsetty, and Rishabh Iyer. Similar: Submodular information measures based active learning in realistic scenarios. Advances in Neural Information Processing Systems, 34:18685 18697, 2021.
Suraj Kothawade, Vishal Kaushal, Ganesh Ramakrishnan, Jeff Bilmes, and Rishabh Iyer. Prism: A rich class of parameterized submodular information measures for guided data subset selection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 10238 10246, 2022.
Anders Krogh and Jesper Vedelsby. Neural network ensembles, cross validation, and active learning. Advances in neural information processing systems, 7, 1994.
Alex Kulesza and Ben Taskar. k-dpps: Fixed-size determinantal point processes. In Proceedings of the 28th International Conference on Machine Learning (ICML-11), pp. 1193-1200, 2011.
B Lakshminarayanan, A Pritzel, and C Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. arxiv. arXiv preprint orXiv:1612.01. 01474, 2016.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017.
14

---

Published in Transactions on Machine Learning Research (7/2022)
Miguel Lázaro-Gredilla, and Anibal R Figueiras-Vite Vidal. Marginalized neural network mixtures for large-scale regression. TEEE transactions on neural nelworks, 21 21(8):1345-1361 2010.
Dennis Lindley. On a measure of the information provided by an experiment. The Annals of Mathemalionl Statistics, pp. 986-1005, 1956.
Quan Long. Multimodal information gain in bayesian design of experiments. Computational Statistics, 37(2): 865 885, 2022.
David JC MacKay. Information-basao objective functions for active data selection. Neural computation, 4(4): 590 604, 1992.
George L Nemhauser, Laurence A Wolsey, and Marshall L Fisher. An analysis of approximations for maximizing submodular set functions i. Mathematical programming, 14(1):265 294, 1978.
Mehdi Neshat, Bradley Alexander, Markus Wagner, and Yuanzhong Xia. A detailed comparison of meta- heuristic methods for optimising wave energy converter placements. In Proceedings of the Genetic and Evolutionary Computation Conference, 2018.
Hieu Nguyen, Joseph Yadegar, Bailey Kong, and Hai Wei. Efficient batch-mode active learning of random forest. In 2012 IEEE Statistical Signal Processing Workshop (SSP), pp. 596 599. IEEE, 2012.
Ian Osband, Zheng Wen, Seyed Mohammad Asghari, Vikranth Dwaracherla, Morteza Ibrahimi,, Xiuyuan Lu, and Benjamin Van Roy. Epistemic neural networks. arXiv preprint arXiv:2107.0880 2021.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine lcarning in Python. Journal of Machine Learning Rescarch, 12:2825-2830, 2011.
Robert Pinsler, Jonathan Gordon, Eric Nalisnick, and José Miguel Hernández-Lobato. Bayesian batch active learning as sparse subset approximation. Advances in neural information processing systems, 32, 2019.
L Prokhorenkova, G Gusev, A Vorobev, AV Dorogush, and A Gulin. Catboost: Unbiased boosting with categorical features. arxiv 2017. ar preprint Xrxiv:1706 100.:09666916 2017.
Pengzhen Ren, Yun Xiao, Xiaojun Chang, Po-Yao Huang, Zhihui Li, Brij B Gupta, Xiaojiang Chen, and Xin Wang. survey of deep active learning. ACM computing surveys (CSUR), 54(9):1 40, 2021.
Tim G. J. Rudner, Zonghao Chen, Yee Whye Teh, and Yarin Gal. Tractabe Function-Space Variational Inference in Bayesian Neural Networks. 2022.
Fotis Savva, Christos Anagnostopoulos, and Peter Triantafillou. Explaining aggregates for exploratory analytics. In International Conference on Big Data, 2018
Alexander Schrijver et al. Combinalorial oplimization: polyhedra and efficiency, volume 24. Springer, 2003.
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. arXio preprint Xrxin:1788. 00488, 2017.
Burr Settles. Active learning literature survey. 2009.
H Sebastian Seung, Manfred Opper, and Haim Sompolinsky. Query by committee, In Proceedings of the fifth annuall workshop on Computational learning theory, pp. 287-294, 1992.
Paul Shannon, Andrew Markiel, Owen Ozier, Nitin S Baliga, Jonathan T Wang, Daniel Ramage, Nada Amin, Benno Schwikowski, and Trey Idcker Cytoscape: software environment for integrated models of biomolecular interaction networks. Genome research, 13(11):2498 2504, 2003.
John Slawe-Taylo, Nello Cristianini, et al. Kernel methods for pattern analysis. Cambridge university press, 2004.
15

---

Published in Transactions on Machine Learning Research (7/2022)
Dominik Slęzak, Marek Grzegorowski, Andrzej Janusz, Michal Kozielski, Sinh Hoa Nguyen, Marek Sikora, Sebastian Stawicki, and Lukasz Wróbel. A framework for learning and embedding multi-sensor forecasting models into a decision support system: A case study of methane concentration in coal mines. Information Sciences, 451:112-133, 2018.
Joaquin Vanschoren, Jan N Van Rijn, Bernd Bischl, and Luis Torgo Openml: networked science in machine learning. ACM SIGKDD Eaploralions Newsleller, 15(2):49 60, 2014.
Christopher KI Williams and Carl Edward Rasmussen. Gaussian processes for machine learning, volume 2. MIT press Cambridge, MA, 2006.
Xueying Zhan, Qingzhong Wang, Kuan-hao Huang, Haoyi Xiong, Dejing Dou, and Antoni B Chan. A comparative survey of deep active learning. orXiv preprint 3rXiv:200..13/ 2022.
Min-Ling Zhang and Zhi-Hua Zhou. Exploiting unlabeled data to enhance ensemble diversity. Dala Mining and Knowledge Discovery, 26:98 129, 2013.
16

---

Published in Transactions on Machine Learning Research (7/2022)
A Information- Theoretic Active Learning
The following section provides a brief overview of information-thcoreti active learning, focusing on the expected information gain/BALD and greedy batch selection. Other information-tteorele quantities can be examined similarly. We refer to Kirsch & Gal (2022) for more details. For other selection methods, we refer to Holzmüller et al. (2022).
Expected Information Gain (EIG) (Lindley, 1956) based on the mutual information. For a Bayesian model, the FIG between the parameters Z and the predictions Y x is defined
[[2 Y x = H - H Y,x] / p(w. y x) log
p(w yy dw dy, p(w)
(36)
where [[22; Y x] is the mutual information between the parameters Z and the model predictions Y given the data H[22 is the entropy of the parameters Ω; and H[22 Y, is the conditional entropy of the parameters Q in expectation over the predictions Ω and the model predictions Y x. The EIG measures the amount of information that can be gained about the parameters Ω by observing the predictions Y x on average, where we think of the entropy H[[2] as capturing the uncertainty about the parameters Q.
Computing the EIG can be difficult when focused on the model parameters, as we need to compute the conditional entropy H( Y, x|- -et this is what many recent. approaches elfectively atlempt to do via the Fisher information (Kirsch & Gal, 2022). (We do not need to compute the entropy of the parameters H[S)], as it does not depend on the data and can be dropped from the optimization objective.)
BALD (Houlsby et al., 2011) examines the expected information gain in prediction space instead of parameter space, using the symmetry of the mutual information:
1[22 |x] I[Y; S x H[Y x] H[Y [2,x - f p(ws. y x) log
p(y x, (x,W) dw dy, p(y
(37)
x)
This can be much easier to evaluate by sampling from Ω without the need for additional Bayesian inference. BatchBALD (Kirsch et al., 2019) is an extension of BALD to the batch acquisition of samples {x²'9) for classification tasks:
1[2 YB X1, xB] H YB X1,
xBl
Σ H[Y x,, S2,
(38)
using the mutual information between the parameters w and the predictions {Yi} on an acquisition candidate batch [x.]. and thus takes the correlations between samples into account.
This joint entropy can be upper-bounded by the corresponding entropy of multivariato normal distribution with the same covariance matrix, while the conditional entropy term is precisely the entropy of a normal distribution given our model assumptions for the observation noise, yielding an upper- upper-bound bound overall.
The same approach can be applied to other information quantitics like the expected predicted information gain (Bickford Smith et al., 2023). Unlike Kirsch & Gal (2022), which examines the connection between Fisher information (weight-space) and information--heeretiee quantities, this exposition shows the connection between predictions and information-
quantities.
Greedy Acquisitinnn a heuristic to find the subset of size in Dpool that maximizes A({x, A((x)),((w)) Finding the optimal solution NP-hard (Schrijver et al., 2003). Kirsch et al. (2019) find that ABatchBALD is submodular and that iteratively adding the sample x4*9, with the highest ABatchBALD ( (x"*9) ,,.., x p(w) to the acquisiiin batch size is -approooooiinnnionn. This is well-known result in submodular optimization and is known as the greedy algorithm (Nemhauser et al., 1978). For the entropy approximation, finding an optimal solution is equivalent to finding the mode of the k-DPP (Kulesza & Taskar, 2011).
p(w)).
17

---

sgenm Published in Transactions on Machine Learning Research (7/2022)
B Results
Table 1: Kernels and selection methods for SotA methods. Taken from Hokmüller et al. (2022). See Holzmüller et al. (2022) for more details.
Known
Selection method
Kernel
Remark
BALD (Houlsby et al., 2011)
MAXDIAG
Nerai
for CP with kernel & for GP with kernel k, proposed for classification
BatchBALD (Kirsch et al., 2019)
MAXDET-P
ky Ruvon
BATT (Ash et al., 2021)
BAIT-FB-P
Krran
ACS-FW (Pinsler et al., 2019)
FRANKWOLPE-P
Core-Set (Sener & Savarese, 2017), FF-Active (Geifimnn & EI- Yaniv,
MAXDIST-TP"
kn
proposed for classification
2017)
BADGE (Ash et al., 2019)
KMEANSPP-P
LCMD
LCMD-TP
kgrad -oso-csshhhh2
This refers to their simpler k-center-greedy selection method.
Table 2: Overview over used data sets. See Hellester-Ripoll et al. (2019); Neshat et al. (2018); Graf et al. (2011); Shannon et al. (2003); Dencke et al. (2014); Anagnostopoulos ct al. (2018); Savva et al. (2018); Friedman (1991); Slęzak et al. (2018). The second column entries are hyperlinks to the respective web pages. Taken from Holzmüller al. (2022).
Short name
Initial pool set size
Tost set size
Number of featuros
Source
OpeuML ID
Full name
192000
48320
14
UCI
SGEMM CPU kerrel performance
syduuy
56320
14400
48
UCI
Wave Energy Converters
ct slices
41520
10700
379
UCI
Relative location of CT slices on axial axis KEGG Metabolic Reaction Network (Undirected)
kegg_undir undir
50407
12921
27
UCI
online video
53748
13756
26
UCI
Online Video Characteristics and Transcoding Time
query
158720
40000
UCI
Query Analytics Workloads
poker
198720
300000
95
UCT
Poker Hand
road
198720
234874
UCI
3D Road Network (North Jutland, Denmark)
mir kmn mng
88123
22350
132
OpenML
42454
mlr kun rog
fried
31335
8153
10
OpenMI,
564
fried
diemonds
41872
10788
29
OpeuML
42225
diamouds
methane
198720
300000
33
OpenML
42701
Methane
stock
45960
11809
OpenMLL
1200
(stokk)
protein
35304
9146
OpenML
42903
poysioochemical-proteee
sarcos
34308
8896
21
GPML
SARCOS data
18

---

Published in Transactions on Machine Learning Research (7/2022)
Dutasets
ct,slices diamonds
kegg_undir methane mir mir_knn_mg
online_video
query road sancos
sgemm
poker
stock
Fried
peotein
wec_sydney
Uniform (1)
-0.5
-0.5 0.0 0.5
-0.5 0.0 0.5
-0.5 0.0 0.5
-0.5 0.0 0.5
-0.5 0.0
0.5
-0.5 0.0
0.5
-0.5 0.0 0.5
BALD
ACS-FN
BatchBALD
BAIT
Core Set
BADGE
L.MD
vs Uniform (1)
Figure 5: Pinal Logarithmic RMSE by regression datasets for DNNs: us D (es Uniform). Across acquisition functions, the performance of black-box methods is highly correlated with the performance white-bo methods, even though black-box methods make fewer assumptions about the model. We plot the improvement of the white-box method (L) over the uniform baseline on the x-axis, so for datasets with markers left of the dashed vertical lines, the white-box method performs better than uniform, and the improvement of the black-box method over the uniform baseline on the y-axis, for datasets with markers above the dashed horizontal lines, the black-box method performs better than uniform. Similarly, for datasets with markers in the region, the black-bon method performs better than the white-box method. The average over all datasets marked with a star *.
Table 3: Average performance of black-box and white-box O batch active learning acquisition functions using DNNs. On average, for five acquisition methods, the black-box method performs better than the white-box method. Cf. Figure 3, which analyzes the final epoch.
Acquisition funettoo
MAE
RMSE
95%
99%
MAXE
Uniform
-1.935
-1.401
-0.766
-0.163
1.107
BALD
-1.794
-1.389
-0.713
-0.221
0.946
BALD
-1.717
-1.278
-0.611
.0.070
1.000
BatchBALD
-1.865
-1.465
-0.792
-0.03
0.892
M BatchBALD
-1.894
-1.462
-0.808
-0.287
0.912
BAIT
1.999
-1.540
-0.895
00.56
0.895
BAIT
-1.892
-1.489
-0.817
-0.328
0.881
ACS-FW
-1.935
-1.436
-0.793
-0.222
1.031
ACS-FW
-1.678
-1.168
-0.509
0.085
Core-Set
-1.988
-1.585
-0.926
-0.435
0.831
Core-Set
-1.924
-1.491
-0.831
-0.308
0.929
BADGE
-2.042
-1.579
-0.931
-0.383
0.948
BADGE
22009
-1.531
-0.897
-0.311
1.009
LCMD
LCMD
2.033
-1.589
-0.940
-0.401
0.913
19

---

Published in Transactions on Machine Learning Research (7/2022)
Table 4: Average performance of block-bor models.
batch active learning acquisition functions on non-differentiabbe
Random Forests
(b) Random Forests (Bagging)
Acquisition function
MAE
MAXE
Acquisition funetion
MAE
RMSE
95%
MAXE
Uniform
-1.516
-0.977
-0.292
0.290
1.380
Uniform
-1.438
-0.931
-0.237
0.317
1.375
BALD
-1.222
-0.815
-0.106
0.365
1.318
BALD
-1.151
-0.771
-0.070
0.386
1.349
BatchBALD
-1.419
-1.070
-0.401
0.064
1.195
BatchBALD
-1.883
-0.923
-0.245
0.205
1.263
BAIT
-1.582
-1.158
-0.485
0.021
1.198
BAIT
-1.304
-0.944
-0.269
0.184
1.254
ACS-FW
-1.502
-0.989
-0.310
0.266
1.379
ACS-FW
-1.20
-0.732
-0.019
0.503
1.471
Core-Set
-1.419
-1.071
-0.43
0.059
1.196
Core-Set
-13369
-1.030
-0.377
0.071
1.194
BADGE
-1.188
-1.150
-0.480
0.070
1,273
BADGE
-1.524
-1.089
-0.113
0.114
1.261
LCMD
-1.564
-1.116
00.55
0.097
1.270
LCMD
-1.496
-1.099
-0.433
0.074
1.219
(c)
Bradient-Boosted Decision Trees
Acquisition funetion
MAE
RMSE
95%
MAXE
Uniform
-1.691
-1.179
-0543
0.065
1.224
BALD
-1.445
-1.044
-0.379
0.143
1.217
BatchBALD
-1.484
-1.092
-0.435
0.080
1.167
BAIT
-1.85
-1.088
-0.433
0.095
1.191
ACS-FW
-0.999
-0.300
0.246
Core-Set
-1.600
-1.215
-0.566
-0.053
1.086
BADGE
-1.728
-1.264
-0.629
-0.048
1.157
LCMD
-1.253
-0.613
20

---

Published in Transactions on Machine Learning Research (7/2022)
Figure 6: DNNs: Error Metrics oter 15 regression datasets. We report mean absolute error (MAE), root mean squared error (RMSE), 95% and 99% quantiles, and the maximum error (MAXE), Averaged over 5 trials.
Mathed
4006
Aspiatie NAAI
1024
NAIT BADGE LOMD
512
250
Aed Fundin Utiloo
ACS BaacIBALD
(b) Acquisition Batch Sizes
010
-016
571 SENOO
1000
2044 4005
Appletton Woann
1104
Asquiaoe har sixc
512
250
286
126
64
107
5:05 Soy DERMI
40066
4099
Methed
2018
Traising Nusis
2048
1024
The Thiiing net
Aeg, Fundiin
RuchBBALDD
ACS-PR
BADGIE
512
512
IMLD
BATT
LOND
(a) Learning Curves
236
256
1.8
0.4
0.
122
agummh 266 Sog nrvon
4995
4096
2048
Masks
2048
Teinng est Masts
Training sct st0e Neahs
1024
1124
512
512
256
256
0.5
0.7
201 1022
Srraree 9:55 Sor urso
AXXXX
21

---

Published in Transactions on Machine Learning Research (7/2022)
Figure 7: Random Forests: Error Metrics over 15 regression datasets (cont 'd) We report mean absolute error (MAE), root mean squared error (RMSE), 95% and 99% quantiles, and the maximum error (MAXE), Averaged over 5 trials.
ACSUPW RADGE LCND
Auprattin NAT Nesto
1024
512
Asd Facibe
BARABALD RATT
250
(b) Acquisition Batch Size
0.5
0.2
011
0.0
4090
2044 400 246
Appieaton Woa
1104
Asquiaio Nar stx
512
512
512
256
126
64
102
1.35
Suuera 5656 Rag VERM
40096
4099
www. 2.0.
2014
Traising sct Nusiss
2048
Aog, Funddos
RaCABALD
ACS-TW
BADGE
RALD
1024
The Thaiing
512
512
(a) Learning Curves
236
256
0.0
0.0
-02
Aagmmn 266 Sig nroo
4995
4096
2011
2048
Masks
Teainig Masts
Training sct s30e Meahs
1024
1124
Training
512
512
256
bqpurnb 9:55 Sor mro0
SXCVIN LEEAA
22

---

Published in Transactions on Machine Learning Research (7/2022)
Figure 8: Random Forests (Bagging): Error Metrics over 15 regression datasets (cont'd), 'd). We report mean absolute error (MAE), root mean squared error (RMSE), 95% and 99% quantiles, and the maximum error (MAXE), Averaged over 5 trials.
4006
ACSUPW RADGE LCND
2004
Nasio
1124
512
Asd Facibe
BAKABALD RATT
350
125
(b) Acquisition Batch Size
-0A
0.5
124
0.3
02
01
000
4005
2044 400 286
20H
Nana
1004
1104
Asquiaio has sixc
512
512
255
64 126
1.5
Piaaons 98.66 Deg MRCEE
41996
4099
20048
Tris aing set size Murii
Aog, Funddos
RaCABALD
ACS-TW
BADGE
RALD
1024
Tra
512
512
(a) Learning Curves
256
256
0.0
8og 5965 auguno
BSIC Rog mom
4995
4095
2048
Training Net sibe Masaa
Training sct s30e Meahs
1024
512
512
28
255
3X0 Soy BEOAE
EIVI Soy uoo
aqpureb 9555 Sor uroaz
23

---

Published in Transactions on Machine Learning Research (7/2022)
Figure 9: Gradient- Boosted Trees with Virtual Ensemble: Ervor Metrics oter 15 regression datasets (cont 'd. We report mean absolute error (MAE), root mean squared error (RMSE), 95% and 99% quantiles, and the maximum eиror (MAXE). Averaged over 5 trials.
4606
ACSUPW RADGE LCND
2004
Nasio
1124
512
Asd Facibe
BAKABALD RATT
336
125
(b) Acquisition Batch Size
012
4005
403
204
2044 246.
1004
1104
Asquiaio har stxc
512
512
235
126
5.00
4006
4096
2018
Traisinng sct Nusis
2048
Aog, Funddos
RaCABALD
RALD
ACS-TW
BADGE
LCND
1024
1024
The Thaiiing
5122
512
(a) Learning Curves
236
256
0.0
-02
agurmn 2666 Sog nrem
4995
a096
2018
2048
ak
Teaining est Masts
Training sct so0e Meahs
1024
11244
Training
512
512
256
1.0
TVV 201 mom
beuurob 9:55 301 uroa
XXVVV DEIAA
24

---

Published in Transactions on Machine Learning Research (7/2022)
B.1 Comparison Neural Networks with Best- Best-Performing Kernels and Selection Methods
Table 5:
Best- pest-peronning white-box kernels and selection methods based on SotA methods according results from Holzmüller et al. (2022). Taken from Holzmüller al. (2022).
Based on
Selection method
Kernel
BALD (Houlsby et al., 2011)
MAXDIAG
BatchBAL.D (Kirsch et al., 2019)
MAXDET-P
BATT (Ash et al., 2021)
BATT-F-P
kgrad- (2k
ACS-FW (Pinsler et al., 2019)
FRANKWOLFE-P
Rigrad - a hype
Core-Set (Sener & Savarese, 2017), FF-Active (Geifman & EI- Yani,,
MAXDIST-P
ligrad ssketch(612)
2017)
BADGE (Ash et al., 2019)
KMEANSPP-P LCMD-TP
T.CMD)
Rggnd -skotei((11)
Table 6: Average performance of black-bor and white-boz O batch active learning acquisition functions using DNNs for Table 5. On average, for five acquisition methods, the black-box method performs better than the while-box method. Cl. Figure 11, which analyzes the final epoch.
Acquisition function
MAE
RMSE
95%
99%
MAXE
Uniform
-1.935
-1.401
-0.766
-0.163
1.107
BALD
-1.794
-1.389
-0.713
-0.221
0.946
BALD
-1.779
-1.372
-0.692
-0.191
0.979
BatchBALD
-1.915
-15122
-0.844
-0.349
0.866
BatchBALD
-1.865
-1.465
-0.792
-0.303
0.892
BAIT
-2.014
-1586
-0.928
-0.413
0.864
BAIT
-1.892
-1.489
-0.817
-0.328
0.881
0 ACS-FW
-1.979
-1.544
-0.892
-0.363
0.920
ACS-FW
-1.678
-1.668
-0.509
0.085
1.278
Core-Set
-1.988
-1.585
-0.926
-0.435
0.831
O Core-Set
-1911
-1.515
-0.845
-0.351
0.865
BADGE
-2.042
-1.779
-0.931
-0.383
0.948
BADGE
-2.009
-1.570
-0.915
-0.387
0.921
LCMD
-2.048
-1.609
-0.965
-0.437
0.874
LCMD
-2.033
-1.589
-0.940
-0.401
0.913
25

---

Published in Transactions on Machine Learning Research (7/2022)
Dutasets
slices diamonds
undir
online.vides
query road
sgemm
methune
stock
fried
mir_knn__nn
wec_sydney
vs Unifrr (
0.5
0.0
0.5
0.5 0.0
0.5
-055 0.0 0.5
0.5
-0.5 0.0 0.5
-0.5 0.0
0.5
-0.5
00.5 0.0 0.5
BALD
ACS-FW
BathhRALD
BAIT
Core. Set
BADGE
L.MD
Uniform (1)
Figure 10: Average Logarithmic RMSE by regression datasets for DNNs:
(us Uniform).
Uniforn (f)
0.0
0.5
0.5 0.0 0.5
0.5 0.0
0.5
0.. 0.0 0.5 BAIT
0.5 0.0 Core-Ser
0.5 0.5
0.5 0.0 0.5
BALDD
ACS-FW
BathhRALD
RADGE
1.CDD
(1)
Figure 11: Final Logarithmic RMSE by regression datasets for DNNs:
(us Uniform).
-088
-12
Acg Futction
Method
Uniform
BAIT
-13
-1.0
ACS-EW BuccBALD
BADGE
MO336
-12
mncan log ROSE
-15
-1.4
Ang. Fanciion
Malasi
-1.6
Thiforn
-17
BALD
Coce Set
BADGE
-188
BuchBALD
LCMD
256
512
1024
2043
4056
64
128
256
512
1024
2048
4296
Teaining
Acpistioon
-13
Acg, Vinction
Method
Unifomn
BuchBALD
-3.4
BALD
BALY
LCMD
ACS-FW
Cam-Sc
moca ing RANSE
-16
-1.7
27
22
27
25
25
27
Eascubli
Figure 12: Mean logarithmic RMSE over 15 regression datasets for Deep Neural Networks using the best- performing white-boz kernels and selection methods from Table 5. Even for the best-performing white-box kernels and selection methods as determined by Holzmiüller et al. (2022), the black-box approach outperforms the white-box approach overall.
26

---

Published in Transactions on Machine Learning Research (7/2022)
Figure 13: DNNs: Error Metrics over 15 regression datasets using the strongest white- box kernels from Table 5. We report mean absouut error (MAE), root mean squared error (RMSE), 95% and 99% quantiles. and the maximum error (MAXE). Averaged over 5 trials.
Mathed
4006
Aspiatie NAAA
1024
NAIT BADGE LOMD
512
250
Aed Fundin Utiloom
ACS BaacIBALD
(b) Acquisition Batch Sizes
01
-01
571 1RAS
0000
4005
2044 286
Appletto Nuann
1104
Asquiaie hat sixc
512
250
126
66
33VE Rry CETEE
2:55 Bay DERMI
40066
4099
Methed
2018
Traising sct Nusis
2048
1024
1024
The Thaiing sdt
Aeg, Fundinn
ACS-PR RuchBBALD
BADGIE
512
512
IMLD
BATT
LOND
(a) Learning Curves
236
256
1.8
0.4
0.2
102
004
agammh 266 Sig nrvo
4995
a096
2018
Masks
2048
Teiinng est Masts
Training sct st0e Neahs
1024
1124
512
512
256
256
0.5
0.7
201 1E2E
Srraren 9:55 3or urom
AXXX
27


Reviews:
[{"review_id": "q8wi8k2cz1", "reviewer": "Reviewer_hT9S", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "STRENGTHS:\n* The scope of the proposed method is broader than many existing methods because it is effective in non-differentiable settings.\n* The proposed method is mathematically sound, the derivation and presentation are clear, and the formulation fits in nicely with recent work that provides a unified perspective for active learning.\n* The relatively minor drop in performance for the proposed method when compared to white-box methods are an encouraging demonstration that the proposed method is effective and useful.\n\nWEAKNESSES:\n\n* The term \"black-box\" is used in different ways in the literature and the meaning in this paper is not fully specified. I assume that \"black-box\" as used in this paper means that the model regression predictions are available, but not model parameters or gradients. Is that right? If black-box means model gradients are inaccessible and the parameters themselves accessible, it seems like MC-dropout and related techniques could be considered black-box. Clearly defining \"black-box\" is essential for the reader to understand the context.\n* Not enough efforts were devoted to presenting other black-box active learning techniques, or at least basic adaptions of existing techniques into a black-box setting. For example, the Core Set method could be used with an auxiliary network (Inception, VAE, etc.) to encode embedding distances even if the target model parameters are unknown to obtain the intermediate embedding. The results would be more compelling with comparison to other black-box baselines, instead of comparing the proposed black-box results only to white-box results. Similarly, for random forests models, it might be helpful to investigate an MC-dropout type method that looks at the predictions of each tree in the forest as a baseline for comparison (even though this would be white-box) to show performance for non-differentiable white-box vs. non-differentiable black-box.\n* The decision to investigate regression rather than classification and regression seems somewhat arbitrary and was not strongly motivated.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work investigates the problem of batch active learning for regression problems where the user has black-box access to a neural network (I assume this means that the model parameters, and therefore related information like gradients are unknown, and only predicted regression outputs are known). A Bayesian prediction kernel based on multinomial sampling is presented and it is shown that this kernel can approximate the covariance of predictions for two input states. This observation, along with a recent framework for active learning from given a kernel definition, defines a black-box active learning method. Experimental results show that the proposed method can nearly match white-box active learning for deep learning across several different baseline methods, and also that the proposed method is effective for non-differentiable models like random forests and GBMs.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "There should be more efforts to compare the proposed method against existing active learning techniques in a black-box setting (possibly with minor modifications). It would also be helpful to discuss which existing methods can be applied or easily adapted to a black-box setting and which necessarily must be applied in the white-box setting (in which case this work provides a novel implementation method).", "broader_impact_concerns": "No section on broader impacts is included, although this does not significantly affect my assessment.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "WYFLRM70qy", "reviewer": "Reviewer_7KgK", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n1. I thought the paper was overall well-written and easy to read. I liked the structure / way concepts were built up.\n2. The idea itself is simple (look at prediction-space instead of model embeddings or derivatives). The simplicity is a good thing. (Disclaimer: I am assuming people have not tried this method / similar methods before in this field.)\n3. The link to non-differentiable models is neat and a strength of the paper, regardless of how raw performance of the models compare to neural networks.\n\n--------------------------\n\nWeaknesses\n\n1. Despite what is said in Strength 2 above, Proposition 3.1 is not new or unexpected, with similar arguments used in the past in non-active-learning works, where the Gaussian approximation / Laplace approximation / linearisation around w* (+ Gauss-Newton approximation) leads to the simple relationship between prediction-space and parameter-space. And very similar arguments, as cited in this paper, were used in Holzmuller et al. (2022). I therefore do not view Prop 3.1 as particularly surprising, rather just a nice property of this method that exists. (This is not a weakness as such, more a comment on how interesting this part is.)\n\n2. Overall I found the experiments section severely lacking: the experiments are enough to show some evidence for why the proposed method might be good, but does not do enough to explore properties of the method. More details / suggestions:\n(a) There are no baselines (aside from uniform sampling) in the non-differentiable models experiments. Why is this? If no suitable baselines exist at all in the literature (I am not aware of any, but I am not an expert in active learning), then perhaps some visualisation (or more explanation) of the intuition provided ('the virtual ensemble might not capture disagreement well enough') would be helpful. I found this section (Experiments with non-differentiable models) very short.\n(b) The experiments show B3AL generally outperforming the white-box methods, but I still do not really know much about its properties. Figure 2 is nice (showing a correlation between performance), but much more can be done. For example, how does performance change as number of ensembles changes? The authors plot only the mean performance, but how high is the standard deviation across different random seeds (as presumably the set of ensembles / disagreement between ensembles changes)? What is the computation cost of B3AL (with 10 ensembles) compared to the white-box versions? \n(c) I notice in Figure 2 that most method/dataset combinations have performance very similar to uniform sampling, with only a few datasets per method performing better than uniform. Figure 1a seems to be hiding this information by taking the mean performance. I think this should be commented on in the paper at least.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces a kernel for batch active learning for regression which uses model predictions instead of model parameters (/gradients of model parameters/similar). This allows the method to be applied to non-differentiable models too, like random forests, and means the method no longer scales with number of parameters/final-layer features. The authors show how their method is related to the previous white-box approaches, and compare performance on the benchmark from Holzmuller et al. (2022), where the black-box method usually out-performs the white-box version, and where there is correlation between performance. ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see Weaknesses section for the more major requested changes. Especially Weakness 2 and the experiments section.\n\nList of minor changes / typos:\n- 2nd line of Related Work: use \\citet{} instead of \\citep{}\n- The authors can give a one-line summary of acquisition function at the beginning of the Related Works section itself instead of later (such as 'it measures the informativeness of...' on page 3).\n- page 5 second paragraph: 'paramters'\n- page 5 just before Sec 3.2: 'examind'\n- Equation 18: 'sim' vs '\\sim'\n- I'm not sure the prior was defined as p(\\psi) before Equation 29\n- Page 8: 'Above gradient kernel...' -> 'The above gradient kernel...'\n- Figure 2 caption: 'markers left of the dashed vertical lines' -> 'right of the...'\n- More important: Figures 1b and 1c are very difficult to read: the font size should be bigger. Also, in general in Figure 1, there are too many overlapping lines, and presentation could be improved.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "wOHFLxOvaL", "reviewer": "Reviewer_pr7j", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The proposed method based on prediction-based kernels is nicely motivated and positioned with respect to the previous works.\n- The method, although simple, indeed leads to well-performing black-box active learning methods which are empirically shown to be competitive with their corresponding white-box methods. So the proposed technique can be a useful contribution to the field of black-box active learning and a nice baseline for future works in this area.\n\nWeaknesses:\n- I think the authors should discuss why the predictive covariance kernel (and also the empirical one) is a positive semi-definite kernel. The posterior gradient kernel from previous works is PSD since it's an inner product in some feature space, but the predictive covariance kernel proposed in this work isn't clear to me.\n- The writing of the paper is clear overall, but there are some places in the technical sections which can use some more polishing and better explanation, to make it easier to understand for users not familiar with the field. For example, first line of Section 3.2.2., are $\\omega_1,\\ldots,\\omega_K$ the model parameters of $K$ models from an ensemble? More importantly, I find the explanation in Section 3.2.4 not easy to understand, so perhaps this section can be made clearer. In equation (28), are you saying that in this case, every member of the ensemble is a model resulting from BMA using equation (28)? What's the dimension of $\\mu(x;\\cdot)$ in equation (27)? Does the symbol $\\psi$ represent an index from $1,\\ldots,K$ (as in equation 28) or a vector (as in equation 27)? Overall, I suggest that in Section 3.2.4, whenever you explain a term (e.g., equation 28), you can immediate point out what does this term correspond to in Sections 3.2.1-3.2.3, this makes it easier for readers to see the big picture.\n- Can add error bars in Figure 1? With error bars, it's difficult to see whether the differences are significant.\n- I find Figure 2 difficult to read. For example, initially I thought the black square in the top left corner refers to the area in of the top left quadrants, which leads to inconsistent interpretations with those from the main text. It took me quite a while to figure out that it actually refers to the entire area above the y=x line. The caption says that on average, the black-box methods perform better the the white-box methods since the stars line in the top left area, but I think that all stars lie pretty close to the y=x line. But I understand that this also shows the strengths of the proposed method since it allows black-box methods to perform on par with the white-box ones. So, I think some modifications are needed for the presentation of Figure 2 and its caption.\n- Fourth last line on page 1: \"such\" -> \"such as\"", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper has proposed a batch active learning method for black-box (potentially non-differnetiable) models in regression tasks. The proposed method is based on a recent development of the kernel-based perspective of active learning and introduces prediction-based kernels, and hence can be used to modify various existing active learning algorithms for the black-box setting. Empirical evaluations verify that black-box methods perform comparably with or even surpasses their white-box counterparts.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "A small suggestion in addition to the points listed under Weaknesses above:\n- One line above equation (1): when can $\\Omega$ be bootstrapped training data? I find this section on page 3 much easier to understand if you just say that $\\Omega$ represent the model parameters, so you can perhaps just say that $\\Omega$ is the model parameters, and then say (perhaps in a footnote) that in some special cases, it can also be others as well such as bootstrapped training data.", "broader_impact_concerns": "I do not have any concern on the ethical implications of this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Xc4rzxWYbA", "reviewer": "Reviewer_h9yU", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n1. Clear motivation about why black-box active learning can be useful\n2. Detailed explanation about related background and the method\n3. Experimental results look good.\n\nWeaknesses:\n1. It's not clear why this paper focuses on regression tasks. Classification tasks are also very popular (if not more popular) than regression tasks. \n2. It's not clear why the (proposed) black-box methods can outperform white-box methods. Intuitively, white-box methods can access more information (model internals, such as architecture, or weights), thus better suited to the specific models. I am not sure why the black-box methods can outperform white-box methods as shown in Figure 1.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes black-box active learning for regression task. Existing works on white box active learning can only be applied to white-box models. This paper addresses this limitation by proposing black box methods which is applicable to black-box models. Experimental results showed that the proposed black-box methods can outperform existing white-box methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please clarify my concerns in the above Weaknesses ", "broader_impact_concerns": "none", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper was reviewed by four reviewers. They find the paper well-written, and the proposed method novel, interesting, and simple. They raised some concerns about presentation and experiments, as well as limitation to regression setting. The authors have successfully addressed all their concerns in the rebuttals. In the end, all the reviewers are positive about the paper. 

I would like to clarify the timeline a little bit. On April 18, the authors reported that they were still running some ablation studies. On May 15, the authors uploaded the revised version of the paper. The delay was caused by some personal issues of the first author, which are completely understandable. I appreciate the efforts in the revision. 
Decision: Accept (poster)