Targeted Active Learning for Bayesian Decision-Making
Authors: Louis Filstroff
Iiris Sundin
Petrus Mikkola
Aleksei Tiulpin
Juuso Kylmäoja
Samuel Kaski
Venue: Accepted by TMLR ()

Abstract: Active learning is usually applied to acquire labels of informative data points in supervised learning, to maximize accuracy in a sample-efficient way. However, maximizing the supervised learning accuracy is not the end goal when the results are used for decision-making, for example in personalized medicine or economics. We argue that when acquiring samples sequentially, the common practice of separating learning and decision-making is sub-optimal, and we introduce an active learning strategy that takes the down-the-line decision problem into account. Specifically, we adopt a Bayesian experimental design approach, in which the proposed acquisition criterion maximizes the expected information gain on the posterior distribution of the optimal decision. We compare our targeted active learning strategy to existing alternatives on both simulated and real data and show improved performance in decision-making accuracy.

Published in Transactions on Machine Learning Research (06/2024)
Targeted Active Learning for Bayesian Decision-Making
Louis Filstroff
louis. louis. jitiiuutttrrf@@eemmmeeelllllll
Univ. Lille, CNRS, Centrale Lille, UMR 9189 CRIStAL, F-59000 Lille, Fronce
liris Sundin
surds.n @iki.
Deportment of Computer Science
Aalto University, Pinland
Petrus Mikkola
petrus. metrus..ikkolia@mma... com
Deportment of Computer Science
Aolto University, Finland Universily of Helsinki, Finland
Alcksei Tiulpin
Research Unát of Health Sciences and Technology University of Oulu, Finland
Juuso Kylmãoja
juuso, kouso.kylmu@@mmii.coom
Department of Computer Science
Aolto University, Finland
Samuel Kaski
kamuel.aakk@aall
Deportment of Computer Science
Aulto Universily, Finland
Reviewed on OpenReview: https: openreview. net/ forum? id= KaPjuuMgmm
Abstract
Active learning is usually applied to acquire labels of informative data points in supervised learning, to maximize accuracy in a sample-efficient way, However, maximizing the supervised learning accuracy is not. the end goal when the results are used for decision-making, for example in personalized medicine or economics. We argue that when acquiring samples soquentially, the common practice of separating learning and decision-making is sub-optimal, and introduce active learning strategy that takes the down-the-ling decision problem into account. Specifically, we adopt a Bayesian experimental design approach, in which the proposed acquisition criterion maximizes the expected information gain on the posterior distribution of the optimal decision. We compare our targeted active learning strategy to existing alternatives on both simulated and real data and show improved performance in Docision-making accuracy.
1 Introduction
Supervised learning techniques aim at learning a function that maps the input x E X' to the outcome (or label) E Y,, based on collection of examples D = {(x, (:),, ,. Whereas having access to thousands of unlabeled data is nowadays easy, obtaining the associated labels is expensive in many applications, such as those involving human experts (e.g., image annotating), or running additional experiments. In this context, active learning (AL) aims at itcratively querying for the most informative data point among pool of unlabeled data (Settles, 2012). In the machine learning literature, the term "active learning" often implies a classification task, but the concept straightforwardly extends to regression, and the same problem arises in

---

Published in Transactions on Machine Learning Research (06/2024)
the statistics literature under the names "optimal experimental design" or "Bayesian experimental design" (BED) (Chaloner & Verdinelli, 1995; Ryan et al., 2016).
Active learning boils down to the selection criterion for the next point to label. Popular strategies include uncertainty sampling (Lewis & Catlett, 1994), expected error reduction (Roy & McCallum, 2001), or expected information gain on the model parameters (Lindley, 1956; MacKay, 1992). All these strategies aim learning a model as accurately as possible with as few queries as possible. However, the accuracy of the model is not the end goal in all scenarios. In this paper, we consider the setting where the model is used for decision- Each action (or decision) is assessed by its utility, and the optimal decision is the one that yields the highest expected utility.
We focus on a decision-making task where the goal is to select the optimal decision for a single test point, while we also extend our method to the case of a test population. Such a scenario arises, for instance, in the topical field of personalized medicine. Basod on the history of previous paticnts, described by patient covariates, the treatments they received, and the observed oulcomes, a model is built to infer the individualized treatment. effect (Wager & Athey, 2018; Shalit et al., 2017; Alaa & van der Schaar, 2017; Yao et al., 2018; Bica et al., 2020). A doctor will then use the model's predictions to choose the best treatment for new patient. In this motivational example, the active learning task addressed in our paper involves identifying the most informative patieetttretm pairs from a predefined pool, which may include sources such as electronic health records (EHR). Accessing data from these records involves stringent legal justification due to privacy concerns, mandating that a physician must have a legitimate reason for retrieval and use in patient treatment. The access process can be lengthy and incur significant financial costs, or the administrative costs per patient can quickly accumulate, becoming intolerable for large datasets.
Traditionally, model learning and decision-making are carried out separately, i.e., the learning phase is blind to the decision-making problem. This is sub-optimal when data can be collected actively, and as such, there is a need for active learning strategies that take into account this downstream decision-making task. This problem of decision-mmainng-no aware active learning has recently received attention by Sundin et al. (2019), who proposed a heuristic strategy for a binary decision--akinn problem. However, their criterion does not. extend to more complex situations, such as multiple-decssion problems, which limits applications.
In this paper, we propose principled selection criterion for decision-making-awaan precisely, we adopt a BED approach, where it is a well-known fact that the opti strategy is to perform expected information gain on the quantity of interest (Chaloner & Verdinelli, 1995). In our setting, we identify that quantity as the epistemic uncertainty on the optimal decision; in other words, the proposed criterion will aim at maximally reducing the uncertainly of the posterior distribution ion of the optimal decision. This is unlike classical BED approaches, which either target model parameters or outcomes. The effect of the proposed methodology versus classical ones is illustrated in Figure 1. Our active learning criterion is optimal in the sense that it maximally reduces uncertainty about which decision yields the highest expected utility for given test point.
active learning. More
We consider performance improvement at first t additional acquisitions for small 1, which is a key measure for scenarios where acquisitions are costly, such as personalized medicine and generally having a human in the loop. Specificallyy our method is applicable when a) the test population of the decision makiing task is known: a single individual x (as in personalized modicine), or a set of individuals, and b) it possible to collect more data on-demand, for example, once the model has been deployed, but e) these queries are costly due to, e.g., requesting new experiments, involving experts, or fulfilling privacy constraints. We empirically demonstrate the advantages of the proposed method with respect to existing AL basclines, both in simulated and real-world experiments.
2 Problem formulation
2.1 Modeling of outcomes
We consider a regression setting with covariates x E Rp and outcomes y R. We further assume that the outcome also depends decision variable d E {1,... K}. Typically, the outcome is observed after an action
2

---

Published in Transactions on Machine Learning Research (06/2024)
P-EIG query
26
34
Posterior probability of the optimal decision
22
-2
0.8
EIG query
12
0.4
34
22
38
0.0
Our query
71
112
ITg
32
38
Figure 1: Mustratitv example of a decision-making task: choose decision from d d= (1,2,3) at point & == -1 (black dashed line). (Left): The function fx models the outcome for decision d = k based learning dataset (colored lines with uncertainty intervals). The learning dataset consists of labeled (x marker) and unlabeled (* marker) data points. (Center): The posterior distribution of the optimal decision helps in making the decision (here the Bayes-optimal decision is d = 1), and assessing its uncertainty. (Right): Evolution of that distribution after querying one additional point. Using the standard EIG criteria (Sec. 2.3) does not help the decision-makin (top, middle), while the proposed targeted AL criterion greatly improves it by reducing its uncertainty (bottom).
3

---

Published in Transactions on Machine Learning Research (06/2024)
has been taken. In the healtheare application, this corresponds to observing the effect of treatment d on patient. We therefore have training set 'D comprising triplets, i.e., D = {( di, Di) )
Denoting by the variable yl(d = k), the goal is therefore to learn the functions fa which map x to yk- In this work, we assume that the y/ are conditionally independent given x, and write
yk = [x) + Ck:
(1)
where OK ~ (N
Moreover, we assume that we are equipped with a functional prior distribution on fk. such as a Gaussian process (GP) or Bayesian neural network, which in turn allows us deal with posterior uncertainty. Indeed, given D2 ( Ut) D d; k), and using the notation fk, to denote fx), we may characterize the posterior distribution p( (Dx) for all x. Lastly, we treat of a hyperparameterr to be estimated with, e.g., maximum marginal likelihood.
2.2 Decision-making problem
For clarity of presentation, when introducing the method we will focus on a single test input, which we denote by &, rather than test population. Nevertheless, the developments presented in the paper can straightforwardly be extended to test population, we will describe in Section 3.2.
The input X is previously unseen data point for which the end-user of the model has to make a decision, i.e., has to choose one among the set of the K available decisions. In our introductory example, x corresponds to the covariales of a patient for whom the doctor has to choose a treatment.
In this context, decisions are assessed through a scalar utility: the higher the better. Utilities can be computed from the outcomes Uk: we write uy = (ūk)), where the r& are known, deterministic functions that map the outcomes to the utilities. In the remainder of the paper, we will assume, without loss of generality, that y.
Given that the models have been trained on D = UR we assume that the user behaves optimally in the sense of (evidential) decision theory, i.e. chooses the decision which yields the greatest expected utility at x. The Bayes-optimal decision' 1 is
/f (iptfihaaalllfhiiDddaaaaaa. ke (1......K)
dBAYES
arg max
(2)
2.3 Bayesian active learning
We assume access to a pool of unlabeled data U = dz))/=)) from which the associated outcomes can be actively queried. We wish to select queries from 24 which are maximally useful for the decision decision-making problem, i.e., queries that reduce uncertainty on the optimal decision for Our primary example involves a scenario where information about the outcome for an unlabeled data point can be retricvod from a database (e.g. containing Electronic Health Records, EHR).
Conventional Bayesian experimental design strategies
Let us consider a standard Bayesian regression formulation, i.c., when the relationship between the input x and the outcome is modeled by likelihood p(y|x, 0), where 0 are latent parameters with prior distribution p(0). We wish to decide on the next point to query. The principled strategy from an foormation-thhoreee perspective is to look for the query that maximizes the expected information gain (EIG) on a quantity of interest, which is defined as the expected reduction the entropy of the posterior distribution of the quantity of interest.
Due the assumption of identity mility function y, this corresponds to risk-neutral Bayes-optimal decision.
4

---

Published in Transactions on Machine Learning Research (06/2024)
Typically, the parameters chosen to be that quantity; we refer to that strategy as P-EIG. In that case, the optimal query is such that
EIG = arg max (Hlp(0|D)] XyEX'
(X) [(), ((x))1))111)
(3)
where we used the notation M[p(.)] to denote the differential entropy of probability distribution (Shannon, 1948). This idea was first suggested by Lindley (1956), and has then been considered by several authors, see for example Bernardo (1979); MacKay (1992); Houlsby et al. (2011); Berrández-Lobatt et al. (2014). Morcover, the criterion of Eq. (3) can be rearranged in a form that computes entropies in the outcome space rather than the parameter space (this has been coined "BALD" by Houlsby et al. (2011)), which allows define it in non-parametric setting (i.e., when fxx) + c):
XP.IIG arg max H[p(vs|x, D) - Bp((D) [M[p(yy)x) fx). X,EX
(4)
see details on the equivalence in Appendix B. Nonetheless, the EIG remains challenging criterion to compute, as it involves nested Monte Carlo estimation (Rainforth et al., 2018), and several recent works have aimed at mitigating this issue (Foster et al., 2019; Zheng et al., 2020).
The other standard option is to consider the quantity of interest to be g. the outcome at a specific & (i.e., not belonging to the unlabeled set) (Krause al., 2008; Dace et al., 2017; Sundin et al., 2018). Let us refer that strategy O-EIG. The optimal query becomes
EOG arg max H[p(ö1&, D)] ()) [[[(()] DU ((x),,))
(5)
X, Ex
Shortcomings for the decision-makee problem
The conventional Bayesian AL criteria of Eqs. (4) and (5) can easily be adapted our setting, and would select the element of U which yields the highest information gain over any of the Sky over any of the 7h. respectively. However, such queries are necessarily helpful to improve the quality of the decision-making Indeed, they may have little to no impact the posterior predictive distributions at x, or, may improve predictions only for decision that has very little probability of being the optimal onc. Such phenomena are displayed on the right panel of Figure 1.
Thus, we present in the next section an active learning strategy which takes the decision-making problem into account by considering the posterior distribution of the optimal decision for x, and which therefore overcomes the aforementioned shortcomings.
3 Targeted active learning criterion
3.1 Posterior uncertainty on the optimal decision
The optimal decision is, by definition, the with the highest expected utility. If we knew the value of fe. exactly for all k; then the optimal decision would be known with 100% certainty. However, since work with a finite sample size, we cannot have access to the value of fR.*, but instead, we characterize posterior distributions p p(&), which in turn leads to the Bayes-optimal recommendation of Eq. (2). As it turns out, by fully taking advantage of the Bayesian framework, we can go beyond mere point recommendation and estimate the posterior uncertainty that decision & is the optimal decision (at x).
Let denote by Thk the posterior probability that decision k is the optimal decision. We further define Doese (x) to be the discrete random variable whose probability mass function is given by the (T+) In other words, Dhest (x) contains the posterior uncertainty on the optimal decision for x. We have
=P
(E(ūxlfk.) max E(UN Sw(x) swxx))
(6)
5

---

Published in Transactions on Machine Learning Research (06/2024)
In the equation above, the conditional expectations are to be understood random variables, and fR. p(J),x(Dz). Given the model assumptions, it turns out that B()u(Jkxx) = SK, We can further write
Th P
fex max fW(x)
(7)
P n K'H
(fe fext
(8)
The events inside Eq. (8) are not independent, and such this cannot be broken down into product of probabilities. An illustrative problem with 3 decisions is displayed on Figure 1, with the current models in the left panel, and the associated probabilities in the middle panel.
It important to note that the randomness of Ja, only comes from lack of information. Such uncertainty is said to be epistemic, and adding more points to the dataset. will reduce this uncertainty (Hüllermeier Waegeman, 2021). Therefore, is more precise to characterize Dbeat (x) as the random variable containing the epistemic uncertainty the optimal decision. We argue that this is the variable of interest in our setting.
3.2 Decision-targeted active learning criterion
Now that we have characterized the posterior distribution of interest (the distribution of the variable we called Dbest())) propose to sequentially select. the data point from U which maximizes the expected information gain about this posterior distribution. We write
(x", dd) arg max (x,,
H[pp(Deest (x)[D)
(9)
((xx, dj, da, bo,))11]
- -Eply,, (xy.
which means that these queries aim at reducing the uncertainty on the optimal decision of x. The criterion of Eq. (9) may be rewritten in simpler form as
(x", d') = arg min Selvi,lxy,0 (xg.d))el
{(x,, dz Na, H)]]
(10)
The full decision-makkig-aawee active learning process illustrated on Figure 2.
Extension to test population
We briefly consider here the scenario where there a collection of previously unseen testing points (x,) ) for which we wish to improve the decision-making. The optimal decision for X; be understood, as before, the one which yields the highest expected utility &; Similarly, we can define Dbeat(x;) as the discrete random variable which contains the (epistemie) uncertainly the optimal decision for each X
The extension of Eq. (10) simply consists in considering the entropy of the joint posterior distribution of Duest (XI),.... Doest (x..). However, this quickly leads to computational issues, as the cardinality of the space is now KNN To alleviate this issue, we propose to minimize an upper bound of the entropy instead by applying the chain rule for differential entropy, which leads to the following criterion
N Hlpp Dpest (b) [(x) dj. 11))
(x",d" d²)) = arg min Extsa, 1x, Day)) (x)
Σ
(11)
The tightness of the bound depends on the degree of dependence between the optimal decisions and the test points. For instance, the optimal treatment is similar for all paticnts, the bound not very tight. Conversely, if the optimal treatments are highly specific to individual patients, the bound becomes significantly tighter.

---

Published in Transactions on Machine Learning Research (06/2024)
Unlabeled pool
D-M-Aware Active Learner Selects query (x", ,d'
Oracle Provides label y*
Posterior of the optimal decision
Add (x*,d",y")
Possible decisions
Training set
Model
Decision-maker
Takes decision with highest expected utility
Figure 2: Decision-man Docision-making-nnee making-aware active learning. Agents are in blue boxes. The active learner is aware of the down-the-line decision-makinn problem, and selects targeted queries for the problem, by taking into account the posterior distribution of the optimal decision. Once the learning phase is over, here assume that the end-user takes the action that yields the highest expected utility.
3.3 Practical implementation
Computing the criterion Eq. (10) requires to solve two computational challenges:
The expectation w.rtt. p(ya, xx,, Da,) is intractable and needs to be approximated;
2. The probabilitics Na not known in closed form either. They need to be estimated in order to compute the entropy of Drest
To approximate the expectation, we resort Monte Carlo approximation. This means that given N, samples ()) drawn from p(ya, [x,, Da,), we have
yd,
Epiva, (xy. (X ((xx, dy, dd, Hb]]]
N,
H[p(D (oooDDee())DUU ((x,, dj va }}}}
(12)
Note that when p(ua, Xx,, Da,) is Gaussian, as is the case in GP regression, we may use a Gauss-Hermite approximation scheme (see Appendix C).
Next, to compute the entropy H[p( DiepDDes (X))),, we nced to know the posterior probabilitics x (given by Eq. (8)). Unfortunately, closed-ffrm solutions do not exist in general, and we resort to a straightforward approximation scheme. We take sets of posterior draws from p(fx.x (D&) for all & to generate posterior samples of (x,s(()), which are then used to estimate the entropy. For simplicity, we estimate the entropy of the multinomial distribution by using empirical estimates of the n from the posterior samples of D (x).
Pseudo-code the algorithm computing the proposed targeted AL criterion is given in Algorithm 1. All the computational burden resides in the model retraining step, which has to be carried out N, x card((f) times solve the optimization problem in Eq. (10). The computational complexity high, but many operations are trivially parallelizable, for example over all elements of U, or even over all Monte Carlo samples. Moreover,
7

---

Published in Transactions on Machine Learning Research (06/2024)
pre-selection strategies may be implemented to avoid computing the criterion for all elements of U, or the selection problem itself could be cast as Bayesian optimization problem.
Algorithm 1 Estimating the criterion Eq. (10) for (xj, dj) E U
C=0;
D Current estimale of Eq. (10)
> Monte Carlo approzimation
for 1= 1=.,..,,NN NN, do
Ddy ~ ~p(ya, sp(u,,1x,,20 (x;)
(1)
(8)
Add (xg, Vá. to the training set Dd,
Optimize GP hyperpareneters associated with decision dj
b Estimale entropy of Dreat(8) with augmented dalasel for -....., K do
Get samples from x(f
end for
Compute estimates of the Ti Compute entropy H. from the Tk
C=++N-1 ( from the training set Daj (x).
Remove (x,,
())
end for
Lastly, emphasize that working with test population, i.e., dealing with Eq. (11) instead of Eq. (10), brings negligible additional computational complexity. Indeed, the only difference is that we would have to estimate several entropy values instead of one, which has a negligible cost compared to retraining the model, as previously stated.
4 Related work
Decision-making-awarr strategies in machine learning. We begin by discussing such strategies in passive learning context. Lacoste-Julien et al. (2011) introduced the so-called loss-calibrated inference framework. They characterized the decision-making problem by loss (i.e., negative utility), which is taken into account to alter the learning objective of variational inference. This work has been extended, e.g., to Bayesian neural networks (Cobb et al., 2018) and to continuous decisions (Kuśmicrczyk et al., 2019). Another line of work, which lackles the computation of expected functions (w.r.t. posterior distribution), is discussed by Rainforth al. (2020). The authors argued that when these functions are known in advance, it is beneficial to take them into account and subscquently proposed framework coined TABI (target-aware Bayesian inference), which enables efficient estimation of such quantities.
Surprisingly enough, the literature is quite sparse when it comes to similar strategies for active learning. Saar-Tsechansky & Provost (2007) proposed two houristics to help choosing which customers to target in marketing campaigns. More recently, Sundin et al. (2019) proposed an active learning criterion based on the Type-S error to improve binary decisions. Several recent works tackled goal-oriented active learning, but none of those consider the pociiiinnmmainggg step that comes after the learning process. For instance, Yan et al. (2018) proposed debiasing query strategy based disagroement- based active lcarning when learning classifiers from logged data (where the labels have been revealed according to logging policy, leading to biased training sets). Their work is also limited to binary decisions. Kandasamy et al. (2019) introduced a reward function and a method based on posterior sampling, and Xu & Kazantsev (2019) introduced a utility function and the use of so-called influence functions, but the words "reward" or "utility" there refer to different metrics of model evaluation. Finally, Zhao al. (2021) proposed an uncertainty-awan AL criterion for classification with 0-1 loss, which focuses only on the reduction of the uncertainty that pertains to the classification error.
Best arm identification in multi-armed bandits. The decision-male -aaking problem we consider can equivalently be presented as the problem of identifying which of the K arms, described by the distributions
8

---

Published in Transactions on Machine Learning Research (06/2024)
of the utilities of each decision at X, is the best (i.e., yields the highest expected utility, or reward). This is known in the multi-armed bandits literature as the "best arm identification" problem, or "pure exploration" problem, which has been studied both from frequentist and Bayesian perspectives (Audibert et al., 2010; Kaufmann et al., 2016; Russo, 2016). A generalization of this problem has recently been introduced under the name "transductive bandits" (Fiez el al., 2019). The objective of such problems differs from the traditional setting of multi-armed bandits, which is to maximize the cumulative sum of rewards.
However, the setting of best arm identification problems differs from ours in the possible arms that can be sampled. In contrast to these problems, we cannot sample from the different arms at X. We can only sample once from a specific set arms defined by the pairs (x, d) E U. This prevents us from using strategies from the multi-armed bandits literature. Instead, by adding new points to the regression models, we aim at better characterizing the distributions of the expected utilities at X.
Bayesian optimization and active learning. Bayesian optimization (BO) refers to class of algorithms for global optimization of black-box functions, where a probabilistic surrogate model such as a Gaussian process is placed on the objective function (Brochu et al., 2010; Garnett, 2023). BO algorithms sequentially select points where the objective function is evaluated, based on some acquisition function that typically balances exploration and exploitation. As such, BO is closcly related to AL, see for example Ling et al. (2016) for a unifying framework of some standard AL and BO algorithms. Conceptually, BO can be seen as a goal-oriented AL strategy, but for the specific decision-making problem of choosing a point that maximizes black-box function. Many acquisition functions can be derived using Bayesian decision theory by choosing an appropriate definition for the utility of the gathered dataset u(D) (Garnett, 2023), which should not be confused with the utility of the outcome discussed in Section 2.2.
Entropy-search multi-fidelity Bayesian optimization Multi-fidelity Bayesian Optimization (MFBO) extends traditional BO by incorporating auxiliary information sources, also termed as low-fidelities (Kandasamy et al., 2016). These fidelities offer a cost-effective means of gathering insights about the objective function, which serves as the primary information source. A strong link exists between the proposed method and an entropy-search based MFBO, which we will explore next.
Consider family of functions fx : [K] + E. indexed by the information sources x E X. The primary information source and the corresponding objective function can be defined as Xtest and fxieus respectively. The auxiliary information sources and their associated functions are denoted by x * Xtest and fxexxent respectively. Our restriction that querying fx (d) for any d E [K] is not possible can be interpreted as the cost of fxi.re (d) always exceeding the total budget. This restriction also applies to any query (x, & U. For any (x, d) E U4, the cost is a constant budget/ T where T is the total number of iterations. The MFBO acquisition problem can thus be formulated as follows: Which information source x and which point d should be chosen in order to maximally assist in finding max fxc.cc (d)?
If we adopt policy aiming to maximize the information gain of arg ma ().....) fxcea (d) (i.c., we set w arg max.e(1....K) fxies Garnett (2023), Equation (6.8)), we are close to the Predictive Entropy Search (PES) method proposed by Hernández-Lobate et al. (2014). However, PES was initially formulated in single-fidelity BO setting. In the literature, there are extensions of entropy-search based methods to multi-fidelity settings, such as the Multi-Fidelity Maximum Seeemm-Ettto-y-aaae
(MF-MES) method introduced by Takeno et al. (2020). Therefore, our acquisition criterion (9) can be interpreted as an MF-PES method. While this specific method is not directly identified in existing litcrature, can be conceptualized as a combination of PES Mierááddez-Lbbtte al., 2014, Equation (2)), and for instance MF-MES (Takeno el al.,
2020, Supplement, Equation (18)).
What remains to be addressed is the design of the multi-fidelit GP. The MFGP kernel can be specified as k((x, d), (x',d") d')) = Σa (x x') <()d d'), where x(d,d') I(d dd) and 6a(x, x') corresponds to the kernel chosen in Section 5.2. With this MFGP model, our method aligns with MF-PES when the multi-fidelity setting is framod as describod above. It is noteworthy that the derived MFGP kernel is unconventional, as it treats the information source kernel ka(x,x) x') as dependent on the data point, with distinct hyperparameters for each d E {I,... KJ. This flexibility allows for more nuanced learning of the correlations between various
9

---

Published in Transactions on Machine Learning Research (06/2024)
information sources x and each d. We hypothesize that this inductive bias, combined with a decision-focused (d-focused) acquisition criterion, may contribute to the superior performance observed in our experiments.
5 Experiments
5.1 Use-cases and datasets
Fully synthetic data. We proceed to generate dataset of 400 points of dimension 5. The covariates are drawn from the slandardized Gaussian distribution. We generate four different outcomes as independent realizations of GPs with squared exponential kernels whose variance and lengthscales are different. These outcomes are then corrupted by Gaussian white noisc. Finally, the decision variable associated with cach point is drawn randomly, but not uniformly, to mimic the imbalance in treatment assigument.
Treatment recommendation. The first use-case focuses on the topical personalized medicine rescarch question of using electronic health records (EHRs) to augment data from randomized controlled trials (RCT). In this setting, the training set contains individuals x, and the outcome y of the treatment that they received. In addition, An example case is EHRs that contain information about the prescription of treatment without follow-up, in which case a new appointment or call needs to be scheduled with the patient in order to acquire the outcome. The objective is to improve the decision of which treatment to give to a new patient &, as in the study of Sundin et al. (2019).
assume record of patients and treatments, for which the outcomes can be acquired.
Experiments are run on the IHDP dataset? (Hill, 2011), semi-synthetic dataset which consists of 747 patients with 25 covariates. The patient covariates come from real randomized medical study from the 80s, however, the outcomes have been artificially generated, implying that all potential outcomes are available. We combine the responses A1, B1, and C1 to obtain 3-decision problem.
Knee osteoarthritis diagnosis. The second use case focuses on symptomatic patients who have a suspicion of knee osteoarthritis (OA) progression in the medial compartment of the right knee. OA is degenerative disorder of the joints, which reveals itself through symptomatic and structural changes. To date, this disease has no cure, but if detected early, its progression could be slowed down via behavioral interventions (Katz et al., 2021). We thus consider the problem of optimizing the diagnostic path for a new patient preciscly, the decision-making problem is to decide when to perform the next follow-up: at 12, 24, 36 48 months, after 48 months. We assume that the doctor is able to query for additional data about previous patients, but that requires a laborious authorization process due to privacy concerns.
More
We construct a datasct from the Ostcoarthritis Initiative (OAI) database², which is a multi-center 10-ycar observational longitudinal study of 4796 subjects (consent obtained from all the subjects; data are de- identified). After pre-procsssing, we obtain our final dataset with 8 covariates (clinical data and an initial imaging-basee based assessment) from 606 patients. The outcome is the joint space width loss over 0. 0.7mm by the time of the follow-up (Neumann al., 2009; Eckstein et al., 2015). A detailed description of the dataset is given in Appendix D).
5.2 Model of the outcomes
All experiments are run with GP regression (Rasmussen Williams, 2006), i.., we assume zero-mean GP prior for the function fk, with kernel mg:
S ~
(13)
Note that this case, posterior distributions p(Sa,> turn out to be Gaussian (standard results are recalled in the Appendix A). We use for all models the squared exponential kernel with automatic relevance determination (ARD-SE). GP hyperparameters (variance, lengthscales), as well the noise variance, are
2 Available online part of the supplementary material of Hill (2011).
paattp:///daal mih. gov/aa//
10

---

Published in Transactions on Machine Learning Research (06/2024)
estimated with maximum marginal likelihood. Python implementation is carried out with the framework GPy (open-source, under BSD licence).
5.3 Protocol and evaluation metrics
Our experimental protocol is follows: each considered dataset is randomly split into training set D, query set U, and test set. Experiments mainly focus on the scenario where the test set is single point &; nonetheless, we also provide additional results for a test population. We then proceed to sequentially acquire Naoq points using the proposed Algorithm and the active learning baselines presented in the next. subsection. All experiments are run with Nacq = 10.
We track the evolution of two metrics, computed both before the active learning phase and after each acquisition, over M different splits of the original dataset. More precisely, given split we track whether the correct decision is returned, with binary accuracy score
Am I(d d""),
(14)
where d BAYES is the Bayes-oplinal decision for Xm returned by the model (i.e., according to Eq. (2)), and d is the ground truth best decision for Xm. We have I(dm: d(n) = if and only if dm d (and zero otherwise). Our second metric is the entropy of the postcrior the optimal decision of the testing point x,,
Hm = H(p( (xm) (Dm))
(15)
All experiments are run with M = 200 replications.
5.4 Baseline active learning methods
The proposed method (acronym D-EIG) is compared with several active learning methods:
Random sampling (RS)- Chooses (x,,d) dj) uniformly random from Ui;
EIG on the parameters (P-EIG) Presented in Section 2.3. Chooses the (xj, di) which yields the greatest expected information gain on its associated GP.
EIG on the outcome (O-EIG) Presented in Section 2.3. Chooses the (x,, dg) which yields the greatest expected information gain on p(ūa, [X, D. This criterion is connected to the classical expected error reduction criterion, see details in Appendix E.
Decision uncertainty sampling (D-US) - A bascline that we introduce. Chooses the (X,, dz) whose optimal decision (i.e., associated to xj) is the most uncertain, evaluated with the entropy of Drest (Xz);
Uncertainty sampling (US) Chooses the (x,, dj) whose postcrior predictive distribution p(ya, [xj, Day) has the greatest variance;
5.5 Results
Experiments are run with a starting training set of size 100 for the synthetic dataset and the OAT dataset, and of size 50 for the IHDP datasct. All experiments were run on a high-performance computing cluster.
For the scenario with a single test point (x), Figure displays the evolution of the average binary accuracy score An over all replications (i.e., the evolution of the proportion of correct decisions). Figure 4 displays the evolution of the average entropy of the postcrior of the optimal decision (the Hm score). For all considered dalasets, the proposed method gives the best results both in terms of improving the decision-making accuracy and reducing the uncertainty on the optimal decision. This is particularly striking in the OAI dataset, where the problem is the hardest (real data and five possible decisions): all alternatives barely improve the decision-makee decision-making at all, whereas the proposed method greatly improves it. More preciscly, the baselines do
hhttps://sheffieldee gi thub. io/GPy/
11

---

Published in Transactions on Machine Learning Research (06/2024)
0.60
0.70
A5
PE0
Progorton of correct decisions
0.55
Dell
irpprtton of correct decisioss
6665
Proportion of deciiios
0.3
c:23
0.30
0.40
0.50
0.50
0.40
0.0
0.55
0.45
10
30
10
Nb of acquisitions
Nb of acquisitins
Nb arquisitions
(a)
(b)
(c)
Figure 3: Mean and standard error of the mean of the accuracy score Am over M = 200 replications of the experiment, with a single test point, w.r.t. the number of AL acquisitions. The proposed largeted active learning criterion D-EIG outperforms all considered AL methods in improving the accuracy of the decision-making. From left to right: (a) Synthetic data. (b) IHDP dataset. (c) OAT datasct.
0.40
16-
Average entropy of best
0.78
10.5
0.
0.70
Average entropy
0.65
0.
2
590
#00 D 000
0.55
000
0.50
260
040
acquistioons
acquisition
acquisitions
(b)
(c)
Figure Mean and standard error of the mean the entropy score Hm (entropy of the posterior of the optimal decision) over M 200 replications of the experiment, with a single test point, w.r.t. the number of AL acquisitions. The proposed largeted active learning criterion D-EIG reduces the uncertainty the optimal decision the fastest among all considered AL methods. From left to right: (a) Syuthetic data, (b) IHDP dataset. (c) OAl dataset.
not yield good performance, with the notable exception of US which has the second-best performance in entropy reduction on the IHDP and OAI datasets. Lastly, despite being targeted to the outcome 8, O-EIG has overall poor performance. This demonstrates the value taking into account the posterior uncertainty on the optimal decision.
For completeness, also include results with test population with N, 50 test points. Figure 5 displays the evolution of the average binary accuracy score Am over all replications. We draw similar conclusions to the single test point scenario; the proposed method outperforms all other considered baselines.
6 Discussion
In this paper, we tackled the problem of decision-makine making-aware active learning, that is, sample-officient performance improvement a down-the-line miiision-maiing problem. To this end, we have proposed directly reduce the uncertainty the posterior distribution of the optimal decision. Experimental work
12

---

Published in Transactions on Machine Learning Research (06/2024)
R5
R5
047
PB8
P-816
Properton of connect dociiions
0-212
Proportinn of conrect decisions
6.02
Proportinn decisions
D-210
D.0
DE5G
2.00
0.42
0:58
0.43
0.43
636
10
No of acquisitions
Nb of acquistions
Nb of acqusitions
(b)
(c)
Figure 5: Mean and standard error of the mean of the accuracy score Am over M 200 replications of the experiment, with test set of 50 points, w.r.t. the number of AL acquisitions. Here, the proposed targeted active learning criterion D-EIG outperforms all considered AT. methods in improving the accuracy of the decision- decision-making, From left to right: (a) Synthetic data. (b) IHDP dataset. (c) OAI dataset.
demonstrated the advantages of the proposed technique compared to classical Bayesian experimental design baseline methods in personalized medicine settings.
The main limitation of the proposed method is its computational complexity, as the current implementation involves many model retraining steps. Computational complexity is tolerable in applications where both the utility of correct decisions and cost acquiring new data points are high, such as in personalized medicine. Nevertheless, future work needed to design lower-complexity and still accurate approximations of the proposed criterion. potential direction is to frame the problem as MFBO, as discussed in Section 4, which would enable leveraging of efficient approximation strategics inherent to entropy-scarch Bayesian oplimization (Hennig & Schuler, 2012; Hernández-LLoaalo et al., 2014; Wang & Jegelka, 2017; Takeno el. al., 2020). Extending the proposed criterion to batch selection, in contrast with the current sequential selection method, will also help. The second limitation of our method is that we considered decisions to be available to the algorithm, and in many real-life situations, this may not be the case, for instance, due to privacy concerns. However, our criterion can be frraight.foraadde extended to tackle this limitation, and we also see this as a direction for future work.
A natural alternative to the proposed active learning criterion is to consider the expected information gain for the maximum utility, instead of focusing the optimal decision (treatment) that maximizes utility. The rationale behind our proposed criterion is to adhere as closcly as possible to the principle: 'Best possible mrett for the patient Considering the information gain for the maximizer of the utility (i.e., optimal treatment) implies a criterion that concentrates the active learning budget to identify the optimal treatment, even if the difference outcomes between two treatments small. scenarios where a good treatment is sufficient and it is not necessary to identify the absolutely best one, the criterion that maximizes the expected information gain for maximum utility be resource-efficient if two more treatments work almost. equally well for the patient.
To conclude, we anticipate that our method will have significant impact in the field of interactive AI with healtheare applications. Specifically, we have shown that the proposed technique can be applied personalized diagnosis and treatment applications. Both of these clinical problems require accurate and reliable docision-makinng tools, which are, however, costly build. Our method is sample-efficient and has decision-making capabilities by design.
Broader Impact Statement
In the personalized medicine scenario, it would of course be unethical to conduct experiments on other subjects only to gain information for specific individual. Our perspective in this paper to retrieve
13

---

Published in Transactions on Machine Learning Research (06/2024)
information from other databases, such RCTs (meaning that such experiments have already been carried out), or to conduct non-invasive experiments such as asking experts about counterfactuals (Sundin et al., 2019). Nonetheless, building fair active data collection is a crucial direction for research in that field (Andrus et al., 2021). Active learning may conccivably tempt unchtical misuse in personalized medicine, by exposing other patients to threats starling from non-consensual use of information to more sinister scenarios. Special attention required prevent misuse while getting the benefits of treatments that active learning promises.
Acknowledgments
This work was supported by the Academy of Finland (Flagship programme: Finnish Center for Artificial Intelligence, FCAI), grants 292334, 294238, 319264, by the UKRI Turing AI World-Leading Researcher Fellowship, EP/ EE//EP//WWOR /W002733/1 and by the Emil Aaltonen foundation. We also acknowledge the Aalto Science-IT project for their computational resources. LF and AT were affiliated with the Department of Computer Science of Aalto University when this research was conducted. This project was supported by the Research Council of Finland (Profi6 336449 funding program); the University of Oulu strategic funding, and the European Union Horizon Europe- HORIZON-HLTH 202.HHHH2222-STTALTTTTO program (grant 101137146).
References
Ahmed M. Alaa and Mihaela van der Schaar. Bayesian inference of individualized treatment effects using multi-task gaussian processes. In Advonces in Neural Information Processing Systems (NTPS), volume 30, pp. 3424 3432, 2017.
McKane Andrus, Elena Spitzer, Jeffrey Brown, and Alice Xiang. What We Can't Measure, We Can't Understand: Challenges to Demographic Data Procurement in the Pursuit of Fairness. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pp. 249 260, 2021.
Jean- Yves Audibert, Sébastien Bubeck, and Rémi Munos. Best arm identification in multi-armed bandits. In Proceedings of the Annual Conference on Learning Theory (COLT), pp. 41-53, 2010.
José M Bernardo. Expected information as expected utility. The Annals of Statistics, pp. 686-690, 1979.
Ioana Bica, Ahmed M. Alaa, Craig Lambert, and Mihaela van der Schaar. From real-world patient data to individualized treatment effects using machine learning: Current and future methods to address underlying challenges. Clinical Pharmacology & Therapeutics, 109(1):87-100, 2020.
Eric Brochu, Vlad M Cora, and Nando De Freitas. A tutorial bayesian optimization of expensive cost functions, with application to active user modeling and hierarchical reinforcement learning. arXiv preprinl arXiv:1012.2500 2010.
Kathryn Chaloner and Isabella Verdinelli. Bayesian experimental design: A review. Statistical Science, 10(3): 273-304, 1995.
Adam D Cobb, Stephen J Roberts, and Yarin Gal. Coss-Calibrated Approximate Inference in Bayesian Neural Networks. In Proceedings of the ICML workshop Theory of deep learning, 2018.
Pedram Daee, Tomi Peltola, Marta Soare, and Samuel Kaski. Knowledge elicitation via sequential probabilistic inference for high-dimensional prediction. Machine Learning, 106(9-10):1599-116)) 2017.
F Eckstein, JE Collins, MC Nevitt, JA Lynch, V Kraus, JN Katz, E Losina, W Wirth, A Guermazi, FW Roemer, et al. Cartilage thickness change an imaging biomarker of knee osteoarthritis progression Data from the FNIH OA biomarkers Consortium. Arthritis & Rheumatology, 6111))3118, 2015.
Tanner Fiez, Lalit Jain, Kevin G Jamicson, and Lillian Ratliff. Soquential experimental design for transductive linear bandits. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
Adam Foster, Martin Jankowiak, Elias Bingham, Paul Horsfall, Yee Whye Tch, Thomas Rainforth, and Noah Goodman. Variational Bayesian optimal experimental design. In Advances in Neural Information Processing Systems (NeurIPS), pp. 44066-1445 2019.
14

---

Published in Transactions on Machine Learning Research (06/2024)
Roman Garnett. Bayesian Optimization. Cambridge University Press, 2023.
Philipp Hennig and Christian J Schuler. Entropy search for information--lficient global optimization. Journal of Machine Learning Research, 13(6), 2012.
José Miguel Hernández-Lobatte Matthew W Hoffman, and Zoubin Ghahramani. Predictive entropy search for efficient global optimization of black-box functions. In Advances in Neural Information Processing Systems (NIPS), volune 27, pp. 918 926, 2014.
Jennifer Hill. Bayesian nonparametric modcling for causal inference. Journal of Computational and Graphical Statistics, 20(1):217-241) 2011.
Neil Houlsby, Ferenc Huszár, Zoubin Ghahramani, and Máté Lengyel. Bayesian active learning for classification and preference learning. In Proceedings of the NIPS workshop Bayesian optimization, eaperimental design and bandits: Theory and applications, 2011.
Eyke Hüllermeier and Willem Waegeman. Alcatoric and epistemic uncortainty in machine learning: An introduction to concepts and methods. Machine Learning, 110(3):457-50) 506, 2021.
Kirthevasan Kandasamy, Gautam Dasarathy, Junier B Oliva, Jeff Schneider, and Barnabás Póczos. Gaussian process bandit optimisation with multi-fidelity evaluations. Advances in neural information processing systems, 29, 2016.
Kirthevasan Kandasamy, Willie Neiswanger, Reed Zhang, Akshay Krishnamurthy, Jeff Schneider, and Barnabas Poczos. Myopic posterior sampling for adaptive goal oriented design of experiments. In Proceedings of the International Conference Machine Learning (TCML), pp. 3222 3232, 2019.
Jeffrey N Katz, Kaetlyn R Arant, and Richard F Loeser. Diagnosis and Treatment of Hip and Knee Osteoarthritis: A Review.. JAMA, 325(6):568 578, 2021.
Emilic Kaufmann, Olivier Cappé, and Aurélien Garivier. On the complexity of best-arm identification multi-armed bandit models. Journal of Machine Learning Research, 17(1):1-42, 2016.
Andreas Krause, Ajit Singh, and Carlos Guestrin. Near-optimal sensor placements in gaussian processes: Theory, efficient algorithms and empirical studies. Journal of Machine Learning Research, 9(2), 2008.
Tomasz Kušmierczyk, Joseph Sakaya, and Arto Klami. Variational Bayesian decision makinn for continuous utilitics. In Advances Neural Information Processing Systems (NeurIPS), pp. 6395 6395-6405, 2019.
Simon Lacoste-Julien. Ferenc Huszár, and Zoubin Ghahramani. Approximate inference for the loss-calibrated Bayesian. In Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS), pp. 416-424, 2011.
David D Lewis and Jason Catlett. Heterogeneous uncertainly sampling for supervised learning. In Proceedings of the International Conference on Machine Learning (ICML), pp. 148 156, 1994.
Dennis V Lindley. On measure of the information provided by experiment. The Annals of Mathematical Statistics, 1006 1956.
Chun Kai Ling, Kian Hsiang Low, and Patrick Jaillet, Gaussian process planning with lipschitz continuous reward functions: Towards unifying bayesian optimization, active learning, and beyond. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 186001866,661 2016.
David J. C. MacKay. Information-Basel Objective Functions for Active Data Selection. Neural Computation, 4(4):590 604, 1992.
G Neumann, Hunter, M Nevitt, LB Chibnik, K Kwoh, H Chen, T Harris, Satterfield, J Duryea, al. Location specific radiographic joint space width for ostcoarthritis progression. Osteoarthritis and cartilage, 17(6):761-708 -76,, 2009.
15

---

Published in Transactions on Machine Learning Research (06/2024)
Tom Rainforth, Rob Cornish, Hongscok Yang, Andrew Warrington, and Frank Wood. On nesting Monte Carlo estimators. In Proceedings of the International Conference on Machine Learning (TCML), pp. 4267-4277 2018.
Tom Rainforth, Goliniski Frank Wood, and Sheheryar Zaidi. Target-aware Bayesian inference: how beat optimal conventional estimators. Journal of Machine Learning Research, 21(8)), 2020.
C.E. Rasmussen and C.K.I. Williams. Gaussian Processes for Machine Learning. MIT Press, 2006.
Nicholas Roy and Andrew McCallum. Toward optimal active learning through Monte Carlo estimation of error reduction. In Proceedings of the International Conference on Machine Learning (ICML), pp. 441-448, 2001.
Daniel Russo. Simple Bayesian algorithms for best arm identification. In Proceedings of the Annual Conference on Learning Theory (COLT), pp. 1417-1418, 2016.
Elizabeth G Ryan, Christopher Drovandi, James McGree, and Anthony N Pettitt. review of modern computational algorithms for Bayesian optimal design. International Statistical Review, 84(1):128 154, 2016.
Maytal Saar-Tsechansky and Foster Provost. Decision-centric active learning of binary-outcome models. Information systems research, 18(1):4-22, 2007.
Burr Settles. Active learning. Morgan Claypool, 2012.
Uri Shalit, Fredrik D. Johansson, and David Sontag. Estimating individual treatment effect: generalization bounds and algorithms. In Proceedings of the International Conference Machine Learning (ICML), pp. 3076-3085, 2017.
Claude E Shannon. Mathematical Theory of Communication. The Bell System Technical Journal, 27(3): 379-423, 1948.
Tiris Sundin, Tomi Poltola, Luana Micallef, Homayun Afrabandpcy, Marta Soare, Muntasir Mamun Majumder, Pedram Daee, Chen He, Baris Serim, Aki Havulinna, et al. Improving genomics-based predictions for precision medicine through active elicitation of expert knowledge. Bioinformatics, 34(13) 10905-199--0 2018.
liris Sundin, Peter Schulam, Eero Siivola, Aki Vehtari, Suchi Saria, and Samuel Kaski. Active learning for decision-making from imbalanced observational data. In Proceedings of the International Conference on Machine Learning (ICML), pp. 6046 6055, 2019.
Shion Takeno, Hitoshi Fukuoka, Yuhki Tsukada, Toshiyuki Koyama, Motoki Shiga, Ichiro Takeuchi, and Masayuki Karasuyama. Multi-fidelity bayesian optimization with max-value entropy search and its paralleiiaaton. In International Conference on Machine Learning, pp. 9334-9345. PMLR, 2020.
Stefan Wager and Susan Athey, Estimation and Inference of Heterogeneous Treatment Effects using Random Forests. Journal of the American Statistical Association, 113(523), 2018.
Zi Wang and Stefanic Jegelka. Max-value entropy scarch for efficient bayesian optimization. In International Conference on Machine Learning, pp. 3627-3635. PMLR, 2017.
Minjie Xu and Gary Kazantsev. Understanding Goal-Oriented Active Learning via Influence Functions. In Proceedings of the NeurIPS workshop on Mochine Learning with Guarantees, 2019.
Songbai Yan, Kamalika Chaudhuri, and Tara Javidi. Active learning with logged data. Tn Proceedings of the International Conference on Machine Learning (ICML), pp. 5521 5530, 2018
Liuyi Yao, Sheng Li, Yaliang Li, Mengdi Huai, Jing Gao, and Aidong Zhang. Representation learning for treatment effect estimation from observational data. In Advances in Neural Information Processing Systems (NTPS), volume 31, 2018.
16

---

Published in Transactions on Machine Learning Research (06/2024)
Guang Zhao, Edward Dougherty, Byung-Jun Yoon, Francis Alexander, and Xiaoning Qian. Uncertainty-aware active learning for optimal Bayesian classifier. In International Conference on Learning Representations (ICLR), 2021.
Sue Zheng, David Hayden, Jason Pacheco, and John W Fisher III. Sequential Bayesian Experimental Design with Variable Cost Structure. In Advances in Neural Information Processing Systems (NeurIPS), pp- 4127-4137, 2020.
17

---

Published in Transactions on Machine Learning Research (06/2024)
A Gaussian process regression
The elements of this section are taken from Rasmussen & Williams (2006), Chapter 2. A Gaussian process (GP) is a stochastic process, i..,, a collection of random variables, such that any finite combination of this collection has a Gaussian distribution. A GP is completely specified by its mean function m, and covariance function (or kernel) K, and we write
f~ ((xx
(16)
It is often assumed that mx) = 0.
We now consider GP regression model
f GP(O,, x(xxxx(x x')), y= f(x) +c,
(17)
(18)
where e e~ (0, )²). That is to say that GP prior is placed on f. the following, we use the notation fx to denote (xx) Given a collection of observations = {(x;, (=) we wish to characterize the posterior distribution a test point &, p(fx(D). The definition of GP implies that
y fx
R [xX,X) + 2 -n(0,
N
(0.
K xxxx))
(19)
x)
and such, by using basic manipulations of the Gaussian distribution, can be shown that
p(f(DD = ((ux, oZ).
(20)
where
Fs = x(x X X + - o x(x
(21)
K X X X X) +
(22)
Consequently, p(y| D) is also Gaussian with mean Us and variance o² oz
B Notes on the expected information gain (EIG)
Let us first consider a standard Bayesian regression model, with likelihood p(ylx, f) and prior p(0), which leads to the characterization of the posterior distribution p(0(D). We take the example of the EIG on 0. We have
EIG(x) H[p( Bol/x.) ((p(O) {(x, v)Di]
(23)
This expression can be rearranged show that the EIG is equal to the mutual information between y and 0 (given x and D), defined
/f ply, 0|x, D) log
ply, 0|x, D) dyd0. plu|x, ()p(0 D)
I(y; 1(y;01x, D)
(24)
The symmetry of the mutual information leads in turn to alternative formulation of the EIG, namely
EIG(x)= H(p(y/x, x D)j [7pppx) 0)]]
(25)
which now computes entropies in the output space (and not the parameter space). Most notably, this does not involve model retraining. This the form most often used in practice.
If we consider non-parametric regression model of the form y f(x) +,, where N/(,0²), easily adapt the expression of Eq. (25)
BIG(x) H/plylx, D)J (Hplylx
(26)
18

---

Published in Transactions on Machine Learning Research (06/2024)
This can be further simplified when dealing with GP regression. In that case, the predictive posterior distribution p(y|x, x, D) is Gaussian, with mean /xx and variance o² + xx Considering that the value of o² is fixed, or estimated, the expression of Eq. (26) becomes
EIG(x) = (log(o² + + - - -log(o²))
(27)
As such, the higher xx the higher the EIG.
C Gauss-Hermite quadrature
We consider computing expectations of the form
B[[(w)] (u)rlu)dy
(28)
where Y is Gaussian random variable with mean and variance o². The Gauss-Hermitee approximation of order N of the previous expression is given by
wif(v2oz, 1)),
(29)
where the T2 are the roots of the Hermite polynomial of order N (denoted by Hn), and the weights wi are given by
2 'n.nnVV
(30)
n2 (H
D Knee osteoarthritis follow-up data details
We considered all the data from the Osteoarthritis Initiative Dataset (OAI; https://ndaa //tee:///daaaaeeee .ih haalhh.ooooai/// with a total WOMAC score over 9 (symplomatic subjects). Subsequently, we selected those subjects, which have early, doubtful, or early radiographic Osteoarthritis at the baseline according to the Kellgren Lawrence grading scoring system.
In our experiments, we used a commonly accepted measure joint space width (JSW) loss over 0.7mm indicator of progression. The JSW was measured from knee X-rays at fixed location (æ 0.250), thus focusing on OA only in the medial compartment of the knee.
The following is the list of variables, which we selected from the OAI dataset (per knee):
Age;
Sex;
Body-mass-index (BMT);
Total WOMAC score;
Indication of varus, valgus, or neither;
Indication of past injury;
Indication of past surgery;
Kelleren-Tawrrece grade;
JSW fixed location (x (x= 0.250).
19

---

Published in Transactions on Machine Learning Research (06/2024)
E O-EIG and expected error reduction
Let us consider that the error measure is the log-loss. Then, in a Bayesian AL framework, the expected error reduction query writes
XEER
arg min B (())))
Σ
Hiplyylz3, DU ((x, w)ill)
(31)
xeu
where Xe lest population. the setting considered in the paper, we have X, = (x), which reduces to
XEER = arg min Big D()l, [H[p(0 x, DU {(x, v))il.
(32)
XEM
which is exactly the O-EIG criterion.
20


Reviews:
[{"review_id": "Nj38GYxcZc", "reviewer": "Reviewer_Wi6b", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Please note that I'm an expert in the topic of Bayesian Active Learning, and as such, it is hard for me to evaluate the novelty of this study and especially the correctness of the discussion of related works properly as I don't know many cited works.\n\n\n**Strengths:**\n- The paper is easy to follow.\n- After comparison with the previous submission, it seems that the authors addressed the concerns of the previous reviewers.\n- The authors clearly discuss the limitations and specific assumptions of the new setting and the method.\n- It seems that the authors consider novel settings for active learning in decision-making, and it is quite well motivated by authors...\n\n\n**Weaknesses:**\n- ... however, it is also very specific. It is hard for me to think about other applications than medicine. As such, I believe this article will be interesting only to a narrow group of readers.\n- Under the authors' goal and assumption, the final algorithm is rather simple and computationally very inefficient, but the authors somehow defend this by assuming that the cost of running the algorithm doesn't matter as obtaining additional information for a selected sample is much more expensive than running even very costly algorithm.\n- While the authors clearly discuss it, the application of the algorithm may motivate unethical/instrumental treatment of some patients. \n- Provided examples of ethical use cases:\n  > Our perspective in this paper is to retrieve information from\nother databases, such as RCTs (meaning that such experiments have already been carried out), or to conduct non-invasive experiments such as asking experts about counterfactuals (Sundin et al., 2019) (...)\n\n  seems not to meet an assumption about the high cost of obtaining information for new data points. But I belive better examples can be found.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose a new active learning approach in the setting where a learner is tasked to make a decision for a single data point that yields the highest return, but before the learner can select a few unlabeled data points and decisions for them and learn about their outcomes (returns). The proposed algorithm is inspired by Bayesian experimental design and aims to take into account the decision-making process and maximize the expected information gain on the posterior distribution of the optimal decision instead of the general quality of predictive performance of the learner. The authors then generalize the framework for the setting of prediction for the population. Finally, the proposed approach is evaluated on 3 different benchmarks, one completely synthetic and two medical datasets against a few baseline active learning methods. The results demonstrate that in both settings (for a single decision and population) and under different budgets for acquiring additional information, the new approach significantly outperforms the baselines.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "None, if more knowledgeable reviewers believe that the current discussion on the related works is now adequate.", "broader_impact_concerns": "I don't have any new concerns in addition to the ones discussed by the authors of the paper - precisely the setting somehow motivates selecting the treatments/making decisions for some patients in such a way as to help gain information that will allow the selection of the best treatment for the other patient. Putting the good of one patient ahead of a few others.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "o9N9uYndH3", "reviewer": "Reviewer_D3ah", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: the problem itself is interesting, and it does seem to be untreated in the literature (surprisingly to me). There is an improved connection to bayesian optimisation (from what I gather there was in a previous iteration), and due reference to related problems in bandits. The empirical evaluation is of a small scale but is sound and shows the method working well. The methodology itself is also very sensible and clearly explained. There are fairly clear directions for future work identified and (some tiny things aside - see below) the work appears to be accurate. \n\nWeaknesses: These are twofold, 1. the size of the contribution (it could have gone further to conduct more extensive evaluation or to provide theoretical guarantees on the method), 2. the connection to applications - I'm not entirely convinced that in the healthcare settings described, this active learning method would actually be used by practitioners and it wouldn't just be more efficient and better for data privacy for a central body to build a predictive model on a larger dataset. A more convincing explanation, a reference to this practice, or an alternative application where the setup feels more commonplace would improve this.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper considers a variant of the traditional active learning, where the focus is not on generally learning a parameter vector, or non-parametric function as well as possible, but on learning which of $K$ functions $f_1,...f_K$ has the largest expectation at some predefined input vector $\\tilde{\\mathbf{x}}$. This captures scenarios where a practitioner knows they have a decision to make for a particular individual in the future and seek to efficiently build a model to decide which option (from a discrete set) to choose for the individual.\n\nThe main contribution of the paper is a method for this active learning problem, which is a modification of expected information gain based  algorithms, which targets a reduction in the entropy of the (discrete) posterior distribution of the optimal action for $\\tilde{\\mathbf{x}}$ rather than a posterior on the underlying functions $f_1,...f_K$ or the stochastic output at an arbitrarily selected $\\mathbf{x}$. The assessment of the method is empirical, and it compares favourably to a range of sensible benchmarks on a fully synthetic dataset and example datasets from healthcare applications.\n\nThere is also brief exploration of the extension of this method to taking decisions for a set of individuals with vectors $\\tilde\\mathbf{x}_j$, $j=1,...N_t$, where the method can be readily extended (by utilising an upper bound on the entropy in lieu of the computationally intractable exact entropy of the joint distribution) and continues to have an improved performance over benchmarks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Are we assuming (in our construction of $D_{best}$ and elsewhere) that the best action is always unique? If $f_k: \\mathcal{X} \\rightarrow \\mathcal{S}$ where $\\mathcal{S}$ is some discrete set, it would seem that this could quite readily not be the case. Perhaps you intend that the $f's$ are implicitly continuous (as you use a GP to model them) and that the chance of them intersecting at $\\tilde{\\mathbf{x}}$ is implicitly vanishingly small, but it would be good practice to make this explicit and/or establish a rule for breaking ties.\n2. Can you add any detail to convince me that the suggested usage in the healthcare applications is more realistic than it seems?\n3. Typos on p5: *cardinality (not cardinal), and the sum in equation (11) should be over $i$ not $t$?\n4. On p6, why is the complexity $N_s \\times card(\\mathcal{U})$ and not $N_s \\times card(\\mathcal{D})$ or similar? I took it to be the case that we would typically use much fewer than $card(\\mathcal{U})$ samples?\n\nIt would be ideal to address 1,3, and 4, at least in rebuttal; and 2. is non-essential, but I think worthwhile nonetheless.", "broader_impact_concerns": "I believe the current statement is adequate.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "rsnQDnTrjE", "reviewer": "Reviewer_GuAx", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strength: Validity of Proposed Method and Impact of the Results\nThe proposed method adopts uncertainty in decision-making (9) as a criterion for selecting labeling points.\nWhile conventional active learning methods use parameters or prediction uncertainty as criteria, this criterion is specific to decision-making problems, making it a valid criterion that aligns with the research objectives.\nFigures 3 and 4 of the experimental results demonstrate that the proposed method outperforms existing active learning methods in improving decision-making accuracy and uncertainty.\nThese results indicate that the proposed method is both valid and effective, considering its design for active learning in decision-making.\n\n### Weakness: Uncertainty in Decision-Making and Uncertainty in Maximum Utility\nThe proposed method adopts uncertainty in decision-making as a criterion for active learning.\nHowever, when two interventions $d$ and $d'$ provide very similar (or equal) utilities $y_d \\approx y_{d'}$, determining the superiority of these interventions requires a large amount of data, leading to increased uncertainty.\nOn the other hand, if the utilities $y_d$ and $y_{d'}$ are nearly equal, the practical impact of choosing either intervention $d$ or $d'$ may be negligible.\nIn such scenarios, rather than using uncertainty in decision-making as in the proposed method, it might be more appropriate to use uncertainty in the maximum utility $\\max_k y_k$ as a criterion, as shown below:\n\n$$\n\\\\mathcal{H}[p(\\\\max\\_k \\\\tilde{y}\\_k | \\\\tilde{x}, \\\\mathcal{D} )] - \\\\mathbb{E}\\_{p(y\\_j | x\\_j, \\\\mathcal{D}\\_{d\\_j})} \\\\mathcal{H}[p(\\max\\_k \\\\tilde{y}\\_k | \\\\tilde{x}, \\\\mathcal{D} \\cup \\\\{(x\\_j, d\\_j, y\\_{d\\_j})\\\\} )]\n$$\n\nI presume that one may superior to another one depending on the problem setting.\nAdding some discussions on the differences between these two approaches and their respective advantages could further enhance this paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors have proposed a method for active learning in Bayesian decision-making problems.\nA decision-making problem involves selecting an intervention $d$ to maximize the utility $y$ given an input $x$.\nThe authors considered actively selecting data from unlabeled data $(x, d)$ where the utility $y$ is unknown, to acquire labels for utility $y$.\nThis approach aims to maximize utility for new inputs $x$ with minimal labeling.\nThe proposed method adopts uncertainty in decision-making as a criterion for selecting points to acquire labels by estimating the most uncertain data points.\nThrough experiments on both artificial and real data, the authors have demonstrated that the proposed method outperforms existing active learning methods.\nIn other words, it effectively improves the accuracy of decision-making with fewer labels.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See Weakness above.", "broader_impact_concerns": "There is no ethical concern.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers are recommending acceptance and after reading reviews, rebuttals and the discussion, I certainly concur. Congratulations on a fine piece of work!
Decision: Accept (poster)