Batch Multi-Fidelity Active Learning with Budget Constraints
Authors: Shibo Li
Jeff Phillips
Xin Yu
Robert Kirby
Shandian Zhe
Venue: NeurIPS 2022 Accept (2022)
Keywords: Multi-Fidelity Active Learning
Budget Constrint
High-dimensional Outputs
Physical Simulation
Computational Physics

Abstract: Learning functions with high-dimensional outputs is critical in many applications, such as physical simulation and engineering design. However, collecting training examples for these applications is often costly, e.g., by running numerical solvers. The recent work (Li et al., 2022) proposes the first multi-fidelity active learning approach for high-dimensional outputs, which can acquire examples at different fidelities to reduce the cost while improving the learning performance. However,  this method only queries at one pair of fidelity and input at a time, and hence has a risk of bringing in strongly correlated examples to reduce the learning efficiency. In this paper, we propose Batch Multi-Fidelity Active Learning with Budget Constraints (BMFAL-BC), which can promote the diversity of training examples to improve the benefit-cost ratio, while respecting a given budget constraint for batch queries. Hence, our method can be more practically useful. Specifically, we propose a novel batch acquisition function that measures the mutual information between a batch of multi-fidelity queries and the target function, so as to penalize highly correlated queries and encourages diversity. The optimization of the batch acquisition function is challenging in that it involves a combinatorial search over many fidelities while subject to the budget constraint. To address this challenge, we develop a weighted greedy algorithm that can sequentially identify each (fidelity, input) pair, while achieving a near $(1 - 1/e)$-approximation of the optimum. We show the advantage of our method in several computational physics and engineering applications.

Batch Multi-Fidelity Active Learning with Budget Constraints
Shibo Li", Jeff M. Phillips", Xin Yu, Robert M. Kirby, and Shandian Zhe School of Computing. University of Utah
Salt Lake City, UT 84112 {shibo, jeffp, xiny, kirby, zhe]@cs utah. edu
Abstract
Learning functions with sigh-dimensional outputs is critical in many applications, such as physical simulation and engineering design. However, collecting training examples for these applications is often costly, e.g., by running numerical solvers. The recent work (Li et al. 2022) proposes the first multi-fidelity active learning approach for high- high-dimensional outputs, which can acquire examples different fidelities to reduce the cost while improving the learning performance. However, this method only queries at one pair of fidelity and input a time, and hence has a risk to bring in strongly correlated examples to reduce the learning efficiency. In this paper, we propose Batch Aulli-Fidelity Active Learning with Budget Constraints (BMFAL-BC), which can promote the diversity of training examples to improve the benefit-cost ratio, while respecting a given budget constraint for batch queries. Hence, our method can be more practically useful. Specifically, propose novel batch acquisition function that measures the mutual information between a batch of multi-fidelity queries and the target function, queries and encourages diversity. The optimization of the batch acquisition function is challenging in that it involves combinatorial search many fidelities while subject to the budget constraint. To address this challenge, we develop weighted greedy algorithm that can sequentially identify each (fidelity, input) pair, while achieving a near (1 - -pprrxxixxiiiioon of the optimum. We show the advantage of our method in several computational physics and engincering applications.
penalize highly correlated
Introduction
Applications, such in computational physics and engineering design, often demand we calculate a complex mapping from oww-dimensional inputs to high-dimensiona, outputs, such as finding the optimal material layout (output) given the design parameters (input), and solving the solution field on mesh (output) given the PDE parameters (input). Computing these mappings is often very expensive, e.g. iteratively running numerical solvers. Hence, learning surrogate model to outright predict the mapping, which is much faster and cheaper, is of great practical interest and importance (Kennedy
and O' H'Haaan 2000, Conti and O' O'Hagan (2010)
However, collecting training examples for the surrogate model becomes another bottleneck, since each example still requires costly computation. To alleviate this issue, [i al. (2022) developed DMFAL, first deep multi-fidelity active learning algorithm. which can acquire examples at different fidelities to reduce the cost of data collection. Low-fidelity examples are cheap to compute (e.g.. with coarse meshes) yet inaccurate; high-fidelity examples are accurate but much more expensive (e.g., calculated with dense grids). See Fig. for an illustration. DMFAL uses an optimization-bans based acquisition method to dynamically identify the input and fidelity at which to query new examples, so as to improve the learning performance, lower the sample complexity, and reduce the cost.
Equal contribution Correspondence to: Jeff M. Phillips, Shandian Zhe.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Despite its effectiveness, DMFAL can only optimize and query at one pair of input and fidelity each time and hence ignores the correlation between consecutive queries. As a result, it has a risk of bringing in strongly correlated examples, which can restrict the learning efficiency and lead to suboptimal benefit-cost ratio. In addition, the sequential querying and training strategy is difficult to utilize parallel computing resources that are common nowadays (e.g., multi-core CPUs/GPUs and computer clusters) to query concurrently and to further speed up.
In this paper, we propose BMFAL-BC, a batch multi-fidelity active learning method with budget constraints. Our method can acquire a batch of multi-fidelity examples at a time to inhibit the example correlations, promote diversity so as to improve the learning efficiency and benefit-cost ratio. Our method can respect given budget in issuing batch queries, hence are more widely applicable and practically useful. Specifically, we first propose novel acquisition function, which measures the mutual information between a batch of multi-fidelity queries and the target function. The acquisition function not only can penalize highly correlated queries to encourage diversity, but also can be efficiently computed by an Monte-Carlo approximation. However, optimizing the acquisition function is challenging because it incurs a combinatorial search over fidelities and meanwhile needs to obey the constraint. To address this challenge, we develop a weighted greedy algorithm. We sequentially find one pair of fidelity and input each step, by maximizing the increment of the mutual information weighted by the cost. In this way, we avoid enumerating the fidelity combinations and greatly improve the efficiency. We prove that our greedy algorithm nearly achieves a (1 ) approximatiinn of the optimum, with a few minor caveats.
For evaluation, we examined BMFAL-BC in five real-world applications, including three benchmark tasks in physical simulation (solving Poisson's, 's, Heat and viscous Burger's equations), a topology structure design problem, and computational fluid dynamics (CFD) task to predict the velocity field of boundary-driven flows. We compared with the budget-aware version of DMFAL, single multi-fideity querying with our acquisition function, and several random querying strategies. Under the same budget constraint, our method consistently outperforms the competing methods throughout the learning process, often by a large margin.
2 Background
2.1 Problem Setting
Suppose we aim to learn a mapping f S TK." -> Rdd where is small but d is large, e.g., hundreds of thousands. To economically learn this mapping, we collect training examples at M fidelities. Each fidelity m corresponds to mapping fm S + IRd., The target mapping is computed at the highest fidelity, i.e. f(x) = (x). The other fm can be viewed as (rough) approximation of f. Note that dm is unnecessarily the same as d for m < M. For example, solving PDEs on a coarse mesh will give
dimenssoonl output (on the mesh points). However, we can interpolate it to the d-dimensional space to match f()) (this is standard in physical simulation (Zienkiewicz et al. 1977) Denote by Am the cost of computing fm (+) at fidelity m. We have x S... A.M
2.2 Deep Multi-Fidelity Active Learning (DMFAL)
To effectively estimate / while reducing the cost, Li et al. 2022) proposed DMFAL, a multi-fidelity deep active learning approach. Specifically, neural network (NN) is introduced for each fidelity m, where a low-dimensional hidden output h (x) is first generated, and then projected to the high-dimensional observation space. Each NN is parameterized by (Am, W m. pm). where A,, is the projection matrix, W.. is the weight matrix of the last layer, and On, consists of the remaining NN parameters. The model is defined as follows,
Xm hx;
hm (x) Wm.ee.. (Xm), (m) Am) + Em:
where X., is the input to the NN at fidelity m. ym (x) is the observed d,, dimensional output, &. ~ (- I) is a random noise, Φe... (Xm) is the output of the second last layer and can be viewed as a nonlinear feature transformation of X Since xmm includes not only the original input x, but also the hidden output from the previous fidelity, i.e., (x), the model can propagate information throughout fidelities and capture the complex relationships (e.g.. nonlinear and nonstationary) between different fidelities. The whole model is visualized in Fig. of Appendix. To estimate the posterior of the model, DMFAL uses a structural variational inference algorithm. A multi multi-variate Gaussian

---

multi-fideliy surrogate to predict solution fields
PDE parameters:
Ut + uur Uxx 0
PDE solvers
low-fidelity, cheap solve
high-fidelity, costly solve
Illustration the motivation and goal with physical simulation as an example. costly solve every PDE from scratch. We aim to train a multi-fidelity surrogate model to directly predict high-fidelity solution fields given the PDE parameters. To further reduce the cost of collecting the training data with numerical solvers, we scek develop multi-fidclin active learning algorithms.
Figure 1:
posterior is introduced for each weight matrix, a( (Wm) = ( (We [ Em) A variational evidence lower bound (ELBO) is maximized via stochastic optimization and the reparameterization trick (Kingma and Welling 2013).
To conduct active learning, DMFAL views the most valuable example at each fidelity the one that can hest help its prediction at the highest fidelity M (i.e., the target function). Accordingly, the acquisition function is defined
a(m, x) = T( y (x)(D) = Am
(HI(Y (D) + (y (D) - Hy y [D))
(2)
Am
where I(:,)) the mutual information, H(--) is the entropy. and D is the current training dataset. The computation of the acquisition function is quite challenging because ym and y M are both high dimensional. To address this issue, DMFAL takes advantage the fact that each low-dimensional
output h,, (x) is nonlinear function of the random weight matrices (W,,..
W m J. Based on the
variational posterior {)(W,)}, DMFAL uses the multi-variate delta method (ehlert 1992 Bickel
and Doksum 2015) to estimate the mean and covariance of = hm hM) ). with which to construct
joint Gaussian posterior approximation for h by moment matching. Then, from the projection in (1 we can derive a Gaussian posterior for [ [ymiX M. By further applying Weinstein-Arone -ronssiinAAoozzajn
identify (Kato. 2013), can compute the entropy terms in conveniently and efficiently.
Each time, DMFAL maximizes the acquisition function which to query. The new example is added into D. and the deep multi-fidelity model is retrained and updated. The process repeats until the maximum number of training examples have been queried or other stopping criteria met.
to identify a pair of input and fidelity at
Batch Multi-Fidelity Active Learning with Budget Constraints
While effective, DMFAL can only identify and query at one input and fidelity each time, hence ignoring the correlations between the successive queries. As a result, highly correlated examples are easier to be acquired and incorporated into the training set. Consider we have found (x", mmm) that maximizes the acquisition function (2).
often the case that single example will not cause significant change of the model posterior {e( W Wn)) (especially when the current dataset D is not very small). When we optimize the acquisition function again, we are likely to obtain a new pair (x", ri²) that is close to (x". mm) (e.g., m m and x* close to x"). Accordingly, the queried example will be highly correlated to the previous one. The redundant information within these examples can restrict the learning efficiency, and demand for more queries (and cost) to achieve the satisfactory performance. Note that the similar issue have been raised in other single-fidelity, pool- active learning works, e.g., Geifman and El- Yaniv 2017 Kirsch et al. 2019); see See
To overcome this problem, we propose BMFAL-BC. batch multi-fidelity active learning approach to reduce the correlations and to promote the diversity of training examples, so that we can improve the learning performance, lower the sample complexity, and better save the cost. addition, we take into account the budget constraint in querying the batch examples, which is common in practice (like cloud high-peeformance computing) (Mendes al., 2020). Under the given budget, we aim find batch of multi-fi fulti-fidelity queries that improves the benefit-cost ratio much possible.

---

3.1 Batch Acquisition Function
Specifically, we first consider an acquisition function that allows us to jointly optimize a set of inputs and fidelities. While it seems natural consider how extend (2) to the batch case, the acquisition function in ( is about the mutual information between ym (x) and yı (x). That means, it only measures the utility of the query (m, x) in improving the estimate of the target function at x itself (ie., yM (x)), rather than at any other input. To take into account the utility in improving the function estimation at all the inputs, we therefore propose new single acquisition function,
a. x) Ep(x') [I (ym NM
where x' E Q and (xx') is the distribution of the input in Q. We can see that, by varying the input x' in the second argument of the mutual information, we are able to evaluate the utility of the query in improving the estimation of the entire body of the target function. Hence, it is more reasonable and comprehensive. Now, consider querying a batch of examples under the budget B, we extend to
Obatch (M = Ep(e)) [T (fym, I
S.t.
my S B,
(4)
where M {mi.... M. X' {x1,.... Xx and D is the current training dataset. We can that the more correlated {ym, (x,) (x,))==1- j=1 the smaller the mutual information, and hence the smaller the expectation in ( Therefore, the batch acquisition function implicitly penalizes strongly correlated queries and favors diversity.
The expected mutual information in is usually analytically intractable. However, we can efficiently compute with an Monte-Carlo approximation. That is, we draw independent samples from the input distribution, xi.....Xx XAA p(x'), and compute
A A
apacch (M,
(5)
It is straightforward to extend the multi-variate delta method used in DMFAL to calculate the mutual information in (5). We can then maximize (5) subject to the budget constraint, Σ Am, < B, to acquire more diverse and informative training examples. In addition, when parallel computing resources (e.g., multi-core CPUs/GPUs and computer clusters) are available, we can acquire these queries in parallel to further speed up the active learning.
However, directly maximizing (5) will incur combinatorial search over multiple fidelities in M, and hence is very expensive. Note that the number of examples n is not fixed, and can vary as long as the cost does not exceed the budget B, which makes the optimization even more challenging. To address these issues, we propose weighted greedy algorithm to sequentially determine each (my. Xy) pair.
3.2 Weighted Greedy Optimization
Specifically, at each step, we maximize the mutual information increment on one pair of input and fidelity, weighted by the corresponding cost. Suppose at step k we have identified set of & inputs and fidelities at which to query, Q+ = ((x,, m,))), To identify a new pair of input and fidelity at step l + 1, we maximize weighted incremental version of (5)
1 I (x)
(1 (x)))
ax+1 m) =
A
Am
s.t. Am +
Σ
Am B,
(6)
MeMe
where 3/- {ym (x,) m,) E 2x). and Mr are all the fidelities in Q.. At the beginning (k 0), we have O 0. 34+ 0 and M 0. Each step, look each valid fidelity, i.e., 1 < Am < B E EnEM, Am. and find the optimal input. We then add the pair (m, x) that gives the largest 0 into Qs. and proceed to the next step. Our greedy approach is summarized in Algorithm

---

Algorithm 1 Weighted-Greeddy( (Am), budget B)
0. Ck + 2: while Ck do
1: k 4- 0, Qx
Optimize weighted incremental acquisition function (6)
(x" m
argmax
Rx+1(xx)) m)
Sm Ok
4:
if Infeasible then
5: break
end
7:
A: k+
8:
Q--1 {(x'
9:
Cx c Cx-1 + Am-
10:
end while
11:
Retur Q
The intuition of our approach is follows. Mutual information is classic submodular func- tion (Krause and Guestrin 2005), and hence if there were no budget constraints or weights, greedily choosing the imput which increases the mutual information most achieves solution within (1 1/e) of the optimal (Krause and Golovin 2014).. However, Leskovec et al. (2007) observed that when items have weight (corresponding to the cost for different fidelities in case) and there is budget constraint, then the approximation factor can be arbitrarily bad. We observe, and prove, that this only occurs as the budget is about to be filled, and up until that point, the weighted greedy optimization achieves the best possible (1 - 1 1/e)-approxxiiii -pprrooxxiiitioonn of the optimal. We can formalize this near (1 - (app)a)appoxmmmatiii in two ways, proven in the Appendix. Let OPT(B) be the maximum amount of mutual information that can be achieved with a budget B.
(Algorithm 7 before any choice of fidelity would exceed the budget, and the total budget used to that point is B < B, then the mutual information of the current solution is within (1 - 1/e) e) of O')(B').
Theorem 3.1. At any step of Weighted-Greed
(Algorithm 7 is run until input-fidelity pair (x, m) that corre- sponds with the maximal acquisition function 0g+1 (x, m)) would exceed the budget, it selects that indel-fidelity pair anyways (the solution exceeds the budget B) and then terminates, the solution obtained within (1 1/e) of OP.
Corollary 3.1. If Weighted-Greed
We next sketch the main idea of how to prove the main result. Adding new fidelity- -input pair (m, x) gives an increment of learning benefit A, 1 (Dis (y y Since we need to trade off to the cost Am, we can view there are Am independent copies of x (for the Ai particular fidelity m), and adding each copy gives Xm benefit. We choose the optimal and that
(D y (x() (D).
maximizes the benefit AL Since all the Am copies of x have the equal, biggest benefit (among all possible choices of inputs in Ω and fidelities in M), we choose to add them first (greedy strategy), which in total gives A, benefit their benefit does not diminish as each copy is added. Via dividing by Am and considering the copies, which each have unit weight, we can apply the standard submodular optimization analysis obtaining (1 O/e)OPT, at least until we encounter the budget constraint.
3.3 Algorithm Complexity
The time complexity of our weighted greedy optimization is O MG) where X is the cost for the lowest fidelity, G' is the cost of the underlying numerical optimization (e.g.. L-BFGS) and acquisition function computation (detailed (Li et al. 2022)). The space complexity O E (r+ d + 1)). which is to store at most B/A, pairs of input locations and fidelities, and their corresponding outputs (ie., the querying results). Therefore, both the time and space complexities are linear to the budget B.
Related Work
As an important branch of machine learning, active learning has been studied for long time (Balcan)
al. 2007 Settles 2009; Balcan et al.] 2009 Dasgupta 201T Hanneke al., 2014). While many prior works develop active learning algorithms for kernel based models, e.g., (Schohn and Cohn 2000
5

---

Tong and Koller 2001 Joshi et al. 2009; Krause et al. 2008 Li and Guo 2013 Huang et al. 2010 recent research tocus has transited to deep neural networks, Gal et al. (011)) used Monte-Carie (MC) Dropout (Gal and Ghahramani 2016) to generate posterior samples of the neural network output, based on which variety of acquisition functions, such as predictive entropy and Bayesian Active Learning by Disagreement (B (BALD) (Houlsby et al., 2011), can be calculated and optimized to query new examples in active learning. This approach has been proven successful in image classification. Kirsch et al. (2019) developed BatchBALD, a greedy approach that incrementally selects set of unlabeled images under the BALD principle and issues batch queries to improve the active learning efficiency. They show that the batch acquisition function based on BALD is submodular, and hence the greedy approach achieves a 1 1 1/c approximation. Other works along this line includes (Geifman and EI- Yaniv 2017: (Sener and Savarese 2018) based on core-set search, (Gissin and Shalev-Shward 2019 Ducoffe and Precioso| 2018) based on adversarial networks or samples, (Ash et al. 2019) based on the uncertainty evaluated in the gradient magnitude, etc.
Different from most methods, DMFAL (Li et al. 2022) conducts optimization- based, rather than pool-based active learning. That is, they optimize the acquisition function in the entire domain rather than rank set of pre-collected examples to label. It is also related to Bayesian experimental design (Kleinegesse and Gutmann 2020). In addition, DMFAL for the first time automatically queries examples at different fidelities, and integrates these examples in a deep multi-fidelity model to improve the active learning efficiency while reducing the cost which is critical in physical simulation and engineering design. The pioneer work of Settles et al. (2008) empirically studies how the human labeling cost can vary in practical active learning. but does not provide a scheme to identify multi- fidelity queries. Our work is an extension of eLi et al. 2022) to generate a batch of multi-fidelity to reduce the query correlations, improve the diversity and quality of the training examples, while respecting a given budget constraint. A counterpart in the Bayesian optimization domain is the recent work BMBO-DARN (Li et al., 2021), which considers batch multi-fidelity queries for optimizing a black-box function. From the high-level view, BMBO-DARN and our work employ a similar interleaving procedure, i.e., determining a new batch of queries by optimizing an acquisition function, issuing the queries to collect new examples, and updating the surrogate model. However, both the learning setting and acquisition function are different. More important, we consider the budget constraint while AMBO-DARN does not. Thereby, the computation and optimization techniques of the two works are totally different. The BMBO-DARN uses Hamiltonian Monte-Carlo samples of the single function output prediction and constructs empirical covariance matrices to compute the acquisition function, while our method and (Li et al., 2022) use the multi-variate delta method and matrix identities to overcome the challenge of the -inediimenioonl outputs. BMBO-- DARN uses alternating optimization to search over multiple fidelities, while our work develops a weighted greedy algorithm with additional theoretical guarantees to respect the budget constraint.
queries
5
Experiment
5.1 Solving Partial Differential Equations
We first evaluated BMFAL-BC in several benchmark tasks of computational physics, i.e., predicting the solution fields of partial differential equations (PDEs), including Heat, Poisson's. s, and Burgers' equations (Olsen-Kette 201T). A numerical solver was used to collect the training examples. High-fidelity examples were generated by running the solver with dense meshes, while low-fidelity examples by coarse meshes. The dimension of the output is the number of the grid points. For example, 50 x 50 mesh means the output dimension is 2, 500. We provided two-fidelity queries for each PDE, corresponding to 16 x 16 and 32 x 32 meshes. We also tested three-fidelity queries for Poisson's 's equation, with a 64 x 64 mesh to generate examples at the third fidelity. We denote the three-fidelity setting by Poisson-3, The input comprises of the PDE parameters and/or boundary/initiall condition parameters. The details are provided in (Li ct al. 2022). We ran the solver at each fidelity for many times, and computed the average running time. We normalized the average running time to obtain the querying cost at each fidelity, X 1, A2 = 3 and ' 10. To collect the initial training dataset for active learning, we generated 10 fidelity-1 examples and 2 fidelity-2 examples for in the two-fidelity setting, and 10, 5, and 2 examples of fidelity-1, 2, 3, respectively, for the threl-fidelity setting. The training inputs of the initial dataset were uniformly sampled from the domain. To evaluate the prediction accuracy, for each PDE, we randomly sampled 500 inputs, calculated the ground-trui truth outputs from much denser mesh: 128 x 128 for Burger's 's and Poisson's 's and 100 x 100

---

for Heat equation. The predictions at the highest fidelity were then interpolated to these denser meshes Zienkiewiez et al. 1977) to evaluate the error.
Competing Methods. Since currently there is not any budget-aware, batch multi-fidelity active learning approach (to our knowledge), for comparison, we first made simple extension of the statc-of-the-art multi-fidelity active learning method, DMFAL (Li et al.) 2022). Specifically, to obtain batch of queries, we ran DMFAL as is, namely, each step acquiring one example by maximizing (2) and then retraining the model, until the budget is exhausted or no new queries can be issued (otherwise the budget will be exceeded). We denote this method by (i) DMFAL-BC. Note that it is still sequentially querying and training inside each batch, but respects the budget. Next, to confirm the effectiveness of the proposed new acquisition function (3) (based on which, we propose our batch acquisition function (.) we ran active learning in the same away as AEFAL-BC, except the
acquisition function is replaced by as (m(,x) where ag is defined by B. To compute as, used Am
an Monte-Carbe approximation similar to (5) where the number of samples A is 20. We denote this method by (ii) MFAL-BC. In addition, we compared with (iii) DMFAL-BC-HR RF, which follows the running of DMFAL-BC, but each time, it randomly selects a fidelity m. then maximize the mutual information T (x), y a (x) (D) - the numerator of (2 - to identify the corresponding input. Similarly, we tested (iv) MFAL-BC-RF, which follows the execution of MFAL-BC, but each time, it randomly selects a fidelity m and maximizes a, (m, x). Again ag is computed by the Monte-Carlo approximation. For all these methods, we used L-BFGS for the input optimization. Finally, we tested (v) Batch-FR-BC. which randomly samples both the fidelity and input at each step (i.e. fully random), until the budget is used up or no more fidelities are available. Then the batch of the acquired examples were added into the dataset altogether to retrain the model. Note that there are other possible acquisition functions, e.g., the popular predictive variances and BALD (Houlsby et al., (201T) D. Their multi-fidelity versions have been tested and compared against DMFAL in (Li et (022)) and turns out to be inferior. Hence, we did not test their possible variants/extensioon// in our paper.
Settings and Results. All the methods were implemented by Pytorch (Paszke et al.] 2019). We followed the same setting as in Li et al. (2022) to train the deep multi-fidelity model (see Sec. 2.2). which employed two-layer NN at each fidelity, tanh activation. and the layer width was selected from {20, 40, 60, 80, 100} from the initial training data. The dimension of the latent output was 20. The learning rate was tuned from (10-4,5 x 10-2, 1,100$5 5 x 10-3 10-3, 10-2). We set the budget for acquiring each batch to 20 (normalized seconds), and ran each method to acquire 25 batches of training examples. We tracked the running performance of each method in terms of the normalized root-mean caaun--quue--eree (nRMSE). We repeated the experiment for five times, and report the average nRMSE vs. the accumulated cost (i.e., the summation of the corresponding X's in each acquired example) in Fig. [] The shaded region exhibits the standard deviation. As we can see, BMFAL-BC consistently outperforms all the competing methods by large margin. At very early stages, all the methods exhibit similar prediction accuracy. This is reasonable, because they started with the same training set. However, along with more batches of queries, BMFAL-BC shows better performance, and its discrepancy with the baselines is in general growing. addition, MFAL-BC constantly outperforms DMFAL-BC. Since they conduct the same one-by-one querying and training strategy, the improvement MFAL-BC reflects the advantage of our new single acquisition function (3) over the one used in DMFAL. Together these results have demonstrated the advantage of our method.
5.2 Topology Structure Optimization
Next, we applied our approach in topology structure optimization, which critical many manu- facturing and engineering design problems, such as 3D printing. bridge construction, and aircraft engine production. A topology structure is used to describe how to allocate the matcrial, e.g., metal and concrete, in a designated spatial domain. Given the environmental input, e.g., a pulling force or pressure, we want to identify the structure with the optimal property of interest, e.g., maximum stiffness. Traditionally, we convert it to a constraint optimization problem, for which we minimize compliance objective with a total volume constraint (Sigmund) 1997). Since the optimization often needs to repeatedly run numerical solvers solve relevant PDEs, the computation is very expensive. We aim to learn a surrogate model with active learning, which can predict the optimal structure outright given the environmental input.
Specifically, our task is to design a linear elastic structure in an L-shape domain discretized in [0, 1] x [0, 1]. The environmental input is load at the bottom half right and described by two
7

---

BMFAL-BC MFAL-BC-RF
MFAL-BC DMFAL-BC-RF
DMFAL-BC Batch-FR-BC
1.0
1.0
0.6
0.4
nRMSE
RRMSE
0.2
0.1
100
200
300
400
500
100
200 Accumulated cost
300
400
500
Accumulated cost
(a) Heat
(b) Poisson
1.0
1.0
0.6
0.4
nRMSS
nRMSE
0.1
0.2
0.1-
0
100
200
300
400
500
100
200
300
100
500
Accumulated cost
Accumulated cost
(c) Burgery
(d) Poisson-3
Figure 2: Normalized root-mean mea--ee---qure error (nRMSE) vs. accumulated data acquiring cost during batch active learning, with budget 20 (normalized seconds) per batch. There are two fidelities in acquiring the examples in (a-c) and three fidelities in (d). The results averaged over runs. The shaded region shows the standard deviation.
parameters: angle (in [0, 7) and location (in [0.5, 1)). Given a particular load, we aim to find the structure that has the maximum stiffness. To calculate the compliance in the structure optimization, we need to repeatedly apply the Finite Element Method (FEM) (Keshavarzzadeh et al. 2018), where the fidelity is determined by the mesh. In the active learning, the training examples can be acquired two fidelities. One corresponds output dimension of the two fidelities is therefore 2, 500 and 5, 625, respectively. The querying cost the normalized average time to find the optimal structure (with the standard approach), X = 1 and Ag = 3. To evaluate the performance, 500 test structures were randomly generated with 100 100 mesh. We interpolated the high-fidelity prediction of each method to the 100 x 100 mesh and then evaluated the prediction error.
x 50 mesh used in the FEM, and the other 75 x 75. The
At the beginning. we uniformly sampled the input (i.e., load angle and location) collect 10 structures at the first fidelity and 2 at the second fidelity, as the initial training set. We then ran all the active learning methods, with budget 20 per batch, to acquire 25 batches of examples. We examined the average nRMSE along with the accumulated cost of acquiring the examples. The results are reported consistently better than all
in Fig. 3a We can see that the prediction accuracy of BMFAL-BC the competing methods during the active learning. The improvement becomes larger when more examples are acquired. The results confirm the advantage of BMFAL-BC.
5.3 Predicting Spatial-Temponee Fields of Flows
Third, we evaluated BMFAL-BC in computational fluid dynamics (CFD). We considered a classical application where the flow is inside rectangular domain (represented by [0. 1] x [0, 1)) and driven

---

BMFAL-BC MFAL-BC-RP
MFAL-BC DMFAL-BC-RF
DMFAL-BC Batch-FR-BC
1.0
1.0
nRMSE
DRMSE
0.6
0.5
0.4
0.3
0.3
0.2
100
200
300
400
500
50
100
150
200
250
Accumulated cost
Accumulated cost
(a) Topology optimization
(b) Fluid dynamics
Figure 3: aRMSE vx. the accumulated cost in learning to predict topological structures and fulid dynamics.
The budget set 20 and 10 for (a) and (b), respectively.
BMFAL-BC-20
BMFAL-BC-30
BMFAL-BC-50
MFAL-BC-20
MFAL-BC-35
MFAL-BC-
DMFAL-BC-20
DMFAL-BC-35
DMFAL-BC-50
MFAL-BC-RF-20
MFAL-BC-RFFE
WFAL-BC-RF-50
DMFAL-BC-RF-20
DMFAL-BC-RF--35
RMFAL-BC-RF-85
Batch-FR-BC-20
Batch-FR-EC-35
Batch-FR-BC-50
10
1.0
06-
ARMSE
nAMSE
ARMSE
0.L-
0.1
200
600
600
200
400
600
20:
400
600
Accumulated cost
Accumulated cost
Accumulated cost
Heat
(b) Poisson
(c) Burgers
Figure 4: nRMSE vs. the accumulated cost under different budgets per batch: B E (20, 35, 50).
by rotating boundaries with a constant velocity (Bozeman and Dalton 1973). The velocities of different parts inside the flow will vary differently, and eventually lead to turbulent fluids. To compute the velocity field in the time spatial domain, we need to solve the incompressible Navier-Stokes equations (Chorin 1968), which is known be computationally challenging, especially under large Reynolds numbers. We considered the active learning task of predicting the first component of the velocity field 20 evenly spaced points in the temporal domain [0, 10). The training examples can be queried at two fidelities. The first fidelity uses a 50 x 50 mesh in the [o, 1] x [0, 1] spatial domain, and the second fidelity 75 75. Accordingly, the output dimensions are 50, 000 and 112, 500; the cost per query is X and A = 3. The input is five dimensional, including the velocities of the four boundaries and the Reynold number. The details are given in (Li al. (022). For testing, computed the solution fields of 256 random inputs, using 128 x 128 mesh We used the cubic-spline interpolation align the prediction made by each method to the 128 128 grid, and then calculated normalized RMSE. To conduct the active learning, we randomly generated 10 and 2 examples in each fidelity as the initial training set. We set the budget to 10, and ran each method acquire 25 batches. We repeated the experiment for five times, and examined how the average nRMSE varied along with the accumulated cost. As shown Fig. 3b BMFAL-BC keeps exhibiting superior predictive performance during the course of active learning. Again, the more examples acquired, the more improvement of BMFAL-BC upon the competing methods. The results are consistent with the previous experiments. Note that throughout these comparisons, we focus on the accumulated

---

cost of querying (or generating) new examples, because it dominates the total cost, and in practice the key bottleneck learning the surrogate model. For example, topology optimization (Sec (.2) and the fluid dynamic experiment, running a high-fidelity solver once takes 300-500 seconds on our hardware, while the surrogate training takes less than 2 seconds, and our weighted greedy optimization of the batch acquisition function (Algorithm[] I takes few seconds. One can imagine for practical larger-scale problems, the simulation cost, e.g., taking hours or even days to generate one example, can be even more dominant and decisive.
5.4 Influence of Different Budgets
Finally, we examined how the budget choice will influence the performance of active learning. To this end, varied the budget B in (20, 35, 50) and tested all the methods for Poisson's ss Burger's and heat equations. We used the same two-fidelity settings as in Sec. 5.1 For each budget, we ran the experiment for five times. We show the average nRMSE vs. the accumulated cost in Fig. 4 As can see, the larger the budget per batch, the better the running performance of BMFAL-BC. This is reasonable, because larger budget allows our method to generate more queries in each batch and in the meantime to account for their correlations or information redundancy. Accordingly, the acquired training examples are overall more diverse and informative. Again, BMFAL-BC outperforms all the competing methods under every budget. The improvement of BMFAL-BC is bigger under larger budget choices. This together demonstrates the advantage of batch active learning that takes into account the correlations between queries.
Conclusion
We have presented BMFAL-BC, budget-aware, batch multi-fidelity active learning algorithm for nigh-dimensione outputs. Our weighted greedy algorithm can efficiently generate batch of input- fidelity pairs for querying under the budget constraint, without the need for combinatorially searching over the fidelities, while achieving good approximation guarantees. The results on several typical computational physical applications are encouraging.
Acknowledgments
This work has been supported by MURI AFOSR grant FA9550-20-1-0038 and NSF CAREER Award IIS-2046295. JP thanks NSF CDS&FF-195335 11S-1816149, CCF-2115677, and Visa Research.
References
Ash, J. T., Zhang, C., Krishnamurthy, A., Langford, J., and Agarwal, A. (2019). Deep batch active learning by diverse, uncertain gradient lower bounds. In International Conference on Learning Representations.
Balcan, M.-F., Beygelzimer, A., and Langford, J. (2009). Agnostic active learning. Journal of Computer and System Sciences, 75(1):78-89.
Balcan, M.-F., Broder, A., and Zhang, T. (2007). Margin based active learning. In International Conference on Computational Learning Theory, pages 35-50. Springer.
Bickel, P. J. and Doksum, K. A. (2015). Mathematical statistics: basic ideas and selected topics. volume I, volume 117, CRC Press.
Bozeman, J. D. and Dalton, C. (1973). Numerical study of viscous flow in a cavity. Journal of Computational Physics, 12(3)3348-363.
Chorin, J. (1968). Numerical solution of the navier-stokes equations. Mathematics of computation, 22(104):755-76.
Conti, S. and O' D'Hagan, A. (2010). Bayesian emulation of complex multi-output and dynamic computer models. Journal statistical planning and inference, 140(3):40--551
Dasgupta, S. (2011). Two faces of active learning. Theoretical computer science, 412(1)) 4177119::767-17711
10

---

Ducoffe, M. and Precioso, F. (2018), Adversarial active learning for deep networks: margin based approach. arXiv preprint arXiv:1802.0984)
Gal, Y. and Ghahramani, Z. (2016). Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pages 1050-1059.
Gal, Y., Islam, R., and Ghahramani, Z. (2017). Deep bayesian active learning with image data. In International Conference Machine Learning, pages 1183-1192.
Geifman, and EI- Yaniv, R. (2017). Deep active learning over the long tail. arXiv preprint SSi1:17111.09411
Gissin, D. and Shalev-Shwartz, S. (2019). Discriminative active learning. arXiv preprint arXiv:1907.06347.
Hanneke, et al. (2014). Theory of disagreement-based active learning. Foundations and Trends@ in Machine Learning, 7(2-))131-309,
Houlsby, N., Huszár, F., Ghahramani, Z., and Lengyel, M. (2011). Bayesian active learning for classification and preference learning. arXiv preprint arXiv: 1112.575..
Huang, S.-J., Jin, R., and Zhou, Z.-H. (2010). Active learning by querying informative and represen- tative examples. In Advances in neural information processing systems, pages 892-900.
Joshi, A. J., Porikli, F., and Papanikolopoulos, N. (2009). Multi-class active learning for image classification. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 2372-2379. IEEE.
Kato, T. (2013). Perturbation theory for linear operators, volume 132. Springer Science & Business Media.
Kennedy, M. C. and O' O'Hagan, A. (2000). Predicting the output from a complex computer code when fast approximations are available. Biometrika, 87(1):1-13.
Keshavarzzadch, V., Kirby, R. M., and Narayan, A. (2018), Parametric topology optimization with multi-resolution finite element models. arXiv preprint arXiv: 100..:088.110000 10367.
Kingma, D. and Welling, M. (2013). Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114,
Kirsch, A., van Amersfoort, J., and Gal, Y. (2019). BatchBald: Efficient and diverse batch acquisition for deep bayesian active learning. In Advances in Neural Information Processing Systems, pages 7026-7037.
Kleinegesse, S. and Gutmann, M. U. (2020). Bayesian experimental design for implicit models by mutual information neural estimation. In International Conference on Machine Learning, pages PMLR.
Krause, and Golovin, D. (2014). Submodular function maximization. Tractability, 3:71-104.
Krause, A. and Guestrin, C. F. (2005). Near-optimal nonmyopic value of information in graphical models. In Proc. of Uncertainty in Artificial Intelligence (UAI).
Krause, A., Singh, A., and Guestrin, C. (2008). Near-optimal sensor placements gaussian processes: Theory, efficient algorithms and empirical studies. Journal of Machine Learning Research, 2(F5))235-284
Leskovec, J., Krause, A., Guestrin, C., Faloutsos, C., VanBriesen, J., and Glance. N. (2007). Cost- effective outbreak detection in networks. In Proceedings of the 13th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 420-
Li, S., Kirby, R., and Zhe, S. (2021). Batch multi-fidelity bayesian optimization with deep auto- regressive networks. Advances in Neural Information Processing Systems, 34:25463-25475.

---

Li, S., Wang. Z., Kirby, R. M., and Zhe, S. (2022). Deep multi-fidelin active learning of high- dimensional outputs. Proceedings of the Twenty-Fifth International Conference on Artificial Intelligence and Statistics.
Li, X. and Guo, Y. (2013). Adaptive active learning for image classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 859-866.
Mendes, P., Casimiro, M., Romano, P., and Garlan, D. (2020). Trimtuner: Efficient optimization of machine learning jobs in the cloud via sub-sampling. In 2020 28th International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems (MASCOTS), pages 1-8, IEEE.
Ochlert, G. (1992). note on the delta method. The American Statistician, 46(1):27-29.
Olsen-Kettle, L. (2011). Numerical solution of partial differential equations. Lecture notes at University of Queensland, Australia.
Paszke, A., Gross, S.. Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, Lin, Z., Gimelshein, N., Antiga, I.., et al. (2019). Pytorch: An imperative style, high-performance deep learning library. In Advances neural information processing systems, pages 8026-8037.
Schohn, G. and Cohn, D. (2000). Less is more: Active learning with support vector machines. In ICML. volume 2, page 6. Citeseer.
Sener, O. and Savarese, S. (2018). Active learning for convolutional neural networks: A core-set approach. In International Conference on Learning Representations.
Settles, B. (2009). Active learning literature survey. Technical report, University of Wisconsin- Madison Department of Computer Sciences.
Settles, B., Craven, M., and Friedland, L. (2008). Active learning with real annotation costs. In Proceedings of the NIPS workshop cost-sensitive learning, volume 1. Vancouver, CA:
Sigmund, O. (1997). On the design of compliant mechanisms using topology optimization. Journal of Structural Mechanics, 25(4):493-524
Tong, S. and Koller, D. (2001). Support vector machine active learning with applications to text classification. Journal of machine learning research, 2(Nov):45
Zienkiewicz, O. C., Taylor, R. L., Zienkiewicz, O. C., and Taylor, R. L. (1977). The finite element method, volume 36. McGraw-hill London.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
12

---

(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasng new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "ud28yyWZX0a", "reviewer": "Reviewer_UwmX", "summary": "\nThe paper proposes BMFAL an extension of DMFAL for batch multi-fidelity active learning with budget constraints. The proposed method is a greedy sequential algorithm that selects datapoints which maximise the information gain. The experiments show that BMFAL outperform a naive extension of DMFAL in the batch multi-fidelity active learning with budget constraints. ", "questions": "\n - I think Section 3.3 could be improved. The algorithm complexity in 3.3 ignores the cost of searching (i.e., performing the argmax) $x \\in \\Omega$. In addition, it ignores the cost of computing the acquisition function which is not negligible. The space complexity can also be simplified to $O(\\frac{B}{\\lambda_1}(r+d))$ since the constant is negligible. \n - Figure 2 is unnecessarily large. The figure can be made smaller and further discussion regarding the limitations of the method should be included.", "limitations": "The authors could include further discussion regarding in what specific areas batch multi-fidelity active learning with budget constraints is used. The discussion should include the potential negative societal impact of those areas.", "rating": 7, "confidence": 2, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\nStrengths:\n - To my knowledge, this is the first method proposed for Batch Multi-Fidelity Active Learning with Budget Constraints. The setting is very specific but it's an important setting. Although not mentioned in the paper, multi-fidelity methods are also very useful for biological sequence design or drug design. In these settings, wet-lab experiments are very expensive, so there are methods to approximate them at various fidelities.\n - The experiments are conclusive and show a clear benefit of BMFAL over DMFAL in the batch setting. \n\nWeaknesses:\n - The experiments are limited to up to 3 fidelities. However, I would imagine that in certain settings we would want a larger number of fidelities. It would be interesting to see how this scales to a larger number of fidelities.\n - No code is included but the details appear sufficient to reproduce the experiments.\n\n\nOverall, I would rate the paper:\n\nNovelty: Medium\n\nClarity: High\n\nSignificance: Medium-High\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "sHscd97H45l", "reviewer": "Reviewer_Uvz4", "summary": "Multi-fidelity learning is applied whenever observations generated from a cheap but less precise surrogate model are combined with limited observations from the actual process, such that a mapping can be learned between low and higher fidelity outputs. This is particularly challenging when the output dimensionality becomes greater at higher fidelities. In this work, the authors focus on the problem of deciding at which fidelity and location of the input domain the next sample points should be acquired. In order to mitigate the risk of successive sequential acquisitions focusing on the same limited region, the authors propose a batch active learning scheme which jointly selects different input points to sample across different fidelities, subject to a computational budget. Via a range of experiments, the authors show how the selection procedure results in more efficient learning than other competing methods when given with the same sampling budget.", "questions": "My largest reservations around this work lie around the lack of citation and connection to _“Batch Multi-Fidelity Bayesian Optimization with Deep Auto-Regressive Networks”_, which shares a lot of the motivation and approach of the work presented here. Again, while I appreciate that the two use-cases have different requirements, the overall significance and novelty of this paper come across as slighter in light of this other work.\n", "limitations": "I appreciated the discussion on the quality of the weighted greedy optimisation procedure, which was also appropriately hinted at in the introduction of this work. As mentioned earlier, providing an anecdotal example of where this could occur (even if synthetic) would be helpful.\n\nThere should not be any ethical or societal issues tied to the publication of this work.\n", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n_[Significance]_ - As evidenced by the variety of application domains featured in the experiments section, multi-fidelity modelling is an emerging topic in many practical areas. This paper tackles an interesting problem within this space that does not appear to have been directly investigated for multi-fidelity regression prior to this work.\n\n*[Clarity]* - The paper is well-written overall and easy to follow. There are a few typos here and there, but these aren’t distracting and should be easily fixed in a second revision.\n\n*[Evaluation]* - I appreciated how the experiments were varied by leveraging datasets sourced from different domains. The choice of methods used for comparison is also appropriate, and the plots effectively illustrate both the overall performance and variance of the methods. The results indicate that the proposed approach consistently performs better than competing methods, which adds credence to the usefulness of the contributions presented in this work. \n\n**Weaknesses**\n\n*[Novelty]* - The paper bears similarities to another recent paper, _“Batch Multi-Fidelity Bayesian Optimization with Deep Auto-Regressive Networks”_, which shares a lot of the motivation and set-up for this work, with the exception that it is centred on the single-output Bayesian optimisation case (rather than the high-dimensional outputs considered here). While I appreciate that the two set-ups are different, there is considerable overlap which makes the lack of any reference to that paper particularly puzzling. I would like to see this featured more prominently in a future revision, as it is currently conspicuous by its absence.\n\n*[Evaluation]* - One disappointing aspect about the evaluation section is that most of the text is devoted to explaining the datasets and the parameters used for the different methods, whereas the insights and findings themselves are limited to one or two brief sentences at the end of each experiment. While the results of the graphs are fairly self-evident, further insights or anecdotal examples of where the batch selection is particularly effective would be welcome.\n\n*[Completeness]* - One interesting limitation highlighted in the presentation of the method is the potential degradation of the greedy optimisation mechanism - however, this isn’t revisited in the experimental evaluation, whereas it would add to the paper’s completeness if the authors could include an example of how/where such behaviour manifests itself.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "RFh-n4AP8Pb", "reviewer": "Reviewer_SBrH", "summary": "This paper proposes a novel batch active learning algorithm for multi-fidelity tasks. While prior work only queries one example at a time, the authors propose novel mutual information objectives that also take into account of the correlation within a batch. Moreover, the authors provide a greedy relaxation algorithm in solving the optimization, which is further shown theoretically to be an (1 - 1/e)-approximation of the optimal objective. Extensive experiments have been conducted on four physics applications while outperforming the existing baseline significantly.", "questions": "1. Is knowing a-priori the costs lambda_m for each fidelity m a reasonable assumption? Does it make sense to assume they are fixed costs?\n2. Other issues mentioned in the weakness section.", "limitations": "1. An advantage of batch active learning is the fact we could parallelize annotation. Now, consider if the cost of annotation comes only from time constraints, then the annotation cost of a batch actually comes from the example that takes the longest to annotate (highest fidelity) rather than the sum of the costs from each fidelity. It is not immediate to me that the method proposed in this paper is sufficient to solve such settings. Therefore, I recommend the authors to properly address as a limitation in their paper.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The paper targets an important issue not addressed by prior work, which is the diversity among collected examples. The objectives are straight-forward and very intuitive.\n2. The algorithm appears to be rather efficient and also enjoys theoretical worst case guarantees.\n3. The experiments clearly shows the proposed algorithm's advantage in the batch setting.\n\nWeaknesses:\n1. The high labeling cost incurred by physical simulations motivated this work on applying active learning. However, when addressing accumulated cost, it does not seem to take into account the extra computation cost of the active learning algorithm. In the time complexity section, it is also lacking a comparison with existing work.\n2. Moreover, a comparison with uniform random sampling strategies may be informative, especially when taking into account the extra computation cost incurred by the active learning algorithm. If they significantly underperforms, I think it's worth addressing in the text at least.\n\n\nMinor Suggestion:\n1. On the last line of equation (6), it's better to put sum_{hat{m}} lambda_{hat{m}} into parentheses. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "McyGURThcpT", "reviewer": "Reviewer_v34g", "summary": "The paper presentes BMFAL-BC, a new batch multi-fidelity active learning approach that takes into account budget constraints. Differently from prior work, BMFAL-BC selects query points at different fidelities ensuring both that the queried points have low correlation among themselves and that the budget is constraint is complied with. The intuition behind ensuring low correlation among the queried points is that higher diversity will contribute to a better learning process, i.e., higher learning efficiency, hence decreasing the required number of queries and the cost of the learning process. The authors prove bounds for their algorithm and compare their approach against state-of the art multi fidelity active learning approaches, showing that BMFAL-BC finds better solutions at cheaper learning costs. ", "questions": "**a)** I don't understand the meaning of \"the cost of computing examples at each fidelity, was set to the normalized average running time\". Does this mean you gathered queries from each fidelity multiple times, measured how long each query took, and set that as the cost? Also, \"the normalized average running time\" of what?\n\n**b)** In the sentence \"For active learning, we randomly generated {10, 2} and {10, 5, 2} training examples\", what does the 10 and 2 stand for in {10, 2}? Same question for {10, 5, 2}.\n\n**c)** When BMFAL-BC selects a batch, does it test all points in that batch? Also, do all points in a batch belong to the same fidelity?\n\n**d)** If my understanding is correct, BMFAL-BC has a budget B per batch, which it can spend to evaluate several queries. In your implementation of DMFAL-BC do you allow it to select as many queries/points as it needs to reach budget B before you consider the batch as complete? Or do you terminate\t the run the first time DMFAL-BC is complete?\n\n**e)** Is the accumulated cost equal to number of evaluated batches x B?\n\n**f)** How many more queries does BMFAL-BC execute compared to DMFAL-BC?\n\n**g)** What is the time complexity/overhead of the proposed method compared to existing methods?\n\n**Comments:**\n- PDE is introduced in line 224 however it is used already in line 25. Maybe introduce it the first time you use it for clarity.\n- Lines 38-42 lack supporting references\n- The sentence in lines 226-227 is not complete.\n- The legend of Figures 2 and 3 could be made easier to read: all methods have BC in their names, so BC is redundant. Also, maybe you could align the methods with the same name, i.e., MFAL-BC would be above MFAL-BC-RF and DMFAL-BC above DMFAL-BC-RF\n- typo in line 485? when --> well\n", "limitations": "No limitations were discussed. Under which conditions can the method be applicable? Do the underlying problems need to adhere to any specific format? Are there any underlying assumptions? For instance with respect to the shape of the function under optimisation?", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Originality:** The authors propose a new method that extends existing work in order to account for budget constraints and thus decrease the cost of the learning process. This is a novel combination and application of existing techniques to the context of batch multi-fidelity active learning. As far as I know, active learning related work is adequately cited however there is significant (and recent) work in the context of multi-fidelity hyper-parameter tuning optimisation which is not mentioned. Specifically, the authors may find works such as [1] and [2] relevant for their research. [1] seems a particularly good citation for line 123.\n\n**Quality:** The claims seem to be sound, however I did not carefully check the math and proofs. The evaluation methods are appropriate and consistent with those used by related work. The weaknesses and limitations could be discussed more explicitly. One possible limitation that is not discussed is the use of one NN for each fidelity. It would be interesting to explore research directions that address this overhead and use only one NN, for example.\n\n**Clarity:** The active learning setting could be better described. It is not clearly explained how each batch of samples actively contributes to the learning. There could be more information on the experiments and parameters tested (in table format) in the appendix. This would make reproducibility easier. Also, some guidance on possible ways to set the budget could aid potential users of this approach.\n\n**Significance:** The problem tackled in the paper is a relevant one and having approaches to optimize the search and learning process at lower costs is valuable for the community. The proposed method is better than existing ones on the evaluated scenarios.\n\n\n[1] Mendes, Pedro, et al. \"Trimtuner: Efficient optimization of machine learning jobs in the cloud via sub-sampling.\" 2020 28th International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems (MASCOTS). IEEE, 2020.\n\n[2] Klein, Aaron, et al. \"Fast bayesian optimization of machine learning hyperparameters on large datasets.\" Artificial intelligence and statistics. PMLR, 2017.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept