Batch Multi-Fidelity Bayesian Optimization with  Deep Auto-Regressive Networks
Authors: Shibo Li
Robert Kirby
Shandian Zhe
Venue: NeurIPS 2021 Poster (2021)
Keywords: Multi-fidelity Bayesian Optimization
Batch Acquisition Function
Full Auto-Regressive Modeling
Multi-fidelity Max-value Entropy Search

Abstract: Bayesian optimization (BO) is a powerful approach for optimizing black-box, expensive-to-evaluate functions. To enable a flexible trade-off between the cost and accuracy, many applications allow the function to be evaluated at different fidelities.  In order to reduce the optimization cost while maximizing the benefit-cost ratio,  in this paper we propose Batch Multi-fidelity Bayesian Optimization with Deep Auto-Regressive Networks (BMBO-DARN). We use a set of Bayesian neural networks to construct a fully auto-regressive model, which is expressive enough to capture strong yet complex relationships across all the fidelities, so as to improve the surrogate learning and optimization performance. Furthermore, to enhance the quality and diversity of queries, we develop a simple yet efficient batch querying method, without any combinatorial search over the fidelities. We propose a batch acquisition function based on Max-value Entropy Search (MES) principle, which penalizes highly correlated queries and encourages diversity. We use posterior samples and moment matching to fulfill efficient computation of the acquisition function, and conduct alternating optimization over every fidelity-input pair, which guarantees an improvement at each step.  We demonstrate the advantage of our approach on four real-world  hyperparameter optimization applications.

Batch Multi-Fidelity Bayesian Optimization with Deep Auto-Regressive Networks
Shibo Li, Robert M. Kirby, and Shandian Zhe
School of Computing. University of Utah
Salt Lake City, UT 84112
shibolcs. utah. edu, wirkykrrroooo wtw..eaaeeeeo edu, che@cs.utah....................a edu
Abstract
Bayesian optimization (BO) is powerful approach for optimizing black-box, expensive-to-evaluate functions. To enable a flexible trade-off between the cost and accuracy, many applications allow the function be evaluated different fidelities. In order to reduce the optimization cost while maximizing the benefit- cost ratio, in this paper we propose Batch Multi-fidelity Bayesian Optimization with Deep Auto-Regressin neural networks construct fully rege-segrese model, which is expressive enough to capture strong yet complex relationships across all the fidelities, so improve the surrogate learning and optimization performance. Furthermore, to enhance the quality and diversity of queries, we develop simple yet efficient batch querying method, without any combinatorial search over the fidelities. We propose a batch acquisition function based on Max-value Entropy Search (MES) principle, which penalizes highly correlated queries and encourages diversity. We use posterior samples and moment matching to fulfill efficient computation of the acquisition function, and conduct alternating optimization over every fidelity-input pair, which guarantees improvement each step. We demonstrate the ad advantage our approach on four real-world hyperparameter optimization applications.
Networks CEMBO-DARY We use set of Bayesian
Introduction
Many applications demand we optimize a complex function of an unknown form that expensive to evaluate. Bayesian optimization (Mockus, 2012 Snoek et al.) 2012) is powerful approach to optimize such functions. The key idea is to use a probabilistic surrogate model, typically Gaussian processes (Rasmussen and Williams] 2006), to iteratively approximate the target function, integrate the posterior information to compute and maximize an acquisition function so as to generate new inputs at which query, update the model with new examples, and meanwhile approach the optimum.
In practice, to enable a flexible trade-off between the computational cost and accuracy, many appli- cations allow us to evaluate the target function at different fidelities. For example, to evaluate the performance the hyperparameters for machine learning model, we can train the model thoroughly, ie., with sufficient iterations/epochs, to obtain the accurate evaluation (high-fideliy yet often costly) or just run a few iterations/epochs to obtain a rough estimate dow-fidelity but much cheaper).
Many multi multi-fidelity BO algorithms (Lam et al. 2015 Kandasamy et al., 2016 Zhang et al. 2017 Song al. 2019 Takeno et al. 2019) have therefore been proposed identify both the fide fitiittes and inputs at which to query, so as to reduce the cost and achieve a good benefit-cost balance. Notwithstanding their success, these methods often overlook the strong yet complex relationships between different fidelities or adopt an over-simplifie assumption, (partly) for the sake of convenience in nalculaiin/mmxxiizzngg the acquisition function considering fidelities. This, however, can restrict the performance of the surrogate model, impair the optimization efficiency and increase the cost. For
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

example, Lam et al. (2013; Kandasamy et al. (201)) learned an independent GP for each fidelity, Zhang al. (2017) used multitask GPs with a convolved kernel for multi-fidellit modeling and have to use simple smoothing kernel (e.g., Gaussian) for tractable convolutions. The recent work (Takeno) et al. 2019) imposes a linear correlation across different fidelities. In addition, the standard one-by- one querying strategy needs to sequentially run each query and cannot utilize parallel computing resources to accelerate, e.g. multi-core CPUs/GPUs and clusters. While incrementally absorbing more information, it does not explicitly account for the correlation between different queries, hence still has a risk to bring in highly correlated examples that includes redundant information.
To address these issues. we propose BMBO- BMBO-DARN, novel batch multi-fidelity Bayesian optimization method. First, we develop a deep regresseeeee model to integrate training examples at various fidelities. Each fidelity is modeled by Bayesian neural network (NN), where the output predicts the objective function value at that fidelity and the input consists of the original inputs and the outputs of all the previous fidelities. In this way, our model is adequate to capture the complex, strong correlations (e.g., nonstationary, highly nonlinear) across all the fidelities to enhance the surrogate learning. We use Hamiltonian Monte-Carlo (HMC) sampling for posterior inference. Next, to improve the quality of the queries, we develop a simple yet efficient method to jointly fetch a batch of inputs and fidelities. Specifically, we propose batch acquisition function based on the state-of-the the- Max-value Entropy Search (MES) principle (Wang and Jegelka, (017). The batch acquisition function explicitly penalizes highly correlated queries and encourages diversity. To efficiently compute the acquisition function, we use the posterior samples of the NN weights and moment matching to construct multi-variate Gaussian posterior for all the fidelity outputs and the function optimum. To prevent a combinatorial search over multiple fidelities in maximizing the acquisition function, we develop an alternating optimization algorithm cyclically update each pair of input and fidelity, which is much more efficient and guarantees an improvement at each step.
For evaluation, we examined BMBO-DARN in both synthetic benchmarks and real-world applications. The synthetic benchmark tasks show that given a small number of training examples, our deep auto- regressive model can learn a more accurate surrogate of the target function than other state-of-the-arr multi-fidelity BO models. We then evaluated EMBO-DARA on four popular machine learning models (CNN, online LDA, XGBoost and Physics informed NNs) for hyperparameter optimization. BMBO-DARN can find more effective hyperparameters leading to superior predictive performance, and meanwhile spends smaller total evaluation costs, as compared with state-of-the-art multi fulti--idelity BO algorithms and other popular hyperparameter tuning methods.
2 Background
Bayesian Optimization (BO) (Mockus et al. 1978 Snoek et al. 2012) is a popular approach for optimizing black-box functions that are often costly evaluate and cannot provide exact gradient information. BO learns probabilistic surrogate model to predict the function value across the input space and quantifies the predictive uncertainty. At each step, we use this information to compute an acquisition function to measure the utility of querying at different inputs. By maximizing the acquisition function, we find the next input at which to query, which is supposed to be closer to the optimum. Then we add the new example into the training set to improve the accuracy of the surrogate model. The procedure repeated until we find the optimal input or the maximum number of queries have been finished. There are a variety of acquisition functions, such as Expected Improvement (EI) (Mockus et al. 1978) and Upper Confidence Bound (UCB) (Srinivas et al. 2010). The recent state-of-the-art addition is Maximum-value Entropy Search (MES) (Wang and Jegelka 2017
a(x) T(J((x).
where I((,-)) :) is the mutual information, f(x) is the objective function value at x, f the minimum, and D the training data collected far for the surrogate model. Note that both f(x) and f are considered as generated by the posterior of the surrogate model given D; they are random variables.
The most commonly used class of surrogate models is Gaussian process (GP) (Rasmussen and Williams 2006). Given the training dataset X = [ xW] and y = [y1,..., UNN GP assumes the outputs y follow multivariate Gaussian distribution, p(y = =Nyy + el), where m is the mean function values at the inputs X, often set to 0, v is the noise variance, and K is a kernel matrix on X. Each [K]; = x(x;, Xj), where sc(.,)) :) is a kernel function. For example, popular one is the RBF kernel, x,) exp (-3--1|x; x, 112). An important advantage of GPs

---

0
(1
fi
0
Figure I: Graphical representation the deep auto-reaaa auto-regressive model in BMBO-DARN, The output at cach fidelity fm (x) (1 & m < M) is calculated by (deep) neural network.
their convenience in uncertainty quantification. Since GPs assume any finite set of function values follow multi-variate Gaussian distribution, given test input &, can compute the predictive (or posterior) distribution p (f(*) & y) via a conditional Gaussian distribution, which simple and analytical.
Multi- Multi-Fidelity BO. Since evaluating the exact value of the object function is often expensive, many practical applications provide multi-fidelity evaluations (fi (x)....... JM (x)] to allow us to choose trade-off between the accuracy and cost. Accordingly, many multi-fidelitg BO algorithms have been developed to select both the inputs and fidelities to reduce the cost and to achieve a good balance between the optimization progress and cost, i.e., the benefit-cost ratio. For instance, MF- GP-UCB (Kandasamy et al. 2016) sequentially queries at each fidelity (from the lowest one, i.e., m 1) until the confidence band is over a given threshold. In spite of its great success and guarantees in theory, MF-GP-UCB uses set of independent GPs to estimate the objective each fidelity, and hence ignores the valuable correlations between different fidelities. MF-PES (Zhang et al.) 2017) uses multi-task GP surrogate where each task corresponds to fidelity, and convolves a smoothing kernel with the kernel a shared latent function to obtain the cross-covariance. The recent MF-MES (Taeeno al. 2019) also builds multi-task GP surrogate, where the covariance function is
(fm (x), fm (x'))
Σ ا=ز (Mmy Mmm's + (1 m') .am)) (x(x
where Omj > 0. 1()) is the indicator function, {umj 15=1 is d latent features for each fidelity m, and (p are d bases kernels, usually chosen as a commonly used stationary kernel, egg., RBF
Deep Auto-Regressive Model for Multi-Fidelity Surrogate Learning
Notwithstanding the elegance and success of the existing multi-fidelity BO methods, they often ignore or oversimplify the complex, strong correlations between different fidelities, and hence can be inefficient for surrogate learning, which might further lower the optimization efficiency and incur more expenses. For example, the state-of-the-arr methods MF-GP-UCB (Kandasamy al.) 2016) estimate a GP surrogate for each fidelity independently; MF-PES (Zhang et al. (017) has to adopt simple form for both the smoothing and latent function kernel (e.g. Gaussian and delta) to achieve an analytically tractable convolution, which might limit the expressivity in estimating the pross-fidelity covariance; MF-MES (Takeno et al., 2019) essentially imposes a linear correlation structure between different fidelities - for any input x, K. (f (x), ( = u Umz + O. where
Um [uml, Umd] and ām Σラ O'yy we use RBF basis kernel (see (2).
To overcome this limitation, develop deep auto-regressive model for multi-fidelity surrogate learning. Our model is expressive enough to capture the strong, possibly very complex (e.g., highly nonlinear, nonstationary) relationships between all the fidelities to improve the prediction (at the highest fidelity). As such, our model can more effectively integrate multi-fidelity training information better estimate the objective function.
Specifically, given M fidelities, we introduce a chain of M neural networks, each of which models one fidelity and predicts the target function at that fidelity. Denote by Xm. Wm. and e'w... (.) the NN input, parameters and output mapping at each fidelity m. Our model is defined as follows.
X. xx; fi (x); ...; fm-
fm (x) = Jww (xm), yo (x) = fm (x) + Em.
(3)
where x1 = x, Sm (x) is the prediction (i.e., NN output) at the m-th fidelity, you (x) is the observed function value, and 6m is random noise, 6m (em 00, T,1). We can see that each input xm.
3

---

consists of not only the original input x of the objective function, but also the outputs from all the previous fidelities. Via a series of linear projection and nonlinear activation from the NN, we obtain the output at fidelity In this way, our model fully exploits the information from the lower fidelities and can flexibly capture arbitrarily complex relationships between the current and all the previous fidelities by learning an NN mapping, fm (x) r/w... (Xm= Ji (x).... fm-1(x)) (x)).
We assign standard Gaussian prior distribution over each element of the NN parameters W
(W..... War J. and Gamma prior over each noise precision, p(Tm) = Gam Tm (Co, bo). Given
Nm. M the dataset D = { (Xmmm Fnm) ( n= m== 1. the joint probability of our model is given by
M
Ni
p(W, T, = ((ve((W))o.) II (Tm( lao, bo) II N (ymml (Xmm), Tm (4)
where [T...... Jar] X {Xam. J. ) [8/m J. and vec(.) is vectorization. The graphical representation of our model is given in Fig. I We use Hamiltonian Monte Carlo (HMC) (Neal et al. 201T) sampling to perform posterior inference due to its unbiased, high-quality uncertainty quantification, which is critical to calculate the acquisition function. However, our method allows us to readily switch to other approximate inference approaches as needed (see Sec. 44. e.g. stochastic gradient HMC (Chen et al. 2014) used in the excellent work of Springeeberg et al. (0166).
Batch Acquisition for Multi-Fidelity Optimization
Given the posterior of our model, we aim to compute and optimize an acquisition function to identify the input and fidelity at which to query next. Popular BO methods query at one input each time and then update the surrogate model. While successful, this one-by-one strategy has to run each query sequentially and cannot take advantage of parallel computing resources (that are often available in practice) to further accelerate, such as multi-core CPU and GPU workstations and computer clusters. In addition, the one-by-one strategy although gradually integrates more data information. it lacks an explicit mechanism to take into account the correlation across different queries, hence still has a risk to bring in highly correlated examples with redundant information, especially in the multi-fidelity setting, e.g., querying at the same input with another fidelity. To allow parallel query and to improve the query quality and diversity, we develop a batch acquiring approach to jointly identify a set of inputs and fidelities at time, presented as follows.
4.1 Batch Acquisition Function
We first propose a batch acquisition function based on the MES principle (Zhang et al. 2017) (see ①.. Denote by B the batch size and by {A } the cost of querying at M fidelities. We want to jointly identify B pairs of inputs and fidelities (x1, m.).... (XB. mB) at which to query. The batch acquisition function is given by
Obatch (X m) 11(fm, (xi).... Sma (xB)).
(5)
ΣB Neml Amz
where X = -xi..... Xn) and m = [m.,..., m).. m) As we can see, our batch acquisition function explicitly penalizes highly correlated queries, encouraging joint effectiveness and diversity if between the outputs {m (x+) are high correlations, the mutual information in the numerator will decrease, Furthermore, by dividing the total querying cost in (5), the batch acquisition function expresses a balance between the benefit of these queries (in probing the optimum) and the price, i.e. benefit-cost ratio. When we set 1, our batch acquisition function is reduced to the single one used in (Tkenoo et al. 2019).
4.2 Efficient Computation
Given X and m, the computation of (5) is challenging, because it involves the mutual information between a set of NN outputs and the function optimum. To address this challenge, we use posterior samples and moment matching to approximate / as a multi-variate Gaussian distribution, where f = [fm, (xi),..., S (x=)]. Specifically, we first draw posterior sample of the NN weights W from our model. We then calculate the output at each input and fidelity to obtain a sample of f,
and maximize (or minimize) fM(() to obtain a sample of f*. We use L-BFGS (Liu and Nocedal

---

1989) for optimization. After we collect L independent samples fif..i)...... ft). we can estimate the first and second moments of h = f" namely, mean and covariance matrix.
L (h, - whj - =)
-
L
E = L- j=1
where each hū- = [Bj We then use these moments to match multivariate Gaussian posterior, p(h(D) = Nh, E). Then the mutual information can be computed with a closed form,
(6) where ΣΦ - Σ B, 1 BJ, ie., the first B B sub-matrix along the diagonal, which is the posterior covariance of f, and o.. E[B + 1, B + 1J. i.e. the posterior variance of f*. The batch acquisition function is therefore calculated from
I(f, (D) (fI)D) + H((ID) Hff, (') = log Exr log o.. log [Σ],
Obatch X, m
(log Eel + log o.. log ΣE)
Amp
Note that E is a function of the inputs X and fidelities m and hence so are its submatrix and elements, Ee and a... To obtain reliable estimate of the moments, we set L = 100 in our experiments. Note that our method can applied along with any posterior inference algorithm, such as variational inference and SGHMC (Chen et al. 22144), as long as we can generate posterior samples of the NN weights, not restricted to the HMC adopted in our paper.
4.3 Optimizing Batch of Fidelities and Inputs
Now, we consider maximizing to identify B inputs X and their fidelities m at which to query. However, since the optimization involves a mix of continuous inputs and discrete fidelities, is quite challenging. A straightforward approach would be enumerate all possible configurations of m, for each particular configuration, run a gradient based optimization algorithm to find the optimal inputs, and then pick the configuration and its optimal inputs that give the largest value of the acquisition function. However, doing so is essentially conducting a combinatorial search over B fidelities, and the search space grows exponentially with B, i.e., (MM)) O log M). Hence, it will be very costly, even infeasible for a moderate choice of B.
To address this issue, we develop an alternating optimization algorithm. Specifically, we first initialize all the queries, @ = {(x), mm),..................... (xB, mB)). say, randomly. Then each time, we only optimize one pair of the input and fidelity (xx; m1 I h < B), while fixing the others. We cyclically update each pair, where each update is much cheaper but guarantees to increase Obatch- Specifically, each time, maximize
Obatch, (X,, m) I(F (fm (f)())).
(8)
Am + ΣjfR Am,
where F-R [fm, (x,)li # k). Note that the computation of (8) still follows (7). We set (xx, m.) to the optimum (x", m) and then proceed to optimize the next input location and fidelity (X++1, m++1)
in @ with the others fixed. We continues this until we finish updating all the queries in Q, which corresponds to one iteration. We can keep running iterations until the increase of the batch acquisition function is less than a tolerance level or a maximum number of iterations has been done. Suppose we ran G' iterations, the time complexity is O(GM B), which is linear in the number of fidelities and batch size, and hence is much more efficient than the naive combinatorial search. Our multi-fidelity BO approach is summarized in Algorithm
Related Work
Most Bayesian optimization (BO) (Mockus, 2012 Snoek et al.) 2012) methods are based on Gaussian
processes (GPs) and a variety of acquisition functions, such as (Mockus et al. 1978) Auer 2002
Srinivas et al. 2010 Hennig and Schuler 2012 Hernández-Looato et al. 2014: Wang and Jegelka
2017] Kandasamy al. 7b; Garrido-Merehhn and Hernández-L.ote
2020). Snoek et al. (2015)
showed Bayesian neural networks (NNs) can also be used as general surrogate model, and has
5

---

Algorithm 1 BMBO-DARN (D, B, M, T. {Am]m=)
Learn the deep auto-regressive model on with HMC.
for t t==== ,T' do
Collect batch of B queries, Q {(x,, my m=))R==> k= with Algorithm 2
B
Query the objective function value at each input X5 and fidelity my Q
D <
D U {(x,, yk, m < < B).
Re-train the deep auto-regressive model on D with HMC.
end for
Algorithm 2 BatchhAcquisttion(((An)))) L, G)
Initialize Q = {
mi) )), ),...,, (xp, randomly.
Collect L independent posterior samples of the NN weights. repeat
for k= l........ do
Use the posterior samples to calculate and optimize (8).
(x", m=m)
argmax Obatch (x, m),
x 1
where 52 is the input domain.
(x,, m&) < (x", m²)
end for until G iterations are done or the increase of Obatch in Return Q
less than S
excellent performance. Moreover, the training of NNs is scalable, not suffering from ((N)) time complexity (N is the number of examples) of training exact GPs. GPs. (Springenbebrggg al. (2016) further used scale adaption to develop a robust stochastic gradient HMC for the posterior inference in the NN based BO. Recent works that deal with discrete inputs (Baptista and Poloczek 2018 or mixed discrete and continuous inputs (Daxberger et al. 2019) use an explicit nonlinear feature mapping and Bayesian linear regression, which can be viewed as one-layer Bayesian NNs.
There have been many studies in fulti-fiielityy (MF) BO, e.g., (Huang et al., 2006 Swersky et al. 2013 Lam al.) 2013; Picheny et al. 2013 Kandasamy al. 2016]] 2017a] Poloczek al. 2017 McLeod et al. 2017 Wu and Frazier 2017). While successful, these methods either ignore or oversimplify the strong, complex correlations between different fidelities, and hence might be
inefficient in surrogate learning. For example, Picheny et al. )): Lam et al. (2013); Kandasamy al. (2016):Poloczee et al. (2017) learned an independent GP for each fidelity; [Song et (2019)
used all the examples without discrimination to train one single GP: Huang et al. (2006); Takeno al. (2019) imposed a linear correlation across fidelities, while Zhang al. (2017) constructed convolutional kernel as the cross-fidelity covariance and so the involved kernels in the convolution must be simple and smooth enough (yet less expressive) to obtain a closed form. Recently, Perrone et al. (2018) developed an NN-based multi-task BO method for hyper- parameter transfer learning. Their model constructs an NN feature mapping shared by all the tasks, and uses an independent linear combination of the mapped features to predict each task output. While we can consider each task as evaluating the objective at particular fidelity, the model does not explicitly capture and exploit the correlations across different tasks given the shared (latent) features, the predictions of these tasks (fidelitics) are independent. The most recent work (Li et al. 2020 also developed an NN-based multi-fidelity BO method, which differs from work in that (1) their model only estimates the relationship between successive fidelities, and hence has less capacity, (2) their work uses a recursive one-dimensional quadrature to calculate the acquisition function, and is difficult to extend to batch acquisitions. In a high level, the chain structure of Li et al. (020)) 's model also resembles deep GP based multi-fidelin models (Perdikaris al. 2017 Cutajar et al. 2019).
Quite few batch BO algorithms have been developed, such as (González et al., 2016 Wu and Frazier 2016: Hernández-- Lobato et al. 2017 Kandasamy et al. (017b). However, they work with single-fidelity queries and are not easily extended to multi-fidelity optimization tasks. Takeno et al. (2019) proposed two batch querying strategies for their MF-BO framework. Both strategies

---

leverage the property that the covariance of a conditional Gaussian does not rely on the values of the conditioned variables; so, there no need to worry about conditioning on function values that are still in query. The asynchronous version generates new queries conditioned on different sets of function values in query (asynchronously)) However, if the conditional parts are significantly overlapping. which might not be uncommon in practice, there is risk of generating redundant or even collapsed queries. Takeno et al. (2019) also talked about a synchronous version. While they discussed how to compute the information gain between the function maximum and batch of function values, they did not provide an effective way to optimize it with the multi-fidelity querying costs. Instead, they suggested simple heuristics to sequentially find each query by conditioning on the generated ones. However, there is no guarantee about this heuristics.
While in our experiments, we mainly use hyperparameter optimization to evaluate our multi-fidelitg BO approach, there are many other excellent works specifically designed for hyperparameter tuning or selection, e.g., the non-Bayesian, random search based method Hyberband (I.i et al.. (017) which also reflects the multi-fidelity idea: it starts using few training iterations/epochs (low fidelity) to evaluate many candidates, rank them, iteratively selects the top-ranked ones, and further evaluate them with more iterations/epochs (high fidelity). BOHB (Falkner et al. 2018) is a hybrid of KDE asynchronous successive halving algorithm for parallel random search over hyperparameters. Domhan et al. (2013); Klein et al. (2017b) propose to estimate the learning curves, and early halt the evaluation of ominous hyperparameters according to the learning curve predictions. [Swersky et al. (2014) introduced a kernel about the training steps, and developed Freeze-thaw BO (Swersky al.] 2014) that can temporarily pause the model training and explore several promising hyperparameter settings for a while and then continue on to the most promising one. The work in (Klein et al., (21 jointly estimates the cost as a function of the data size and training steps, which can be viewed as continuous fidelities, like in (Kandasamy al. 2017a Wu and Frazier 2017).
based BO (Bergstra et al., ( and Hyperband. Li et al. (2018) further developed
Experiment
6.1 Surrogate Learning Performance
We first examined if BMBO-DARN can Iearn a more accurate surrogate of the objective. We used two popular benchmark functions: (1) Levy (Laguna and Marti 2005) with two-fidelity evaluations, and (2) Branin (Forrester al. 2008 Perdikaris et al. 2017) with three-fidelity evaluations. Throughout different fidelities are non inniinaarnnntatonnaary transforms. We provide the details in the Appendix. Methods. We compared with the following multi-fidelity learning models used in the state- of-the-art BO methods: (1) ME-GP-UCB (Kan- dasamy et al. 2016) that learns an independent
Levy
nRMSE
MNLL
NE--GP-UBB
0.831 + 0.195
1.824 + 0.276
MF-MES
0.581 + 0.032
1.401 0.031
GP for each fidelity. (2) MF-MES (Takeno et al.,
SHTL.
0.443 + 0.009
1.208 + 0.026
2019) that uses multi-output GP with linear correlation structure across differrnnt outputs (fi- delities), (3) Scalable Hyperparameter Transfer Learning (SHTL) (Perrone et al.) 2018) that uses an NN to generate latent bases shared by all the tasks (fidelities) and predicts the output of each task with linear combination of the bases. (4) Deep Neural Network Multi-Fidelity BO (DNN- MFBO) (Li et al. 2020) that uses chain of NNs to model each fidelity, but only estimates the relationship between successive fidelities.
DNN-MFBO
0.365 + 0.035
1.081 + 0.011
BMBO-DARN
0.348 + 0.021
1.072 + 0.016
Branin
MF-GP-UCB
0.846 + 0.147
1.976 + 0.208
MF-MES
0.719 + 0.099
1.796 + 0.128
SHITL
0.835 1 0.218
1.958 1 0.646
DNN-MFBO
0.182 1 0.022
0.973 0.013
BMBO-DARN
0.158 1 0.016
0.965 1 0.005
Table 1: Surrogate learning performance Branin function with three-fidelln training examples and Levy function with two-fidelity examples: normalized root- mean mean-square- square-error (nRMSE) and mean sean--eeeaie-loggooooe likelihood (MNI..). The results were averaged over five runs.
Settings. We implemented our model with PyTorch (Paszke et al. (019) and HMC sam- pling based on the Hamiltorch library (Cobb and Jalaian| 2021) (https:/ /tght/////tuub..../// com/ AdamCobb/ /hanhibbh//aaia. hami ttrrch). For each fidelity, we used two hidden layers with 40 neurons and tanh activation. We ran HMC for 5K steps to reach burn in (by looking at the trace plots) and then produced 200 posterior samples with every 10 steps. To generate each sample proposal, we ran 10 leapfrog steps, and the step size was chosen as 0.012.
7

---

We implemented DNN-MFBO and SHTL with PyTorch as well. For DNN-MPBO, we used the same NN architecture as in AMBBO-DARN for each fidelity, and ran HMC with the same setting for model estimation. For SHTI,, we used two hidden layers with 40 neurons and an output layer with 32 neurons to generate the shared bases. We used ADAM (Kingma and Bal 2014) estimate the model parameters, and the learning rate was chosen from (10-1, 5 x 10- 10-5, 5 x 10-3, 10- We ran IK epochs, which are enough for convergence. Note that we also attempted to use L-BFGS to train SHTL, but it often runs into numerical issues. ADAM is far more stable. We used Python implementation of MF-MES and ME-GP-UCB, both of which use the RBF kernel (consistent with the original papers).
Results. We randomly generated {130, 65) examples for Levy function at the two increasing fidelities, and {320, 130, } examples for Branin function at its three increasing fidelities. After training, we examined the prediction accuracy of all the models with 100 test samples uniformly sampled from the input space. We calculated the normalized root-mean-square-errn regative-ogg-ikellhood likelihood (MNLL.. We repeated the experiment for 5 times, and report their average and standard deviations in Table. D As we can see, for both benchmark functions, BMBO-DARN outperforms all the competing models, confirming the advantage of our deep auto-regressive model in surrogate learning. Note that despite using a similar chain structure, DNN-MFBO is still inferior to MMBO-DARN, implying that our fully auto-regressive modeling (see ( can better estimate the relationships between the fidelities to facilitate surrogate estimation.
(nRMSE) and mean-
6.2 Real-World Applications
Next, we used AMBO-DARD to optimize the hyperparameters of four popular machine learning models: Convolutional Neural Networks (CNN) (Fukushima and Miyake| 1982] LeCun al., 1990) for image classification, Online Latent Dirichlet Allocation (LDA) (Hoffman et al. 2010) for text mining, XGBoost (Chen and Guestrin 2016) for diabetes diagnosis, and Physics- Informed Neural Networks (PINN) (Raissi et al. 2019) for solving partial differential equations (PDE).
Methods and Setting. We compared with the state-of-the-art multi-fidelity BO algorithms men- tioned in Sec. 6.1] (1) MF-GP-UCB, (2) MF-MES, (3) SHTL, and (4) DNN-MFBO. In ad- dition, we compared with (5) MF-MES-Bater (Takeno et al. 2019). the (asynchronous) paral- lel version of MF-MES, (6) SF-Batch (Kandasamy et al. 2017b) (https: /tt::///gtthb com/ kirthevasank/ http:/pppppaaeeeeeaeeaaea---eeep ts). a single-fdellity GP-based BO that optimizes posterior samples of the objective function to obtain batch of queries. (7) SMAC3 (ht / com/ com/auutomm/SMMCC) BO based on random forests, (8) Hyperband (Li et al. 2017) (https) //github. com/ aut om Http///aaanttteeeeeeeeeee er) that conducts multi muuti-fidelity random search over the hy- perparameters, (9) BOHB (Falknrr et al. 2018) that uses Tree Parzen Estimator (TPE) (Bergstra et al.) 2011) to generate hyperparameter candidates in Hyperband iterations. We also tested our method that queries at one input and fidelity each time (B- = 1), which we denote by BMBO-DARN-I. We used the same setting as in Sec. 6.1 for all the multi multi-fidelity methods, except that for SHTL, we ran 2K epochs in surrogate training to ensure the convergence. For our method, we set the maximum number of iterations in optimizing the batch acquisition function (see Algorithm 88 to 100 and tolerance level to 10-3 For the remaining methods, e.g., SMAC3 and Hyperband, we used their original implementations and default settings. For all the batch querying methods, we set the batch size to 5. All the single fidelity methods queried at the highest fidelity.
Convolutional Neural Network (CNN). Our first application is to train CNN for image classifica- tion. We used CIFAR-10 10 dataset (https: //t:///wwwa www. cs toronto. wed//krriz from which we used 10K images for training and another 10K for evaluation. To optimize the hyperparameters, we considered thrce fidelities, i.e., training with 1, 10, 50 epochs. We used the average negative log-loss (nLL) to evaluate the prediction accuracy of each method. We considered optimizing the following hyperparameters:
far. html),
convolutional layers ranging from [1,4], # channels in the first filter (18, 136]), depth of the dense layers ([1, 8)), width of the dense layers ([32, 2080J), pooling type (["max", " "average"]), "),, and dropout rate ([10-3 3 0.99I). We optimized the dropout rate in the log domain, and used continuous relaxation of the discrete parameters.
Initially, we queried at 10 random hyperparameter settings at each fidelity. All the methods started with these evaluation results and repeatedly identified new hyperparameters. We used the average running time at each training fidelity as the cost: X : X2 Xa 1 : 10 50. After each query, we evaluated the performance of the new hyperparameters the highest
8

---

BMBO-DARE
MF-GP-UCB
1MBO--AAN-1
MF-MES
HyperBand BOHB
DNN-MFBO
MF-MES-Batch SF-Batch
SHTL
Negative Log Loss
2.0
Perplexity
1200
1.5
1000
800
600
1000
2000 3000 4000 5000 6000
7000
8000
200
400
600
800
1000
1200
Accumulated Cost (Time in seconds)
Accumulated Cost (Time in seconds)
(a) CNN
(h) Online I.DA
Log nRMSE
0.1
Log ARMSE
-0.2
-0.3
-0.4
30 Accumulated Cost (Time seconds)
10 15
25
1000 2000
3000 4000 5000 6000 7000 8000
Accumulated Cost (Time seconds)
(c) XGBoost
(d) PINN
Figure 2: Performance vs. accumulated cost (running time) in Hyperparameter optimization tasks. For fairness, all the batch methods queried new examples sequentially, iee, no parallel querying was employed. The results were averaged over five runs. Note that MF-GP-UCB, MF-MES and MF-MES-Bate often obtained very close results and their
overlap much.
training level. We ran each method until 100 queries were issued. We repeated the exper- iment for times and in Fig. 2h report the average accuracy (nLL) and its standard devi- ation for the hyperparameters found by each method throughout the optimization procedure. Online Latent Dirichlet Allocation (LDA).
Our second task is train online LDA (Hoffman et al. 2010) to extract topics from 20Nups (http://qwann /top:////woe... com/ com/~jasoni
1750
DUED DARN DAAN-1
1750
21503
corpus
20Newsgroups//
We used 5K documents
HIS
1000
for training, and 2K for evaluation. We used the implement from the scikit-learn library (https
//scikit-lears: org/ sttble/bb. considered optimizing the following hyperparam-
We
(a) CNN
(b) Online LDA
eters: document topic prior E 110 1].
topic word prior 1) E [10 110-311, 11. learn- Figure 3: Average time to generate queries (including
surrogate training).
ing decay R
[0.51, 1), learning offset
To E [1, 10, 20, 50, 100, 200), E-step stopping tolerance E [10-5,10-1], document batch size (2,4, 8, 16, 32, 64, 128, 256), and topic number K E [1, 64]. We optimized (t, 7,, k and e the log domain, and used a continuous relaxation of the discrete parameters. We considered three fidelities training with 1, 10 and 50 epochs, and randomly queried 10 examples at each fidelity to start each method. We evaluated the performance of the selected hyperparameters in terms of perplexity (the smaller, the better). In Fig. b,, we reported the average perplexity (and its standard deviation) of each method after five runs of the hyperparameter optimization.
XGBoost. Third, we trained an XGBoost model (Chen and Guestrin, 2016 predict a quantitative measure of the diabetes progression (https: htt/aarhiivi ve. ics uci edu/ m./ /dattat tasets/ diabetes). The dataset includes 442 examples. We used two-thirds for training and the remaining one-third for evaluation. We used the implementation from the scikit-learn library. We optimized the following hyperparameters: Huber loss parameter a E (0.01, 0.1). the non-negative complexity pruning parameter (00.01, 100)), fraction of samples used to fit individual base learners (00.1, 1)).

---

fraction of features considered to split the tree ((0.01,1), 1),, splitting criterion (["MAE", "MSE"D. minimum number of samples required to split an internal node (2,,)), and the maximum depth of individual trees (1, 16). The hyperparameter space is 12 dimensional. We considered three fidelities - training XGBoost with 2, 10 and 100 weak learners (trees). The querying cost is therefore A X²= X 1 5 50. We started with 10 random queries at each fidelity. We used the log of nRMSE to evaluate the performance. We ran 5 times and report the average log-nRMSE the identified hyperparameters by each method in Fig. 2
Physics-informed Neural Networks (PINN). Our fourth application to learn PINN to solve PDEs (Raissi et al. 2019). The key idea of PINN is to use boundary points to construct the training loss, and meanwhile use set of collocation points in the domain to regularize the NN solver to respect the PDE. With appropriate choices of hyperparameters, PINNs can obtain very accurate solutions. We used PINNS to solve Burger's equation (Morton and Mayers 2005) with the viscosity 0.01/z. The solution becomes sharper with bigger time variables (see the Appendix) and hence the learning is quite challenging. We followed (Raissi et al., 2019) to use fully connected networks and L-BFGS for training. The hyperparameters include NN depth (1,,)), width ([L, 64), and activations (8 choices: Relu, tanh, s gimoid. id, their variants, etc.). Following (Raissi et al. 2019), we used 100 boundary points as the training set and 10K collocation points for regularization. We used 10K points for evaluation. We chose 3 training fidelities, running L-BFGS with 10, 100, 50K maximum iterations. The querying cost (average training time) is \ d2 A = 1 10 50. Note that in fidelity 3, L-BFGS usually converged before running 50K iterations. We initially issued 10 random queries at each fidelity. We ran each method for 5 times and reported the average log nRMSE after each step in Fig. 2
Results. As we can see, in all the applications, BMBO-DARN used the smallest cost (i.e, running time) to find the hyperparameters that gives the best learning performance. In general, BMBO-DARN identified better hyperparameters with the same cost, or equally good hyperparameters with the smallest cost. BMBO-DARN-I outperformed all the one-by-one querying methods, except that for online LDA (Fig. 2b) and PINN (Fig. D). it was worse than DNN-MFBO and Hyperband at the early stage, but finally obtained better learning performance. We observed that the GP based baselines (MF-MES, MF-GP-UCB, SF-Batch, etc.) are often easier to be stuck in suboptimal hyperparameters, this might because these models are not effective enough to integrate information of multiple fidelities to obtain a good surrogate. Together these results have shown the advantage of our method, especially in our batch querying strategy. Finally, we show the average query generation time of BMBO-DARN for CNN and Online LDA in Fig- 3 (including surrogate training). It turns out BMBO-DARN spends much less time than MF-MES using the global optimization method DIRECT (Jones et al. 1998), and comparable to ME-GP-UCD and DNN-MFBO. Therefore, BMBO-DARN is efficient to update the surrogate model and generate new queries.
Conclusion
We have presented BMBO-DARN, a batch multi-fidelity Bayesian optimization method. Our deep auto-regressive model can serve better surrogate of the black-box objective. Our batch query- ing method not only is efficient, avoiding combinatorial search over discrete fidelities, but also significantly reduces the cost while improving the optimization performance.
Acknowledgments
This work has been supported by MURI AFOSR grant
References
Auer, P. (2002). Using confidence bounds for exploitation-exploraiin trade-offs. Journal of Machine Learning Research, 3994
Baptista, R. and Poloczek, M. (2018). Bayesian optimization of combinatorial structures. In International Conference on Machine Learning, pages 462-471.
Bergstra, J., Bardenet, R., Bengio, Y., and Kégl, B. (2011). Algorithms for hyper-parameter op- timization. In 25th annual conference on neural information processing systems (NIPS 2011).
10

---

volume 24. Neural Information Processing Systems Foundation.
Chen, T., Fox, E., and Guestrin, C. (2014). Stochastic gradient Hamiltonian Monte Carlo. In International conference on machine learning, pages 1683-1691. PMLR.
Chen, and Guestrin, C. (2016). Xgboost: scalable tree boosting system. Proceedings of the 22nd aem SigKDD international conference on knowledge discovery and data mining, pages 785-794.
Cobb, A. D. and Jalaian, B. (2021). Scaling Hamiltonian Monte Carlo inference for Bayesian neural networks with symmetric splitting, Uncertainty in Artificial Intelligence.
Cutajar, K., Pullin, M., Damianou, A., Lawrence, N., and González, J. (2019). Deep gaussian processes for multi-fidelity modeling. arXiv preprint arXiv:1903.07322
Daxberger, E., Makarova, A., Turchetta, M., and Krause, A. (2019). Mixed-variable Bayesian optimization. arXiv preprint arXiv: 1907.0139.
Domhan, T., Springenberg. J. T., and Hutter, F. (2015). Speeding up automatic hyperparameter optimization of deep neural networks by extrapolation of learning curves. In Twenty-fourth international joint conference on artificial intelligence.
Falkner, S., Klein, A., and Hutter, F. (2018). BOHB: Robust and efficient hyperparameter optimization at scale. In International Conference on Machine Learning, pages 1437-1446. PMLR.
Forrester, A., Sobester, A., and Keane, A. (2008). Engineering design via surrogate modelling: a practical guide John Wiley & Sons.
Fukushima, K. and Miyake, S. (1982). Neocognitron: self-organizing neural network model for mechanism of visual pattern recognition. In Competition and cooperation in neural nets, pages 267-285. Springer.
Garrido-Merchán, C. and Hermández-Lobato (2020). Dealing with categorical and integer- valued variables in Bayesian optimization with gaussian processes. Neurocomputing. 380:20-35.
González, J., Dai, Z., Hennig, P., and Lawrence, N. (2016). Batch Bayesian optimization via local penalization. Artificial intelligence and statistics, pages 648-657, PMLR.
Hennig, P. and Schuler, C. J. (2012). Entropy search for information--effcciet global optimization. Journal of Machine Learning Research,
Hermández-Lobato, J. M., Hoffman, M. W., and Ghahramani, Z. (2014). Predictive entropy search for efficient global optimization of black-box functions. In Advances in neural information processing systems, pages 918-9 1--926.
Hernández-Lohate J. M., Requeima, J., Kyzer-Knapp, E. O.,, and Aspuru-Guzik, A. (2017). Parallel and distributed Thompson sampling for large-scale accelerated exploration of chemical space. In International conference machine learning, pages 1470-14799 PMLR.
Hoffman, M., Bach, F. R., and Blei, D. M. (2010). Online learning for latent dirichlet allocation. In advances in neural information processing systems, pages 856-864. Citeseer.
Huang, D., Allen, T. T., Notz, W. I., and Miller, R. A. (2006). Sequential kriging optimization using multiple-fided evaluations. Structural and Multidisciplinary Optimization, 32(5):369-335
Jones, D. R., Schonlau, M., and Welch, W. (1998). Efficient global optimization of expensive box functions. Journal of Global optimization, (3(4):455-499
Kandasamy, K., Dasarathy, G., Oliva, J. B., Schneider, J., and Póczos, B. (2016). Gaussian process bandit optimisation with multi-iideliyy evaluations. In Advances in Neural Information Processing Systems, pages 992-1000.
Kandasamy, K. Dasarathy, G., Schneider, J., and Póczos, B. (2017a). Multi-fidelity Bayesian optimisation with continuous approximations. Proceedings the 34th International Conference on Machine Learning- Volume 70, pages 1799-1800 JMLR. org.

---

Kandasamy, K.. Krishnamurthy, A., Schneider, J., and Poczos, B. (2017b). Asynchronous parallel Bayesian optimisation via Thompson sampling. arXiv preprint arXiv:1705.09223.
Kingma, D. P. and Ba, J. (2014), Adam: A method for stochastic optimization. arXiv preprint 112.v:1412.6980.
Klein, A., Falkner, S., Bartels, S., Hennig, P., and Hutter, F. (2017a). Fast Bayesian optimization of
machine learning hyperparameters large datasets. In Artificial Intelligence and Statistics, pages PMLR.
Klein, A., Falkner, S., Springenberg, J. T., and Hutter, F. (2017b). Learning curve prediction with Bayesian neural networks. 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net.
Laguna, M. and Martí, (2005). Experimental testing of advanced scatter search designs for global optimization of multimodal functions. Journal of Global Optimization, 33(2):235-25) 255..
Lam, R., Allaire D. L., and Willcox, K. E. (2015). Multifidelity optimization using statistical surrogate modeling for non-hierarchical information sources. In 56th AIAN/ACC//ASS/SSC Structures, Structural Dynamics, and Materials Conference, page 0143.
LeCun, Y., Boser, B. E., Denker, J. S., Henderson, D., Howard, R. E., Hubbard, W. E., and Jackel, L. D. (1990). Handwritten digit recognition with a back-propagation network. In Advances in neural information processing systems, pages
Li, L., Jamieson, K., DeSalvo, G., Rostamizadeh, A., and Talwalkar, A. (2017). Hyperband: A novel bandit-based approach to hyperparameter optimization. The Journal of Machine Learning Research, 18(1):676566816
Li, L., Jamieson, K., Rostamizadeh, A., Gonina, E., Hardt, M., Recht, B., and Talwalkar, A. (2018). Massively parallel hyperparameter tuning. arXiv preprint arXiv: 180.05934.
Li, S., Xing, W., Kirby, R., and Zhe, S. (2020). Multi-fidelity Bayesian optimization via deep neural networks. In Advances Neural Information Processing Systems.
Liu, D. C. and Nocedal, J. (1989). On the limited memory BFGS method for large scale optimization. Mathematical programming, 45(1-3):503-522
McLeod, M., Osborne, M. A., and Roberts, S. J. (2017). Practical Bayesian optimization for variable cost objectives. arXiv preprint arXiv: 1703:0303..
Mockus, J. (2012). Bayesian approach to global optimization: theory and applications, volume 37. Springer Science & Business Media.
Mockus, J., Tiesis, V., and Zilinskas, A. (1978). The application of Bayesian methods for seeking the extremum. Towards global optimization, 2(117-129):2.
Morton, K. W. and Mayers, D. (2005). Numerical solution of partial differential equations: an introduction. Cambridge university press.
Neal, R. M. et al. (2011). Meme using Hamiltonian dynamics. Handbook of markov chain monte carlo, 2(11):2.
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang. E., Vito Z.,, Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S. (2019). Pytorch: An imperative style, high- performance deep learning library. Wallach, H., Larochelle, H., Beygelzimer, A., d'Alché-Bur Buc. F., Fox, E., and Garnett, R., editors, Advances in Neural Information Processing Systems 32, pages 8024-8035. Curran Associates, Inc.
Perdikaris, P., Raissi, M., Damianou, A., Lawrence, N., and Karniadakis, G. E. (2017). Nonlinear information fusion algorithms for data-efficient multi-fidelity modelling. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, 473(21989::016075))
12

---

Perrone, V., Jenatton, R., Seeger, M. W., and Archambeau, C. (2018). Scalable hyperparameter transfer learning. In Advances in Neural Information Processing Systems, pages 6845-6855.
Picheny, V., Ginsbourger, D., Richet, Y., and Caplin, G. (2013). Quantile-based optimization noisy computer experiments with tunable precision. Technometrics, 55(1):2-13.
Poloczek, M., Wang. J., and Frazier, P. (2017). Multi-information source optimization. In Advances in Neural Information Processing Systems, pages
Raissi, M., Perdikaris, P., and Karniadakis, G. E. (2019). Physics-infornen neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378:686-707.
Rasmussen, C. E. and Williams, C. K. I. (2006). Gaussian Processes for Machine Learning, MIT Press.
Snoek, J., Larochelle, H., and Adams, R. P. (2012). Practical Bayesian optimization of machine learning algorithms. In Advances in neural information processing systems, pages 2951-2959.
Snoek, J., Rippel, O., Swersky, K., Kiros, R., Satish, N., Sundaram, N., Patwary, M., Prabhat, M., and Adams, R. (2015). Scalable Bayesian optimization using deep neural networks. In International conference on machine learning, pages 2171-2180.
Song, J., Chen, Y., and Yue, Y. (2019). general framework for multi-fidelity Bayesian optimization with Gaussian processes. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 3158-3167.
Springenberg. J. T., Klein, A., Falkner, S., and Hutter, (2016). Bayesian optimization with robust Bayesian neural networks. In Advances in neural information processing systems, volume 29, pages 4134-4142.
Srinivas, N., Krause, A., Kakade, S., and Seeger, M. (2010). Gaussian process optimization in the bandit setting: regret and experimental design. In Proceedings of the 27th International Conference on International Conference on Machine Learning, pages 1015-1022.
Swersky, K., Snoek, J., and Adams, R. P. (2013). Multi-task Bayesian optimization. In Advances in neural information processing systems, pages 2004-2012.
Swersky, K. Snoek. J., and Adams, R. P. (2014). Freeze-thaw Bayesian optimization. arXiv preprint arXiv:1406.3890
Takeno, S., Fukuoka, H., Tsukada, Y., Koyama, T., Shiga, M., Takeuchi, I,, and Karasuyama, M. (2019). Multi-fidelity Bayesian optimization with max-value entropy search. arXiv preprint 1rXiv:1901.0827.
Wang, Z. and Jegelka, S. (2017). Max-value entropy search for efficient Bayesian optimization. In Proceedings the 34th International Conference on Machine Learning- Volume 70, pages 3627- 3635. JMLR. org.
Wu, J. and Frazier, (2016). The parallel knowledge gradient method for batch Bayesian optimization. Advances in Neural Information Processing Systems, 29:3126-3134
Wu, J. and Frazier, P. I. (2017).
sotinuouus-fielliy Bayesian optimization with knowledge gradient. In NIPS Workshop on Bayesian Optimization.
Zhang, Y., Hoang, T. N., Low, B. K. H., and Kankanhalli, M. (2017), Information-basee multi- multi-fidelity Bayesian optimization. In NIPS Workshop Bayesian Optimization.
13


Reviews:
[{"review_id": "yu0JbEUDdGe", "reviewer": "Reviewer_Hcby", "summary": "In multi-fidelity black-box optimization, f() can be queried at points x at different levels of precision. Higher levels of precision have higher costs. Prior work on Bayesopt for mfbbo used GP models for multi-fidelity observations that make simplistic assumptions about correlations across fidelities. In response, the authors propose instead using a flexible autoregressive model for modeling the sequence of fidelity scores for a given x. Bayesian inference on the model is performed using HMC.\n\nThe paper also introduces a batched form of max-value entropy search and an algorithm for optimizing this complex acquisition function that works well in practice.\n\nThe experimental results are impressive. The proposed method outperforms a large number of popular methods from the literature.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "=originality=\nThe paper provides multiple novel methods (surrogate model, batch acquisition function, etc) that will be of broader interest to the Bayesopt community.\n\n=clarity=\nOverall, good. See comment below about the problem statement, though.\n\n=significance=\nMulti-fidelity black-box optimization has applications in a number of important domains, including neural network hyper-parameter tuning. This paper benchmarks against a variety of recent algorithms from the literature and demonstrates an impressive performance improvement. \n\n=comments=\nThe paper would be improved by a more formal problem statement. Lines 92-106 are largely focused on a discussion of related work. It's unclear to me, for example, why eq (2) is necessary. It would be helpful to better explain the action space for the algorithm. For example, do fidelities have to be queried in increasing order of cost? If you query f() at high fidelity, does that mean that you get a low-fidelity measurement for free? This is the case for neural network training (where high fidelity corresponds to more training steps), but not true in general.\n\n I'm assuming that for each x you can evaluate f at any given fidelity. However, the parametrization of the autoregressive model in (3) suggests that predicting at a given fidelity requires having data for every available measurement of x at lower fidelity. This places a considerable restriction at the action space that isn't reflected in a problem definition or in sec 4.3. There are ways to parametrize an autoregressive model such that it can handle missing data, but this is not discussed.\n\nAlso, it was unclear to me why you need to have a separate model for each fidelity in (4). Wouldn't there be a shared model that could model all of the fidelities, and thus would have more training data?\n\nI'm not that familiar with the latest literature on max-value entropy search. Can you comment on what is novel in your usage of it. It was unclear to me, for example, what parts of sections 4.1 and 4.2 are new.\n\nThe experiments section contains a thorough ablation analysis of the impact of various surrogate models. This helps provide evidence for the importance of one of the paper's contributions. However, the paper's other contributions (batch acquisition function, strategy for optimizing the AF) are not analyzed. I'm curious, for example, what would happen if you used a simpler acquisition function that doesn't account for interactions within the batch or used something other than MVES.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "no", "review_text": "", "extra_scores": {"ethical_concerns": "no", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "qUMdO1-llDJ", "reviewer": "Reviewer_gu5R", "summary": "Multi-fidelity methods are prevalent in problem settings where high-quality (but difficult to acquire) observations are enhanced with lower-quality (but more easily obtained) samples of a function. In this paper, the authors focus on the application of multi-fidelity modelling to hyper parameter tuning, whereby different model settings are verified with varying fidelity constraints (such as epochs in a training loop) in order to achieve the best accuracy-cost trade-off. The model introduced here is based on chained neural networks, whereby the input at each fidelity layer combines the input in the original domain for the function to be estimated, along with the outputs from the preceding fidelities. In order to carry out BO, the authors develop a batch acquisition technique that encourages diversity across the samples being drawn. This is in contrast to other techniques where new samples are drawn one a time, thus incurring the risk of there not being sufficient diversity across successive iterations. In order to achieve this goal, the authors also propose an alternating optimisation scheme for iteratively selecting samples across different fidelities.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths\n\n- Having a multi-fidelity model that can be trained end-to-end is an appealing property, as it allows for learning the interactions across fidelities in a unified manner rather than disjointed. The batch acquisition function also arises quite naturally from the formulation of the model, meaning that the authors do not have to rely on a multitude of empirical tricks and implementation hacks to meet the desired goal.\n\n- The performance of the method is evaluated over over benchmark functions (for assessing the quality of the multi fidelity model in isolation), while four hyper-parameter optimisation set-ups are used to validate the practical effectiveness of the BO proposal. This variety of experiments successfully addresses the different potential uses that the methods proposed in the paper might have. \n\n- The paper is very well-written and organised. The literature review adequately covers related work, and the experimental evaluation also includes a wide variety of competing methods, some of which were implemented by the authors themselves.\n\n\n### Weaknesses\n\n- The importance of having a multi-fidelity model that is either chained or trained end-to-end has already been explored by Perdikaris et al (2017) and Cutajar et al (2019) for deep GP model constructions. I think that discussing this connection could be beneficial to the paper as the superior uncertainty estimates that are typically expected of GP models should be relevant to the iterative BO procedure explored here.\n\n- It would be interesting to more explicitly state the potential limitations of the proposal here. For example, how well does the MF model scale to large input dimensions, either in terms of computational efficiency or expressivity? The paper successfully leverages weaknesses in other methods for motivating the work work described here, but there is little insight into which conditions might cause this method to underperform.\n\n- In the same vein of the above, the paper doesn’t really hint at how the method described here can be improved or extended further in future work.\n\n\n### Writing and Clarity\n\nThe paper is very well-written and a pleasure to read. The proposal is well-motivated and the multi-fidelity batch acquisition function is properly explained. One possible suggestion would be to include a visualisation of the multi-fidelity model itself (and how the output of the various layers feeds into the next) within the main paper itself.\n\nSome very minor typos:\n- *L2:* between *the* cost and accuracy\n- *L15:* reference to *alternating optimization* is a bit confusing here.\n- *L41:* between successive queries *and* is at higher\n- *L83:* most commonly used *class of surrogate models are Gaussian processes*\n- *L114* might limit the *expressivity*\n- *References:* Certain terms (e.g. Bayesian) and conference names need to appear as capitalised in the references.\n\n\n### Overall Recommendation\n\nBesides putting forward an interesting problem to solve, the authors propose a well-rounded solution and provide multiple experiments that showcase the diversity of settings this work can be applied to. Just as importantly, the paper and its writing are already in a state fit for publication, which gives me greater confidence in recommending it for acceptance.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "As highlighted in the main review, I think it would interesting for the authors to expose the limitations for this work more clearly. Having more insight into what could constitute future work would also be appreciated. There is no immediate societal impact to consider here.", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "15"}, "extra_text": {}}, {"review_id": "aFuInLqBeEx", "reviewer": "Reviewer_zV6Z", "summary": "The paper suggests a methodology for performing multi-fidelity Bayesian Optimziation (BO) using a particular (deep auto-regressive) Neural Network model as the surrogate. The authors formulate a batch acquisition strategy based on max-value entropy search, and propose an approximation using posterior sampling and cyclic optimization that renders the optimization of the acquisition function tractable. The method is evaluated on both synthetic and real experiments, where it shows improved performance over existing approaches.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n### High-level summary\n- The problem setting is relevant and of interest to the community.\n- The modeling approach seems quite an incremental contribution over previous work, and it is unclear what the effect of the model on the performance of the overall approach is.\n- There is some but not much methodological novelty in the approach; it essentially uses a standard batch MES formulation with a (slightly modified compared to previous work) BNN model and  some approximations of the acquisition function to render computations tractable. The main novelty is in these approximations; however, I am not convinced of the merits suggested approach.\n- The writing is generally ok (some wording issues, repetitive in some places), overall the paper is relatively easy to follow.\n- The empirical evaluation includes a good number of baselines, but is quite sparse on details and thus not clear; this section needs to be improved. In the current form, it's hard to assess the significance of the results. \n- Overall, a potentially interesting approach, but too many open questions in the current draft.\n- Not making the bar for NeurIPS in the current form.\n\n\n### Notes\n- The authors appear to suggest that all GP approaches have computational complexity of N^3: There area  number of of ways around this (e.g. linearge conjugate gradient solves instead of Cholesky-based approaches, interpolation / sparse GPs, variational GPs, etc.). These have some other challenges but they should at least be mentioned as ways to reduce computational complexity.\n- In all the examples considered in the paper, the different fidelities correspond to different \"lengths\" (epochs, weak learners, LBFGS-B steps). An alternative approach to pre-committing to the length of the evalution is to to not specify the fidelity of a candiate priori but perform early stopping while obtaining partial results. Such approaches are at least as good as pre-committing (since that is a special case of the early stopping setting). The paper should discuss the early stopping line of work; please also explain why you consider the pre-comitting approach.\n- Modeling:\n  - The modeling approach is not particularly well motivated and the contribution seems quite incremental compared to previous work.\n  - Specifically, the main difference to the model of Li et al. is that fidelities here depend on the output of all previously modeled fidelities. This doesn't really cause any technical challenges and is a reasonably setup, but it's not clear from the paper what the effect of this change is.\n  - In particular, there is no evaluation of the model fits, and a lack of an ablation study comparing to the simpler model from Li et al. means that it's not possible to separate the effect of the model and the effect of the different approach to optimizing the acquisition function (in the case of sequential optimziation).\n  - This point is particular pertinent since in all of the real-world examples considered, one would expect a relatively simple (and not complex) relationship between the fidelities. It's not clear that in any of the real world the examples the additional model capacity obtained by the modified structure would be all that helpful.\n  - The model is motivated with \"complex, strong (nonstationary, nighly nonlinear) correlations across fidelites\": Can you give some more conrete examples to provide some motiviation/intuition for where/in which practical applications this is / would be the case?\n  - Model fitting fo BNNs is known to be much more finnicky than other approaches. The authors state they fit models by \"looking at the trace plots\", which does not seem particularly robust / automatable (e.g. \"step size was chosen as 0.012\" - why and how? how hard was it to determine this value?)\n- Acquisition Function:\n  - Sec 4.1 is a bit light on discussing similarities and differnces to other batch acquisition functions. How does this approach relate to the asynchronous variant in Takeno et al.? It would also be useful to discuss the relation to the GIBBON batch MVES approximation developed in [1].\n  - There seems to be a bit of a tension between the modelinga approach using a BNN (much more flexible than existing GP-based formulations) and the efficient computation scheme that is based on moment moment matching.\n    - The approach is based on Laplace approximation, which is presumably a pretty coarse approximation. If that is true, what is the motivation for building a complicated BNN model in the first place? It seems that the cases in which the more expressive model helps would likely be those where the approximation used in the computation of the acquisition function will be poor.\n    - It's not clear how well the Laplace approximation works for this problem. It woudl be helpful to verify in simulations on real problems that this is in fact a reasonable thing to do.\n  - The coordinate descent approach optimization is reasonable but has obvious issues that should be discussed (e.g. this can get easily get stuck in local optima).\n- Experiments:\n    - The difference between BMBO-DARN and DNN-MFBO isn't particularly large (would be good to see p-values of a hypothesis test for the difference).\n    - The real-world application evaluation seems to be quite comprehensive, a few models and quite a few baselines. However, additional ablation studies are necessary to understand the impact of the modeling change and the impact of the acquisition function formulation (see above).\n    - There is lack of detail on the evaluation criteria. Specifically, it is not clear to me what exactly the accumulated cost is, and how it is computed in the batch setting (e.g. how is it defined if there are heterogeneous fidelities that take different times to evaluate?). Reading this section, I was left confused and concerned that there might be an apples-to-oranges comparison of non-batch vs. batch methods going on. IN particular, the fact that BMBO-DARN-1 is a varaint that \"queries at one input and fidelity each time (B = 1)\" performs worse thatn BMBO-DARN is surprising, since purely sequential optimization should perform much better than any batch variant if given the same fidelities to choose from. Essentially I have two ways to read this: (1) The cost function is not actually summed across the cost of the batches running in parallel (i.e. only overall wall time is counted, not total CPU time), or (2) the BMBO-DARN-1 only evalues at the highest fidelity. EIther version would be somewhat misleading (in the latter case an obvious baseline that would need to be added would be one that only allows low fidelity evaluations to make sure the improvements observed here don't just come from the fact that using lower fidelities is generally enough). Please clarify this and be explicit about the evaluation metrics and the benchmark procedure.\n    - Why don't the losses all start at the same value if the initial points are shared between methods?\n    - How many function evaluations are actually performed in the experiments?\n- Miscellaneous:\n    - Somewhat repetitive discussion of \"other approaches simplifying things\" (introduction, background, related work): should consolidate this somehow as the repeated discussion does't really add anything\n- Wording (selected):\n    - \"most commonly used surrogate is GP\"\n    - \"this might because these models are not effective\"\n\n\n\n### Questions\n- You mention \"the standard one by one querying strategy\": Can you elaborate on this? I don't think this is true, there has been a lot of work on batch and fully asynchronous candidate generation in the past (see e.g. ), e.g. via sequential conditioning on pending points.\n- \"running L-BFGS with 10, 100, 50K maximum iterations\": Is this correct? this seems like an obscene difference in fidelities. It would also seem odd with the average training time. Or is the idea that 50K basically just means \"LBFGS until convergence\"?\n- Timings in Figure 2:\n    - Is this the average time per candidate or the average time per batch?\n    - How much time is spent on model fitting, how much on acqf optimization?\n- What MF-MES implementation is used? The paper just states \"We used a Python implementation of MF-MES and MF-GP-UCB\". It seems from Figure 2 that whatever was used is extremely slow; it seems quite possible to substantially reduce this time (and likely also optimization performance) with a proper implementation.\n- Is there an assumption that the fidelities are ordered? If so, what if that's not the case? Will things still work ok? It seems like the auto-regressive model might not be the best approach in that case.\n  - How could this approach work with continuous fidelities? These are common e.g. in HPO for ML training;  incorporating some prior information about the structure information should help with the modeling.\n\n\n### References\n[1] Moss, M., et al., GIBBON: General-purpose Information-Based Bayesian Optimisation. arXiv:2102.03324, 2020\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "-There is more room for discussing limitations; see main review for details. \n- No concerns regarding social impact. \n\n", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5"}, "extra_text": {}}, {"review_id": "YwSjAqXW6e", "reviewer": "Reviewer_JVN7", "summary": "The paper introduces a series of Bayesian neural networks that predict the performance across different, discrete fidelities. The models for higher fidelities use the prediction on lower ones as inputs, which makes this model auto-regressive. the authors use this flexible model and combine it with a new acquisition function that aims to minimize the mutual information of a batch of potential query points and the current belief over the optimum. By designing a heuristic to optimize the acquisition function, the authors are able to apply their model to the problem of Batch BO.\n\nIn the experiments, the proposed method is compared to other state-of-the-art methods for regular BO, batched BO and multifidelity BO.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": " I think the paper contains a very interesting method for modeling multiple fidelities/tasks with a cascade of Bayesian neural networks. Based on the presented results. the method seems very sample efficient, but the overhead is unclear. To my surprise, the sequential version of the algorithm performs worse than the batch one, which I would like to hear the authors' feedback on.\nMissing related work on batch BO and missing details on the parallelism of the presented experiments, also lower my impression of the paper a bit.\nI think that the paper contains two orthogonal ideas: the DARN model itself, and a clever way to approximate the mutual information for multiple points. Because the effects of those are not separated by an ablation study, the effects of contributions both make are not separable.\n\nThe paper is easy to follow, although I would suggest to move some of the details of the benchmarks to the appendix and expand the discussion about the acquisition function (which I found too dense) and a more detailed analysis of the results.\n\n## Specific remarks/questions:\n\n- In the discussion of equation (2), I am missing the OG multi-task BO reference\n\n  Swersky, K., Snoek, J., & Adams, R. P. (2013). Multi-task bayesian optimization.\n\n  equation 2 is one particular way of approximating the MTBO kernel, which is more general.\n- The discussion about the limitations of multi-fidelity BO is missing MTBO and potentially stacked GPs from: \n\n  Golovin, D., Solnik, B., Moitra, S., Kochanski, G., Karro, J., & Sculley, D. (2017, August). Google vizier: A service for black-box optimization. In Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining (pp. 1487-1495)\n\n  which shows as a different way of approximating the full covariance across tasks.\n- About equation (5): In the normal context of batch BO, the whole batch is suggested at once, then evaluated and the results incorporated into the model. If this pre-batch query and model-update is followed here, wouldn't $max \\lambda_{m_k}$ be more meaningful if the evaluations are in parallel? Otherwise querying a batch wouldn't give any advantage.\n- The related work section is missing a discussion about batch BO methods, including (but not limited to):\n\n  Wu, J., & Frazier, P. (2016). The parallel knowledge gradient method for batch Bayesian optimization. Advances in Neural Information Processing Systems, 29, 3126-3134.\n\n  González, J., Dai, Z., Hennig, P., & Lawrence, N. (2016, May). Batch Bayesian optimization via local penalization. In Artificial intelligence and statistics (pp. 648-657). PMLR.\n\n- The method labeld as SHTL, is referred to as ABLR (adaptive Bayesian linear regerssion) in the paper itself. The method also uses L-BFGS to optimize all parameters, which was replaced by ADAM here. Not sure how much that impacts performance, but it should be pointed out in the paper.\n\n- It is nowhere mentioned if evaluations where carried out in parallel. If they were, it should be stated how many parallel evaluations where performed. If they were all done sequentially, I would like to understand why the BMBO-DARN-1 performs worse. Usually, batch methods are outperformed by their sequential counterpart when the batch is evaluated sequentially, but become useful when the evaluations can be performed in parallel.\n\n- BMBO-DARN is very performant across all shown benchmarks, but I wonder how much each contribution has, i.e. how does the auto regressive model compare against a simpler model, but with the same acquisition function, and how would the DARN model work with MF-MES.\n\n- The performance is shown versus the  accumulated cost, so I assume the overhead of the HMC sampling and the elaborate acquisition function is not included. Especially for the XGBoost example, I would assume the overhead to be substantial. What would these plots look like if this overhead is included?\n\n- Following up on the previous point, could you please comment on the average evaluation time for the benchmarks at the different fidelities? It is really hard to judge how many configurations have been evaluated across the different benchmarks.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors failed to address any limitations of the method. A discussion about the choices of the fidelities, and what happens if they are poorly chosen is missing. A discussion about the overhead of the model would also be appreciated.", "review_text": "", "extra_scores": {"ethical_concerns": "No.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Ultimately, I am recommending this paper for acceptance because I believe the approach to multi-fidelity optimization proposed by the authors is sufficiently novel and the experimental evaluation is sufficiently strong, and I am thus in agreement with Reviewer gu5R and Reviewer Hcby that, after substantial incorporation of feedback from the reviewers, the paper is solid. Furthermore, I believe that you do indeed already include several of the ablation studies asked for (e.g., BMBO-DARN-1 vs MF-MES is equivalent to an ablation of how much the DARN matters). While I believe the paper has flaws (see below), I believe most of these flaws are in argumentation rather than in execution, and would not require changes significant enough to warrant rejection.

With that said, I want to be clear that several of the criticisms raised by Reviewer zV6Z and Reviewer JVN7 seem valid to me, and for at least a few of them, I found the author response to be unconvincing.

First, the claims made on lines 143-149 and repeated in the author feedback that it is expected for batch acquisition to be more query efficient than sequential acquisition seem specious to me, and I would remove or rephrase them. If I am selecting a set of points x_1,...,x_k, clearly there is nothing inherent about knowing the labels y_1,...,y_{k-1} that forces a worse choice of x_{k}. Rather, certain sequential acquisition functions like EI are known for being myopic. Having access to more information should always enable more informed decisions than having access to less information. Saying that sequential decision making can "underrate the correlations between consecutive queries" does not seem rigorous enough to explain this claim. Rather, batch acquisition is typically more efficient than sequential optimization in terms of total wall-clock time, and indeed the results presented in Figure 1 are given in terms of time. While the advantages of your approach in terms of time elapsed are clear, I would **strongly recommend** the authors additionally use the supplementary materials to report results in terms of query efficiency to remove this confusion.

Second, your claim that "our topic/work is multi-fidelity batch BO and is irrelevant to early-stopping" seems overly dismissive of Reviewer zV6Z's concern. Every experiment in Figure 4 falls precisely into the setting Reviewer zV6Z is describing, where training can be stopped at any time (e.g., early stopping) rather than pre-determining a set of training lengths to use as epochs, as for example Freeze-thaw BO (Swersky et al., 2014) does, which you yourself cite. While of course in the general setting, fidelities can be arbitrary discrete measures of precision that do not lend themself to early stopping, the experiments you choose to run in your paper do indeed fit the setting considered by Swersky et al. 2014 and others. I would strongly encourage the authors to reconsider which criticisms may be valid and which may be dismissed when preparing the camera ready version, as many of the questions raised here are likely to be shared by readers of the paper at large.
Decision: Accept (Poster)