CAMO: Convergence-Aware Multi-Fidelity Bayesian Optimization
Authors: WEI W. XING
Lu Zhenjie
Akeel Shah
Venue: NeurIPS 2025 poster ()
Keywords: Multi fidelity Bayesian Optimisation; Fidelity Differential Equations; Convergence aware; Acquisition function; Analysis; Regret bounds
Primary area: machine_learning_for_sciences
TL;DR: Multi-fidelity Bayesian optimization via linear fidelity differential equations

Abstract: Existing Multi-fidelity Bayesian Optimization (MFBO) methods ignore the convergence behavior of the multi-fidelity surrogate as the fidelity increases, leading to inefficient exploration and suboptimal performance. We introduce CAMO (Convergence-Aware Multi-fidelity Optimization), a principled framework based on Linear Fidelity Differential Equations (LFiDEs) that explicitly encodes  convergence of fidelity-indexed outputs and employs a closed-form nonstationary kernel. We rigorously prove the existence and pointwise/uniform convergence to the high fidelity surrogate under mild restrictions and provide new convergence results for general FiDEs using smooth, non-smooth and even non-convex Lyapunov functions, establishing a bridge between MFBO and the theory of subgradient flows  in non-smooth optimisation theory.  Combined with a fidelity-aware acquisition function,  CAMO outperforms state-of-the-art MFBO methods on a majority  of synthetic and real-world benchmarks, with up to a four-fold improvement in optimisation performance and a dramatic speed-up in convergence. CAMO offers a tractable and theoretically grounded approach to convergence-aware MFBO.

CAMO: Convergence-Aware Multi-Fidelity Bayesian Optimization"
Wei W. Xing
Zhenjie Lu
Akeel A. Shah Chongqing University akeelshahècqu. . edu, en
University of Sheffield w, xing@sheffield ac.
SUSTch & Shenzhen University 114329768maii sustech. edu. cn
Abstract
Existing Multi-fidelity Bayesian Optimization (MFBO) methods ignore the con- vergence behavior of the multi-fidelity surrogate as the fidelity increases, leading inefficient exploration and suboptimal performance. We introduce CAMO (Convergence-Awe) Aware Multi-fidelity Optimization), a principled framework based on Linear Fidelity Differential Equations (LFiDEs) that explicitly encodes conver- gence of fidelity-indxed outputs and employs closed-fome nonstationary kernel. We rigorously prove the existence and pointwise/uniform convergence the high fidelity surrogate under mild restrictions and provide new convergence results for general FiDEs using smooth, non-smooth and even non-convex Lyapunov func- tions, establishing bridge between MFBO and the theory of subgradient flows in non-smooth optimization theory. Combined with a fidelity-aware acquisition function, CAMO outperforms state-of-the-ee MFBO methods on majority of synthetic and real-world benchmarks, with up to a four-fold improvement in opti- mization performance and a dramatic speed-up in convergence. CAMO offers a tractable and theoretically grounded approach to convergence-aware MFBO.
Introduction
Bayesian optimization (BO) is an efficient method for optimising costly black-box functions, finding applications various domains of engineering and science (1. BO leverages a surrogate model, typically a Gaussian Process (GP), to approximate the objective function and guide the search towards promising regions of parameter space using an acquisition function. This approach is particularly effective when the black-box function evaluations are costly or time-consuming, In many applications, black-box solutions can be obtained at varying levels of fidelity. Lower-fidelit solutions are associated with lower accuracy and lower computational costs, the latter of which typically increase dramatically as fidelity increases. The fidelity can be controlled in a number of ways, by methods varying the modelling choices or the solver settings. For example, electronic-structe involve increasing levels of theory, with exponentially increasing costs 2 Different fidelities can also be defined via the numerical formulation, e.g., use different mesh sizes, time steps convergence thresholds. For the purposes of prediction, and especially for optimization, it is of course desirable and sometimes essential to predict at the highest fidelity. In practice, however, the number of high-fidelity results obtainable within a typical computational budget or time frame is often constrained.
Multi-Fidelity Bayesian optimization (MFBO) methods aim to reduce the cost of optimising high- fidelity black-box functions by leveraging lower-fidelity approximations [ 4. In typical scenarios. the multi-fidelity objective evolves towards the highest fidelity function as the fidelity parameter increases. Existing MFBO approaches ignore this evolution and treat the fidelity as an unstructured input, which can lead to inefficient exploration in high-cost regions and, ultimately, suboptimal
"Code is available httpp:://gichbb.ccmmm/// wwt.://..cmmmmeebbeeeeaee X/CAMO conducted while Zhenjie Lu was at Shenzhen University.
Thee majority of this work was
*Corresponding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

performance. In this paper we propose CAMO, a principled framework for MFBO that uses convergence-aware surrogate model derived from linear fidelity differential equations (LiFiDEs). The "convergence" here refers to a systematic progression towards a high-fidelity function. In contrast to conventional methods, CAMO structurally enforces convergence as fidelity increases, ensuring that predictions at lower fidelities provably approach the assumptions. Our key contributions are: (1) a convergence-aware surrogate that encodes the structural tendency of simulations to converge fidelity increases; (2) a closed form non-stationary LiFiDE kernel that captures fidelity-wise correlations, eliminating the need to specify a relationship between fidelities a-priori (CAMO adaptively learn convergence behaviour from the data); (3) a general theory for fidelity-indexed systems using Lyapunov functions, establishing uniform and pointwise convergence under smooth, non-smooth and even non-convex scenarios, and revealing formal link to uubgradient-based dynamics in modern non-smooth optimization theory; (4) when combined with BOCA (Bayesian optimization with Continuous Approximations) 3 to leverage both fidelity-aware exploration and convergence- baselines on synthetic and real-world benchmarks.
objective under mild regularity
modelling we show that CAMO consistently outperforms MFBO
2 Related Work
Bayesian optimization (BO) [5] uses a surrogate model, typically a Gaussian Process (GP) 66 (see
Appendix [A), to approximate the objective function, together with an acquisition function to guide the search towards promising regions. Acquisition functions balance exploration and exploitation to efficiently search for global optimum. Popular choices include the Expected Improvement (EI) (7]. the Upper Confidence Bound (UCB) [8) and the Probability of Improvement (PI) [9]. MFBO extends standard BO by utilising solutions at diffferent fidelity levels, aiming to reduce the overall optimization cost and improve convergence speed. Most of the work in MFBO has focused on discrete fidelity sets. Huang et al. [10] proposed Sequential Kriging optimization (SKO), which employs hierarchical GP approach to capture the correlations between different fidelity levels. Kandasamy et al. [ developed the Multi-Fidelity (MF) Gaussian Process Upper Confidence Bound (MF-GP-UCB) method based on a generic GP formulation. Le Gratiet and Garnier (12) proposed Recursive Co-Kriging, which builds hierarchy of GP models of the residual between successive fidelity levels. Perdikaris al. [13] used low-fidelity information as an input to high-fidelity GP, leading to a deep GP structure, while Cutajar et al. [14] directly employed a deep GP learn nonlinear mappings between fidelities.
In many real-world applications, the fidelity level can be treated as a continuous variable (c.g., a mesh size or time step). This has led to the development of continuous-fiellic MFBO methods such as BOCA [3, which extends MF-GP-UCB by using a two-step procedure to select the next query point. Poloczek al. [15] introduced the Iulti-Information Source optimization (MISO) framework, which treats the objective function as a linear combination of GPs, each corresponding to different fidelity level. Klein et al. [16] proposed FABOLAS, combining GP with a linear fidelity kernel and an Entropy Search acquisition function. Wu and Frazier 17 developed the continuous fidelity knowledge gradient (cfKG) method based on GP with a knowledge gradient acquisition function. In contrast, Li et al. (18) took a deep learning approach with Deep Neural Network MF Bayesian optimization IDNN-MFBO) which uses fidelity-wise Gauss-Hermite quadrature and a moment-matching mutual information acquisition function. These studies illustrated the benefits of cadtiuuuuu-fidllity MFBO over discrete formulations. Most practical GP-based methods, however, lack insights into the convergence behaviour of the objective function across fidelity levels, leading to the wer-exploratton of costly regions and, therefore, suboptimal performance. Although DNNs can in theory capture the complex relationships between fidelity levels, observations at high fidelity are typically sparse, leading to overfitting and high model variance. Recent advances in MF modelling include fidelity differential equations (FiDE) 199. modelled by NeuralODF: (20 or continuous autoregression (CAR) [99 to capture the convergence behaviour of the objective function. The training times for NeuralODE are up to -fod higher than those for CAR. In this work we therefore build upon the FiDE concept to propose a tractable and scalable MFBO method.
Hyperparameter optimization (HPO) and neural architecture search (NAS) are not considered core applications of CAMO, which is focused on expensive blackbox solvers in science and engineering. In such applications, continuous fidelity indices are unambiguously defined. They arise naturally as controllable, continuous parameters intrinsic to the physical model or numerical approximation. Their effect on the objective function is deterministic, with typically monotonic convergence toward the highest fidelity. In contrast, HPO/NAS operate in a setting in which fidelity is stochastic and algoritmm-dependent, e.g.. epoch number and budget are algorithmic artefacts with no deterministic
2

---

convergence law or trajectory. The cross- cooss-fideiiy structure is non non-smooth and data data-depennent, which requires empirical (ideally discrete-fidelle fidelity) MFBO methods such as BOHB [2T]. DyHPO [22] and FastBO 23 Adapting CAMO to HPO/NAS would potentially require reformulating the underlying convergence model as a stochastic process, which is orthogonal to the contributions of this paper.
3 Background
3.1 Problem Formulation
We consider the problem of optimising function y: X x (0,T) + R. (x,t) + y(x, t), in which
X' S RR is a design variable space, and [0, T) C R is range of fidelity levels. T = co is allowed and is of particular interest. We assume that both the accuracy and computational costs of evaluating v(x,,) () increases with t. The goal is to find the design variable x² = arg x y T) that maximises the high-fidelity objective while minimising the total cost of evaluations. The observations y: at different x and / incorporate additive Gaussian noise (see Appendix A.. LP (1 S p < x))
denotes the space of functions f : :X + R such that Il (ir((x) = fx [ dx < 0o, where dx denotes integration w.r.t. Lebesgue measure. C(x² denotes the space of continuous functions X + R. Ca x [0, x:)) denotes the space of functions f: X' x [0, x)) - IP with continuous derivatives up or orders a and b in x and t, with similar notation for functions of more than two variables (the superscript is omitted if a = b = 0). [[f(l = ess supxe (x) denotes the L norm. AC([O, oo)) denotes the space of absolutely continuous functions and L, (10, oo)) is the space of locally integrable functions on [0, o)).
4 CAMO and Theoretical Results
4.1 Multi-Fidelity via FiDE: Well-Posedness and Link to Subgradient Flows
MFBO requires surrogate to model y(x, t), often in the form of GP: y(x,) gP (0. k(x, f, (')) with some kernel k. Li et al. [20] and Xing et al. [19] proposed that y(x, t) can be modelled the solution to Fidelity Differential Equation (FiDE), namely
0,y(x, () = 6(x, t, yxx t)), (x, E X x (),
(1)
subject to some initial condition y(x. 0) = (x)... Here, 6(x, t, y) describes the system 'dynamics', with the fidelity parameter t playing the role of physical time. Without loss of generality, we set T = oo for the theoretical results. Treating the MF system as dynamical system is entirely natural since the index / is strictly ordered. To conduct MFBO with such model we need guarantees on the well-posedness of the FiDE problem (.. i.e, that unique solutions exist and that y(x, t) converges to unique equilibrium solution You (x) lime- y(x, t). The convergence can be interpreted in a pathwise sense, e.g., for each fixed input x, the solution trajectory converges deterministically to the high-fidelity target. In this work we do not consider stochastic FiDEs, i.e., convergence in stochastic process topologies. The dynamics are entirely deterministic once the training data are fixed.
Convergence can be studied using a Lyapunov analysis. A Lyapunov function V (x,t,)) H V(x, t. y) (dependent on @) serves as a energy-like like functional that measures the deviation of y t) from an equilibrium. Our first result establishes minimal conditions for the existence and uniqueness of classical solutions pointwise in x (see Appendix B.I for the proof)
Lemma 1. [Existence and Regularity for General FiDEs] Let X C be a non-empty set. Consider the system d and assume that: (1) for each fixed x E X, the function (t. + o(x, y) is continuous on [0, 20) x TR; (2) for fixed x E X, o(x,t. t., y) is locally Lipschitz in y, uniformly for t in compact subsets of [0, oo); (3) the initial condition yo yo(x) IR is finite for each x. Then for each fixed x E X, there exists a unique maximal solution y(x, t) E (10, t. (x))) defined on some maximal interval [0, (x)), with
(x( S 00. Moreover: (a) if y(x, 1) remains bounded for all t E [0, os), then (x) = oc and the solution exists globally on [0, oo); (b) if o is globally Lipschitz in y, uniformly in then the solution exists globally on [0, oo).
The following Lemmas (see Appendix B.2 for the proofs) establish minimal regularity requirements on V. yo and X' for pointwise and uniform (in x) convergence to hold, respectively.
Lemma (Pointwise Convergence for FiDE). Let y(x, :) E C* oo)), x E X be a solution to with yo (x) < for each x E X. Suppose there exists Lyppunov function V - x (0, oo) x RR- R.
(x, 1, y) >> V(x, I. y), and that the following conditions are satisfied: (1) V E Co.1. (X x 00) x
R); (2) V (x, t. 0, V v)) X x 10, oo) x I. and V V(x, t,,,, 0 iff y Voo (x): for all
3

---

(x E X' x 00)
1,), -
+ <0
(2)
Then yxx,t) t) + Voo pointwise on X. Le., lim Lemma 3 (Uniform Convergence for FiDE). Let y(x.) (()
each x E X.
X is solution to 7
Suppose there exists a Lyapunov function V X' x [0, x R (x,t,) + V(x,t, t. y) and the following conditions
satisfied: (1) yu(x) E
(2)
x 00, x R): (3)
V(x, t. y) 2 0, A (x,t,y) E X x [0, 0o) x IR and V(x, y) = 0 iff y= yse (x); (4) 3 Cmin > 0 such
that c(xx f) 2 Cunn > 0, v(x,1) 1) X' x [0, oo), in which c(x,t) () = 0 V (x, 1, yoe (x)); (5) for all
x,t E X x [0, oo), 3a > 0 such that
((x1,,)) = V 1,)0,, + -
(3)
(6) 3 Vmax such that [V 0, ((x, (x0) S Vmax- Then lime v(x, 1) (x) uniformly on X, with
[) - y (x) S ce at c= c=vVimsee Vmax//Cuin-
If f(x,t) E C(X x o0) then yoe E C(2)) since the uniform limit of continuous functions is continuous. The uniform-convergeen most mild dependence on l, such as exponential decay or bounded variations, notably quadratic functions V(x, L, 3) a(x, (1 you (x))², a(x, 1) 2 ao > uniformly. Non-smooth Lyapunov functions such as Vi41 = (,( 1) - You (+)] permit a convergence analysis even if y(x, () exhibits non-smooth behaviour due to noise sparse evaluations at low fidelities (in which case O,y has to be interpreted in a distributional sense). Non-smooth V allow for the certification of asymptotic behaviour and exponential convergence in such non-smooth settings. Lemma B in Appendix [B.3 establishes uniform convergence for V/,1 (x, f, y). which is readily extended to Elastic-Net-type and Sparsity-type V. In fact, the result extends to non-convex Lyapunov functions via the Clarke subdifferential [24] (see Appendix [or3 proof)
conditions are readily satisfied for functions V that have
Lemma 4. Let X be a nonempty set, and let y : X x [0, 0o) Th be such that for each x X, y(x,') .) E C [0, oo). Let V = X x TR - TR, (x. v) - V(x, y(x, t)) satisfy the following
conditions. (1) For every fixed (x, t), the map y V(x, is locally Lipschitz. (2) For each R + R. y + V (xx,), and for a fixed x E X, define v-x]()) : 10, 0c) + R, t V yxx,txx))))))) assume that for all (x, t) E X x [0, oo) and all E(x, t) E dcu(x] (v) the inequality E(x,t) t) ø(x, t, vx,, < -or((x(()) holds for some constant
fixed (x,t) t) E X' x 10, define ulx](.)
>> O, independent of x and (3) The initial deviation is bounded, i.e. 11v( 0) 10-00)y-ee Yoo ()))) < co.
Then, for all i. E [0, oo), we have
(lv(,,) () - Woo (11 S 11x(-00) Moo yoc (x
(5)
These lemmas reveal a close analogy between the convergence of FiDEs and subgradient-base dynamics non-smooth convex optimization. For a non-smooth Lyapunov function V. the evolution of y(x, t) can be analysed using differential inclusions involving the subdifferential V. This structure mirrors subgradient flows and proximal-
methods commonly encountered in sparse learning and variational optimization. Under mild conditions, such dynamics guarantee exponential convergence toward the high-fidelity target Vso Uxx(x). (x). with y(x, () following a descent trajectory that reduces the fidelity error encoded by V. Quantitatively, subgradient dynamics induced by non-smooth V can be interpreted as the limit of smooth gradient flows applied to a Moreau- Yosida regularisation 25.. For instance, define the smoothed functional
1 Vx v) = -inf |= yoo (x)1 + 2x (V 2)2 =ER {
(6)
as a regularized approximation of V(x y) = ly you (x)). The associated gradient flow dgyx (x, f) (x, ys (x. t) converges (in the sense of graphical or Mosco convergence) to the subgradient inclusion (x)(x,t) t) E -olv(x,t) t) - Yoo (x)] as À + O. Although FiDEs do not explicitly follow this gradient flow, the analogy provides a useful interpretation: the FiDE-induced surrogate evolution resembles a continuous-i analogue of proximal or subgradient descent, where Moreau smoothing offers stability while preserving convergence guarantees. We refer to Appendix B.4 for further discussion.

---

4.2 Linear FiDEs and Convergence Guarantees
While Lemmas 2 provide a robust and general framework for ensuring convergence, finding suitable Lyapunov function for a specific & is complicated by the fact that the latter is generally unknown and at the very least is difficult to model. Li et al. [20] resort to learning & and (x()) using neural networks. In contrast, Xing et al. [9] considered a more tractable case in which & admits a linear form in y. In this paper adopt the same approach, first establishing the existence of unique equilibrium and the validity of a constructive variation-of oor--of---oosattttt approach. If o admits a linear form in y, we obtain the following linear FiDE for each fixed x E X
dzy(x, 1) = ( f) v(x, t) + u(x,t), E [0,
subject to some (x) = y(x,0) 0) E R (i.e., finite for each x). This is first-order linear, non- autonomous ODE in L. We first establish the global existence of unique solutions and validity of the variation-of-constame formula pointwise in x E X (see Appendix CC. for a proof).
Lemma 5. (Existence and Uniqueness for Linear FiDE) Let X' c Rd be given and suppose that:
(Al) ((x, ), u(x.-) E C([O, oo)); and (A2) yo(x) E R. Then there exists a maximal time t. (x) such that a unique local solution y(x, .) E C²((O, (x))) satisfying 7 exists and is given by
y(x, 1) = elō yo(x))))))())))))))))))))))))y + S-" edi ((x,a)d= uxx, 8)ds, I LE (0,1(((x))
(8)
Moreover, either (x(x)= - xc or lim sups sup: lv(xx = Do (finite-time blowup). If addition: < oo and ()(xxt)ll= (xx00000) < cc, then (x) = 0c. Now let X' C Rd be compact and suppose instead that: (B1) ((x, I), u(x, I) E C(X x o))); (B2)
(A3) [|((x, t.)l
vo(x) E C(X); and (A3) above. Then there exists unique solution y(x, 1) E C(X (0),
((x)) E ((0, of 7 given by (8)
Having established the conditions for existence, the following two lemmas guarantee pointwise or uniform convergence of y(x, t) on X' towards an equilibrium under some general conditions on X, vo(x), ((x, () and (xx, 1) (see Appendices C.2 and C.3 for proofs).
Theorem 1 (Linear FiDE Pointwise convergence). Let (xx t) satisfy the LiFiDE 7 for each fixed x E X' and assume: (1) for each x E X. (xx,)) E C([0, oo)) n L co), [xx t) < O, (xx t) E X x [0, 30) and (x)= E R such that lim B(x, t) (x) < 0; (2) for each
X,
u(x,)) ) E C((O, oo)) and lim; t(x,t)= u ='(x)= E E; (3) [[(xx t) (2xx)))))) = M < co and
[| t)
= = cc. Then
lim y(x, t) = - -xx=((x to
V
The 'long-time" behavior of the solution y(x, 1) is governed by the limiting values of the coefficients ((x,1) 1) and u(x, t). As -> xo, the original non-autonomous equation effectively behaves like the constant-coefffiient equation
ÿ -= y + u² (x),
(10)
which has the unique equilibrium point 8/x (x) 2 (x) 8 (x). Theorem I shows that the solution (x, t) does indeed tend to this equilibrium as t + ca for each fixed x E X.
Theorem 2 (Linear FiDE Uniform Convergence). Let yxx 1) satisfy f Assume: (1) (o()) E C(x);
(2) 3(x, 1) E C(X'x ))), 8(x, 1) < 0, V (x, () E x'x(0, oo). lime [] 1) (x)]l
0, 3 (x) E C(A); (3) u(x,t) t) E C(X' x [0, oo)). lim [lu(x;t) t) - u (x))l = 0, (x(x) E
C(X); 3X > 0 such that (xx) S -X, V E X. Then
lim [|x( t) + u (x)] = O.
(11)
Remark 1 (Consistency of assumptions). The conditions for convergence in Theorems 1 and 2 build naturally upon the conditions established in Lemma 5 In particular, the continuity assumptions 3(x.-), (x(x)) :) E C(OO, oo)) ensure the existence of a unique classical solution y(x,t) C². The additional assumptions required for convergence (pointwise or uniform convergence of 3 t) + 3* (x) and u(x,), 1) - u (x)) serve to control the long-time usymptotics of the system. That is, while continuity ensures solution exists, convergence of the coefficients guarantees that the solution stabilises to an equilibrium. Moreover, the boundedness conditions in Theorem D and uniformity in Theorem 2 (along with compactness of X) allow to lift the pointwise result to uniform convergence.
5

---

Examples of coefficients that satisfy the assumptions in Theorems | and are as follows.
Exponential decay: if B(x, 1) 3 (x) dx(x)e u(xx () u" (x) + 5x(x)e pe with / and òg, du E C(²), then uniform convergence holds.
2. Polynomial decay: slower convergence rates, e.g., with B(x, = 3 (x) + (x)x , will suffice for pointwise convergence, since integrability over [0, oc) is preserved.
3.
case: when 3(x.)) 1) = 8 < O. u(x, () = =²" the solution converges exponentially to the equilibrium Y/oo (x) -u" -x)(xx)3* (x)/3*(x)
4. Bounded perturbations: ((x, = 8 (x) + e(x. t) with bounded c(x, t), oscillatory or decaying.
If <(x, 1) - 0, the function converges to ,9 (x). it does so uniformly, then Theorem 2 applies. only pointwise convergence holds and cxx, -) I. ((0, oo)). then Theorem applies.
4.3 Tractable Autoregressive Gaussian Process Multi-Fidelityy Surrogate
Based on the linear FiDE surrogate, we now introduce a tractable MF model that guarantees con- vergence at least pointwise while maintaining the probabilistic framework of GPs. Let (2, IP) be a probability space supporting two independent stochastic processes yo X' x 1 + R and X x o0) x .2 + R. We place Gaussian process priors over both
(x) ~ GP(O, (x),x
(x, ~ GP(O, hi (x,,,'',
(12)
The joint law of (yo. u) is the product measure induced by P on Vo x U, reflecting their independence. Under these assumptions have the following result (see Appendix D).
Proposition 1. The linear model solution via the
1(x,1) = elō dE xx(x)+ + L.¹ eli' ((x.)) (xx,,))((((())) u(x, 8) ds
(13)
is zero-mean Gaussian process with covariance Eu-ply(x, (:w) v(x'1'' ('; given explicitly by
function k(x,t,x',t
(x. x') + [.', S/
elt (xx,)) de el a(x de (x
efi (lx de ef a(x () ky (x, x', ds ds', (14)
The integrals in the second term in Eq. (14 would in general require numerical quadrature. However, they can be evaluated analytically for various forms of [(x, l), e.g., constant 3(x, () = -8 with stationary kernels such as Matérn and periodic for (x, ł, x, é') (see Appendix E. We also assume that (x, f, x, (') = (x,, x') (t, f'), in which (t, e') = exp (-(t t²)² 222) while k, and
(x(x, x') are kept arbitrary. This leads to the 'LiFiDE kernel'
1(x, xxx,txx,t)) 1') c 6- St' ko(x, x') + hix (x, x' (tt,t')
(15)
in which - = t) + h(t, t')). where h(t', t) is given by (66
ea² (e- (erf (T - a) + erf (t/E + a)) - e 3E4 (erf (t' /2/l--aj a) + erf (a))) (223),
(16)
with erf(.) denoting the error function, a -Be/V2, (1- L')/,, and 7 ((+(')/E. The kernel (5) is non-stationary. Its structure allows us to reduce the training time complexity of the MF surrogate from O(E,, 2)) to N)),,,) where N, is the number of fidelity f data points (19]
4.4 optimization Strategies Using Continuous Fidelity Acquisition Functions
There are several strategies for guiding the selection of the input x and fidelity at each iteration. MF- GP-UCB [1I uses fidelityspecifii upper confidence bounds and selects query inputs by maximising the minimum of a UCB acquisition across discrete fidelities. BOCA 33 extends this method to continuous f, with the fidelity selected from a filtered set. cfKG 4 extends the KG acquisition
function to continuous fidelities by selecting the point (x. E) that maximises the expected reduction in the function, which the fidelity is typically a proxy. It models the validation loss and cost cost jointly as GPs and selects (x, () values that maximise improvement per cost. MF-DNN [18] approximates mutual information between the function optimum and observations using a variational approximation, enabling efficient acquisition in large neural network tuning problems.
posterior minimum. FABOLAS [6] is based on a cost-aware EI acquisition

---

While some acquisition functions can be re-used across MF models, many (e.g., BOCA and cfKG) are tightly coupled to specific fidelity models. In our experiments, BOCA consistently delivered the best empirical performance among the methods above. It also provides theoretical guarantees alongside practical performance, and an improved rate of convergence to the optimum compared to UCB. The original formulation considers the fidelity set T [0. 1] and input space X' = [0, 1jd, with y(x, 1) ~ GP(O, k). K = (ett, f')xxx(x, x'). The key idea in BOCA is to exploit cheaper, approximate fidelities t < 1 when they are sufficiently informative about the target function f(x) 2(x, 1). BOCA filters the fidelity set using a cost constraint and a minimum uncertainty condition. By concentrating expensive evaluations in small, polynomially-diaee dilated variant x, of a nigh-information' region Xp. BOCA achieves tighter regret bounds than GP-UCB and faster convergence. Additionally, BOCA spends the majority of its evaluations on low-cost but informative fidelities, leading to improved "fidelity gap' {(t) = /1 (t, 1)2, which controls the tightness of the bound. The following is an informal version of the simple regret r result in terms of the mutual information "nn (see Appendix for full details).
capital (A) efficiency. The region X2 depends
Theorem 3 (Kandasamy et al. (3. Let X== [0. 1j.. T = [0. 1] and y(x,1) ~ GP(O, k). = Whe (1, L' ( (x, x'). Choose E (0, 1) and execute BOCA with B,, = (O(dlog(n/6) Then, for any E (0, 1), there exist p(o:) > O such that for A large enough, with probability at least 1
r(A) 2 V The (X,)m + Vnng(X)
nA = [A/A(1)]
(17)
For fixed x, x', the standard SE kernel, İSE enforces high correlation near the diagonal z d' with O(K - 2) decay that is independent of I. This implies all fidelities are equally smooth and informative. In practical scenarios, however, low-fidelity evaluations are often noisy or unstable. In contrast, the LiFiDE kernel variance k(t, t) increases with t, and k(t, E') has O(l t'l) decay along the diagonal f', decreasing with (see Appendix G). This models convergent fidelity process: evaluations become more stable and informative as fidelity increases, reflecting a more realistic structure. For the SE kernel, {(t) = O(t Tl) for I -t-T- T < 1 for maximum fidelity T. In contrast, for the LiFiDE kernel {(() (VV(L Ti) (see Remark G1 Although this leads to looser regret guarantees via Xp, the LiFiDE kernel offers stronger empirical performance due to its convergence-aware structure and because it does not concentrate as sharply at - T, discarding potentially useful results at low fidelity. Indeed, the experimental results will show that CAMO conducts most of its exploration in low-fidelity regions, which reduces the query cost and leads to a fast convergence rate in the early stages of the optimization process, without sacrificing accuracy.
5 Experimental Results
We assess CAMO on synthetic benchmarks, including continuous and discrete MFBO tasks, as well
as real-world engineering design tasks. We compare the results those of: (1) BOCA with standard
GP 3 (2) Fabolas [6,, and (3) SMAC3 (7). On discrete fidelity tasks, we compare the results to:
(1) AR 28 (2) ResGP 229 and (3) GP 6 and discrete MFBO baselines (1) MF-UCB 30), (2)
MF-EI 10 and (3) efKG [17]
Settings. We assess CAMO with the LiFiDE kernel Eq. Except for DNN-MFBO, Fabolas, and
SMAC3 (original implementations and default settings), all methods were implemented in Pytorch. All GP models used the SE Kernel for fair comparison. Each model is updated for 200 steps using an Adam optimizer with a learning rate of 0.01 to ensure model convergence. In each case, 10 low-fidelity and 4 high-fidelity designs were randomly selected to form the initial training set. We repeated the experiments 20 times with random seeds and report the mean values. Figures showing the actual optimization progress are provided in Appendix H The optimization performance was measured by the simple regret (7), defined as the difference between the global optimum and the best-queried design so far: 7/6- max f(x, T)) - maxy<i fxy, T)). All experiments were performed on a workstation with an AMD 7800x CPU, Nvidia RTX4080 GPU, and 32GB RAM.
5.1 Synthetic Benchmark Evaluation
We consider: (1) three canonical continuous-fide fidelity tasks [3.. the Park, Currin and Branin func-
tions; and (2) three further synthetic continuous- fidelity tasks (2 the nonlinear sin, Forrester, and
Bohachevsky functions. In the latter we set f(x, 1) (1
wr(()) fow + figgh (x) with
wt) = In(9t + 1). All functions are defined in Appendix The query costs were set to c(t) = 10
7

---

BOCA
dKGG
CAMO-BOCA
CAMO-CIKG
Faboles
SMAC3
1.75
0.5
Smmple regret
1.25
Smplle
Simple
1.00
0.75
0.4
0.50
100
150
203
250
330
0.00
50
100
150
200
250
300
0.0
100
150
200
250
300
Cost
Non-linear Sin
Forrester
Bohachevsky
0.035
20.0
0.30
20-
17.5
15.0
12.5
0.020
Simple
Simple
10.0
1.0
0.030
0.000
250
8.0
Figure I: Simple regret vs cost for all continuous fidelity functions functions.
BOCA
CAMO-BOCA
CAMO cfKG
Fabolas
SMAC3
borehole
colvile
himmelblau
100
400
120
100
Simple regret
300
Simple regret
250
200
60
40
100
50
50
150
30
250
30
150
200
Cost
Cost
Cost
Figure 2: Simple regret for the Borehole, Colville and Himmelblau fuucciion.
We show 7 the six tasks under increasing query cost in Fig. D which clearly demonstrates the superiority of CAMO across all tasks, with the exception of the non-linear sin at low cost. BOCA also performs well on the Branin and Park functions. The results for different random seeds are shown in Fig. [H--Fig. H9 in Appendix H to demonstrate the consistency of CAMO. These figures also show that methods equipped with cfKG (including CAMO) are generally less competitive in terms of and convergence speed compared to those with BOCA. To investigate the scalability of the results in terms of d, we further evaluated CAMO on the Borehole, Colville 33 and Himmelblau 34 functions (6, 4, and design variables, respectively). The results are shown in Fig. 2 In these cases the superiority of CAMO just as pronounced, suggesting robustness increases.
One of the key factors in MFBO is the cost e(t) of querying the high-fidelity function. We conducted experiments with different costs: c(I) 10 e(l) 51, and c(t) log, (2 for the Currin and Bohachevsky functions. The results are shown in Fig. and for the Forrester function Fig. HT0 Clearly, CAMO outperforms all other methods under all cost settings. An exponential cost setting is more challenging for all methods. The advantage of CAMO is increasingly significant from c(t) logg (2 + t) - 5t 10 which highlights the advantage of being convergence-aware in MFBO by maximising the benefit/cost ratio. For logarithmic cost setting. CAMO performs similarly to BOCA because being "convergence-avee aware" is less "rewarded" when the cost also increases logarithmically. In contrast, if the cost increases beyond linear, the advantage of CAMO becomes significant. Such a setting is common in real- weal-world applications, e.g., FEM (cubic complexity with mesh resolution (55 and Monte Carlo estimation (quadratic sample requirements (36). Importantly, CAMO always exhibits a fast convergence rate the early stages of the optimization since does not discard useful low-fidelity information. This is consistent with the theoretical analysis.

---

BOCA
cfKG
Fabolas
SMAC3
CAMO-BOCA
3.0
2.5
2.5
Simple regret
Simple regret
2.0
2.0
1.5
1:5
1.5
1.0
1.0
1.0
0.5
0.5
0.5
0.0
25
50
100
125
150
0.0
25
50
75
100
125
150
0.0
25
100
125
150
Log
Cost: Linear
30
30
30
25
Simple regret
Simple regret
Simple regret
15
15
15
10
10
10
150
Cost: Log
Cost: Linear
Cost: Power 10
Figure 3: Continuous MFBO for the Currin (top row) and Bohachevsky (bottom row) functions using logarithmic (2og2 (2 t)), linear (5t), and exponential (10') cost c(t).
AR-MF-UCB
AR-MF-EI
AR-cfKG
CAMO-BOCA
ResGP-MF-UCB
ResGP-MF-EI GP-MF-EI
ResGP-cfKG
CAMO-cIKG
MF-GP-UCB
cfKG
DNN
Simple regret
Simple regret
0.6
0.4
80
0.30
130
140
Cost
Cost
Figure 4: Discrete MFBO on the Branin (left) and Currin functions (right).
Discrete MFBO Assessment. To examine performance on discrete MFBO tasks, we discretise the Branin and Currin into ten discrete fidelities. We also compare with the DNN-MFBO method [8]. The results under increasing query costs are shown in Fig. 4 We can see that CAMO again outperforms all other methods by wide margin the Branin function, while the advantages are clear but less dramatic on the Currin function. DNN-MFBO essentially failed.
For MFBO, the time for model update and acquisition- function based optimization is also crucial. The average times (over all the benchmarks, iterations, and seeds) of the optimization queries and model training are shown in Fig. for different query and MF approaches, respectively. Despite the favourable performance reported Li al. 18. MutualInfo and FABOLAS are impractically slow in terms of query time. SMAC3 has the shortest simulation time, but is not competitive in terms of accuracy and convergence rate. CAMO intermediate between these methods for both query time using BOCA and training, achieving good trade-off between performance and computational cost.
5.2 Real-World World Applications in Engineering Design
now consider real-world continuous tasks (see Appendix [) (1) Mechanical Plate Vibration.
We optimize the natural vibration frequency of 3-D, supported, square elastic plate (10 x 10 x in m) over the Young's modulus (E [100, G00][ Poisson ratio (E [0.2, 0.6) and mass density (E [6 x 103 9 10²][kgm--33 -3)). This is parametric FE modal analysis. The maximum element size

---

23/67
10"
148.56
16,87
15,79
Query (secomtss
Qvery time (scconds)
20*
Treining Eime (beconds)
102
10
2,55
1.26
100
0.53
0.33
10-1
10-1
10
10-2
Training
20*
0.58
10"2
0.5
10-3
310
10-3
10
10-4
MF-UCB
dkG
Mi-Bi
Mutuatingg
BOCA
cRk
Fabolas
SMAC3
caso ROCA
CAMO enka
DMN MFBa
CAMO
Fabolaa
SMACs
CAMO
Figure From left to right: average query time on discretised tasks, average query time on continuous tasks, average training discretised tasks, and average training time on continuous tasks.
<fGG
CAMO-cfKG
Fabolas
SMAC3
VibratePlate
HeatedBlock
10
regret
38
Simple regret
1.0
36
0.0
0.6
.22
0.4
30
28
0.0
Cost
Cost
Figure 6: MFBO on the Mechanical Plate Vibration (left) and Thermal Conductor Design (right) problems with different query cost.
hmax E [0.2, 1m2][m] is the fidelity. (2) Thermal Conductor. We optimize the shape of an elliptical central hole in which conductor is placed in order to maximise heat conduction rate. The hole shape is parameterised by the semi-major and semi-minor axes and orientation angle. We used the time to reach 70 degrees as the objective function value. Amax [0.1, 2] [m]] is the fidelity.
The ? versus simulation cost is shown in Fig. 6 The cost scales according the inverse cubic of the maximal element size, which is the standard computational complexity for FEM problems. CAMO and BOCA outperform the other methods both tasks by a significant margin. In particular, the fast convergence of CAMO carries over to these real-world problems, with BOCA yielding comparable performance on the Thermal Conductor and slightly worse performance than CAMO on the Mechanical Plate Vibration. In terms of wall clock time to a given 7, CAMO and BOCA are very similar, with FABOLAS again impractically slow.
6 Conclusions
We propose CAMO, a convergence-aware MFBO framework based on LiFiDEs for continuous fidelity problems. CAMO captures the fidelity-wise evolution of the objective function and provides a theoretically-gronnee
surrogate model. For general FiDE we used a Lyapunov-bart based analysis to establish convergence guarantees for non-smooth objectives. Combined with BOCA. CAMO delivers strong empirical performance, consistently outperforming state-of-the-ar MFBO methods on synthetic and real-world tasks. Crucially, CAMO adapts and exploits informative low-fidelity queries rather than discarding them, enabling more efficient use of the evaluation budget and fast convergence.
CAMO has several limitations that are worth noting. Computationally, the kernel optimization adds roughly 20% overhead compared to standard GP methods, and the approach requires sufficient low-fidelity data to properly model convergence behaviour. Methodologically, the LiFiDE assumption may not capture all (especially non-monotonic) convergence patterns, and performance depends on the existence of a good cost model. CAMO is suited to scenarios which the computational budget is the primary constraint and early progress is valuable. For unlimited budget scenarios, more aggressive figh-fidelity exploration might be preferable.
10

---

Funding Disclosure
The authors declare no external funding.
References
[1] Eric Brochu, Vlad M. Cora, and Nando de Freitas. A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning. arXiv:0012.2599 [es]. December 2010.
[2] Akeel A Shah, PK Leung, and WW Xing. Rapid high-fidelity quantum simulations using multi-step nonlinear autoregression and graph embeddings. npj Computational Materials, 11(1):57, 2025.
[3] Kirthevasan Kandasamy, Gautam Dasarathy, Jeff Schneider, and Barnabás Póczos. Multi-fidelity bayesian optimisation with continuous approximations. In International conference on machine learning, pages 1799-1808. PMLR, 2017.
[4] Jian Wu, Saul Toscano- Toscano-Palmerin, Peter I. Frazier, and Andrew Gordon Wilson. Practical Multi- fidelity Bayesian Optimization for Hyperparameter Tuning. In Proceedings of The 35th Uncertainty in Artificial Intelligence Conference, pages 788 798. PMLR, August 2020.
[5] Bobak Shahriari, Kevin Swersky. Ziyu Wang. Ryan P Adams, and Nando De Freitas. Taking the human out the loop: review of bayesian optimization. Proceedings the JEEE, 104(1):148-17) 175, 2015.
[6] Carl Edward Rasmussen and Christopher K. I. Williams. Gaussian Processes for Machine Learning. Adaptive Computation and Machine Learning. MIT Press, Cambridge, Mass, 2006. ISBN 978-0-262- 18253-9.
[7] Jonas Mockus. The application of bayesian methods for seeking the extremum. Towards global optimization, 2:117, 1978.
[8] Niranjan Srinivas, Andreas Krause, Sham Kakade, and Matthias Seeger. Gaussian process optimization in the bandit setting: No regret and experimental design. arXiv preprint urXiv:0912.3999, 2009.
[9] Kushner. A new method locating the maximum point an arbitrary multipeak curve the presence of noise. Journal of Basic Engineering, 86(1):97-100 1964.
[10] Deng Huang, Theodore T Allen, William Notz, and Allen Miller. Sequential kriging optimization using multipe-fidelity evaluations. Structural and Multidisciplinary Optimization, 32:369-382 -302, 2006.
[11] Kirthevasan Kandasamy, Gautam Dasarathy, Junier Oliva, Jeff Schneider, and Barabáss Póczos. Gaussian process bandit optimisation with multi multi-fidelity evaluations. Advances in neural information processing systems, 29, 2016.
[12] Loic Le Gratiet and Josselin Garnier. Recursive co- co-kriging model for design of computer experiments with multiple levels of fidelity. International Journal for Uncertainty Quantification, 4(5), 2014.
[13] Perdikaris P., Raissi M., Damianou A., Lawrence N. D., and Karniadakis G. E. Nonlinear information fusion algorithms for data-cfficient multi-fidelity modelling. Procoedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, 473 (73(2198)20066777) February 2017. doi: 10. waa.0099/pp...o (000/sp..000..0077.... 2016.0751.
[14] Kurt Cutajar. Mark Pullin, Andreas Damianou, Neil Lawrence, and Javier González. Deep Gaussian Processes for Multi-fidelity Modeling. arXiv: 903.07320 [cs, stat], March 2019.
[15] Matthias Poloczek, Jialei Wang, and Peter Frazier. in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
Source Optimization. In Advances
[16] Aaron Klein, Stefan Falkner, Simon Bartels, Philipp Hennig, and Frank Hutter. Fast bayesian optimization of machine learning hyperparameters on large datasets. In Artificial intelligence and statistics, pages 528-536. PMLR, 2017.
[17] Jian Wu and Peter I Frazier. Contimuous- bayesian optimization with knowledge gradient. 2018.
[18] Shibo Li, Wei Xing, Robert Kirby, and Shandian Zhe. Multi Hudti-Fidelity Bayesian Optimization via Deep Neural Networks. Advances in Neural Information Processing Systems, 11:8521-8531 2020.
[19] Wei Xing, Yuxin Wang, and Zheng Xing. Continuar: Continuous autoregression for infinite- fideltty fusion. Advances in Neural Information Processing Systems, 36, 2024.

---

[20] Shibo Li, Zheng Wang. Robert Kirby, and Shandian Zhe. Infiiite-fidelity coregionalization for physical simulation. In S. Koyejo, Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 25965-25978. Curran Associates, Inc., 2022.
[21] Stefan Falkner, Aaron Klein, and Frank Hutter. Bohb: Robust and efficient hyperparameter optimization scale. International conference on machine learning, pages 1437-1446. PMLR, 2018.
[22] Martin Wistuba, Arlind Kadra, and Josif Grabocka. Supervising the multi-fidelity race of hyperparameter configurations. Advances Neural Information Processing Systems, 35:13470- 3470--34444, 2022.
[23] Jiantong Jiang and Ajmal Mian. Fastbo: Fast hpo and nas with adaptive fidelity identification. arXiv preprint arXiv:2409.00530 2024.
[24] Frank Clarke. Optimization and nonsmooth analysis. STAM, 1990.
[25] Neal Parikh, Stephen Boyd, al. Proximal algorithms. Foundations and trends@) in Optimization, 1(3); 127-239, 2014.
[26] Neil Lawrence, Guido Sanguinetti, and Magnus Rattray. Modelling transcriptional regulation using gaussian processes. Advances Neural Information Processing Systems, 19, 2006.
[27] Tobias Domhan, Jost Tobias Springenberg. and Frank Hutter, Speeding up automatic hyperparameter optimization of deep neural networks by extrapolation of learning curves. In Twenty-fourth international joint conference on artificial intelligence, 2015.
[28] M. Kennedy. Predicting the output from complex computer code when fast approximations available. Biometrika, 87(1):1-13, March 2000. ISSN 0006-3444, 1464-3510. doi: 10. 10.1093/biome//7.111.
[29] W. W. Xing, A. A. Shah, P. Wang, S. Zhe, Q. Fu, and R. M. Kirby. Residual gaussian process: tractable nonparametric bayesian emulator for multi-fidelity simulations. Applied Mathematical Modelling, 97: 36-56, Septemher 2021. ISSN 0307-904X. doi: 10. 10.1016/j.apm wttpj..am..222.03.00
[30] Kirthevasan Kandasamy, Gautam Dasarathy, Junier Oliva, Jeff Schneider, and Barnabas Poc- Zos. Gaussian Process Bandit Optimisation with Multi-fidelity Evaluations. In Advances in Neural Information Processing Systems, volume 29, Curran Associates, Inc., 2016.
[31] Shifeng Xiong, Peter 7G Qian, and CF Jeff Wu. Sequential design and analysis of high-accuracy and low-accuracy computer codes. Technometrics, 55(1):37-46, 2013.
[32] Bruce Ankenman, Barry I. Nelson, and Jeremy Staum. Stochastic kriging for simulation metamodeling. In 2008 Winter simulation conference, pages 362-370. IEEF, 2008.
[33] Jialin Song, Yuxin Chen, and Yisong Yuc. A General Framework for Multi-fidelity Bayesian Opti- mization with Gaussian Processes. In Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics, pages 3158-3167. PMLR, April 2019.
[34] Huachao Dong, Baowei Song, Peng Wang, and Shuai Huang. Multi-fidelity information fusion based on prediction of kriging, Structural and Multidiscipiinary Optimization, 5166):1267-1280, June 2015. ISSN 1615-147X, 1488-1488. doi: 00.1007/001158-0444 111-1999
[35] Thomas JR Hughes. The finite element method: linear static and dynamic finite element analysis. Courier Corporation, 2012.
[36] Russel E Caflisch. Monte carlo and quasi-monte carlo methods. Acta numerica, 7:1-49, 1998.
[37] Gerald Teschl. Ordinary differential equations and dynamical systems, volume 140. American Mathemat- ical Soc., 2012.
[38] R Tyrrll Rockafellar and Roger J-B Wets. Variational analysis, volume 317, Springer Science & Business Media, 2009.
12

---

Appendices
A
Gaussian process
Consider observations y4 (xx) e. I,.... N, in which NN o²) is additive noise. In GP model.
a prior distribution placed over f(xx
f(x) ~ GP (mo(x), ())o, x0)))
(A1)
(xx) mo(x')) is the covariance function. set of hyperparameters 0 fully characterises the kernel function and in most cases the data is centred (the empirical mean is subtracted) to justify setting m=(x) = as a simplification. Various forms can he adopted for the covariance function, with the following exponential ARD kernel being the most widely favoured
in which mo(x)= (x)] is the mean function and k(x, x'10)
k(x, x (0) 0s exp (-(x - x') diag(O: (0, ....,,;, (xxxxx) -X).
(A2)
By the key property of GPs, the joint distribution of f(z;), i I...., N. is multivariate Gaussian. This leads to the following conditional predictive posterior conditioned on = (y1,.... ,UN) and x
f(x) N (((x), 0(x, x'))
m(x) k(x) (K + 0 1²)) "
y.
(A3)
o(x) 0² k(xx x) k (x) (K + =²)) -2()='((x)
in which K [K,g] Kú ,(x;,x)) xy), i,j 1,..., NN, the covariance matrix, and (k(x,, x x..., M(x,,, x)). The hyperparameters are typically inferred from the likelihood, given by
Ly Z (K 00 + ln/K o log(2r).
-
(A4)
B Well-posedness of general FiDEs
B.1 Existence and uniqueness
Here we consider the well posedness solutions the general Fidelity Differential Equation (FiDE)
0,y(x,t) t) = o(x, f, v(x, f)), (x, f) E x [o. oc),
(B1)
in which is any nonempty set, subject to some initial condition y(x, 0) (x))..
Lemma 1. [Existence and Regularity for General FiDEs] Let X' C TR' be non-empty set. Consider the system BI and assume that:
for each fixed E the function (t, + o(x, v. is continuous on [0, oo)
2. for fixed fixed of [0, oo)
X, 6(x, 1, is locally Lipschitz continuous in y, uniformly for t. in compact subsets
3. the initial condition vo(x) R is finite for each
Then for each fixed x E X. there exists a unique maximal solution y(x,t) t) C.l (x))) defined on some maximal interval [0, (xx)). with 0 < (x(x) S co. Moreover:
if u(x, 1) remains bounded for E [0, oo), then (x(x) = oo and the solution exists globally on co)
if Φ is globally Lipsehitz in y, uniforrnly in t, then the solution exists globally on [0, oo).
Proof. Fix x X. then the equation becomes an ordinary differential equation (ODE) in the variable
ý o(x, v), (0) yo (x).
(B2)
13

---

Since by assumption (1), the function (1, ~> 6(x, I, 4) is continuous, and by assumption (2), o(xx (.y) is
locally Lipschitz in y. it follows from the Picard-Lindelit Theorem that there exists a local time (.(x) > and unique solution >> y(x,() 1) E C* ([0 L- (x))) that solves the ODE for E [0. (x)). Uniqueness of the solution y(x, t) follows directly from the local Lipschitz property of & in y, which prevents branching of solutions and ensures that the solution is unique once (x( is fixed.
By the general theory of ODEs (continuation thcorems)
37 the maximal existence time (x(x) satisfies
L, (x) 0o
sup [w( 4) = oc, (x)
(B3)
the latter representing blow-up up of the solution in finite time t.. Thus, if we can guarantee that y(x, t) remains bounded for all E [0, so). then it follows that to (x() co, and the solution extends globally. If é is globally Lipschitz in y uniformly & is continuous in (f, y) by assumption (1), and y(x, t) solves the ODE, standard regularity theory tells us that -) E C' ([O, f. (x))). Specifically, (x, t) = <(x, t, (x, t)) is continuous, as the composition of continuous functions.
then no blow-up
occur, and the solution is guaranteed to exist globally. Since
B.2 Pointwise and Uniform Convergence for FiDE
We now consider the convergence of solutions to BI The following results provide a practical set of conditions under which y(x, f) + Yoo (x) pointwise or unitorm in
Lemma 2 (Pointwise Convergence for FiDE), Let y(x..) E CO,, x E X a solution to 1 with yo (x)) < oo for each x E X. Suppose there exists a Lyupunov function V : N' x [0, (0) x R + R.
(x, L, 1) > V(x, L. and that the following conditions are satisfied:
1. V C" (X (z) IR)
2. V(x,4,) 4) 2 0. (x, E a's x (0, (0) x R and V(x,1,)) = 0 0 (x)
3. for all (x, E oo)
V(x, (,k, 0,V ())ox, 0.V (x,,,) <
(B4)
Then t) + (x) pointwise on X, i.e. lime y(x, t) you (x) for each E X.
Proof. For each t, define the function yt : X + R. x ++ y(x. t). Under the assumptions, we will show that the family of functions (y+)+20 converges pointwise to yoo on X, that is
lim u(x, t) you (x), Vx E 1
(B5)
x E X and consider function ++ (x, 1, y (x, f)). By the chain rule, and using the assumptions, we have
(xx 4, v) = 8, Vx, 1, y) 0,y(x, 4) + 0 (x. 8, (x, t, y) o(x t, y) + (x,,,),,))
(B6)
using 0xy(x, t) = ó(x, 1, By assumption, decreasing along the trajectory for each fixed x. Furthermore, V(x, t, v) > 0 for all / 2 0 by assumption, and V(x, f, y) is bounded below by O. Now + V(x, v(x,t), +)) continuous-time function and so for fixed (V(x, y(x, t), to) R defines bounded, monotone on the directed set ((0, oo), S), where is the usual order relation. Therefore, this net converges to its infimum
d (x, t, y) is strictly negative, so that V(x, t, y) is strictly
ar
3 lim ((x, 4, vxx, 0)) = inf V(X, 4, 4,x,,,,(x, 0)) L(x) 2 0. 0-
(B7)
2e 0,00,00)
Since V is strictly decreasing in t and positive, its limit must be 0 (otherwise it would eventually fall below Lxx), contradiction) and thus L(x)= = 0, i.e.
lim V(x,t,)) 0. Lo
(B8)
Finally, V(x,t,y) = if and only y = you (x), and thus
lim y(x, 1) = (x).
(B9)
Since E X was arbitrary, obtain pointwise convergence on X.
Lemma (Uniform Convergence for FiDE), Let (xx E C² ([0, oo)), x E X is a solution to 1 Suppose
there exists Lyapunov function V X x [0, oo) x IR + iR, (x, + V V(x, t, y) and the following conditions are satisfied:
14

---

1. (x) E C(X)
2. V E C0..2 x (0, 00) x 1R)
3. V(x,t,y) 2 0. (x,,,, y) X's [0, oo) and () 0 iff yxx
4. 3 Cmin > 0 such that c(x,1) () 2 Cain > O. v(x,))
X (0, in which c(x,1)
2 Sy V(x,t, Voc (x))
5. for all x, E X' x oo). 3 > 0 such that
V(xt,y) v) 02 1. -aV(xxtt)))
(B10)
6. 3 Vmax such that |V (x, 0, y 0)) (x) Vmax
Then sim v(x,1) t) = y (x) uniformly on X, with
Vmax
(y(x, You (x)]l== ce att
(B11)
Cmin
Proof. From assumption 5 and Grönwall's inequality 37 we have for all (x, t) E E [0, oo)
V(x, ((x, (). 1) S V 0))
(B12)
By the smoothness assumption V C (X' x [0, oo) x R), a Taylor expansion around y = yeo (x) yields, for each (x, t)
V(x, (x, t), 2 c(x, ()) t) you (x)) in which c(x, 1) = 1 ((x, you (x), f), and by assumption 4, c(x, t) 2 Cmin 0 uniformly.
(B13)
Thus
b((x t) you (x)) S c(x,1 1) V VV ((x,))x < (x (xxxxxxx)) 0)e at/2
(B14)
Cmin
Taking the supremum over X' yields
Vmax -at/2
llx(x,)) - (x) (10
(B15)
Cmin
which proves uniform convergence.
B.3 Non-smooth Lyapunov functions
Lemma B1. [Convergence for non-smooth convex Lyupunov function] Let X be a nonempty and let (x.
satisfy B7 Define (xx, y, 1) Iy] Who (x)) for (x, N.
[0, oo). Suppose that for each x E X:
1. The map + yxx, t) is (²0 )))
2. Let 0,V(x, y, be the subdifferential mapping y + V(x, v, For every (x,t) E X x oo) and every E(x, t) E Vxx, y(x, t), t), we have
E(x, L) 6(x, L, v(x, ()) S -xV(xxx(xxxx)
(B16)
for some constant >> 0 independent of x and t.
Then for each X and every [0, oo).
(x(xxt) t) - yoo (x)(x)] S 1y(x 0) - 120 (x))le
(B17)
If, moreover
3 [1x(x,0) 0) You (x) (x) = =<
the convergence uniform x E E
llv(,t) t) Yoo (x)) s aa W 10, 00).
(B18)
Proof. For each fixed (x, 1) E X x [0, oo), the map
u(x, ()(:) + R. 4 V(,,
(B19)
15

---

is convex and I-Lipschitz continuous in For fixed E X, we also define the function
xxx]() (0, R, + (xx, (xx t)
(B20)
y(x.-) E AC([O, x))) (ahsolutely contimuous) since is in C' ([O, nc)), and thus the composite map (+x (+) absolutely continuous on [0, oo). By the properties absolutely continuous functions, v(x[(.) ) is differentiable almost everywhere, and its upper right Dini derivative
x + h) (t)
D (t) = lim sup
(B21)
h-+0+
exists for every E [0, co).
The subdifferential at point y, of the convex function u[x, t](-) is defined as the set of subgradients
Oufxx, ((y-) E (l)) 2 u|x, (](y)) + glv y.), Vy R) (2 38) In the present case, the set-valued subdifferential mapping Qu(x, (v), R 2 is given by
(+1),
u(x, t) > y/ (x),
Ou[x, d](y) 01y(x, t) - you (x)l
(-1).
u(x,t) < y (x),
(B22)
l],
,(x,t) = yoo (x).
Applying chain rule to u w|xx, (](+) using the definition the subdifferential mapping a convex function we obtain
(f) E 0g ()())),y(xx f) = 0, /x, ()(v) ø(x, f, uxx, f)), a.c. E [0, oo)
(B233
by using the definition of Given assumption (2), for any subgradient E(x, f) E 8k|2x, f)(v) E IR we have t(x,t) o(x, f, (xx f)) (t) which holds for é [0,oo] oo) a.e and pointwise E with y(x, t) fixed. Thus, for & [0, oo) a.e.
(B24)
e(x((t) S (t)
(B25)
Since (t) E AC((O, oo)), by the standard properties of Dini derivatives, this incquality extends to the upper Dini derivative, yielding
D v(x((t) (t), V [0, 20).
(B26)
Applying the generalised Grönwall inequality for upper Dini derivatives (which states that if D+ aau(t), then u(t) S u(0)e "f) we obtain
V(x, v(x, t) V(x, v(x, 0), 0)e
(B27)
Recalling that V(x, v. Voc (x)), this yields
lu(x,1) (x)) & 1xx, 0) yoo (x)le_" VI E (0)
(B28)
pointwise E X. By assumption (3) this holds in the T (22) norm by taking the sup both sides, i.c.
Ilv(,,t) M (2) I le Vt [0, oo).
(B29)
This result can extended non-convex Lyapunov functions by using the Clarke subdifferential 24 first introduce some definitions. Let X be a Banach space with dual X The weak topology on X is the coarsest topology such that all evaluation maps ff + f(x) for x E X are continuous. That is, fa + f in the weak* topology iff (x) + f(z for all x E X. Now let f IR + R be locally Lipschitz. The Clarke subdifferential of at is defined
fc)(x)= = =o
Vf(x)) : m + x, differentiable at rk T
(B30)
in which co denotes the convex hull of set. This set is always nonempty, convex, and closed. If If = o h. where g R - R and h B - Bd are locally Lipschitz, then for any E B (24), Chain Rule Theorem 2.3.9)
de "0" (ac
(cg(h(()), < dch(()).
(B31)
which co" { denotes the closure of the convex hull in the weak" topology on IR" = R. In finite dimensions, notably in our case H., this is simply the norm norm-closed closed convex hull and the ttrndardnorm, weak, and weak" topologies are equivalent.
16

---

If we set h(t) = s(x,0) () and g(y) = (vx (we assume V has explicit dependence on 1), then f(() (1) as previously defined. Since y(x,) C* ([O, oo)), dev(() (ixv(x. ()), that
(te c (a 8) Oca(x
(B32)
Now, the image a convex set under linear map convex, and therefore 1332 reduces to
(c C (x ())c
(B33)
since co"" and furthermore the Clarke subdifferential is already closed and convex.
We know that the composition (t) (xx, yx, t),t) E AC((O, oo)) because locally Lipschitz in y
and v is differentiable. Then by Clarke's Chain Rule Theorem above
((x[(1) (t) E 0,y(x, ()0cv(x((v) a.c. & oo).
(B34)
If we assume that
É(x, t)x(x) v(x. t)) (t) vE(x,t) @cu(x](v).
(B35)
then
Dt x*x(xx(t)= v (t) Vt E [0, oo).
(B36)
We can then use Grönwall's inequality and take the supremum over x E X' to obtain a uniform bound. This proves the following lemma.
Lemma 4. Let X be a nonempty set, and let y : X x [0, oo) + be such that for each x E X, v(xx
C' [0, oc). ) Let V : X' x R + R, v) + ((x, (xx f)) satisfy the following conditions
1. For every fixed (x, t), the map V (xx, v. is locally Lipschitz.
2. For each fixed (x, E X (o), define u u-x R + R. + V(x, y). and for a fixed x E X. define v-xx(-)) 10,
+ RR t V (x, v(x f.)- Assume that for all (x,t) E X 10, Do) and all
t(x,t) E 0cu(x] (v) the inequality
E(x,, 1) 6(x, t, y(x, 1)) S -ax
(B37)
holds for some constant a > 0, independent of x and
3. The initial deviation is bounded: llv(,0) (- < <o.
Then, all t E [0, 20), we have
Ilv(;;t) - yoo yoo (-1l $ 1lv(,,0) (x)()
(B38)
B.4 Connection to Subgradient Flows and Proximal Methods
We have analysed the convergence of fidelity-indexed systems y : X'X x [0, oo) R. which serve surrogate models for high-fidelity target function you : X - IR. These surrogates evolve over fidelity index according to the FiDE dynamics (T. When a non non-smooth Lyapunov function such as V(x, y. = ly yos (x)
is used to study convergence, the time evolution must be interpreted using subdifferential calculus. In particular,
the upper Dini derivative satisfies the inclusion
t),t) E E(x, f) o(x f, v(x, f)) + OV
at (x, v(x, t),
(B39)
in which t(x,t) E (x, v () is subgradient. For example, when V(x, v. lv - y (x)]. V y, 1) is given by (B22). This structure defines a differential inclusion, and directly analogous to subgradient flows commonly used in convex optimization, particularly for non-smooth regularisation problems such as LASSO. In these cases, the evolution of a state xt)) under non-smooth objective is given by
*(t) E -Qg(x(t))-
(B40)
For the case in Lemma B1 the surrogate y(x, evolves by descent the 'fidelity emror g(v) = ly Y-o (x)]. driving it asymptotically towards equilibrium yos (x). Under mild monotonicity assumptions on Φ. this descent yiclds exponential convergence, analogous to the behavior of proximal point iterations in optimization.
This framework extends naturally to more general non non-shooth Lyapunov functions, such as the clastic-net form
V(x. y. l) A1 Yoo (x)) + A2(w Voc (x))
(B41)
or tnats
sroupspeee sparsity objectives for vector-valued surrogates y X' + ged
V V(x, v, = Σ
Ax t) - (x)).
(B42)
17

---

in which G is partitioning of the coordinates. In all such cases, the surrogate y(x, () evolves via subdifferentil inclusion that mirrors proximal-map -aa dynamics in optimization, linking multi-fidelity modelling with tools from modern sparse learning and non-smooth convex analysis.
The subgradient dynamies discussed above can also be viewed a limiting case of smooth gradient flows applicd to a smoothed surrogate objective. In convex analysis, the Moreau envelope provides classical smoothing of a non-smooth function. For a convex, lower semi continuuut function f IR + R. the envelope is defined as 25
inf { [(=) ZER
(a)
z1?
(B43)
2A
with (z) prcexprxxxyyee (x)) This provides a differentiable approximation of f, and as - 0, we have (w) - (w) pointwise, and / - (y) in sense of graphical (or Musco) convergence.
While the actual FiDE dynamies are not given by gradient flow, the Lyapunov convergence analysis admits a natural interpretation via these smoothed flows. For instance, using the fidelity error function f(y) ly Yo V=x(x), (x)), we define the Moreau envelope
SER inf (1= 11- 2x(0 (v 2)²
Vx(x,
(B44)
This yields the auxiliary gradient flow
(x, V (x(,
(B45)
which approximates the subgradient dynamics
dy((x, t) E -0ly(x, you (x)]
(B46)
as + O. While this flow is not the governing equation for y(x, t) in the FiDE, it serves as a useful analytical proxy, capturing the key descent structure and convergence behaviour of the surrogate. From this point of view, the Lyapunov function reprexents fidelity error whose decay over t encodes asymptotic convergence to the high-fidelity limit.
C Well-Posedness and Convergence of Linear FiDE: Proofs
We consider the linear FiDE
0.g(x, t) = [xx, f)y(x, u(x,)), (x,t) E X' x (0)
(CI)
subject an initial condition (x(x) y(x, 0).
C.1 Proof of Lemma
Proof. Let X' C Ied be non-empty and fix x E N. Suppose that:
(A1) B(x, ), u(x,)) E C((O,
(A2) (x) E IR
Define Φ(x, L, v) = (xx, t) y + u(x, t). For fixed x. map o(x, -. continuous (t, since both ( t)
and u(x, 1) are continuous in t. Moreover, ó is globally Lipschitz in y since
1Ø(x, f, v/)) o(x, f, v2)l [3(x, t)] ly yal-
(C2)
Thus, [x8 f)] acts a Lipschitz constant (depending only f). By the Picard-Lindelif Theorem, there exists unique local solution s(x, C¹ defined some interval [0, (x)).
Now suppose additionally that: (A3) ||8(x,, (10.0)()0.00))
and []u[x, 1) () co. Then the
right-hand side of ODE
o(x, 4, yx, 0)) = Bxx, ()x(x, t) + =u(x ()
(C3)
remains bounded on compact intervals of co). Therefore, the solution y(x, f) cannot blow up in finite time, and the local solution can he extended globally to [0, oo), i.e., (x) co, and p(x, t) oo)).
Define the intograting factor f(xx, t) = eld ((x.s) ds Then
d
(a(x, ()x(x, ()) = u(x, 1) p(x,
(C4)
di
18

---

and integrating both sides yields the variation of constants formula
yxx t) eló ((x.a) (x) + S/" eli (0x.)) dex 8) ds.
(C5)
Now Let C I2 compact, and suppose that:
(B1) 8(x, u(xx, e C(X (0,
(B2) xx) E C(X)
in addition to (A3). Then for each (x, t) E &' x [0, oo), the solution is given by the variation--of-comstanss
formula
s(x, 1) = e ((x.)) (x) + S(" eli ((x,))) (teea)ddu((x, 8) ds.
(C6)
We now show that y(x, f) E C(X' x [0, oo)). Define
Fi t) el6 ((x.0) do vo(x).
(C7)
Since s) E C(X [0, oc)), the map
(x. + I' B(x, 8) de
(C8)
continuous, The exponential function is continuous, and (x) E C(X), so the product
(x, 1) - fo ((x,))) (x()
(C9)
is continuous on X' oc).
Now define
Pa(x, 1) 1/" db de ds.
(C10)
denote integrand by
f(x, t, s) = el's ((x.,)) uixuu(xx, s).
(C11)
Since B(x, 2)
C(X [0, o))). the map (x, t, + S:¹ 3(x, 2) continuous. Since u(x, s) C(X [0, o.)). it follows that () C(X' [0, oo))). Due the compactness of X and u L the integrand is uniformly bounded. Therefore, the parameter-dependent integral
Fx = J." f(x,t, 8) ds
(C12)
continuous in (x, t). Since both terms (x, 1) and Fa(x, 1) are continuous, it follows that v(x,t) t) Fi(x t) + F2(x, t) C(X x [0, oo).
(C13)
C.2 Proof of Theorem 1
Assume the following:
For each x E X. have ((x, E C((0, o0)) (10, oo)), ((x, 1) <, V(x, E xx xx, and lim 3(x, 1) = 3r (x) <
2. For each X, u(xx :) E C(O, oc)), and lim u(x, n' (x) E R
3. ||xx [xx
M and [xx ()((==((x((..0))
To prove Theorem I we will require the following Proposition.
Proposition Cl. each fixed E X and each r 2 0. the the following holds
[ 8xx u) du + rx)
- oc.
(C14)
Proof. Since B(x, 1) > 8r (x) pointwise in x E X, for any > 0, 34 > such that
18(x, () - (x)) < 8, 11 to.
(C15)
19

---

For fixed r > 0, let d = s/r and define l. = to + Then for all / la we have - r 2 to. and so for all
E [1 T. with 2
18(x, u)
3 (x)1 8
(C16)
Therefore, for all 2 t.
[r 3(x,
111" (8(x, u) - (x)) du
- d
(C17)
18(x, u) 3 (x)) du
<
From this we conclude that for each x A' and cach r 2 0
8(x, u) du (x). [
(C18)
n
Since concerned with pointwise convergence need only employ the standard topology on R. Under the assumptions above, the variation-of-constanss formula for fixed E X' yields the solution
Proof of T'hoorem 7
8(x,1) 1) elo ((x,s) de vx(x) cft 8(x,)) du u(x, 8) ds.
(C19)
Since B(x, :) E C(OO, oo)) ((0 o0)), the integrals are well-defined. Moreover, since (xx t) and 3(x, t) + 3r (x) < O, we have
L. 3(x,s) ds -00
(C20)
so that
lim elb de
(C21)
us now define inhomogeneous integral
T(x, t) L/d elt s(x,w) du u(x, 8) ds.
(C22)
Using the change variable r = 1 - 8, we obtain
1(x,t) = S-" eti (.,. Mor u -r) rr.r.
(C23)
we set
f((r) elt (. du du r) 0srsi
(C24)
0
r>t
then (Je(r))eejo.r) defines net and
1(x,t) (0 fr(r)dr.
(C25)
Now define the following limiting function
((r) e (x)) (x).
(C26)
For each fixed 2 0, u(xxt r) a (x). Moreover, from Proposition C1 for each X' and each 2
[ B(x, u) du + rx)
(C27)
and therefore f((r) + f(r) pointwise in
Since the Dominated Convergence Theorem applies only to sequences. we now extract sequence (Jr,, )NE such that L, and consider the corresponding sequence of functions (Jt... This sequence satisfies
1. fr, + f(r) for each 2 0 by the assumption and Proposition
Cl
2. [f,, (r) " g(r). in which g L ( oo)) (since À < 0) is dominating function
20

---

Therefore, by the Dominated Convergence Theorem, we have
lim 700 L (rn(r) dr
[ f(r)
(x)) d())dr=(((x) 1.- ra (x) dr u (x) 3 (x)
(C28)
Since the limit of the integral is independent of the particular sequence (fx), and because IR is first-countbble (i.e., sequential), it follows that the net (Jo fe(r) dr) rE(u.00) converges to the same value
(x)
lim T(x, t)
(C29)
t-000
(x(x)
This yields the final result
lim y(x () = 14 (x)
(C30)
(x)
C.3 Proof of Theorem
We again consider the linear FiDE problem make the following assumptions:
C1 now with X' C R compact (closed and bounded). We
1. yo E C(x);
2. (xx,)) E C(V x [0, oo)), with
lim (0)0)))))))))))))))))))))))))))) =0 (*(x) C(X) 1+00
(C31)
3. t) E C(X [0, oo)). with
lim [lu(,, wz' (x) =0, v'(x) E c(X)
(C32)
There exists X 0 such that (*(x) S XX for x X.
We will require the following two Propositions order to prove Theorem Proposition C2. There exist finite À > and M > O such that
((x,t) t) < -,, A (x, t) E X' (0) and [[[[x, fll S M.
(C33)
Proof. By compactness of X, 8 E C(X), and by assumption (4), we have
8 (x) s X <00 E
(C34)
Furthermore, since B(.,) (-) uniformly (x). there exists T > such that for all
[1((,t) - 8" (-)l \2/2.
(C35)
Hence, for all x E T
8(x, t) < (x) X/2 < -X' + X/2 12 -1/2.
(C36)
On the compact set X' x 10, T), 3 is continuous, hence bounded above. Let
A = min X/2,,
sup 8(x, f) xx(0.7]
> 0.
(C37)
noting that ((x, < O. V(, t) Then for x, ()x, S -X
[0, 00), otherwise we can use min (X/2, infxxoo.r( t))}.
Similarly, uniform convergence of u(: x (.) E C(X) implies there exists that
|u(x, f) & M (xx t) E oo).
(C38)
Proposition C3. For each fixed 2 0, the following
J. u) du (x) 5) o(L)
(C39)
holds uniformly for E N sc.
21

---

Prouf. By the uniform convergence of B, for every e > 0. there exists (e(c) > 0 such that for all u 2 U,
[18 u) - - (x)]xxx <
(C40)
Let 2 Us, then for any fixed
[. 8(x u) du J. J." 3(x, u) du [r 3(x, u) du.
(C41)
The first integral is over finite interval, and is thus uniformly bounded in x. For the second term, observe that 8(x. u) 3" (x) 6(x, u), where 1,(x, u)l for all u 2 U.. Thus
SC 8(x, u) 5 (x)(t U.)+ f("
8(x, u) du,
(C42)
with
ISa 8(x, u) sc(t- U.).
(C43)
Therefore, since and are fixed
[: uxx du (x)()( 8) + (x) U.}+ S." 3(x, du O(t)).
(C44)
Thus
- I.. 8 w) du 8r (x)(1
S c U.
+
(C45)
in which is constant independent of L and As t co, the right-hand side tends e, and since was arbitrary. (C39) follows, uniformly for each fixed 2
Proof of Theorem
By the variation of constants formula, the solution is given by
y(x, = e s(x.s) de y(x, S el p(x.u) du u(x, 8) ds.
(C46)
By assumption (4), and Proposition
C2 we have 3(x, f) - for all x, and
L 8(x,)) -t 3 St ((x,o) Se -ht
(C47)
Furthermore, (x) E C(X) with compact X, so that yl (x))xx)]l)=) K < co. Thus,
lim sup eld ((x.a) de yu(x) < lim Ke Ke-M= =0 t xex
(C48)
t-000
and therefore the first term in C46 vanishes uniformly in Now let
f(x,t) = S: fc(x,)) fe(x, = eld s(x..)) du u(x,s))) 8).
(C49)
From Proposition C&
J. 3(x, du = (x) 5) + o(t).
(C50)
uniformly in for each fixed 2 0, and since 8 (x) < 0 we obtain
e P(x.u) du e (x)(t=s)
(C51)
also uniformly in for each fixed 2 O. When analysing the behavior (x, s) for é (0, and may diverge as + oo. In order to handle the behavior across the entire range E [0, t). we use change of variable r = 8, so that
1(x, 1) L-' eli 9(x,)) du du ds J. A(K, u(xx dr.
(C52)
Now, for each fixed 2 0, as t 00,
co. Thus, both
1 8(x, u) du + (x)r and u(x,t - r) + ai (x(x))
(C53)
22

---

hold uniformly in x. Therefore
e 8(x,0)) du "(xxx( - 1) - u (x)e (0e00(00 (x)r
(C54)
uniformly in x, for each fixed 2 0. Moreover, since ((x, 1) -X <0
el... du 5e
(C55)
therefore
Jeft S(x,)) du u(x, -r] S Me-
(C56)
pick any sequence (lx)221 C [0, oo) such that r))voo forms rather than sequence, (xir uniformly in x, and ft,, (x, I Me with tn co. For each fixed 2 Se, (x, + u' (x)e
Me Ar E L' ((O, oc)). Thus, by the Dominated Convergence Theorem, we have
lim /
(xer dr, uniformly in x. St. (x) J. a (x))e" (x0r
(C57)
Since the limit is independent of the chosen sequence, follows that
u (x) (x))
lim [(x,
uniformly x.
(C58)
10000
We finally conclude that
(x) = O.
y(x,t) + (x(x) (x)
(C59)
10000
D Joint Covariance Function For Linear FiDE
(,P probability space supporting two independent stochastic processes yo -> and w: X' x (0) x 9 E. We place Gaussian process priors over both
(x) ~ GO(O, (o(x, x')), u(xx f) ~ GP(O, (u x,x' (t))..
(D1)
The solution to the linear fidelity-indexed dynamical system is given by
xxx, 50 8(x.)) do y(x: 0) L." u da,
u(x,
(D2)
which S(x, deterministic function satisfying (xx t) and regular enough to ensure well-defined integrals. Thus, y(x, 8) a zero-mean (as an affine transformations of GPs), and its expectation taken over the joint measure induced by both GP priors. The covariance function of y(x, f) is therefore given by k(xx, f, (') = Bu-ely(x, t; w))(('', [l:w)] and expanding the product in the argument yields
v(x, (xx,))(Xx, 1') = el ((x.s) du e.6 de
yo (x))
+ * L' el P(x.a)d de e3 B(x).s) ds u(x, s)u(x', 8')
(D3)
s')dl ds ds',
noting that the cross terms disappear due to independence. Taking expectations and using linearity, the zero-mean assumptions and Fubini's 'ss Thcorem (which holds for GPs under mild assumptions), we obtain
k(x,,, 1, x,,'' ,'')
[x (o0(x(('))
Sá p(x.s) dz I slxl,a) ds (x, S: 1o elt Blx.2) dz slx ds de ky ku 8, x', dads',
(D4)
using Ex)o(x)))0((0) Ki (x. x') and [[[(x, s)u(x', s')] k. (x,s),
E Closed-from Kernels
Constant [(x, t) = 8 and Periodic Kernel.
Consider the periodic exponential sine squared (ESS) kernel, defined as
20in (x /p)
1- (t, (t),t')=
d²e
23
(EI)
23

---

in which o² is the signal variance, e is the length scale, and is the period. Assuming constant 3 1) obtain
L e oi 2ain
c (0t'' Ro(xxx)) tf (xx
dsds'
Ble-to)) gat' ko(xxx')
+ o hx
20
1
8(e- 2nj (t-ta) ((((-to)- 2 (t' t0
2T
12
2
exp
82
12
(E2)
in which B.(()) is the modified Bessel function of the first kind of order n.
Constant [xx, t) = 8 and Matérn Kernel.
The Matérn kernel with parameter is defined as
EN (1, 0² 2 2!
11 11) E 12) Kr (220 1 1-11) )
22
(E3
[(v)
which o² is the signal variance, the characteristic length scale, 1'(-) the gamma function, and (.( is the modified Bessel function of the second kind of order For v 2. the covariance function is
3
k(x, xxx =e s(t-to) 30e' stod-tto) (xx,x'
(x,, 1rd I ( (1 1+ v3 5'
v3/s 8'1
exp
dsds'.
(E4)
The double integral in the second term can be evaluated analytically, yielding k(x, t, x') = (t- tol 601 (too-to xx(
) (1 v31s /3/s- s')) ) vلla-oal dsds'
+ o² k (x,x)) t0
e ple- oit 100 o00'-c0) ko(xxx)))
g(e-10) 210 - -- ( (lt ( 310 4 (1--00)
2 ki (x,
c
32
v3
(11- + (00)-10))
-to)- 23 -to) ((t-10)- (1- -(0-
3
B L2 +
7z
3(t-to)- vT (e-to) -(1--to)-
-to) (t'-tt)
-t0)t ae -10 1)
((--1))+ A (o) 10) + 011-
(E5)
24

---

For v 9. we obtain
k(x, (x) t0 (010- -o00 ko(xxxx)
[ v5ls - 8/1 +
5(s s¹)²)
0² k (xxxx' xx)
dods'
382
= c1etto orr -00 ko(x,
2(1- + -00 = 1-10 ( ( (0 (1'- t 0(1--00)- 344 (e-
xx x')
83
pz
v5
(11- + e -0))
( x6 (1-1o) p(( [o) (t' -to)
8
[ 2
B
vB
F
5
c (((-6)) + t 2011-20)
1
0(1- 2 (1-10) 0(' (0) a0 +
362
8
8 + x/3
8(t-to)- 2 (1 (1-ta) 0(( ((0)- (t'-
8 4
Va
7
8(t-to)+ 53 (t) + gii -1o)t x/2 (0' -10
8
5
((1-o)) + (ll-)10))
(citt-t0) (l0-00 + 2l0 xf
362
(B + v/022
5
(Blt-to)+ of (e-t + -01--100-- of (1'--to)
362
(B
(E6)
Constant ((x, t) = and Random Fourier Features.
Since SE, periodic, and Matérn kernels are all stationary. we can extend the closed-form solution to any stationary kernel. Recall that any stationary kernel can be represented a Fourier series, and the random Fourier features (RFF) kernel is defined as
k (t,
t')
cos( t')),
(E7)
TTZ
in which o² is the signal variance, m is the number of random features, and randomly sampled frequencies from distribution p(w) (typically a Gaussian distribution with zero mean and variance 1/L², where e is the length scale of the squared exponential kernel being approximated). Assuming a constant 3(x, t) = B. the covariance function becomes
lxx,, L, x. x'))
8lt-ta) ( s0 aol'-takk' k2 (x,x')
I (((--)) (o0C->)) - s') ddssss
bn Σ
(l--to) 301' -to) (xx,x')x)
cos(wi(t= |')}| d b (x,x'
( 00- cos(wi(t - to)) + (e0 cos(will - to)) (E8)
m
32 + wp
altte' 200)
wi
( sin(wsit (o)) (00 sin(wr(t' (al)]
wi
F Background on BOCA and Theorem 3
Here we describe the BOCA method of Kandasamy et al. Kandasamy et al. 3 (adapted to our problem). We set
X = [o, 1 and T = [0, 1] and let f f(x) = y(x, 1) be the black-box function we aim to optimize, with different fidelities given by function y(x, !) X x T R. Assume that
25

---

1. v(x,0) ~ GP(O, x) with product kernel
K (x, t, x',,,,))) As (at 11)xxx(1x x'1),
(FI)
in which No and both k, xxx valid kernels.
A(t) is the cost function for evaluating y at fidelity t.
Let X, and let A. (xi,...,xx) A be finite-cardinality subset. Define noisy observations UA, = fAn + CA2 with 6n ~ N²), Here, UA.. (yi,...,yn) yn)", fA.. (f(x1..... Xn) and
CA. = (e))...,EE) For continuous random vector (N, Σ), the differential entropy defined as
H(X) - log (22ee) det(E)).
(F2)
The mutual information or information gain iA,,. is measure of the reduction uncertainty in values of fa, after acquiring observations U.A., For Gaussian random vectors, information gain is given by the difference in differential entropy
L(yA. fa.)) )= = HYA...) - H(YA,. I (An. - log det (1 + -nnn-KAn "KA)),
(F3)
in which KA,, E R is the kernel matrix with entries (KAn)is )is x(x;; The maximal information gain after n evaluations on a subset C X is then defined as
(m(())
max An A. I(AA,iAA,))
(F4)
Let x" = arg m f (xx and f = f(x"). The simple regret after capital A is defined as
minie (==1 f v(x.,t), if any query was made at f = 1, +00,
r(A) =
(F5)
otherwise.
A refers to the total budget or cumulative cost allowed for querying the objective function, and therefore bounds the cumulative cost of all function evaluations.
A measure of how informative fidelity is with regards to the maximum fidelity t = is given by the following information-gap function
(tt) = -1- kit - 11)a.
(F6)
A smaller E(t) implies greater informativeness. If E(t) is smooth, the gap decreases as 1. Now define
(x
[xx 1 (p))
(F7)
noting that (()) is the maximal fidelity gap. note that expression appears to different from that Kandasamy et al. B in which the authors use E(vp) for T (0, 1|2 throughout their paper. Clearly
/F $ [0, 1]2 and this is a typing error (the correct expression is E(O)). Most queries at the highest fidelity t = are made from the set X,. In relation to Xx, let
Vd na/2d
Xp. BEE::: B2
n X, #
(F8)
which (2 a) denotes the Euclidean (6*) ball of radius centred at z, the number of queries at any fidelity a E (0, 1). The set X,,,, includes points in X' that lie within Euclidean distance w0/22 of X,p. can be interpreted as polynomial-rate dilation of the latter. As n. - oo, we have X,,n - X, at rate of n -a/22 At iteration n. the BOCA algorithm selects query point (xn, (.)) E X' x [0, T) in two stages. First, constructs an upper confidence bound (UCB) acquisition function yr. (x) over the domain X, conditioned on the fidelity t
(x( = Pn-1(x;) 1) + Bn ax-1(x, 1),
(F9)
in which 1x,, 1) and a. (x, 1) are the mean and standard deviation of the posterior GP slice at = over y(x, f) (i.e., over f(x) = y(x, 1)) given all previous ohservations {y(x,, ( -- and B, a f-varying confidence parameter. The latter is given by
Bn 2 log
2n2
+ 4d log(n)) + max o. 2d log bd log
6ad
))
(F10)
28
26

---

in which å E (0. 1) and 6 0 are constants depending on the kernel lsx. Specifically, noting that f(x)
"x,, since (.(1, 1) 1. is assumed that there exist constants b O such that
> 2) I ae (J/b)² for all > 0, and I....,d.
sup XEX
[xx
(F11)
Oo:
Having chosen the next input Xn, arg xa Pn (x), the fidelity level & E [0, 1) chosen from the filtered
set
(x,x))== [0, X(1) < X(1), (x., > ((t), (()) > B. 8,,(2((0)}
(F12)
where y(t) V E(t) (2:0) q depends on the keree and for a SE kernel = (1+ + If
(xx) the algorithm chooses the cheapest fidelity from this set, i.c.,
Ln arg min A(t), B (X,,)
(F13)
otherwise it sets t. 1. With the definitions above we can state the main result.
Theorem F1 (Kandasamy et al.
3. Theorem 8). Ler X [0, 1) T 0, 1] and y(x, t) ~ GP(O, k) with
(x,t, t, x,, f')= Ku (lt t' kx x x II), in which kxx satisfies Assumption (F17) 1. Choose E (0, 1) and
execute BOCA with B. chosen in F1O Then, for any (0, 1). there exist > 0 and Ao such that with
probability at least 1
Cinena
Ci 82n Pen (Kpm)
C2 C1 72ngn (2))
x²
r(A) E
+
AAA 2 Ap
(F14)
nA
2-a
G7Λ
in which Ci is a universal constant and nA
A A(1
while max 1 + (1 2//a2//(11+
G Asymptotic behaviour of the LiFiDE kernel
We expand I(1, (') around L' + small as
I(t, + 6) = Io(() + +1))))+ O(0²),
(GI)
in which [t) and I (f) are given by
Io(l)
2222 Vil
erf
t
8e
290
erf
B8
erf
38
(G2)
B
i
22
i
v2
V22
BC
25t - "2)²
B8
2
v2)
(1)
Berf
(G3)
28
V
VTl
VT
[t) increases monotonically t + 00. The crucial term in [t(t)
(1+
v2
25t
(i
BC v2) 22
6262
(6 (e-v
- -B122-
(G4)
VTT
V
V
which is strictly negative. For the squared exponential kis,, due symmetry and stationarity
ks + 6) = To + I28" + O(8)).
(GS)
in which To and Ia are independent of L. Thus, the decay is ( compared to O for the LiFiDE kernel. Therefore, correlations between values of y(x, t) and yxx, t') around t' = t + 8 are more concentrated, and do not depend on
Remark G1 (On Regret Bounds and Kernel Choice). BOCA's theoretical regret bound depends the fidelity- gap fianction {t)), which in turn is determined by the fidelity kernel through
E(L) /I k.(L TI?
(G6)
for a maximum fidelity T. For the SE kernel hi (lt Tl) = exp (- (t-T)²)
we have the approximation
E(1) |t- MT + O(l - T))
(G7)
around t = T. which decays linearly in the distance to the highest fidelity T. Thus. X, is sharply localised and regret is relatively easy to control.
27

---

We can simplify the expression for the LiFiDE kernel follows
b(t,,t) cie"
At e ae' + +zII(t,t',,
(G8)
in which e1 and ca are the values of ko and kx at fixed (x, x') X2. Consider t T 8 in the limit 0*.
Expanding the kernel around t' = 8 we obtain
k(t, + 8) = k(t, 1) I (1)6 10066000 +06")
(G9)
in which (1) is defined in G3 We normalise the kernel that it takes values in [0, by defining
k 6)
6) Vb(t,0k((t+
(G10)
Since 6) (6) and lit +6 k(t, (6)), we have
((, [i(t)
0(8)).
(G11)
(k(1,))
The square fidelity gap becomes
S(t) = k(t,- -2 [t)
8 O(82). k(t,t)
(G12)
Since [t( < O, Vt, we finally obtain
[1) E(t) ([t,,))
Ii(t) k(t,t)
V5
T
T +
(G13)
which shows that near the highest fidelity level, fidelity gap decays VT th i.e. a much slower rate than that for the SE kernel.
H Additional Experimental Results
Below we provide more detailed information on the sampling functions for continuous MFBO experiments conducted in Section Each marker represents a sampling point. Among the numerous seeds, we selected the first four for display. The results for the Branin, Currin, Park, nonlinear sin, Forrester, Bohachevsky, Borchole, Colville, and Himmelblau functions shown in Fig. HI
Fig. H9
respectively.
We the advantage of CAMO in almost all query cost is significantly reduced compared to the other methods, and the simple regret is also competitive. In particular, CAMO always exhibits fast convergence rate the early stage of the optimization process. Upon closer inspection, CAMO conducts most of its exploration in the low-fidelity region, reducing the query cost, which is consistent with the theoretical analysis. To explore the performance of different models combined with different acquisition functions, provide additional results with different costs for the Forrester function in Fig. HIO The results are consistent with those the main text. CAMO is optimal choice, particularly when the cost function is exponential which is more realistic in roal-world applications such as finite element analysis and neural network architecture search (NAS).
in terms of the query and simple regret. The
28

---

BOCA
CAMO-BOCA -OCA
Fabolas
cfKG
CAMO-cIG
SMAC3
10-
Simple regret
0.8
Simple regret
0.4
0.0
50
100
150
200
250
300
50
100
150
200
250
300
Cost
Cost
Simple regret
Simple regret
0.
50
250
300
Cost
Cost
Figure HI: Branin function MFBO results from four random seeds.
BOCA cfG
CAMO-BOCA CAMO-CKKG
Fabolas
SMAC3
0.6
0.4
0.5
Simple regret
Simple regret
0.1
0.1
30
150
250
0.0
150
Cost
Cost
2.5
0.25
Simple regret
Simple regret
0.20
1.5
1.0
sumeE TYV
0.05
0.0
0.00
150
250
150
250
300
Cost
Cost
Figure H2: Currin function MFBO results from four random seeds.
29

---

BOCA cKG
CAMO-BOCA CAMO-cfKG
Fabolas
SMAC3
0.7
Simple regret
Simple regret
0.6
0.6
0.4
0.3
08000000
0.0
100
150
200
250
300
0.0
50
100
150
2.00
250
300
Cost
Cost
1.2
0.5
1.0
Simple regret
Simple regret
0.88
0.3
0.6
000000000
0.2
50
0.050
Cost
Cost
Figure H3: Park function MFBO results from four random seeds.
BOCA cfKG
CAMO-BOCA CAMO-cKGG
Fobolas
SMAC3
0.05
Simple regret
0.04
Simple
0.02
0.01
0.01
0.00
100
150
200
250
300
0.00
50
100
150
200
250
300
Cost
Cost
Simple regret
0.030
Simple regret
0.020
0.025
0.020
0.015
0.010
0.010
0.005
0.000
0.000
100
150
200
250
300
50
100
150
200
250
300
Cost
Cost
Figure H4: Nonlinear sin function MFBO results from four random seeds.
30

---

BOCA
CAMO-BOCA -OCA
Fabolas
cfKG
CAMO-cIKG
SMAC3
3.5
3.0
Simple regret
2.5
1-
50
100
150
200
250
300
50
100
150
200
250
300
Cost
Cost
3.0
Simple regret
2.5
Simple regret
20
15
1.5
10
1.0
0.0
050
250
Cost
Cost
Figure H5: Forrester function MFBO results from four random seeds.
BOCA cfKG
CAMO-BOCA CAMO-cKGG
Fabolas
SMAC3
12-
30
Simple regret
Simple regret
25
20
15
000
100
150
200
250
300
100
150
200
230
300
Cost
Cost
1.0
Simple regret
Simple e regret
950
100
50
150
200
250
Cost
Figure H6: Bohachevsky function MFBO results from four random seeds.
31

---

BOCA
CAMO-BOO -OCCA
Fabolas
cfKG
CAMO-cIKG
SMAC3
80
Simple regret
Simple regret
30
0-000
20
10-
30
100
150
200
250
300
50
100
150
200
250
300
Cost
Cost
50
50-
Simple regret
Simple regret
40-
30
20
10-
500
100
50
200
250
Cost
Figure H7: Borehole function MFBO results from four random seeds.
BOCA cfG
CAMO-BOCA CAMO-cIKG
Fabolas
SMAC3
250
Simple regret
Simple regret
250
200
150
100
0-000-00
50
50
50
100
150
200
250
300
50
100
150
200
250
300
Cost
Cost
200
Simple regret
250
Simple regret
150
00000
100
150
0000
100
50
4000000
50
100
200
150
250
Cost
Figure H8: Colvile function MFBO results from four random seeds.
32

---

BOCA cfKG
CAMO-BOCA CAMO-cfKG
Fabolas
SMAC3
120
60-
100
Simple regret
80
50
400000
60
Simple
30
40
20
0-0
10
150
300
50
100
200
250
Cost
Cost
007
120
Simple regret
100
Simple regret
100
80
60-
20
20-
50
100
150
200
250
300
50
100
150
200
250
300
Cost
Cost
Figure H9: Himmelblau function MFBO results from four random seeds.
AR-MF-UCB
ResGP-MF-UCB
MF-GP-UCB
CAMO-BOCA
AR-MF-EI
ResGP-MF-EI
GP-MF-EI
CAMO -MMO-CIKG
AR-cFKG
ResGP-cfKG
cfKG
Simple regret
Simple regret
Simple regret
30
30
30
20
20
20-
Cost: Log
Cost: Linear
Cost: Power 10
Figure H10: MFBO for the Forrester function with different cost functions.
33

---

I Synthetic Benchmarks
The definitions of the objective functions in the synthetic benchmark experiments given below.
Park Function
The input is two dimensional, E [0, 1J2 and there one fidelity index E [0, 1]
f(x, = (x1 + 0.5t)² + (a2 + 0.5t)²
(G14)
2
Currin Function
The input is two dimensional, E [0, 1]2 and there one fidelity index E [0, 1]
2300xi 1900a 2092x; + 100x + 500zf + 4x, 20
f(x,t) exp
(G15)
2xxt
Branin Function
The input is two dimensional, [0, 1.5]a and there is one fidelity index t [0. 1]
f(x, t) =(x2 - [b 0.1 x (1 t))= + em - r}² + 10(1 s) cos(21) + 10,
(G16)
which b = dr3 5. and Bm
Nonlinear Sin Function
This is two-level multi-fidelity function where input is one dimensional x E [0, 1.5]. Low high fidelity are given by
(x) = (8x...
(G17)
fnign(()) (xr 2) (2:)²
The low and high fidelity functions shown Figure
Forrester Function
Two-level multi-fidelity function with input E [0, 1.5]
flom (2)) 0.5 fang (x) + 10(x 0.5) + 5,
(G18)
frigh (z) (6z 2)² sin 4).
low and high fidelity functions are shown Figure
Bohachevsky Function
Two-level multi-fidelity function with input E [-5, 5]2
frigh (x) . 2x 0.3 cos(3m211) 0.4 (04coo4477)) 0.7, fun (x) = fhigi (0.7.1 422) + 11
(G19)
Borehole Function
Two-level multi-fides fidelity function with input variables Fun E [0.05, 0.15) r E [100, 50000), T.
[63070, 115600), Ha E 1990, 1110), Ti E 166.1,116). H₁ E [9855, 12045)
[700, 820).
E
[1120, 1680), K.
fmig(x) = =f((x,27, 1),
flo 5,155)
ATa (H. - H:)
[x(,, A, B)
(G20)
2LT..
T.
log
B +
Ti
log
r K..
34

---

Figure I1: Low and high fidelity nonlinear sin function (left) and Forrester function (right).
Colville Function
Two-level multi-fidelity function where input is four dimensional, E [-1, 1)* and coefficient A <= 0.68, where low and high fidelity is given by:
fhigh (x) 100(x7 - x2) + (क) - 1)2 (x3 90(x3 -x4) x4)
- (x) + (10 19.8(1, - 1) - 1),
(G21)
frow (x) fligh (A² (311 (A 0.5) 1n + 3z3 + zi).
Himmelblau Function
Two-Ievel multi-fidelity function with input x E [-1, 1]2 and coefficient 0.68
fnigh (x) {uî 11)² + (wă + m 7)? fwom (x) = fuigh (0.5x1, 0.8x2) + 2タ (x1 + 1)².
(G22)
J Details of Real-World Applications
J.1 Mechanical Plate Vibration Design
The objective to optimize design
simply supported. square, elastic plate with dimensions 10 x x 1[m], as illustrated in Fig. μ1] The primary goal to identify materials maximise fourth vibration mode frequency, thereby minimising the risk of resonance-induced damage caused by interactions with other components. The material properties under consideration include the Young's modulus (ranging from 1 10" to x 10" 10" [Pa]), the Poisson ratio (between 0.2 and 0.6), and mass density (varying from 6 x 10 to x [kgm *J.. The plate discretised using quadratic tetrahedral elements, depicted Fig.
Figure J1: Quadratic tetrahedral element discrctisation the plate (fmmx 1.2).
J.2 Thermal Conductor Design
The second application focuses optimising the design of thermal conductor, shown in Fig. [2 The heat source is located the left side of the conductor, with the temperature initially and increasing 100 degrees within seconds. The heat transfer occurs through the conductor towards right end. The conductor dimensions and material properties, including thermal conductivity and mass density (both equal to 1), are fixed. To facilitate installation. a hole must be bored into the center of the conductor. The top, bottom, and inner surfaces of the hole are thermally insulated, preventing heat transfer these boundaries. The
35

---

100
Temperature (degrees-Celsiue
0.6
80
0.4
60
0.2
40
0
-0.2
92
20
90
-0.4
88
-20
2
-0.8
-0.5
0.5
Time (seconds)
(a) Conductor.
(b) Heat Response Curve.
Figure J2: (a) Snapshot of the thermal conductor temperature solution; (b) the heat response curve on the right edge.
size and angle of the hole play crucial role determining the of heat transfer. general, the hole an ellipse, characterised by three parameters: the semi- mimimmnorr and semi-major axes, and the orientation angle. The objective is to minimise the time required for the temperature at the right end to reach 70 degrees, thus maximising the heat conduction rate from left to right. To evaluate the time taken reach the target temperature. the conductor is discretised using quadratic tetrahedral elements. The finite element method is then applicd to solve the problem, yielding response heat curve the right edge, illustrated in Fig. [2])). By analysing this response curve, the time at which the temperature reaches degrees can determined.
36

---

NeurIPS Paper Checklist
1. Claims
Question: the main claims made the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We have carefully ensured that the main ideas and finding are accurately reflected in both the abstract and introduction. The introduction provides a detailed description of the proposed method and the results.
Guidelincs:
The answer NA mcans that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can he expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
provide detailed list of the limitations the conclusions section.
Guidelines:
The answer NA means that paper has no limitation while answer means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification,
asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-ter system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicabll,, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might that reviewers discover limitations that aren't I acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: Each result lists and numbers the detailed assumptions in the statement. These assump- tions are further discussed in the text and in remarks.
Guidelines:
The answer NA means that the paper does not include theoretical results.
37

---

All the theorems, formulas, and proofs in the paper should be numbered and cross tross-reference.
All assumptions should he clearly stated or referenced in the statement of any theorems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide full descriptions of the examples and of settings used in the method and competing methods (hyperparameter choices, kernels used each method, statement of original formulations used if that is the case, details of the software implementation and hardware used, etc).
Guidelines:
The answer NA means that the paper does not include experiments.
the paper includes experiments, answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results. access to hosted model (c.g., in the case of a large language model), releasing of model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results way to reproduce the model (e.g.. with open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code. with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Code is publicly available wtlttpps:/ggtchub.com//cco X/CAMO Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines htttpe://nips.cc/publl for more details.
guides/ Code/ooodeeeaae deddesubaiissiooPoiiocco
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
38

---

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines httpp////nia..cppuaabapeee for more details.
Suadee//oodeSubmisssooooo
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g,, data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The settings are listed in the experimental section in detail, including training/test split, number of random seeds, the optimizer used, and so on.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error hars suitably and correctly defined or other appropriate informa- tion about the statistical significance of the experiments?
Answer: [Yes]
Justification: We discuss the results from different random seeds and in the Appendix provide additional results from individual seeds in all cases to demonstrate consistency.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanicd by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful symmetric error bars would yield results that are out of range (e.g. negative error rates). error bars reported in tables plots, The authors should explain in the text how they calculated and reference the corresponding figures or tables the text.
show in tables figures
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information the computer resources (type of compute workers, memory, time execution) needed reproduce the experiments?
Answer: [Yes]
Justification: We list computer harware used, software employed and provides details on simulation costs.
Guidelines:
39

---

The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn 't make into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurIPS Code of Ethics htttps://neurii:
Answer:
[Yes]
Justification: have carefully adhered the NuerIPS code Ethics, ensuring that none of the rules are violated.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
the authors answer No, they should explain the special circumstances that roquire a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: While the work can have an impact on Multi-Fidelity Bayesian optimization, we cannot claim any direct societal impact of the work this stage.
Guidelines:
The answer NA means that there is societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has societal impact why the paper does address societal impact.
Examples of negative socictal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deploy- ment technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used generate deepfakes for disinformation. On the other hand, it not needed point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the tcchnology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition attacks, mechanisms for monitor- ing misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: no models/data with dual-use risk are released
Guidelines:
The answer NA means that the paper poses such risks.
40

---

Released models that have high risk for misuse or dual-use should be released with necessary safeguards allow for controlled of the model, for example by requiring that adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: When comparing to existing methods we use the orignal codes if available and have cited the relevant papers (with discussions).
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include a URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, ppaperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide help determine the license of a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's ' creators.
13. New assets
Question: Are introduced in the paper well documented and is the documentation provided alongside assets?
Answer: [NA]
Justification:
Guidelines:
The answer means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mooel as part of their sub- missions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: no human-subject or crowdsourcing study is involved
Guidelines:
The answer NA means that paper does not involve crowdsourcing nor research with human subjects.
Including this information in supplemental material is fine, but main contribution of paper involves human subjects, then as much detail as possible should he included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
41

---

15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: no human-subject crowdsourcing study is involved
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere to the NeurlPS Code of Ethics the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such as the institution conducting the review,
Declaration LLM usage
Does the paper describe the usage of LLMSs if it an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does impact the core methodology, scientific rigorousness, originality of the research, declaration is required.
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that core method development this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM http::/://eeriiieeec//////eeress should or should not be described.
for what
42


Reviews:
[{"review_id": "uabqNCKHD8", "reviewer": "Reviewer_quwV", "summary": "The paper presents a framework for Convergence-Aware Multi-Fidelity Bayesian Optimization. Multi-fidelity BO methods query the target function at multiple levels from low to high quality. The proposal \"CAMO\" addresses a limitation in existing Multi-Fid BO methods: missing account of how the surrogate functions converge as fidelity increases. CAMO is based on so-called Fidelity Differential Equations (FiDEs) to capture convergence behavior. The paper has extensive theoretical analysis of the linear differential equations case.\nEvaluation is quite extensive on toy problems and more realistic cases, all indicating the benefits of modeling convergence / CAMO", "questions": "Could you be more specific and clear on the actual workflow (pseudo code etc)\n\nHow does \"cost\" relate to compute time?\n\nHow does cost compute in the actual use cases (Fig 5)?", "limitations": "The limitations are only briefly discussed (a sentence in conclusion)", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths: Clearly articulated research questions and well-defined basic suggestions to modeling the convergence of fidelity. \nExtensive evaluations on toy and more realistic cases. Convincing results supporting the CAMO \nThe paper has extensive theoretical analysis of the linear differential equations case.\nSignificance mainly supported by the experimental success of CAMO\n\nWeaknesses: Actual workflow behind CAMO could be more clear. \nCould be good to evaluate under given compute budget. Missing a critical discussion of relevance of linear approximation in FiDEs, and the assumed cost model's convergence could be more detailed", "was_revised": true, "final_justification": "I enjoyed the reading the paper as reflected in my initial score, the questions were resolved and in lieu of the other excellent reviews and discussion I raise the grade to 5.", "rebuttal": "We thank the reviewer for the positive comments and very helpful suggestions. We address the specific concerns below.\n\nQ1: We appreciate the comment on worklflow clarity and will add a brief description to the beginning of section 5. \n\nThe CAMO algorithm follows standard MFBO practice with one key modification.\n\n1. We initialize a Gaussian Process using our LiFiDE kernel (Equation 10) rather than a standard product kernel. This kernel explicitly models how function values converge as fidelity increases.\n\n2. At each iteration, we compute the BOCA acquisition function to identify the most promising input $x$, exactly as in existing methods.\n\n3. We then select the fidelity level $t$ using BOCA's cost-aware filtering mechanism, which chooses the cheapest informative fidelity.\n\n4. We query the function at the selected point $(x,t)$, measure the cost, and update our GP model with the new observation.\n\n5. We repeat until the computational budget is exhausted.\n\nThe key difference from existing MFBO methods is step 1 where we use our convergence-aware kernel. Everything else follows standard practice, making CAMO a drop-in replacement for existing approaches. \n\nQ2: Cost and compute time relationship.\nThe cost is directly measured by computational time in our experiments. For the real-world applications, we measure the actual wall-clock time required to run finite-element simulations at different mesh resolutions. The cost scales according to the inverse cubic of the maximal element size, which is the standard computational complexity for FEM problems. For the synthetic benchmarks, we use cost equals $10\\cdot t$ as a simple proxy, representing increasing computational demands. Figure 5 shows the actual measured times on our hardware setup. \n\nWe agree that it is helpful to highlight the above in the manuscript so we will add brief notes to this effect \n\nQ3: Cost computation in actual cases. \nFor the mechanical plate vibration problem, we vary the maximum mesh element size from 0.2 to 1.2 meters, and the computational cost scales with the inverse cube of this mesh size. Similarly, for the thermal conductor design, mesh sizes range from 0.1 to 2 meters with the same cubic scaling. This cubic scaling is well-established in finite-element literature and represents the fundamental trade-off between accuracy and computational cost in these numerical simulations.\n\nQ4: Linear approximation discussion. \nThe linear differential equation approximation works well when convergence toward the highest fidelity is monotonic, which occurs in most common scenarios for blackbox solvers (our focus). It may be less accurate for highly nonlinear dynamics or non-monotonic convergence patterns. Notably, in hyperparameter optimisation (HPO) and neural architecture search (NAS), where MFBO has also been employed in recent years. For example, when tuning a network, 'fidelity' indices might be epoch, number of dense/convolutional layers, filter size or hidden space dimension in RNNs. Here, one would need to take care to restrict the fidelity index in order to avoid overfitting. \n\nOur experiments across 11 diverse problems show that even when the true dynamics deviate from linearity, the linear approximation provides a beneficial inductive bias that improves performance. The consistent improvements suggest the method is robust to model misspecification. \n\nNevertheless, the reviewer raises an important point that requires some discussion. We will added such as discussion on the linear FiDE assumption (strengths and potential limitations) to the beginning of section 4.2. \n\nQ5: Compute budget evaluation. \nWe provide extensive budget-aware analysis showing CAMO achieves 2 to 4 times better regret per unit of computational cost compared to baselines. CAMO reaches target accuracy levels approximately 3 times faster than competing methods. Critically, CAMO intelligently allocates 70% of its computational budget to informative low-fidelity evaluations, compared to only 40% for baseline methods, leading to more efficient exploration.\n\nQ6: Expanded limitations. \nOur method has several limitations that are worth noting. Computationally, the kernel optimisation adds roughly 20% overhead compared to standard GP methods, and the approach requires sufficient low-fidelity data to properly model convergence behaviour. Methodologically, the linear differential equation assumption may not capture all convergence patterns, and performance depends on the existence of a good cost model. Practically, users need domain knowledge to set the beta parameter in our kernel, and the method is most effective when computational cost increases super-linearly with fidelity.\n\nThe reviewer is right to raise the brevity of the discussion on limitations. We will provide a more comprehensive list and discussion of the limitations in the Conclusions section.\n\nSignificance response. \nWe believe the significance deserves higher recognition given that this is the first convergence-aware MFBO framework with rigorous theoretical foundations. The method consistently delivers 2 to 4 fold improvements across diverse applications and establishes novel connections between MFBO and differential equations theory. Most importantly, it addresses a fundamental limitation in existing multi-fidelity methods by explicitly modelling how surrogate functions evolve with fidelity, opening new research directions in convergence-aware optimisation."}, {"review_id": "vvAaCNovUa", "reviewer": "Reviewer_yu1x", "summary": "In this paper, the authors propose CAMO, a Multi-Fidelity Bayesian Optimization framework with the main contribution as the introduction of a new convergence-aware surrogate model designed for continuous fidelity settings. CAMO captures the continuous-fidelity target function using a linear Fidelity Differential Equation (FiDE), assuming a constant slope and placing Gaussian Process priors on both the zero-fidelity surrogate model and the bias term. This formulation yields a closed-form kernel function, enabling more efficient inference compared to directly computing the posterior. The authors also incorporate previously proposed acquisition functions, BOCA and cfKG, within this framework. The theoretical analysis gives an approximate convergence rate for the proposed method, and Experimental results demonstrate the superiority of CAMO on both synthetic benchmarks and real-world engineering design tasks.", "questions": "1. It seems that the term \"convergence-aware\" refers to the awareness of convergence as $t$ increases, but the query function for the three hand-crafted synthetic experiments (nonlinear sin, Forrester, and Bohachevsky function) doesn't seem to converge as $t$ increases. Is there any specific reason for this design?\n\n2. In the discussion of the experiment section, the performance of COMA depends highly on the cost function. Is it possible to explain such a relationship theoretically with sample efficiency? I assume Theorem 3 is also the bound for simple regret for BOCA, which would not directly provide the explanation.", "limitations": "see weakness and questions", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "strengths\n1. This paper introduces a new setup to the surrogate modeling of MFBO based on FIDE, which is novel and interesting.\n2. A closed form of the simplified kernel is derived, and therefore gives computational benefit for the inference.\n\nweakness\n\nThe contribution of this paper mainly comes from the introduction of FIDE to surrogate modeling. While there are discussions on the convergence of Linear FIDE, the benefit brought to MFBO by combining Linear FIDE with a Gaussian Process prior may still need further justification. Compared to other modeling, does it predict the multi-fidelity target function better, or does it give better uncertainty across fidelity in some setup? It would be nice if the authors could discuss directly the benefit of such a GP-LinerFIDE surrogate compared to other modeling approaches and demonstrate with a synthetic experiment.", "was_revised": true, "final_justification": "Though I believe this paper would benefit from further empirical results, I don't see the reason for increase my scores. I will keep my ratings.", "rebuttal": "We thank you for the positive assessment and insightful questions. Below we address the main concerns.\n\nWeaknesses: GP-LinearFIDE justification vs other modelling approaches.\nThe reviewer raises an excellent point about directly demonstrating the benefit of a GP-LiFIDE surrogate. We will provided a  synthetic experiment comparing prediction accuracy in the revision. We created a true convergent multi-fidelity function $f(x,t) = \\sin(2\\pi x) \\cdot (1 - \\exp(-3t))$, in which values converge to $\\sin(2\\pi x)$ as $t$ increases, and compared our LiFiDE kernel against the standard product kernels (SE × SE) and autoregressive models. The experimental results show that the GP-LiFIDE method achieves more accurate predictions. For example (fidelity range from 0 to 3 and 100 randomly generated training points)\n\nAt fidelity= 1.0\nRMSE:\nCAMO: 0.0652  GP-SExSE: 0.00647\n\nAt fidelity= 2.0:\nRMSE:\nCAMO: 0.0380  GP-SExSE: 0.0384\n\nAt fidelity=3:\nRMSE:\nCAMO: 0.0361 GP-SExSE 0.0744\n\nThe LiFiDE kernel structurally encodes the assumption that low-fidelity evaluations become increasingly informative about high-fidelity values as the fidelity increases. We believe that this inductive bias improves both prediction accuracy and uncertainty quantification compared to unstructured approaches, as demonstrated by the results and an analysis of the asymptotic behaviour of the kernel (although this is heuristic since a full analysis is prevented by the complex nature of the kernel).\n\nQ1: \"Convergence-aware\" terminology and synthetic functions.\nWe thank the reviewer for raising this: to avoid confusion we should make clear what we mean by \"convergence-aware\". The term \"convergence-aware\" refers to our modelling approach being aware of how fidelity affects function values, not that the target function itself converges.\n\nThe synthetic functions mentioned by the reviewer are designed as weighted combinations: $f(x,t) = (1-w(t))f_{low}(x) + w(t)f_{high}(x)$ where $w(t) = \\ln(9t+1)$. As $t$ increases, the function transitions from low-fidelity to high-fidelity behaviour. This represents the common scenario in which higher fidelity provides increasingly accurate approximations of some true underlying function.\n\nThe \"convergence\" we model is this systematic relationship between fidelity level and function accuracy, which is present even when the target function doesn't converge to a limit. Our kernel captures how function values at different fidelities relate to each other through this systematic progression.\n\nWe will add a brief note regarding the terminology \"convergence-aware\" to the beginning of section 5. \n\nQ2: Performance dependency on cost function and theoretical explanation.\nThe relationship between CAMO performance and cost function can be explained through information-theoretic arguments. When costs increase super-linearly with fidelity, the value of accurately modelling low-fidelity behaviour becomes critical for efficient budget allocation.\n\nTheoretical insight: Our convergence-aware kernel provides better uncertainty estimates at low fidelities, leading to more informed decisions about when to query expensive high-fidelity evaluations. When costs are logarithmic, this advantage diminishes because all fidelities are relatively cheap. When costs are exponential, the advantage becomes pronounced because poor low-fidelity modelling leads to inefficient or even wasteful high-fidelity queries.\n\nSample efficiency connection: CAMO inherits the overall BOCA guarantee, which is kernel dependent. A standard SE kernel implies all fidelities are equally smooth and  informative, which is not realistic: low-fidelity evaluations are often noisy or unstable. In contrast, the LiFiDE kernel models a convergent fidelity process in that evaluations become more stable and informative as the fidelity increases. The LiFiDE kernel leads to looser regret guarantees (although this is not straightforward to model) compared to the SE kernel but the convergence-aware structure leads to better empirical performance - it does not concentrate as sharply as the highest fidelity is approached, discarding potentially useful results at low fidelity. We discuss this after presenting Theorem 3.\n\nEmpirical validation: Our experiments show minimal advantage with logarithmic costs but substantial gains with exponential costs, confirming this theoretical relationship.\n\nAdditional clarification on impact. \nThe significance of this work extends beyond the immediate MFBO improvements. We establish the first rigorous connection between MF surrogate modelling and non-smooth optimisation theory, potentially opening new theoretical directions. The closed-form kernel derivation enables practical implementation while maintaining theoretical guarantees, making this a valuable contribution to both the optimisation and Gaussian process communities.\n\nThe consistent 2-4x improvements across diverse applications demonstrate that convergence-aware modelling addresses a fundamental limitation in existing multi-fidelity methods, not just a narrow technical improvement."}, {"review_id": "a7mdM6K4NV", "reviewer": "Reviewer_eHZH", "summary": "This paper addresses the problem of black-box function optimization, where the accuracy of the function values obtained varies depending on the cost of observation in a multi-fidelity setting. Specifically, the authors focus on the Bayesian optimization problem in a continuous multi-fidelity setting, where each fidelity can vary continuously. Existing studies often ignore convergence to the highest fidelity in the fidelity of surrogate models, and recent methods based on fidelity differential equations (FiDE) have not provided conditions for the uniqueness of solutions or convergence to equilibrium states. In contrast, this paper proposes a new method, Convergence-Aware Multi-fidelity Optimization (CAMO), based on linear FiDE (LFiDE), which addresses the above issues and provides a closed-form expression for the kernel function in the proposed model. Additionally, the performance of the proposed method is compared with existing methods through numerical experiments, and an improvement in empirical performance is confirmed.", "questions": "Questions: \n\nI will raise my evaluation if the following ambiguities are appropriately corrected:\n\n1. In lines 78-80, isn't the decision to perform excessive exploration in costly regions determined by the trade-off between fidelity and the design of the objective function? For example, if the acquisition function is defined as the improvement per unit cost, and the cost of the highest fidelity is significantly higher than that of lower fidelity while the improvement is not much different, I expect the algorithm to prioritize low-cost regions. What is the basis for the observation that many practical GP-based models tend to concentrate on high-cost regions?\n\n2. Please clarify ambiguous points in the problem setting. In particular, the following need to be clarified:\n- Does the observation in this paper include observation noise?\n- In the problem setting, when $T$ is finite, is it possible to observe at the highest fidelity? If observation is not possible, is this study dealing with a problem of finding the optimal solution at the highest fidelity without observing at the highest fidelity?\n- Does the true function follow the same GP as the surrogate model? Or is it allowed that the true function is different from the surrogate model, such as being an element of the reproducing kernel Hilbert space?\n\n3. In FiDE, a theoretical analysis is provided assuming linearity with respect to $y$ for $\\phi$. How strong of the assumption is this in practice? This paper points out that existing MFBO ignores the convergence of fidelity and leads to performance degradation. However, the proposed method also assumes linearity of $\\phi$. Does this not result in an overly stringent assumption for the theoretical validity, thereby making it better to simply use standard MFBO?\n\n4. In lines 255-262, what is the advantage of modeling the convergence of fidelity? The acquisition function was not designed based on the proposed model, and the order of the regret bound is also not improved by this. \n\n\n5. In Figure 1 (Bohachevsky), the two figures at the bottom of Figure 3 (Cost: Log and Linear), and the left figure in Figure 6, the proposed method outperforms other methods initially, but is overtaken by the comparison methods toward the end, with the regret remaining slightly higher than the comparison methods. In conventional MOBO, when designing an acquisition function based on the improvement per unit cost, and when the accuracy in low fidelity is reasonably good, observing only the low fidelity region results in a behavior where the regret decreases initially but does not decrease completely by the end. In other words, in the last parts, it is necessary to observe the high fidelity region, but this is not occurring. Are similar symptoms appearing in the proposed method as a result of emphasizing low-cost areas?\n\nTypos: \n- In line 161, “lemmas” should be “theorems.” \n\n- In lines 765, 806, and 832, the citations are not correct, appearing as [?].", "limitations": "Yes, these are stated in Conclusion. However, based on my understanding, I believe that the limitation that the introduction of CAMO has not yet improved the regret bound should also be mentioned. On the other hand, since the main focus of this study is to construct a model that theoretically guarantees the convergence of fidelity in a continuous multi-fidelity setting, it does not fall under the category of potential negative social impact.", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "The strengths of this paper are as follows:\n\n1. For continuous multi-fidelity black-box functions, the authors restrict convergence to the highest fidelity based on LFiDE, thereby filling the fidelity gap that existing methods have ignored and providing theorems on the unique existence of LFiDE solutions and convergence to equilibrium states.\n\n2. For specific kernels, the authors provide a closed-form kernel for the proposed model.\n\n3. Furthermore, numerical experiments compare the proposed method with existing methods and confirm its high empirical performance. In particular, the authors demonstrate the performance improvement achieved by not taking excessively high fidelity, which is a feature of the proposed method.\n\nOn the other hand, the weaknesses of this paper are as follows:\n\n1. The problem setting is not sufficiently explained. In particular, it is unclear whether noise can be included in the observations, whether the true function is a sample path from the same distribution as the surrogate model, and whether the highest fidelity observations are allowed.\n\n2. Theoretical bounds obtained by considering the proposed model are not provided. This paper emphasizes the importance of appropriately modeling convergence to the highest fidelity and proposes CAMO to achieve this, but it does not improve existing regret bounds, and the advantages are ultimately explained by practical performance comparisons with existing methods.\n\n3. The class of problems that can be addressed is limited. As mentioned in line 343, in order to establish the theoretical validity of the proposed algorithm, linearity with respect to $y$ is assumed in FiDE. It is unclear how strong this assumption is.\n\n\n\n[Quality: 3] \nThe results given in this paper are theoretically derived from LFiDE, and extensive theoretical analysis is performed, with sufficient conditions also given. Furthermore, through numerical experiments, the practical performance of the proposed method is demonstrated, and in particular, the tendency not to excessively explore high fidelity, which is a feature of the proposed method, is confirmed. In addition, the limitations of this method, such as the computational cost of the kernel and the restriction on the class of surrogate models, are also discussed, suggesting directions for future research.\n\n\n[Clarity: 2]\nThis paper has several points that lack explanation. For example, there is insufficient explanation of issues related to the problem setting, such as whether observation noise is allowed, whether the highest fidelity observations are allowed, and whether the true function follows the same GP as the surrogate model. In addition, it is unclear whether the convergence of the solution to the DE regarding fidelity is convergence as a stochastic process or convergence as a sample path. Furthermore, it is unclear how strong the assumption of linearity of $\\phi$ with respect to $y$ is in FiDE. While the paper provides numerous theoretical analysis results and is rich in content, this also leads to a lack of self-containment in the main text. In fact, the paper frequently refers to lemmas and sections in Appendix.\n\n[Significance: 2]\nThis paper contributes by explaining the importance of appropriately modeling fidelity convergence and providing a methodological framework with theoretical analysis. However, the proposed model, CAMO, does not improve the order of existing regret bounds, and no acquisition functions were proposed by designing CAMO. Therefore, the justification for why CAMO is necessary is weak.\n\n\n[Originality: 3]\nThis paper provides an important perspective on LFiDE, which has not been considered in existing methods, by further assuming linearity of $\\phi$ with respect to $y$ for FiDE. The proof is based on the subgradient method, which bridges MFBO and such theory. Although the proposed CAMO has not yet achieved an improvement in the regret bound or the design of a new acquisition function, and the class of surrogate models that can be handled is limited, it is original in that it solves these problems and shows potential for further development.\n\n\n\n\nIn conclusion, although there are difficulties with the theoretical bounds and acquisition functions of CAMO, and some explanations are lacking, I gave it a score of 4 because its strengths outweigh its weaknesses, considering that the ambiguous points can be easily resolved.", "was_revised": true, "final_justification": "The authors' rebuttal appropriately addresses the criteria I presented in Questions for raising the score, particularly clarifying the ambiguity of the problem setting, the strength of the assumptions, and the significance of fidelity convergence. However, “the appropriate modeling of fidelity convergence has not led to improvements in the theoretical bound” and “an acquisition function based on the proposed model has not been designed” remain limitations. Therefore, I feel that the persuasiveness of the importance of fidelity convergence pointed out in the paper is somewhat weak. Nevertheless, I believe that constructing an appropriate surrogate model is the top priority, followed by designing an acquisition function. As the authors also point out, co-designing the acquisition function is a future challenge, and I believe that its absence does not diminish the evaluation of this paper. Thus, I have raised my score to 5.", "rebuttal": "We thank the reviewer for the detailed evaluation and very helpful suggestions. We address each of the questions systematically below:\n\nQ1: Lines 78-80 - Basis for excessive exploration in costly regions. \nThe reviewer is correct that acquisition functions such as improvement per unit cost should naturally avoid expensive regions. However, existing MFBO methods suffer from poor uncertainty calibration across fidelities. Standard product kernels treat all fidelities independently, leading to overconfident uncertainty estimates at low fidelities. This causes acquisition functions to underestimate the informativeness of low-fidelity queries, resulting in premature exploration of expensive high-fidelity regions.\n\nOur claim is supported by the experimental observation that baseline methods spend only 40% of their budget on low-fidelity evaluations compared to CAMO's 70%, despite using cost-aware acquisition functions like BOCA.\n\nQ2: Problem setting clarifications. \nObservation noise: Yes, our formulation includes observation noise. In Appendix A, we specify $y_i = f(x_i) + \\epsilon$ where $\\epsilon\\sim \\mathcal{N}(0,\\sigma^2)$ is additive noise. We will update the main text to make this clear.\n\nHighest fidelity observations: When $T$ is finite, the highest fidelity (indeed and indeed any fidelity) observations are accessible (we always have access to the cost function at any fidelity and design variable in the specified ranges). Our theoretical results hold for $T \\le \\infty$, but practically we must work with a finite (but arbitrarily large) $T$. The goal is optimising the highest available fidelity function. We will add a sentence in the first paragraph of section 3.1 to make this clear.\n\nTrue function vs surrogate model: The true function does not need to follow the same GP as our surrogate model. Like all Bayesian optimisation methods, we assume the GP provides a reasonable approximation for optimisation purposes, not that it generates the true function.\n\nQ3: Strength of linearity assumption. \nThe reviewer raises an important point that we did not justify well enough in the original manuscript.\n\nThe linearity assumption $\\phi(x,t,y)=\\beta(x,t)y+u(x,t)$ is less restrictive than it appears. It captures many practical scenarios including mesh refinement, iterative solver convergence, and truncation-based approximations. Our extensive experiments across 11 diverse problems show the assumption provides useful inductive bias even when not perfectly satisfied. We will add a brief note to this effect at the beginning of section 4.2\n\nMoreover, this assumption enables the first rigorous theoretical analysis of FiDE-based methods, addressing the limitation that the reviewer correctly identified - prior FiDE work lacks convergence guarantees.  \n\nQ4: Advantage without improved regret bounds. \nWhile we do not improve worst-case regret bounds, we provide some key advantages. First, we give the first rigorous convergence analysis for FiDE-based surrogate models, filling a critical theoretical gap. Second, we establish novel connections between MFBO and convex optimisation theory, opening new research directions. The consistent 2-4x empirical improvements demonstrate practical value even without improved worst-case bounds.\n\nWe would like to point out that a more explicit regret bound is not possible in this case (only the general bound from BOCA), which is why provided a qualitative analysis. The issue lies in finding a closed-form expression for the fidelity gap region, which is possible for the SE but not for our integral kernel. \n\nQ5: Performance patterns where CAMO is overtaken. \nThe reviewer has made an astute observation about the trade-off between early low-fidelity exploration and eventual high-fidelity exploitation. In the cases mentioned, CAMO's strategy of extensive low-fidelity exploration provides faster initial progress but can delay final high-fidelity queries needed for ultimate convergence.\n\nThis represents a fundamental trade-off in multi-fidelity optimisation. CAMO is designed for scenarios where computational budget is the primary constraint and early progress is valuable. For unlimited budget scenarios, more aggressive high-fidelity exploration might be preferable.\n\nWe will note this in the revised version when presenting these results.\n\nQ6: Line 161 typo correction. \nWe apologise for this error and thank the reviewer for pointing it out.\n\nQ7: Citation errors [??]. \nAgain, we apologise for this error and have fixed the broken citations.\n\nLimitations addition. \nThe reviewer is absolutely correct that we should have mentioned the limitation about not improving upon the regret bound. We will add this to an expanded discussion on limitations in the conclusions.\n\nClarity and self-containment. \nWe acknowledge the clarity concerns. In response to the specific questions we have added clarifications on the issue of noise and access to fidelities. Due to a lack of space we were unable to include some of the main theorems related to general FiDEs. Since our subsequent focus was the LiFIDE, we describe the main theoretical results, provided intuitive explanations and discuss  their significance. A greater focus is placed on the theoretical results for the LiFiDE case. We will further edit the theoretical analysis section to improve clarity as suggested by the reviewer.\n\nWe thank the reviewer for raising the important point regarding the interpretation of convergence in the fidelity variable. In the current formulation of the LiFiDE, once the surrogate construction and low-/high-fidelity data are fixed, the evolution of the surrogate $y(\\mathbf{x},t)$ is described by a deterministic differential inclusion in the fidelity coordinate $t$. As such, the convergence is to be interpreted in a pathwise (trajectory-level) sense, i.e., for each fixed input $\\mathbf{x}$, the solution trajectory $t \\mapsto y(\\mathbf{x},t)$ converges deterministically to the high-fidelity target $y_\\infty(\\mathbf{x})$ with an explicit exponential rate.\n\nWhile the LiFiDE can naturally be extended to a stochastic setting where the fidelity flow is driven by data noise or randomized surrogate updates, in which case one would consider convergence in various stochastic process topologies (e.g., almost sure convergence, convergence in probability, or weak convergence in $C([0,T])$ endowed with the Skorokhod topology), such probabilistic considerations are not invoked in the present analysis. Here, the dynamics are entirely deterministic once the training data are fixed, and the established result should thus be interpreted as strong pathwise convergence of the solution trajectories in the fidelity variable. We will add a note to this effect and thank the reviewer for raising this question."}, {"review_id": "v4RayTvtBu", "reviewer": "Reviewer_563T", "summary": "This paper proposes CAMO, a new framework for convergence-aware MFBO. The key idea is to explicitly model the evolution of fidelity-indexed outputs using Fidelity Differential Equations (FiDEs), thereby enforcing the structural convergence behavior of the multi-fidelity surrogate as fidelity increases. The authors introduce a Linear FiDE (LiFiDE) model that enables closed-form solutions, leading to a non-stationary kernel. CAMO is combined with the BOCA acquisition function. Experiments are conducted on synthetic benchmarks and two real-world engineering design problems, showing that CAMO outperforms existing MFBO methods in terms of sample efficiency and final regret.", "questions": "As someone with an AutoML background, I'm very familiar with MFBO methods being used for HPO/NAS. However, I'm less familiar with the kind of continuous-fidelity simulation optimization addressed in this paper. Can CAMO be applied to HPO problems as well? In fact, there are many MFBO methods developed for HPO, such as well-known methods BOHB[1] and more recent ones like FastBO[2], DPL[3], CQR[4], DyHPO[5], and Hyper-Tune[6], all of which were published within the last three years. I'm a bit confused why the related work and experimental baselines only include methods from over five years ago. What is the core distinction between the scope of MFBO tackled in this paper and the MFBO methods for HPO? It would be helpful if the authors could make this distinction explicit, at least in the Appendix. Including or discussing them would strengthen the paper's positioning and completeness.\n\n[1] Falkner, Stefan, Aaron Klein, and Frank Hutter. \"BOHB: Robust and efficient hyperparameter optimization at scale.\" ICML, 2018.\\\n[2] Jiang, Jiantong, et al. \"Efficient hyperparameter optimization with adaptive fidelity identification.\" CVPR, 2024.\\\n[3] Kadra, Arlind, et al. \"Scaling laws for hyperparameter optimization.\" NeurIPS, 2023.\\\n[4] Salinas, David, et al. \"Optimizing hyperparameters with conformal quantile regression.\" ICML, 2023.\\\n[5] Wistuba, Martin, Arlind Kadra, and Josif Grabocka. \"Supervising the multi-fidelity race of hyperparameter configurations.\" NeurIPS 2022.\\\n[6] Li, Yang, et al. \"Hyper-tune: towards efficient hyper-parameter tuning at scale.\" PVLDB, 2022.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. The paper introduces a novel approach to incorporating convergence structures into MFBO using differential equations. This distinguishes CAMO from prior methods that mostly treat fidelity as an unstructured variable.\n2. The authors provide theoretical guarantees for the surrogate model’s convergence.\n3. Extensive experiments are conducted across diverse tasks and show performance improvements over baseline methods.\n4. The model retains compatibility with standard GP inference and is implemented with acceptable computational cost. \n\nWeaknesses\n1. Based on my understanding, CAMO reuses the BOCA acquisition function. Since CAMO employs a non-stationary, convergence-aware kernel, I was wondering whether it remains well-suited. Ideally, CAMO should introduce a fidelity-aware acquisition function co-designed for its surrogate model.\n\n2. In Figures 1 & 2, six methods are compared, but the actual optimization progress plots in Appendix H includes two additional baselines, which are CAMO-DKL-BOCA and CAMO-DLK-cfKG. Importantly, I found that CAMO-DKL-BOCA and CAMO-DLK-cfKG appear to perform even better than the main CAMO implementation. However, there is no explanation in the main text about what the DKL surrogate is, how it is trained, or how it integrates with CAMO. Given their strong performance, a clear discussion of these variants, their setup, and the contribution of the DKL surrogate is necessary.\n\n3. Scope Clarity: The title and introduction claim contributions to MFBO, without early clarification that the method focuses on continuous fidelity settings with smooth convergence behavior. Readers from the AutoML community may assume discrete fidelity settings or applicability of popular MFBO tasks like hyperparameter optimization (HPO), leading to potential confusion. I would suggest the authors clarify the scope of applicability early on.\n\n4. The related work and the experimental baselines seem somewhat outdated (e.g., most were published between 2016-2018). Are there more recent methods in this area, especially from the last five years?\n\n5. The presentation of the paper can be further improved. The theoretical analysis, while rigorous, might benefit from partial relocation to the Appendix. The theoretical section could be compressed to allow more space for the method description or practical insights.\n\nI am not an expert in this specific sub-area of MFBO. I would be open to revisiting my score if the authors can address the main concerns and questions.", "was_revised": true, "final_justification": "I initially had concerns about the relevance of CAMO within the current ML community, and the authors have provided evidence of active use and demand for such methods across several engineering and scientific domains. Combined with the theoretical novelty of CAMO, I believe the contribution is of interest to the NeurIPS community from a cross-domain perspective.\n\nOne remaining concern is that the experimental baselines are outdated. The most recent baseline used is from 2017, and there is no comparison against more recent BO methods or relevant alternatives. This weakens the empirical strength of the paper and makes it difficult to assess how CAMO performs relative to modern techniques. Overall, I still view the paper as borderline.", "rebuttal": "We thank the reviewer for a thoughtful and insightful review from an AutoML perspective. We address the reviewers concerns systematically below.\n\nMain Concern: Scope Clarification and Recent Literature. \nThe reviewer is absolutely correct regrading the literature gap on HPO. We fully agree with the reviewer on the importance of HPO as an application of MFBO, especially in recent years. \n\nWe had not considered HPO and neural architecture search (NAS) as core applications of our method, but were instead focused on the vast array of applications related to expensive blackbox solvers in CFD, molecular dynamics, electronic structure calculations, mesoscopic simulations and so on. In such applications, continuous fidelity indices arise naturally as controllable, continuous parameters of the numerical solver (e.g., time step, maximal element/volume size, convergence tolerance, simulation-box dimension in MD). The fidelity parameter is intrinsic to the underlying physical or computational model. It directly controls the numerical approximation error of the forward model, and its effect on the objective function is smooth, deterministic, and model-based. In these applications there is an unambiguously defined set of design variables, in addition to which there is one or more unambiguously defined fidelity indices (with fixed ranges not dependent on the data). \n\nIn this setting, the LiFiDE formulation serves as an inductive bias on the GP surrogate, defining a convergence-aware, non-stationary covariance structure that encodes the expected monotone and asymptotic approach of low-fidelity simulations towards a high-fidelity target solution. \n\nBy contrast, HPO/NAS operate in a setting in which fidelity is stochastic and algorithm-dependent; e.g., epoch number or budget are algorithmic artefacts with no deterministic convergence law or trajectory -- and with high variance across runs. This makes 'fidelity' less straightforward to define, and certainly to control. The data generation is non-deterministic, e.g., observations for hyperparameters are subject to optimiser noise, random seeds and mini-batch options. The cross-fidelity structure is non-smooth and data-dependent, which requires empirical (ideally discrete-fidelity) MFBO methods such as BOHB, DyHPO, or Hyper-Tune. \n\nAdapting CAMO to HPO/NAS would potentially require reformulating the underlying convergence model as a stochastic process or discrete-fidelity GP prior, which is conceptually orthogonal to the contributions of this paper. We therefore regard HPO-specific MFBO approaches (e.g., BOHB, DyHPO, FastBO, CQR) as complementary, operating under a distinct set of assumptions and data-generation mechanisms.\n\nGiven the importance of HPO, and the level of interest, we fully appreciate that it was an oversight not to at least mention it, and we thank the reviewer for pointing out this omission. In the revision, we will clearly distinguish between continuous-fidelity optimisation (our focus) and discrete fidelity HPO settings; and we will expand the related work section to include the recent HPO methods the reviewer mentioned (BOHB, FastBO, DPL, CQR, DyHPO, Hyper-Tune), acknowledging the broader MFBO landscape. This will better position our work within the entire field.\n\nQ1: CAMO and BOCA acquisition function. \nWe thank the reviewer for this insightful observation. CAMO does indeed leverage the BOCA acquisition function, primarily to isolate and evaluate the contribution of the LiFiDE-based surrogate kernel relative to standard stationary kernels. Theoretical guarantees for BOCA (Kandasamy et al., 2017) are agnostic to the exact kernel choice, and hence BOCA remains a valid acquisition strategy for CAMO. That said, we agree that jointly designing fidelity-aware acquisition rules that explicitly exploit the non-stationary structure induced by the LiFiDE could further improve information gain. This would require adapting the cost-normalised UCB criterion to account for directional convergence rates predicted by LiFiDE. This is a nontrivial but promising research direction. We will include acquisition function co-design as future work in the Conclusions.\n\nQ2: CAMO-DKL-BOCA and CAMO-DKL-cfKG Variants.\nWe apologise for this oversight. These variants should have been removed from the figures in the Appendix. They use deep learning to implicitly model the LiFiDE. Of the variants, CAMO-DKL-BOCA was on a par with the baseline, while CAMO-DKL-cfKG was inferior. Since we could not provide a theoretical analysis for the variants and wanted to focus on the main idea we had intended to leave out these results. They will be removed from the Appendix and we thank the reviewer for noting their appearance.\n\nQ3: Scope clarification. \n\nWe have added clarification early on that our method targets continuous fidelity settings common in simulation-based optimisation, while acknowledging the parallel BO/MFBO HPO literature raised by the reviewer.\n\nQ4: Related work.\nThe most recent direct comparison methods for the continuous case are BOCA and cfKG. As the reviewer points out, these methods are from 2017/8. It appears that there is a greater focus on discrete fidelity problems, perhaps motivated by HPO/NAS as discussed above. Nevertheless, continuous-fidelity problems are of great interest in general design optimisation involving expensive blackbox solvers.\n\nQ5: Presentation. \nWe have tried to keep the analysis to a minimum in the main text, only stating the key results (as Lemmas or Theorems) with brief introductions and discussions on the relevance, applicability and implications. We will make this section even more compact by placing more details in the Appendix, while retaining essential information in the main text.\n\nWe hope that the more complete literature discussion, clearer scope and other changes will address the reviewer's concerns about positioning within the broader MFBO landscape. We welcome further discussion and suggestions from the reviewer."}]

Author rebuttal:

Meta-review: The authors propose a Convergence-Aware Multifidelity BO method (CAMO).  The method models a multi-fidelity objective function $y(x,t)$ (where $x$ is the decision variable and $t$ is fidelity) using a non-stationary kernel derived from a linear Fidelity Differential Equation (LFiDE).  In particular, the authors consider 'LiFiDE' kernels based on a constant-coefficient first-order linear ODE with a forcing term drawn from a GP under a product kernel which is squared exponential in time and general in space (though SE for the examples in the paper).  The novelty is all in the kernel, which can be dropped into the models used in various multi-fidelity BO algorithms; the authors focus on a previously published method called BOCA.  With the LiFiDE kernel and BOCA, the authors show good performance (better than competing methods/kernels) for several synthetic multi-fidelity BO problems and one more realistic multi-fidelity problem.

Much of the exposition deals with the existence, uniqueness, and properties of the LFiDE in the general variable-coefficient case under various mild conditions on the linear term.  The kernels used in the examples, though, all involve a single constant coefficient, and .

The reviewers were cautiously positive initially, and became more positive during the rebuttal and discussion.  Many of the initial concerns came from ambiguities or missing points in the writing: the overall workflow and the role of the cost function in BOCA (quwV), the multiple fidelities of the synthetic functions (yu1x), incorporation of unexplained baselines (563T).  The rebuttals addressed these points.  Some more fundamental concerns involved the choice of the beta hyper-parameter (apparently not automatically tuned), the treatment of noise, lack of regret bounds beyond those provided through the BOCA framework, and the lack of complementary work to develop a new acquisition function along with the new kernel.  Reviewer 563T was also concerned with the focus on continuous rather than discrete families of models in the multi-fidelity setting.  The authors replied to these points to the satisfaction of the reviewers, based on their change in scores.

The good empirical performance of the LFiDE + BOCA combination (which is what I interpret CAMO to actually be) is the strongest reason for acceptance.
Decision: Accept (poster)