Title: ContinuAR: Continuous Autoregression For Infinite-Fidelity Fusion
Venue: NeurIPS 2023
Accepted: yes
Keywords: Gaussian process

autoregression

multi fidelity

nonparametric Bayesian
Abstract: Multi-fidelity fusion has become an important surrogate technique, which provides insights into expensive computer simulations and effectively improves decision-making, e.g., optimization, with less computational cost. Multi-fidelity fusion is much more computationally efficient compared to traditional single-fidelity surrogates. Despite the fast advancement of multi-fidelity fusion techniques, they lack a systematic framework to make use of the fidelity indicator, deal with high-dimensional and arbitrary data structure, and scale well to infinite-fidelity problems. In this work, we first generalize the popular autoregression (AR) to derive a novel linear fidelity differential equation (FiDE), paving the way to tractable infinite-fidelity fusion. We generalize FiDE to a high-dimensional system, which also provides a unifying framework to seemly bridge the gap between many multi- and single-fidelity GP-based models. We then propose ContinuAR, a rank-1 approximation solution to FiDEs, which is tractable to train, compatible with arbitrary multi-fidelity data structure, linearly scalable to the output dimension, and most importantly, delivers consistent SOTA performance with a significant margin over the baseline methods. Compared to the SOTA infinite-fidelity fusion, IFC, ContinuAR achieves up to 4x improvement in accuracy and 62,500x speedup in training time.

ContinuAR: Continuous Autoregression For Infinite-Fidelity Fusion

Abstract

Multi-fidelity fusion has become an important surrogate technique, which provides insights into expensive computer simulations and effectively improves decision-making, e.g., optimization, with less computational cost. Multi-fidelity fusion is much more computationally efficient compared to traditional singlefidelity surrogates. Despite the fast advancement of multi-fidelity fusion techniques, they lack a systematic framework to make use of the fidelity indicator, deal with high-dimensional and arbitrary data structure, and scale well to infinitefidelity problems. In this work, we first generalize the popular autoregression (AR) to derive a novel linear fidelity differential equation (FiDE), paving the way to tractable infinite-fidelity fusion. We generalize FiDE to a high-dimensional system, which also provides a unifying framework to seemly bridge the gap between many multi- and single-fidelity GP-based models. We then propose ContinuAR, a rank-1 approximation solution to FiDEs, which is tractable to train, compatible with arbitrary multi-fidelity data structure, linearly scalable to the output dimension, and most importantly, delivers consistent SOTA performance with a significant margin over the baseline methods. Compared to the SOTA infinite-fidelity fusion, IFC, ContinuAR achieves up to 4x improvement in accuracy and 62,500x speedup in training time.

1 Introduction

Contemporary scientific and engineering endeavors depend significantly on analyzing highly complex systems, where the repeated execution of large-scale differential equation numerical simulations, often with intricate interconnections, is essential. For instance, in the design of a systemon-chip (SoC), more than 80% of the design time is spent on analysis based on different types of simulations, e.g., timing analysis and yield optimization. Such an intense computational demand prompts the use of a data-driven surrogate model, which essentially acts as a functional approximation for the input-output mapping of a simulation. To enhance convergence efficiency, especially in scenarios involving recurrent simulations, such as those encountered in Bayesian optimization (BO) [1] and uncertainty quantification (UQ) [2].

Zheng Xing

Graphics& Computing Department Rockchip Electronics Co., Ltd Fuzhou, China, 350003. zheng.xing@rock-chips.com

Traditional surrogates are trained on many high-fidelity simulation results, which are still computationally expensive to generate. To make high-fidelity predictions [3] while further reducing the computational burden, it is a possible way to combine low-fidelity results. More specifically, we can run low-fidelity simulations based on simplified equations (e.g., reducing the levels of physical detail) or coarse solver setups (e.g., using a coarse mesh, a large time step, a lower order of approximating basis, and a higher error tolerance) to generate cheap but inaccurate results, which offer a chance to train a rough model with low cost. The multi-fidelity fusion techniques then improve such a rough model by using only a few high-fidelity simulation samples. In total, the computational cost is dramatically reduced. Multi-fidelity fusion, owing to its efficiency, has garnered growing interest in BO [4, 5], UQ [6], and surrogate modeling [7].

While many state-of-the-art (SOTA) fusion approaches have been rapidly developing, most focus on improving the model accuracy or scalability for large-scale problems. They normally assume a small number of fidelities (say, less than five), and a particular fidelity data structure (i.e., the output space is well aligned and the high-fidelity samples' corresponding inputs must form a subset of the low-fidelity inputs). However, in practice, a natural multi-fidelity problem can be much more complicated. For instance, the data does not admit any particular structure. Furthermore, the number of fidelities is in general countably infinite, very few works actually utilize the fact that the fidelity levels are implicitly quantified by a continuous variable, e.g., the number of nodes in a mesh and there are infinite number of fidelities. These limitations seriously hinder the applications of multifidelity-based methods, e.g., multi-fidelity BO and multi-fidelity Monte Carlo.

Recently, Li et al. [8] propose the first infinite-fidelity fusion, IFC, which utilizes NeuralODE [9] to resolve these challenges. Despite its success, IFC is difficult to train and scale poorly to highdimensional problems. To make a further step towards practicality while preserving tractability and accuracy, we propose the first tractable infinite-fidelity fusion framework, linear fidelity differential equations (FiDEs), and its rank-1 solution, ContinuAR, to deliver a powerful yet tractable fusion model. The novelty of this work is as follows,

We propose the first linear fidelity differential equation (FiDE) and its general solution, paving the way to tractable infinite-fidelity fusion.

We extend FiDE to FiDEs, which handle the common high-dimensional simulation problems. Furthermore, FiDEs bridge the gap between multi-fidelity and single-fidelity surrogates and serve as a unifying framework for many existing GP-based surrogates, revealing some future directions for classic surrogate models (a.k.a emulators) with multi-fidelity fusion.

We propose ContinuAR, a tractable and efficient rank-1 solution to FiDEs. It is scalable to infinite fidelity, capable of handling high-dimension problems, compatible with arbitrary multi-fidelity data structure, and delivering SOTA accuracy with low computational cost.

2 Backgronud

2.1 Statement of the problem

Provided with multi-fidelity dataset D i , where i = 0 , . . . , T , each set comprising entries { t i , x ( i ) n , y ( i ) n } N i n =1 , with t i representing the fidelity indicator (e.g., the number of nodes in a mesh generation), where a larger t indicates a more accurate solution; x ( i ) ∈ R Q denotes the system inputs (e.g., a vector containing parameters found in the system of equations or initial-boundary conditions for a simulation), where Q is the dimensionality for x ; y ( i ) ∈ R D indicates the vectorized outputs associated with x , where D is the dimensionality for y ; T is the total number of fidelities. In general, higher fidelity simulations, being closer to the ground truth and more costly to acquire, result in a limited number of available samples, i.e., N 0 > N 1 > · · · > N T . In most works, e.g., [10-12], the system inputs of higher-fidelity are chosen to be the subset of the lowerfidelity, i.e., X T ⊂ · · · ⊂ X 2 ⊂ X 1 . We call this the subset structure for a multi-fidelity dataset, as opposed to arbitrary data structures, which we will resolve in Section 4. Our objective is to estimate the function y ( T ) ( x ) based on observations at various fidelities {D i } T i =1 .

2.2 Autoregression (AR)

The classic AR model [3] only considers the scalar problem (i.e., y ( i ) ∈ R ) and imposes a Markov property for its multi-fidelity formulation. Considering that t 0 and t T are indicators (e.g., time steps) for the low- and high-fidelity data, respectively, AR defines

where ρ is a factor transferring knowledge from the low-fidelity in a linear fashion, while u ( x ) aims to encapsulate the remaining information. Assuming a zero mean Gaussian process (GP) prior [13] for y (0) ( x ) and u ( x ) , denoted as y (0) ( x ) ∼ N (0 , k 0 ( x , x ′ )) and u ( x ) ∼ N (0 , k u ( x , x ′ )) , the high-fidelity function similarly adheres to a GP. This results in an elegant joint GP applicable to the combined observations y = [ y (0) ; y ( T ) ] ,

where [ K (0) 0 ] ij = k 0 ( x (0) i , x (0) j ) ; [ K (0 T ) 0 ] ij = k 0 ( x (0) i , x ( T ) j ) ; [ K ( T 0) 0 ] ij = k 0 ( x ( T ) i , x (0) j ) ; [ K ( T ) 0 ] ij = k 0 ( x ( T ) i , x ( T ) j ) ; [ K ( T ) u ] ij = k u ( x ( T ) i , x ( T ) j ) . We can see that the joint likelihood is still Gaussian, which admits a tractable solution for model training as in a standard GP. Furthermore, the predictive posterior is also a standard GP posterior, which can effectively utilize lowand high-fidelity data. Moreover, by decomposing the likelihood and predictive posterior into two separate components, Le Gratiet [10] managed to reduce the complexity from O (( N 0 + N T ) 3 ) to O (( N 0 ) 3 +( N T ) 3 ) using a subset data structure, where X T ⊂ X 0 .

2.3 Fidelity Differential Equation

In scientific numerical simulations, the fidelity can be determined by quite different factors, e.g., using simplified/complete equations to describe the target system or implementing a dense/sparse mesh to discretize the domain. For problems where the fidelity factor cannot be easily quantified, the classic multi-fidelity methods, e.g., AR, and other methods should work just fine. However, it is more frequent to find that the fidelity indicators have valuable information in them. For instance, the fidelity is often indicated by the number of elements in a finite element simulation or time steps in a forward/backward Euler timing scheme. In these cases, we should treat the fidelity indicators as continuous variables and utilize the information they carry to boost the predictive accuracy. To this end, Li et al. [8] propose a general formulation of infinite-fidelity models,

Although this formulation is general enough to cover all multi-fidelity models, it lacks an insightful interpretation-it is not clear how to design the function φ ( x , t, y ( x , t )) . A clever workaround is to use a neural network to approximate φ ( x , t, y ( x , t )) [8], which introduces many challenges such as 1) requiring a large amount of training data, 2) expensive computational cost for backpropagation even with adjoint method, and 3) instability as we can see how easy a simple nonlinear system can lead to chaotic behaviors (e.g., the Lorenz system [14]).

3 Proposed Method

3.1 Linear Fidelity Differential Equation

To remedy the over-general formulation of Eq. (3), we first revisit the classic AR and reveal its connection to a linear ODE's forward Euler solution. Rewrite Eq. (1) as

Proposition 1. The linear fidelity differential equation (FiDE):

and take the limit ( t T -t 0 ) → 0 , we derive Proposition 1 (see Appendix B for detailed derivations).

If we take a forward Euler solution to solve Eq. (5) discretized at each fidelity, we recover the classic AR in Eq. (1). Let us stick to the continuous formulation and utilize calculus techniques [15], we can derive the general solution to Eq. (5), where C ( x ) is a function of x , and B ( t ) = ∫ t β ( τ )d τ is the antiderivative of β ( t ) . In order to design β ( t ) such that the solution converges to the ground truth as t → ∞ , a detailed stability analysis is necessitated. General stability considerations for systems of this kind are often complex and may warrant the application of Lyapunov's direct method. For the sake of maintaining a tractable model, we make the simplifying assumption that β ( t ) = β is a constant value. Provided that β > 0 , our system is guaranteed to converge to what we define as the "ultra-high-fidelity ground truth" as

t →∞ . Setting t 0 as the lowest-fidelity indicator, we derive a general solution (see Appendix C)

To design a subtle model with enough flexibility while trying to maintain tractability, we place zero mean GP priors for y ( x , t 0 ) and u ( x , t ) (with all data being normalized), i.e.,

Similarly to the latent force model [16], the resulting general solution y ( x , t ) is a also GP with zero mean and covariance with the analytical form:

Alvarez et al. [16] demonstrate that a simple ODE system, devoid of x , yields a nonstationary output covariance with a closed-form solution when employing a squared-exponential (SE) kernel for u ( t ) . This result, however, is not directly applicable to the FiDE due to the inherent x dependency.

To improve model flexibility, we enable the usage of arbitrary kernels, e.g., deep kernel [17], by implementing a Monte-Carlo integration and reparameterization trick [18] to conduct efficient backpropagation for model training-the integral approximated by 1 M ∑ M i,j e -β ( t i + t j -2 t 0 ) k u ( x , τ i , x ′ , τ j ) , where t i and t j are M are random samples from [ t 0 , t ] and [ t 0 , t ′ ] , respectively.

3.2 FiDEs: Multi-Variate Extension For High-dimensional Problems

In practice, the outputs of a simulation are generally high-dimensional [19], i.e., y ( t ) ∈ R D . We hereby generalize Eq. (5) for a more general formulation that describes a multi-variate system:

Proposition 2. The linear fidelity differential equations (FiDEs) for a multi-variate system:

where B ( t ) ∈ R D × D and S ∈ R D × R are affine transformations; u ( x , t ) ∈ R R is a source function.

We can derive a general solution for Eq. (10) with some calculus as model design guideline (see Appendix D). For instance, for a constant B , all eigenvalues of B must have positive real parts to keep the system stable. To derive an efficient model, we define a constant diagonal matrix B and derive a tractable general solution

with B d being the d -diagonal element of B . Due to the linearity of the integral in G d ( u r ( x , t )) , we can derive the output correlation with an analytical form:

where

We recognize that this model is a generalization of semiparametric latent factor model (SLFM) [20], which is a special case by setting t = t 0 or B d = 0 to consider a single-fidelity problem.

3.3 ContinuAR: A Rank-1 Solution to FiDEs

A special yet practical case of SLFM is the intrinsic model of coregionalization (IMC, also a rank-1 approximation to LMC [21]), where all u r ( x , t ) share the same kernel function, i.e.,

Here ⊗ means the Kronecker product. Similar IMC assumptions are made in ResGP, a popular modification of AR for high-dimensional output fusion [12], which shows promising results in

many physics applications. Based on their conclusions and our intention to keep our model simple and efficient, we define B = β I following our solution to FiDE and place an IMC model for the lowest-fidelity model, i.e., y ( x , t 0 ) ∼ N ( 0 , k 0 ( x , x ′ ) ⊗ H ) , where H is the output correlations. Substituting the new u ( x , t ) and B back into Eq. (13), we get the output correlation k y ( x , t, x ′ , t ′ ) = Cov[ y ( x , t ) , y ( x ′ , t ′ )] =

which uniquely defines a rank-1 solution to FiDEs. We call it ContinuAR.

Lemma 1. Autokrigeability in ContinuAR: the particular values of the spatial correlation matrix H and SS /latticetop do not matter in the predictive mean as they will be canceled out.

The proof is given in Appendix E for clarity by basically deriving the predictive mean of ContinuAR. Since the predictive mean is the main concern in high-dimensional problems [8, 12], we simply set H = SS /latticetop = I to significantly improve model efficiency without introducing any additional error in the mean predictions. The computational complexity w.r.t the output dimension is reduced from O ( D 3 ) to O ( D ) and the memory consumption is reduced from O ( D 2 ) to O (1) .

4 Efficient Training and Inference Through Subset Decomposition

We have implicitly marginalized out the underlying function u ( x , t ) based on its GP prior and derived the output covariance of Eq. (16). Given a set of observations Y = [ Y (0) ; Y (1) ; . . . ; Y ( T ) ] , we have the joint distribution where /vector y (0) = vec ( Y (0) ) is the vectorization; [ K ( kl ) ] ij = k y ( x i , t k , x j , t l ) is the shorthand notation of output correlation Eq. (16). For a small number of total training data of all fidelity, we can simply opt for a maximum likelihood estimation (MLE) for the joint likelihood

where Σ is the whole covariance matrix in Eq. (17) and /vector y = [ y (0) , . . . , y ( T ) ] /latticetop . However, this approach will soon become invalid, particularly in a multi-fidelity scenario where we expect many low-fidelity simulations.

Since the integration of Eq. (16) can be done by parts, K ( kl ) admits an additive structure exactly as in AR. We can follow Gratiet and Cannamela [22] to decompose the joint likelihood Eq. (18) into independent components provided that corresponding inputs strictly follow a subset structure, i.e., X T ⊂ · · · ⊂ X 2 ⊂ X 1 . For problems with only a small number of fidelity (a small T ), the subset structure may not be too difficult to satisfy. However, for the infinite (countable) fidelity setting, such a requirement is not practical. Here, we derive a decomposition by introducing virtual observations ˆ Y for each fidelity such that Y ( T ) satisfies the subset requirement for the completed set { Y ( T -1) , ˆ Y ( T -1) } . ˇ Y ( T ) is the part of Y ( T ) that forms the subset of Y ( T -1) (with a selection formulation ˇ X ( T ) = E ( T ) X ( T -1) , where X ( T -1) corresponds to the previous-fidelity outputs Y ( T -1) ). The joint likelihood can then be decomposed,

is the updated additive kernel with the uncertainty of the predictive variance ˆ Σ ( T ) of the virtual points, whose computation details are given later in Section 4.1;

is the additive/residual part of the kernel from ( T -1) to T ;

is the additive part for the completed outputs from ( T -1) to T ; ∆ T is the time interval between t T -1 and t T ; log p ( Y (0: T -1) ) is the log likelihood for the previous T fidelities, which is obtained by calling Eq. (19) recursively. The detailed derivation is preserved in Appendix I due to the space limitation. Through the decomposition of Eq. (19), the computation complexity is reduced from O (( D ∑ T i =0 N t i ) 3 ) to O ( D ∑ T i =1 ( N t i + N t i -1 -| X ( t i ) ∩ X ( t i -1 ) | n ) 3 ) . Here, | · | n indicates the number of samples. Furthermore, when data at a new fidelity is obtained, we only need to modify the joint likelihood slightly by adding new blocks, which will be handy for active learning or Bayesian optimization. Model training is conducted easily by maximizing the joint likelihood Eq. (19) with respect to the hyperparameters using gradient-based optimizations.

4.1 Predictive Posterior

Since the joint model (17) is a Gaussian, the predictive posterior for the highest fidelity can be derived as in standard GP with a large covariance matrix that requires inversion for once. Similar to the derivation of an efficient joint likelihood in the previous section, we drive an efficient predictive posterior y ( x ∗ , T ) ∼ N (¯ y ( x ∗ , T ) , Σ ( x ∗ , T )) (see Appendix H), where

with two selection matrixes that follow:

In these equations, [ ˜ k ( TT ) a ∗ ] j = k y ( x ∗ , T, x j , T ) -k y ( x ∗ , T -1 , x j , T -1) for x j ∈ { X ( t T ) } is the kernel additional part between T and T -1 for x ∗ and X ( t T ) ; ˜ Σ ( t T ) is the predictive variance based on { X ( T -1) , ˆ X ( T ) } of T fidelity model; [ ˆ k ( TT ) a ∗ ] j = k y ( x ∗ , T, x j , T ) -k y ( x ∗ , T -1 , x j , T -1) for x j ∈ { ˆ X ( t T ) } is the kernel additional part between between T and T -1 for x ∗ and ˆ X ( t T ) ; ˆ Σ ( t T ) is the predictive variance based on { ˆ X ( T ) } .

5 Related Work

As stated, FiDEs serve as a unifying framework that unifies many SOTA multi-fidelity and single-fidelity surrogates. We made connections to some popular models in Table 1.

Multi-Variate GPs are commonly used surrogates that can be recovered by setting t = Const or T = 0 for the FiDEs, which gives us the fundamental semiparametric latent factor models (SLFM) [20]. SLFM is a simplified Linear model of coregionalization (LMC) [23, 29]. Based on SLFM, intrinsic model of coregionalization (IMC) reduces the computational complexity by setting a rank-1 approxi-

Model

Assumptions under FiDEs

SLFM [20]

T

= 0

;

IMC [23]

T

= 0

;

u

(

x

)

∼

N

(

0

,

k

u

(

x

,

x

′

)

⊗

I

)

HOGP [24]

T

= 0

; R=1;

[

SS

/latticetop

]

ij

=

k

d

(

z

i

,

z

j

)

GPRN [25]

T

= 0

,

S

GP

(

x

)

AR [3]

D

= 1;

B

(

t

) =

β

(

t

i

)

ResGP [12]

R

= 1

,

B

(

t

) = 0

,

S

=

I

,

NAR [11]

B

(

t

) = 0

,

G

(

u

(

x

, t

))

GP

(

x

,

y

, t

i

)

,

S

=

I

MF-BNN [26]

B

(

t

) = 0

,

G

(

u

(

x

, t

))

BNN

(

x

,

y

, t

i

)

,

S

=

I

SC [27]

B

(

t

) = 0

,

G

(

u

(

x

, t

))

PCE

(

x

,

y

, t

i

)

,

S

=

I

DC [28]

B

(

t

) = 0

,

G

(

u

(

x

, t

))

GP

(

x

,

y

, t

i

)

S

=

ResPCA

(

Y

)

or

S

i

=

PCA

(

Y

(

i

)

)

IFC [8]

B

(

t

) = 0

,

u

(

x

, t

)

,

NN

(

x

,

y

, t

)

,

S

=

I

ContinuAR

B

(

t

) =

β

I

,

S

=

I

mation u ( x ) ∼ N ( 0 , k u ( x , x ′ ) ⊗ I ) ; HOGP [24] improves IMC by letting [ SS /latticetop ] ij = k d ( z i , z j ) with tensor decomposition, where z are latent vectors. Higdon et al. [19] further simplify SLFM using singular value decomposition (SVD) to obtain S . For a full review of the GP-based multi-variate surrogates, the readers are referred to [21] for an excellent review. To overcome the fixed bases of SLFM, GP regression network (GPRN [30, 25, 31]) introduces another GP to model a flexible S .

Tractable Fusion is the name we give to multi-fidelity models where the joint outputs form a GP. AR[3] is the most fundamental and tractable solution to FiDEs for a univariate problem, i.e., D = 1 , which allows it to take a (discrete) t-dependent B ( t ) = β ( t i ) without leading to an intractable solution. To deal with high-dimensional problems, ResGP [12] avoids the difficulty involved with B ( t ) by setting it to zero and uses a conditional independent GP for u ( x , t i ) (equivalent to R = 1 , S = I ).

Intractable Fusion refers to methods where the joint output is no longer a GP. Normally, they assume B ( t ) = 0 to avoid the integral and replace the antiderivative G ( u ( x , t )) in Eq. (13) with a regression model that takes y (at the previous fidelity) as model input. For instance, the popular nonlinear AR (NAR) [11] uses GP ( x , y , t i ) to replace G ( u ( x , t )) ; Deep coregionalization (DC [28]) extends NAR with a residual PCA to capture S ; Wang et al. [7] further introduce a fidelity variating S ( t i ) to increase model flexibility at the cost of significant growth in the number of model parameters and a few simplifications in the approximated inference; Li et al. [26] take the advances of recent advancement of deep learning neural network (NN) and place a Bayesian neural network (BNN) to replace G ( u ( x , t )) ; A similar idea is proposed by Meng and Karniadakis [32] who add a physics regularization layer to a deep NN; Li et al. [33] propose a Bayesian network approach to multi-fidelity fusion with active learning techniques for efficiency improvement. To account for the missing uncertainty propagation in intractable fusion methods, Cutajar et al. [34] use approximation inference at the price of overfitting and scalability to high-dimensional problems; In the UQ community, multi-fidelity fusion has been implemented using stochastic collocation (SC) [27], which essentially uses a polynomial chaos expansion (PCE) to approximate G ( u ( x , t )) under FiDEs. All the above methods do not generalize to infinite-fidelity problems. The seminal work IFC [8] uses a NeuralODE to solve general FiDEs, leading to a great challenge in model training.

Algorithm. The solutions of FiDEs are closely related to the latent force models (LFMs) [16, 35], where they focus on dynamic processes without x . For tractable fusion, Le Gratiet [10] improves the efficiency of AR by decomposing the likelihood functions based on a subset multi-fidelity data structure, which is relaxed through our non-subset decomposition in Section 4.

6 Experiment

To assess ContinuAR, we compare it with (1) AR [3], (2) ResGP [12], (3) MF-BNN 2 [33], and (4) IFC 3 [8], which are the most closely related SOTA methods for high-dimensional multi-fidelity fusion, particularly with infinite fidelities. Note that AR and ResGP are modification versions according to [36] instead of their original versions that cannot deal with non-subset or high-dimensional problems. ContinuAR, AR, and ResGP are implemented using Pytorch. All GP-based methods use the RBF kernel for a fair comparison. MF-BNN and IFC are conducted using default settings from their source codes. As presented in IFC's original paper, there are two variations of IFC, one with deep learning (IFC-ODE) and the other one with Gaussian process ODE (IFC-GPODE), which shows better results. Thus, we show the results of IFC-GPODE in our experiments. All GP-based models are trained with 200 iterations whereas MF-BNN and IFC with 1000 iterations to reach convergence. All experiments are run on a workstation with an AMD 5950x CPU, Nvidia RTX3090 GPU, and 32 GB RAM. All experiments are repeated five times with different random seeds, and the mean performance and its standard deviation are reported.

6.1 Multi-Fidelity Fusion for Benchmark PDEs

We first assess ContinuAR in three canonical PDE simulation benchmark problems, namely, Heat, Burgers', and Poisson's equations as in [28, 37-39], which produces high-dimensional spatial/spatial-temporal fields as model outputs. The multi-fidelity results are generated by solving the corresponding PDEs using finite difference with mesh nodes of { 4 2 , 8 2 , 16 2 , 32 2 , 64 2 } , which is also used as the fidelity index t for ContinuAR and IFC. We use interpolation to upscale the lower fidelity fields and record them at the high-fidelity grid nodes to provide uniform output across different fidelity. Equation parameters in the PDEs and parameterized initial or boundary conditions represent the corresponding inputs. Please refer to Appendix J for comprehensive details of simulation setups.

IFC has an extremely high computational cost which scales poorly to output dimension (see Table 1 for experiments with N 0 = 32 and η = 0 . 5 ). For Poisson's equation, considering the 5x more iterations required, IFC requires about 62,500x more training time than ContinuAR on a CPU. It

will take 32 days just to finish the Poisson experiment in our experimental setup. Thus, we do not consider IFC a practical method for high-dimensional problems because its training cost is almost as expensive as running a high-fidelity simulation; we only apply it to relatively low-dimensional problems, namely, Heat equation and the later real-world application experiments.

Classic Subset Assessment . We follow the classic experiment setup where the training samples are consistently increased; the lower-fidelity data forms a superset of the higher-fidelity data for this experiment. To deliver concrete results, the high-fidelity training samples are reduced at rate η , i.e., round ( η | X ( t i +1 ) | n ) = | X ( t i ) | n , while the removed samples are randomly selected. For each experiment, we gradually increase the number of lowest-fidelity training data N 0 from 32 to 128 and calculate predictive accuracy (RMSE). The statistical result for five repeated experiments (with different random seeds) is demonstrated in Fig. 2. The superiority of IFC and ContinuAR indeed highlight the benefits of harnessing the useful information hidden in the fidelity indicators t i . All method benefits from a larger η ; MF-BNN is unstable due to its model complexity. ContinuAR outperforms the competitors with a large margin consistently in call cases (with up to 6.8x improvement). Averaging over all experiments, ContinuAR achieves 3.5x, 2.5x, and 8.7x accuracy improvements over the best competitor for Heat, Burger's, and Poisson's equation, respectively. Testing error against training time for Heat N 0 = 32 and η = 0 . 5 is shown in Fig. 1, where ContinuAR achieves 2.8x and 603x improvement in RMSE and training time. Detailed mean error fields (see Appendix for the computational details) are also provided in Fig. 4, which clearly reveals the superiority of ContinuAR by producing minimal red regions (high error) and maximal blue regions (low error).

Non-subset Assessment . We then assess the compatibility of ContinuAR for non-subset training data, which is inevitable in many-fidelity problems. The setup is similar to subset assessment with the same decreasing rate η , except that the training data for each fidelity is randomly selected without forcing a subset structure. The results are reported in Fig. 3. The results are consistent with the subset assessment, and ContinuAR outperforms the competitors with a large margin (with up to 3x improvements), whereas MF-BNN performs poorly as usual. Averaging over all experiments, ContinuAR achieves 1.9x, 1.6x, and 1.5x accuracy improvements over the best competitor for Heat, Burger's, and Poisson's equation, respectively. The mean error fields are demonstrated in Fig. 4, which draws the same conclusion as the subset assessment.

Ours error

ARerror

resGPerror

Ourserror

ARerror

resGPerror

0.40

0.25

0.20

0.17

·0.15

0.05

0.10

80.0

80.0

60.0

60.0

40.0

40.0

20.0

20.0

0.0

0.0

0.67

·0.15

0.64

0.10

0.60

0.06

0.57

0.01

0.15

0.15

0.10

0.10

0.05

0.05

0.00

0.00

6.2 Multi-Fidelity Fusion for Real-World Applications

Next, we assess ContinuAR with real-world applications. Particularly, we look at two practical simulation applications: topology optimization (TopOP) and Plasmonic nanoparticle arrays (PNA) simulation, both of which are known for their high computational cost and render the need for multifidelity fusion. The detailed problem descriptions and simulation setups are given in Appendix J. The TopOP data contains five fidelity solutions of dimensionality of 1600 based on simulations on a mesh concatenating { 64 , 256 , 576 , 1024 , 1600 } nodes, whereas the PNA data has five-fidelity outputs of two dimensions.

The same subset and non-subset assessments are conducted, and the results are reported in Fig. 5. We can see that the advantage of ContinuAR is clear but less pronounced due to the complexity of the data. Averaging over all experiments, ContinuAR achieves 1.1x and 1.4x improvements over the best competitor in TopOP and PNA, respectively. Although taking averagely 192s for training (140x more than ResGP and AR) for PNA, IFC only shows small improvements over ResGP and AR in non-subset settings, but not in the subset settings. In contrast, the training time of ours is only 5.7s (4.4x more than ResGP and AR) and the improvements are significant in all cases. Detailed mean error field in Fig. 4 shows a significant reduction in red region volume for ContinuAR.

Cost Evaluation. Finally, to assess ContinuAR in a more realistic setting, we relax the requirement that low-fidelity samples are more than high-fidelity samples and randomly pick samples for each fidelity (ensuring that each fidelity has a minimal of four samples) for each models. We do not test under different seeds because the randomness of setting is already sufficient to assess the robustness of the models. The computational cost for generating the training data against accuracy results for all assessment datasets are reported in Fig. 6. The superiority of ContinuAR over the competitors is consistent with previous experiments. The average accuracy improvements over all settings of ContinuAR over the best competitors is 4.3x, 9.3x, 2.7x, 1.2x, and 1.3x for Heat equation, Burger's equation, Poisson's equation, TopOP, and PNA, respectively.

7 Conclusions

In this work, we propose a unifying framework FiDEs to pave the way for tractable infinite-fidelity fusion and a novel solution ContinuAR, which shows a significant improvement over the SOTA competitors. We expect FiDEs and ContinuAR can lead to the development of surrogate-assisted systems to deliver cheaper, more efficient, and eco-friendly scientific computation. The limitation of this work includes the rank-1 assumptions and further examination in highly-complex real systems.

References

Bobak Shahriari, Kevin Swersky, Ziyu Wang, Ryan P. Adams, and Nando de Freitas. Taking the Human Out of the Loop: A Review of Bayesian Optimization. Proceedings of the IEEE, 104(1):148-175, January 2016. ISSN 0018-9219, 1558-2256. doi: 10.1109/JPROC.2015.2494218.

Apostolos F. Psaros, Xuhui Meng, Zongren Zou, Ling Guo, and George Em Karniadakis. Uncertainty quantification in scientific machine learning: Methods, metrics, and comparisons. Journal of Computational Physics, 477:111902, 2023. ISSN 0021-9991. doi: https://doi.org/10.1016/j.jcp.2022. 111902.

M. Kennedy. Predicting the output from a complex computer code when fast approximations are available. Biometrika, 87(1):1-13, March 2000. ISSN 0006-3444, 1464-3510. doi: 10.1093/biomet/87.1.1.

Matthias Poloczek, Jialei Wang, and Peter Frazier. Multi-Information Source Optimization. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.

Jialin Song, Yuxin Chen, and Yisong Yue. A General Framework for Multi-fidelity Bayesian Optimization with Gaussian Processes. In Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics, pages 3158-3167. PMLR, April 2019.

L. Parussini, D. Venturi, P. Perdikaris, and G.E. Karniadakis. Multi-fidelity Gaussian process regression for prediction of random fields. Journal of Computational Physics, 336(C):36-50, May 2017. ISSN 0021-9991. doi: 10.1016/j.jcp.2017.01.047.

Zheng Wang, Wei Xing, Robert Kirby, and Shandian Zhe. Multi-Fidelity High-Order Gaussian Processes for Physical Simulation. In International Conference on Artificial Intelligence and Statistics, pages 847855. PMLR, March 2021.

Shibo Li, Zheng Wang, Robert Kirby, and Shandian Zhe. Infinite-fidelity coregionalization for physical simulation. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 25965-25978. Curran Associates, Inc., 2022.

Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.

Loic Le Gratiet. Bayesian analysis of hierarchical multifidelity codes. SIAM/ASA Journal on Uncertainty Quantification, 1(1):244-269, 2013.

Paris Perdikaris, M Raissi, Andreas Damianou, N D. Lawrence, and George Karniadakis. Nonlinear Information Fusion Algorithms for Data-Efficient Multi-Fidelity Modelling, volume 473. Royal Society. doi: 10.1098/rspa.2016.0751.

W. W. Xing, A. A. Shah, P. Wang, S. Zhe, Q. Fu, and R. M. Kirby. Residual gaussian process: A tractable nonparametric bayesian emulator for multi-fidelity simulations. Applied Mathematical Modelling, 97: 36-56, September 2021. ISSN 0307-904X. doi: 10.1016/j.apm.2021.03.041.

Carl Edward Rasmussen and Christopher K. I. Williams. Gaussian Processes for Machine Learning. Adaptive Computation and Machine Learning. MIT Press, Cambridge, Mass, 2006. ISBN 978-0-26218253-9.

Edward N Lorenz. Deterministic nonperiodic flow. Journal of atmospheric sciences, 20(2):130-141, 1963.

William E Boyce, Richard C DiPrima, and Douglas B Meade. Elementary differential equations and boundary value problems. John Wiley & Sons, 2021.

Mauricio A Alvarez, David Luengo, and Neil D Lawrence. Linear latent force models using gaussian processes. IEEE transactions on pattern analysis and machine intelligence, 35(11):2693-2705, 2013.

Andrew G Wilson, Zhiting Hu, Russ R Salakhutdinov, and Eric P Xing. Stochastic variational deep kernel learning. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016.

Diederik P. Kingma and Max Welling. Auto-Encoding Variational Bayes. arXiv:1312.6114 [cs, stat], December 2013.

Dave Higdon, James Gattiker, Brian Williams, and Maria Rightley. Computer Model Calibration Using High-Dimensional Output. Journal of the American Statistical Association, 103(482):570-583, June 2008. ISSN 0162-1459, 1537-274X. doi: 10.1198/016214507000000888.

Yee Whye Teh, Matthias Seeger, and Michael I. Jordan. Semiparametric latent factor models. In International Workshop on Artificial Intelligence and Statistics, pages 333-340. PMLR, January 2005.

Mauricio A. Álvarez, Lorenzo Rosasco, and Neil D. Lawrence. Kernels for Vector-Valued Functions: A Review. Foundations and Trends® in Machine Learning, 4(3):195-266, 2012. ISSN 1935-8237, 19358245. doi: 10.1561/2200000036.

Loic Le Gratiet and Claire Cannamela. Cokriging-based sequential design strategies using fast crossvalidation techniques for multi-fidelity computer codes. Technometrics, 57(3):418-427, 2015. doi: 10. 1080/00401706.2014.928233.

Michel Goulard and Marc Voltz. Linear coregionalization model: tools for estimation and choice of cross-variogram matrix. Mathematical Geology, 24(3):269-286, 1992.

Shandian Zhe, Wei Xing, and Robert M. Kirby. Scalable High-Order Gaussian Process Regression. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 2611-2620. PMLR, April 2019.

Andrew Gordon Wilson, David A Knowles, and Zoubin Ghahramani. Gaussian process regression networks. In Proceedings of the 29th International Coference on International Conference on Machine Learning, pages 1139-1146. Omnipress.

Shibo Li, Wei Xing, Robert Kirby, and Shandian Zhe. Multi-Fidelity Bayesian Optimization via Deep Neural Networks. Advances in Neural Information Processing Systems, 33:8521-8531, 2020.

Akil Narayan, Claude Gittelson, and Dongbin Xiu. A Stochastic Collocation Algorithm with Multifidelity Models. SIAM Journal on Scientific Computing, 36(2):A495-A521, January 2014. ISSN 1064-8275. doi: 10.1137/130929461.

Wei W. Xing, Robert M. Kirby, and Shandian Zhe. Deep coregionalization for the emulation of simulationbased spatial-temporal fields. Journal of Computational Physics, 428:109984, March 2021. ISSN 00219991. doi: 10.1016/j.jcp.2020.109984.

Pierre Goovaerts et al. Geostatistics for natural resources evaluation. Oxford University Press on Demand, 1997.

Shibo Li, Wei Xing, Robert M. Kirby, and Shandian Zhe. Scalable Gaussian Process Regression Networks. In Twenty-Ninth International Joint Conference on Artificial Intelligence, volume 3, pages 2456-2462, July 2020. doi: 10.24963/ijcai.2020/340.

Trung Nguyen and Edwin Bonilla. Efficient variational inference for gaussian process regression networks. In Artificial Intelligence and Statistics, pages 472-480. PMLR, 2013.

Xuhui Meng and George Em Karniadakis. A composite neural network that learns from multi-fidelity data: Application to function approximation and inverse PDE problems. Journal of Computational Physics, 401: 109020, January 2020. ISSN 0021-9991. doi: 10.1016/j.jcp.2019.109020.

Shibo Li, Zheng Wang, Robert Kirby, and Shandian Zhe. Deep multi-fidelity active learning of highdimensional outputs. In Gustau Camps-Valls, Francisco J. R. Ruiz, and Isabel Valera, editors, Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, volume 151 of Proceedings of Machine Learning Research, pages 1694-1711. PMLR, 28-30 Mar 2022.

Kurt Cutajar, Mark Pullin, Andreas Damianou, Neil Lawrence, and Javier González. Deep Gaussian Processes for Multi-fidelity Modeling. arXiv:1903.07320 [cs, stat], March 2019.

Simo Särkkä, Mauricio A. Álvarez, and Neil D. Lawrence. Gaussian process latent force models for learning and stochastic control of physical systems. IEEE Transactions on Automatic Control, 64(7): 2953-2960, 2019. doi: 10.1109/TAC.2018.2874749.

Yuxin Wang, Zheng Xing, and WEI W. XING. GAR: Generalized autoregression for multi-fidelity fusion. In Advances in Neural Information Processing Systems, 2022.

Rui Tuo, C. F. Jeff Wu, and Dan Yu. Surrogate Modeling of Computer Experiments With Different Mesh Densities. Technometrics, 56(3):372-380, July 2014. ISSN 0040-1706, 1537-2723. doi: 10.1080/ 00401706.2013.842935.

Mehmet Onder Efe and Hitay Ozbay. Proper orthogonal decomposition for reduced order modeling: 2d heat flow. In Proceedings of 2003 IEEE Conference on Control Applications, 2003. CCA 2003., volume 2, pages 1273-1277. IEEE.

Maziar Raissi and George Em Karniadakis. Machine Learning of Linear Differential Equations using Gaussian Processes. Journal of Computational Physics, 348:683-693, November 2017. ISSN 00219991. doi: 10/gbzfgr.

Supplementary Materials

A Gaussian process

Due to its capacity to handle complex black-box functions and quantify uncertainty, the Gaussian process (GP) is often a popular selection for the surrogate model. For now, let's contemplate a simplified scenario where we possess noise-influenced observations, denoted as y i = f ( x i ) + ε , i = 1 , . . . , N . In a GP model, a prior distribution is placed over the true underlying function of f ( x ) , whose input is x :

employing mean and covariance functions,

where the hyperparameters θ control the kernel function and E [ · ] is the expectation. Centering the data allows us to assume the mean function as a constant, m 0 ( x ) ≡ m 0 . Other options, like a linear function of x , are possible but are seldom employed unless there is prior knowledge of the function's shape. Various forms can be adopted for the covariance function, with the ARD kernel being the most widely favored.

Starting here, we remove the specific mention of θ 's dependence on k ( x, x ′ ) . The hyperparameters θ 1 , . . . , θ l are termed as length-scales in this case. When x is a constant parameter, f ( x ) represents its random variable. In contrast, a set of values, f ( x i ) , where i = 1 , . . . , N , constitutes a partial realization of the GP. Realizations of GPs are deterministic functions of x . The key characteristic of GPs is that the joint distribution of η ( x i ) , for i = 1 , . . . , N , is a multivariate Gaussian.

We can derive the model likelihood by assuming the Gaussian distribution of the model deficiency ε ∼ N (0 , σ 2 ) , along with the utilization of the prior (A.1) and the existing data.

where K = [ K ij ] is the covariance matrix, in which K ij = k ( x i , x j ) , i, j = 1 , . . . , N . The hyperparameters θ are frequently obtained through point estimations, utilizing the maximum likelihood (MLE) of Eq. (A.4) with respect to θ . The joint Gaussian distribution of y and f ( x ) also possesses a mean value of m 0 1 and a covariance matrix.

where k ( x ) = ( k ( x 1 , x ) , . . . , k ( x N , x )) T . The conditional predictive distribution at x is obtained by conditioning on y .

Given by µ ( x ) , the expected value is E [ f ( x )] , and v ( x ) represents the predictive variance. The transition from Eq. (A.5) to Eq. (A.6) is critical, as the prediction posterior of this wake relies on a comparable block covariance matrix.

B From AR to LiFiDE

We first revisit the classic AR and reveal its connection to an ODE's forward Euler solution. Rewrite Eq. (1) as follows,

where α ≡ γ -1 . We then divide both sides by the constant ( t T -t 0 ) indicating the fidelity difference and absorb the constant into ( ρ -1) and u ( x , t 0 ) and write

Since α and v ( x , t 0 ) are values and function to be estimated, we can absorb the constant t T -t 0 into α and v ( x , t 0 ) and write

where β 0 ≡ α /t T -t 0 and u ( x , t 0 ) ≡ v ( x , t 0 ) /t T -t 0 . We recognize that is a explicit solution of a different equation. If we take the limit of ( t T -t 0 ) → 0 , we have

which is in Proposition 1. Notice that, in this equation, we turn the constant β into a function β ( t ) , which allows us to control differential level of information transfer depending on its fidelity level as a general formulation.

Taking a forward Euler solution to solve Eq. (A.10), we have

We recognize that this formulation is exactly the same as the classic AR with ρ = 1 + ∆ t β ( t ) with a residual GP ∆ tu ( x , t ) .

C A General Solution to LiFiDE

We derive the general solution to the derived linear fidelity differential equation (LiFiDE)

which is a standard non-homogeneous first order differential equation. We know that for the homogeneous equation, i.e., u ( x , t ) = 0 , the general solution is

where C ( x ) is a functional of x , and B ( t ) = ∫ t β ( τ )d τ is the antiderivative of β ( t ) . Thus, we assume the general solution of the non-homogeneous equation is of the form y ( x , t ) = v ( x , t ) e -B ( t ) , where v ( x , t ) is a functional of x and t . Substituting this into Eq. (A.12), we have

Integrating both sides, we have

Putting this back into the assumed solution, we have

If we assume a constant for β ( t ) , i.e., β ( t ) = β , then we have

For a practical from where the integral starts from t 0 , the lowest-fidelity index, we have

where we have assumed u ( x , t ) = 0 for t < t 0 because we are not interested in t < t 0 . Substituting this into Eq. (A.17), we have

This formulation is equivalent to Eq. (A.17). However, it allows for more flexibility in practice as we can give start the model with the lowest-fidelity t 0 and then use the model to predict the higher-fidelity t > t 0 .

D General Solutions to LiFiDEs

Consider LiFiDEs taking this general form

where B ( t ) and S are matrices, y ( x , t ) is a vector of unknown functions, and u ( x , t ) is a non-zero vector. The general solution is the sum of the homogeneous solution y h ( x , t ) and a particular solution y p ( x , t ) . The homogeneous solution comes from the homogeneous equation:

where c is a constant vector. The particular solution comes from the non-homogeneous equation and can be obtained using the variation of parameters. The method involves finding a function v ( x , t ) such that:

is a solution to the non-homogeneous system. Substituting y p into the non-homogeneous system gives:

which can be integrated to find v ( x , t ) . Thus, the particular solution is:

The general solution to the non-homogeneous system is the sum of the homogeneous and particular solutions:

This has the solution:

which is similar to the solution to LiFiDE in Eq. (A.17). Note that the matrix exponential computation is done by

which is not easy to compute. However, if the matrix B ( t ) is a constant matrix β I , then the matrix exponential can be computed by which gives us a efficient way to derive a solution to LiFiDEs,

E Proof of Lemma 1

Lemma 1. Autokrigeability in ContinuAR: the particular values of the spatial correlation matrix H and SS /latticetop do not matter in the predictive mean as they will be canceled out.

Proof. The autokrigeability in ContinuAR is easy to derive once we derive the predictive posterior for the subset case in Appendix G and the non-subset case in Appendix H. We can see the the matrix H and SS /latticetop are canceled out in the predictive mean.

F Joint Likelihood of FiDEs-1

Wederive the joint likelihood of FiDEs for observations Y = [ /vector y (0) ; /vector y (1) ] as an illustrating example. For clarity, we slightly abuse the notations by replacing the previous spatial correlation H in the main paper with S (0) and SS /latticetop with S (1) . The joint probability for Y is

Eq. (A.30) reveals that the for a two fidelity problem, ContinuAR is equivalent to AR. Thus, the likelihood decomposition for subset structure data [10] also holds for ContinuAR, which lays out the foundation for our fast inference algorithm for non-subset problems.

G Predictive Posterior With Subset Structure

In order to conduct the proof of decomposition for the likelihood function with non-subset data, we need to derive the form of predictive posterior distribution with multi-fidelity with subset structure i.e., X ( T ) ⊂ , · · · , ⊂ X (0) , first. Here, we take the simplest situation, i.e., T = 1 as an example, and the assumption X (1) ⊂ X (0) holds. To stay consistent with AR, we denote ρ = e -β ( t 1 -t 0 ) for clarity. Following the format of the covariance

matrix in Eq. (A.30) and Eq. (A.6), the mean function and covariance matrix have the following expression,

where k (0) ∗ = k 0 ( X ∗ , X (0) ) , k (00) ∗ = k 0 ( X ∗ , X ∗ ) , K (00) = k 0 ( X (0) , X (0) ) denotes the covariance in the lowest fidelity, and k (1) a ∗ = k u 1 ( X ∗ , X (1) ) , k (11) ∗ = k u 1 ( X ∗ , X ∗ ) and K (11) a = k u 1 ( X (1) , X (1) ) .

Notice that Eq. (A.32) decomposes the predictive posterior into two parts one related to the low-fidelity data and the other to the high-fidelity data. This lays the foundation for our later derivation for the non-subset structure. In the following section, we will derive the predictive posterior for non-subset structure, and show that the autokrigeability also holds.

H Predictive Posterior With Non-Subset Structure

In this section, we derive the mean function and covariance matrix in the predictive posterior for non-subset structure following the previous two-fidelity setup.

Non-subset part

Virtual complement set

X(t-1)

x( = E(x(-1)

x(-1) =Ex()

As we know, once the ˆ Y (1) is decided, the high-fidelity posterior part can be written as the subset posterior distribution, in the following way,

where

This is depends on the conclusion in Eq. (A.31). Σ (2) ∗ and ¯ y (2) are posterior covariance matrix and mean function.

At the same time, we know that ˆ Y (1) are samples from fidelity 1 model, therefore the probability of getting ˆ Y (1) can be written as:

where the Σ (1) ∗ ⊗ S means the posterior covariance matrix of sampling ˆ Y (1) from lower fidelity model and ¯ y (1) are predicted mean for the non-subset data. Therefore, the posterior distribution of non-subset data structure is

where ˜ y and Γ is defined by the following equations,

where E (2) m denotes the selection matrix which selects the non-subset parts between two fidelities. ˆ X (1) = ˆ X (1) = ( E (2) m ) /latticetop [ X (1) , ˆ X (1) ] and E (2) n also denotes the selection matrix but which selects all inputs for the second fidelity, X (2) = ( E (2) n ) /latticetop [ X (1) , ˆ X (1) ] . Please see Fig. 7 for the illustration of the selection matrix for the multi-fidelity data structure.

After that, we can simplify the determinant and the exponential parts by decomposing them into different parts and using the Sherman-Morrison formula to obtain conclusions.

For the determinant part, we can derive

For the exponential parts in Eq. (A.37),

Therefore, the likelihood of the posterior distribution is

From the upper formula, for the non-subset data structure, the posetrior mean and covairance matrix are,

In Eq. (A.42), we also prove that matrix S does not take part in posterior mean function which means autokrigeability still holds in the non-subset data structure. By recursively applying this conclusion, we can easily extend it to a multi-fidelity problem and we show the detailed application in our main paper.

I Decomposition of Joint Likelihood With Non-Subset Structure

As we have shown that the autokrigeability holds in subset and non-subset data structure, we thus assume an identical spatial correlation, S = I , for easy computation acceleration (with the cost of losing the accuracy in the predictive variance). Due to the identical spatial correlations, we no longer need vectorization. First, we decompose the joint likelihood L into several independent parts,

Once the ˆ Y (1) is fixed, the probability of part 1 in Eq. (A.43) can be written as,

Then is the part 2 in Eq. (A.43), it is based on the posterior distribution of lower fidelity, zu

where we use the N (2) n to denotes the missing point in second fidelity corresponding with fidelity 2, which means the ˆ Y (2) parts, yellow part, in Fig. 7.

Therefore, we can combine the Eq. (A.44) with the Eq. (A.45), the integral part in Eq. (A.43) can be written as,

After that, we try to decompose the vector Y as,

The Y is divided into subset parts ˇ Y and non-subset part ˆ Y . Then, we consider the data fitting part by substituting Eq. (A.47) into Eq. (A.46), which gives the method to make the Eq. (A.46) calculable,

where φ = ( ˇ Y (2) , ˆ Y (2) ) /latticetop -( ˇ Y (1) , 0 ) /latticetop . After that we try to simplify (A.48) by different parts with Sherman-Morrison formula.

First, we consider the determenant part,

Then, we gather terms with φ and simplify them as,

After that, we consider the interaction part between φ and ¯ Y (1) ,

Finally, we simplify the terms related with ¯ Y (1) ,

Putting all the parts together, the joint likelihood for the R = 1 is,

Therefore, we can see that the joint likelihood of three different fidelities with non-subset dataset can be decomposed as three independent model to train. For problems with arbitrary number of fidelity, we can easily apply this conclusion recursively to decompose them into the summation structure, which is scalable to the number of training data and fidelity level and is easy to code up.

J Experiment in Detail

J.1 Canonical PDEs

Three canonical PDEs are under consideration: Poisson's equation, Heat equation, and Burger's equation. These equations play pivotal roles in scientific and technological applications [41-43]. They present typical simulation scenarios that include high-dimensional spatial-temporal field outputs, nonlinearities, and discontinuities. These scenarios are frequently used as benchmark problems for surrogate models [28, 37-39]. x and y denote the spatial coordinates, with t representing the time coordinate, a departure from the notation used in the primary paper. The notation in the appendix is employed solely to improve comprehension and does not influence or correlate with the main text.

Let's start with Burger's equation , a renowned nonlinear hyperbolic PDE commonly used to depict various physical phenomena such as fluid dynamics [42], nonlinear acoustics [48], and traffic flows [49]. It is often employed as a standard benchmark for multiple numerical solvers and surrogate models [50, 51] due to its ability to generate discontinuities (shock waves) based on a standard conservation equation. Here is the equation in its viscous form,

Within the context, u represents volume, x pertains to spatial location, t signifies time, and v stands for viscosity. The parameters are defined as follows: x ranges between 0 and 1 (in meters), t spans from 0 to 3 (in seconds), and the initial condition is set as u ( x, 0) = sin( x π / 2) with homogeneous Dirichlet boundary conditions. We conducted uniform sampling of viscosities v within the interval of [0.001,0.1] (in milliPascal seconds) as the input parameter for generating the solution field.

Solving the problem in the space and time domains involves the utilization of finite elements employing hat functions and backward Euler, respectively. To produce solutions at various fidelities, the solver addresses the PDEs by employing discretized spatial-temporal domains consisting of regular rectangular meshes with grid points ranging from 8 2 , 16 2 , 32 2 , 64 2 , 128 2 , facilitating simulations from low to high fidelity.

Next, the focus shifts to Poisson's equation , a common elliptic PDE employed in mechanical engineering and physics to simulate potential fields, including gravitational and electrostatic fields [41]. Expressed in the form of

It represents an extension of Laplace's equation [53]. Encountered frequently in physics, Poisson's equation, despite its simplicity, serves as a fundamental test case for surrogate models, as highlighted in various studies [37, 54]. Within our experiment, we apply Dirichlet boundary conditions to a 2D spatial domain, where x ranges over [0 , 1] × [0 , 1] . The input parameters involve fixed values assigned to the four boundaries and the central point of the rectangular domain, each ranging from 0 . 1 to 0 . 9 . To generate the matching potential fields as outputs, we sampled the input parameters uniformly. Employing a first-order center differencing scheme and regular rectangular meshes, we solved the PDE using the finite difference approach. Simulations of five different fidelities were generated using meshes containing grid nodes 8 2 , 16 2 , 32 2 , 64 2 , 128 2 .

Heat equation , established in 1822 to depict time-dependent changes in heat fluxes, remains a fundamental PDE. Its scope extends beyond heat fluxes, finding applications in diverse scientific domains such as probability

P2

P1

P3

theory [55, 43] and financial mathematics [56]. As a result, it is widely adopted as a representative model. Here is the heat equation:

where k is the materials conductivity q V is the rate at which energy is generated per unit volume of the medium ρ is the density and c p is the specific heat capacity. The flux rate at the left boundary, ranging from 0 to 1 at x = 0 , the flux rate at the right boundary, ranging from -1 to 0 at x = 1 , and the thermal conductivity, ranging from 0 . 01 to 0 . 1 , serve as the input parameters. Within a 2D spatial-temporal domain x ∈ [0 , 1] and t ∈ [0 , 5] , we set up the Neumann boundary condition at x = 0 and x = 1 . Additionally, we define u ( x, 0) as H ( x -0 . 25) -H ( x -0 . 75) , where H ( · ) represents the Heaviside step function. Using finite difference in space and backward Euler in time domains, the equation is solved. The spatial-temporal domain is discretized into a 16 × 16 regular rectangular mesh for the first (lowest) fidelity solver, while a refined solver employs a 32 × 32 mesh for the second fidelity. The computed result fields are on a 100 × 100 spatial-temporal grid. Solving the equation involves applying a finite difference approach in the spatial domain and employing reverse Euler in the temporal domain. We discretize the spatial-temporal domain into a regular rectangular mesh, using 8 2 , 16 2 , 32 2 , 64 2 , 128 2 nodes to produce simulation results at five varying fidelity levels.

J.2 Multi-Fidelity Fusion for Topology Optimization

In a topology structure optimization problem, we employ ContinuAR to determine the optimal topology structure, maximizing mechanical metrics such as stiffness, for a material layout comprising alloys and concrete. This is accomplished by considering various design parameters including external force and angle. Particularly with the advancements in 3D printing methods, where materials are deposited in small increments, topology structure optimization has become a crucial technique in mechanical design, applied in areas like airfoils and slab bridges. Notably, the computational intensity of topology optimization stems from the need for gradientbased optimization and mechanical simulations. The demanding nature of a high-fidelity solution, requiring extensive discretization mesh and imposing substantial computational burdens in both space and time, exacerbates the situation.

Gaining popularity is the utilization of data-driven methods that offer suitable structures to facilitate the process [57]. In this study, we examine the topology optimization of a cantilever beam. Utilizing the efficient implementation [58], we conduct density-based topology optimization, aiming to minimize the compliance C while adhering to volume constraints V ≤ ¯ V .

Utilizing the SIMP scheme [59], we convert continuous density measurements into discrete, optimal topologies. System inputs include the position of point load P 1 , the angle of point load P 2 , and the filter radius P 3 [60]. The issue is addressed using a standard mesh containing nodes in the set 32 2 , 40 2 , 48 2 , 56 2 , 64 2 .

J.3 Multi-Fidelity Fusion for Plasmonic nanoparticle arrays

Using the Coupled Dipole Approximation (CDA) approach, we compute the extinction and scattering efficiencies Q ext and Q sc for plasmonic systems of different scatterer quantities in the final illustration. CDA represents a technique for emulating the optical response of an ensemble of identical, non-magnetic metallic nanoparticles, each with dimensions significantly smaller than the wavelength of light (in this case, 25 nm). In this work, we define Q ext and Q sc as the QoIs. Our proposed method allowed us to construct surrogate models efficiently, incorporating up to three fidelities. We examined particle arrays generated by Vogel spirals, where the interaction of incident waves with particles significantly influences the magnetic field. Consequently, the local extinction field produced by plasmonic arrays is greatly impacted by the number of nanoparticles. The configurations of Vogel spirals, characterized by particle numbers in the set 2 , 50 , 200 , 500 , 1000 , defined

the parameters for five-fidelity simulations. The parameter space was determined to be λ ∈ [200 , 800] nm, α vs ∈ [0 , 2 π ] rad, and a vs ∈ (1 , 1500) . These parameters represent the incidence wavelength, the divergence angle, and the scaling factor, respectively. Inputs were chosen using a Sobol sequence. As the number of nanoparticles increases, the computing time required for CDA execution grows exponentially. Hence, the suggested sampling approach leads to substantial reductions in computational costs.

By solving the corresponding linear equation, one can compute the local field E loc ( r j ) for each nano-sphere, given N metallic particles with the same volumetric polarizability α ( ω ) situated at vector coordinates r i . This calculation allows for the determination of the response of a plasmonic array to electromagnetic radiation, as per the solution [61] of the local electric fields, E loc ( r j ) .

/negationslash where the incident field is denoted by E 0 ( r i ) , where k represents the wave number in the background medium, and /epsilon1 0 signifies the dielectric permittivity of vacuum ( /epsilon1 0 = 1 in the CGS unit system). ˜ G ij is derived from 3 × 3 blocks of the overall 3 N × 3 N Green's matrices for the i th and j th particles. When j = i , ˜ G ij becomes a zero matrix. Otherwise, it is computed as

Expressed as ̂ r ij , the unit position vector represents the distance from particle j to i , denoted by rij = | r ij | . Solving Eq A.54 and A.55 enables the computation of the overall local fields E loc ( r i ) , which in turn determines the scattering and extinction cross-sections. Further information on the numerical approach is available in [62].

Derived from the normalization of the scattering and extinction cross-sections concerning the array's entire projected area Q ext and Q sc are considered. Our focus was on the Vogel spiral class of particle arrays, as described in [63].

In a Vogel spiral array, the n -th particle's radial distance and polar angle are denoted by ρ n and θ n , respectively. Hence, the unique definition of the Vogel spiral configuration includes the incidence wavelength λ , the divergence angle α vs , the scaling factor a vs , and the number of particles n .

J.4 Detailed Prediction Analysis

Defining the average RMSE field y (EF) allows us to thoroughly examine the prediction error.

The square root is taken element-wise, where ˜ y i represents the prediction and y i is the true ground value.

The average RMSE field corresponding to the Heat equation, Burger's equation, Poisson's equation, and TopOP problem with a decreasing rate of η = 0 . 5 , lowest-fidelity training samples, N 0 = 128 and 128 testing samples are shown in Fig. 4 (left) for the subset assessment and Fig. 4 (right) for the non-subset assessment. Plasmonic nanoparticle arrays (PNA) have only two output variables and thus we do not show the average RMSE field for it.

For the classic subset assessment in Fig. 4 (left), we can see that clear that ContinuAR outperforms the competitors in all cases with a large margin by showing more blue areas and only tiny red areas. For Heat equation, the error is significantly reduced in most areas except for the bottom area of the domain, where a tiny thin bar of red area is shown. For the Burger's equation, ContinuAR shows some checker board pattern in the error field, which is probability caused by the conditional independence assumption. Nonetheless, the error is significantly reduced in most areas the largest error is also reduced. For the Poisson's equation, different method has it own error patterns. ContinuAR show a more blue area in the left part of the domain, while AR has high-error areas everywhere except the center. For the TopOP problem, ContinuAR show a significantly reduced error by a significant reductions of deep red areas. The AR as the deepest blue ares, indicating its good performance. However, it also has a lar areas of red, indicating its poor performance overall.

For the classic non-subset assessment in fig: errorplot both (right), the overall conclusion is similar to the subset assessment. ContinuAR outperforms the competitors in all cases with a large margin by showing more blue areas and tiny red areas. The overall error pattern for most methods are similar to the subset assessment with subtle difference. For instance, for the Burger's equation, checker board pattern disappear, indicating a successful improvement by using training data across the input domain. The error for Poisson's equation is

also significantly reduced particularly on the right part of the domain. TopOP turns out to be the most stable problem as the error patterns for all methods are almost the same as the subset assessment.

K Additional Experimental Results

K.1 Likelihood Evaluation

Here are some additional results in Figure 9, which shows the negative log-likelihood (without the constant term and thus the result can be negative) of ContinuAR, ResGP, and AR. Surprisingly, the NLL of our method is better than ResGP and AR more significantly than the comparisons in RMSE. We believe that this is because the log-likelihood is more sensitive to the uncertainty of the prediction. Since our method is a joint learning (as discussed in the previous question), it is able to capture the uncertainty as a joint model whereas the other methods treat each fidelity separately.

K.2 Two-fidelity Evaluation

In order to understand the limitation of our method, we follow the classic subset experiment setting with η = 0 . 5 and reduce the number of fidelity from five to two. The results are shown in the Figure 10. In this case, we often see that ContinuAR is almost identical to the classic AR, losing its advantages as an infinite-fidelity fusion method. Also, when the number of training data is scarce, our method does not perform better compared to other baselines. There are certainly other factors that may affect the performance, such as the choice of fidelity and B(x). We will investigate this in future work.

Supplementary References

Loic Le Gratiet. Bayesian analysis of hierarchical multifidelity codes. SIAM/ASA Journal on Uncertainty Quantification, 1(1):244-269, 2013.

Steven C Chapra, Raymond P Canale, et al. Numerical methods for engineers. Boston: McGraw-Hill Higher Education,.

TJ Chung. Computational fluid dynamics. Cambridge university press.

Krzysztof Burdzy, Zhen-Qing Chen, John Sylvester, et al. The heat equation and reflected brownian motion in time-dependent domains. The Annals of Probability, 32(1B):775-804.

Wei W. Xing, Robert M. Kirby, and Shandian Zhe. Deep coregionalization for the emulation of simulationbased spatial-temporal fields. Journal of Computational Physics, 428:109984, March 2021. ISSN 00219991. doi: 10.1016/j.jcp.2020.109984.

Rui Tuo, C. F. Jeff Wu, and Dan Yu. Surrogate Modeling of Computer Experiments With Different Mesh Densities. Technometrics, 56(3):372-380, July 2014. ISSN 0040-1706, 1537-2723. doi: 10.1080/ 00401706.2013.842935.

Mehmet Onder Efe and Hitay Ozbay. Proper orthogonal decomposition for reduced order modeling: 2d heat flow. In Proceedings of 2003 IEEE Conference on Control Applications, 2003. CCA 2003., volume 2, pages 1273-1277. IEEE.

Maziar Raissi and George Em Karniadakis. Machine Learning of Linear Differential Equations using Gaussian Processes. Journal of Computational Physics, 348:683-693, November 2017. ISSN 00219991. doi: 10/gbzfgr.

NSugimoto. Burgers equation with a fractional derivative; hereditary effects on nonlinear acoustic waves. Journal of fluid mechanics, 225:631-653.

Kai Nagel. Particle hopping models and traffic flow theory. Physical review E, 53(5):4655.

S Kutluay, AR Bahadir, and A Özde c s. Numerical solution of one-dimensional burgers equation: explicit and exact-explicit finite difference methods. Journal of Computational and Applied Mathematics, 103(2): 251-261.

A. A. Shah, W. W. Xing, and V. Triantafyllidis. Reduced-order modelling of parameter-dependent, linear and nonlinear dynamic partial differential equation models. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, 473(2200):20160809, April 2017. ISSN 1364-5021, 1471-2946. doi: 10.1098/rspa.2016.0809.

Maziar Raissi, Paris Perdikaris, and George Em Karniadakis. Physics informed deep learning (part i): Data-driven solutions of nonlinear partial differential equations. arXiv preprint arXiv:1711.10561.

S Persides. The laplace and poisson equations in schwarzschild's space-time. Journal of Mathematical Analysis and Applications, 43(3):571-578.

I.E. Lagaris, A. Likas, and D.I. Fotiadis. Artificial neural networks for solving ordinary and partial differential equations. IEEE Transactions on Neural Networks, 9(5):987-1000, Sept./1998. ISSN 10459227. doi: 10.1109/72.712178.

Frank Spitzer. Electrostatic capacity, heat flow, and brownian motion. Probability theory and related fields, 3(2):110-121.

Fischer Black and Myron Scholes. The pricing of options and corporate liabilities. Journal of political economy, 81(3):637-654.

Wei Xing, Shireen Y. Elhabian, Vahid Keshavarzzadeh, and Robert M. Kirby. Shared-Gaussian Process: Learning Interpretable Shared Hidden Structure Across Data Spaces for Design Space Analysis and Exploration. Journal of Mechanical Design, 142(8), August 2020. ISSN 1050-0472, 1528-9001. doi: 10.1115/1.4046074.

Erik Andreassen, Anders Clausen, Mattias Schevenels, Boyan S. Lazarov, and Ole Sigmund. Efficient topology optimization in matlab using 88 lines of code. Structural and Multidisciplinary Optimization, 43(1):1-16, Jan 2011. ISSN 1615-1488.

Martin Philip Bendsoe and Ole Sigmund. Topology optimization: Theory, methods and applications. Springer, 2004.

Tyler E. Bruns and Daniel A. Tortorelli. Topology optimization of non-linear elastic structures and compliant mechanisms. Computer Methods in Applied Mechanics and Engineering, 190(26):3443 - 3459, 2001. ISSN 0045-7825.

Charles-Antoine Guérin, Pierre Mallet, and Anne Sentenac. Effective-medium theory for finite-size aggregates. JOSA A, 23(2):349-358, 2006.

Mani Razi, Ren Wang, Yanyan He, Robert M. Kirby, and Luca Dal Negro. Optimization of Large-Scale Vogel Spiral Arrays of Plasmonic Nanoparticles. Plasmonics, 14(1):253-261, February 2019. ISSN 1557-1955, 1557-1963. doi: 10.1007/s11468-018-0799-y.

Aristi Christofi, Felipe A Pinheiro, and Luca Dal Negro. Probing scattering resonances of vogels spirals with the greens matrix spectral method. Optics letters, 41(9):1933-1936, 2016.

Peer reviews:
comments: SUMMARY:
The author proposes a general auto-regression model for multi-fidelity fusion. By simplifying the ODEs over the fidelity indicator in a linear form, close-form solutions can be derived. And the computational efficiency can be further improved using a rank-1 approximation. The experiment results also show superior performance of the method.

STRENGTHS:
1. A general linear fidelity differential equation is proposed. It serves as an simplified version of IFC and generalized version of IMC.
2. Close-form solutions provide a more efficient way to conduct multi-fidelity fusion with infinite fidelities, especially for high-dimensional problems.
3. The effectiveness of this approach is demonstrated through the testing of both simulated and real-world data.
4. The paper exhibits a clear structure and is straightforward to comprehend.

WEAKNESSES:
1. In experiment section, it seems the baseline methods are tested with default settings without fine tuning. The comparisons are not completely fair.
2. One big benefit using the ODE formulation is to extrapolate since the ODE formulation can capture the underlying dynamics between different fidelities. In the paper, there is no discussion regarding this matter.
3. \eta is set 0.5 and 0.75 in all experiment settings. It might be a bit high.

QUESTIONS:
1. Could you also provide some fidelity interpolation and extrapolation results?
2. Could you also give some results with much less high-fidelity samples.

LIMITATIONS:
The authors did not address the limitations in the paper.

I do not believe there is any potential negative societal impact of their work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-17

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nMulti-fidelity models are widely used for combining training data obtained from information sources with different degrees of precision or accuracy. More specifically, this allows for the combination of greater quantities of noisier but more cheaply-obtained examples with more faithful (but limited) data. In this work, the authors describe an extension to infinite-fidelity fusion that incorporates information contained within the fidelity indicator itself, while also mitigating issues relating to training time and complexity, as well as scalability to high-dimensional outputs. The authors also formulate a surrogate model than unifies a large selection of pre-existing multi- and single-fidelity models. Experiments on synthetic and real-world data indicate that the model obtains significant performance improvements over competing techniques, without incurring an unreasonably large speed penalty (compared to IFC).\n\nSTRENGTHS:\n- The problems investigated in this work, along with the associated solutions, are non-trivial, and the authors diligently include detailed derivations for all their contributions. \n- The improvements over IFC are well-motivated in this work, and I appreciated how there was a strong emphasis on computational complexity and training stability. Both of these are highly prized by practitioners, and I would expect the performance improvements reported here to be transferrable to other problem domains as long as the training process is stable.\n\nWEAKNESSES:\n- The paper is currently quite dense and difficult to follow at times. While I appreciate that the authors present several varied contributions here, I believe the presentation of the main paper could be improved further to highlight the key takeaways while deferring detailed derivations to the supplementary.\n- The paper bears very strong writing similarities to *GAR: Generalized Autoregression for Multi-Fidelity Fusion* by Wang et al., where some sentences are nearly copied in their entirety with only a single word replaced here and there. This is especially noticeable in the *Introduction* and *Background* sections of the paper, as well as some of the *Related Work*. The contributions themselves are different, although I am surprised that this 2022 paper is only given a cursory reference given the degree of similarity in the problem statement and experimental set-up.\n- Maybe I missed this while reading the paper, but while is the IFC method listed as IFC-GPT in the figures and tables?\n- A handful of limitations for this method are listed at the very end of the paper, but these currently come across as an afterthought. I would prefer to see additional ablation studies or synthetic examples showing specific situations where the proposed models may not work as well as expected.\n\nQUESTIONS:\nI have listed my concerns with the paper in the *Weaknesses* section. I encourage the authors to focus on these comments when continuing the discussion on the paper.\n\nThe contributions in this paper are insightful, and could inspire further research within the community. However, I currently have major concerns on the writing and novelty of the paper (which are very similar to a pre-existing paper that is only trivially referenced in the submission - I am personally not comfortable with this degree of overlap), as well as clarity and presentation. In view of the above, I am currently inclined towards rejecting this submission, but I look forward to reading the feedback from other reviewers as well as the author rebuttal.\n\nLIMITATIONS:\nThere shouldn't be any immediate negative societal impact resulting from this work.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a Gaussian process (GP) based multi-fidelity model that makes use of fidelity indicators. This paper extends the autoregression two-fidelity formulation to a linear fidelity differential equation. By assuming the lowest fidelity function and all the residual functions follow GP, a joint GP model of all the fidelity can be derived. To allow flexible kernel choices, the integral in the kernel function is approximated with Monte Carle samples. In the case of multi-dimensional observations, the proposed model assumes a coregionalization formulation. To further speed up inference, the proposed model relies on the assumption that the inputs of high fidelity are a subset of the inputs of low fidelity. The proposed method compared to state-of-the-art multi-fidelity methods on both synthetic and real data sets and shows significant improvement on mean prediction accuracy.\n\nSTRENGTHS:\n- This paper extends the common autoregression multi-fidelity formulation to a linear  linear fidelity differential equation, which results in a joint GP model over the observations of all the fidelity.\n- The proposed method makes an explicit assumption about the role of fidelity indicator in the model, which allows it to use this information for modeling.\n- With a sophisticated GP model, the proposed method requires less training time and works better with low data compared to the neural network based multi-fidelity method.\n- The proposed method significantly outperforms state-of-the-art multi-fidelity methods on both synthetic and real data.\n\nWEAKNESSES:\n- The modeling assumption in the linear fidelity differential equation formulation is quite restrictive, which may not be applicable for many real world problems. For example, this model is not very effective if low fidelity data is only good at certain area, i.e., the knowledge transferring factor needs to depend on input x.\n- For simplicity, the proposed method assumes that $\\beta(t)$ is a constant. This means that the knowledge transferring factor is full determined by the fidelity indicator, which may be too restrictive for the use case where the fidelity indicator only shows the order of fidelity not the relative quality.\n- The proposed method jointly models the observed data of all the fidelity under a single GP model, which does not scale well when a lot of data are available.\n\nQUESTIONS:\n- In the experiments, the authors attribute the performance improvement to the usage of fidelity indicator. However, compared to AR, two factors potentially contribute to the performance difference: the joint modeling of all the fidelity data (instead of pairwise modeling of two consecutive fidelities) and the explicit usage of fidelity indicator in the linear fidelity differential equation. I wonder which factor is more important to the performance difference.\n- A big benefit of using GP is uncertainty quantification. In the experiments, only the accuracy of mean prediction is compared. I wonder what is the performance of the proposed method in terms of test log-likehood compared to the methods like AR.\n\nLIMITATIONS:\nThe limitation of the proposed method has not been sufficiently discussed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes an implementable and concrete version of Li et al's infinite dimensional fidelity DE, an method of fusing simulations at different levels of fidelity/resolution, to trade off between computational tractability and statistical accuracy.\n\nSTRENGTHS:\nif i understand correctly the \"infinite-fidelity\" model of Li et al has many attractive data fusion properties for varaible-resolution simulation, but is not implementable.\nThe claim of this paper is that an this specific parameterisation to the infinite-fidelity approach can be implemented with the GP induced by a linear ODE with a GP prior over functional inputs (to the ODE), which induces a GP posterior. Some fancy work with inducing points is done to make this tractable in practice.\n\nWEAKNESSES:\nmany small typos, and some odd phrasing that undermine my confidence in the results. See questions.\n\nThe paper seems not to be about design of experiments but rather sharing uncertainty between low and high-fidelity simulations that have already been performed, and yet lacks a justification for the informativeness of the low-fidelity simulations.\n\nI suspect this paper could be great with some typo- and bugfixes, but in the current form I hesitate recommend with confidence. There are too many confusing things to be sure I have understood the paper correctly.\n\nI think a simple diagram or two could have made this much clearer.\n\nQUESTIONS:\nequation (1) is central in the paper, and yet I don't understand it. Perhaps I am misreading, or perhaps a typo? in what sense is it _auto_-regressive? This seems to be a linear relation between two fidelities, $t=0,t=T$. Do you want it to relate two variable fidelities, not simply the two fidelities $i=0$ and $i=T$?\nThings look more autoregressive in eq (4) but I'm not sure I understand that either. What is $\\Delta$ doing here? Maybe this is a  notational quibble, but It is multiplying the difference between $t_T$ and $t_0$. If we let $\\Delta(t_T-t_0)\\to 0$, does that mean that $t_T\\to t_0$ *and* $\\Delta\\to 0$ simultaneously? I think it makes more sense if we delete $\\Delta$ and let $t_T\\to t_0$, esp in the light of (5)\n\nAlso, what is the relationship between times and fidelities? Does the system we are looking at have a time dimension, and each simulation run has a different time and spatial discretization? In which case, how do I interpret something like $t_T$ as in eq(4)? Shouldn't each fixed fidelity have a _sequence_ of different timesteps at which the entire simulation is evaluated?\n\nin (19) our virtual sites look a lot like the inducing points of sparse GPs. Is that how I should be interpreting them?\n\nl70:\"the system inputs of higher-fidelity are chosen to be the subset of the lower-fidelity, i.e., $\\mathbf{X}^T \\subset \\cdots \\subset \\mathbf{X}^2 \\subset \\mathbf{X}^1$.\" OK, I think I must be confused; what exactly is being subset here? If the higher-fidelity model is sampled over a denser mesh than the lower one, for example, then its data points should be, if anything, a superset of the mesh points of the lower-fidelity model. So I guess it is not mesh points; what is in a subset relation with what then?\n\nl177: I got lost trying to understand the subset selection here. Can you diagram it? I suspect this is very simple, but I just can't parse the sentence \"setting, such a requirement is not practical. Here, we derive a decomposition by introducing virtual observations $\\hat{\\mathbf{Y}}$ for each fidelity such that $\\mathbf{Y}^{(T)}$ satisfies the subset requirement for the completed set $\\left\\{\\mathbf{Y}^{(T-1)}, \\hat{\\mathbf{Y}}^{(T-1)}\\right\\} . \\check{\\mathbf{Y}}^{(T)}$ is the part of $\\mathbf{Y}^{(T)}$ that forms the subset of $\\mathbf{Y}^{(T-1)}$ (with a selection formulation $\\check{\\mathbf{X}}^{(T)}=\\mathbf{E}^{(T)} \\mathbf{X}^{(T-1)}$, where $\\mathbf{X}^{(T-1)}$ corresponds to the previous-fidelity outputs $\\left.\\mathbf{Y}^{(T-1)}\\right)$.\"\n\nGenerally are we missing something from the framing? What even is the fusion problem, if we already have a fixed library of simulations?  If I have run a high-fidelity simulation already then I have a maximally dense mesh of points for some version (with maximal $T$; Then what do I  gain by fusing it with lower-fidelity runs also? If the simulation is deterministic, which it seems to be, then we might as well just take that and go home. I thought that the fusion setting made sense in an _adaptive_ design-of-experiments setting where we might start from low fidelity model and up-sample as necessary to reduce overall uncertainty until we are satisfied. the model in this paper seems to assume a fixed set of simulation runs then pool them; but why would we do this, rather than simply throw out all but the highest-fidelity model? Is there some quantification of uncertainty we get from the lo-fi models which is not apparent at the high fidelity?\n\nBonus question about related research: The setting of the LiFiDEs, as one of the \"tractable fusion\" methods (l222) looks a lot like the \"probabilistic numerics\" setting -see e.g. https://www.probabilistic-numerics.org/research/pde/ where there are multi-resolution and meshless methods. Can you position this work in relation to that literature?\n\nLIMITATIONS:\nThe authors are transparent about the limitations of the method. Could probably test in settings where the model likelihoods are poorly approximated by Gaussians. Possibly some of the example problem do this; I have not have time to check the appendices to confirm this, however.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}