Infinite-Fidelity Coregionalization  for Physical Simulation
Authors: Shibo Li
Zheng Wang
Robert Kirby
Shandian Zhe
Venue: NeurIPS 2022 Accept (2022)
Keywords: Multi-Fidelity Learning
Surrogate Modeling
Physical Simulation

Abstract: Multi-fidelity modeling and learning is important in physical simulation related applications. It can leverage both low-fidelity and high-fidelity examples for training so as to reduce the cost of data generation yet still achieving good performance. While existing approaches only model finite, discrete fidelities, in practice, the feasible fidelity choice is often infinite, which can correspond to a continuous mesh spacing or finite element length. In this paper, we propose Infinite Fidelity Coregionalization (IFC). Given the data, our method can extract and exploit rich information within infinite, continuous fidelities to bolster the prediction accuracy. Our model can interpolate and/or extrapolate the predictions to novel fidelities that are not covered by the training data. Specifically, we introduce a low-dimensional latent output as a continuous function of the fidelity and input, and multiple it with a basis matrix to predict high-dimensional solution outputs. We model the latent output as a neural Ordinary Differential Equation (ODE) to capture the complex relationships within and integrate information throughout the continuous fidelities.  We then use Gaussian processes or another ODE to estimate the fidelity-varying bases. For efficient inference, we reorganize the bases as a tensor, and use a tensor-Gaussian variational posterior approximation to develop a scalable inference algorithm for massive outputs. We show the advantage of our method in several benchmark tasks in computational physics. 

InfiniteFidelity Coregionalization for Physical Simulation
Shibo Li, Zheng Wang, Robert M. Kirby, and Shandian Zhe
School of Computing. University of Utah
Salt Lake City, UT 84112 {shibo, wzhut, kirby, che)@cs.utah.edd utah.
Abstract
Multi-fidelity modeling and learning is important in physical simulation related ap- plications. It can leverage both low-fidelity and high-fidelity examples for training as to reduce the cost of data generation yet still achieving good performance. While existing approaches only model finite, discrete fidelities, in practice. the feasible fidelity choice is often infinite, which can correspond mesh spacing or finite element length. In this paper, we propose Infinite Fidelity Coregionalization (IFC). Given the data, method extract and exploit rich information within infinite, continuous fidelities to bolster the prediction accuracy. Our model interpolate and/or extrapolate the predictions novel fidelities that are not covered by the training data. Specifically, we introduce a www-dimensiona latent output as continuous function of the fidelity and input, and multiple with a basis matrix to predict high-dimensional solution outputs. We model the latent output neural Ordinary Differential Equation (ODE) capture the complex relationships within and integrate information throughout the continuous fidelities. We then use Gaussian processes or another ODE to estimate the fidelity-varying bases. For efficient inference, we reorganize the bases as a tensor, and use a tensor Gasoo-G variational posterior approximation develop scalable inference algorithm for massive outputs. We show the advantage of our method in several benchmark tasks in computational physics.
continuous
Introduction
Many scientific and engineering applications demand physical simulations, for which the task is mainly to solve partial differential equations (PDEs) at a domain of interest. For example, to estimate the temperature change at the end of part, one might need to solve transient heat transfer equations over the part (Incropera et al., 2007). Due to the high cost of running numerical solvers, in practice it is often important to train a surrogate model (Kennedy and O' "Hagan 2000 Conti and O HHagan 2010). Given the PDE parameters and/or parameterized boundary/inttall conditions, we use the surrogate model to predict the tigh-dimensional solution field, rather than run the numerical solvers from scratch. In this way, we can greatly reduce the cost, because computing the prediction for a machine learning model is often much more efficient and faster.
However, we still have to run the numerical solvers to generate the training data for the surrogate model, which is costly and can be bottleneck. To alleviate this issue, wise strategy is to conduct multi-fidelity learning. High-fidelig examples are generated via dense meshes (or smaller finite elements), hence accurate yet expensive to compute; low-fidelity examples are generated with coarse meshes, which are much cheaper for computation yet quite inaccurate. Despite the (significant) difference in quality, the low-fidelity and high-fidelin examples strongly correlated since they based on the same equation(s) or physical laws. Many multi multi-fidelity surrogate modeling and learning methods have therefore been developed to effectively combine examples of different fidelities to
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

improve the prediction accuracy while reducing the cost of data generation, e.g. (Perdikaris et al.
2017 Parussini et al. 2017 Xing et al. 2021aa Wang et al., 202I).
While successful, the exiting methods only model finite, discrete fidelities, which usually corresponds to several pre-specified meshes (or finite elements). However, since the spacing of the mesh (or the length of finite elements) is continuous, its choice can be infinite and therefore corresponds to infinitely many fidelities. To extract and take advantage of rich information within these infinite, continuous fidelities, we propose IFC, an infinite-fidelity coregionalization method. Our model can flexibly estimate the complex relationships among these fidelities to bolster the predictive performance, and scale to high- -igh-dimessiona outputs, which are common in physical simulation. Specifically, we first introduce -ow-dimensional latent output, which continuous function of the input and fidelity. We model the latent output as an ordinary differential equation (ODE), where the dynamics (fidelity derivative) is a neural network with the input as the latent output itself plus the original input, i.e. neural ODE (Chen et al.) 2018.. In this way, we can capture the complex re relationships within and integrate the information throughout the continuous fidelities. To predict nigh-dimensional outputs, we multiply the latent output with a basis matrix. We place Gaussian process prior over the basis elements or use another demeentwwise ODE to capture the basis variations along with the fidelity. For scalable inference of the GP bases, we re-organize the basis matrix as a tensor and introduce tensor-Gaussian
Gaussian distribution as the variational posterior. In this way, not only can we capture the strong posterior dependency among the massive basis elements, posterior covariance matrix (which can be huge) and greatly reduce the parameters. We then use the Kronecker product properties and ODE solvers to develop an efficient variational inference algorithm.
also avoid estimating the full
For evaluation, we tested our method for predicting the solution fields of three benchmark PDEs, including Poisson's, Heat and Burger's equations. We also applied IFC in topology structure optimization and computational fluid dynamics (CFDs). The output dimension for these tasks varies from thousands to hundreds of thousands. In all the cases, IFC outperforms the state-of-the-art multi-fidelity learning methods by large margin. In addition, we examined the performance of IFC in making predictions with novel fidelities (other than the training fidelities). It shows that our model with the ODE bases can extrapolate the prediction to new fidelities higher than (i.e., more accurate than) the training fidelities. This opens up possibility to achieve high-fidelitg predictive performance by only using low fidelity data.
2 Background
Linear Model of Coregionalization. Many tasks demand learning a function of high high-ddmennsionl outputs, where the dimension of the input is relatively low. For example, given the scalar viscosity (input), want to predict the solution of the viscous Burger's equation a 128 x 128 grid some domain of interest (output). A popular and classical high sigh-dimessiona output regression method is Linear Model of Coregionalization (LMC) (Jounel and Huijbregts 1978). which introduces a low dimensional latent output h(x) = [bg (x),... ... he [x]] where each h2 R + and is the input dimension. LMC models the actual high- dimenssoil output E Rd by linearly combining the latent output elements with a basis matrix B- = [b.....,bbr bx]].
K
f(x) = Σ h = B h(x)
where K < d and each by E Rd. To flexibly estimate each latent output, we often use a Gaus- sian process (GP) prior (Rasmussen and Williams 2006). GP is a popular approach to estimate single-output functions. In general, suppose given the training data D = {(x1 (1),.,))..... Vi),..., (XN. MN }. we want learn a function g R² + R. With the GP prior over g. the function values [x(x)).... ... g(xx)] follow a multivariate Gaussian distribution, p(g X)
.N m, K), where m is the mean function evaluated at the training inputs, usually set 0, K is the covariance matrix, and each element K ij k(x;, x,) is a covariance (kernel) function of the inputs. The are often assumed to be generated from a Gaussian noise model, p(y g) = A N(ylg, I²I) where o² is the noise variance. We can marginalize out g to obtain the marginal likelihood, p(y[X)= = (yy/0, K K I)). The kernel parameters and noise variance can be estimated by maximizing the marginal likelihood. Due to the Gaussian form, given the new input x", the predictive distribution of g(x") is straightforward to compute, which is a conditional Gaussian.
observations y = [V....., yNl

---

m as a nonlinear function While we can jointly estimate the latent outputs and bases in I an effective approach is to conduct Principled Component Analysis (PCA) on the training data to identify the bases B, and then use the singular values as the training outputs to learn the latent functions hyg (x) with standard GP regression. We refer to this method as PCA-GP (Higdon et al. 2008).
Multi- f Coregionalization. Practical applications often allow us to collect data with varying fidelities to enable a trade-off between the cost and efficiency. For example, in physical simulation, one can adjust the mesh spacing or length of the finite elements in the numerical solver to generate solution examples at different fidelities. Many multi-fidelity models have been developed to synergize training examples of different fidelities. For example, (Xing et al. 2021a) recently proposed deep residual coregionalization which sequentially learns M PCA-GP models ....... Las. for the given M fidelities. At each fidelity m, it first uses the lower fidelity models to make predictions and then compute the residual error between the low-fidelity predictions and the training outputs at the current fidelity. Based on the residual, it performs PCA to find the bases and estimates the latent output via GP regression,
1m PCA-GP(X train R (rain
R train
R
Y-train
Σ 7- 1, (X
where (Xtrain Y is the training inputs and outputs at fidelity m, and (,(.) the prediction made by the PCA-GP at fidelity j. The prediction at the highest fidelity M is obtained by summing the predictions of all the M models. Other than the sequential training. the recent works of Wang et al. (202T): [i et al. (022) jointly learn the bases and latent output for every fidelity. To estimate the relationship of successive fidelities, they model the latent output at fidelity of the latent output at fidelity m
hm (x) = a (xm= (x) = B. hx).
(3)
where Bm is the basis matrix at fidelity m. h..(-) and (-) are the latent outputs at fidelity m and m - 1, respectively, and f. (x) is the prediction at fidelity m. To fulfill this auto-regression, Wang et al. (2021) proposed matrix GP prior over ax-) that introduces an additional dependency on the bases, while Li et al. (2022) used a (deep) neural network to model (-)).
3 Model
Despite their success, the existing multi-fidelity approaches only model or estimate the relationships between finite, discrete fidelities. In physical simulation, these fidelities often correspond to several specific mesh spacings or finite element lengths. However, since the mesh spacing or element length is continuous, we actually have infinitely many possible choices, which correspond to infinitely many fidelities. Among the continuous, infinite fidelities are much richer information or relationships that can be valuable to promote the predictive performance. To extract and take advantage of this information, we propose IFC, an infinite-fidelity coregionalization model.
Specificallyy, since the fidelity m can be viewed as continuous (corresponding to the continuous mesh spacing and finite element length), we model the latent output as a continuous function of the input and fidelity, i.e., h(x, m.. Inspired by the residual coregionalization of [ing et al. (202 (see (2), we model the latent output- which can he viewed as low-rank summary of the actual high-dimensional output- as the latent output at the preceding (lower) fidelity, plus an adjustment/coreecioon for the current fidelity,
h(m, x) = h(m - A, x) + 4b.
(4)
where A > is an infinitesimal and sa is the correction term. To capture the complex yet strong relationship with the proceeding fidelity m - A, we model sb as a function of the latent output at m A. the current fidelity m, and the input: &b = &b (m, h(m x,)) Since lim sb 0, is natural assume xb
A00
A Φb. Therefore, we have
h(m, x) = h(m - A, x) + A & (m, h(m - A, x).
Moving h(m A, x) to the left, dividing the equation by A, and taking the limit of A to zero, arrive at an ODE model,
oh(m, ob (m, h(m,)),x).
(5)
dm
3

---

Without loss of generality. we assume the lowest fidelity is O. We then model the initial state of the ODE, i.e., the latent output at the lowest fidelity, as function of the input x,
(6) To flexibly estimate 33 and Φ. we parameterize them as neural networks. The advantage of our modeling is two-fold. First, according to (5) and (6), the prediction an arbitrary fidelity m is h(m,x) x) = h(0, x) + fo & (u, h(,, x). x) de, which integrates the predictions from all possible lower fidelities. Thereby, it enables us to exploit information from infinite, continuous fidelities. Second, learning dynamics Φ via neural networks enables us to capture the complex relationships among these continuous fidelities so as to bolster the predictive performance. The above component is an instance of neural ODE (Chen et al. 2018), and a continuous extension of the auto-regressive model
h(0,x) x) (xx).
Similar to LMC (see high-dimensional output at fidelity m. However, the bases can vary along with the fidelity. To flexibly capture such variations, we propose two methods.
①..
we multiply the latent output h(x, m) with a basis matrix B to obtain the
IFC-GPODE We model each basis element biy as function of the fidelity and place GP prior,
biy(m) ~ GO( k(m,m'))
(7)
where x(..-) is the kernel function. Suppose we have collected a set of training examples DD = {(Xn: Mi, yn]] We use Gaussian noise model to sample the observed data. The joint distribution is given by
d K IIII N ((b,, 0, K) =1k=
N II n=1
N( (,, Xm),
(8)
where X = (xi,..., XN). y = (y1,...,yN) B fby)hssaaassk bi (11((1)...... bey (ST)) is the basis values at different fidelities, {sy)T=1 are the distinct fidelities in the data, K is the kernel matrix on {sy}. and B, = [bs (mm))ssssd.1s fidelity my. Note that the latent output h(m,,, x.) is the state of the ODE system in (5) and (6).
is the basis matrix at
IFC-ODE2 Our second method to model each element bij with another ODE system,
(b:)(m) Ybee, m), (iy(0) = Vijo
dm
where y is parameterized by neural network. In this way, we can flexibly capture the evolution of the bases along with the fidelity. The joint distribution is
N
p()X)) = II Wyy, (,,((m, (a) x o²I)
(10)
==1
where both B, and h are computed from ODE solvers.
4 Algorithm
We now present the inference algorithm. Both IFC-GPODE and IFC-ODE" demand we compute the gradient of the learning objective w.r. to the ODE parameters and initial states, i.e., the parameters for & and B in (5) and (6) and for 7 and Vig in ( This can be efficiently done by applying automatic differentiation during the numerical integration in ODE solvers (e.g., the Runge-Kutte method (Dormand and Prince 1980)). However, the computational graph can be memory intensive. When the memory is insufficient, we can use the adjoint state approach instead (Pontryagin) 1987] Chen et al. 2018), which constructs an adjoint backward ODE system. The gradient is computed by solving the adjoint ODE. We refer to the details in (Chen et al. 2018).
We estimate the parameters of IFC-ODE by maximizing the log joint probability 10) via stochastic optimization, which relatively straightforward. The learning of IFC-GPODE, however, is much more challenging in that we need to estimate the posterior distribution of the bases at the observed fidelities, B (bry) siiaa skKk. which consists of dKTT elements. The posterior distribution does not have closed form, and resort to the variational inference framework (Wainwright and

---

Jordan 2008). Since these bases are coupled in both the GP prior (across the fidelities) and the likelihood (across the outputs), they are strongly dependent in the posterior. Hence, it is natural to introduce multi-variate Gaussian distribution for B as the variational posterior to capture these dependencies. However, since the output dimension is often large, say, hundreds of thousands, the computation and storage of the posterior covariance matrix (dK' dK'T) is prohibitively costly or even infeasible. To sidestep this issue, one might consider the commonly used mean-field variational approximation (Wainwright and Jordan 2008), which uses fully factorized posterior. However, doing this will lose all the posterior dependencies and can result in inferior inference quality.
To address this issue, we use an idea of (Zhe et al. 2019 Li et al. 2021) to fold the output space into an R dimensional tensor space, d1 x... dR where d = 11 dr. For convenience, we set dj .... dR Vd. Then we can arrange B as d1 .... dR K x T tensor. To fully capture the posterior correlations while still achieving a compact parameterization, we introduce a tensor- Gaussian distribution as the variational posterior for the bases B. The tensor-Gaussian is straightforward extension of the matrix Gaussian distribution,
(BB) = TN (BIU, E,...., ER. ER+I, ER+2) = - (vec(B))vec((() E, & .... @ ER+2), (II)
where U is the posterior mean, and Σ, is the posterior covariance at each mode (1 S S 2 To ensure the positive definiteness, we parameterize each covariance matrix by Σ, = L, where L, is a lower-trianuular matrix (i.e., the Cholesky decomposition form). In this way. the total number of
T(T+1) Consider parameters for the posterior covariance reduced to Σ d+(d,+1) + K[K+1] + 2
2
10,, K 10, and = 100 as an example. If we fold the output into a three-dimension tensor, ie., R = 3, we only need 2 x 10 dKKT parameters to represent the whole dKT x dK'T covariance matrix. Thus, the number of variational parameters dramatically reduced (> 99.99%) while our variational posterior can still represent the strong posterior dependencies.
We then construct the variational evidence lower bound (ELBO) with the censor-Gaussian posterior L ()) log p(B,)|x) We maximize the ELBO to obtain the variational parameters U
((B)
and LL, ODE parameters, and noise variance o², We leverage the Kronecker product properties Stegle et al. 2011 to decompose the full covariance matrix and to simplify the ELBO,
= (BL))())))((B) +
(B) [logp(Y, Yn Xn. B)]
(12)
where
(L(g(B)(P((B)
= tr 1K- ER+2)
II r=1 tr (x(E)) +
R+1
2 ttr(K trr(K--Unp U R+2)
1
2 +
Σ R+2 dKT
dK log det(K)
log det 2,,,
(13)
2
2 r=1 dp
d
Eg [log p( Y Xn. B) log (og(2Ta")
202 (yTTyn
2y, 2y, E, (B, kn + Lr(B., [B, B,] 2 kn
where 2n himm, xx). U R+2 obtained by unfolding the mean tensor at mode R + 2, giving a T x dK matrix, E, [B.] is obtained by fetching the -th slice of U at mode R+ 2 and reshape it as d x K matrix, and E,, [ B.] (II ur(E,) ER+ 1 + E. [B.]] E. [B.]] The computation restricted to the covariance matrices at each mode and hence is much more efficient. Note that we can always choose enough large R ensure dæ is small (e.g., 100) that the computation of the mode covariance matrix is efficient and cheap. We can use any gradient- based optimization method to maximize the ELBO.
Related Work
Linear model of coregionalization (LMC) (Matheron 1982 Goulard and Voltz 1992) is a classical
framework to extend the standard GP regression for multi-output function estimation. There have been many instances and variants, such as intrinsic coregionalization (Goovaerts et al. 1997). PCA- GP (Higdon al. 2008), KPCA-GP (Xing et al.] 2016). and IsoMap-GP (Xing al. 2015). GP regression networks (GPRNs) (Wilson et al. 2012) place a GP prior over the basis elements in
5

---

LMC and model the bases as functions of the input as well. While more flexible, it brings in additional computational challenges. In addition to LMC. other oulti- regression approaches
include convolution GPs (Higdon 2002, Boyle and Frean 2005 Alvarez et al., 2019) and multi-task GPs (Bonilla et al., 2007| |007][2008| Rakitsch et al. 2013). They use kernel convolution and matrix GP priors to model the multiple function outputs. The sparse GP approximations were applied for
large output dimensions (Alvarez and Lawrence 2009. Álvarez et al. 2010). A great survey is given in (Alvarez al., 2012). The recent work of Zhe et al. (2019) tensorized the output space and learned set of coordinate features to scale up to massive outputs and to capture the output correlations. To scale up GPRNs to high-dimensional outputs. Li et al. (202T) tensorized the bases and latent output, and developed a structural variational inference with matrix Gaussian and tensor Gaussian posteriors. They also used the Kronecker product properties to simplify the computation. Hence, our approximation technique is similar to these works.
To fulfill multi-fidelity training, Perdikaris et al. (2017); Cutajar et al. (2019) learned sequence of GP regressors, where each GP is for one fidelity, and models the output as a function of the input and the prediction at the previous fidelity. Their model is an instance of deep GPs (Damianou and Lawrence, 2013 Hebbal et al. 2019). However, their method might not be amenable to a large number outputs, since these outputs will serve as a part of the input to the GP at the next fidelity, and henceforth greatly increase the input dimension of that GP model. [Wang et al. (2021) addressed this issue by fulfilling an auto-regressive structure over the low-dimensional latent outputs. They used a matrix GP prior to sample the latent output as function of the latent output at the previous fidelity, the input, and the bases. Li et al. (2022) instead used auto-segressive neural networks to model the latent output, and developed an active learning algorithm to dynamically query at new inputs and fidelities. In addition, recently Hamelijnek et al. (2019) developed a multi-resolution. multi-task (output) regression method based on GPRNs and mixtures of experts (Rasmussen and Ghahramani] 2002). which intends to integrate data collected by sensor networks. The network nodes can have multiple resolutions. Other most recent multi-fidelit models include (Wang and Lin 2020 Wu et al. 2022 Xing et al. 2021b), etc. All these works assume the fidelities are fixed and finite, and model the re relationship between these discrete fidelities. Our work is different in that we point out the presence and value of continuous fidelities, especially in physical simulation, and we develop a new method to capture and leverage the rich Knowledge/relationships within the continuous, infinite fidelities to further enhance the predictive performance.
Experiment
6.1 Predicting Solution Fields of Partial Differential Equations
We first tested IFC for predicting the solution fields of several benchmark PDEs in computational physics, including Poisson's, Heat and Burger's is equations KOlten-Ketle 2011). To collect the training data, we run the numerical solvers with several meshes. Denser meshes give examples of higher fidelities. The output vector comprises of the solution values on the grid. For instance, a mesh of size 50 x 50 corresponds to an output vector of 2, 500 dimensions. For Poisson's 's and Heat equations, we generated training examples of four fidelities, using 8 x 8, 16 x 16, 32 x 32 and 64 x 64 meshes, respectively. For Burger's equation, we used 16 16, 24 x 24, 32 x 32 and 64 x 64 meshes to generate four-fidelity training data. For all the PDEs, the number of training examples for each fidelity (from the lowest to highest) 100, 50, 20, and 5, respectively. For testing, we generated 128 examples with the highest fidelity. Both the training and test inputs were uniformly sampled from the domain (but non-overlapping). The input includes the parameters of the PDE, the boundary and/or the intial conditions. The input dimension for Poisson's, 's Heat and Burger's equations is five, three and one, respectively. Hence, the task is in essence to learn an low-to-high high mapping that maps the parameters that index PDE to the solution field of that PDE. The data generation followed the details as provided in (Wang et al. 202T).
Competing Methods. We compared with the following state-of-the-at art multi- fidelity high- dimensional output learning methods.
(1) DRC (Xing et al., 2021a (ttaat)tttt///: :/tp:::://tttttt thub. com/ wayXing/DC) deep residual coregionalization, which performs LMC on the residual error of the pre- dictions from the lower fidelities. The final prediction is the summation of the LMC prediction across all the fidelities. See Sec. 2 (2) MFHoGP Wang et al. ht2l/httss://githuubb. com GregDobby/ Multi-FFielity-HHa High- Order- Caussian PrrpeiiannPPcccsss--eeee for www.saalllssssllaaaliiiaaaaaaaaa uses matrix GP prior to construct nonlinear coregionalization (NC) model, and connects multiple
which

---

IFC-ODE
+
+
0.6
0.8
0.40
nRMSE
nRMS
ARMSE
0.25
0.4
0.2
0.15
0.2
10
15
20
10
15
20
10
15
20
K
K
K
(a) Poisson
(b) Heat
(c) Burgers
Figure 1: Normalized RMSE predicting solution fields of Poisson's. s, and Heat and Burger's equations. K is the dimension of the latent output.
NC models for multi-fidelity learning, one for each fidelity. To capture the correlation between successive fidelities, the matrix GP prior samples the latent output as random function of the latent output in the previous fidelity. (3) DMF (Li al. 2022)st //gi thub. com/ hts//shibo111 DMF AL neural network (NN) based
learning approach, where each NN models one fidelity. To synergize different fidelities, the latent output of each NN is fed into the NN for the next fidelity. The high-dimensional prediction at each fidelity is obtained through a linear transformation of the latent output. To verify if IFC can indeed better integrate information of distinct fidelities, we also tested (4) SF, the single-fidelity degeneration
our model, where prediction is f(x) = (gha(x), where Bn is static basis matrix, and ho (-) is neural network. SF uses all the training examples without differentiation. We denote our ODE based method using the GP prior over the basis matrix
by (5) IFC-GPODE and another latent ODE over each basis element by (6) IFC-ODE".
Settings and Results. All the methods were implemented by PyTorch (Pazkke except that DRC was implemented by MATLAB. For our method, we used torchdiffeq li- brary (https://githin)
2019).
to solve ODEs and compute the gradient w.r.t ODE parameters and initial states via automatic differentiation. the Runge-Kutta method of order with adaptive steps. For GP related models, in- cluding DRC, MFHoGP and IFC-GPODE, we used the square exponential (SE) kernel. The length-scale parameter was initialized to one. For our method, each component (db
We used
IFC ODE
+
IPC GPODE + SF +
DRC
MPHoGP +
DMF
0.9
B3 and y in Eq. 55. and (9)
employed two hidden layers with tanh the activation function. To handle continuous (infinite) delities, mapped the lowest
0.4
ARMSE
nRMSE
0.3
0.7
fidelity to m O, and highest 1. For simplicity, use a linear mapping from the
0.1
mesh size to the fidelity value m. Suppose the mesh for m = so sp, and for m 1 is 81 Then the fidelity of an arbitrary
0.5
15
10
15
20
K
K
(a) Topology Optimization
(b) Fluid Dynamics
mesh is m(s)
(s so)/(s1 - So). Note that this Figure 2: Normalized RMSE in predicting the optimal topological struc- just one way of indexing the tures and spatial-temporal pressure field of fluids.
mesh size (or spacing) by fidelity
values there can arbitrary other ways. The complex, possibly nonlinear relationships between the fidelities (or meshes) are captured by our neural ODE component (see (5)). Since DRC, MFHoGP and IFC demand the output dimension be same across different fidelities, we set the output dimension to the one at the highest fidelity (which is 64 64 4, 096), and used interpolation

---

TFCOOF IFCOFODE
(a) Poisson's 's equation
(b) Fluid dynamics 10
Figure 3: Local prediction errors. The leftmost column in (a) and (b) is the original solution. The other columns are the error fields based the prediction of cach method. The lighter the color, the smaller the error.
0.8
IFC-ODE IFC-GPODE
IFC-ODE: IFC-GPODE
nRMSE
6.2
nRMSE
0.2-
0.0
8
18
24
32
40
45
56
64
72
80
86
96
104112100
128
0.0
16
24
32
40
46
56
84
72
80
88
96
104 112120111
Mesh size
Mesh size
(a) Poisson's 's equation
(b) Hear equation
Figure 4: Normalized RMSE of the predictions with various fidelity values (m E 2.14.. The x-axis shows the corresponding mesh size, where corresponds to the 8 mesh and 2.14 the 128 128 mesh. The largest training fidelity (m corresponds to the mesh.
(or down sampling) to obtain lower dimension predictions to the data Zienkiewicz et al. 1977). For IFC-GPODE, the output is folded into a two-dimensional tensor. For DMF, we also used two hidden layers for each NN, and tanh activation, which consistent with the setting in (Li al. 2022). The number of neurons per layer was chosen from {10, 20, 30, 40, 50, 60). We found that more layers for both our method and DMF did not improve predictive performance. In addition, other activation functions, such as ReLU and LeakyReLu worsened the performance. This is consis- tent with the typical choice of the activation function physics-informat neural networks (Raissi) et al. 2019). We ran ADAM (Kingma and Ba 2014) to train all the models, except DRC that uses L-BFGS estimate the latent output (the maximum number of iterations was set 1,000). We used ReduceLROnF Pateeuu (al-Kababji al.) 2022) scheduler to adjust the learning rate from [10-3, 3,10- We set the maximum number epochs to 5,000, which ensured the convergence of every method. We verified K - the latent output dimension setting, we repeated the experiment for five times. The average normalized root-mean root-mean-ssuure--on -quare--rror (nRMSE) and the standard deviation of each method are reported in Fig.
from (5, 10, 15, 20). For each
As we can see, both IFC-GPODE and IFC-ODE" consistently outperform all the competing methods by a large margin. The prediction errors of IFC-GPODE and IFC-ODE? are much closer, as compared with their difference from the other methods. The both versions of IFC greatly outperforms SF, single-fidelity degeneration, and in most case SF is also worse than the competing finite fidelity models. This together shows the advantage of our fnfilite-fidelity modeling, and the improvement indeed from more effective usage of the training information across dinstinct fidelities.

---

6.2 Topology Optimization
Next, we applied IFC in predicting the optimal topology design structures. Topology optimization (TO) an important task in engineering design and manufacturing. In general, given the environmen- tal constraint, e.g. an external force, our goal is to find a layout of the give materials (e.g.. alloys) that maximizes/minimizes a property of interest, e.g., stiffiness. The standard TO solves a constraint optimization problem that includes a compliance objective and total volume constraint (Sigmund 1997). The computation of the objective often demands for solving associated PDEs, and hence is quite computationally expensive. Hence, we learn a surrogate model to predict the optimal structure outright given the constraint (input). We considered the design problem in (Keshavarzzadeh et al. 2018), which aims to find structure (discretized in [0. 1] x [0, 1)) with the maximum stiffness under a load on the bottom right half. The load (input) is expressed by the location (in [0.5, 1) and angle (in [0, 3 JD. The strength of the load is fixed. During the optimization, we need to repeatedly run numerical solver. To learn the surrogate model, we generated training examples with four fidelities, corresponding to 50 x 50, 60 x 60, 70 x 70 and 80 x 80 meshes. Again, we generated 100, 50, 20, 5 for each fidelity, and 128 examples at the highest fidelity for testing. We repeated the experiment for five times. The average nRMSE and standard deviation IFC-ODE" achieve much higher prediction accuracy than all the competing methods in all the cases. It is interesting to that the performance of our method kept improving with the increase of the latent output dimension. This might be because more latent output elements can summarize and propagate the fidelity information more comprehensively and accurately.
shown in Fig. 2h. IFCDE and
6.3 Computational Fluid Dynamics
Third, we applied IFC in predicting the simulation results of computational fluid dynamics. We considered flow driven by rectangular boundaries (Bozeman and Dalton 1973). The rectangular is in the domain [0, 1] x [0. 1]. Each of the four boundaries has a prescribed velocity. The spatial-tempont field can be computed by solving the incompressive Navier-Stokes (NS) equations (Chorin) 1968), which is known to be costly due to the complex behaviors under large Renolds numbers. We were interested in predicting the pressure field of the flow along with time in [0, 10). given the Reynolds number [10, 500]. We simulated training examples of four fidelities, with spatial meshes of size 32 x 32, 48 48, 64 64 and 80 x 80 respectively. The number of time steps was set to 20. Hence, the output dimension (at the highest fidelity) is 128,000. Similar to the previous experiments, we collected 100, 50, 20, and 5 examples for each fidelity, and 128 examples at the highest fidelity for testing. We examined the prediction accuracy of each method. For IFC-GPODE, the output is folded as 20 x 80 x 80 tensor. We repeated the experiment for five times and report the average nRMSE in Fig. 2b. We can see that, consistent with the previous comparison results, IFC (both versions) greatly outperforms all the competing baselines, which confirms the advantage of IFC in predicting complex physical simulation results.
Furthermore, to investigate the local errors in predicting individual solution outputs, we randomly selected four test examples for Poisson's is equation and fluid dynamics. We examined the absolute error of each method in predicting every output. For fluid dynamics, we restrict the prediction at t = 10. We visualized the error field for each example in Fig. 3 and c. As we can see, in most cases, the competing methods have dominant errors at several local places, e.g., MFHoGP in Fig. 3a (first three examples) and DRC in 3 b. By contrast, the local errors of IFC-GPODE and IFC-ODE? distributed much more uniformly, and much smaller than the competing methods (lighter colors). That means, their performance is much less restricted by a few local regions. This also leads to a better global error.
6.4 Interpolation and Extrapolation in Fidelities
Since our method models the output as the function of continuous fidelity m, it can predict the solution outputs at arbitrary m that is different from the training fidelity values, i.e., interpolation and extrapolation. Note that the current finite, discrete fidelity approaches cannot make such predictions. To examine the performance of our model in interpolating and extrapolating the fidelity of prediction, we tested on Poisson's and Heat equations. We generated four-fidelity training data, including 256, 128, 64, and 32 examples for the first, second, third and fourth fidelity, respectively. The corresponding mesh size is 8 x 8, 16 x 16, 32 x 32, and 64 x 64. The lowest fidelity m = O. and highest m 1. We set the latent output dimension to 20 and trained our model accordingly.

---

We then used the model to predict the solution at a variety of m values, which corresponds to new meshes. For example, m 1.29, 1.57, 2.14 correspond meshes of size 80 x 80, 96 96 and 128 x 128, respectively. We viewed the "gold-standard" solution mesh, under which we generated 256 test examples. We varied m E [0, 2.14), and examined the corresponding prediction errors as compared with the gold-standard solution. The results are reported in Fig. 4 As we can see, within the range of training fidelities, i.e., 0 I m S 1 and the corresponding mesh size less than 64 x 64, the prediction error of IFC-GPODE is consistently smaller than that of IFCODDE", especially at very low fidelities (e.g. the 8 x grid). IFC-GPODE achieves the smallest prediction error at 64 x 64). the performance of IFC-GPODE drops. By contrast, while when m < 1, the prediction error of IFC-ODE is slightly worse than IFC- when > 1, the performance of IFC-ODE? keeps improving: achieves the smallest error at the largest m (i.e., m = 2. corresponding to the 1 (i.e., highest training fidelity). The nRMSE of IFC-ODE2 at m = 1 and m = 2.14 is 0.036 vs. 0.018 and 0.074 vs. 0.061, for Poisson's 's and Heat equations, respectively. The results show that IFC- PECGE is better in interpolation but IFC-DDE is promising in extrapolation. This might be attributed to the GP used IFC-GPODE, which known interpolate well yet not good at extrapolation (Rasmussen and Williams 2006 The improved extrapolation performance of IFC-ODE" can be particularly usetul in practice. It allows us to train the surrogate model only using lower fidelity examples, but we can still expect to gain higher fidelity predictions, i.e., more accurate than the training data. Therefore, we can avoid generating very high-fideliig examples for training to further reduce the cost.
solved with the 128 x 128
1, i.e., the highest training fidelity. When m 1 (mesh size bigger than
128 x 128 mesh), which is smaller than the prediction
One major limitation of IFC is that the training is much slower than the other methods. For example, hitrreccccittttiinnnoo time of DRC, MFHoGP, DMF, on the dataset for Poisson's equation, the average and 7.84 seconds, respectively (K = 20). For IFC-GPODE and IFC-ODE? is 0.02, 1.05, 0.04, 4.95 the fluid dynamics, the average per-epochl-iteration time is 0.04, 1.28, 0.10, 14.65 and 21.54 seconds, respectively (K = 20). This mainly arises from the intensive computation in back-propagating the gradient throughout the numerical integration in the ODE solver. One might improve the speed by using lower order solvers or larger step-sizes, which, however, can hurt the accuracy of the gradient computation. Note that, after training, the prediction of IFC is instantly fast (as fast as the competing methods), because simply doing numerical integration is very efficient.
Conclusion
We have presented IFC, an infinite coregionalization method for physical simulation. Through ODE based modeling, our method can capture and integrate information from infinite, continuous fidelities to facilitate learning. Our algorithm can scale up to high-dimensional outputs. The experimental results have shown an encouraging improvement upon the existing finite, discrete fidelity methods. In the future, plan develop an active learning scheme for our model to further reduce the training data amount and to maximize the benefit-cost ratio.
Acknowledgments
This work has been supported by MURI AFOSR grant FA9550-20-1-0038 and NSF CAREER Award IIS-2046295.
References
Al-Kababji, A., Bensaali, F., and Dakua, S. P. (2022). Scheduling techniques for liver segmentation: Reducelronplateau vs onecyclelr. arXiv preprint arXiv:2202.06373.
Alvarez, M. and Lawrence, N. D. (2009). Sparse convolved gaussian processes for multi-output regression. In Advances in neural information processing systems, pages 57-64.
Álvarez, M., Luengo, D., Titsias, M., and Lawrence, N. (2010). Efficient multioutput gaussian processes through variational inducing kernels. Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pages 25.32.
10

---

Alvarez, M., Ward, W., and Guarnizo, C. (2019). Non-linear process convolutions for multi- gaussian processes. The 22nd International Conference on Artificial Intelligence and Statistics, pages 1969-1977.
Alvarez, M. A., Rosasco, L., Lawrence, N. D., et (2012). Kernels for vector-valued functions: A review. Foundations and Trends@ in Machine Learning, 4(3):195-266.
Bonilla, E. V., Agakov, F. V., and Williams, C. K. (2007). Kernel multi-task learning using task- specific features. In Artificial Intelligence and Statisties, pages 43-50.
Bonilla, E. V., Chai, K. M., and Williams, C. (2008). Multi-task gaussian process prediction. In Advances in neural information processing systems, pages 153-160.
Boyle, P. and Frean, M. (2005). Dependent gaussian processes. Advances in neural information processing systems, pages 217-224.
Bozeman, J. D. and Dalton, C. (1973). Numerical study of viscous flow in a cavity. Journal of Computational Physics, 12(3):348-363.
Chen, R. T., Rubanova, Y., Bettencourt, J., and Duvenaud, D. K. (2018). Neural ordinary differential equations. Advances in neural information processing systems, 31.
Chorin, A. J. (1968). Numerical solution of the navier-stokes equations. Mathematics of computation, 22104))745--
Conti, S. and O' Hagan, A. (2010). Bayesian emulation of complex multi-output and dynamic computer models. Journal of statistical planning and inference, 140(3):640 661661..6661...
Cutajar, K., Pullin, M., Damianou, A., Lawrence, N., and González, J. (2019). Deep gaussian processes for multi-fidelity modeling. arXiv preprint arXiv:190..07320.
Damianou, A. and Lawrence, N. (2013). Deep gaussian processes. In Artificial Intelligence and Statistics, pages 207-215.
Dormand, J. R. and Prince, P. J. (1980). A family of embedded runge-kutta formulae. Journal of computational and applied mathematics, 6(1):19-26.
Goovaerts, P. et al. (1997). Geostatistics for natural resources evaluation. Oxford University Press on Demand.
Goulard, and Voltz, M. (1992). Linear coregionalization model: tools for estimation and choice of cross- varis-var matrix. Mathematical Geology, 24(3):269-266
Hamelijnck, O., Damoulas, T., Wang, K., and Girolami, M. (2019). Multi-resolution multi-task gaussian processes. arXiv preprint arXiv:1906.084:
Hebbal, A., Brevault, L., Balesdent, M., Talbi, E.-G., and Melab, N. (2019). Multi-fidelity modeling using DGPs: Improvements and a generalization to varying input space dimensions. In NeurlPS Workshop on Bayesian Deep Learning.
Higdon, D. (2002). Space and space-time modeling using process convolutions. In Quantitative methods for current environmental issues, pages 37-56. Springer.
Higdon, D., Gattiker, J., Williams, B., and Rightley, M. (2008). Computer model calibration using high-dimensione output. Journal of the American Statistical Association, 103(482):570-588
Incropera, F. P., Lavine, A. S., Bergman, T. L., and DeWitt, D. P. (2007). Fundamentals of heat and mass transfer. Wiley.
Journel, A. G. and Huijbregts, C. J. (1978). Mining geostatistics, volume 600. Academic press London.
Kennedy, M. C. and O'Haga,, A. (2000). Predicting the output from a complex computer code when fast approximations are available. Biometrika, 87(1):1-13.

---

Keshavarzzadeh, V., Kirby, R. M.. and Narayan, A. (2018). Parametric topology optimization with multi-resolution finite element models. arXiv preprint arXiv:1808.1033 10367.
Kingma, D. P. and Ba, J. (2014), Adam: A method for stochastic optimization. arXiv preprint ar2iv:141..6980.
Li, S., Wang. Z., Kirby, R. M., and Zhe, S. (2022). Deep multi-fidelity active learning of high- dimensional outputs. Proceedings of the Twenty-Fifth International Conference on Artificial Intelligence and Statistics.
Li, S., Xing, W., Kirby, R. M., and Zhe, S. (2021). Scalable Gaussian process regression networks. In Proceedings of the Twenty-Nin -inth International Conference on International Joint Conferences on Artificial Intelligence, pages 2456-2462.
Matheron, G. (1982). Pour une analyse krigeante des données régionalisées. Centre de Géostatistique, Report N-732, Fontainebleau.
Olsen-Kettle, (2011). Numerical solution of partial differential equations. University of Queensland, Australia.
Lecture notes at
Parussini, L., Venturi, D., Perdikaris, and Karniadakis, G. (2017). Multi-fidelity gaussian process regression for prediction of random fields. Journal of Computational Physics, 336:36-50.
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019). Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32.
Perdikaris, P., Raissi, M., Damianou, A., Lawrence, N., and Karniadakis, G. E. (2017). Nonlinear information fusion algorithms for data-efficient multi-fidelity modelling. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, 273(2198):201077511
Pontryagin, L. S. (1987). Mathematical theory of optimal processes. CRC press.
Raissi, M., Perdikaris, P., and Karniadakis, G. (2019). Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378:686-707.
Rakitsch, B., Lippert, C., Borgwardt, K., and Stegle, O. (2013). is all in the noise: Efficient multi-task gaussian process inference with structured residuals. In Advances in neural information processing systems, pages 1466-1474.
Rasmussen, C. E. and Ghahramani, Z. (2002). Infinite mixtures of gaussian process experts. In Advances in neural information processing systems, pages 881-888.
Rasmussen, C. E. and Williams, C. K. I. (2006). Gaussian Processes for Machine Learning, MIT Press.
Sigmund, O. (1997). On the design of compliant mechanisms using topology optimization. Journal of Structural Mechanics, 25(4):493-524
Stegle, O.. Lippert, C., Mooij. J. M., Lawrence, N. D., and Borgwardt, K. (2011). Efficient inference in
gaussian models with' jidd observation noise. Advances in neural information processing systems, pages 630-638.
Wainwright, M. J. and Jordan, M. I. (2008). Graphical models, exponential families, and variational inference. Now Publishers Inc.
Wang, Y. and Lin, G. (2020). Mfpe-net: Multi-fidelity poostee-c neural process. arXiv preprint arXiv:2010.01378.
Wang, Z., Xing, W., Kirby, R., and Zhe, S. (2021). Multi-fidelity high-order Gaussian processes for physical simulation. In International Conference on Artificial Intelligence and Statistics, pages 847-855. PMLR.
12

---

Wilson, A. G., Knowles, D. A., and Ghahramani, Z. (2012). Gaussian process regression networks. In Proceedings of the 29th International Coference on International Conference on Machine Learning, pages 1139-1146. Omnipress.
Wu, D., Chinazzi, M., Vespignani, A., Ma, Y.-A. and Yu, R. (2022). Multi-fidd Multiifidelitty hierarchical neural processes. arXiv preprint (iV:2206.008722
Xing. W., Shah, A., and Nair, B. (2015). Reduced dimensional gaussian process emulators of parametrized partial differential equations based on isomap. In Proceedings of the Royal Society of London A: Mathematical, Physical and Engineering Sciences, volume 471, page 20140697. The Royal Society.
Xing, W., Triantafyllidis, V., Shah, A., Nair, P., and Zabaras, N. (2016). Manifold learning for the emulation of spatial fields from computational models. Journal of Computational Physies. 326:666-699 690.
Xing, W. W., Kirby, R. M., and Zhe, S. (2021a). Deep coregionalization for the emulation of simulation-based spatial-temporal fields. Journal of Computational Physics, 428:109984
Xing. W. W., Shah, A. A., Wang, P., Zhe, S., Fu, Q., and Kirby, R. M. (2021b). Residual Gaussian process: A tractable nonparametric Bayesian emulator for multi-fidelity simulations. Applied Mathematical Modelling, 97:36-56.
Zhe, S., Xing, W., and Kirby, R. M. (2019). Scalable high-order gaussian process regression. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 2611-2620.
Zienkiewicz, O. C., Taylor, R. L.,, Zienkiewicz, O. C.,, and Taylor, R. L.. (1977). The finite element method, volume 36. McGraw-hill London.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofss of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (c.g., with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
13

---

(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsoureing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "xhi45D5WSH", "reviewer": "Reviewer_rxat", "summary": "This is an interesting paper that seeks to propose novel machine learning methods to extract information within continuous and infinite fidelities to bolster the prediction accuracy. The key novelty here is the means to develop a infinite fidelity coregionalization method by introducing a low-dimensional latent output and multiply it with a basis matrix for solution output prediction. The experimental data is comprehensive. Overall, the work is well done and provides some new methodology to this field.", "questions": "1. The interpolation and extrapolation study was only provided on Poisson's and Heat equations, which are problems with relatively smooth solutions and this is possibly why the model can extrapolate to even high fidelities. Can the authors perform interpolation and extrapolation studies on Burger's equation and CFD examples?\n2. What is the Reynolds number in the CFD example? How does the algorithm perform on high Reynolds number (such as transient flow and turbulence flow) regimes?\n3. The authors have commented that a shallow NN (2 hidden layers) is sufficient. Does the number of required layers, as well as other hyperparameters, vary for different examples?", "limitations": "The discussions on limitations are adequate.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\nA novel infinite fidelity coregionalization method is proposed, which improves upon existed models with finite and discrete fidelities. The paper is well-written. Empirical results on different types of PDE problems are provided.\n\nWeakness:\nOne of the main advantage of multi-fidelity models is are the improve learning and sampling efficiencies. Therefore, besides the comparison on accuracy, it would be helpful to see the training time comparison for at least 1 of the problems. It would also help the readers to assess the method if the authors can present a comparison when using different amount of training data.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Ow8frKfLkyK", "reviewer": "Reviewer_vVWD", "summary": "This paper introduced IFC, an infinite-fidelity coregionalization method for physical simulation. They designed ODE-based modeling to capture information for continuous fidelities and combined it with  Gaussian processes or another ODE to estimate the fidelity-varying bases. The result shows their method outperforms the baselines among several benchmark prediction tasks at the highest fidelity level.", "questions": "1. What's the definition of fidelity $m$ (continuous case)? \n2. Figure 4 result is interesting, especially the extrapolation part. It will be great if you can add similar figures for the rest three experiments to see whether this trend happens in general.\n3. I don't see the case that the dataset has continuous and infinite fidelity. But your method considering continuous and infinite fidelity improves the performance. It will be great if you can explain more about the reasons.\n4. baselines are missed [1, 2, 3]. Including them will make the experiment results more strong.\n\n[1] Wang, Yating, and Guang Lin. \"MFPC-Net: Multi-fidelity Physics-Constrained Neural Process.\" arXiv preprint arXiv:2010.01378 (2020).\n[2] Wu, Dongxia, et al. \"Multi-fidelity Hierarchical Neural Processes.\" arXiv preprint arXiv:2206.04872 (2022).\n[3] Xing, Wei W., et al. \"Residual Gaussian process: A tractable nonparametric Bayesian emulator for multi-fidelity simulations.\" Applied Mathematical Modelling 97 (2021): 36-56.\n\n5. It will be great if the author could include other metrics for accuracy (negative log-likelihood) and uncertainty quantification (Continuous Ranked Probability Score [4] or mean interval score [4,5]).\n\n[4] Gneiting, Tilmann, and Adrian E. Raftery. \"Strictly proper scoring rules, prediction, and estimation.\" Journal of the American statistical Association 102.477 (2007): 359-378. \n[5] Wu, Dongxia, et al. \"Quantifying uncertainty in deep spatiotemporal forecasting.\" arXiv preprint arXiv:2105.11982 (2021).\n\n6. Does the proposed model support varying input and output dimensions at different fidelity levels?\n7. Does the proposed model support non-subset multi-fidelity data?\n\n\n", "limitations": "No potential negative societal impact I can see.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. Well-written paper in general.\n2. the experiment is extensive. Five benchmark study included. \n3. The performance of the proposed model seems good.\n\nWeaknesses:\n1. I disagree with the statement \"in practice, the fidelity choice is often continuous and infinite\". For all five experiments in the paper, the data has finite and discrete fidelities. In practice, people pre-generate the simulation data. The fidelity level is also pre-selected, which is also finite and discrete. Although the performance of the proposed method is good.\n2. The definition of fidelity $m$ seems confusing and not consistent. In the background, $m$ looks like a discrete value 1,2,3,.... but starting from the model section, $m$ becomes a continuous value. There's no explanation of how to map a fidelity level to $m$ value. \n3. Only one evaluation metric nRMSE is included. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "C4vXmUGM1Ex", "reviewer": "Reviewer_u8Vk", "summary": "The authors propose two novel methods for coregionalization, i.e., projecting multiple grid resolutions onto a common grid, similar to superresolution. While previous works assume a discrete number of resolutions, the proposed work contributes the first coregionalization with a *continuous* change in resolution. The methods combine neural-ODEs and GPs to project low-resolution inputs, parameters, and BCs onto a higher-resolution grid. The main idea is by training a neural ODE to interpolate the meshes, the NODE can learn from other meshes in the latent sstate and outperform discrete methods that learn one model per mesh. The authors provide background, methodology, and support the claims with extensive empirical results. The empirical results confirm that the proposed methods, IFC-ODE and IFC-GPODE, both outperform discrete methods for coregionalization.\n", "questions": "3.1 What are the stochastic variables, parameters, ICs, or BCs that vary in between samples in train and test dataset and what are their distributions? It remained unclear to me whether the proposed work is an emulator of a single PDE solution at different resolutions or an emulator of the PDE solver that works at different resolutions and parameters, ICs, BCs. L249 indicates that the proposed work emulated a PDE solver, but I would need confirmation by the authors.\n\n3.2 Can the authors add a complete in-/output diagram to the camera-ready version? It is unclear to me if the state with lowest- or 1-lower fidelity is used as input. The quality of the results would positively surprise me if, e.g., in the case of Navier Stokes, an 8x8 solution is the input and the output is an accurate 64x64 solution.  \n\n3.3 What are the assumptions on the grid that are being made?\n", "limitations": "4.1 The authors do not address limitations or potential negative social imapcts. \n\n4.2 It is not mentioned on which kind of grids or finite element topologies the proposed model would work. I am assuming that it would only work for equispaced grids. \n\n4.3 It would be helpful to explain the implications of Gaussian assumptions during modeling.\n\n4.4 The authors claim that \"the surrogate model [is trained] only using low-fidelity examples, but we can still expect to obtain high-fidelity predictions, i.e., more accurate than the training data\" [L389-391]. It seems to me that there is not sufficient evidence to support this statement and I would alter or explain it for a camera-ready version. I do not fully understand how the proposed model could create, for example, higher-order frequencies that have been seen in the training phase. \n", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "1. Strength:\n1.1 The broader research topic of ML-based surrogate modeling of PDEs is significant to computational fluid dynamics, climate modeling, chemistry, biology, etc. The narrow topic of coregionalization, i.e., projecting data from various resolutions to a common grid, or learning from data of various resolutions is relevant to practical settings. The more narrow topic of infinite resolutions would allow for higher flexibility in using the method in practice.   \n\n1.2 The authors provide extensive empirical results on five problem settings and compare to five relevant methods. While the comparative methods could have been selected broader, e.g., including GAN-, Flow-, or Diffusion-based superresolution, the selection seems sufficient. The empirical results support the claims of the paper.\n\n1.3 The author choose advanced methodology to handle GPs in high-dimensional settings. \n\n2. Weaknesses:\n2.1 It is still a bit unclear what exactly the contributions are if Zhe et al., 19 and Li et al., 21 have already provided scalable GPs for regression of latent outputs and bases. The authors acknowledge this in L218. I am assuming that the paper is the first paper to do learning-based coregionalization with continuous fidelities. Section 4 needs to clarify that matrix Gaussian distribution is taken from a different paper and only applied to coregionalization, here, as far as I understood. The clarity of the paper could be improved by adding a 'list of contributions' at the end of the intro.  \n\n2.2 The related works section is very detailed with respect to the most similar works. The related works section could be improved by mentioning a practical use-case of coregionalization, and deterministic and stochastic superresolution methods with GANs, Flows, or diffusion-based methods. \n\n2.3 The method is quite complicated as it mixes matrix GPs and neural ODEs. I am not sure if introducing active learning to the mix would make for a very interesting paper as it might become really complicated to use this method in practice. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept