Infinite-Fidelity Coregionalization  for Physical Simulation

Abstract: Multi-fidelity modeling and learning is important in physical simulation related applications. It can leverage both low-fidelity and high-fidelity examples for training so as to reduce the cost of data generation yet still achieving good performance. While existing approaches only model finite, discrete fidelities, in practice, the feasible fidelity choice is often infinite, which can correspond to a continuous mesh spacing or finite element length. In this paper, we propose Infinite Fidelity Coregionalization (IFC). Given the data, our method can extract and exploit rich information within infinite, continuous fidelities to bolster the prediction accuracy. Our model can interpolate and/or extrapolate the predictions to novel fidelities that are not covered by the training data. Specifically, we introduce a low-dimensional latent output as a continuous function of the fidelity and input, and multiple it with a basis matrix to predict high-dimensional solution outputs. We model the latent output as a neural Ordinary Differential Equation (ODE) to capture the complex relationships within and integrate information throughout the continuous fidelities.  We then use Gaussian processes or another ODE to estimate the fidelity-varying bases. For efficient inference, we reorganize the bases as a tensor, and use a tensor-Gaussian variational posterior approximation to develop a scalable inference algorithm for massive outputs. We show the advantage of our method in several benchmark tasks in computational physics. 

Reviews and author responses:

 Thanks for answering my questions. I've raised my score based on the response.
Good luck

 C6: Does the proposed model support varying input and output dimensions at different fidelity levels?

R6: Great question. Since the input to our model is the identify information of the problem, such as PDE parameters and IC/BC parameters, which we do not assume change along with the fidelity of the solver, the input dimension is fixed in our model. Our model supports varying output dimensions. Although the prediction of our model (at any fidelity) is of a fixed dimension, we use interpolation or down-sampling to align with the actual dimensions in the data (see line 295-298). 


C7: Does the proposed model support non-subset multi-fidelity data? 


R7: Yes, our model allows an arbitrary set of inputs and outputs at each fidelity. Our model does not require that the inputs of higher fidelity examples must be a subset of the inputs at lower fidelities. 


 Thanks for your valuable and insightful comments. Here are our responses. C: comments; R: response.


C1: I disagree with the statement "in practice, the fidelity choice is often continuous and infinite". For all five experiments in the paper, the data has finite and discrete fidelities. In practice, people pre-generate the simulation data. 


R1: Thanks for the question. We mean to point out that the range or the possible choice of  the fidelity in simulation is often continuous and infinite. That’s because the fidelity usually corresponds to the mesh spacing or finite-element length, which are continuous in nature. 
This does **not** contradict to the fact that the actual simulation data only includes a finite number of fidelities, because the dataset itself can only be finite, and cannot cover infinitely possible fidelities. As an analogy, suppose an ML model includes the temperature as one feature. Obviously, the range of the temperature is continuous, and it can take infinitely many values. However, no matter how much data we collect, we can only observe finite distinct temperature values, because the dataset is always finite, which cannot cover all possible temperature values. Note that the difference between our method and existing works is in the modeling perspective, rather than in the data used. The training data can be the same. Our method models the fidelities and their relationship in the whole range (continuous space), including those appearing in the data and those not. The existing works only focus on the a few fidelities present in the data and their relationships. For such set of finite fidelities, we can index them by integers, that’s why they are viewed as “discrete”.  We will highlight these to improve the clarify of our paper. 


C2: The definition of fidelity $m$ seems confusing and not consistent. In the background, looks like a discrete value 1,2,3,.... but starting from the model section, $m$ becomes a continuous value. There's no explanation of how to map a fidelity level to value. What's the definition of fidelity (continuous case)?


R2: Thanks for the comments and question. As discussed in R1, the existing works introduced in the background focus only on the fidelities present in the data. They used integers to index this set of finite fidelities, that’s why $m=1,2,3,\ldots$ In our work, since we notice that the fidelity is often determined by mesh-spacing, finite element length, and/or other continuous control variables in the simulation, and hence the fidelity is continuous in nature. In addition to the finite fidelities observed in the data, there are infinitely many other choices. Accordingly, in the model section, we use continuous $m$ to index the fidelity. Actually, we did explain in our experiment how to map a fidelity level to the value --- we used a simple linear mapping from the mesh length (fidelity level) to the fidelity value $m$; see line 282-293 for details. Thanks for the questions. We will clarify these differences in our paper. 


C3: Figure 4 result is interesting, especially the extrapolation part. It will be great if you can add similar figures for the rest three experiments to see whether this trend happens in general. 


R3: Great suggestion. We will surely supplement the figures for the rest experiments. 


C4: I don't see the case that the dataset has continuous and infinite fidelity. But your method considering continuous and infinite fidelity improves the performance. It will be great if you can explain more about the reasons. 


R4: As discussed in R1 and R2, since the dataset is finite, it can only include a finite number of fidelities, despite the range or choice of the fidelity can be continuous and infinite. The critical difference of our method from the existing works is in *the modeling perspective*, rather than the data. The dataset is the same. **The existing works only models those finite fidelities present in the data, and ignores infinitely many other fidelities.  Our work models the fidelities in their entire range, which is continuous and infinite, including both those present in data and those not.** We believe, capturing the relationships and integrate the information throughout a much *richer* set of fidelities can allow us to further improve the performance of the surrogate model. 


C5: baselines are missed [1, 2, 3]. Including them will make the experiment results more strong. It will be great if the author could include other metrics for accuracy (negative log-likelihood) and uncertainty quantification (Continuous Ranked Probability Score [4] or mean interval score [4,5]).


R5: Thanks for providing the references of these excellent works! We will cite and discuss them, and supplement the comparison in our paper. We will also add evaluation results based on other metrices as you suggested. 


 C8: it is not mentioned on which kind of grids or finite element topologies the proposed model would work


R8:  Please see R6. 


C9: It would be helpful to explain the implications of Gaussian assumptions during modeling 


R9: Great suggestion. The Gaussian likelihood (see Eq8&Eq10) is essentially equivalent to the square loss, the most commonly used loss function in machine learning and data science. The Gaussian prior over the basis elements (see Eq8) arises from the Gaussian process (GP) prior over the basis as a function of the fidelity; see Eq7 (the finite project of GP is Gaussian). GP is a powerful nonparametric prior over functions; it assumes the function is sampled by the realization of a Gaussian process governed by some covariance (kernel) function.  It does not pre-assume any parametric form of the function, and only models the correlations between the function values. Accordingly, it can automatically capture the complexity of the function (e.g., multilinear and nonlinear) according to the data. GP has been widely used surrogate learning (e.g., the competing baselines (Xing et al., 2021) and (Wang et al., 2021) ) and numerous other machine learning applications.  Thanks for your suggestion. We will supplement these explanations in our paper. 


C10: The authors claim that "the surrogate model [is trained] only using low-fidelity examples, but we can still expect to obtain high-fidelity predictions, i.e., more accurate than the training data" [L389-391]. It seems to me that there is not sufficient evidence to support this statement and I would alter or explain it for a camera-ready version. I do not fully understand how the proposed model could create, for example, higher-order frequencies that have been seen in the training phase.


R10: Thanks for your insightful comments. We never claimed that our model can guarantee to extrapolate the prediction to higher fidelities (than training data). Even in our experiments, only IFC-ODE^2 shows good extrapolation yet IFC-GPODE does not, and we did acknowledge that (see Fig. 4 and line 385-387). We mean to point out that, if it is the case that the extrapolation gives even higher-fidelity prediction in some problem, it will be very useful because we can avoid too expensive, very high-fidelity simulations for that problem. We will alter the tone and text to express our thought more accurately. Also, thanks for the example regarding higher-order frequencies. We do agree it is challenging, but if from the training data, our ODE model can capture how the frequency varies with the increase of the fidelity (for a toy example: the frequency grows quadratically with the fidelity increase), we might still be able to infer new frequencies that have not been seen in the training data. Of course, if the frequency change has nothing to do with the fidelity change, our model will be unable to infer new frequencies because it violates our model assumption. We will supplement all these discussions about the scope and limitations of our methods. 


 C4: What are the stochastic variables, parameters, ICs, or BCs that vary in between samples in train and test dataset and what are their distributions? It remained unclear to me whether the proposed work is an emulator of a single PDE solution at different resolutions or an emulator of the PDE solver that works at different resolutions and parameters, ICs, BCs. L249 indicates that the proposed work emulated a PDE solver, but I would need confirmation by the authors.

R4: Great question. Our work is “an emulator of the PDE solver that works at different resolutions and parameters, ICs, BCs” rather than “a single PDE solution”. That is, the input to our models consists of PDE parameters and/or parameterized IC/BCs and designated fidelity; our models predict the corresponding solution field at that fidelity. These parameters and variables are sampled uniformly to generate the training and test datasets (but we ensure there is no overlap).  This is the same as our competing baselines, e.g., (Wang et al., 2021). As mentioned in our paper (line 251), the data generation, including the PDE parameters, ICs/BCs, and the solvers, follow (Wang et al., 2021). The details are provided in the appendix of the paper (Wang et al., 2021). We will highlight these in our paper. Here we give a brief summary. 

For Burger’s equation, the PDE parameter is the viscosity and varies from [0.001, 0.1] in the training and test samples, with the fixed IC and BC: $u(x,0) = sin(x\pi/2)$ with the homogeneous Dirichlet boundary condition.


For Poisson’s equation, we use a rectangular domain and the Dirichlet boundary condition. The values of the four boundaries and the center of the domain are used as the input to our model,  hence varying among the samples.


For heat equation, we use a 2D spatial-temporal domain with the Neumann boundary condition. The input parameters to our model includes the flux rate of the left boundary, flux rate of the right boundary, and the thermal conductivity. They vary among the training and test samples. 


For computational fluid dynamics, it is described by 2D spatial-temporal domain (see line 340-342). The input parameters to our model include the tangential velocities of the four boundaries and the Reynolds number, which vary among the samples. 


For topology structure optimization, the input parameters to our model consist of the location and angle of the load on the structure, and vary among the training and test examples (see line 326-329). The detail of solving this problem is given by (Keshavarzzadeh et al.,2018) (cited by our paper at Sec 6.2).  

C5: Can the authors add a complete in-/output diagram to the camera-ready version? It is unclear to me if the state with lowest- or 1-lower fidelity is used as input. 


R5: Great suggestion! We will surely add such a diagram to make our paper more clear.  Actually, the state with ``lowest- or 1-lower fidelity’’ is indeed used as the input to compute the state with higher fidelities. This can be seen from the infinitesimal view of our ODE model:  see Eq4, the unlabeled equation under Eq4, and the surrounding text. We can see the state at a higher fidelity is determined by the state at the lower fidelity (with $\Delta$ difference). From the wholistic view, if we write down the general solution of the ODE model, it is given by
$$h(m,\mathbf{x}) = h(0, \mathbf{x}) + \int_0^m \phi(\tau, h, \mathbf{x}) \text{d} \tau$$
where $h(0, \mathbf{x})$ is the initial state, which corresponds to the lowest fidelity. We can see that the state with fidelity $m$ is computed by both the initial state and all the states at lower fidelities ($<m$). Therefore, the information from all the lower, continuous (infinite) fidelities are integrated to make predictions at the target fidelity. We will enrich our presentation to highlight this point.  

C6: What are the assumptions on the grid that are being made? 

R6: We actually do not make any specific assumption on the grid (position, shape, topology, size, etc.). This is up to the particular problem and the solver choice. We only assume the change of the grid (e.g., dense/coarse) can change the fidelity, and there can be infinite, continuous choices, e.g., based on the length of the intervals or finite elements. We believe this is a reasonable and widely applicable assumption. 


C7: The authors do not address limitations or potential negative social impacts. 


R7: Thanks for reminder. In principle, our method can be used in any physical simulation related applications. One potential negative social impact can occur if our method is used to design fatal weapons. One limitation of our method is that the training involves ODE solvers and is sequential in nature. Hence, it is not obvious about how to utilize parallel computing resources to accelerate the training. We will supplement all these discussions in our paper.


 Thanks for your valuable and insightful comments. Here are our responses. C: comments; R: response. 

C1: It is still a bit unclear what exactly the contributions are if Zhe et al., 19 and Li et al., 21 have already provided scalable GPs for regression of latent outputs and bases. ... I am assuming that the paper is the first paper to do learning-based coregionalization with continuous fidelities. Section 4 needs to clarify that matrix Gaussian distribution is taken from a different paper and only applied to coregionalization... The clarity of the paper could be improved by adding a 'list of contributions' at the end of the intro.

R1: Thanks for the great comments and suggestions. We do agree that the tensorization and Kronecker product properties have already been used in the prior works (as cited and acknowledged in our paper). Our contribution is obviously *not* the invention of these tricks. Instead, we believe the contribution is the novel combination of these techniques with ODE solvers (and/or adjoint state methods), to address the learning challenges of our newly proposed ODE-GP mix, which is the first model "to do learning-based coregionalization with continuous fidelities". We will follow your suggestions to make clarifications, and highlight the difference with the prior works, so as to make our contributions more clear.  

C2: The related works section is very detailed with respect to the most similar works. The related works section could be improved by mentioning a practical use-case of coregionalization, and deterministic and stochastic superresolution methods with GANs, Flows, or diffusion-based methods.

R2: Great suggestion! We will add the references and discussions accordingly. 


C3: I am not sure if introducing active learning to the mix would make for a very interesting paper as it might become really complicated to use this method in practice.

R3: Thanks for the concern. We believe active learning is a promising direction since it can help us to further reduce the simulation cost (in data collection) while improving the efficiency of surrogate learning. We do agree the potential complexity or challenge in developing an effective active learning approach for our models: how to design an acquisition function, and how to optimize it to find the new input and fidelity at which to query. But once the active learning approach is ready, its usage can be pretty convenient --- it just repeats three steps: identifying new query points (input and fidelity), acquiring the examples by calling off-the-shelf simulators, and retrain the surrogate model. All the three steps can be done automatically, with little human intervention. Therefore, despite the potential risk, we are still willing to study active learning in the future work. 

 Thanks for your valuable and insightful comments. Here are our responses. C: comments; R: response. 

C1: … besides the comparison on accuracy, it would be helpful to see the training time comparison for at least 1 of the problems … if the authors can present a comparison when using different amount of training data…

R1: Great suggestions! We do agree. We will supplement the training time of all the methods, and conduct a comparison with varying the training data amount. 

C2: The interpolation and extrapolation study was only provided on Poisson's and Heat equations, which are problems with relatively smooth solutions and this is possibly why the model can extrapolate to even high fidelities. Can the authors perform interpolation and extrapolation studies on Burger's equation and CFD examples?

R2: Thanks for the insightful comments and great suggestion. We do agree that the solutions of Poisson's and heat equations can be quite smooth. But the extrapolation performance of our  model mainly depends on if our model can accurately capture the *solution variation* (not the solution itself) along the fidelity increase. For example, the solution might be relatively less smooth at both fidelity $m-\Delta$ and fidelity $m$; However, if their change (or difference) w.r.t the fidelity change $\Delta$ is smooth (see Eq 4&5), our model might still be relatively easy to learn and utilize such change to obtain a good extrapolation. Thanks for your suggestion. We will conduct such studies on Burger’s equation and CFD problems to further investigate the performance of our method. 

C3: What is the Reynolds number in the CFD example? How does the algorithm perform on high Reynolds number?...

R3: The Reynolds number is one input (i.e., the PDE parameter) to our model, and is sampled from [10, 500], which is consistent with the experiment conducted in the baseline work (Wang et al., 2021). We will test on larger ranges to examine the performance of our method on much higher Reynolds numbers. 

C4: Does the number of required layers, as well as other hyperparameters, vary for different examples? 

R4: For our method, we used the same hyperparameters for all the tasks in the experiments (2 hidden layers, 40 neurons per layer, tanh activation, Rk45 solver, etc.). For DMF (baseline), all the hyperparameters do not change except for the layer width, and we selected it for different tasks following the original paper (Li et. al. 2022); see line 300-301. We will highlight this point in our paper. 


 This is an interesting paper that seeks to propose novel machine learning methods to extract information within continuous and infinite fidelities to bolster the prediction accuracy. The key novelty here is the means to develop a infinite fidelity coregionalization method by introducing a low-dimensional latent output and multiply it with a basis matrix for solution output prediction. The experimental data is comprehensive. Overall, the work is well done and provides some new methodology to this field. Strength:
A novel infinite fidelity coregionalization method is proposed, which improves upon existed models with finite and discrete fidelities. The paper is well-written. Empirical results on different types of PDE problems are provided.

Weakness:
One of the main advantage of multi-fidelity models is are the improve learning and sampling efficiencies. Therefore, besides the comparison on accuracy, it would be helpful to see the training time comparison for at least 1 of the problems. It would also help the readers to assess the method if the authors can present a comparison when using different amount of training data. 1. The interpolation and extrapolation study was only provided on Poisson's and Heat equations, which are problems with relatively smooth solutions and this is possibly why the model can extrapolate to even high fidelities. Can the authors perform interpolation and extrapolation studies on Burger's equation and CFD examples?
2. What is the Reynolds number in the CFD example? How does the algorithm perform on high Reynolds number (such as transient flow and turbulence flow) regimes?
3. The authors have commented that a shallow NN (2 hidden layers) is sufficient. Does the number of required layers, as well as other hyperparameters, vary for different examples? The discussions on limitations are adequate.

 The authors propose two novel methods for coregionalization, i.e., projecting multiple grid resolutions onto a common grid, similar to superresolution. While previous works assume a discrete number of resolutions, the proposed work contributes the first coregionalization with a *continuous* change in resolution. The methods combine neural-ODEs and GPs to project low-resolution inputs, parameters, and BCs onto a higher-resolution grid. The main idea is by training a neural ODE to interpolate the meshes, the NODE can learn from other meshes in the latent sstate and outperform discrete methods that learn one model per mesh. The authors provide background, methodology, and support the claims with extensive empirical results. The empirical results confirm that the proposed methods, IFC-ODE and IFC-GPODE, both outperform discrete methods for coregionalization.
 1. Strength:
1.1 The broader research topic of ML-based surrogate modeling of PDEs is significant to computational fluid dynamics, climate modeling, chemistry, biology, etc. The narrow topic of coregionalization, i.e., projecting data from various resolutions to a common grid, or learning from data of various resolutions is relevant to practical settings. The more narrow topic of infinite resolutions would allow for higher flexibility in using the method in practice.   

1.2 The authors provide extensive empirical results on five problem settings and compare to five relevant methods. While the comparative methods could have been selected broader, e.g., including GAN-, Flow-, or Diffusion-based superresolution, the selection seems sufficient. The empirical results support the claims of the paper.

1.3 The author choose advanced methodology to handle GPs in high-dimensional settings. 

2. Weaknesses:
2.1 It is still a bit unclear what exactly the contributions are if Zhe et al., 19 and Li et al., 21 have already provided scalable GPs for regression of latent outputs and bases. The authors acknowledge this in L218. I am assuming that the paper is the first paper to do learning-based coregionalization with continuous fidelities. Section 4 needs to clarify that matrix Gaussian distribution is taken from a different paper and only applied to coregionalization, here, as far as I understood. The clarity of the paper could be improved by adding a 'list of contributions' at the end of the intro.  

2.2 The related works section is very detailed with respect to the most similar works. The related works section could be improved by mentioning a practical use-case of coregionalization, and deterministic and stochastic superresolution methods with GANs, Flows, or diffusion-based methods. 

2.3 The method is quite complicated as it mixes matrix GPs and neural ODEs. I am not sure if introducing active learning to the mix would make for a very interesting paper as it might become really complicated to use this method in practice. 
 3.1 What are the stochastic variables, parameters, ICs, or BCs that vary in between samples in train and test dataset and what are their distributions? It remained unclear to me whether the proposed work is an emulator of a single PDE solution at different resolutions or an emulator of the PDE solver that works at different resolutions and parameters, ICs, BCs. L249 indicates that the proposed work emulated a PDE solver, but I would need confirmation by the authors.

3.2 Can the authors add a complete in-/output diagram to the camera-ready version? It is unclear to me if the state with lowest- or 1-lower fidelity is used as input. The quality of the results would positively surprise me if, e.g., in the case of Navier Stokes, an 8x8 solution is the input and the output is an accurate 64x64 solution.  

3.3 What are the assumptions on the grid that are being made?
 4.1 The authors do not address limitations or potential negative social imapcts. 

4.2 It is not mentioned on which kind of grids or finite element topologies the proposed model would work. I am assuming that it would only work for equispaced grids. 

4.3 It would be helpful to explain the implications of Gaussian assumptions during modeling.

4.4 The authors claim that "the surrogate model [is trained] only using low-fidelity examples, but we can still expect to obtain high-fidelity predictions, i.e., more accurate than the training data" [L389-391]. It seems to me that there is not sufficient evidence to support this statement and I would alter or explain it for a camera-ready version. I do not fully understand how the proposed model could create, for example, higher-order frequencies that have been seen in the training phase. 


 This paper introduced IFC, an infinite-fidelity coregionalization method for physical simulation. They designed ODE-based modeling to capture information for continuous fidelities and combined it with  Gaussian processes or another ODE to estimate the fidelity-varying bases. The result shows their method outperforms the baselines among several benchmark prediction tasks at the highest fidelity level. Strengths:
1. Well-written paper in general.
2. the experiment is extensive. Five benchmark study included. 
3. The performance of the proposed model seems good.

Weaknesses:
1. I disagree with the statement "in practice, the fidelity choice is often continuous and infinite". For all five experiments in the paper, the data has finite and discrete fidelities. In practice, people pre-generate the simulation data. The fidelity level is also pre-selected, which is also finite and discrete. Although the performance of the proposed method is good.
2. The definition of fidelity $m$ seems confusing and not consistent. In the background, $m$ looks like a discrete value 1,2,3,.... but starting from the model section, $m$ becomes a continuous value. There's no explanation of how to map a fidelity level to $m$ value. 
3. Only one evaluation metric nRMSE is included. 
 1. What's the definition of fidelity $m$ (continuous case)? 
2. Figure 4 result is interesting, especially the extrapolation part. It will be great if you can add similar figures for the rest three experiments to see whether this trend happens in general.
3. I don't see the case that the dataset has continuous and infinite fidelity. But your method considering continuous and infinite fidelity improves the performance. It will be great if you can explain more about the reasons.
4. baselines are missed [1, 2, 3]. Including them will make the experiment results more strong.

[1] Wang, Yating, and Guang Lin. "MFPC-Net: Multi-fidelity Physics-Constrained Neural Process." arXiv preprint arXiv:2010.01378 (2020).
[2] Wu, Dongxia, et al. "Multi-fidelity Hierarchical Neural Processes." arXiv preprint arXiv:2206.04872 (2022).
[3] Xing, Wei W., et al. "Residual Gaussian process: A tractable nonparametric Bayesian emulator for multi-fidelity simulations." Applied Mathematical Modelling 97 (2021): 36-56.

5. It will be great if the author could include other metrics for accuracy (negative log-likelihood) and uncertainty quantification (Continuous Ranked Probability Score [4] or mean interval score [4,5]).

[4] Gneiting, Tilmann, and Adrian E. Raftery. "Strictly proper scoring rules, prediction, and estimation." Journal of the American statistical Association 102.477 (2007): 359-378. 
[5] Wu, Dongxia, et al. "Quantifying uncertainty in deep spatiotemporal forecasting." arXiv preprint arXiv:2105.11982 (2021).

6. Does the proposed model support varying input and output dimensions at different fidelity levels?
7. Does the proposed model support non-subset multi-fidelity data?


 No potential negative societal impact I can see.

Meta-review: The paper tackles the multi-fidelity simulation problem by modeling the grid variation with NODE, coupled with a GP. Experiments on multiple physical simulators show better performance compared to baselines. Please also report computational efficiency and sample complexity in the final version.  

Decision: Accept