Data-driven Multi-Fidelity Modelling for Time-dependent Partial Differential Equations using Convolutional Neural Networks

Authors: freja t petersen
allan p engsig-karup
Venue: ICLR_cc_2024_Workshop_AI4DiffEqtnsInSci
Published: 2024-02-07 09:22:41

Abstract: We present a general multi-fidelity (MF) framework which is applied through utilizing flexible-order explicit finite difference numerical schemes using convolutional neural networks (CNNs) by combining low-order simulation data with higher order simulation data obtained from numerical simulations based on partial differential equations (PDEs). This allows for improving the performance of low-order numerical simulation through learning from the data how to correct the numerical schemes to achieve improved accuracy. Through the lens of numerical analysis we evaluate the accuracy, efficiency and generalizability of constructed data-driven MF-models. To illustrate the concept, the construction of the MF models uses CNNs and is evaluated for numerical schemes designed for solving linear PDEs; the heat, the linear advection equation and  linearized 1D shallow water equations. The numerical schemes allow for a high level of explainability of data-driven correction terms obtained via CNNs through numerical analysis of truncation errors. It is demonstrated that data-driven MF models is a means to improve the accuracy of LF models through operator correction.

Main paper: 
 introduction: 
 Convolutional neural networks (CNNs) are kernel-based local approximators which can be applied in the context of numerical solvers for partial differential equations. We investigate the usability of CNNs in a multi-fidelity (MF) setting using explicit flexible-order finite difference numerical solvers. The aim of generating a multi-fidelity model is to take advantage of combining expensive and often sparse high-accuracy data with cheap and easily obtainable low-accuracy data to improve accuracy of lower fidelity numerical models.Decades ago, neural networks were established as function approximators by Hornik et al. (1989) , which was later established in practice by Rico-Martinez et al. (1995) and González-García et al. (1998) . This is fundamental for solving e.g. dynamical systems and differential equations using neural networks. More recently, the universality of CNNs was established (Zhou, 2020) .Recently, the use of multi-fidelity techniques has been evolving using neural network based approaches. The term multi-fidelity techniques refers to methods that operate with different fidelities. The word fidelity is a broad term, which describes the overall quality or "truthness" of the model. The higher the fidelity, the closer to the truth the model is. The fidelity of the model can be controlled in terms of grid-discretization, order of the numerical method, and how well a model approximates the true system. Examples of previous neural network-based multi-fidelity models include a composite neural network that learns from multi-fidelity data (Meng & Karniadakis, 2020) , a long short-term memory network for multi-fidelity surrogate modeling (Conti et al., 2023) and a neural networkbased acceleration of a high-order discontinuous Galerkin solver (de Lara & Ferrer, 2023) . In the context of initial value problems for PDEs, methods using multi-resolution wavelet operator learning have been successful in beating state-of-the-art in terms of obtaining high accuracy for flexible resolutions (Gupta et al., 2021; 2022; Xiao et al., 2023) . All of these methods use multi-layer-perceptron (MLP) type networks, which are global in nature and hence difficult to scale. CNNs, on the other hand, use localized kernels that are executed across the domain. The advantage of this is that the computational complexity scales linearly with the input dimension. There are a few examples of CNN-based MF-models, including the a deep CNN-based MF-model for temperature field prediction (Zhang et al., 2023) and a CNN-structure called Deep FDM for enhancing finite difference methods (Kossaczká et al., 2023) . The latter argues for a strong connection between CNNs and numerical schemes. This work builds upon this idea, and explores it even further.The above section describes the use of neural networks and multi-fidelity models for solving PDEs. This work contributes to the area by creating a strong connection between numerical theory and CNNs, and the contributions include: 1) the formulation of a general framework for developing multi-fidelity models for future research and 2) data-driven multi-fidelity experiments with numerical simulators of different complexities. This work serves as a proof of concept, highlighting the potential of the proposed CNN-based multi-fidelity methods. 
methodology 2.1 a general multi-fidelity framework: 
 Consider the general initial value problem (IVP):EQUATIONwhere u ∈ R n holds n state variables and u 0 = u(t 0 ) is the initial state at time t = t 0 , and f (•) is assumed Lipschitz continuous with respect to the first argument. In the setting of time-dependent partial differential equations, it is common to discretize the PDEs using the method of lines (MoL) in two steps, where the first step involves numerical discretization in space to generate a semi-discrete system in the form of an IVP. The subsequent step, is to then solve this spatially discretised system via finite differences in the form of ordinary differential equation (ODE) solvers that provide stability and efficiency for the temporal integration. The general solution to the IVP can be stated asEQUATIONFor governing equations such as nonlinear PDEs, the integral cannot be readily evaluated. Instead it is possible to resort to numerical approximation.Following Chen & Xiu (2021) that introduced the generalised Residual Neural Networks (gResNet) for data-driven modeling of autonomous dynamical systems, we introduce an operator, L : R n → R n , such that:u(t k+1 ) ≈ L(u(t k )) ↔ u(t k+1 ) = L(u(t k )) + τ,where τ is the error (TE) of the numerical scheme that is dependent on the underlying solution u.To perform numerical analysis, the TE can be expanded in terms of a Taylor series to understand the structure of the leading order terms wrt. to the numerical discretization parameters.In the context of MF-models, L can be considered a low-fidelity (LF) operator, which maps the state variables in a time step from time t k to time t k+1 = t k + ∆t with ∆t the time step. The LF operator is assumed to be a convergent numerical scheme for finding approximate solutions to the IVP equation 1. Consider now the operator: F : R n → R n . This operator can be subject to function approximation, e.g., via numerical discretization or neural networks, and the update takes the form:EQUATIONIn the framework of equation 3, F(u(t k )) ≈ τ can be interpreted as a correction to the LF operator L(u(t k )), and the framework is therefore characterized as a multi-fidelity model when the correction is constructed based on utilizing data of different fidelities in a learning procedure.Consider a method of lines discretization of the IVP in equation 1 onto a uniform grid, such that u(x j , t i ) denotes the jth grid point of the spatial discretization and i is the time step. Consider an explicit low-order FDM with a stencil of half-width K and a TE, τ F DM . A temporal step with the FDM approximating equation 2 is: u(x j , t i+1 ) = F F DM (u(x j , t i )) + τ F DM , for i = 0, 1, ..., N T and j = 0, ..., m + 1 and where xj = [x j-K , ..., x j , ..., x j+K ]. The CNNbased MF-model consists of the FDM being the LF-operator, L, and an additive correction term represented via a CNN being an approximation to the TE term τ F DM . For example, the TE of the low-order FTCS-scheme for the heat equation is( ∆t 2 -∆x 2 12 ) ∂ 4 u ∂x 4 .This term is the difference between a fourth order and a second order method, which use the same time-stepping scheme. It is possible to approximate this TE using a local finite difference stencil for the fourth derivative and apply this stencil anywhere on the domain or for any initial value. The motivation behind using a CNN for approximating this term is that the convolutional layers consist of local kernels similar to those of a finite difference stencil, except they are trained using data from the higher-order numerical method. This work therefore investigates to what extent a CNN can represent the truncation error of the low-order method compared to a high-order method, which uses a different time-stepping scheme, and hence obtain a kernel-based local approximator to the truncation error.The architectures of the CNNs used for the testing in the following consist only of convolutional layers and activation functions (as described in appendix A.2). To limit the computational expense for the online phase of the simulation, the CNNs should be relatively shallow (i.e. few layers). Ultimately, a CNN of only two layers with five channels, kernels of size five and no activation functions is selected.The MF-framework is tested on the one-dimensional heat equation and the one-dimensional linear advection equation as well as the system of equations given by the linearized one-dimensional shallow water equations (LSWE) assuming small-amplitude waves. The heat equation is given by: ∂uEQUATION)where κ is the diffusion coefficient. The advection equation is given by: ∂uEQUATIONwhere v is the advection velocity. The LSWE are given by:∂u ∂t = -g ∂η ∂x , ∂η ∂t = -d ∂u ∂x (6)where g is the local graviational acceleration and d is the mean depth of the water and with initial values u(x, 0) = u 0 and η(x, 0) = η 0 . u is the velocity and η is the surface elevation of the water.Implementation details are found in appendix A.3. Table 1 shows the analytical solutions for testing the efficiency and accuracy of the MF-models for the heat and advection equation. The solutions are periodic and described on the domain x ∈ [-π, π]. The numerical models are initialized using the analytical solutions at time step 0, u 1 (x, 0) and u 2 (x, 0), respectively. The numerical solution corresponding to u 1 , is used for training the MF-model, whereas u 2 is only used for testing the generalization of the MF-model to unseen solutions. The data for training and testing is generated by sampling values of a, b and c from a uniform distribution. For the LSWE, a traveling wave solution is used as described in table 2 on the domain x ∈ [0, 1]. The constants are described in table 5 in appendix, where H is sampled in order to introduce variation in the training data.The equations are simulated using explicit finite difference methods (FDMs). The CNN is trained to correct a second or first order FDM using a fourth order FDM as reference. Hence, this paper relies on the use of flexible order FDMs and the strength of the MF-model is to avoid the strict stability requirements of using a higher order method. In the case of the LSWE, the HF-data used for training is based on a finer spatial and temporal discretization step by doubling the number of spatial grid points, resulting in a MF-model which captures both a higher order accuracy as well as a finer discretization. The training procedure is presented in appendix A.4Table 1 : Analytic, periodic solutions to the heat and advection equation. Considering the top row, the MF-model achieves better performance both in terms of accuracy and efficiency. The MF-model improves the error of the LF-model, while using fewer FLOPs than the HF-model, since it avoids the strict stability requirement. For the heat equation, there seems to be a saturation point in terms of accuracy, since the slope of the graph decreases with m. For the advection equation, it seems that a second order convergence rate is obtained, yet with lower error than the second order Lax-Wendroff scheme. These results show that the MF-model serves as an improved numerical scheme with properties (truncation errors and computational requirements) that are midway between known numerical schemes.u 1 (x, t) u 2 (x, t) Heat c + ae -b 2 π 2 t sin(bπx) c + a cos(bπx)e -b 2 π 2 t + sin( b 4 πx)e -b 2 16 π 2 t Advection a sin(bπ(x -vt)) a cos(bπ(x -t)) + sin( b 4 π(x -t))The bottom row, showing results for second initial value function u 2 (x, 0) without retraining the CNN, is promising. It is seen that the error of the MF-model is still lower than the error of the LF-model. Though, for the heat equation, the errors are closer to the LF-model than in figure 1a .For the advection equation, the result is approximately the same as in figure 1b , despite the fact that the CNN is trained on a completely different initial value problem. This indicates a property of generalizability of the CNN to unseen initial values. Considering the trade-off between FLOPs and errors, the advantage of the MF-model is not as clear for the heat equation as it is for the advection equation. For the heat equation, the computational requirement of the MF-model is midway between the LF-and HF-model, but the HF-model achieves much better accuracy. Hence, the MF-model would probably not be the first choice. Though, for the advection equation, the generalization properties together with the fact that the error of the MF-model is comparable to the HF-model while using fewer FLOPs, the advantage of the MF-model is more clear.  
conclusion and outlook: 
 In conclusion, this paper shows the potential of CNN-based MF-models for achieving an explainable and interpretable MF-model, which succeeds in correcting the error of a low-order explicit finite difference solver to achieve higher order for a set of linear time-dependent PDEs. The paper further shows the potential to combine the use of flexible order methods with varying grid discretizations in order to ensure speed-up of the MF-model. In ongoing work, we will consider the construction of MF-models for nonlinear and more complex time-dependent partial differential equations where flexible-order numerical schemes are utilized for data generation.For the advection equation, the CNN takes two time steps as input (N in = 2) and in the other cases, only the previous time step is taken as input (N in = 1). The CNN takes the current state as input and outputs a correction to the FDM approximation of the next time step. The heat equation is simulated using a forward-time-central-space (FTCS) discretization, which is first order accurate in time, and second and fourth order accurate in space for the low-and highfidelity simulations, respectively. The low-order simulation for the advection equation is based on the upwind method, which is first order accurate in time and space, and the high-fidelity simulation is based on the Lax-Wendroff method, which is fourth order accurate in the spatial discretization. The time-step, ∆t, is chosen small enough such that the order of the method corresponds to the spatial order. This means that the fourth order methods use a more fine time-step than the loworder methods. The LSWE are simulated using an explicit fourth order Runge-Kutta method for the time integration and a second and fourth order scheme for the spatial discretization of the low and high-fidelity models, respectively. The time-stepping of the LF-and HF-models is the same.Tables 3 and 4 The time-step is determined using the relation ∆t = c(∆x) q , where c and q are given in the table. Note, even though these equations can be used to model physical systems, this work is primarily an academic contribution, and, therefore, units are left out. In a physical system the range of the variable x would be given in meters and the time would be given in seconds.Table 3 : Specifications of the simulation domain for the two test-equations. N train is the size of the training set, N val is the size of the validation set, T end is the end-time of the simulation, x-range is the range of the spatial domain. Table 5 shows the characteristic constants of the traveling wave solution to the LSWE.Table 4 : Specifications of the simulation domain for the two test-equations and for each type of model (HF is high-fidelity and LF is low-fidelity). The problem-dependent, c, and the order of the numerical scheme, q, correspond to the relation ∆t = c(∆x) q , which differ for each equation and model.  
 
 Appendix: = 2π L = 2π m -1Specifications of the simulation with LSWE can be seen in table 6 . The first table shows the domain range and the grid resolution, and the second table shows specifics of the LF-and HF-models. The LF-model is based on a stencil with a half-width of α = 1 (stencil of size 3), and the HF-model has a stencil with half-width α = 2 (stencil of size 5). The choice of implementation method means that the restrictions on ∆t in the relation ∆t = c(∆x) is the same for the two models. Figure 5 shows the analytical solution to the LSWE for a chosen value of a. Table 6 : Specifications of the simulation domain the LSWE and for each type of model (HF is high-fidelity and LF is low-fidelity). N train is the size of the training set, N val is the size of the validation set, T end is the end-time of the simulation, x-range is the range of the spatial domain, m is the number of interior grid points and ∆x is the grid spacing. In the second table, the α column denotes the half-width of the stencil used for the spatial derivative in the explicit fourth-order 5stage Runge-Kutta method used for the simulation. A higher α means a higher order, since α is the half-width of the finite difference stencil. The training procedure is outlined in algorithm 1. One for-loop surrounds two sets of nested forloops. The outer-most loop, iterates through a number of epochs (iterations), and makes sure that the training and validation procedures are repeated N epochs times. In the first set of nested for-loops, the training procedure is carried out. The outer loop iterates through the time-steps whereas the inner loop iterates through the training data. Thus, the loop is constructed such that all training data is visited for one time-step before moving on to the next time-step. For each time step, the function output is calculated using a low-fidelity finite difference method, F LF , and an additive CNN, F CN N .The loss, when compared to the correct output obtained from the training data, is calculated using a mean-squared-error (MSE) function. The gradient of the loss-function is used to update the weights of the CNN according to the Adam optimizer.The next set of nested for-loops constitutes the validation procedure. It is only accessed if the training loss is below some tolerance in order to save computation time. For the validation procedure, the outer loop iterates the data and the inner loop iterates the time-steps, such that one set of data is iterated through before moving on to the next. The CNN is not updated in the validation loop. The validation loss is based only on the last time step. If the validation loss is lower than for the previously saved model, the new model is saved. 



Reviews: review: title: title: This article presents a data-driven multi-fidelity (MF) modeling approach that achieves better accuracy compared to LP approach and better FLOPS compared to HP approach on heat equation, linear advection equation, and linearized shallow water equations.
main_review: main_review: A) The authors can use more dimensions in the study to compare LF, HF and MF. From Figure 1 and Figure 2, at some times, the error of HF is much smaller than that of MF, but it's FLOPS is not much greater than that of MF. Therefore, the author needs to explain the superiority of MF from more perspectives.
B) The description of solutions in Section 2.3 is a bit confusing. I hope the author can provide further explanations.
C) The discussion of related work is very thin, there have been many efforts to learn from low fidelity complex PDEs and solve or forecast PDEs at other resolutions and fidelities. Here are some recent examples:
- "Non-linear operator approximations for initial value problems." In International Conference on Learning Representations (ICLR). 2022.
- "Coupled Multiwavelet Operator Learning for Coupled Differential Equations." In The Eleventh International Conference on Learning Representations. 2022.
- "Multiwavelet-based operator learning for differential equations." Advances in neural information processing systems 34 (2021): 24048-24062.
Particularly, the initial value problem has been actively considered in the neural operator literature and so it should be accurately discussed.
D) As the author stated, the performance of the model needs to be verified on more PDEs.
E) What are the main factors that affect the multi-fidelity and why the CNNs are the right approach? I think this is an interesting study but need to be put I the general context so that we can see when we should use it in combination with other techniques.