Extension of Physics-informed Neural Networks to Solving Parameterized PDEs

Authors: woojin  cho
minju  jo
haksoo  lim
kookjin  lee
dongeun  lee
  hong
noseong  park
Venue: ICLR_cc_2024_Workshop_AI4DiffEqtnsInSci
Published: 2024-02-09 18:54:41

Abstract: In this paper, we address PINNs’ problem of repetitive and time-consuming training by proposing a novel extension, parameterized physics-informed neural networks (P$^2$INNs). P2INNs enable modeling the solutions of parameterized PDEs via explicitly encoding a latent representation of PDE parameters. With the extensive empirical evaluation, we demonstrate that P$^2$INNs outperform the baselines both in accuracy and parameter efficiency on benchmark 1D and 2D parameterized PDEs and are also effective in overcoming the known “failure modes”.

Main paper: 
 introduction: 
 Scientific machine learning (SML) (Baker et al., 2019) has evolved rapidly, requiring exact satisfaction of important physical laws. Among various deep-learning models that encode such physical characteristics (Raissi et al., 2019; Lee & Carlberg, 2021; Cranmer et al., 2020b; Lee et al., 2021; Satorras et al., 2021) , physics-informed neural networks (PINNs) (Raissi et al., 2019) are gaining traction in the research community. This is primarily due to their sound computational formalism, which enforces governing physical laws to learn solutions.PINNs parameterize the solution u(x, t) of partial differential equations (PDEs) using a neural network u Θ (x, t) that takes the spatial and temporal coordinates (x, t) as input and has Θ as the model parameters. During training, the neural network minimizes a PDE residual loss (cf. equation 11) denoting the governing equation, at a set of collocation points, and a data matching loss (cf. equation 10 and equation 12), which enforce initial/boundary conditions, at another set of collocation points sampled from initial/boundary conditions. This computational formalism enables to infuse the physical laws, described by the governing equation F(x, t, u), into the solution model and, thus, is denoted as "physics informed". PINNs have shown to be effective in solving many different PDEs, such as Navier-Stokes equations (Shukla et al., 2021; Jagtap & Karniadakis, 2020; Jagtap et al., 2020) . While powerful, PINNs suffer from several obvious weaknesses. W1) PDE operators are highly nonlinear (making training extremely difficult); W2) Repetitive trainings from scratch are needed when solutions to new PDEs are sought (even for new PDEs arising from new PDE parameters in parameterized PDEs).There have been various efforts to mitigate each of these issues: (For addressing W1) curriculumlearning-type training algorithms that train PINNs from easy PDEs to hard PDEs1 (Krishnapriyan et al., 2021) , and (for addressing W2) meta-learning PINNs (Liu et al., 2022) ; or directly learning solutions of parameterized PDEs such that u Θ (x, t; µ µ µ), where µ µ µ is a set of PDE parameters. However, there has been a less focus on addressing both problems in a unified PINN framework. The first two encoders g θp and g θc are added to generate better representations for the PDE parameter and the spatial/temporal coordinate. We also carefully customize the manifold network g θg . In this figure, we provide the CDR equation as an example.To mitigate the both issues in W1 and W2 simultaneously, we propose a variant of PINNs for solving parameterized PDEs, called parameterized physics-informed neural networks (P 2 INNs). P 2 INNs approximate solutions as a neural network of a form u Θ (x, t; µ µ µ) (for resolving W2) and show reasonable accuracy (cf. Table 1 ) even for harder PDEs (for resolving W1). A novel modification in our approach is to explicitly extract a hidden representation of the PDE parameters by employing a separate encoder network, h h h param = g Θp (µ µ µ), and uses this hidden representation to parameterize the solution neural network, u Θ (x, t; h h h param ). Rather than simply treating µ µ µ as a coordinate in the parameter domain, we infer useful information of PDEs from the PDE parameters µ µ µ, constructing the latent manifold on which the hidden representation of each PDE lies.To demonstrate the effectiveness of the proposed model, we demonstrate the performance of the proposed model with well-known benchmarks (Krishnapriyan et al., 2021) , i.e., parameterized CDR equations. As studied in Krishnapriyan et al. (2021) , certain choices of the PDE parameters (e.g., high convective or reaction term) make training PINNs very challenging (i.e., harder PDEs), and our goal is to show that the proposed method is capable of producing approximate solutions with reasonable accuracy for those harder PDEs. 
proposed methods: 
 Now we introduce our parameterized physics-informed neural networks (P 2 INNs). In essence, our goal is to design a neural network architecture that effectively emulates the action of the parameterized PDE solution function, u(x, t; µ µ µ).For P 2 INNs, we propose a modularized design of the neural network u Θ (x, t; µ µ µ), which consists of three parts, i.e., two separate encoders g θp and g θc , and a manifold network g θg such thatEQUATIONwhere Θ = {θ c , θ p , θ g } denotes the set of model parameters (cf. Figure 1 ). The two encoders, g θc and g θp , take the spatiotemporal coordinate (x, t) and the PDE parameters µ µ µ as inputs and extract hidden representations such that h h h coord = g θc (x, t) and h h h param = g θp (µ µ µ). The two extracted hidden representations are then concatenated and fed into the manifold network to infer the solution of of the PDE with the parameters µ µ µ at the coordinate (x, t), i.e., û(x, t; µ µ µ) = g θg ([h h h coord ; h h h param ]).The important design choice here is that we explicitly encode the PDE parameters into a hidden representation as opposed to treating the PDE parameters merely as a coordinate in the parameter domain, e.g., (x, t, µ µ µ) is combined and directly fed into our ablation model, called PINN-P, for our ablation study in Appendix. With the abuse of notation, P 2 INNs can be expressed as a function of (x, t), parameterized by the hidden representation: u Θ (x, t; µ µ µ) = u {θc,θg} (x, t; h h h param ). This expression emphasizes our intention that we explicitly utilize the PDE model parameters to characterize the behavior of the solution neural network. Encoder for equation input The equation encoder g θp reads the PDE parameters, and generates a hidden representation of the equation, denoted as h h h param . We employ the following fully-connected (FC) structure for the encoder:EQUATIONwhere σ denotes a non-linear activation, such as ReLU and tanh, and F C i denotes the i-th FC layer of the encoder. D p means the number of FC layers of g θp . We note that h h h param has a size larger than that of µ µ µ in our design to encode the space and time-dependent characteristics of the parametrized PDE. Since highly non-linear PDEs show different characteristics at different spatial and temporal coordinates, we intentionally employ relatively high-dimensional encoding.Encoder for spatial and temporal coordinate input The spatial and temporal coordinate encoder g θc generates a hidden representation h h h coord for (x, t). This encoder has the following structure:EQUATIONwhere F C i and D c denote the i-th FC layer and the number of FC layers of g θc , respectively.The manifold network g θg reads the two hidden representations, h h h param and h h h coord , and infer the input equation's solution at (x, t), denoted as û(x, t; µ µ µ). With the inferred solution û, we construct two losses, L u and L f . The manifold network can have various forms but we use the following form:EQUATIONHere, ⊕ represents the concatenation of the two vectors; D g denotes the number of FC layers of g θg .Model training is performed by minimizing the regular PINN loss. With the prediction û produced by P 2 INNs, our basic loss function consists of three terms as follows:EQUATIONwhere L u , L b , and L f enforces initial, boundary conditions, and physical laws in PDEs, respectively, and w 1 , w 2 , w 3 ∈ R are hyperparameters. In general, the overall training method follows the training procedure of the original PINN (Raissi et al., 2019) . The only exception is that the PDE residual loss associated with multiple PDEs is minimized in a mini-batch whereas in the original PINN, the residual of only one PDE is minimized. To be more specific, in each iteration, we create a mini-batch of {µ µ µ i , (x i , t i )} B i=1 , where B is a mini-batch size. We randomly sample the collocation points and, thus, there can be multiple different PDEs, identified by µ µ µ i , in a single mini-batch.  
evaluation: 
 In this section, we test the performance of P 2 INNs on the benchmark PDE problems, 1D CDR equations and 2D Helmholtz equations (cf. Appendix A), both of which are known to suffer from the failure modes. For 1D CDR equations, we employ 6 different equation types stemmed from CDR equations with the varying parameters as listed in Table 2 . The experimental results are summarized in Table 1 and Figure 2 . To evaluate the performance of the model, we measure the L 2 relative and absolute errors between the solution predicted by the model and the analytic solution. Due to space reasons, we put detailed experimental setups and results are in Appendix.Methodology. We train PINN and PINN-R for each parameter configuration in each equation type -in other words, there are as many models as the number of PDE parameter configurations for an equation type. To train P 2 INNs, however, we train it with all the collocation points of the multiple parameter configurations in each equation type, following the training method in Section 2.2. Therefore, we have only one model in each equation type. See Appendix for detailed discussion.Experimental results. The values for 6 CDR equations in Table 1 represent the averaged errors for the coefficient ranges of 1 to 5, 1 to 10, and 1 to 20 (cf. Table 3 ). In Table 1 , whereas PINNs show fluctuating performance, our P 2 INNs show stable performance for all the equation types. The most notable improvement ratios are observed for the reaction and the convection-diffusion-reaction equations. For instance, PINN marks an absolute error of 0.5825 whereas P 2 INN achieves an error of 0.0029 for the reaction equations, i.e., 200 time smaller error. Since large coefficients incur equations difficult to solve, PINNs commonly fail in the range.For 2D Helmholtz equations, we train models with a ∈ [2.5, 3.0] with interval 0.1, and then test them with interval 0.05. Notably, as shown in Figure 2 , P 2 INNs consistently shows good performance in both cases where a is a seen parameter (a = 2.7) and an unseen (a = 2.75) parameter. However, PINN and PINN-R struggle, despite the fact that both of these values are within the seen(trained) parameter range for two models. This reaffirms the robustness and efficacy of our proposed P 2 INN approach in higher-dimensional settings, where there are non-trivial boundary conditions. A DATASETSFor simplicity but without loss of generality, we assume the parameterized 1D CDR equations and 2D Helmholtz equations (cf. equation 6 and equation 7). To generate the ground-truth data, we use either analytic or numerical solutions. In case of 1D CDR equations, we analyze the target equations with three types of initial conditions u(x, 0): two Gaussian distributions of N (π, (π/2) 2 ) and N (π, (π/4) 2 ), and a sinusoidal function of 1 + sin(x). To solve the equation, we use the Strang splitting method (Strang, 1968) . For 2D Helmholtz equations, we obtain the exact solution by calculating it directly.We consider parameterized CDR equations:EQUATIONThe equation describes how the state variable u changes over time with the existence of convective (the second term), diffusive (the third term), and reactive (the fourth term) phenomena. Here, β is a coefficient about how fast transportable the equation is, ν is a diffusivity for the diffusion phase, and ρ is a scaling parameter about spreading velocity. Note that we choose the well-known Fisher's form ρu(1 -u), which was used in Krishnapriyan et al. (2021) , as our reaction term.Each of these individual PDEs has their own importance and has been studied extensively:1. Convection-diffusion equations are used in fluid dynamics, particle chemistry, computational finance, and so on, 2. Reaction-diffusion equations are popular in the domain of biophysics and mathematical biology, 3. Convection equations, diffusion equations, and reaction equations are for describing transport, diffusive, and reactive phenomena, respectively in simplified settings.In total, there are six classes of Convection-Diffusion-Reaction equations, each of which has its own importance in science. For each dataset, we select 1,000 collocation points, 256 initial points, 100 boundary points, and 1,000 test points.A.2 2D HELMHOLTZ EQUATIONSEQUATIONWe employ the specific Helmholtz equations which were used in McClenny & Braga-Neto (2020) as benchmark PDEs (cf. equation 7), and it can be directly solved as equation 8. The Helmholtz equations describe the behavior of state variable u(x, y) in a 2D space, accounting for the effects of wave propagation, and a source term represented by q(x, y). Here k is a parameter related to wave frequency, while a 1 and a 2 control the spatial variations of the source term. In our experiments, we set k to 1 and the parameters a 1 and a 2 to a common value a. For each dataset, we select 1,000 collocation points, 400 boundary points, and 100 test points.Our goal is to develop a method to solve parameterized PDEs via the computational formalism of PINNs' overcoming W1 and W2. With this in mind, we first attempt to obtain intuitions from the visual inspection of solution snapshots displayed on the (x, t)-coordinate space (Figures 5 and 6 ).The first set of the examples is shown in Figure 5 : The ground-truth solutions of convection equations (top row) and reaction equations (bottom row) with varying parameters β and ρ, respectively. . We note that the solution in the last column reflects the first two columns' solutions. Therefore, there also exist similarities across different equation types.As we vary the PDE parameter, e.g., increasing β, we obtain gradually changing solutions (i.e., becoming more oscillatory, as we go left from Figure 5 (a) to Figure 5(c) ). This suggests that model parameters of PINNs for varying PDE parameters could have similar values and this can be leveraged in the training of PINNs.This observation indeed has been investigated in Krishnapriyan et al. (2021) to solve hard PDEs for PINNs. With a higher convective term (large β), the PDE becomes a hard problem for PINNs to solve due to the spectral bias (Rahaman et al., 2019) (i.e., the solution is highly oscillatory in time).Thus, Krishnapriyan et al. (2021) proposed a curriculum-learning algorithm which starts to feed an easier PDE and gradually increases β until it reaches the target value. This approach, however, drops all the intermediate model parameters obtained in the course of training. Instead, in our approach, we utilize all PDE information to train a single model for the solutions of parameterized PDEs.The second set of examples (Figure 6 , the solutions of different types of PDEs) provide a similar observation as above: even for different classes of PDEs (e.g., convection, diffusion, and convectiondiffusion equations), the solutions gradually change, which can be leveraged in training PINNs.Motivation #1: a latent space of parameterized PDEs may exist. Since PDEs with similar parameter settings share common characteristics, we conjecture that solutions of parameterized PDEs can be embedded onto a latent space and reconstructed by using a shared decoder network.Motivation #2: it will be more effective to solve similar problems simultaneously. Considering the similarities between solutions parameterized by similar PDE parameters, we conjecture that training can be improved by attempting to solve all those similar problems together -multi-task learning approaches are also based on the same intuition (Kendall et al., 2018; Ruder, 2017) .Motivated by the observations, we develop a new approach that alleviates the two known weaknesses W1 and W2 (cf. Section 1).There is plenty of literature dealing with PDEs. We briefly introduce related work mostly used in this work.Machine Learning Methods for Solving Partial Differential Equations. Traditional numerical methods such as finite element methods and finite difference methods have clear pros and cons (Patidar, 2016; Li & Bettess, 1997; Srirekha et al., 2010) . The more accurate the results, the more expensive the calculation of numerically approximated formulas. It means that to earn more accurate solutions, it needs to use finer grids, which implies more cost. To alleviate those cons, researchers were attracted to machine learning approaches (Karniadakis et al., 2021; Cuomo et al., 2022) . After various trials like using the Galerkin or Ritz method (Rudd & Ferrari, 2015) , PINNs proposed a transformative way of using deep learning for solving general governing PDEs in a physically sound and easy-to-formulate computational formalism (Raissi et al., 2019) . As elaborated above, PINNs, however, possess weaknesses which must be addressed (Krishnapriyan et al., 2021) (Yu et al., 2018; Li et al., 2020; Gupta et al., 2021) but PINNs still have its potential for mainly focusing on governing equations which describe physical phenomena.Physics as Inductive Biases. There have been various strategies to impose physical constraints on neural networks (Cranmer et al., 2020a; Rudd & Ferrari, 2015; Lee et al., 2021) . Most of them focus on imposing constraints for outputs or injecting specific physical conditions into neural networks. As a simple but effective solution, PINNs directly impose physical conditions into neural networks by using a governing equation itself as a loss (Raissi et al., 2019) . This loss function is called L f . In this way, PINNs can learn the residual error of the governing equation. If initial conditions are given, we can add an initial error loss term L u . Furthermore, if there are specific boundary conditions, we can specify boundary conditions in L b .With the prediction û produced by P 2 INNs, our basic loss function consists of three terms as follows:EQUATIONand L u , L f and L b are defined as follows:EQUATIONEQUATIONEQUATIONwhere N u , N f , and N b are the cardinalities of the sets of initial conditions, collocation points, and boundary conditions; w 1 , w 2 , w 3 ∈ R are hyperparameters. The first and the second terms denote the data matching loss L u and the PDE residual loss L f , respectively. In addition, we separately add the boundary condition term L b , forcing their values equal at both top and bottom parts (see Figures 5 and 6 ).We compare P 2 INNs with three baselines. PINN is the original design based on fully-connected layers with non-linear activations in Raissi et al. (2019) , and PINN-R is its enhancement by using residual connections, which was used in Kim et al. (2021) . PINN-seq2seq (Krishnapriyan et al., 2021) is a model that applies the seq2seq learning method to the PINN model, sequentially learning data over time. We divided the entire time into 10 steps. In addition, we define one ablation model for our method, called PINN-P, which has the same structure as original PINN, but the PDE parameters µ µ µ is treated as a coordinate in the parameter space, i.e., (x, t, µ µ µ).Each baseline and ablation model is trained in the following way:1. PINN, PINN-R, and PINN-seq2seq do not read PDE parameters, such as β, ν, ρ and a, but are trained separately for each of the coefficient settings. 2. PINN-P, an ablation model of P 2 INNs, is able to process PDE parameters and is trained for all coefficient settings in each equation type. 3. Therefore, PINN, PINN-R, and PINN-seq2seq require many trained models for solving parameterized PDEs whereas PINN-P and our method require a single trained model to solve them.Metrics. The relative error and the absolute error of the i-th equation are defined as the averages of ∥ ûiu i ∥ 2 / ∥u i ∥ 2 and ∥ ûiu i ∥ 2 , where i ∈ {1, ..., N e } and N e is the number of equations used for the task. At this time, the errors are measured for each test points and the average value is used. We test with 3 seed numbers and report their mean. Our baselines, PINN, PINN-R, and PINN-seq2seq, are designed with 6 layers, and the dimension of hidden vector is 50. For training, we employ Adam optimizers with learning rate of 1e -3. For our method, we set D p , D c , and D g to 4, 3, and 5 respectively. In the loss function in Eq. equation 5, we set w 1 , w 2 , and w 3 to 1. We use a hidden vector dimension of 50 for g θc and g θg , and 150 for g θp . For g θg . Considering that our method is able to solve multiple equations with one model, the total model size for our method is much smaller than other baselines (see Appendix K).F ARCHITECTURAL DETAILS OF PINN-P ... We propose PINN-P as an ablation model of our P 2 INN. Unlike P 2 INN, PINN-P does not have a separate encoder for PDE parameters, so that PDE parameters enter the model with coordinates. As shown in Figure 7 , PINN-P consists of l-stacked fully-connected layers. For a fair comparison with P 2 INNs, we set size of hidden vector to 150 and l to 6, making the model size similar to P 2 INNs.To benefit the community, the code will be posted online. The source code for our proposed method and the dataset used in this paper are attached.H EXPERIMENTAL RESULTS ON 1D CDR EQUATION Following the notational conventions in curriculum learning, we use the terms, "easy" and "hard," to indicate data that are easy or hard for neural networks to learn.CONCLUSIONPINN is a highly applicable and promising technology for many engineering and scientific domains. However, due to the highly nonlinear characteristic of PDEs, PINNs show very poor performance in certain parameterized PDE problems. In addition, there is a weakness that the model must be re-trained from scratch to analyze a new PDE. To solve these chronic issues, we propose parameterized physics-informed neural networks (P 2 INNs), which can learn similar parameterized PDEs simultaneously. Through this approach, it is possible to overcome the failure situation of PINNs that could not be solved in previous studies. To ensure that it is effective in general cases, we use more than thousands of CDR equations and show that P 2 INNs outperform baselines in almost all cases of the benchmark PDEs. 
 
 Appendix:  


Decision: The reviewer mentions the novelty, clarity and relevance of the proposed work to the workshop. The authors tackle the problems with PINNs and verify their proposed approach on the failure modes paper from Krishnapriyan et al., 2021. I also vote for acceptance and would like the authors to address the reviewer's comments on the meaning of the physical PDE parameters and updated references in the final version. Also, the test problems such as convection are cited in the appendix from Krishnapriyan et al., 2021, which would also be good to add to the main body.

Reviews: review: title: title: A notable work based on the reduction of the errors with regards to some baselines and the application of physics knowledge in machine learning.
main_review: main_review: The level of detail of the methods implemented in this work is excellent. It is relatively straightforward to understand what the authors did. Also, they state very clearly the motivation and the novelty of their study, and it helps to capture the importance of their contribution. I consider that this work is worth accepting it.

Pros:
* The topic of their work (physics-informed machine learning models) is very relevant in the scientific community currently.
* They provide all relevant information of the data, model architecture, baselines, training procedure and objective. The paper is very informative.
* The results reached are quite impressive, specially for the reaction equations, with regards to the tested baselines. 
* With their proposed model they are tackling two weaknesses of the CDF equations, so it is more efficient that all other methods that tackle each weakness separately.

Cons:
* I appreciate that the authors explained the meaning of the equations and most of their parameters. However, since this is a Physics-informed machine learning model, I feel they should have explained also the physical meaning of the tested values of the parameters. In most of the cases they only say which numbers they used, but not actual physical explanation of most of them.
* The bibliography is quite old. There is not any mention of related work from 2023 nor 2024, (the most updated one is only one work from 2022), which makes me think that they are not comparing their method to a more state-of-the-art model.