Characterizing possible failure modes in physics-informed neural networks

1 Introduction . Partial differential equations ( PDEs ) are commonly used to describe different phenomena in science and engineering . These PDEs are often derived by starting from governing first principles ( e.g. , conservation of mass or energy ) . It is typically not possible to find analytical solutions to these PDEs for many real-world settings . Thus , many different numerical methods ( e.g. , the finite element method [ 44 ] , pseudo-spectral methods [ 9 ] , etc . ) have been introduced to approximate their solutions/behavior . However , these PDEs can be quite complex for several settings ( e.g. , turbulence simulations ) , and numerical integration techniques , which typically update and improve a candidate solution iteratively until convergence , are often quite computationally expensive . Motivated by this—as well as the increasing quantities of data available in many scientific and engineering applications—there has been recent interest in developing machine learning ( ML ) approaches to find the solution of the underlying PDEs ( and/or work in tandem with numerical solutions ) . As a result , the area of Scientific Machine Learning ( SciML ) —which aims to couple traditional scientific mechanistic modeling ∗Equal contribution 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . ( typically , differential equations ) with data-driven ML methodologies ( most recently , neural network training ) —has emerged . In this vein , there have been a number of ML approaches to incorporate scientific knowledge into such problems while keeping the automatic , data-driven estimates of the solution [ 2 , 17 , 33 , 39 ] . A recent line of work involves Physics-Informed Neural Network ( PINN ) models , which aim to incorporate physical domain knowledge as soft constraints on an empirical loss function , that is then optimized using existing ML training methodologies . To some degree , PINNs are an example of “ grafting together ” domain-driven models and data-driven methodologies . However , there are important subtleties with this , and we identify several possible failure modes with a naive approach . We then illustrate possible directions for addressing these failure modes . Background and problem overview . Many of the problems with a PDE constraint fit the following abstraction : F ( u ( x , t ) ) = 0 , x ∈ Ω ⊂ Rd , t ∈ [ 0 , T ] , ( 1 ) where F is a differential operator representing the PDE , u ( x , t ) is the state variable ( i.e. , parameter of interest ) , x/t denote space/time , T is the time horizon , and Ω is the spatial domain . Since F is a differential operator , in general one must specify appropriate boundary and/or initial conditions to ensure the existence/uniqueness of a solution to Eq . 1 . In the context of PDEs , F can be taxonomized into a parabolic , hyperbolic , or elliptic differential operator [ 23 ] . Quintessential examples of F include : the convection equation ( a hyperbolic PDE ) , where u ( x , t ) could model fluid movement , e.g. , air or some liquid , over space and time ; the diffusion equation ( a parabolic PDE ) , where u ( x , t ) could model the temperature distribution over space and time ; and the Laplace equation ( an elliptic PDE ) , where u ( x ) could model a steady-state diffusion equation , in the limit as t→∞ . One possible data-driven approach is to incorporate domain information by applying Eq . 1 as a “ hard constraint ” when training a NN on the data . This can be formulated as the following constrained optimization problem , min θ L ( u ) s.t . F ( u ) = 0 , ( 2 ) where L ( u ) is the data-fit term ( including initial/boundary conditions ) , and where F is a constraint on the residual of the PDE system under consideration ( i.e. , the “ physics ” knowledge in the equation itself ) . As mentioned before , for many practical use cases , it is not possible to derive closed form solutions for these problems , and it is often quite difficult to solve problems of the form of Eq . 2 , with F ( u ) as a hard constraint . Another ( related but different ) data-driven approach is to impose the constraint as a “ soft constraint ” on the outputs of the NN model , min θ L ( u ) + λFF ( u ) , ( 3 ) L ( u ) = Lu0 + Lub . ( 4 ) Here , Lu0 and Lub measure the misfit of the NN prediction and the initial/boundary conditions ( which are pre-specified/given as input to the problem ) , and θ denotes the NN parameters ( which takes ( x , t ) , and possibly other quantities , as inputs and then outputs u ( x , t ) ) . Furthermore , λF is a regularization parameter that controls the emphasis on the PDE based residual ( which we ideally want to be zero ) . The goal is then to use ML methodologies ( stochastic optimization , etc . ) to train this NN model to minimize the loss in Eq . 3 . In particular , the NN is trained to minimize this modified loss function , where the modification is to penalize the violations of F ( u ) for some λF ≥ 0 . However , even with a large training dataset , this approach does not guarantee that the NN will obey the conservation/governing equations in the constraint Eq . 1 . In many SciML problems , these sorts of constraints on the system matter , as they correspond to physical mechanisms of the system . For example , if the conservation of energy equation is only approximately satisfied , then the system being simulated may behave qualitatively differently or even result in unrealistic solutions . We should also note that this approach of incorporating physics-based regularization , where the regularization constraint , LF , corresponds to a differential operator , is very different than incorporating much simpler norm-based regularization ( such as L1 or L2 regularization ) , as is common in ML more generally . Here , the regularization operator , LF is non-trivially structured—it involves a differential operator that could actually be ill-conditioned , and it does not correspond to a nice convex set ( as does a norm ball ) . Moreover , LF corresponds to actual physical quantities , and there is often an important distinction between satisfying the constraint exactly versus satisfying the constraint approximately ( the soft constraint approach doing only the latter ) . Main contributions . The contributions of this paper are as follows : • We analyze PINN models on simple , yet physically relevant , problems of convection , reaction , and reaction-diffusion . We find that the vanilla/regular PINN approach only works for very easy parameter regimes ( i.e. , small PDE coefficients ) , but that it fails to learn relevant physics in even moderately more challenging physical regimes , even for problems that have simple closed-form analytical solutions . For many cases , the vanilla PINN approach achieves almost 100 % error , as compared to the ground truth solution , even after extensive hyperparameter tuning . ( See §3 for details . ) • We analyze the loss landscape of trained PINN models and find that adding/increasing the PDE-based soft constraint regularization ( LF in Eq . 3 ) makes it more complex and harder to optimize , especially for cases with non-trivial coefficients . We also study how the loss landscape changes as the regularization parameter ( λF ) is changed . We find that reducing the regularization parameter can help alleviate the complexity of the loss landscape , but this in turn leads to poor solutions with high errors that do not satisfy the PDE/constraint . ( See §4 for details . ) • We demonstrate that the NN architecture has the capacity/expressivity to find a good solution , thereby showing that these problems are not due to the limited capacity of the NN architecture . Instead , we argue that the failure is due to optimization difficulties associated with the PINN ’ s soft PDE constraint . ( See §5 for details . ) • We propose two paths forward to address these failure modes through ( i ) curriculum regularization and ( ii ) posing the learning problem as a sequence-to-sequence learning task . First , in curriculum regularization , we start by imposing the PDE constraint ( LF ) with small coefficients , which are progressively increased to the target problem ’ s settings as the model gets trained . This gives the NN an opportunity to first train with easier constraints , before it is exposed to the target constraint which could be hard to optimize from the beginning . Second , we show that changing the learning problem to a sequence-to-sequence learning problem can reduce the PINN error , again without any change to the NN architecture . In this setup , the NN is trained on a time segment , instead of the full space-time , which could be more difficult to learn . The task is then to predict the solution and reduce the loss only over smaller time segments . We extensively test both approaches and show that they can reduce the error by up to 1-2 orders of magnitude as compared to regular PINN training , and in many cases can better capture “ sharp ” features in the solution . ( See §5 for details . ) • We have open sourced our framework [ 26 ] which is built on top of PyTorch both to help with reproducibility and also to enable other researchers to extend the results . 2 Related work . There is a large body of related work , and here we briefly discuss the most related lines of work . Machine learning and PDEs . ML approaches for PDE problems have been increasing rapidly in recent years [ 13 , 19 ] . A number of tools and methodologies now exist to solve scientific problems by combining ML and domain insights [ 14 , 20 , 27 , 28 , 38 ] . As mentioned earlier , a popular approach to combine ML and physical knowledge is to include aspects of the PDE term as part of the optimization process via regularization . A notable aspect of such an approach is that the NN can be trained only on data that comes from the governing equation ( s ) itself ( though additional data can be included as well , if available ) , i.e. , with a relatively small amount of data . This has garnered interest and shown successful results in a wide variety of science and engineering problems and applications [ 3 , 11 , 16 , 29–31 , 43 ] . However , there have also been issues observed with this formulation . For example , it did not work well for stiff ordinary differential equations ( ODEs ) describing chemical kinetics [ 15 ] , for certain heterogeneous media problems [ 7 ] , or for certain fluid flow problems [ 10 ] . Furthermore , PINN models have been analyzed in the context of neural tangent kernels ( i.e. , towards the infinite width limit ) to study their convergence [ 36 , 37 ] . This work found some cases where the model failed ( such as when the target function exhibits “ high frequency features ” ) and showed some preliminary solutions via the lens of the neural tangent kernel . It has been argued that some of these problems may be due an imbalance in back-propagated gradients in the loss function during training , and a learning-rate annealing scheme has been proposed to mitigate this [ 35 ] . Physical priors and constraints in NNs . Imposing physical priors and constraints on NN systems is common in SciML problems , as a way to try to enforce a property of interest . This idea has been introduced in different forms in the past ( for instance [ 5 , 18 , 25 , 28 , 32 ] ) . Some approaches have focused on embedding specialized physical constraints into NNs , such as conservation of energy or momentum [ 4 , 12 ] or multiscale features [ 34 ] . While methods focusing on constraining the output of the NN are more common , it is difficult to enforce such constraints exactly in ML settings . Previous work has tried to impose hard constraints in ML ( both within the context of SciML and otherwise ) [ 6 , 21 , 22 , 24 , 40 ] , although this can be computationally expensive , and does not guarantee better results or convergence .

Summary: This work studies two specific (yet essential) PDE learning cases in the PINN framework. Notably, the work proposes two main conclusions to explain the difficult training in PINN networks:  - the loss landscape varies greatly with the PDE coefficient values (and works well with only low diffusion/conduction coefficient),  - the loss landscape (at the end of training) varies greatly with the physics informed penalty coefficient.   The main explanation to such difficulties lies in the condition number associated to PINN penalty. The work proposes two solutions: a warm-up, and a sequence to sequence approach.