Multilevel physics informed neural networks (MPINNs)

1 INTRODUCTION . The approximation of the solution of partial differential equations ( PDEs ) by artificial neural networks ( ANNs ) dates back to the 90s ( Lagaris et al. , 1998 ) , but it is only in these last years that this topic fully emerged and gave rise to an active field of research . ANNs have been used for many different purposes in this field : for the numerical solution of either direct problems ( Di Muro & Ferrari , 2008 ; Raissi et al. , 2019 ; Rudd , 2013 ; Mishra , 2018 ; Sirignano & Spiliopoulos , 2018 ; Luo & Yang , 2020 ) or of inverse problems ( Raissi & Karniadakis , 2018 ; Raissi et al. , 2017b ) , to reconstruct the equation from given data ( Long et al. , 2018 ; Rudy et al. , 2017 ; Schaeffer , 2017 ) , to learn dynamics from incomplete information and physical priors ( Ayed et al. , 2019 ; De Bézenac et al. , 2019 ) . ODE solvers have even been used for supervised learning problems ( Chen et al. , 2018 ) . Also the interpretation of successful ANNs like ResNET as ODE discretization schemes is an important direction ( Ruthotto & Haber , 2019 ; Lu et al. , 2018 ) , as well as the study of the issues of stability and robustness for different schemes ( Haber et al. , 2019 ; Chang et al. , 2018 ) . In this field , the most famous network architecture is the one of PINNs ( physics informed neural networks ) , introduced for the first time in Raissi et al . ( 2019 ; 2017a ; b ) . Since their introduction , the PINNs architecture has encountered a growing interest and the good performance observed in practice have been later supported by theoretical results . In Mishra & Molinaro ( 2020b ; a ) the authors propose rigorous estimates on the generalization error of PINNs approximating the solution of the direct and inverse problems for PDEs data assimilation problems in terms of the training error and number of training samples , while convergence results for the sequence of approximations to the solution of linear second-order elliptic and parabolic PDEs when the number of data grows are proposed in Shin et al . ( 2020 ) . Despite their good performance , the training of such networks may still represent a challenge in case of difficult problems , such as highly nonlinear problems . In this case really large networks may be needed to correctly represent the sought solution , leading to the need of solving a large-scale optimization problem , for which standard training methods may show a slow convergence , and it may be difficult to properly tune the learning rate . When dealing with linear PDEs , multigrid ( MG ) methods are by far the most effective methods for the solution of large scale problems ( Hackbusch , 1985 ; Briggs et al. , 2000 ; Trottenberg et al. , 2000 ) . The improved performance of MG methods derives from the fact that alternating relaxations among fine and coarse grids allows us to more efficiently reduce all the components of the error , smooth and oscillatory ones . The same behaviour can be observed in multilevel optimization for nonlinear problems ( Gratton et al. , 2008 ; Groß & Krause , 2009 ; Kočvara & Mohammed , 2016 ; Lewis & Nash , 2005 ; 2013 ; Nash , 2000 ; 2014 ; Wen & Goldfarb , 2009 ; Calandra et al. , 2021 ) , which still exploits representations of the problem at different scales . If in MG methods the residual of the nonlinear equation is transferred from a grid to the other , in such schemes the variables of the optimization problems are the object of the transfer operators . It is evident that the application of such techniques to neural network training is not straightforward , as the transfer operators usually used in MG and multilevel optimization ( standard interpolation and restriction operators ) requires the quantities that need to be transferred to possess an underlying geometrical structure in order to be effective . In the case of a training problem , the variables subject to optimization are the weights and biases of the network , which do not possess any geometrical structure . To derive a multilevel method for the training problem , it is necessary to design multilevel transfer operators differently and it is not evident how to do that . Previous contributions have been made in this direction in Calandra et al . ( 2020 ) . The authors propose a strategy to transfer the weights that is inspired to algebraic multigrid ( Ruge & Stüben , 1987 ; Brandt , 2000 ) , which however requires the knowledge of the Hessian matrix of the loss function , which is impractical for large scale problems . Moreover the proposed approach is designed for one layer neural networks , which limits the range of applications of the approach . In this work we avoid this problem by following a completely different path : diverging from classical multilevel optimization methods and remaining closer to MG methods , we propose a multilevel PINN approach ( MPINN ) , based on writing the solution of the PDE as a sum of two terms , a fine and a coarse one . Each term is a PINN depending on a different number of parameters and trained on a different training set , which are optimized independently the one from the other , as each PINN has its own weights and training points that are not transferred from a network to the other one . As in classical MG , the method proceeds by alternating relaxations on the two levels , which in this case are epochs of training of each PINN . As it is common in the multilevel literature , the approach is presented in the two-levels framework , but by a recursive scheme it can be naturally extended to the multilevel case . In such case , the solution of the PDE will be written as the sum of a finite number of increasingly coarser networks . Interestingly , this approach allows us to reproduce the acceleration typically observed in classical MG methods , in the context of the training of PINNs . Related work The idea of exploiting multiple scales in learning is not new . We mention for instance Haber et al . ( 2018 ) ; Ke et al . ( 2017 ) that propose multiscale methods for convolutional neural networks , which connect low-resolution and high-resolution data , leading to new training strategies that gradually increase the depths of the CNN while re-using parameters for initializations . A similar idea is employed in Cai & Xu ( 2019 ) , but in the context of partial differential equations . Multi-scale deep neural networks are introduced , based on specific techniques to convert the learning or approximation of high frequency data to that of a low frequency ones . Still in this context , Fan et al . ( 2019 ) introduce neural networks with a novel multiscale structure inspired by hierarchical matrices , which are used to approximate discrete nonlinear maps obtained from discretized nonlinear partial differential equations , such as those arising from nonlinear Schrödinger equations . In Chung et al . ( 2017 ) the authors propose multiscale recurrent networks , that can capture the latent hierarchical structure in temporal sequences by encoding the dependencies with different timescales . With respect to these methods , our is closer in spirit to classical multigrid and enables at the same time a multilevel structure in the space of the network ’ s parameters and in that of the samples . An approach similar in spirit to ours , which is derived from another classical technique used for the solution of PDEs ( domain decomposition rather than multigrid ) is extended physics informed neural networks ( XPINNs ) ( Jagtap & Karniadakis , 2020 ; Hu et al. , 2021 ) . This approach has been theoretically and empirically proved to improve standard PINNs by using multiple sub-networks with different complexities for each subdomain . This is very related to our work : instead of decomposing the domain space , MPINNs are decomposing the solution components for fine and coarse parts . Structure of the manuscript The manuscript is organised as follows . In Section 2 we briefly review the standard PINNs architecture and in Section 3 the standard multigrid method . Section 4 represents the main contribution of this work , where we introduce the multilevel PINNs . In Section 5 , we show numerical evidence of the advantage of the MPINNs over classical PINNs . Conclusions and perspectives are presented in Section 6 . 2 PHYSICS INFORMED NEURAL NETWORKS . PINNs ( Raissi et al. , 2019 ) are neural networks that are trained to approximate the solution u ( z ) of a partial differential equation . To do that , a loss function is defined that is composed of two terms , a term taking into account the physical information and a data-fidelity term . Consider for instance a stationary PDE written as : D ( z , u ( z ) ) = f ( z ) , z ∈ Ω , B ( z , u ( z ) ) = fB ( z ) , z ∈ ∂Ω , where Ω ⊆ Rd , d ≥ 1 , is a connected subset , ∂Ω is the boundary of Ω , D is a differential operator , B is an operator defining the boundary conditions , and f , fB : Rd → R are given functions . For this problem the loss function reads as : L ( p ) = RMSEres ( p ) +RMSEdata ( p ) ( 1a ) RMSEres ( p ) = λr Nr ‖D ( z , ûN ( p ; zr ) ) − f ( zr ) ‖2 , ( 1b ) RMSEdata ( p ) = λm Nm ‖ûN ( p ; zm ) − u ( zm ) ‖2 , ( 1c ) where zr is a vector of Nr sample points in Ω in which the residual of the PDE is evaluated and zm is a vector of Nm sample points in Ω ∪ ∂Ω in which the values of the solution are known ; λr , λm are weights to balance the two components and ûN ( p ; z ) is the sought approximation , which during training is a function of p , the set of N weights and biases . Notice that training a PINN does not require the discretization of the operatorD , asD ( ûN ( p ; z ) ) can be directly computed . After training we get the desired approximation ûN ( p∗ ; z ) to u ( z ) , which depends on the number of parametres used to parametrize the network , N , which is a hyperparameter fixed before the training . 3 CLASSICAL MULTIGRID METHODS . In this section we briefly review the basic idea behind classical multigrid ( MG ) methods ( Hackbusch , 1985 ) . Consider a linear system arising from the discretization of a PDE : Au = f. Assume to have at disposal some approximation v to the exact solution . There are two important measures of v as an approximation to u . One is the error , which is given simply by e = u− v , and is however just as inaccessible as the exact solution itself . However , a computable measure of how well v approximates u is the residual , given by r = f − Av . The residual is simply the amount by which the approximation v fails to satisfy the original problem Au = f . Assuming that the solution of the linear system is unique , we have r = 0 if and only if e = 0 ( however , it may not be true that when r is small in norm , e is also small in norm ) . Using the definitions of r and e , we can derive the so-called residual equation , an extremely important relationship between the error and the residual : Ae = r. Given the approximation v , it is easy to compute the residual . To improve the approximation v , we might solve the residual equation for e and compute a new approximation using the definition of the error u = v + e. Usually a relaxation scheme is used to solve the linear systems . Many relaxation schemes possess the so called smoothing property , meaning that they are efficient in eliminating the oscillatory modes of the error , while they tend to leave the smooth ones . This is a limiting property of such methods , which is corrected by multigrid methods thanks to this key observation : passing from a fine grid Ωh to a coarse grid ΩH , a mode becomes more oscillatory . The relaxation method will thus efficiently remove the oscillatory components of the error if used on a coarse grid . Mutigrid methods obtain good speed ups by alternating relaxations among fine and coarse grids . The main scheme of MG is the following . Assume to have discretized the problem with two different resolutions , obtaining two subproblems : a fine one Ahuh = fh and a coarse one AHuH = fH , and assume to have at disposal two linear operators R ( restriction ) and P ( prolongation ) to transfer information from a grid to the other one . A V-cycle of MG on two levels follows this scheme : • Relax ν1 times on Ahuh = fh to obtain an approximation vh • Compute the residual rh = fh −Avh . • Project the residual on the coarse level rH = Rrh • Relax ν2 times on the residual equation AHeH = rH to obtain eH • Correct the fine level approximation vh = vh + eh , where eh = PeH . This procedure is the basis of what is called the correction scheme . Having relaxed on the fine grid until convergence deteriorates , we relax on the residual equation on a coarser grid to obtain an approximation to the error itself . We then return to the fine grid to correct the approximation first obtained there . Multiple V-cycles can be performed and the procedure can be extended in a recursive way to more than two levels . It is well-known that MG outperform Gauss Siedel method and already two-grid algorithm can result in tremendous reduction in the iteration count ( Mazumder , 2016 ) .

Summary: The authors adopt ideas from multigrid methods in solving PDEs to Physics Informed Neural Networks (PINNs). After giving a standard introduction to PINNs and Multigrid methods, they describe their approach.  They decompose the PDE solution as a sum of a coarse and a fine term, and train independent PINNs to learn each term. The coarse PINN is chosen to have smaller capacity than the fine PINN, and is also (potentially) trained on fewer datapoints on the domain. Training proceeds by alternating training between the two PINNs: $\nu_1$ epochs on the fine PINN, followed by $\nu_2$ epochs on the coarse PINN, and repeat.  The authors test their method on a linear non-homogeneous 1-D elliptic equation (trained with Adam or BFGS), a 2-D nonlinear equation, and 1-D Burger's equation. They compare a standard PINN vs a larger PINN vs their method (MPINN).