Multilevel physics informed neural networks (MPINNs)
Authors: Elisa Riccietti
Valentin Mercier
Serge Gratton
Pierre Boudier
Venue: ICLR 2022 Submitted (2022)

Abstract: In this paper we introduce multilevel physics informed neural networks (MPINNs). Inspired by classical multigrid methods for the solution of linear systems arising from the discretization of PDEs, our MPINNs are based on the classical correction scheme, which represents the solution as the sum of a fine and a coarse term that are optimized in an alternate way. We show that the proposed approach allows to reproduce in the neural network training the classical acceleration effect observed for classical multigrid methods, thus providing a PINN that shows improved performance compared to the state-of-the-art. Thanks to the support of the coarse model, MPINNs provide indeed a faster and improved decrease of the approximation error in the case both of elliptic and nonlinear equations.


Under review as a conference paper at ICLR 2022
MULTILEVEL PHYSICS INFORMED WORKS (MPINNS)
NEURAL
NET-
Anonymous authors Paper under double-blind review
ABSTRACT
In this paper we introduce multilevel physics informed neural networks (MPINNs). Inspired by classical multigrid methods for the solution of linear sys- tems arising from the discretization of PDEs, MPINNs are based on the classical correction scheme, which represents the solution of the PDE as the sum and coarse term that are optimized in an alternate way. We show that the pro- posed approach allows us to reproduce in the neural network training the classi- cal acceleration effect observed for classical multigrid methods, thus providing a PINN that shows improved performance compared to the classical ones. Our tests show that, elliptic and nonlinear equations MPINNs are less sensitive the choice of the learning rate than classical PINNs, that they can reach the same ap- proximation error PINNs employing less operations, and in many cases provide a faster and improved decrease of the approximation error.
fine
INTRODUCTION
The approximation of the solution of partial differential equations (PDEs) by artificial neural net- works (ANNs) dates back to the 90s (Lagaris et al.) 1998), but it is only in these last years that this topic fully emerged and gave rise to an active field of research. ANNs have been used for many different purposes in this field: for the numerical solution of either direct problems (Di Muro & Ferrari 2008 Raissi et al.. 2019 Rudd 2013) Mishra 2018 Sirignano & Spiliopoulos 2018 Luo & Yang (020) or of inverse problems (Raissi & Karniadakis, 2018 Raissi et al. 201 7b), to recon- struct the equation from given data (Long al. 2018 Rudy et | 2017), to learn dynamics from incomplete information and physical priors (Ayed et al. 2019; De Bézenac et al.,
2019). ODE solvers have even been used for supervised learning problems (Chen et al. 2018). Also the interpretation of successful ANNs like ResNET ODE discretization schemes is important direction (Ruthotto Haber 2019 Lu et al., 2018), as well as the study of the issues of stability and robustness for different schemes (Haber al. 2019 Chang al. 2018).
In this field, the most famous network architecture is the one of PINNs (physics informed neural networks), introduced for the first time in Raissi al. (2019) 2017a b). Since their introduction, the PINNs architecture has encountered a growing interest and the good performance observed in practice have been later supported by theoretical results. In Mishra & Molinaro (0200b a) the authors propose rigorous estimates on the generalization error of PINNs approximating the solution of the direct and inverse problems for PDEs data assimilation problems in terms of the training error and number of training samples, while convergence results for the sequence of approximations to the solution linear second-order elliptic and parabolic PDEs when the number of data grows are proposed in Shin et al. (2020).
Despite their good performance, the training of such networks may still represent challenge in case of difficult problems, such highly nonlinear problems. In this case really large networks may be needed to correctly represent the sought solution, leading to the need of solving a large-scale optimization problem, for which standard training methods may show slow convergence, and it may be difficult to properly tune the learning rate.
When dealing with linear PDEs, multigrid (MG) methods are by far the most effective methods for the solution of large scale problems (Hackbusch) 1985 Briggs et al.) 2000 Trottenberg et al.) 2000). The improved performance of MG methods derives from the fact that alternating relaxations among

---

Under review as a conference paper at ICLR 2022
fine and coarse grids allows to more efficiently reduce all the components of the error, smooth and oscillatory ones.
The same behaviour can be observed in multilevel optimization for nonlinear problems (Gratton
ct al.
2008
GroB & Krause 2009 Kočvara & Mohammed 2016 Lewis & Nash 2005 2013 Nash
2000
2014 Wen & Goldfarb 2009 Calandra et al. 2021), which still exploits representations of the
problem at different scales. in MG methods the residual of the nonlinear equation is transferred from a grid to the other, in such schemes the variables of the optimization problems are the object of the transfer operators. It is evident that the application of such techniques to neural network training not straightforward. as the transfer operators usually used in MG and multilevel optimization (standard interpolation and restriction operators) requires the quantities that need to be transferred to possess an underlying geometrical structure in order to be effective. In the case of training problem, the variables subject to optimization are the weights and biases of the network, which do not possess any geometrical structure. To derive a multilevel method for the training problem, is necessary to design multilevel transfer operators differently and it is not evident how to do that.
Previous contributions have been made in this direction in Calandra et al. 2020)) The authors pro- pose a strategy to transfer the weights that is inspired to algebraic multigrid (Ruge & Stüben 1987, Brandt 2000), which however requires the knowledge of the Hessian matrix of the loss function, which is impractical for large scale problems. Moreover the proposed approach is designed for one layer neural networks, which limits the range of applications of the approach.
In this work we avoid this problem by following completely different path: diverging from classical multilevel optimization methods and remaining closer to MG methods, we propose a multilevel PINN approach (MPINN), based on writing the solution of the PDE as a sum of two terms, a fine and a coarse one. Each term is a PINN depending on a different number of parameters and trained on a different training set, which are optimized independently the one from the other, as each PINN has its own weights and training points that are not transferred from a network to the other one. As in classical MG. the method proceeds by alternating relaxations on the two levels, which in this case are epochs of training of each PINN. As it is common in the multilevel literature, the approach is presented in the two-levels framework, but by a recursive scheme it can be naturally extended the multilevel case. In such case, the solution of the PDE will be written the sum of finite number of increasingly coarser networks. Interestingly, this approach allows us to reproduce the acceleration typically observed in classical MG methods, in the context of the training of PINNs
Related work The idea of exploiting multiple scales in learning is not new. We mention for in- stance Haber et al. (2018); : Ke et al. (2017) that propose multiscale methods for convolutional neural networks, which connect fow-resolution and high-resolution data, leading to new training strategies that gradually increase the depths of the CNN while re-using parameters for initializations.
A similar idea is employed in Cai & Xu 2019)), but in the context of partial differential equations. Multi-scale deep neural networks are introduced, based on specific techniques to convert the learning or approximation of high frequency data to that of a low frequency ones. Still in this context, Fan et al. (2019) introduce neural networks with novel multiscale structure inspired by hierarchical matrices, which are used to approximate discrete nonlinear maps obtained from discretized nonlinear partial differential equations, such as those arising from nonlinear Schrödinger equations.
In Chung et al. (017) the authors propose multiscale recurrent networks, that can capture the latent hierarchical structure in temporal sequences by encoding the dependencies with different timescales.
With respect to these methods, our is closer in spirit to classical multigrid and enables at the same time a multilevel structure in the space of the network's 's parameters and in that of the samples.
An approach similar in spirit to ours, which is derived from another classical technique used for the solution of PDEs (domain decomposition rather than multigrid) is extended physics informed neural networks (XPINNs) (Jagtap & Karniadakis] [020, Hu et al.) [021]. This approach has been theo- retically and empirically proved to improve standard PINNS by using multiple sub-networks with different complexities for each subdomain. This is very related to our work: instead of decomposing
the domain space, MPINNs are decomposing the solution components for fine and coarse parts.
Structure of the manuscript The manuscript is organised as follows. In Section 2 we briefly review the standard PINNs architecture and in Section 3 the standard multigrid method. Section
2

---

Under review as a conference paper at ICLR 2022
represents the main contribution of this work, where introduce the multilevel PINNs. In Section we show numerical evidence of the advantage of the MPINNs over classical PINNs. Conclusions and perspectives presented in Section 6
S
2 PHYSICS INFORMED NEURAL NETWORKS
2019) are neural networks that are trained to approximate the solution u(z) of partial differential equation. To do that, a loss function is defined that is composed of two terms, term taking into account the physical information and a data-fidelity term. Consider for instance stationary PDE written as:
PINNs (Raissi et al.
(2 u(=)) f(=), E B(2, u(=)) = fB(=). 2 E a92
where Ω C R.d d 1, is connected subset, an is the boundary of 52,
differential operator, B is operator defining the boundary conditions, and f. fB I R are given functions. For this problem the loss function reads as:
C(p) = RMME Ere (p) + RRMS E (p)
(Ia)
X" [[ uN (p; z²)) f {(=")|]
RMS Erea (p)
(Ib)
N
Xm
RMSF (p)
(p: =m) )12
(1c)
N'
where z" is vector of N sample points in which the residual of the PDE is evaluated and n is vector of Nm sample points in U 0Q in which the values of the solution are known; X", A are weights to balance the two components and å (p: 2) is the sought approximation, which during training is function of p. the set of N weights and biases. Notice that training PINN does not require the discretization of the operator D, as ( (p; z)) can be directly computed. After training get the desired approximation ú, (p" 2) to u(2). which depends the number of parametres used to parametrize the network, N, which a hyperparameter fixed before the training.
CLASSICAL MULTIGRID METHODS
In this section we briefly review the basic idea behind classical multigrid (MG) methods (Hackbusch) 1985). Consider a linear system arising from the discretization of a PDE:
Au= = f.
Assume to have at disposal some approximation to the exact solution. There are two important measures of u as an approximation to u. One is the error, which is given simply by = u v, and is however just as inaccessible as the exact solution itself. However, computable measure of how well o approximates u is the residual, given by r = Av. The residual is simply the amount by which the approximation fails to satisfy the original problem Au = Assuming that the solution of the linear system is unique, we have r = if and only if e = 0 (however, it may not be true that when is small in norm, is also small in norm). Using the definitions of r and e, we can derive the so-called residual equation, an extremely important relationship between the error and the residual:
Ae = r.
Given the approximation v, it is easy to compute the residual. To improve the approximation v. we might solve the residual equation for e and compute new approximation using the definition of the error u = v + e. Usually a relaxation scheme is used to solve the linear systems. Many relaxation schemes possess the so called smoothing property, meaning that they are efficient in eliminating the oscillatory modes of the еттоr, while they tend to leave the smooth ones. This is limiting property of such methods, which is corrected by multigrid methods thanks to this key observation: passing from fine grid S2 to a coarse grid Q2 a mode becomes more oscillatory. The relaxation method will thus efficiently remove the oscillatory components of the error if used on a coarse grid. Mutigrid methods obtain good speed ups by alternating relaxations among fine and coarse grids. The main scheme of MG is the following. Assume to have discretized the problem with two different resolutions, obtaining two subproblems: and assume to have at disposal two linear operators R (restriction) and P (prolongation) to transfer information from grid to the other one. A V-cycle of MG two levels follows this scheme:
fine A, = fu and coarse
Aн fH,
3

---

Under review as a conference paper at ICLR 2022
Relax Vi times A₁ U4 fr. to obtain approximation 2/
Compute the residual rh = /n Av
Project the residual on the coarse level rHH Rra
Relax 22 times on the residual equation Ageн TH obtain En
Correct the fine level approximation t/ - 0 + Ex. where en = Pe#
This procedure is the basis of what is called the correction scheme. Having relaxed on the fine grid until convergence deteriorates, we relax on the residual equation on coarser grid obtain an approximation to the error itself. We then return to the fine grid to correct the approximation first obtained there. Multiple V-cycles can performed and the procedure can be extended recursive way to more than two levels. It is well-known that MG outperform Gauss Siedel method and already two-grid algorithm can result in tremendous reduction in the iteration count (Mazumder 2016).
MULTILEVEL PINNS
Inspired by classical MG, we write the solution of our problem as the sum of two terms:
in (pn; =h) + u (PH: zH),
where ún (PNi =h.) corresponds the "fine" term and is PINN parametrized by h weights and biases and trained on fine set 24, of samples, which has enough expressive power to correctly approximate the solution of the PDE, while ú (Prr EH) corresponds the "coarse" term and is PINN parametrized by H < h parameters and trained on coarse set of samples zи- The training inspired to the classical multigrid scheme, which alternates relaxations at coarse and fine level. If we assume that D is linear and that A = 0, we can recover exactly the classical MG scheme. We define the fine and coarse problems as follows:
min (ph) = |D an (ph: 2/)) f(*f)1?; Ph
(2)
N% 1
min (H()) = |D dn (pH: 25)) r(2
(3)
PH
NH
The training in this case follows the following scheme:
Perform epochs problem 2 obtain approximation in (ph, 2) of u(z)
Compute the residual (2 f((]) D2 ùn)
Project the residual on the coarse level Fи R(rr)
Perform V/2 epochs the residual problem in equation 3 to obtain ün (pHr
Correct the fine level approximation ûn (Ph. 24.) + (())(PH,²H))
The main difference between MG and this approach lies the definition of the transfer operators R and P. In MG they are linear operators, usually standard interpolation (PMG) and full-weighting restriction (RMG), which are directly applied to the variables of the problem. In this case instead of applying the operators to the variables of the optimization problem p. the parameters of the network, we apply them to the underlying geometrical variable 2, and thus we define:
R( (ph, 2))) un (PH, RMG*)) P(④n (PH, =B)) àn (pr, PMG*H)
thus the restriction of neural network is still a neural network, but with less parameters and that is evaluated on smaller set of grid points.
In the most general case, we can still use this idea to design multilevel PINN, by transferring directly the networks from one level to the other rather than the residual, and alternating the mini- mization at the two levels to correct the approximation at fine level by the information computed at coarse level. We therefore alternate the minimization of two different losses:
4

---

Under review as a conference paper at ICLR 2022
(ph) = RM ErMs + RMM Edatah (ph)
(4a)
X
RMS R.ea (Ph)
D(2 un (phi 2/) + uH (z())) f(z+)11
(4b)
Am [ (phi =0)= un (H - u(>;">)1
RMSEdatah
(4c)
N
(μ(P)) RMSE E (PH) RMSE E (
(5a)
X
RMSEETEHE (PH)
DD (2(2)) + uz (4)) f(
(5b)
NB
H
RMSEaarap (pH)
un [u + u (pri = ul u(
(5c)
Ng
where (z) = û, (p;iz) z) for h, is the approximation computed at the end of each cycle at coarse set of training points 2и, but
fine/coarse level. Notice that at coarse level üg is trained then the found approximation can be evaluated at a larger set of points when is transferred to the fine level. Notice also that during the fine level training just the fine weights pk of the fine network in are optimized, while u (z) is a fixed term issued from the previous cycle. Conversely at coarse level un (z) is kept fixed. The procedure is described in Algorithm
Algorithm 2-levels training of PINNs
1: Input: starting weights pi. Ph
2: procedure MPNNN f PH)
3: Set uzr (z) un (p (Hiz).
4: for i=1,2..... do
5:
Perform V11 epochs for the minimization of equation yielding weights př and set (2) = ün (ph, z).
6:
Perform 1/2 epochs for the minimization of equation yielding parameters pi, and set u (2) uи 2)
7: end for
8: return ū(p),,) =) + ú(PH,
9: end procedure
NUMERICAL RESULTS
In this section we validate the performance of the proposed MPINNs the solution of ID and 2D elliptical and nonlinear equations. All the codes are implemented in pytorch and the runs are performed using GPUs on Google Colab. The multilevel strategy can be coupled with any opti- mization scheme to perform the training. Here, employ two different training strategies: ADAM and | For ADAM, the learning rate is experimentally tuned for each problem and the same for all networks. More precisely, for the 1D problem set the starting value to 5e-2, and decrease it by 0.9 after every 150 iterations; for the 2D problem we set the starting value to 5e-2 and we decrease by 0.9 every 10 iterations; and for Burger's equation we the starting value to le-3 and decrease it by 0.99 every 500 iterations. For LBFGS we use the Wolfe linesearch routine provided by the method. The training of all the networks is stopped after a maximum number of iteration is reached, this number is large enough to let all the networks reach a steady state. In all the runs we set x X" = 1. We build 2/ = by taking Nm No equispaced points on the boundary of the considered domain, and we build the points =f, 2H as the resulting grid. In all the tests we compare a MPINN composed of a fine network with h neurons per layer and a coarse network with Il neurons per layer, the corresponding one level PINN with layers of h neurons (PINN h) and one level PINN (PINN h). which have the same total number of parameters as MPINN. The values of h. H, h and the number of layers are specified for each test. The performance of the method
h
I.FGS..

---

Under review as a conference paper at ICLR 2022
is evaluated by the RMSE between the groundtruth and the computed approximation, evaluated on a grid of unseen points and in terms of floating point operations performed during training. This quantity is computed the number of flops required for the matrix-vector products operations dur- ing the training. For the MPINN takes into account both the operations performed at fine and at coarse level. This gives an idea of the expected gain in the computational time, as the current implementation of MPINNs is not optimized and thus it does not allow us to directly compare the computational time.
5.1 ID ELLIPTICAL EQUATION
We consider first 1D linear problem:
-())) u(= f(2)
(6)
with f(z) = -m7 + 1) sin(T=) - (a²x² + 1) sin(arz) with a = on the domain Q = 1- with w(-1) = =11)
0. The solution of this problem is sin(xz) + sin (in(T)))). We study the asymptotic behaviour of the MPINNS on Problem [6] that is we consider 1-hidden layer networks with an increasing number of neurons in the hidden layer. In this case h = + H. The values of h, H are specified in Table For each network we report the median and the IQR of the RMSE and training loss computed on grid of unseen points and of the training loss over 10 runs with random starting weights. We choose V11 1/2 5 in Algorithm | All the networks are trained on a training set of 60 sample points, while the RMSE is evaluated on finer grid of 100 points.
(h
MPINN
PINN h
PINN h +
RMSE
(50,25)
1.3e-04, 3.1e-04
1.3e-04, 1.2c-04
7.0c-04, 4.3e-03
Loss
2.0c-04, 2. 1c
5.3e-04. 3. le-00
2.3c-04, 1.2e-03
RMSE (200,100)
2.0e-04,
I.le-03, 2.9c-03
2.0c-03, 2.5e-03
Loss
1.8c-03, 1.2c-03
2.9e-04,6.6e-03
4.3c-03, 1.9e-02 >
RMSE (300,150)
1.4e-03, 5.2e-03
6.Ie-03, 9.5e-1
Loss
2.0e-02, 2.5e-02
1.9e-02, 3.6e-02
>
3. Median and IQR for the RMSE and loss over 10 independent runs with different initial guesses for the weights of the networks.
Table 1: Problem 6
Results are reported in Table We observe that MPINNs in many runs provide a lower approxima- tion error than standard PINNs. The detail of the evolution of the loss functions and of the RMSE on unseen points during training is reported in Figure 1 for a test with (h, H) (50, 25). These results show that MPINNs are less sensitive than PINNs to the choice of the learning rate: the rate has been optimized for the case (h, H) (50, 25) and was used for all the runs. Increasing h MPINNs still perform well with this choice, while the performance of PINNs deteriorates and new tuning of the learning rate becomes necessary, especially for the largest network. This is particularly desirable feature in a network, as the tuning of the learning rate may be tedious.
the plots we have reported the losses for the different networks all the same plot, stress however that they not directly comparable, the PINNs and MPINNs (cf. equation [ and equation ( The aim of the plots is to assess the development of the training and the choice of the learning rate.
Remark 5.1
they are defined in different ways for
However, even with LBFGS, which uses Wolfe linesearch and therefore the learning rate is auto- matically tuned to ensure the decrease of the loss function, MPINNs achieve a better approximation error than PINNs for this test problem, ef. Figure The advantage becomes even more important in the case of deeper networks: in Figure 2 left we use a 1 -iddden layer network, while in Figure 2 right we use 9-hidden layer network, with / 200, H 100, all trained by LBFGS.
Besides these favorable features of MPINNs, another important advantage is the fact that, even when the same error level of the PINNs is reached, this is achieved in less iterations. MPINNs provide in most runs a quicker decrease of the error and thus a solution of comparable accuracy to that of PINNs in a reduced number of both iterations and floating point operations, cf. Figure
Multigrid techniques are particularly effective when solving PDEs whose solution has different modes. In Table we thus study the behaviour of the methods for larger values of o. In this case the

---

Under review as a conference paper at ICLR 2022
Loss function
10
MPNNN
10r
MPFNN
PANN
109
10"
10
10-4
10
10a2
109
10:3
1073
10'7
14000
1.5
groundtruth
PINN
KFNNN
PINN h+H
PINN
10-1
IMSE
10n
0.0
1011
10
-1.0
-15
1011
1.00
0.50
0.25
0.25
0.75
168
Figure 1: Problem 6] = 3. Comparison of MPINN, PINN h, PINN h. + H for h = 50, = 25, ADAM. Up: loss function (left) and RMSE (right) values along the iterations. Bottom left: compar- ison of true solution of the PDE and approximations computed by PINN 200 and MPINN. Bottom right: evolution of the RMSE as function of the number of floating point operations required for the training of the networks (total number of operations for matrix-vector products).
MPNNN
MPINN
PIN
PINN
PRNN
PINN
10-1
10a0
10
10"
10"
10
10
20
Figure 2: Problem a = 3. Comparison of MPINN, PINN h, PINN h H for h = 200, Hl = 100, LBFGS. RMSE over unseen points along the iterations. Left: 1-layer network. Right: 9-layer network.
solution is highly nonlinear. We use networks trained over 400 fine points and 200 coarse points, with /: 50, H = 25, h. = 54, 5 layers and we use LBFGS. We can observe that the multilevel strategy indeed effective also in this context. MPINNs provide RMSE one order of magnitude lower than classic PINNs and with lower IQR.
MPINN
PINN
PINN
3.0e-3,
3.0e-3
1.5e-2, 2.2c-2
1.7e-2, 3.0e-2
10
1.0e-2, 3.1e-2
1.3e-1, 2.8e-1
4.0e-2, 1.8e-1
3.0e-2, 1.0e-1
1,0
1.7e-1, 1.4
Table 2: Problem varying a, h 50, H 25, h = 54, N₁ = 400, NH 200, layers. Median and IQR for the RMSE error over independent runs with different initial guesses for the weights of the networks.

---

Under review as a conference paper at ICLR 2022
MPINN PINN PINN h-H
MPINN PINN
PINN h--
5M5E
RMSE
10"1
10-1
10"1
10"1
2000
4000
6200
0000
10000
12000
14000
0.2
10 109
iterations
Figure 3: Problem 6 = 3. Comparison of MPINN, PINN h, PINN + H for h 50, H = 25,
ADAM. Left: RMSE over unseen points along the iterations. Right: Number of operations along the iterations.
In the Appendix we discuss the choice of the hyperparameters specific to the multilevel strategy, cf. section B
5.2 2D NONLINEAR EQUATION
We consider now a 2D nonlinear problem:
-Au + ae" = f in [0, x [0,
(7)
w(0, y) 0, u(, 0, Vy [0.1]
uzz 0) = 0. u(x,1) 0, Vr E [0.1]
where f has been chosen in order to have the following funetion exact solution:
u(x; y) [sin( + sin((m=)][sinm(rr)) + sin(3ry)
depicted in Figure 6 We consider again 1-hidden-layer networks and we compare MPINN, PINN h, and PINN h + H. all trained by ADAM, we train the fine network on grid of 484 points and coarser grid of 400 points. The results of the tests reported in Table More results are available the Appendix, for different choices of N₁, Results are similar to the TD) case, again see that MPINNs less sensitive than classic PINNs the choice of the learning rate.
the coarse
(h,, H)
MPINN(h, H)
PINN h
PINN h + H
RMSE
(30,15)
5.5e-2, 1.4e-2
8.1e-2, 4.0e-2
5.0e-2, 1.0e-2
Loss
1.2e-1, 0.6e-1
4.4e-1, 2.0e-1
1.3e-1, 0.5e-1
RMSE (100,50)
3.0e-2, 1.3e-2
3,8e-2, 2.2e-2
2,8e-2, 3.9e-
Loss
2.5e-2, 1.4e-2
7.0e-2, 1.4e-2
4.6e-2, 5.0e-1
RMSE (200,100)
3.0e-2, 3.0e-2 (8/10)
3.0e-2,
3.7c-2 (8/10)
Loss
3.0e-2, 6.0e-2
5.2.c-2, 1.le-1
>
RMSE (300,150)
3.0c-2, 3.0e-2 (7/10)
Loss
5,4c-2, 4.1e-2
>1
>1
Table Problem 7 = 0.1. Median and IQR of the RMSE and loss over 10 independent runs with different initial guesses for the weights of the networks. When numbers in parenthesis are reported it means that not all the 10 runs were successful, i.e. that the achieved RMSE was larger than 1.
5.3 BURGER'S S EQUATION
Burgers' equation is fundamental partial differential equation occurring in various areas of applied mathematics. For a given field u(z, t) and diffusion coefficient v, the general form of Burgers' equation in one space dimension is the dissipative system:
Ou
Ou
= pn u
+ u
(8)
Øt
Oz
d
8

---

Under review as a conference paper at ICLR 2022
We have conducted preliminary tests on this equation, cf. Table 4 As in the other cases, MPINN can reach solution approximations of accuracy comparable to those obtained by classical PINNs, but performing far less operations. We consider here PINNs with 3 hidden layers. We compare MPINN composed of a PINN with 20 neurons and a PINN with 10 neurons in the hidden layers, and two standard PINNs with 40 and 30 neurons in the hidden layers. The PINNs are trained training set of 40 spatial points and 120 temporal points. The MPINN uses 80 time points on the coarse grid. The decrease of the RMSE as a function of the number of operations is depicted in Figure 4 (left). On the right, where we report the RMSE along some iterations, we can observe the effect of the coarse iterations in details. We distinguish in red the RMSE obtained after a set of coarse iterations, and in blue the one obtained after a set of 5 fine iterations. We can see that the coarse iterations help to make the RMSE decrease faster.
MPINN 1.0.-1,0.4e-1 2.6e-2, 0.6e-2
PINN 40
PINN 30
RMSE
1.8e-1, 0.4e-1
1.7e-1, 1.5e-2
Loss Operations 1
2.8e-2, 1.1e-1
8.5e-3, 7.7e-2
6.1
3.5
Table 4: Problem 8 Number of operations and median and IQR of the RMSE error and loss function over 10 independent runs with different initial guesses for the weights of the networks.
6x100-
PRUN 40
PBON 30
7.0
coarse
fnee
MPNNN
6.5
4x
6.6
644
2x10
62-
6.0
operatioon
1:10
0.0
25
5.0
7.5
10.0
12.5
15.0
17.5
20.0
Figure 4: Problem 8 Left: RMSE as function of the number of floating point operations. Right: Focus on subset of V-cycles. The blue stars mark the RMSE at the end of a fine pass in the V-cycle, while the red stars mark the RMSE at the end of a coarse pass in the V-cycle.
CONCLUSIONS AND FUTURE PERSPECTIVES
We have presented MPINNs, family of multilevel PINNs, whose training based on the standard correction scheme used by multigrid methods, state-of-the-a methods for the solution of linear elliptical equations. Miming the correction scheme we represent the solution of the PDE as the sum of a fine and a coarse networks, which are optimized in an alternate way. We show the results of tests performed on and 2D problems, with elliptical and nonlinear differential operators. The preliminary results show good performance of the MPINNs compared to standard PINNs both in terms of accuracy reached in the solution and in terms of operations in the training. MPINNs also appears to be much less sensitive than PINNs to the choice of the learning rate. As meaningful future research perspectives we target two directions: a deeper numerical investigation of MPINNs, in particular the of deeper recursion scheme, described in Appendix A, and the development of a sound convergence theory for our method.
Broader impact The broader impact of deep learning methods for solving PDEs has been detailed in Um et al. (2021).
Reproducibility
We have included our code in the supplementary material for reproducibility.

---

Under review as a conference paper at ICLR 2022
REFERENCES
I. Ayed, E. de Bézenac, A. Pajot, J. Brajard, and P. Gallinari. Learning dynamical systems from partial observations. ArXiv preprint 1902.11136, 2019.
A. Brandt. General highly accurate algebraic coarsening. Electronic Transuctions on Numerical Analysis, 10:1-20, 2000. URL BRlLhtt:://eudh
brgg/oc. 0ttcc/120880 120680
W. Briggs, V. Henson, and S. McCormick. A Multigrid Tutorial, Second Edition. Society for Industrial and Applied Mathematics, second edition, 2000. doi: 10. https: /ppbs: s siam. org/ http//oi//aa /10 http://aaaa......111.... 1117771.770088719553333
URL
W. Cai and Z. Xu. Multi-scale deep neural networks for solving high dimensional PDEs. ArXiv preprint 1910.11710, 2019.
H. Calandra, S. Gratton, E. Riccietti, and X. Vasseur. On multilevel levenberg-mangee for the training of artificial neural networks and its application the solution of partial differential
method
equations. Optimization Methods and Software, pp. 1-26, 2020. doi: 10. 10010100000008.000000 1775828. URL httpt::/ddooi wwt.oor.gg///.......... 1080/1055
2020. 1772222177788
H. Calandra, S. Gratton, E. Riccietti, and X. Vasseur. On high-order multilevel optimization strate- gies. SIAM Journal Optimization, 31(1):307-330, 2021. doi: 10.1137/19M12553555, URL
https://doi.orgg 1tt.://...111719111111
B. Chang, L. Meng, E. Haber, L. Ruthotto, D. Begert, and E. Holtham. Reversible architectures for arbitrarily deep residual neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32 (1), 2018.
R. Chen, Y. Rubanova, J. Bettencourt, and D. Duvenaud. Neural ordinary differential equations. In 32nd Conference on Neural Information Processing Systems (NeurIPS 2018), 2018.
J. Chung, S. Ahn, and Y. Bengio. Hierarchical multiscale recurrent neural networks. In 5th Interna- fional Conference on Learning Representations, ICLR 2017, 2017.
E. De Bézenac, A. Pajot, and Gallinari. Deep learning for physical processes: Incorporating prior scientific knowledge. Journal of Statistical Mechanics: Theory and Experiment, 2019(12): 124009, 2019.
G. Di Muro and S. Ferrari. constrained-optimizzaion approach to training neural networks for smooth function approximation and system identification. In 2008 IEEE International Joint Con- ference on Neural Networks (IEEE World Congress on Computational Intelligence), pp. 2353- 2359, 2008. htRp:tt:///Kk doi org/ 1109/IJCNN 2008 4 4634124
Y. Fan, L. Lin, L. Ying, and L. Zepeda-Nunez. chical matrices. Multiscale Modeling & Simulation, 1744):118-12113 2019. doi: 10.1137/ 18M1203602. URLhttps
multiscale neural network based on hierar-
doii /tt.d/o..oggoogeeeeee 10olgggo... 1111 101.1117/88M11036022 8M1203022
Gratton, A. Sartenaer, and Ph. L. Toint. Recursive trust-region methods for multiscale nonlinear optimization. SIAM Journal Optimization, 19(1):114444, 2008. doi: 10. 10.1137/050023011. URL https::/ http//oi..ogg///// ott.oo/////001110000000000000000
C. Groß and R. Krause. On the convergence of recursive trust-region methods for multiscale nonlin- optmization and applications to nonlinear mechanics. SIAM Journal Numerical Analysis, 47(4):30443069,, 2009.
E. Haber, L. Ruthotto, E. Holtham, and S. Jun. Learning across scales- multiscale methods for convolution neural networks. Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
E. Haber, K. Lensink, F. Treister, and L. Ruthotto. Imexnet a forward stable deep neural network. In International Conference on Machine Learning, pp. 2525-2534, PMLR, 2019.
W. Hackbusch. Multi- Multi-grid methods and applications, volume 4 of Springer Series in Computational Mathematics. Springer- Verlag, Berlin, Heidelberg, 1985. URL https: /www springer. com us http://s/boook/11888411171266
10

---

Under review as a conference paper at ICLR 2022
Zheyuan Hu, Ameya D Jagtap, George Em Karniadakis, and Kenji Kawaguchi. When do tended physics-informed neural networks (xpinns) improve generalization? arXiv preprint arXiv:2109.094 09444, 2021.
Ameya D Jagtap and George Em Karniadakis. Extended physics- inniiis-difomeddd neural networks (xpinns): A generalized space-time domain decomposition based deep learning framework for nonlin- ear partial differential equations. Communications in Compulational Physics, 280)2200--,, 2020.
T.W. Ke, M. Maire, and S. X. Yu. Multigrid neural architectures. In Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4067-4075, 2017. URL https: .org/1........ .ttp//11110000 1109/ CCPR.CCR.... 201P2.2017.. .433
M. Kočvara and S. Mohammed. A first-order multigrid method for bound-constrained convex op- timization. Optimization Methods and Software, 31(3):622-644, 2016. doi: 10. 10.100//10556788. 2016. 1462.71 URL https: org/ /tt.0////...110005505500700
2016 011
I. E. Lagaris, A. Likas, and D. Fotiadis. Artificial neural networks for solving ordinary and partial differential equations. IEEE Transactions on Neural Nerworks, 95):987-1000, 1998. ISSN 1045-
9227. URL Stttst::: /oi. ctop wwtp://///..11//...... /010://////0...220/011111////// 712111 712178
R. M. Lewis and S. G. Nash. Using inexact gradients in a multilevel optimization algorithm. Com- putational Optimization and Applications, 56(1):39 61, 2013. URL httpt://doo....oo /ztp///..ooggl..../
10C7/s10589-0133 9546
R.M. Lewis and S.G. Nash. Model problems for the multigrid optimization of systems governed
by differential equations. SIAM journal on Scientific Computing, 26(6):1811-137, 2005. URL
https: doi. www/oo......oo.oooooo.o.o....... 11137//11482750222220
Z. Long, Y. Lu, X. Ma, and B. Dong. PDE-Net: Learning PDEs from data. In Proceedings of the 35th International Conference on Machine Learning. PMLR 80, pp. html
2018. URL
mlr. press/
Y. Lu, A. Zhong, Q. Li, and B. Dong. Beyond finite layer neural networks: Bridging deep archi- tectures and numerical differential equations. In International Conference on Machine Learning. pp. 3276-32662 3285. PMLR, 2018
T. Luo and H. Yang. Two-layer neural networks for partial differential equations: Optimization and generalization theory. ArXiv preprint 2006. 15733, 2020.
Mazumder. Stability and convergence of iterative solvers. Numerical Methods for Par- tial Differential Equations Chupter 4, pp. 169-218. Academic Press, 2016. ISBN 978-0-12- 849894-1. doi: http:://da..gg///ooooof/77222222 1.000444 URL http::ttt///t/tt sciencedirect. cwm. httpc///ccccicenaaaaaaaaaa//////
S. Mishra. A machine learning framework for data driven acceleration of computations of differential equations. Technical Report 2018-28, Seminar for Applied Mathemat- ics. ETH Zürich, 2018. URL https: sam wath. wtw..ch...ch...... chasamş $reports/ r wfinal/ Baanaa/repprrt221//20-----2
pdf
Mishra and R. Molinaro. Estimates on the generalization error of Physics Informed Neural Networks (PINNs) for approximating class of inverse problems for PDEs. ArXiv preprint 2007.01138, 2020a.
Siddhartha Mishra and Roberto Molinaro. Estimates on the generalization error of physics informed neural networks (PINNs) for approximating PDEs. ArXiv preprint 2006./6/44 2020b.
S.G. Nash. A multigrid approach to discretized optimization problems. Optimization
Methods and Software, 14(1-2):99-116,, 2000. URL hatps:://dat /oi. wroeioorg/111100
S.G. Nash. Properties of class of multilevel optimization algorithms for equality constrained problems. Optimization Methods and Software, 29 2014. URL //doi.
org 10. arc/100.000/1110 100.11000115567700 759577

---

Under review as a conference paper at ICLR 2022
M. Raissi, P. Perdikaris, and G. E. Karniadakis. Physics Informed Deep Learning (Part I): Data- driven Solutions of Nonlinear Partial Differential Equations. ArXiv preprint 1711.10561, 2017a.
M. Raissi, P. Perdikaris, and G. E. Karniadakis. Physics Informed Deep Learning (Part If): Data- driven Discovery of Nonlinear Partial Differential Equations. ArXiv preprint 1711.10566, 2017b.
M. Raissi, P. Perdikaris, and G. E. Karniadakis. Physics-informee neural networks: A deep learn- ing framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378:686-707, 2019.
Maziar Raissi and George Em Karniadakis. Hidden physics models: Machine learning of nonlinear partial differential equations. Journal of Computational Physics, 357:125 141, 2018 ISSN 0021-9991. doi: jo.p016f
2017 2017.11. 039
K. Rudd. Solving Partial Differential Equations Using Artificial Neural Nenworks. PhD the- sis, Cornell University, 2013. URL http: //1:::///sss mae. Co scrreel...duPaattTTee... KeithRuudPhD.ppdl
S. H. Rudy, S. L. Brunton, J. L. Proctor, and J. N. Kutz. Data-driven discovery of partial
differential equations. Science Advances, 3(4), 2017. doi: 10.1126/sciadv. 2012///cciav..00614....444 1602614.
URL
sciencemag http::/advancee http://vvaaeeee.ciiinennaa..a..a org. hatg:/ooneene o /ttpt/////eeee////
W. Ruge and K. Stüben. Algebraic multigrid. In Multigiii methods chapter pp. 73 73-130. SIAM, 1987. URL https:: s am. wwtg//o///ii http//10.1111///11 137/1. 9781 611 191119715 ch4
L. Ruthotto and E. Haber. Deep neural networks motivated by partial differential equations. Journal of Mathematical Imaging and Vision, pp. 1-13, 2019.
H. Schaeffer. Learning partial differential equations via data discovery and sparse optimization. Proceedings of the Royal Society of London A: Mathematical, Physical and Engineering Sci- ences, 473(2197), 2017. ISSN 1364-5021. URL
2016 0446
Y. Shin, J. Darbon, and G. E. Karniadakis. On the convergence of physics informed neural networks for linear second-order elliptic and parabolic type pdes. ArXiv preprint 2004.0 (004.01806, 2020.
J. Sirignano and K. Spiliopoulos. DGM: A deep learning algorithm for solving partial differential equations. Journal of computational physics, 375:1339-1364 2018
U. Trottenberg, C. W. Oosterlee, and A. Schuller. Multigrid. Elsevier, Nor-York, 2000. URL https: http://:wwwwwwwlwee........ http://www...aa.oooooooooooaaaaa 978 0- 08 047956-9
K. Um, R. Brand, Y. Fei, Holl, and N. Thuerey. Solver-in-the-logg Learning from differentiable
physics interact with iterative pde-solvers. ArXiv preprint 2007.00016, 2021.
Z. Wen and Goldfarb. A line search multigrid method for large-scale nonlinear optimization. SIAM J. on Optimization, 20(3):1478-1503, December 2009, ISSN 1052-6234. doi: 10. 08071524X. URL http: doi.iix.iiii....oooo www.::o.......oooooo
A APPENDIX DEEPER MULTILEVEL, SCHEMES
It is well-known that multigrid techniques outperform standard Gauss Siedel method and already two-grid algorithm can result in tremendous reduction in the iteration count (Mazumder) 2016). We have shown that very good results can be obtained by two-levels MPINNs as well.
However, as for standard multigrid, the procedure can be extended to the case of more than two levels in recursive form. This is described in Algorithm where we assume to use L levels. We write u(2)) Σ (p,, 2), with â (pt, 2) for 1 < < L PINN depending and trained on training sets 2 (in the algorithm we assume for simplicity that 2!" = 2/ := 21), and assume that for both the parameters set and the samples sets the dimension increases with (1 is
of parameters
12

---

Under review as a conference paper at ICLR 2022
the minimum level). At each level I, Mi training iterations are performed on the network with the following loss:
L((pt) RMSE (eMSErea(PPi) + F
(9a)
RMS
Ni | D(z (p(p;; 21) +
Σ ux(zi)) f(21)1?
(9b)
1=1,
RM (M
L (pt 2))
+
Σ ui(z)) - u(zi)11.
(9c)
Nı
1=1,1g1
Algorithm 2 V-cycle of MPINNs with L levels (recursive form)
1: Input: level number I (1 < 1 S L, 1 is the minimum level), starting values of the networks
parameters p;, i = 1....,, L
2: procedure pt, (=)= MPINN(1.P
-....... 3: for = ,, + do set (2) (.(p..2)
4: end for
5: Iterate in times to minimize equation to obtain pi
6: Set pt = Pj
7:
i = then return P1, (2)
8:
else
9:
P1 p1 1-1,w-1(z)) MPINN 1,p, i= L). Set ú(=) (p): Ei (z(2).
end if end procedure
12:
The numerical investigation of this framework is left a meaningful future research perspective.
B APPENDIX HYPERPARAMETERS SETTING
In this section we discuss the influence of the choice of the hyperparameters (those specific to the multilevel method) on the performance of MPINNs. We discuss in particular the choice of N and H with respect to N₁, and h, respectively.
In the runs in 55. we selected the same number of points the fine and in the coarse grids. In
Table 5 we study the influence of the choice of N := N2z as compared to N₂, == N2 in problems These results are also reported in Figure 5 left. The number of operations is computed as
ratio of the number of operations for H and the number of operations for H 150, the maximum value used. We choose (h, H) (200, 100). This is of course not a comprehensive study, but we see that decreasing the number of points the accuracy achieved by the MPINN also decreases, even if the behaviour is not strictly regular. If too few points are used the MPINN does not converge.
NH
25
50
60
70
100
150
RMSE (median)
2.3
8.0e-4
9,4e-4
2.8e-4
4.5e-4
2.3e-4
RMSE (IQR)
2.3
1.9e-3
1.6e-3
3.1e-3
3.1e-4
3.0e-4
Operations
0.85
0.88
0.89
0.84
0.94
Table 5: Problem a = 3. MPINN with (h, H)) = (200, 100). N₁. = 150 and different choices of
NH := Nr. Median and IQR (10 independent runs) of the RMSE error on unseen points and ratio of the number of operations performed with respect to the choice N = 150.
have chosen H h/2, in the spirit of classical multigrid methods, but larger or smaller networks can be employed. In Table 6 we study the influence of the choice of H with respect fixed h in problems 4 5 these values are also reported in Figure 5 right.
In the runs in section 5.1
13

---

Under review as a conference paper at ICLR 2022
RMSE 0.008
RMSE
109
0.004
0.000
10a4
120 140
Figure 5: Problem 6 == =3 3. MPINN with h = 200, No Nu = 150, varying H, ADAM. Median and IQR (10 independent runs) of the RMSE error on unseen points varying parameter H and ratio of the number of operations performed with respect to the choice H = 150.
We choose 200. This is again a comprehensive study, but can see that decreasing the number of neurons the accuracy achieved by the MPINN also decreases and the IQR of the results increases. However, the MPINN converges even with few neurons.
The choice of these parameter should thus been based on compromise between desired accuracy and available training time.
10
25
50
60
70
100
150
RMSE (median)
3.2c-3
7.8e-4
4.6e-4
1.7e-4
3.1e-4
1.6e-4
2.4e-4
RMSE (IQR)
5.2e-3
6.8e-4
5.5e-4
9.0e-5
4.5e-4
1.9e-04
2.2e-4
Operations
0.88
0.89
0.91
0.93
0.96
0.98
Table 6: Problem 6 a
3, MPINN with 200, N₁ NH 150, varying H, ADAM. Median
and IQR of the RMSE error over 10 independent runs with different initial guesses for the weights of the networks.
MPINN(h,
H)
PINN(h)
PINN(h)
RMSE
OP
RMSE
OP
RMSE
OP
5
1.6e+00.6.4e-00
3.8e+11
5.2e-02.1.1e-02
3.8e+11
4.6e-02.338e-03
8.6e+11
15
4.7e-02.2.7e-02
3.9e+11
5.2e-02,1.1e-02
3.8e+11
4.6e-02.3.8e-03
8.6e+1
25
4.0e-02,8.0e-4
4.1e+11
5.2e-02,1.1e-02
3.8e+11
4.6e-02,3.8e-03
8.6e+11
35
3.9e-02,7.9e-03
4.3c+I1
5.2e-02.111e-02
3.8c+11
4.6c-02,3.8c-03
8.6e+11
Table 7: Problem 7
0.1. Median and IQR of the RMSE error over 10 independent runs with different initial guesses for the weights of the networks and number of operations (OP).
We consider in Table 7 the effect of Nu in the 2D problem in section 5.2 We consider 3- layers networks with 50, H 25, h 54 and fixed learning rate of 1.5e-3 for all networks. We observe that for a too small number of points, the coarse network does not converge and does the other hand, with a coarse grid of reasonable size, our method obtains results comparable or even better than the classical PINNs of size + H (OP) comparable
allow the convergence of the method.
PINNs of size h. h.e sampling of the coarse grid is, as in the ID case, hyperparameter in its own right that must be balanced in order to obtain a coarse network that is good enough improve convergence while limiting its impact on the computational cost.
while keeping a number of operations
14

---

Under review as a conference paper at ICLR 2022
1.0
15
0.5
0.2
0.5
0.8
Figure 6: Solution to problem 7
15


Reviews:
[{"review_id": "gCoe9kW5Fgj", "reviewer": "Reviewer_Ji4t", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Multigrid training for PINNs could be potentially interesting. In particular, I agree with the motivation of the authors that the particular structure of MPINNs could make training easier by imposing a kind of inductive bias. That being said, the results in this paper are too preliminary to be accepted.\n\n1-D Linear Elliptic equation:\n- It seems like the two PINNs you use only differ by capacity of the network. I thought the main motivation for Multigrid was that the PINN with less capacity would be trained on fewer points?\n- The equation is chosen so that you get two different modes in the solution. It would be interesting to vary $\\alpha$ to see whether MPINN training helps as the difference in frequency between the modes increases. I feel like the main motivation for this toy problem could be the different modes. It would be interesting to see it for more difficult parameter settings rather than only $\\alpha = 3$ or $5$.\n- Why the distinction between Adam for the 1-layer networks and BFGS for the 4-layer networks? Since BFGS uses n^2 memory, it seems Adam would be more appropriate for larger networks.\n- How were the learning rates for the different problems tuned? I am not entirely convinced that the difference in performance is not from using different learning rates / hyperparameters for the different problems.\n- You mention that MPINNs are more robust, but I am not sure I agree with the definition of robustness. It seems you are defining a budget of training epochs, and if the method does not converge in time, it is considered not robust. This would just mean the method is \"slower\", but not necessarily \"less robust\".\n- It seems you are reporting mean + std of the RMSE. Can you report median + IQR instead?\n\n2-D Nonlinear equation:\n- Could you give some intuition on what the equation represents and what the solution looks like?\n- 400 points vs 484 points does not seem like much of a difference between \"fine\" and \"coarse\". How did you choose those numbers?\n- Results do seem less convincing in this case.\n\nBurger's equation:\n- What optimizer did you use?\n- It seems you are picking different networks for each PDE. Here you have 3 hidden layers. How did you choose these sizes?", "summary_of_the_paper": "The authors adopt ideas from multigrid methods in solving PDEs to Physics Informed Neural Networks (PINNs). After giving a standard introduction to PINNs and Multigrid methods, they describe their approach.\n\nThey decompose the PDE solution as a sum of a coarse and a fine term, and train independent PINNs to learn each term. The coarse PINN is chosen to have smaller capacity than the fine PINN, and is also (potentially) trained on fewer datapoints on the domain. Training proceeds by alternating training between the two PINNs: $\\nu_1$ epochs on the fine PINN, followed by $\\nu_2$ epochs on the coarse PINN, and repeat.\n\nThe authors test their method on a linear non-homogeneous 1-D elliptic equation (trained with Adam or BFGS), a 2-D nonlinear equation, and 1-D Burger's equation. They compare a standard PINN vs a larger PINN vs their method (MPINN).", "summary_of_the_review": "The framework is potentially interesting, but the experiments in the paper do not convince me. It would be helpful to see more difficult problems, as well as multilevel problems with n > 2. Furthermore, there are a lot of details on hyperparameter tuning that is missing, leaving the question open whether the stated performance improvements are due to the hyperparameters or the MPINN idea.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "DfCfhaSizsh", "reviewer": "Reviewer_9RP9", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Using multigrid methods as inspiration for training and designing PINNs is neat. Splitting the solution into two terms is interesting, because this scheme presents the model with an easy and a hard task (learning fine-scale structure is typically more challenging than learning coarse-scale structures). The proposed scheme can also be viewed as some form of data augmentation and it would be interesting to discuss this aspect in more detail. Further, it would be interesting to discuss whether there is some connection between the proposed multilevel learning scheme and Curriculum Learning.\n\nOverall, the presentation of the materials is very clear and the paper reads nicely. However, a major shortcoming of the paper are experiments. \n\nI would like to see the following questions addressed:\n\n* How do you construct $z_h$ and $z_H$? How do you determine whether $z_H$ is not expressive enough? Is this an automatic process or does this require manual tuning? How is the performance affected if $z_H$ is varied from less coarse to more coarse?\n\n* How does the proposed multilevel PINN compare to other state-of-the art PINNs or to classical scientific computing methods for the problems under consideration? For papers that have mainly an empirical flavor it is typically good practice to show some benchmark studies. Burger's Equation is often considered as a standard problem, hence it should not be too difficult to provide some baselines for this problem. \n\n* In Table 2 and 3, the difference between the proposed model and the ablation models appear to be marginal in terms of the RMSE. For example, is there a practical difference between 1.0e-3 and 1.2e-3 for the problems under consideration?\n\n* Why does PINN h and h+H start to fail if the number of parameters are increased? Do you regularize these models?\n\n* How did you chose the tuning parameters for the different models? I played around with the provided notebook \"MPINN_2D\" for a few minutes and I was able to improve the performance of the standard PINN quite a bit by slightly increasing the learning rate (2e-3), introducing a small amount of weight decay (5e-3) and switching off the learning rate scheduler. The PINN_15 achieves now an RMSE of 0.092 as compared to 0.14 by the proposed multilevel PINN. I am pretty sure that the performance can be further improved with some more careful fine-tuning of the hyperparamters. Tuning models are crucial and this needs to be addressed.\n\n* The term 'robustness' typically means something different in the ML community. I am not exactly sure what you mean by `increased robustness of MPINNs'.", "summary_of_the_paper": "This paper proposes a new physics informed neural network (PINN) that is motivated by multigrid methods. The idea of the proposed multilevel PINN is to write the solution for a given problem as sum of two terms, where the first term models fine-scale structures and the second term models coarse-scale structures. This approach yields models that converge faster and reduce the approximation error as compared to standard PINNs. The performance is demonstrated on both 1D and 2D problems.", "summary_of_the_review": "This paper introduces a new multilevel PINN for learning solutions of PDEs. The presented ideas are novel and of interest to the scientific machine learning community. One shortcoming of this paper is that the experiments appear to be too preliminary, i.e., no benchmarks are provided and the models are not carefully fine-tuned. This makes it difficult to judge the advantage of the proposed method as compared to other state-of-the-art PINNs. At this point I feel that the paper is slightly below the acceptance threshold, but I am willing to change my score if the authors can address my questions above. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "890mE_rmELb", "reviewer": "Reviewer_EmLF", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is not well written in English. But since this MPINNs show interesting point that it can run faster and provide smaller approximation error for some PDE equations than using original PINNs, I recommend this paper to be published after polishing the language and also addressing all my below comments.\n\nMy comments are as follows:\n\n1.\tIn the third line on page 5, the formula for R and P has extra ‘)’ at the end. On the same page after the Equation (6c) line, it should be ‘at the end of each cycle’ instead of ‘at the and of each cycle’. In this paper, there are too many ‘really’ and ‘then’.\n2.\tIn this paper, all the experiments are done with h=2*H. Why do you choose 2 here? How does this number matter for the MPINNs training? Some explanations about how to choose this number should be added.\n3.\tOn page 6, the authors state “Indeed, when the number of parameters increase the peculiar structure of the MPINN allows to make the training process easier and as a result the network reaches a better approximation.” The authors should explain why MPINN works better instead of just stating the observation.\n4.\tOn page 7, on the last two lines the authors say “In 2D problems the effect of the size of the training set on the cost is far more important than in the 1D case, therefore we train the fine network on a grid of 484 points and the coarse one on a coarser grid of 400 points”. What is the rule of thumb to choose the grid size? \n5.\tOn page 8, the Figure 3 uses h=10 case while in Table 3 it starts with h=30. Since Figure 3 and Table 3 are talking about the same experiment “2D NONLINEAR EQUATION”, why do you choose different h for table and plot?\n", "summary_of_the_paper": "The manuscript entitled “MULTILEVEL PHYSICS INFORMED NEURAL NETWORKS\n(MPINNS)” introduces multilevel physics informed neural networks (MPINNs), which is inspired by classical multigrid methods for the solution of linear systems arising from the discretization of PDEs. The authors showed MPINNs provide a faster and smaller approximation error in the case both of elliptic and nonlinear equations and a more robust asymptotic behavior.\n", "summary_of_the_review": "This paper proposed the multilevel physics informed neural networks (MPINNs), which is inspired by classical multigrid methods for the solution of linear systems arising from the discretization of PDEs. It showed some interesting results, but those results are lacking of explanation in depth. The authors are mostly just stating the observations, which is not enough to meet the standards of a good paper. \n\nIn summary, I would not recommend its publication in ICLR until all my comments are addressed.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "7iTLLHB4QF8", "reviewer": "Reviewer_Ugje", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The method of MPINNs makes sense and seems to be novel. Although it is motivated via the classical multigrid method with the operators P and R, the final method is a very simple alternate optimization with two sub-networks (for two levels) where one sub-networks minimizes the loss for ‘fine’ points, and another minimizes the loss for ‘coarse’ points. Because the solution of PDEs can contain fast/fine components as well as coarse/slow components, the proposed method makes sense and I would expect that this works well for such PDEs. In this regard, I really like the paper and recommend acceptance subject to very minor revisions for several concerns that I explain below. \n\nWhile the idea and method are great, the demonstration of the method can be improved a lot. The present experimental results are somewhat inconclusive because of the experimental settings and the definitions of the plotted values.  For the experimental setting, the authors mention that the learning rates are experimentally tuned, and the best-decreasing strategy is adopted for each network. This can be problematic without presenting more details, as the learning rates can be the reason for the reported superior performance of MPINNs. I would recommend reporting the results with various learning rates or fixing the learning rates a priori by directly using the learning rates of previous work of PINNs. In the current form, I cannot tell if we are overfitting the data and problems with the tuning of learning rates of MPINNs. At least, the authors should report the learning rates for each case.\n\nThe idea is very related to extended physics informed neural networks (XPINNs), which was theoretically studied by Hu et al [1]:\n\n[1] Hu et al. When Do Extended Physics-Informed Neural Networks (XPINNs) Improve Generalization?\n\nThe previous work [1] has theoretically and empirically shown that XPINNs can improve standard PINNs by using sub-networks, each for different subdomains with different complexities of the sub-solutions. This is very related to MPINNs. Instead of decomposing the domain space, MPINNs are decomposing the solution components for fine and coarse parts. Therefore, the authors should cite the previous work [1] to mention this relationship. Moreover, unlike the previous work on XPINNs [1], the present paper does not provide any theoretical results, which should be also mentioned as a limitation of MPINNs, when compared to XPINNs, at this moment. In sum, the relationship and the relative limitation should be made clear.\n\nFor the definitions of the plotted values, the authors should use a metric that can be comparable for all methods. Instead of these loss and objective used for training, the authors can plot the value of errors for the solution of PDEs, by using a set of new unseen data points. \n\nWhat are the precise definitions of RMSE and loss used in the tables and figures?\n\nI am not entirely convinced by the results where the performance degrades as we increase h and H. Do you observe the same phenomena with BFGS instead of ADAM? It is possible that this can be caused by bad training practices by the authors. I would suggest plotting the training loss over epoch for different h and H as well to see what is going on there.\n\nThe paper makes several random claims without any evidence or support. I recommend deleting those claims from the paper. For example, the authors say that it shows improved performance compared to the state-of-the-art, but no state-of-the-art performance is cited or compared against in this paper. After Table 1, it reads “We observe that MPINNs clearly outperform classical PINNs.”, but it is not clear at all because of the uncertain learning rates and the stopping criteria. The following claim is also not supported by any evidence in this paper and it is an opinion of the authors instead of a finding in a scientific paper: “It is clear that structuring the parameters in a clever way is more beneficial than just augmenting their number, to gain both computational time and expressivity.”\n\nIn table 2, F is not used and the following sentence should be deleted in its caption: ”F” means a failure in all the runs. \n\nIn table 4, is “0.04.6e-2” typo? On page 5, “the and of” should be replaced by “the end of”. \n\n", "summary_of_the_paper": "This paper proposes multilevel physics informed neural networks (MPINNs). When compared to standard PINNs, MPINN uses additional networks for different levels with fine and coarse terms in the view of two-levels. By reclusive definition, this can be generalized to any multi-levels. The method is motivated via classical multigrid method, and is empirically studied with 3 simple ODEs/PDEs.    ", "summary_of_the_review": "While there are several concerns and weak evidence to support the main claims, the idea is very novel and significant. I infer from the method and its idea that the additional evidence to support the main claims will come by sooner or later, either by this paper or other authors. There are so many incremental papers without any originality that have been published just because of large-scale experiments for demonstrations. In contrast, this paper has weak demonstrations but a good original idea. Whereas both types of papers have their own merits, I think that the latter type contributes to our community, science, and academia in a longer-term. Accordingly, I recommend this paper for acceptance, subject to minor revisions. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper develops an instance of physics informed neural network inspired from multigrid methods for solving PDEs. The proposed framework describes the solution of a PDE problem as the sum of terms operating at different resolutions. Training is performed by an iterative optimization algorithm that alternates between the different resolution models. Experiments are performed on 1D and 2D problems.

All the reviewers agree on the originality and the potential of the proposed method. They however all consider that the current version of the work is too preliminary both in the form and in the content. The experimental contribution should be developed further with tests performed on more complex problems and complementary analyses. Some of the claims should be given more evidence or moderated. It also appeared during the discussion that the models are not well tuned, making the results inconclusive. The authors are encouraged to develop and strengthen their work.
Decision: Reject