Characterizing possible failure modes in physics-informed neural networks
Authors: Aditi Krishnapriyan
Amir Gholami
Shandian Zhe
Robert Kirby
Michael W. Mahoney
Venue: NeurIPS 2021 Poster (2021)
Keywords: scientific machine learning
physics-informed neural networks
regularization
constrained optimization
unconstrained optimization

Abstract: Recent work in scientific machine learning has developed so-called physics-informed neural network (PINN) models. The typical approach is to incorporate physical domain knowledge as soft constraints on an empirical loss function and use existing machine learning methodologies to train the model. We demonstrate that, while existing PINN methodologies can learn good models for relatively trivial problems, they can easily fail to learn relevant physical phenomena for even slightly more complex problems. In particular, we analyze several distinct situations of widespread physical interest, including learning differential equations with convection, reaction, and diffusion operators. We provide evidence that the soft regularization in PINNs, which involves PDE-based differential operators, can introduce a number of subtle problems, including making the problem more ill-conditioned. Importantly, we show that these possible failure modes are not due to the lack of expressivity in the NN architecture, but that the PINN's setup makes the loss landscape very hard to optimize. We then describe two promising solutions to address these failure modes. The first approach is to use curriculum regularization, where the PINN's loss term starts from a simple PDE regularization, and becomes progressively more complex as the NN gets trained. The second approach is to pose the problem as a sequence-to-sequence learning task, rather than learning to predict the entire space-time at once. Extensive testing shows that we can achieve up to 1-2 orders of magnitude lower error with these methods as compared to regular PINN training.

Characterizing possible failure modes in physics-informed neural networks
Aditi S. Krishnapriyan" .1,2, Amir Gholami*
Shandian Zhe?, Robert M. Kirby², Michael W. Mahoney
LLwwrence Berkeley National Laboratory. ?University of California, Berkeley. University of Utah, 'International Computer Science Institute (aditikl, amirgh, eberkeeyyw edu, {zhe, kirby) utah edu
Abstract
Recent work in scientific machine learning has developed so-called physics- informed neural network (PINN) models. The typical approach to incorporate physical domain knowledge as soft constraints on an empirical loss function and use existing machine learning methodologies train the model. We demonstrate that, while existing PINN methodologies can learn good models for relatively trivial problems, they can easily fail to learn relevant physical phenomena for even slightly more complex problems. In particular, situations of widespread physical interest, including learning differential equations with convection, reaction, and diffusion operators. We provide evidence that the soft regularization in PINNs, which involves PDE-based differential operators, can introduce number of subtle problems, including making the problem more ill-conditioned. Importantly, we show that these possible failure modes are not due to the lack of expressivity in the NN architecture, but that the PINN's setup makes the loss landscape very hard to optimize. We then describe two promising solutions to address these failure modes. The first approach is to use curriculum regularization, where the PINN's loss term starts from simple PDE regularization, and becomes progressively more complex as the NN gets trained. The second approach is to pose the problem as petpuence-t--squeeeee than learning to predict the entire space-time at once. Extensive testing shows that we can achieve up to 1-2 orders of magnitude lower error with these methods as compared to regular PINN training.
analyze several distinct
learning task, rather
Introduction
Partial differential equations (PDEs) are commonly used to describe different phenomena in science and engineering. These PDEs are often derived by starting from governing first principles (e.g.. conservation of mass or energy). It is typically not possible to find analytical solutions to these PDEs for many real-world settings. Thus, many different numerical methods (c.g., the finite element method [44], pseudo-spectral methods [9], etc.) have been introduced to approximate their solutions/behavior However, these PDEs can be quite complex for several settings (e.g., turbulence simulations), and numerical integration techniques, which typically update and improve candidate solution iteratively until convergence, are often quite computationally expensive. Motivated by this-as well as the increasing quantities of data available many scientific and engineering applications- thhre has been recent interest in developing machine learning (MI.) approaches to find the solution of the underlying PDEs (and/or work in tandem with numerical solutions). As result, the area of Scientific Machine Learning SSiMM)-whicl aims to couple traditional scientific mechanistic modeling
"qqual contribution
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

(typically, differential equations) with data-driven MI methodologies (most recently, neural network training)-has emerged. In this vein, there have been a number of ML approaches to incorporate scientific knowledge into such problems while keeping the automatic, data-driven estimates of the solution [2, 17, 33, 39].
A recent line of work involves Physics-Informet Neural Network (PINN) models, which aim to incorporate physical domain knowledge as soft constraints on an empirical loss function, that is then optimized using existing ML training methodologies. To some degree, PINNs are an example of "grafting together" domain-drive
models and data-driven methodologies. However, there are important subtleties with this, and we identify several possible failure modes with a naive approach. We then illustrate possible directions for addressing these failure modes.
Background and problem overview. abstraction:
Many the problems with a PDE constraint fit the following
F(u(z, t)) = O, TEΩC C Rd, t E [0,
(I)
where is a differential operator representing the PDE, u(x, t) is the state variable (i.e., parameter of interest), z/tt denote space/time, T is the time horizon, and {2 is the spatial domain. Since F is a differential operator, general one must specify appropriate boundary and/or initial conditions to ensure the existence/uniqueness of solution to Eq. 1. In the context of PDFs, F can be taxonomized into a parabolic, hyperbolic, or elliptic differential operator [23]. Quintessential examples of J include: the convection equation (a hyperbolic PDE), where u(xr, f) could model fluid movement, e.g., air or some liquid, over space and time; the diffusion equation (a parabolic PDE), where u(x, t) could model the temperature distribution over space and time; and the Laplace equation (an elliptic PDE), where u(x) could model a steady-state diffusion equation, in the limit as / - oc.
One possible data-driven approach is to incorporate domain information by applying Eq. 1 as "hard constraint" when training a NN on the data. This can be formulated as the following constrained optimization problem,
min C(u) s.t. F(u)= 00
where C(u) is the data-fit term (including initial/boundary conditions), and where F is a constraint on the residual of the PDE system under consideration (i.e., the "physics" knowledge in the equation itself). As mentioned before, for many practical use cases, it is not possible to derive closed form solutions for these problems, and it is often quite difficult solve problems of the form of Eq. 2, with F(u) hard constraint.
Another (related but different) data-driven approach is to impose the constraint "soft constraint" on the outputs of the NN model,
min L(u) + Ax)( 8
[(u) = Lus + L.s.
(4)
Here, Lug and Lu, measure the misfit of the NN prediction and the initial/boundary conditions (which are pre-specified/given as input to the problem), and 0 denotes the NN parameters (which takes tx, and possibly other quantities, as inputs and then outputs u(x: t)). Furthermore, 'r is regularization parameter that controls the emphasis on the PDE based residual (which we ideally want to be zero). The goal is then to use ML methodologies (stochastic optimization, etc.) to train this NN model to minimize the loss in Eq. 3. In particular, the NN is trained to minimize this modified loss function, where the modification is to penalize the violations of F(u) for some X 2 O. However, even with large training dataset, this approach does not guarantee that the NN will obey the conservation/gee goveonnnggonnnggggggoooooogn equations in the constraint Fq. 1. In many SciMI. problems, these sorts of constraints on the system matter, as they correspond to physical mechanisms of the system. For example, if the conservation of energy equation is only approximately satisfied, then the system being simulated may behave qualitatively differently or even result unrealistic solutions.
We should also note that this approach of incorporating physics-based regularization, where the regu- larization constraint, Ly, corresponds differential operator, very different than incorporating much simpler norm-based regularization (such as L1 or Lę regularization), as is common in ML more generally. Here, the regularization operator, L, non-trivially structured-ie involves a differential operator that could actually be ill-conditioned, and it does not correspond to a nice convex set (as does

---

norm ball). Moreover, Lx corresponds to actual physical quantities, and there is often important distinction between satisfying the constraint exactly versus satisfying the constraint approximately (the soft constraint approach doing only the latter).
Main contributions.
The contributions of this paper are as follows:
We analyze PINN models on simple, yet physically relevant, problems of convection, reaction, and reaction-diffusion. We find that the vanilla/regular PINN approach only works for very easy parameter regimes (i.e., small PDE coefficients), but that it fails to learn relevant physies in even moderately more challenging physical regimes, even for problems that have simple closed-form analytical solutions. For many cases, the vanilla PINN approach achieves almost 100% error, as compared to the ground truth solution, even after extensive hyperparameter tuning, (See $3 for details.)
We analyze the loss landscape of trained PINN models and find that adding/increasing the PDE-based soft constraint regularization (Ly in Eq. 3) makes it more complex and harder optimize, especially for cases with non- con-trivial coefficients. We also study how the loss landscape changes as the regularization parameter (Az) is changed. We find that reducing the regularization parameter can help alleviate the complexity of the loss landscape, but this in turn leads to poor solutions with high errors that do not satisfy the PDE/constraint. (See 84 for details.)
We demonstrate that the NN architecture has the capacitylexpressivity to find a good solution, thereby showing that these problems are not due to the architecture. Instead, we argue that the failure is due optimization difficulties associated with the PINN's s soft PDE constraint. (See $5 for details.)
limited capacity of the NN
We propose two paths forward to address these failure modes through (i) curriculum reg- ularization and (ii) posing the learning problem as a sequence-to-sequene learning task. First, in curriculum regularization, we start by imposing the PDE constraint (Cx) with small coefficients, which are progressively increased the target problem's 's settings as the model gets trained. This gives the NN an opportunity to first train with easier constraints, before it is exposed to the target constraint which could be hard to optimize from the beginning. Second, we show that changing the learning problem to exquence-to-scquenee learning problem can reduce the PINN error, again without any change to the NN architecture. In this setup, the NN is trained be more difficult to learn. The task is then to predict the solution and reduce the loss only over smaller time segments. We extensively test both approaches and show that they can reduce the error by up to 1-2 orders of magnitude as compared to regular PINN training, and in many cases can better capture "sharp" features in the solution. (See $5 for details.)
time segment, instead of the full space-time, which could
We have open sourced our framework [26] which is built on top of PyTorch both help with reproducibility and also to enable other researchers to extend the results.
2
Related work
There is a large body of related work, and here we briefly discuss the most related lines of work.
Machine learning and PDEs. ML approaches for PDE problems have been increasing rapidly in recent years [13, 19]. A number of tools and methodologies now exist to solve scientific problems by combining MI. and domain insights [14, 20, 27, 28, 38). As mentioned earlier, popular approach to combine ML and physical knowledge is to include aspects of the PDE term as part of the optimization process via regularization. A notable aspect of such an approach is that the NN can be trained only on data that comes from the governing equation(s) itself (though additional data can be included as well, if available), i.e., with a relatively small amount of data. This has garnered interest and shown successful results in wide variety of science and engineering problems and applications [3, 11, 16, 29-31, 43).
However, there have also been issues observed with this formulation. For example, it did not work well for stiff ordinary differential equations (ODEs) describing chemical kinetics [15], for certain heterogeneous media problems [7], or for certain fluid flow problems [10]. Furthermore, PINN models have been analyzed in the context of neural tangent kernels (i.c.. towards the infinite width
3

---

limit) to study their convergence [36, 37]. This work found some cases where the model failed (such as when the target function exhibits "high frequency features") and showed some preliminary solutions via the lens of the neural tangent kernel. It has been argued that some of these problems may be due an imbalance in back- paoo-propde gradients in the loss function during training, and learning-rate -aate annealing scheme has been proposed to mitigate this [35].
Physical priors and constraints in NNs. Imposing physical priors and constraints NN systems common in SciML problems, as a way to try to enforce a property of interest. This idea has been introduced in different forms in the past (for instance [5, 18, 25, 28, 32]). Some approaches have focused on embedding specialized physical constraints into NNs, such as conservation of energy or momentum [4, 12] or multiscale features [34]. While methods focusing on constraining the output of the NN are more common, it is difficult to enforce such constraints exactly in ML settings. Previous work has tried to impose hard constraints MI. (both within the context of SciML and otherwise) [6, 21, 22, 24, 40], although this can be computationally expensive, and does not guarantee better results or convergence.
3 Possible failure modes for physics-informed neural networks
In this section, we highlight several examples where the PINN formulation defined in Eq. 3 does not predict the solution well. We first demonstrate this with two different types of simple, canonical PDE/ODE systems which have simple analytical solutions: convection $3.1), and reaction (A). We then also include a diffusion component by looking at the -iittiooniiffuioon problem ( ($3.2). Note that the convection problem has linear PDE constraint, and aeaeffoonrraactioodddffisiiiiiii problems both have non-linear PDE terms.? 2 We show that PINNs can only learn simple problems with very small parameter values (e.g., small convection or reaction coefficients). We demonstrate that these models fail to learn the relevant physical phenomena for non-trivial cases (e.g., relatively larger coefficients). As we will see, while adding the physical constraint as soft regularization may be easier to deploy and optimize with existing unconstrained optimization methods, this approach does come with trade-offs, including that in many cases the optimization problem becomes much more difficult to solve.
Experiment setup. We study both linear and non-linear PDES/ODES, and we vary the convection, reaction, and diffusion coefficients for each problem (hereafter, we refer these as PDE coefficients). For each problem, we aim to minimize the loss function in Eq. 3. We use 4-layer fully fully-connected NN with 50 neurons per layer, a hyperbolic tangent activation funetion, and randomly sample collocation points (x, f) on the domain. Furthermore, all the systems that we consider have periodic boundary conditions. We enforce this through an extra term the loss function that takes the difference between the predicted NN solution each boundary. We train this network using the L-BFGS optimizer and sweep over learning rates from 1e-4 to 2.0.² After training the PINN, we measure the La relative and absolute errors between the PINN's predicted solution and the analytical solution.
û u ll2 and the absolute error is N ΣN || â - u 1l2. where N is the number of evaluation grid points, is the predicted solution by the PINN, and u is the true solution. For all cases, we run models at least ten times with different preset random seeds, and we average the relative and absolute errors in u(x, t). For each loss function, á is the output of the NN and shorthand for á (0,
The L2 relative error is
N N 1
|| 2 llz
Note that for convection, this does not necessarily mean that the mapping from initial to final solution is also linear. This just means that the terms in the PDE are linear.
For reasons that are only partially understood, L-BFGS methods tend to perform better for existing PINN problems. While variants of stochastic gradient descent are much more popular in computer vision. natural language processing, and recommendation systems, we found that they underperform in comparison to L-BFGS.

---

Relavve ener -n Absolute ener
109
10
Ralative w(
100
175
6.7
10
125
6a5
10
Alsolutt
10-4
-035
-025
10"
10
10
10
10
10
30
1.00
10.00
(a) Error for different
(b) Exact solution for 3
(c) PINN solution for
30
Figure 1: Prediction error for ID convection ( $3.1) prohlem, when 3 is changed. The PINN has difficulty predicting the solution past certain timestep, but is able the boundary conditions. Additional figures for different values can seen Fig. C.I.
3.1 Learning convection
Problem formulation. We first consider a one-dimensional convection problem, a hyperbolic PDE which is commonly used to model transport phenomena:
Ou
3 Iu
Ōt Oz =0, ZE t
(5)
u(x,0) 0) h(x)) FEΩ.
Here, 3 is the convection coefficient and h(x) is the initial condition. For constant 3 and periodic boundary conditions, this problem has simple analytical solution:
Uanalytical (z, t) F-1 (F(h(z))e-0kk)
(6)
where F is the Fourier transform, i 1- and Is denotes frequency in the Fourier domain. The general loss function for this problem (corresponding to Eq. 3) is
Nu Σ ((0) No ==1
N
Oú.
Ou 2 +
+
X
+ 3
LB.
N
Ôt
Oz
==1
where - NN, xx is the output the NN, and LB is the boundary loss. For periodic boundary conditions with Ω 2T), this loss is:
1 LB
No Σ (i(0,0, 0, -0(0--(02200 1,,022m,1 2(0,22m.))
(8)
We use the following simple initial and periodic boundary conditions:
u(z, 0) sin(x), w00, t) u(2m,t) t).
Observations. We apply the PINN's soft regularization to this problem, and we optimize the loss function in Eq. 7. After training, measure the relative and absolute errors between the PINN's predicted solution and the analytical solution, as reported in Fig. ((a). As one can see, the PINN is only able achieve good solutions for small values of convection coefficient, and it fails when becomes larger, reaching a relative eтrоr of almost 100% for 3 > 10. We also provide visualization of the exact and PINN solution Fig. (b-c). One can clearly see that the PINN is unable learn the solution. As we will later show, the NN architecture does have enough capacity to find the solution, but the training/optiiization problem is very difficult to solve with PINNs (and importantly, it may require extensive hyperparameter tuning which is often not feasible in practice).
3.2 Learning reaction-diffusion
Problem formulation. We next look at a reaction-diffusion system, where we add a diffusion operator to the reaction equation discussed above. Note that for pure diffusion, the solution dissipates
5

---

M.
64
04
(a) Exact solution for p
(b) PINN solution for p
Figure 2: Prediction error for 1D reaction- reaction-diffusion $3.2) problem. We can clearly see that the PINN has difficulty predicting the solution (especially the "sharpness" of the solution) and is unable to capture the correct behavior. Additional figures for different values can be seen in Fig. D.I.
a steady-state of uniform/constant distribution, which may be trivial to lear. Therefore, we consider studying the reaction-diffusion system:
du
8u
pu(l =0 x TE 9, te (0,T),
(10)
Or
ulz,0) h(z), x e Ω.
Here, (v > 0) is the diffusion coefficient. The solution of such a system can solved for via Strang splitting, i.e., splitting the equation into two separate models (a reaction component and diffusion component):
du di = pu u)
(11)
du 82
dt = Ox²
For each timestep, solve the reaction equation through Eq. 15 (in $A). The diffusion equation has the following analytical solution:
Uanalytical (z, t) ( t"))) -
(12)
where u(z,t t
t²) is the solution at the th time step. We solve the reaction equation for each timestep, and then use the reaction solution as the initial condition to solve the diffusion component and get the final solution.
The general loss function for this problem is,
1 ((0)
2 +
wb)
N₂
(13)
N₂ N₂
82 dx
A
pū(1 ú1)
+ Lis,
at
where Lg is the boundary loss. Similar to the previous example, periodic boundary conditions be enforced by including L8 from Eq. as an extra term the loss.
that the PINN also fails to learn reaction- diffusion. We illustrate a case in Fig. with 5, when y/ = 5. The PINN achieves a high relative error of 93%. Here, diffusion component. Additional figures for different P values can be seen in Fig. D. In particular, for v the PINN achieves relative enro of 50%. Here, we see that it is unable capture the "sharper" transitions, though it can predict the center of the solution a little better.
Observations. Similar to the previous
clearly that the PINN is unable capture either the reaction
4 Diagnosing possible failure modes for physics-informed NNs
Thus far, we have shown that PINNs can result in high errors even for simple physical regimes, in particular for PDEs/ODEs with non-tri con-trivial convection/reaciio//lffiisssss coefficients. Here,

---

Lock
(a) B
1.0
(b) 10.0
(c)
20.0
(d) 8 30.0
(e) 3 40.0
10
20
30
40
Relative error
7.84
10-3
1.08 x 10
7.50 x 10
8.97
10-1
9.61 X 10
Absolute error
3.17 X 10
6.03 X 10 4.32 x 10
5.42 x 10 5.82 X 10
Figure 3: Loss landscapes for varying values of B, for the convection example $3.1. The loss landscape is more smooth at low 3, and becomes increasingly more complex as 3 increases, which can make the optimization problem more difficult. In particular, at higher B, the optimizer gets stuck in certain regime. These results support that adding the PDE soft regularization term results in more complex optimization loss landscape.
demonstrate that one of the underlying reasons for this arises due to the PDE-based soft constraint of Cy, which makes the loss landscape difficult to optimize. We first (in $4.1) analyze the loss landscape to illustrate how increasing this soft regularization can lead to more complex loss landscapes, thus leading to optimization difficulties. We then (in ŠB) demonstrate how this is related to regularizing with differential operators, which can result in ill-conditioning.
4.1 Soft PDE regularization and optimization difficulties
Here, we analyze how the loss landscape changes for different regimes for the convection problem in $3.1 with/without the soft regularization in PINNs. We show that adding the soft regularization can actually make the problem harder to optimize, i.e., the regularization leads to less smooth loss landscapes. For all the experiments, across the first two dominant Hessian eigenvectors and computing the corresponding loss values. This tends to be more informative than perturbing the model parameters in random directions [41, 42).
plot the loss landscape by perturbing the (trained) model
Figure shows the loss landscape for the convection problem (discussed in $3.1), for different values. Interestingly, the loss landscape relatively low 8 1 is rather smooth, but increasing further results in complex and non-symmetric loss landscape. is also evident that the optimizer has gotten stuck in a local minima with very high loss function for large / values.
Finally, we study the impact of changing the weight/multiplier for the soft regularization term (i.e., the À parameter in Eq. 3), which can be relevant in improving PINN performance [35]. While we find that tuning À can help change the error, it cannot resolve the problem, as shown in Fig. E. Note that as the regularization parameter is increased, the loss landscape becomes increasingly more complex and harder to optimize (additionally, see the z-axis scale).
Expressivity versus optimization difficulty
In this section, first show that the failure modes we observed not necessarily due the specific NN architecture that we used in our experiments. In particular, we show that the NN model does have the expressivity/capacity
learn the convection/reccioodddfitttss coefficient cases where the vanilla PINN method fails. Additionally, in the process of demonstrating this, we also describe two methods that lead to significantly lower error rates. partieular, we show that changing the learning paradigm to curriculum regularization can make the optimization problem easier to solve (as discussed in $5. 1). Second, we show that posing the problem as sequence-to-sege to-seueeccee learning may lead to better results than learning the entire state-space at once (as discussed in $5.2).
5.1 Curriculum PINN Regularization
One may contend that the failure modes shown in $3 may be because the NN does not have enough capacity. Here, we show that this is not the underlying reason. To do so, devise a "curriculum
7

---

-egifar traiiing Caniostur trawing
30
25
2.5
030
X
193
10
5-
00.00
00.00
-275
0.
3.4
14
38
-1.0
50
32
64
66
16
Training duration
(a) Curriculum
(b) Regular training PINN solution for B = 30
(c) Curriculum training PINN
regularization schematic
solution for B 30
Figure 4: Schematic outlining curriculum regularization and example result for ID convection from $3.1 The training procedure for regular PINNs training versus curriculum PINN training for the convection example in $3.1. The regular PINN training only involves training regularization starts a lower B. trains a model, and then uses the weights of this model to reinitialize NN for training the next 3. The curriculum training approach is able to do significantly better (by almost two orders of magnitude).
30, while curriculum
Regular PINN
Curriculum training
ID convection: B = 20
Relative error
7.50 x 10-
9.84 x 10-3
Absolute error
4.32 10
5.42 x 10-3
ID convection: B = 30
Relative error
8.97 10-1
2.02 10 10-2
Absolute error
5.42 x 10-1
1.10 x 10-2
ID convection: 40
Relative error
9.61
10
5.33 10 10-2
Absolute error
5.82
10
2.69 x 10
Table 1: Training the PINN gradually more difficult problems improves performance. 1D convection example in $3.1. The curriculum training approach achieves significantly better errors.
regularization" method to warm start the NN training by finding good initialization for the weights. Instead of training the PINN to Iearn the solution right away for cases with higher B1p, we start by training the PINN on lower 31p (casier for the PINN to learn) and then gradually move to training the PINN higher Blp, respectively. We test these results for the examples in $3.1 and $A. This is somewhat analogous to curriculum learning in ML [1], but applied by progressively making the PDE/ODE harder to solve.
Figure 4 shows the training procedure for an example convection case ( (3.1)) with = 30. As Fig. 4(c) shows, the curriculum regularization approach results in much more accurate solution than regular PINN training. With curriculum regularization, the relative error almost two orders of magnitude lower. Additionally, this is true across all the other regimes that found regular PINNs fail, as shown in Tab. 1. In Fig. E.2, we also show that curriculum regularization not only decreases error significantly, but also decreases the variance of the error. In Fig. E.3, we see that curriculum regularization results in much smoother loss landscape as compared to regular PINN training.
Curriculum regularization also works well for the reaction example in SA. In this case, we start by training with low p value (reaction coefficient), and then increase gradually to higher p values. The results can be seen in Fig. E.4. We can see that the error 0.1 0.6 orders of magnitude lower for 2 - (when the regular PINN error is not as high), and then greatly decreases error by 1-2 orders of magnitude for = - 10. As we discussed before, PINN has difficulty learning sharp features for high values of p. However, the curriculum regularization overcomes this, for p 10, in Fig. E.4(c).
5.2 Sequence-to-sequene learning vs learning the entire space-time solution
The original PINN approach of [28] trains the NN model to predict the entire space-time at once (i.e., predict u for all locations and time points). In certain cases, this can be more difficult to learn. Here, we demonstrate that it may be better to pose the problem as a requence-to-sequene (seq2seq) learning task, where the NN learns to predict the solution at the next time step, instead of all times.

---

0.0
0.4
0.6
0.8
1.0
(a) Regular PINN training
(b) requence-to-ssquenee learning (model
trained every At)
x Initial condition points
Boundary points
Collocation points
Figure 5: Schematic outlining seq2xeq learning. contrast to regular PINN training, the solution in seq2seq learning is predicted for only one At step at a time. Then, the predicted solution used as the initial condition for the next segment. To allow fair comparison, we keep the total number of collocation points to be exactly the same in either approach. That collocation points for seq2seq learning in the right, and keep be the same the corresponding segment in left figure.
is
do not increase the number of
Entire state space
AI
0.05
Al = 0.1
V == 2, =
Relative error
5.07 x 10-1
2.04 x 10-2
1.18 X 10
Absolute error
2.70 x 10
1.06 x 10-
6.41 X 10 10-3
3, =
Relative error
7.98 10-1
1.92 10
1.56 x 10
Absolute error
4.79 x 10-1
1.01 x 10-2
8.17 X 10-3
== -= 5
Relative error
8.84 10
2.37
10-2
1.59 X 10
Absolute error
5.74 10-1
1.15 10
8.01 X 10-3
V 5.p =
Relative error
9.35 x 10-1
2.36 X 10-2
2.39 x 10
Absolute error
6.46 x 10
1.09 X 10
1.15 x 10-2
6, p
Relative error
9.60 x 10-1
2.81 x 10-2
2.69 X 10-2
Absolute error
6.84 x 10-1
1.17 X 10-2
1.28 x 10-2
Table 2: Predicting the entire state space versus discretizing the state space (hee, seq2seq learning) for ID reaction-diffusion $3.2). The seq2seq learning achieves lower error for buth At 0.05 and Al 0.1, in comparison the PINN's approuch of predicting the entire state space
This way,
use marching in timt scheme to predict different sequences/time points. Note that the only data available here is from the PDE itself, i.c., just the initial condition. We take the prediction t At and use this the initial condition make prediction at t 2At, and This is schematically outlined in Fig. 5.
We test this scheme by using the exact same NN architecture as in previous sections, and we report the results in Tab. E. for the convection problem $3.1, Tab. E.2 for the reaction problem of $A, and Tab. 2 for the the learning is posed as a seq2seq problem (i.e., predicting the state space with a "time marching scheme" of one timestep prediction at a time) the PINN approach of predicting the whole state space at once.*
-iittioondiffiio problem of $3.2. We compare the relative/absolute error when
We explore the following cases where the PINN does poorly, varying p, and coefficients:
"To have fair comparison between two methods (time marching versus predicting entire state space at once), for the time marching method, we use the same number collocation (interior) points for both. For example, for T= [0, 1]. if we use 1000 collocation points to predict the entire state space, then for At 0.1 we 100 collocation points per section.

---

M
(a) Exact solution for p
(b) Regular PINN prediction
(c) seq2seg PINN prediction for 5, V
for P 5,
Figure 6: Predicting the entire state space vs seq2seq learning for 1D reaction-diffsion. The regular
PINN is unable capture the "sharp" and/or diffusive features correctly. However, the seqZseq learning approach is able capture correct solution, and achieves almost two orders of magnitude lower error.
1) For ID convection ( $3.11). higher 3 values from 30-40.
2) For ID reaction (A), P coefficients from 5-10.
3) For ID reaction-diffusion ($3.2), fixed p = 5 and v coefficients from 2-6.
For these cases, we find that posing the problem as seq2seq learning results in significantly lower error. The difference particularly striking for the reaction and the seq2seq PINN model decreases error by almost two orders of magnitude. An example case is shown Fig. 6, where the seq2seq approach able to recover solution, while regular PINNs does very poorly. Note that this behavior also has analogues with numerical methods used in scientific computing, where space-time problems are typically harder to solve, as compared to time marching methods [8]. Intuitively, since the problem is ill-conditioned, restricting the dimensions is expected to help. Furthermore, the underlying function/mapping of the input to the solution should be much simpler to approximate over smaller time span, compared the full time horizon.
diffusion cases, where
These initial results are promising, and further developments may lead to still better ways of using PINNs and learning PDEs. In particular, using more sophisticated methods predict timesteps across the state space may provide improved performance, may including more sophisticated seq2seq approaches and tuning the regularization parameter (i.e., amount of constraint added).
6 Conclusions
PINNs SciML more generally-hold great promise for expanding the scope of ML methodology to important problems in science and engineering. For these problems, however, integrating ML methods with PDE-based domain-driven constraints as a soft regularization term can lead to subtle and critical issues. In particular, show that this approach results in failure modes for learning relevant physics commonly used in different fields of science. fundamental PDE problems of diffusion and convection and showed
have fundamental limitations which
To show this, picked that the PINN only works for very simple cases, failing to learn the relevant physical phenomena for even moderately more challenging regimes. We then analyzed the problem characterize the underlying reasons why these failures occur. In particular, we studied the PINN loss landscape behavior and found it becomes becomes increasingly complex for large values diffusion or convection coefficients, and with/without non non-homogencous forcing. We also discussed that the problem is not necessarily due to the limited capacity of the NN, but that is partly an optimization problem resulting in the PDE- baseb soft constraint used in PINNs. Furthermore, PINN approach of solving for the entire space-time at once may not be efficient, and instead posing the problem as a and related issues will be critical if we hope to go beyond existing cut-and engineering more intimate connection between scientific methodologies and MI. methodologies. This will be needed to deliver on the promise of PINNs and SciML more generally.
showed that the
learning task can provide lower enror rates. Addressing these approaches, toward
10

---

7 Acknowledgements.
We are thankful to Shashank Subramanian for his feedback and contributions. We also acknowledge helpful discussions with Prof. George Biros, Geoffrey Negiar, and Daniel Rothchild. ASK was supported by Laboratory Directed Research and Development (LDRD) funding under Contract Number DE-ACO2-05CHI 1231 LBNL and the Alvarez Fellowship in the Computational Research Division at LBNL. AG was supported through funding from Samsung SAIT. MWM would also like acknowledge the UC Berkeley CLTC, ARO, NSF, and ONR. The UC Berkeley team also acknowledges gracious support from Intel corporation, Intel VLAB, Samsung. Amazon AWS, Google Cloud, Google TPU Research Cloud, and Google Brain (in particular Prof. David Patterson, Dr. Ed Chi, and Jing Li). Our conclusions do not necessarily reflect the position the policy of our sponsors, and no official endorsement should be inferred.
References
[1] Y. Bengio, J. Louradour, Collobert, and J. Weston. Curriculum learning. In Proceedings of the 26th annual international conference machine learning, pages 41-48, 2009.
[2] S. L. Brunton, B. R. Noack, and P. Koumoutsakos. Machine learning for fluid mechanics. Annual Review of Fluid Mechanics, 52:477-508, 2020.
[3] Y. Chen, L. Lu, G. E. Karniadakis, and L. Dal Negro. Physics-informed neural networks for inverse problems in nano-optics and metamaterials. Optics express, 2888):1 2020.
[4] M. Cranmer, S. Greydanus, S. Hoyer, P. Battaglia, D. Spergel, and S. Ho. Lagrangian neural networks. arXiv preprint arXiv:2003. 04630, 2020.
[5] M. Dissanayake and N. Phan-Thien. Neural-network-baale based approximations for solving partial differential equations. communications in Numerical Methods in Engineering, 1053):195-201, 1994.
[6] P. L. Donti, D. Rolnick, and J. Z. Kolter. Dc3: A learning method for optimization with hard constraints. arXiv preprint arXiv:2104. 12225, 2021.
[7] V. Dwivedi. N. Parashar, and B. Srinivasan. Distributed learning machines for solving forward and inverse problems in partial differential equations. Neurocomputing, 420:299-316. 2021.
[8] K. Eriksson, D. Estep. P. Hansbo, and C. Johnson. Computational differential equations. Cambridge University Press, 1996.
[9] B. Fornberg. A practical guide to pseudospectral methods. Cambridge university press, 1998.
[10] O. Fuks and H. A. Tchelepi. Limitations of physics informed machine learning for nonlinear two- phase transport in porous media. Journal of Machine Learning for Modeling and Computing, 1 (1), 2020.
[11] N. Geneva and N. Zabaras. Modeling the dynamics of pde systems with physics-constrain deep auto-regressive networks. Journal of Computational Physics, 403:109056 2020.
[12) S. Greydanus, M. Dzamba, and J. Yosinski. Hamiltonian neural networks. Advances in Neural Information Processing Systems, 32:15379- 2019.
[13] J. Han, A. Jentzen, and F. Weinan. Solving high-dimensional partial differential equations using deep learning. Proceedings of the National Academy of Sciences, 115(34):505-8510 2018.
[14] O. Hennigh, S. Narasimhan, M. A. Nabian, A. Subramaniam, K. Tangsali, Z. Fang, M. Rietmann, W. Byeon, and S. Choudhry. Nvidia TM, An ai-accelerated multi-physie simulation framework. In International Conference on Computational Science, pages 447-461. Springer, 2021.
[15) W. Ji, W. Qiu, Z. Shi, S. Pan, and S. Deng. Stiff-pinn: Physies-informed neural network for stiff chemical kinetics. arXiv preprint arXiv:2011.04520 2020.

---

[16] X. Jin, S. Cai, H. Li, and G. E. Karniadakis. Nsfnets (navier-stokes flow nets): Physics-informed neural networks for the incompressible navier-stokes equations. Journal of Computational Physics, 426: 129:10 2021.
[17] G. F. Karniadakis, I. G. Kevrekidis, L. Lu, P. Perdikaris, S. Wang, and L. Yang. Physics- informed machine learning. Nature Reviews Physics, 3(6):422-44)) 2021.
[18] E. Lagaris, A. Likas, and D. I. Fotiadis. Artificial neural networks for solving ordinary and partial differential equations. IEEE transactions on neural networks, 95))987-1000, 1998.
[19] Z. Long, Lu, X. Ma, and B. Dong. Pde-net: Learning pdes from data. In International Conference Machine Learning, pages 3208-3216 PMLR, 2018.
[20] L. Lu, X. Meng, Mao, and G. E. Karniadakis, Deepxde: A deep learning library for solving differential equations. SIAM Review, 63(1):208-22) 228, 2021.
[21] L. Lu, R. Pestourie, W. Yao, Z. Wang, F. Verdugo, and S. G. Johnson. Physics-informed neural networks with hard constraints for inverse design. arXiv preprint arXiv:2102.04626, 2021.
[22] P. Márquez-Neila, M. Salzmann, and P. Fua. Imposing hard constraints on deep networks: Promises and limitations. arXiv preprint 1rXiv:100.002025 2017.
[23] P. Moin. Fundamentals of engineering numerical analysis. Cambridge University Press, 2010.
[24] Y. Nandwani, A. Pathak, Singla, et al. A primal dual formulation for deep learning with constraints. Advances in Neural Information Processing Systems, 2019.
[25] D. Parisi, M. C. Mariani, and M. A. Laborde. Solving differential equations with unsupervised neural networks. Chemical Engineering and Processing: Process Intensification, 42(8-9):715- 721, 2003.
[26] C.
possible failure modes in physics-infrmmed http:/ggghubboomma/////////aaaee
neural networks.
fin---aallaee---messs 2021.
[27] C. Raackaukas,, Y. Ma, Martensen, C. Warner, K. Zubov, R. Supekar, D. Skinner, A. Ramad- han, and A. Edelman. Universal differential equations for scientific machine learning. arXiv preprint arXiv:2001.04338 2020.
[28] M. Raissi, P. Perdikaris, and G. E. Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378:686-7077 2019.
[29] M. Raissi, A. Yazdani, and G. E. Kariiadaiis. Hidden fluid mechanics: Learning velocity and pressure fields from flow visualizations. Science, 367(6481) 2020.
[30] Sahli Costabal, Y. Yang, Perdikaris, D. Hurtado, and Kuhl. Physics-informet neural networks for cardiac activation mapping. Frontiers in Physics, 8:42, 2020.
[31] Sirignano and K. Spiliopoulos. Dgm: A deep learning algorithm for solving partial differential equations. Journal of computational physics, 1775:333-1330 2018.
[32] B. P. van Milligen, V. Tribaldos, and J. Jiménez. Neural network differential equation and plasma equilibrium solver. Physical review letters, 35(20)33594, 1995.
[33] L. von Rueden, S. Mayer, K. Beckh, B. Georgiev, S. Giesselbach, R. Heese, B. Kirsch, J. Pfrom- mer, A. Pick, R. Ramamurthy, et al. Informed machine learning-a taxonomy and survey of integrating knowledge into learning systems. arXiv preprint /9X3v
2019.
[34] B. Wang, W. Zhang, and W. Cai. Multi-scale deep neural network (mscalednn) methods for oscillatory stokes flows in complex domains. arXiv preprint 1:Xiv:2009.12729, 2020.
[35] S. Wang, Y. Teng, and Perdikaris. Understanding and mitigating gradient pathologies in physics-informed neural networks. arXiv preprint arXiv:2001.04536. 2020.
12

---

[36] S. Wang, H. Wang, and P. Perdikaris. On the eigenvector bias of fourier feature networks: From regression to solving multi-scale pdes with physics-informa informed neural networks. arXiv preprint arXiv:2012.10047, 2020.
[37] S. Wang, X. Yu, and P. Perdikaris. When and why pinns fail to train: A neural tangent kernel perspective. arXiv preprint arXiv:2007.14577 2020.
[38] E. Weinan, J. Han, and A. Jentzen. Deep learning baasedg-basedd numerical methods for high- dimensional parabolic partial differential equations and backward stochastic differential equa- tions. Communications in Mathematics and Statisties, 5(4):349 -30,, 2017.
[39] Willard, X. Jia, Xu, M. Steinbach, and V. Kumar. Integrating physics-based based modeling with machine learning: A survey. arXiv preprint arXiv:2003.44919 2020.
[40] K. Xu and E. Darve. Physics constrained learning for data data-driven inverse modeling from sparse observations. arXiv preprint arXiv:2002. 10521, 2020.
[41] Z. Yao, A. Gholami, Q. Lei, K. Keutzer, and M. W. Mahoney. Hessian- analysis of large batch training and robustness to adversaries. Advances in Neural Information Processing Systems, 2018.
[42] Z. Yao, A. Gholami, K. Keutzer, and M. W. Mahoney. Pyhessian: Neural networks through the lens the hessian. In 2020 IEEE International Conference on Big Data (Big Data), pages 581-590. IEEE, 2020.
[43] Y. Zhu, N. Zabaras, P.-S. Koutsourelakis, and P. Perdikaris. Physics-constrained deep learning for high-dimensional surrogate modeling and uncertainty quantification without labeled data. Journal of Computational Physics, 394:56-81 -1,66-11222 2019.
[44] O. C. Zienkiewicz, R. L. Taylor, P. Nithiarasu, and J. Zhu. The finite element method, volume 3. McGraw-hill London, 1977.
13


Reviews:
[{"review_id": "qTlbiUogHm", "reviewer": "Reviewer_Cx4H", "summary": "This work studies two specific (yet essential) PDE learning cases in the PINN framework. Notably, the work proposes two main conclusions to explain the difficult training in PINN networks: \n- the loss landscape varies greatly with the PDE coefficient values (and works well with only low diffusion/conduction coefficient), \n- the loss landscape (at the end of training) varies greatly with the physics informed penalty coefficient. \n\nThe main explanation to such difficulties lies in the condition number associated to PINN penalty. The work proposes two solutions: a warm-up, and a sequence to sequence approach.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "First, I highlight that the paper addresses a significant problem of sciML community.\nThe presented work is well-written and straightforward. The presented arguments are sound and can be leveraged by the community to address the evoked problems.\n\nConcerning the proposed solutions. The investigation of Section 5.1 is interesting but only superficial, and therefore cannot lead to a solid conclusion. Also, the presentation of the solution 5.2 is in my opinion unclear.\n\nTwo main flows appears in the paper:\n1) The causes of the flows in learning PINNs are well identified however it is difficult to state whether the evoked flows were taken into consideration when designing the solutions. \n2) The part on the sequence-to-sequence learning is very poorly described and not clear, despite being a possible solution to the described problems even though experimental results are not compelling.\n\n\nMy questions are: \n1) About the warm states: how come such warm states perform better ? What is the distribution of the weights at the end of training vs in the beginning or vs a regular initialization ?\nCan the gradients of the parameters with respect to the loss be investigated in such a case ?\n2) A simple approach to the problem would be to gradually increase the PINN penalty, progressively enforcing the constraint. Can such a simple approach solve the condition problem ?\n3) Despite the vanishing/exploding gradients in RNN, the NLP community successfully leveraged simple optimization tricks such as gradient clipping ? Can the authors comment on the possible applicability of such optimization strategy ?\n4) Can’t interior domain points for supervision help solving the ill-condition of the learning ?\n\n\n____\n### post rebuttal\n____\n\nThe novel experimental results shows the advantage of the proposition to solve the highlighted issue. \nI have decided to raise my score to 6.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "see above.", "review_text": "", "extra_scores": {"ethical_concerns": "NA", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}, {"review_id": "WMlDwCHP5Qd", "reviewer": "Reviewer_4iUv", "summary": "The authors provide a detailed empirical analysis of the numerical difficulties encountered when training so-called \"physics informed neural networks\" (PINNs), a recent class of models which incorporate physical (often differential) equations into the loss function of a neural network. The authors perform a systematic analysis on two relatively simple but common use-cases in scientific literature, convection and diffusion equations, and demonstrate the PINNs fail to adequately learn the underlying dynamics of the system even in simple cases. They show that these failures are due to the effect of the convection/diffusion coefficients and forcing terms on the conditioning of the dynamical system, i.e. larger coefficients or forcings tend to make the problem increasingly ill-conditioned, thereby impacting characteristics of the loss function. The authors then propose two possible workarounds for these issues; improved initialization via a warm start or preconditioning and re-framing the problem as a sequence-to-sequence learning task.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, the paper is very well written, the experiments are thorough and well designed, and the results are both interesting and of great practical value for researchers and practitioners in scientific machine learning. The theoretical analysis of the Lagrange optimization problem, while relatively informal, provides valuable insight to readers who are less familiar with the field of non-convex optimization and should encourage researchers to think carefully about how arbitrary regularization terms affect the underlying optimization problem in PINNs and other similar methods.\n\nUltimately, the results presented by the authors might not be terribly surprising to those with a background in scientific/numerical computing, where the problem of ill-conditioned differential operators in numerical integration and optimization has been grappled with for decades. However, it's important to confirm intuition and expectation with rigorous experimentation, which is exactly what the authors have done here.\n\nIn summary, this work provides valuable experimental confirmation and analysis of pathologies in PINNs and will, I think, be a valuable contribution to the field. The paper appears pretty much ready to publish as-is, but I will make a few suggestions below.\n\n1. I missed a discussion on discretization strategies, both for the temporal and spatial derivatives. It is briefly mentioned in the supplementary materials that a uniform spatial grid spacing is used for convection (?), but that's all I can find. This is somewhat interesting since uniform grid spacing is typically the easiest case, so if the PINN can't even do that correctly, then it would be reasonable to expect it to fail on non-uniform grids as well. However, I would think that the resolution of the discretization would affect the numerical stability of the loss term, as finer meshes would result in more accurate gradients (assuming finite differencing is used) but also potentially more floating point error. Ideally, it would be nice to see an experiment demonstrating the sensitivity of the results presented here to discretization choices, but a brief discussion would also suffice, I think.\n\n\n2. It would be nice to see a few more details on training, e.g. the number of training samples (collocation points?), discretization details (as mentioned previously), the convergence criterion, etc.\n\n\n3. The results in figure D.1 are a bit puzzling to me. The authors claim \"As the regularization parameter is increased, the loss landscape becomes increasingly more complex and harder to optimize (see the z-axis scale)\" but this supposed increase in complexity is not apparent (at least to me) in the plots. The value of lambda that produced the lowest error values ($\\lambda = 0.01$) has a very similar loss landscape plot to the worst value ($\\lambda = 10.0$). In the latter case, the loss values are of course much higher, but that's to be expected as the value of $\\lambda$ itself partially determines the scale of the loss term. It would be more informative to see a breakdown of how each loss term (initial condition, boundary conditions, and the unscaled differential equation) changes in response to the value of $\\lambda$. I don't think the raw loss value is very informative.\n\n4. Minor stylistic suggestion: The qualifier \"Interestingly,\" is used, in my opnion, a bit too frequently throughout the text.\n\n5. I was a bit confused to find empty sections in the appendix and had to go to the supplementary material to find the figures/tables. I'm not sure if this is a standard formatting choice of NeurIPS or not, but I would suggest at least adding notes directing the reader to the supplementary materials, or removing the section headers from the main document altogether.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations and direction for future work are clearly discussed. I don't see any major societal impact issues, outside of perhaps concerns about potential misuse of PINNs in scientific research. However, the authors' work should, if anything, help ameliorate this.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "Mqw07MvHprt", "reviewer": "Reviewer_sTZN", "summary": "The paper studies the behavior of Physics Informed NNs (PINNs) on two common PDEs of physical relevance. It observes that PINNs fail to learn a good solution under standard training regime when the convection coefficient or viscosity coefficient is high, even when the closed-form solution does not depend on the constants.\n\nThe authors then seek to explain this behavior from an optimization standpoint. They plot the loss landscape along the two dominant hessian eigenvectors. They hypothesize that the optimization problem is ill conditioned due to the regularization term in the optimization objective that enforces the pde constraint.\n\nThey also run experiments to argue that sequence to sequence learning might work better for these ill conditioned problems.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think the paper tackles an important problem in PINNs, but it could go deeper in its investigations.\n\n- When diagnosing whether the loss landscape is ill conditioned, would it be possible to get more information about the spectrum of the hessian? It would be valuable to know bounds on the condition number, or some other investigations on the eigenvalue spectrum.\n- The authors give a heuristic scaling law for the condition number. Could you give a bit more detail on the derivation? Would it be possible to empirically verify this? You could compute the condition number for the operator discretized on a finer and finer grid.\n- The authors mention the condition number of the PDE operator, but isn't what we care about the condition number wrt the NN parameters? During optimization, we change the NN parameters and hope that doesn't lead to large changes in output. It would be interesting to know how the conditioning changes depending on NN architecture.\n- In the warm start experiments, from my understanding you only run these on problems where the solution does not depend on the parameter. In that case, wouldn't the weights not change when you move to a larger coefficient problem, since you warm start from the solution? It would be interesting to see warm start experiments on problems where the solution does depend on the coefficients.\n- The sequence to sequence solution at the end is mentioned, but the results seem mixed. It seems to work much better for diffusion vs convection. Do you have any insight on why this could be the case?\n\nI am borderline in this paper, currently leaning towards a rejection, but open to changing my score if most of my comments above are addressed.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations mentioned above.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hours"}, "extra_text": {}}, {"review_id": "GzfqbfrJ-B-", "reviewer": "Reviewer_LJn4", "summary": "In this paper the authors inspect the popular Physics Informed Neural Network (PINN) set of models and characterize their behavior (specifically their failure modes) in the context of two popular PDE systems, namely diffusion and convection. In each case, the authors demonstrate that PINNs learn effective representations in simple regimes while failing to learn representations of the PDE systems in more complex regimes. In addition, authors also demonstrate that it is not the inability of the neural network to fit the functions (due to some potential inability to represent non-trivial functions), but rather the way the PINNs incorporate the PDEs into the learning pipeline that is the reason for the failure. Finally, the authors also propose casting the PINN problem as a sequential prediction task (i.e., predicting all collocation points per time period) as opposed to tasking the neural network with simultaneously learning the full spatial and temporal domain. \n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall the paper is clearly written and appropriately organized. But for a few minor typos, there are not many problems with the text or narrative structure.\n\n**Positives:**\n1. In this paper, authors tackle an important question of characterizing the performance of the popular PINN model [1], in the context of two canonical PDE systems, namely the convection and diffusion PDEs and do indeed demonstrate failure modes of PINNs.\n\n\n2. Specifically, authors work in the context of 1D PDEs and showcase that for increasing values of equation coefficients (i. \\beta i.e., convection coefficient in the case of the convection equation; ii. \\nu i.e., viscosity in the case of the diffusion equation), PINN behavior degrades. Figures 1,3 in the paper demonstrate how the PINN prediction errors scale with increase in value of the equation coefficients (i.e., convection coefficient and viscosity respectively).\n\n\n\n3. Further, the authors in their experimental setup section clearly outline a rigorous evaluation procedure wherein results have been calculated based on multiple runs of a PINN model optimized with L-BFGS and executed using different random seeds. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "1. The proposed results although clearly demonstrative of certain failure modes in PINNs, offer an incomplete characterization and the solutions proposed (specifically the sequence-to-sequence approach) to alleviate the failure models is not fully detailed and demonstrated for sophisticated higher order PDEs (e.g., second order coupled PDEs like Navier-Stokes equation).\n\n2. The claim that the reason for PINN based failure is due to the “subtle problems”  introduced through the PDE based soft constraint of L_F is not fully characterized or explained. What are the set of “subtle problems” that are being discussed? One that is definitely demonstrated is that the loss-landscape gets more complicated. However, other problems could have to do with the interplay between the loss terms which aren’t discussed.\n\n    Speaking specifically to the point of the “interplay” between loss terms at different values of equation coefficients, it is imperative to demonstrate the influence of each of these loss terms on PINN learning dynamics to fully contextualize the failure. For example: at different values of $\\beta$ (or $\\nu$), how do the gradient distributions change w.r.t the gradient distributions of the other terms in the loss? \n\n4. The model re-formulation in 3.1.2 and 3.2.2 is unclear. For example, in the case of 3.1.2, are the $L_B$ , $L_U$ terms replaced by those in equation 9 and the value of the forcing term `q` replaced with the new value on line `178`? If so, wouldn’t $\\beta$ although absent in the analytical solution due to detailed simplifications), still have an influence on the PINN results?\nSpecifically (assuming the above point of $\\beta$’s influence on the PINN gradient holds true), it should be inspected how the gradient dynamics with increasing $\\beta$ (and correspondingly for increasing $\\nu$ in section 3.2.2) influence the learning especially at the initial training epochs of the network. Have these dynamics been inspected and if they have, what are the learnings?\n\n5. Has the addition of standard regularization terms (`L1, L2`, in conjunction with current PINN losses) been inspected and if so does that have a positive effect on the learned representation? \n\n\n6. There has been some previous work inspecting the effect of various physics based regularizers in PINN [2] and certain adaptive learning rate schemes have been proposed and shown to be effective. It is important to contextualize the current set of results in the context of this previously published work to fully characterize the failure modes of PINNs. \n\n**References:**\n\n[1]  Raissi M, Perdikaris P, Karniadakis GE. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics. 2019 Feb 1;378:686-707.\n\n[2] Wang S, Teng Y, Perdikaris P. Understanding and mitigating gradient pathologies in physics-informed neural networks. arXiv preprint arXiv:2001.04536. 2020 Jan 13.\n\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper addresses an interesting and important topic, and performs a careful and well motivated analysis, producing very practical lessons. There was active discussion of this paper -- 3 out of 4 reviewers raised their scores as a result, and all 4 reviewers vote for acceptance. I am therefore recommending that this paper be accepted.
Decision: Accept (Poster)