On the Role of Fixed Points of Dynamical Systems in Training Physics-Informed Neural Networks
Authors: Franz M. Rohrhofer
Stefan Posch
Clemens Gößnitzer
Bernhard C Geiger
Venue: Accepted by TMLR ()

Abstract: This paper empirically studies commonly observed training difficulties of Physics-Informed Neural Networks (PINNs) on dynamical systems.
Our results indicate that fixed points which are inherent to these systems play a key role in the optimization of the in PINNs embedded physics loss function.
We observe that the loss landscape exhibits local optima that are shaped by the presence of fixed points.
We find that these local optima contribute to the complexity of the physics loss optimization which can explain common training difficulties and resulting nonphysical predictions.
Under certain settings, e.g., initial conditions close to fixed points or long simulations times, we show that those optima can even become better than that of the desired solution.

Published in Transactions on Machine Learning Research (01/2023)
On the Role of Fixed Points of Dynamical Systems in Training Physics-Informed Neural Networks
Franz M. Rohrhofer
frohrhoferSaem.org
GmbH Research Center for Data-Driven Business & Big Data Anolytics
Sandgasse 86/4, 8010 Gruz, Austria
Stefan Posch
stefun.poschhlee..ee
LEC Cmbll Large Engines Competence Center Inffelgasse 19, 8010 Graz, Austrio
Clemens Gößnitzer LEC GmbHi Large Engines Competence Center Inffeldgasse 19, 8010 Graz, Austria
Bernhard C. Geiger Know-Center Center GmbH Research Center for Pata-Driven Business 8 Big Data Analytics Sandgasse 36/4, 8010 Gruz, Austria
geiger@iece.org
Reviewed on OpenReview: https: // optp//vee.....ee..e net/ form?? id= 56TTmVrggu
Abstract
This paper empirically studies commonly observed training difficulties of Physics-Informed Neural Networks (PINNs) on dynamical systems. Our results indicate that fixed points which are inherent to these systems play a key role in the optimization embedded physics loss function. We observe that the loss landscape exhibits local optima that shaped by the presence of fixed points. We find that these local optima contribute to the complexity of the physics loss optimization which culties and resulting nonphysical predictions. Under certain settings, e.g., initial conditions close to fixed points or long simulations times, we show that those optima can even become better than that of the desired solution.
the in PINNs
explain common training diffi-
1 Introduction
Dynamical systems governed by differential equations and are ubiquitous in many scientific disciplines including economics, biology, physics and engineering. The upsurge in scientific machine learning has led to the development of sophisticated approaches, such as Gauss-Markov processes (Scoober et al. 2014) numerical Gaussian processes (Raissi et al., 2018),, that are applicable to those systems and superior to classical methods. In the field of deep learning, state-of-tho-at methods have advanced by incorporating (at least)) some part the underlying physics, e.g., learned through data (Brunton et. al. 2016) or embedded by design (Sanchez- Goneclzz-Gozzlzzzz ct al.] 2020). Among those methods are physiccs-informed neural networks (PINNs) which are the prime paradigmn of plysics-informed machine learning (Raissi al.) 2019 Karniadakis
et al.) 2021). Their seamless integration of data and physical constraints has pushed PINNs into vast number of applications on dynamical systems, including system identification [Raissi
2018 hidden state
inference (Raissi et al. 2020 and surrogate modeling (Sun et al. 2020
Since PINNs are capable solving differential equations a fully mesh-free and time-contimuons manner, one promising field of application the numerical simulation of dynamical systems. In those applications, labeled training data is scarce and typically only used to specify the corresponding initial and boundary conditions (TC/BC). For complete and unique definition of the forward problem, the IC/BC either

---

Published in Transactions on Machine Learning Research (01/2023)
included in the loss function explicitly enforced using specific network architectures. Both variants, however, rely on the optimization of the embedded physics loss function, iee, on minimizing residuals on the governing differential equations, evaluated at collocation points which are randomly sampled inside the computational domain. Optimization success and accuracy thus particularly depend on the complexity of the studied dynamical system and the corresponding physics loss function.
1.1 Training Difficulties of Physics-Informen Neural Networks
In general, issues in the optimization of PINNs are manifold and often cause incorrectly predicted system dynamics. complete description of all reported training difficulties in PINNs is exhaustive; we thus focus our discussion on issues and proposed remedies which appear in the context of dynamical systems and relate what is discussed in our work.
1.1.1 Conflicting Objectives.
Several ongoing discussions address conflicts in the optimization of multiple objectives as one root cause of convergence issues in PINNs (Wang et al.] 2021a). It has been shown that different weighting of the objectives, bound to physical, IC/BC and data constraints, effective for PINNs and can accomplish a successful training. The weighting conducted either by hand-tuned loss weights or with adaptive weighting schemes that adjust the weights during network training, such as in Maddu et al. (2021) Jin et al. (2021) With a focus on coping with imbalanced gradients, those methods are generally used to improve the PINN's performance and to select an optimum point on the Pareto front (Rohrhofer et al. 2021). Furthermore, gpeciallydesiinned network architectures enable hard encoding of IC/BC and physical constraints (Lu et al. 2021 Raissi et al. 2019). These approaches circumvent conflicts by reducing the overall number competing objectives.
1.1.2 Propagation Failure.
Most relevant for our discussion are recent works that focus on a purported failure mode of PINNs in which the learned system dynamics does not represent the solution that is specified by the IC/BC. reason for this, it has been argued, that propagation of the solution from the enforced conditions to interior points is disrupted for a certain region in the computational domain, which often yields the trivial (zero) solu- tion (Daw et al.) 2022). To mitigate this issue, several remedies have been proposed. One focus lies in improving network initializations to reduce the bias towards flat output functions, e.g., by learning in simu- soidal space (Wong et al. 2021). Another type of methods propose rewcighting of collocation points (Wang) let al. 2022) or resampling them during network training [Leiterizz & Pflüger 2021) In those methods, importance or density of collocation points propagates from the enforced conditions to interior points dur- ing network training which, in a causalitt-reppecttngg manner, promises to mitigate the propagation failure. Since it generally argued that with an increasing domain size the physics loss optimization becomes more al., 2021), other methods focus on the extent of the computational domain. Often referred to sequence-to-seet to-sequence learning domain decomposition, those methods comprise approaches that divide the original spatio-temporal domain into smaller subdomains which are easier to solve (Jagtap & Karniadakis 2021). Furthermore, approximation issues of PINNs in the presence of high-frequency or multi- scale features have been explained by the spectral bias of PINNs with proposed remedies found in Wang
complex (Krishnapriyan
et al. 2021bb
1.2 Our Contribution
As shown in the last section, the literal ure is abound with techniques that try to mitigale commonly observed training difficulties PINNs but explanations why training on dynamical systems often fails seems incom- plate to us: We suspect that not only the trivial zero solution, but also fundamental properties, e.g., fixed points of dynamical systems play a key role in training (failures) of PINNs. Based on this, our contribution in this paper will be as follows.
2

---

Published in Transactions on Machine Learning Research (01/2023)
We show two simple dynamical systems that stable and unstable fixed points contribute to the optimization complexity of the physics loss function and influence the rate of training success. (Section 3.1.1
We empirically demonstrate that under certain settings, e-g., IC close to fixed points and long simulation times, nonphysical predictions become economical with better minima than that of the desired solution. (Section 3.1.2
We further demonstrate that PINN training for complex dynamical systems is also affected by fixed points inherent to these systems. (Section 3.2
We visually capture that the physics loss landscape is being shaped by the presence fixed points, which form local optima/ saddle points that slow down the gradient-based optimization and might prevent successful PINN training. (Section 3.3
We provide empirical evidence that the complexity of the physics loss landscape reduces for smaller computational domains and connect this to successful techniques used for training PINNs. (Sec- lions 3.3 and
2 Background
2.1
Dynamical Systems
In this work we consider dynamical systems that can be described by differential equations of the form:
up= Fluj,
(1)
where the solution function w - u(t,z) x) in general depends on time [0, and space x E C R", Ut denotes the (partial) derivative of u w.r.t time, and FF is an arbitrary, potentially nonlinear differential operator dictating the system dynamics. In the numerical simulation of dynamical systems, IC are imposed to define the initial state of the system through
u(to 0, uz(x), Vz E Ω.
2
For the dynamical systems considered in this work, the spatial domain Q is compact, i.e. closed and bounded, which further requires the specification of BC on the boundary an in order to guarantee the uniqueness of the solution:
Bluj = 0, Vz E an,
(3)
where boundary operator, specifying periodic, Dirichlet and/or Neumann BC.
2.2 Physics-Infonl Informed Neural Networks
Fully-connected neural networks (FC-NNs) are most common among PINNs due to their good trade-off between simplicity and expressive power. Thus, we use FC-NNs approximate the unknown solution function of I with u(t,z) z u(t, x; ut( where e R= are the weights and bias terms of the network. Common activation functions the hyperbolic tangent (tanh) Sigmoid linear unit (SILU, swish), which render the approximated solution function and derivatives smooth
PINNs use automatic differentiation (AD) (Baddin et al. 2018 to obtain (partial) derivatives of the network's output with respect to its inputs. In order to retrieve the derivatives of the network solution, AD requires to pass discrete evaluation points, called collocation points, through the network in a feed-forward operation. The collocation points define the data set for penalizing residuals of the differential equation. The physics loss residual for dynamical system [ is given by:
f(t,z) == (t,e (t)
(4)
Thuss mesh-free and time-continuous the context of differential equations.
3

---

Published in Transactions on Machine Learning Research (01/2023)
where we u to denote the derivative of the network function ug w.r.t time t. Following the standard PINN formulation, the sum of squared residuals at all collocation points yields the physics loss function:
N1 [2( N Σ ='),
(5)
with the collocation points {t x' sampled from the entire computational domain (t,x) E () x Ω. These points do not need any label and can be either fixed during PINN training or re-sampled before each training epoch. In the initial formulation of PINNs, additional data constraints are used to enforce the IC/BC by
No.
(0) = N
Σ ug x"") - u((", x²))
(6)
where is given by the r.h.s. of (2 and 33
Both losses 55
and
6 are combined by scalarization of
multi-objective optimization through
L( (L. L(()).
(7)
where À represents a weighting factor, which here by default set to x= 1, unless explicitly stated otherwise. As an alternative to this (standard) formulation of PINNs, special network architectures have been proposed that ensure IC/BC are satisfied explicitly. We refer to these PINNs as being hard constrained.
2.3 Stability, Fixed Points Steady-Start State
Fixed points w² of dynamical system are given by the roots of the nonlinear function F in equation
Flu1]=0 =0 O.
In general, they can be either stable, asymptotically stable or unstable. For stable fixed point, any trajectory close to it will stay close, whereas for asymptotically stable fixed point close trajectories will In contrast, unstable fixed point is repulsive and even the smallest
further converge to it f deviation will cause any close trajectory move away from it as t + co. While it is hard to determine all fixed points and their asymptotic properties in dynamical systems, the trivial zero solution ut = a fixed point for many systems, such as for harmonic oscillators, pendulums or fluid flow.
In the context of ordinary differential equations (ODEs) fixed points of dynamical systems are constant solutions which often studied from the perspective of stability, to characterize the asymptotic properties of solutions and trajectories close to it. For dynamical systems governed by partial differential equations (PDEs), fixed points appear in obtaining certain solutions to given system. Some applications aim at obtaining steady-state solutions, i.e. solutions that do not change over time as the state variables are changed and, thus, for which equation (8) holds truc. When soeking those solutions, it the asymptotic property of an underlying stable fixed point which determines what happens to the system in the long term after has been initiated. Other applications, however, consider the repulsive property of an unstable fixed point, bring the system out from steady-state and to obtain transient solution. An particular example for the latter is simulating vortex shedding which appears in fluid dynamics and will be also studied in this work.
3 On the Attractivity of Fixed Points in Physics-Informen Neural Networks
According to (8) if the parameters 0 of the PINN are such that ug u² corresponds to fixed point, then Fluol = 0 and the physics loss in (4 vanishes. Further, for PINN architectures with finitely many layers and neurons per layer, and for continuous activation functions, the network solution uo changes continuously with 0. Thus, 0' a sufficiently small perturbation of 6, v/ is approximately a fixed point solution, i.c., Flug] 0. As a consequence, the physics loss for u/g will be positive, but small. Hence, fixed points correspond global minima of the physies loss with non-trivial basins of attraction. The PINN can thus only learn "correct' behavior from the given IC/BC data. Regardless of whether this TC/BC data is incorporated
4

---

Published in Transactions on Machine Learning Research (01/2023)
using hard constraints loss term such as 6 the resulting loss landscape seen by the gradient- based optimizer will be affected by the fixed point solution and its basin of attraction.
In this section we now perform experiments to test the hypothesis that fixed points play a key role in the training of PINNs. In the first part, we perform experiments on two dynamical systems governed by ODEs which are the undamped pendulum and a simple toy example. Due to their simplicity, those systems are studied in multiple settings, including the variation of IC, simulations length, network and optimizer settings. In the second part, we show that our findings are also applicable to complex dynamical systems which are governed by PDEs. We perform experiments two complex dynamical systems which are the Navier- Stokes equations and the Allen-Call equation. Those systems are known to be challenging for PINNs and we relate commonly observed training difficulties to fixed points inherent to these systems.
3.1 Fixed Points in Ordinary Differential Equations
In the following, we consider two simple ODEs:
the undamped pendulum dynamics and a simple toy example. These examples are chosen because they exhibit stable (pendulum), asymptotically stable (toy example), and unstable (both) fixed points. We further try to solve these systems using either vanilla (i.e., multi-objective, for the pendulum) and hard constrained (for the toy example) PINNs. For both examples we now stick the convention of ODEs and denote the unknown solution function by y(t).
Undamped Pendulum Equation. The undamped pendulum dynamics are given by second-order ODE
i F =
9 sin (y),
(9)
with y denoting the angle from the vertical to the pendulum (in radians), and l and g representing the length of the rod and magnitude of the gravitational field, respectively. Here we set = 1 and g = 9.81 and consider for our discussion the angle y in units of degrees.
The second-order ODE 9 can be converted into coupled systems of two first-order ODEs which are in
the form of equation 1 Since PINNs are able to also handle higher-order differential equations, we use single neural network yo (L) to approximale the solution function and solve the second-order ODE.
This system exhibits two fixed points, a stable fixed point y² = 0 at the pendulum's natural rest position, and an unstable fixed point y" = 180 at the upright position. For comparison in our experiments, we create reference solutions using a Runge-Kutta fourth-order method.
Toy Example Equation. The toy example is defined a one-dimensional system with a single ODE given by
j ==
(10)
This system exhibits three fixed points, two of which located at y² = 11 are asymptotically stable, and one at y" 0 which is unstable (see Figure D For this system, the analytical solution exists and is provided in Appendix C.1 The simplicity of this toy example further allows for hard constraining the IC by setting
(t) = yo + t (t))
(11)
which per definition fulfills equation
2
3.1.1 Rate of Training Success in the Presence of Fixed Points
We now study the success rate of training PINNs on the above mentioned systems. We declare training successful if the L2 relative error (oo(t))y(t]la//// [l 22 is below 15%, when comparing the PINN's prediction with the reference solution after sufficiently long training time. This threshold allows clear separation of the training o
For the toy example nonphysical predictions are exclusively influenced by the unstable fixed point at y = 0 (see Figure [c)). For the undamped pendulum, however, we further classify whether an unsuccessful training,
2Code can be found GitHub at htttpe//giitubb.com wwwtchub..co//torhhae cow.//rohhhoeer//tte.. fixeed_______________ jpited__pitte httpe////tthbb...///oohohheeeeet
Results for further thresholds are provided in Appendix and suggest similar qualitative conclusions.
5

---

Published in Transactions on Machine Learning Research (01/2023)
Table 1: Undamped Pendulum. Rate of training success across different system settings (7 and yo) and network architectures (size and activation function). Triplcts in the main table represent in percentage (%) and in the respective order, cases of successful training, attracted by stable fixed point, and unstable fixed
point (see Appendix [.1 for details on the classification). Bold triplets represent low (< 5%) success rate.
T
2.5
5
7.5
yo
25
100"
175
25
100
175'
25
100
175'
tanh
98/ 08/2/0
100/0/0
100/0/0
0/100/0
90/10/0
0/100/0
0/100/0
0/100/0
0/61/39
4x50 swish
100/0/0
100/0/0
98/0/22
56/14/0
80/20/0
0/100/0
0/100/0
1/99/0
0/91/9
sin 100/0/0
100/0/0
100/0/0
65/35/0
93/7/0
0/93/7
2/98/0
24/75/1
0/94/6
tanh 100/0/0
100/0/0
99/0/11
100/0/0
97/0/3
92/0/8
49/31/20
84/0/16
1/29/70
8x100
swish 100/0/00
100/0/0
100/0/0
100/0/0
100/0/0
57/42/1
100/0/0
100/0/0
1/92/7
sin 100/0/0
100/0/0
98/0/2
55/0/45
60/0/40
39/15/46
32/0/68
29/0/71
0/26/74
subsequently the nonphysical prodiction, violates the physics by either being attracted by the stable (y" 09) or the unstable fixed point (y* = 180%). This classification is based on y and ÿ. i.e., in phase space, evaluated at t = T (see Appendix B.1 for details).
For both systems, train 100 randomly initialized PINNs with different IC yo and simulation time T. The IC for the undamped pendulum are enforced using the multi-objective loss 7 (vanilla PINN) with a zero initial angular velocity, i.e,, úe = O. Training is performed for 50k epochs using the Adam optimizer with default settings for the moment estimates. We repeat training for different setups, including network architectures and optimizer settings (see Appendix B.2 for details). Software and hardware specifications in use are found in Appendix A
Table T shows the rate of training success across different network architectures for the experiments on the undamped pendulun and an initial learning rale of = 0.001. The outcome of experiments using different optimization settings and for the toy example can be found in Appendix B.2 and C.2 In general, we observe severe training difficultics across all experiments with only a minor number of cases lcading to a success rate of 100%.
Influence of Initial Condition yo- From the results it is apparent that PINNs are sensitive the choice of initial conditions. We obscrve that, in general, IC close to fixed points lead to lower rate of training succoss compared to those that start far. For the undamped pendulum this is evident by comparing in Table the rates for yo 100 with that of yo 25 and yo = 175". We made similar observations for the toy example that also shows a lower rate of success for IC close to the unstable fixed point (see Appendix C.2
Influence of Simulation Time T. Across different settings and for both systems, we observe that training becomes more difficult as the simulation time is increased. Likewise it can be seen that reducing the simulation time accounts for a higher success rate. The influence of the simulation time in terms of the physics loss complexity will be further analyzed in Section 3.3
Influence of Network and Optimization Settings. We observe slight improvement in terms of a higher success rate as the network size is increased. Still, none of the tested network architectures could resolve the training issues at long simulation times and IC close to the (unstable) fixed point. Thus, we conclude that the observed training difficulties are bound to the optimization complexity, rather than insufficient expressive power of the network. As reported in Appendix B.2 we also perform an ablation study using different optimization settings in terms of the learning rate, number of collocation points, loss weighting and network initialization. In comparison to the baseline model (4x50, tanh, from Table D optimization setting yields notable changes to the rate of training success.
3.1.2 Fixed Points Becoming Economical Solutions
Next, we demonstrate that, when the IC is very close to a fixed point, training may result PINN predictions that approach these fixed points, even if the resulting behavior is nonphysical. Furthermore, we show that those nonphysical predictions can even become better minima than that of the desired optimum. The
6

---

Published in Transactions on Machine Learning Research (01/2023)
Data-Guided
Physics-Driven
109
X
Physics Driven
Guided
10:2
10
(a)
2
5
10
10
Ls,ml
109
10"4
10
10"d
10
10
10.23
10-12
25000
30000
25000
50000
10-1 10
10
10-1
10a
(b)
Epoch
(d)
Epoch
Figure 1: Toy Example. Predicted trajectories and learning curves for the (a).(b) data-guided PINN and (c),(d) physics-driven PINN. (e) The minimal physics loss across all epochs vs. absolute distance wetween IC and the unstable fixed point indicate that nonphysical predictions become better minima than the true solution. Five PINN instances were trained for each initial value (sequential color code) and approach (crosses and pluses). Markers were randomly shifted horizontally to reduce overlap
latter further renders convergence to the true solution unfeasible. To rule out effects from multi-objective optimization, we focus this subsection on the toy example implemented by a PINN with hard constraints. Similar observations on the undamped pendulum can be found in Appendix B.4
For this experiment, time of 7 = 10. To show that our chosen PINN architecture has sufficient expressive power to learn the true solution, we implemented the following, data-guided PINN as control: First, 10 labeled data points are sampled from the analytical solution, equidistantly from the compulational domain. In the first 25k epochs of training, these 10 points support training via (7), while in the remaining 25k epochs, the PINN is trained using the physics loss only. The first training phase thus guides the gradient-based optimization into the basin of attraction of the analytical solution, while the second training phase ensures that the physics loss is minimized. Indeed, as shown in Figure (b), the data-guided strategy successfully learns the analytical solution. We compare this approach to training with the physics loss only, i.e., without the use of labeled data, which we refer to as the physics- driven PINN. For cach approach and IC we train five uniquely initialized PINN instances with the same optimization seltings found in 3.1.1
4x50 network architecture with tanh activation and choose simulation
Influence of Initial Condition yo- For the toy example, Figure [c)-(e) show that the PINN predictions impacted by the unstable fixed point (y* 0) achieve lower physies losses as the IC gets closer to it. Indced, for small values of yo, the physics loss become even smaller than the physies loss achieved by the data-
guided PINN (see Figure I(e),, i.c., the prediction impacted by the unstable fixed point seems to become better minimum for PINN optinization than the true solution. This renders finding the true solution unfeasible, even for optimization methods that are not based on gradients (e.g., PSO-PINNs as in Davi & Braga-Neto 2022) or else).
3.2 Fixed Points in Partial Differential Equations
In the following, we broaden our discussion on two complex dynamical systems which are governed by PDEs: the Navier-Stokes equations and Allen-Cahn equation. We show that common training difficulties and freseedtly-obberveed nonphysical predictions on these systems can be explained by the presence of fixed points inherent to the experimental setups.

---

Published in Transactions on Machine Learning Research (01/2023)
Navier-Stokes Equations. The Navier-Stokes equations govern fluid flow and build coupled system of nonlinear PDFs. For the two-dimensional case, the unknown solution functions are u(t, v(t,a") and F),, representing the fluid velocity in x- and y-direction and pressure, respectively. The system equations for transient fluid flow, representing conservation of momentum in x- and y-direction, are given by
Ue = Fulu, u == (ut + vuy) P= + Re (ugg + Ugu).
(12a)
= Fu Mu, u.pl (umz + v0y) Py + Re Reeeeeerr + Ugy).
(12b)
In this work we consider incompressibilitty of the fluid which is given by the continuity equation
+ bvy O.
(13)
In order to hard constrain this additional PDE, we introduce a stream function (t Ø) with u dh and v = -V,, similar to what has been done in Raissi et al. (2019). Consequently, a single neural network can used to approximate (t, Z) and po(t, ã),, which per definition fulfills the contimuity equation 13
Allen-Cahn Equation. The Cahn equation describes a reaction-diffusion systems which used simulate phase separation in multi-component alloy systems. The highly non-linear PDE is given by
Ut
Flu) Ni + 72("- w²).
(14)
The following proposition (proof is given in Appendix E characterizes non-trivial fixed point inherent to this system:
Proposition 1. Consider PINN instance, where the collocation points are drawn from continuous
distribution supported the computotional domain approzimate the solution to the Allen-Cahn equation
1. 1] x [0, and suppose that the PINN is used to
14 Then, for the function
0,
I E 0.5,, 0.5]
u(z,t)
(15)
-1,
[0.5,
the physics loss (0) = with probability one.
3.2.1 Fixed Point Leading to Steady-State
For this experiment we consider the Navier-Stokes system 12 and aim at simulating vortex shedding, which is well-studied phenomenon and benchmark simulation in fluid dynamics. The simulation aims at capturing the characteristic properties of oscillating fluid flow, which appears in the wake of a (round) body that is passed by the fluid at a certain velocity. Of particular interest, in terms of stability, is that this transient and periodic flow obtained by the influence of a fixed point, which becomes unstable above a critical Reynolds number Re, 48 (Tang & Aubry, 1997p. The symmetry breaking, usually achieved by numerical instabilities in classical methods, pushes the symmetrical fluid flow out from steady-state, which conforms to the unstable fixed point, and into the oscillating vortex street.
Experimental Setup. The experimental setup for simulating vortex shedding can be found in Appendix D We choose Re - 100, thus a Reynolds Number high enough to cause the vortex vortex shedding. A database of direct numerical simulation data (DNS) for this Reynolds number can be found in Boudina (2021). This database contains the developed vortex shedding flow field and will be be used as training data for our experiment. We choose fixed 8x100 neural network architecture with tanh activation and consider following (two) training strategies: One PINN instance is used as control and trained on three consecutive periods of vortex shedding, ie., E [0, 18), with collocation points sampled from the same time domain. We refer to this data-guided PINN, which thus is fully supported by training data and only ask to correctly Iearn the reference solution. The second plysicc-driven PINN, however, is trained on labeled data coming only from the time domain E [0. 3), which represents 50% of the first vortex shedding period. This should impose the initial sequence of vortex shedding, and by sampling collocation points the full-time domain (t [0, 18)), the physics-driven PINN is asked to continue the simulation beyond the domain of reference data by minimizing residuals of the Navier-Stokes equations
12). Further details on optimization and data settings be found in A Appendix D
8

---

Published in Transactions on Machine Learning Research (01/2023)
Data-Guided
Physics-Driven
(a)
(b)
Fxlu, v.
Fylu, v,
1
0.5
0.0
0
0.5
(c)
5
15
(d)
5
15
Data-Guided
Physics-Driven
Figure 2: Navier-Stokes Equations. Substantial difference in velocity magnitudes | at t 12 predicted by the (a) data-guided and (b) physics-driven PINN. (c)-(d) The time evolution of the nonlinear operators Fe and Fy at (z, = (3.0, (red rectangle) indicates that the physics-driven PINN is influenced by the unstable fixed point which, apparently as Fz + 0 and Fy + 0 as bocomes large, attracts a stcady-stato solution. The green shaded area represents the initial training sequence for the maysics-drived PINN.
Results. Figure 2 shows for the fully-trained data-guided and physics-driven PINN the predicted velocity magnitudes |ū|] at t = 12, and the time evolution of the nonlinear operators Fr and Fg at a depicted spatial coordinate (x, = 035.. The data-guided PINN indeed resembles the true vortex shedding dynamics (within a certain accuracy not given), which verifies that the network architecture is sufficient large to learn the vortex shedding dynamics over the given time domain. For the physics-driven PINN, however, we observe that outside the domain of reference data the prediction starts to substantially deviate from that of the data- guided. The time evolution of the nonlinear operators Fz and Fy gives further insights, and indicates that the physics-driven PINN approaches steady-state solution with the fixed point properties ( as it is evident by F. + 0 and F, + as becomes large. While this behavior is only shown for particular spatial coordinate, similar observations are made on the entire spatial domain (not shown). These observations, together with the fact that the approached solution (Figure b))) is not the trivial zero solution, suggests that the training is influenced by the underlying unstable fixed point, which originally accounts for the vortex shedding dynamics. We note that evaluating the physics loss for both PINN instances shows that the minimal loss achieved by the physics-driven PINN (L 8.28- 10- 5) lower than that of the data-guided PINN (Ly, 2.07 10-4). We further note that have also tested different network and optimization set setlings, as well as (hard constrained) BCs and different computational domains, but we were not able to achieve, for any of these settings using standard PINNs, a different qualitative behavior as shown here, The challenge of this optimization problem was similarly reported in Chuang & Barba (2022) and can be resolved, e.g., by using truncated Fourier decomposition with PINNs Raynaud et al. 2022
3.2.2 Fixed Point Slowing Down Convergence
For this experiment we consider the Allen-Cahn equation
14] and define the IC and periodic BC as
u(0, x) = x² cos(TT))
x E [-1,], t [0.
(16a)
u(t, 1) = u(l,- -1), u, (,,(,, 1) = uz -1),
(16b)
(16c)
where we set 71 = 0.0001 and 722 = 5 in equation 14 This particular example was also studied in Mattey & Ghosh (2022) and Wight & Zhao (2020) and showed severe training difficulties for PINNs. In both works, the standard PINN approach resulted in ignoring the IC and learning the trivial zero solution. While a weighted PINN with À 100 in Wight & Zhao (2020) showed a slight improvement (by at least learning the
9

---

Published in Transactions on Machine Learning Research (01/2023)
50k Training Epochs
200k Training Epochs
1.0
1.0
1.0
10
0.5
0.5
0.5
0.5
0.0
(a)
t'= 0.0
t==
1.0
==0
('= 0,5
t= 1.0
u(x]l-ee
1.0
ubx]c-r
0.5
0.0
0.0
-0.5
00.5
-1.0
-1.0
(b)
Reference
Prediction
(d)
Reference
Prediction
Figure 3: Allen- Cahn Equation. Prediction of the PINN trained with (a) 50k and (c) 200k epochs using gradient-based optimization. (b), (d) While the prediction of the PINN trained with 200k epochs is in good agreement with the reference, the PINN trained with only 50k epochs still trapped at a solution which resembles for larger values of the non-trivial fixed point [5] This suggests that the training influenced by the fixed point that seems slow down the mradient-basee optimization (will be further analyzed in Section 3.3.2..
IC correctly), both works eventually propose methods that mitigate the observed training difficulties and fall into the category of domain decomposing adaptive collocation point sampling. The reference solution to this experimental setup can be found in Raissi et al. 2019).
Experimental Setup. For this example, we adopt the weighted PINN approach and choose a loss weight A 100 to put greater weight on the IC. We consider a 6x100 network and optimize the composite loss function that enforces the (weighted) IC, BC, and physics loss residuals on equation (14] on the full time domain & [0, 1). We train one PINN instance using Adam with 50k epochs and second instance with much longer training duration of 200k epochs, both with initial learning rate of a 0.001. At each training epoch, data for the IC/BC and collocation points are sampled anew with sizes Nic = 128, NBC 128 and Noot = 1024, respectively.
Results. Figure 3 shows the prediction of both instances on the entire spatio-temporal domain and on selected time steps. We observe that while both instances correctly learn the IC (and BC), the PINN which is trained on only 50k epochs approaches a solution that resembles the non-trivial fixed point given by (15) We note that these results coincide with that shown in Wight & Zhao (2020), although network and optimization settings. To our surprise, the PINN instance that iont the training for another 150k epochs successfully manages to escape from this suboptimal solution and converges to a solution that is in good agreement with the reference (with minor differences in the late time domain). This observation again suggests that the training is influenced by the non-trivial fixed point (15 that seems to slow down the gradient-based optimization. This particular example, together with the corresponding learning curves, will be further analyzed in the next section.
used different
3.3 Optimization Landscape in the Presence of Fixed Points
Finally, we investigate the eflect of fixed points on the physics loss landscape. In particular, we evaluate for the toy example and Allen-Cahn system the physics loss function on solutions that were found by the PINN instances and influenced by (close) fixed points. We also assess the effects of reducing the simulation length T by limiting the time domain up to which collocation points are sampled and used evaluate the physics loss
10

---

Published in Transactions on Machine Learning Research (01/2023)
y(t)
T=
T=
1,0
X 0.5
9>
(a)
2
4
6
(c)
01
(d)
01
t
Learning Curves
T =
10
10
Epach 25000
Epach 50000
10
25000
50000
01
81
(b)
Epoch
(e)
(f)
Figure 4: Toy Example. (a) PINN prediction and (6) corresponding learning curves for training example that barcly manages to escape from its suboptimal location. (c)-(f) Physics loss landscape evaluated on collocation points which are sampled up to T. The intermediate nouphysical prediction (diamond) clearly forms an attractive location in the loss landscape, which gradually disappears as is reduced.
3.3.1 Initial Condition Far From Fixed Point
For the toy example in Figure [d) we observe that most PINN instances suffer from slow convergence, indicated by the presence of platcaus the learning curves. Only a few examples show successful escape from those undesired optima which is evident by distinct drop the learning curves. Those cases are primarily reported at TC that comparably far from the unstable fixed point at y" 0. To obtain better understanding of this phenomenon, we plot the physics loss landscape and PINN prediction for instance (yo = 0.5 and T = =) 8) that barely manages to escape from its suboptimal location. The two plot directions 01 and 0g2 are particularly chosen to point from the initial network state (8") to an intermediate (025k) and the final state (g50k) of the network training. Further visualization details are given in Appendix
Results. Figure 4 shows the loss landscape for two directions 0, and 92, where panel (c) represents the loss landscape seen curing the PINN optimization. Additionally, in panel (a) and (b) a prediction and training sequence of the PINN is given, which shows that the gradient-based optimization first gets trapped in local optimun. After approximately 42k epochs, the optimization manages to converge lo the correct solution. We clearly detect the global minimum, which corresponds to this solution, in the upper right region (plus) of the loss landscape. We further observe that for long simulation times, i.c. in panel (c)- (e), a local minimum or saddle point (diamond) forms, which seems to be attractive to the gradient- based optimization (ef. Figure D and apparently conforms to the nonphysical prediction approaching the unstable fixed point. With decreasing T, i.e. with a reduced time domain on which the physics loss is evaluated, this local optimum gradually vanishes and the global optimum becomes casier to reach due to the better (in terms of optimization) shape of the loss landscape. This explains why we observe in Table 4 high rate of training success when using T = 2.5.
3.3.2 Initial Condition Close To Fixed Point
For our final investigation, we use the PINN instance from the AllenCCahn system which initially was also trapped a nonphysical prediction and finally escaped from the training continued (cf. Figure 3 In this example, the IC given by (16a) and the resultant dynamics closely pass the fixed point (15) which seemed to affect the PINN training. The PINN optimized using composite loss function with
11

---

Published in Transactions on Machine Learning Research (01/2023)
t.5
t'= = 1.0
T= 1.0
T= 0.5
H
u(x]]==e
0.5
0.0
0.0
-0.5
-0.5
-1.0
01
01
(a)
(6)
(d)
X
Learning Curves
0.1
T 0.01
10a
Epoch 80000
Epoch
10
10
0
100000
200000
01
01
(c)
(f)
(g)
Epoch
Figure 5: Allen-Cahn Example. (o).(b) PINN prediction and (c) corresponding learning curves for training example that barely manages to escape from its suboptimal location. (f)-(g) Physics loss land- scape evaluated on collocation points which are sampled up to T. The intermediate nonphysical prediction (diamond) clearly forms an altractive location in the loss landscape, which disappears as is reduced sig- nificantly.
À = 100. However, since the IC and BC were learned correctly, we only focus on visualizing the physics loss landscape. We plot the loss landscape in two particular directions 0 and 02 that point from the initial network state (8") to an intermediate (880k) and the final state (8200k) of the network training. Further visualization details are given in Appendix F
Results. Figure 5 shows the loss landscape for two directions On and 02, where panel (d) represents the loss landscape seen during the PINN optimization. Additionally, in panel (a),(b) and (c) predictions and the training sequence of the PINN is given, which shows that the gradient-based optimization first gets trapped in local optimum (diamond). We note that the PINN instance which was only trained on 50k epochs in Section 3.2.2 was still trapped at this local optimum which apparently conforms to the nonphysical prediction approaching the non-trivial fixed point (15)- After approximately 90k epochs, however, the optimization manages to escape from this optimum and finally converges to the solution which is good agreement with the true system dynamics. The minimum corresponding scen in the upper right region (plus) of the loss landscape. For this example, the simulation length T has to be reduced significantly, to minimum of about T = 0.01, in order to observe the suboptimal location (diamond) completely vanishing from the loss landscape. This observation is in accordance with the domain decomposition method introduced in Mattey & Ghosh (2022) which divides the full time domain in 50 segments, thus using a step size of At 0.02.
this solution can be clearly
4 Discussion and Limitations
Our study and results suggest that fixed points of dynamical systems, irrespective of whether they stable or unstable, lead to the formation of attractive optima in the physics loss landscape which influences the PINN training. This was demonstrated on series of experiments using two simple dynamical systems described by ODEs, and two complex dynamical systems described by PDEs.
On the Role of Fixed Points. As discussod in Section 2.3 fixed points of dynamical systems are given by the roots the nonlinear function Flu] in equation ( Physics loss residuals (4) are thus small by definition
since Flu) R 0 and, thus, 0 in the vicinity of fixed points (see preparatory discussion in 3 These
12

---

Published in Transactions on Machine Learning Research (01/2023)
properties seem to affect the training of PINNs: True fixed point solutions, i..., solutions that are constant (ODEs) or steady-state (PDEs), trivially fulfill the physics loss function and, thus, these solutions yield local optima in the physics loss landscape. Those optima are attractive to the gradient descent optimization, as we could show in our experiments. Additionally, the fact that residuals are small in the vicinity of fixed points may make more economical for the PINN to violate prescribed system dynamics locally, for the sake of settling at a "simpler" solution (Steger et al.). Effectively, fixed points, together with the L2 losses in allow PINNs to trade between severely violating the physics locally or approximating it inaccurately the entire computational domain.
Validity domain. The question may arisc as to which applications, in particular, the role of fixed points is critical in the training of PINNs. While, in general, this question is hard to answer due to the countless ways of studying dynamical systems, we can draw parallels between the systems studied in this work to provide an indication for similar expected behavior. First. of all, our studied dynamical systems are time-continuous and described by or least can be brought into the form of equation (I). Here we note that for any of our studied systems the differential operator F is not explicitly dependent on time 1. This restricts our consideration to so-called autonomous differential equations. However, fixed points predominantly appear only in those specific systems, and many of the dynamical systems studied in the litcrature of PINNs and found in engineering problems are autonomous, such as fluid flow, diffusion processes or Hamiltomian systems. Whether or not a fixed point in a particular application will influence the PINN training is determined by several factors as concluded from our experiments. Our results indicate that having an IC near a fixed point or a trajectory that passes by a fixed point closely, combined with extended simulation times, can increase the chances of encountering training difficulties due to the fixed point. We further believe that in the context of Lyapunov stability the neighborhood of fixed point, determined by the differential operator F, has immediate effect on the influence of the fixed point in the PINN training.
Remedies and Proposed Solutions. In Section L.1 we have already stated several remedies that have been proposed in order to overcome commonly-obsere was not to propose a new entry to this list of remedies, but rather to give an explanation and further insights why some of them may work for the specific type of training difficulties faced on dynamical systems. Proposed solutions in this list range from techniques that reduce the computational domain, i.c., the simulation time in the context of dynamical systems, to methods that reweight resample the dataset of collocation points. Our results showed that the simulation length T has an immediate effect on the optimization landscape, and that smaller computational domains are often characterized by smoother landscapes in which undesired minima, corresponding to fixed points, are less pronounced or disappear altogether. We believe that several of the proposed collocation point sampling schemes affect the optimization landscape a similar manner, effectively reducing the computational domain to regions in which collocation points are sampled densely (e.g., close to the boundary of the computational domain, as in Daw et al. (2022) or Wang et al. (2022) Finally, as we have argued in Section 3, only the provision of IC/BC data can prevent a PINN getting stuck in an undesired fixed point solution. Even so, the fixed point may still be attractive, leading either to small violations of the physics loss (as shown on the toy example in Section 3.1.2p or to small violations of the IC. This may partly explain the success of loss weighting schemes, and the fact that sometimes greater weights on the IC/BC loss are required to arrive at the desired solution |Wann et al. 2021a Maddu et al. 2021 Jin ct al. 2021).
training difficulties in PINNs. The aim of this paper
Limitations and Further Work. One may argue that the trade-off inherent in PINNs, or the multi- objective nature of their vanilla formulation 7 should be vacuous, as the true solution satisfies both physics and the IC/BC. Thus, nomphysical predictions should never represent a better optimum than the desired, physical solution. This is true, however, only for PINNs with unlimited expressive power. In practice, the expressivity of a neural network is always limited by its (necessarily) finite size. Therefore, the mentioned trade-off is effective, and we have reason to believe that there are settings where the desired solution does not correspond to a global optimum (cf. Figure I Future work shall investigate this aspect from a more theoretical perspective, instantiating approximation theorems for neural networks for PINNs
Further, one may argue that some of the observed nonphysical predictions simply appear because the PINN has not sufficiently converged. In other words, training was not long enough to depart from the (flat)
1C, which may correspond to a trivial solution of the differential equation (Wong et al. 2021 Leiteritz
13

---

Published in Transactions on Machine Learning Research (01/2023)
A large part of the literature propagation failures (see Section 1.1.2 points in this direction. Further, such a statement is supported by the fact that the physics loss of, e-g., the solution approaching the stable fixed point in Figure 6 is high, and by the late transitions to the correct solutions in Section 3.3 Indeed, we do not claim that minima of the physics loss formed by the presence of fixed points always correspond (good) minima of the full loss small computational domains), turn into saddle points that slow down convergence, or achieve wider basin of attraction and/or smaller loss than the minimum corresponding to the true solution, in the extreme case where ICs are very close to unstable fixed points.
& Pftüger 2021
these minima may disappear entirely (e.g., for
Finally, our investigations are based on selection of dynamical systems. We chose these systems because they are intuitive to understand, yet still exhibit nontrivial dynamics. Moreover, the particular choice of these systems allowed us to separate the effect of different types of fixed points and to at least partly exclude other explanations for the training difficulties of PINNs. Future work shall be devoted studying fixed points and stcady-state solutions, and to the wider spectrum of asymptotic properties of solutions to dynamical systems.
5 Conclusion
In this paper, we studied the physics loss optimization PINNs when applied to dynamical systems governed by differential equations. Our results revealed that nouphysical predictions appear as attractive optima in the physics loss landscape and seem to stem from the presence of fixed points inherent to dynamical systems. These minima saddle points potentially disrupt and trap the gradient descent optimization, leading to commonly observed convergence issues in PINNs. Reducing the computational domain yielded a greater rate of training success and, in general, reduced the complexity of the physics loss optimization. In the future, we believe that interdisciplinary research that includes advances in deep learning, stability theory and/or /or a further undorstanding of the underlying physics may improve physics-informed machine learning or benefit from it.
References
Atilim Gunes Baydin, Barak A Pearlmutter, Alexey Andreyevich Radul, and Jeffrey Mark Siskind. Automatic differentiation in machine learning: survey. Journal of machine learning research, 18, 2018.
Mouad Boudina. Numerical simulation data of a two-dimensional flow around a fixed circular cylinder, Junc 2021. URL http:///ooo..r//.////// wtt.://///eeeeoo.000000 http:e///oo..oog//....////eeeeoe 5039610
Steven Brunton, Joshua Proctor, and J Nathan Kutz. Discovering governing equations from data by sparse identification of nonlinear dynamical systems. Proceedings of the national academy of sciences, 113 (15):3932 3937, 2016.
Pi-Yuch Chuang and Lorena A Barba. Experience report of chysics-informet neural networks in fluid simulations: pitsalls and Irustration. arXie preprint arXiv:2205.14244 2022.
Caio Davi and Ulisses Braga-Neto. Pso-pinn: Physics-infie informed neural networks trained with particle swarm optimization. arXiv preprint wxxi:22022.01943 2022.
Arka Daw, Jie Bu, Sifan Wang, Paris Perdikaris, and Anuj Karpatne. Rethinking the importance of sampling in physics-informan neural networks. arXiv preprint arXto:2207.02338 2022.
Ameya D Jagtap and George E Karniadakis. Extended physics-informed neural networks (xpinns): A gen- eralized space-time domain decomposition based deep learning framework for nonlinear partial differential equations. In AAAI Spring Symposium: MLPS, 2021.
Xiaowei Jin, Shengze Cai, Hui Li, and George Em Karniadakis. NSFnets (Navicr-Stokes flow nets): Physics- informed neural networks for the incompressible Navier-Stokes equations. Journal of Computational Physics, 426:109951, 2021.
14

---

Published in Transactions on Machine Learning Research (01/2023)
George Em Karniadakis, Ioannis G Kevrekidis, Lu Lu, Paris Perdikaris, Sifan Wang, and Liu Yang. Physics- informed machine learning. Nature Reviews Physics, 3(6):422 4400 2021.
Aditi Krishnapriyan, Amir Gholami, Shandian Zhe, Robert Kirby, and Michael W Mahoney. Characterizing possible failure modes in physics-informed neural networks. Advances in Neural Information Processing Systems, 34:26548 26560, 2021.
Raphael Leiteritz and Dirk Pfttggrr How to avoid trivial solutions in physics-informed neural networks. arXiv preprinl ar Xiv:21 12. 05620, 2021.
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. Advances in neural information processing systems, 31, 2018.
Lu Lu, Raphael Pestourie, Wenjie Yao, Zhicheng Wang, Francesc Verdugo, and Steven G Johnson. Physics- informed neural networks with hard constraints for inverse design. SIAM Journal on Scientific Computing, 43(6))B1105 B1132, 2021.
Suryanarayana Maddu, Dominik Sturm, Christian L Müller, and Ivo F Sbalzarini. Inverse Dirichlet Weight- ing Enables Reliable Training of Physics Informed Neural Networks. Machine Learning: Science and Technology, 2021.
Revanth Mattey and Susanta Ghosh. A novel sequential method to train physics informed neural networks for allen cahn and cahn hilliard equations. Computer Methods in Applied Mechanics and Engineering, 390: 114474, 2022.
Maziar Raissi. Deep hidden physics models: Deep learning of nonlinear partial differential equations. The Journal of Machine Learning Research,
-95,, 2018.
Maziar Raissi, Paris Perdikaris, and George Em Karniadakis. Numerical gaussian processes for time- dependent and nonlinear partial differential equations. SIAM Journal on Scientific Computing, 40(1): A172 A198, 2018.
Maziar Raissi, Paris Perdikaris, and George Karniadakis. Physics-informet neural networks: deep learn- ing framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378:686 707, 2019.
Maziar Raissi, Alireza Yazdani, and George Em Karniadakis. Hidden fluid mechanics: Learning velocity and pressure fields from flow visualizations. Science, 367(6481):1026 1030, 2020.
Gaétan Raynaud, Scbastion Houde, and Frederick P Gosselin. Modalpinn: an extension of physics-informed neural networks with enforced truncated fourier decomposition for periodic flow reconstruction using limited number of imperfect sensors. Journal of Computational Physics, pp. 111271, 2022.
Franz M Rohrhofer, Stefan Posch, and Bernhard C Geiger. On the Pareto Front of Physics-Informed Informed Neural Networks. arXiv preprint 0tl.:2105000822 2021.
Alvaro Sanzaez-Gonzalez, Jonathan Godwin, Tobias Pfaff, Rex Ying, Jure Leskovee, and Peter Battaglia. Learning to simulate complex physics with graph networks. In International Conference on Machine Learning, pp. 8459 8468. PMLR, 2020.
Michael Schober, David K Duvenaud, and Philipp Hennig. Probabilistic ode solvers with runge-kutta means. Advances neural information processing systems, 27, 2014.
Sophie Steger, Franz M Rohrhofer, and Bernhard C Geiger. How pinns cheat: Predicting chaotic motion of a double pendulum. In The Symbiosis of Deep Learning and Differential Equations II.
Luning Sun, Han Gao, Shaowu Pan, and Jian-Xun Wang. Surrogate modeling for fluid flows based on plysics-constrained deep learning without simulation data. Computer Methods in Applied Mechanics and Engineering, 361:1125 12732, 2020.
15

---

Published in Transactions on Machine Learning Research (01/2023)
Shaojie Tang and Nadine Aubry. On the symmetry breaking instability leading to vortex shedding. Physics of Fluids, 909):2550 2561, 1997.
Sifan Wang, Yujun Teng, and Paris Perdikaris. Understanding and mitigating gradient flow pathologies physics-informed neural networks. SIAM Journal Scientific Compuding, 43(5):A3055-A33) A3088, 2021a.
Sifan Wang, Hanwen Wang, and Paris Perdikaris. On the eigenvector bias of fourier feature networks:
From regression to solving multi-scale pdes with physics- neural networks. Computer Methods Applied Mechanics and Engineering. 384:113938, 2021b.
Sifan Wang, Shyam Sankaran, and Paris Perdikaris. Respecting causality is all you need for training physics- informed neural networks. aixiv preprinl orXiv:2203.07400 2022.
Colby L Wight and Jia Zhao. Solving allen-cahn and cahn-hilliard equations using the adaptive physics informed neural networks ai preprint orXiv:2007.04444 2020.
Jian Cheng Wong, Chinchun Ooi, Abhishek Gupla, and Yew-Soon Ong- Learning sinusoidal spaces with ionsics-informed neural networks. Xr preprint 2r0 09338, 2021.
16

---

Published in Transactions on Machine Learning Research (01/2023)
Successful Training
180"
true
PINN
0"
-100"
Attracted by Stable Fixed Point
180
true
PINN
-160
-5
(b)
Attracted by Unstable Fixed Point
true
PINK
-5
(c)
Figure Undamped Pendulum. Representative examples of training cases from Table Training outcomes are classified as either (a) successful training, (b) attracted by stable fixed point, or (c) unstable fixed point. Left: predicted trajectories as function of physical time. Right: predicted trajectories in phase space show whether their end positions y and lie on, inside outside the orbit of the true pendulum dynamics. Green and red lines /circles represent stable and unstable fixed points, respectively.
A Software and Hardware Specifications
All code in our experiments is implemented in Python version 3.8 and TensorFlow version 2.9.1. Computa- tions are performed on a Nvidia Tesla T4 GPU with a memory size of 16 GB.
B Additional Content to Undamped Pendulum
This section provides further results in addition to the in Section 3.1.1 and 3.1.2 presented experiments.
B.1 Classification of Training Outcomes in Phase Space
In the main part of this paper, we classify unsuccessful training outcomes to be either influenced by the stable (y* 0%) or the unstable fixed point (y* = 180%). This classification conducted by determining the trajectories' end position in phase space, i.e., by evaluating y and (see Figure 6). Based on this, end positions that lie inside the periodic orbit of the true pendulum dynamics are classified being attracted by the stable fixed point, whereas positions that lie outside the orbit considered as being attracted by the unstable fixed point. Here we note that trajectories of unsuccessful training outcomes by default leave the periodic orbit which renders this separation possible.
B.2 Rate of Training Success Optimization Settings
In Table [ all PINN instances are optimized with an initial learning rate a = 0,001, number collocation points N. 64, loss weighting factor and as network initialization the Glorot uniform initializer. In addition to this, we also test different optimization settings using the 4x50 network architecture with tanh activation base model.
17

---

Published in Transactions on Machine Learning Research (01/2023)
Table 2: Undamped Pendulum. Rate of training success across different system and optimization settings using the 4x50 network architecture with tanh activation. Triplcts in the main table represent in percentage (%) and in the respective order, cases of successful training, attracted by stable fixed point, and unstable fixed point. The tested optimization settings are learning rate (a), number of collocation points (Ne), loss weighting factor (A) and network weights initialization (Init.) with He denoting the Нe uniform initialization. Baseline model (see Table 1 uses a = 0.001, No = 64, 11 and Glorot uniform initialization. Bold triplcts
represent low (< 5%) success rate.
T
2.5
5
7.5
90
25
100
175
25
100
175
25
100
175
Baseline
98/2/00
100/0/0
100/0/0
0/100/0
90/10/00
0/100/0
0/ 0/100/0
0/100/0
0/61/39
0.01
37/0/ 37/0/63
36/0/64
60/0/40
16/0/84
19/0/81
7/0/93
0/7/93
13/0/87
0/0/100
0.001
0/100/0
0/100/0
0/100/0
0/100/0
0/100/0
0/100/0
0/100/0
0/100/0
0/100/0
16
96/4/0
100/0/0
100/0/0
0/100/0
4/96/0
0/100/0
0/100/0
0/100/0
0/100/0
Ne
256
100/0/0
100/0/0
100/0/0
0/ 0/100/0
100/0/0
35/64/
0/99/1
0/100/0
0/89/11
0/100/0
61/39/0
93/7/0
0/100/0
0/99/1
0/85/0
0/100/0
0/100/0
0/100/0
10
100/0/0
100/0/0
100/0/0
11 11/89/0
42/0/0
6/85/9
0/99/1
0/100/0
0/35/
Int.
He
100/0/0
98/0/2
100/0/0
0 0/98/2
93/7/0
0/98/2
0/98/2
0/98/2
0 0/T3/27
Results to this experiment can be found in Table 2 where we included the baseline model with default optimization settings from Table as reference. In general, we observe that none of the tested optimization settings yields substantial improvement for IC close to fixed points and long simulation times.
B.3 Rate of Training Success Further Thresholds
In the main part of the paper, we use for the classification of successful training a threshold of 15% in terms of the Le relative error. To demonstrate that particular choice of this threshold does not contradict qualitative conclusions made in the main part, we further provide results using different thresholds. In particular, we validate the in Table presented training cases now with a threshold of 5% and 25% in terms of the L2 relative error.
The results can be found Table 3 For compactness, we only show the results for the 4x50 and 8x100 architecture with tanh activation. As apparent in the table, no substantial differences in the classified training outcomes can be observed when using different thresholds.
B.4 Fixed Points Becoming Economical Solutions
In the main part of this work, Figure shows for the toy example that nonphysical predictions can become better minima than that of the desired solution when the IC close to fixed point. To demonstrate similar qualitative observations for the undamped pendulum, we perform experiment similar to that in Section 3.12
In particular, we use a 8x100 network architecture with tanh activation and choose a simulation time of T = 10. Since for this simulation time physics-driven PINN instances will hardly converge to the true solution, we implement the same data-guided strategy as presented in Section 3.1.2) We include a total number of 100 labeled training points in the first half of the training. The data sampled from the Runge- Kutta solution, equidistantly in the computational domain. In the second half, the training continues with the physics loss optimization only. Indeed, show in Figure b), the data-guided strategy successfully converges to the true solution, representing successful outcomes and sufficient expressive power for the chosen PINN architecture. We again compare this approach to physics-driven training, i.c., without the use of labeled training data. We repeat for each IC the experiment with 10 uniquely initialized PINN instances per training strategy. We note that none of the physics-driven instances converges to the true solution (scc Figure b))). The unsuccessful training outcomes are further classified into whether the subscquent nonphysical prediction violates the physics by either being attracted by the stable or the unstable fixed point (see Figure D
18

---

Published in Transactions on Machine Learning Research (01/2023)
Table 3: Undamped Pendulum. Rate of training success using differently set thresholds in terms of the L2 relative error for the classification of successful and unsuccossful training. Triplets in the main table represent in percentage (%) and in the respective order, cases of successful training, attracted by stable fixed point, and unstable fixed point. Bold triplets represent low (< 5%) success rate.
T
2.5
5
7.5
Mo
25"
100
175'
25
100"
175
25"
100"
175"
L2 5%
98/2/0
100/0/0
100/0/0
0/100/0
90/10/0
0/100/0
0/100/0
0/100/0
0/61/39
4x50
L2 < 15%
98/2/0
100/0/0
100/0/0
0/100/0
90/10/0
0/100/0
0/100/0
0/ 1/100/0
0/61/39
L2 25%
99/1/0
100/0/0
100/0/0
0/100/0
90/10/0
34/66/0
0/ 0/100/0
0/100/0
0/61/39
L2 <, 3%
43/37/5 20
84/0/16
0/29/71
100/0/0
97/0/3
43/49/8
43/37 20
84/0/16
0/29/71
8x100
L2 15%
100/0/0
100/0/0
99/0/11
100/0/0
97/0/3
92/0/8
49/31/20
84/0/16
1/29/70
L2 <. 25%
100/0/0
100/0/0
99/0/1
100/0/0
97/0/3
92/0/8
55/25/22
84/0/16
1/29/70
In Figure 7a ()) we report the minimal physics loss values across all epochs for each training outcome. We observe, similar to the behavior in the toy example, that the PINN predictions attracted by the unstable fixed point (y* = 180%) achieve lower physics losses as the IC gets closer it. Furthermore, for yo = 175 the nonphysical solution becomes a better optimum than that of the desired solution. As a direct consequence, any physics-driven instance converges to it.
Minimal Physics Loss L
10-1
10:3
Successful Training (Data GuldeG) by Stable Fixe (Physlcs- Drivves)Drivnnn Attracted by Unstable Fixed Paint (Physics Driven)
10
(a)
La Relative Error
10
10a1
25*
50*
75
100*
125*
150*
175*
(b)
Figure 7: Undamped Pendulum. (a) Minimal physics loss across all epochs. (b) L2 relative error. While the data-guided PINNs converge the true solution (blue), the physics-driven PINNs yield incorrect system dynamics (large Le relative errors) by being either attracted by the stable (green) unstable (red) fixed point (see Figure G). For
175 the nonphysical solution becomes better optimum than that of the desired solution. In the figure, markers were randomly shifted horizontally to reduce overlap.
19

---

Published in Transactions on Machine Learning Research (01/2023)
2.5
5
7.5
0.001
0.01
0.1
0.001
0.01
0.1
0.001
0.001
0.1
tanh
100
100
100
12
14
4x50
swish
100
100
100
69
72
100
0
2
sin
100
100
100
13
5
61
3
12
tanh
100
100
100
0
3
0
1
10
8x100 swish
100
100
100
91
100
100
0
0
0
sin
100
100
100
1
13
0
3
14
Table 4: Toy Example. Rate of training success across different system settings (T and yo) and network
architectures (size and activation function). Numbers in the main table represent in percentage (%) cases of successful training. Bold numbers represent low (< 5%) success rate.
C Additional Content to Toy Example
C.1 Analytical Solution
The analytical solution to 10
is given by
1 -0 e e-2)
-1)e
-1/2
1 +
for 1 2 > 0,
v(()
0
for yo 0,
-1/2
1
1 -1) e-20)
for 0 > yo -.
C.2 Rate of Training Success
As defined in Section 3.1.1 we declare training successful the L2 relative error is below 15%. We show the rate of training success for the toy example using different network architectures in Table Similar to observations on the undamped pendulum (see Table we observe a low rate of training success for IC close the unstable fixed point (y" = 0) and long simulation times.
D Vortex Shedding
In this section we provide additional information to the experimental setup for simulating vortex shedding, including the computational domain, boundary conditions, well network and optimization settings. A publicly available database of direct numerical simulation data can be found in Boudina (2021).
D.1 Computational Domain and Boundary Conditions
The computational domain for this experiment where a cylindrical body is placed (x, (0 with diameter of d = 1. For our main experiment, the top/bottom boundary is considered as moving wall with no-slip conditions, where for the outlet zero- gradient conditions are applied. The cylinder boundary is chosen no-penetration and no-slip condition.
to the compact space (ar, v) 1= x|-1)00111001 10),
These BC are also shown in Figure 8
Here we note that we have also performed experiments with a different computational domain and different BCs (top/bottom boundary with symmetry wall and zero-gradient, and outlet with zero pressure), but none of the tested settings led to the desired vortex shedding motion, but showed similar qualitative behavior presented in the main part of this work.
20

---

Published in Transactions on Machine Learning Research (01/2023)
Cylinder
Outlet
u 0
Ou
0
Inlet
v 0
Tz
Ap
0
du
0
Oz
0
Op 0
BV
2z
Wall
dy
By
Figure Fluid Dynamics. Boundary conditions for vortex shedding.
D.2 Optimization and Data Settings
The overall loss function in this experiment is composed of the respective loss functions for the BCs, the physical and additional data constraints:
I(O)= Linlet (0) Loutict()) (0) + (f) LCylinder (0) (0) (.) IDaaaa
(17)
where Lf and L.g. are the physics loss functions for 12al and 12b respectively. Here, IDataa imposes the (initial) sequence of the developed vortex shedding flow field, which for the physics-driven PINN is 50% of the first period, and for the data-guided PINN three consecutive periods.
The data set for this initial sequence is sampled from the reference database with batch size of NIC.batch = 1024. At each batch iteration, collocation points and training data for the BC are sampled anew with sizes No = 1024, Ninlet 128, Noutlet = 128, Nwal = 256 and NCylinder = 128.
As already stated the main part of this work, we introduce a stream function (tt, F) with u v/ and u = - enforce contimuity, i.c., conservation of mass for an incompressible fluid. A single 8x100 neural network with tanh activation functions is then used to approximate (1) and po (t,
Different optimization settings have been tested throughout our work, but none of them led to the desired vortex shedding for the physics-driven PINN. For the sake of simplicity, we only state the settings used for the experiment presented in the main part of this paper: Optimization is performed using Adam with default settings for the moment estimates and a total number of 10k epochs. The initial learning rate is set to a = 0.001 and an exponential decay with rate 0.9 and step 1000 applied. Training for this setting and for the in Section listed software/ hetwwarrhardwrreee hhadwaree took about 7h.
E Proof to Allen-Cahn Equation
Proof. For the proof, note that for 15),, we have that 4, We hence remain investigate the right-hand
side of [14] To this end, note that u(x, () is piecewise constant. Specifically, we have that uz 0
for r' E [ 1, 1] \ {-0.5, 0.5], hence also Uex (x') 0. Since further, for all [-1,1] we have that (x, = u(x, t). the right-hand side of [1h is zero on x' E
Suppose now that the collocation points on which (0) is evaluated are drawn from a continuous distribution supported on [-1, 1]x [o. 71, i.e. from a distribution that is absolutely contimuus w.r.t. the Lebesgue measure on R Now, the subset (-0.5, 0.5) x [), is Lebesgue null set of IR2 hence the probability that collocation points are drawn from this set is zero. Since the physies loss evidently zero outside of this set, this completes
the proof.
21

---

Published in Transactions on Machine Learning Research (01/2023)
F Visualizing the Physics Loss Landscape
Visualization of the loss landscape is based on the work Li ct al. (2018) and is a two-dimensional projection. We plot the loss landscape (2) with two specific directions 01 and 02 as stated in the main part of this work. basic Gram-Schmidt process then used to obtain an orthonormalized set of the two directions. The loss landscape for different simulation times is obtained by evaluating the physics loss function on a total number of 1024 collocation points, sampled from the time domain E [].
For the toy example, loss values greater than 0() 0.2 were truncated to highlight the interesting domain in Figure 4 (f) Similarly, loss values greater than L(01, 02) 10 were truncated for the Allen- Cahn example where we also use a logarithmic presentation of the loss landscape to highlight the interesting domain in Figure 5 (g).
22


Reviews:
[{"review_id": "pk722PxDXv", "reviewer": "Reviewer_J9px", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## strength:\nThe message of this paper is quite simple: PINNs may get stuck in some local minima that correspond to fixed points of the dynamics. \n\n## weaknesses:\n- The experiments are very toyish (small neural networks and easy tasks). It is unclear if larger/deeper neural networks would achieve the same conclusion. \n- I found the notation $u_t$ for the derivative quite confusing since it is eventually mixed with $u_\\theta$ and $u_{\\theta,t}$ where the subscript $\\theta$ does not correspond to any derivation. \n- The paper does not investigate/propose solutions on how to fix that issue. However, I acknowledge that this may be beyond the scope of this paper. \n\n\n## summary\n\nOverall I feel like since this paper is an experimental paper, the evidence proposed by the author is relatively weak (toy experiments and small NNs) So, I feel that this paper is currently slightly under the acceptance threshold. Stronger experiments would be required to strengthen the claims. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper investigates the limitations of physics-informed neural networks (PINNs). \nThey show that (stable and **unstable**) fixed points in the PDE contribute to the optimization difficulties of PINN since they are attractive (in terms of loss minimization) for the PINN, thus leading the PINN to learn non-physical solutions. ", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- I would like to see experiments with larger/deeper NNs to see if this problem remains there. \n- Less toyish tasks. \n\nMinor: I would clarify the notation $u_{\\theta,t}$", "broader_impact_concerns": "No concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "gF2Oel45OA", "reviewer": "Reviewer_KDWh", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- the research question above sounds like a novel aspect of training physically-informed machine learning methods, and it is well grounded in the literature  as described by the authors in the introduction\n- the results are underpinned by the experiments on different models and illustrate the given phenomenon\n\nWeaknesses:\n- the main concern is about the sufficiency of evidence: the generalised conclusions of impact of fixed points  on the optimisation of physics-informed neural networks only rely upon a limited number of experiments; therefore the evidence for the claims should be improved, and that is the reason behind the answer on the question \"Are the claims made in the submission supported by accurate, convincing and clear evidence? \" is currently No (see comments in the requested changes section)\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper suggests, through a series of experiments over two toy examples and one complex dynamical system, Navier-Stokes equation, that fixed points shape local optima of the loss landscape. This would help inform the architecture and training of the model. The authors provide empirical evidence to support this claim. ", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Figure 1: it is important to clarify why cannot the given performance be just explained by the previously documented phenomenon when trained neural networks are giving good interpolations but bad extrapolations (see Ziyin et al, 2020; Davini et al, 2021)?\n- It is important to show the phenomenon for multiple complex dynamical systems. This may include varying the parameterisation of Navier-Stokes equation such as varying Reynolds number, as well as other dynamic systems (maybe Lorenz system?). The main concern, otherwise, is that while there is evidence on two particular toy problems, the evidence of generalisation to the more complex problems such as solving Navier-Stokes equation is not entirely conclusive. \n-In Table 1, although the general meaning is clear, the reviewer struggled to find the exact procedure for evaluation of the following values: 'attracted by stable fixed point, and unstable fixed point'. It is important that the authors clarify upon it. \n- Figure 4 gives good qualitative motivation why would the fixed point affect optimisation landscape; however, is it possible to conclusively clarify this for multiple tasks by showing landscapes for different tasks?\n\nDavini, David, Bhargav Samineni, Benjamin Thomas, Amelia Huong Tran, Cherlin Zhu, Kyung Ha, Ganesh Dasika, and Laurent White. \"Using physics-informed regularization to improve extrapolation capabilities of neural networks.\" (2021)\nZiyin, Liu, Tilman Hartwig, and Masahito Ueda. \"Neural networks fail to learn periodic functions and how to fix it.\" Advances in Neural Information Processing Systems 33 (2020): 1583-1594.", "broader_impact_concerns": "No broader impact concerns registered", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "MPHlJ21He7", "reviewer": "Reviewer_jz2A", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The paper formulates a clear hypothesis to diagnose the challenges in training PINNs.\n- PINNs are a timely and important topic in the machine learning literature, especially for applied mathematics and computational sciences. \n\nWeaknesses:\n\n- This study does not engage in any real hypothesis testing, rather it simply provides two minimal examples where fixed points influence the convergence of learning. Is it always the case? What about at the bottom of a quartic potential, where there's a stable fixed point with large fluctuations?\n\n- The evidence is purely numerical and is guided by models that are very simplistic. It's hard to connect the results obtained in the paper to the Navier Stokes solution presented at the beginning. \n\n- There is not a compelling explanation to demonstrate that this phenomenon is generic. The argument appears to be by analogy. \n\n- The connection to the Navier-Stokes example isn't very clear. The dynamics resembles an unstable fixed point, but is it?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper examines numerically the inconsistent training outcomes of physics informed neural networks (PINNs) in some simple systems after motivating the problem by discussing an unsuccessful application to the Navier-Stokes equation. The authors hypothesize that a lack of convergence arises due to fixed points in the dynamics of the target dynamical system. The effect of fixed points is studied from the perspective of the loss landscape and optimization dynamics for 1) learning the dynamics of an undamped pendulum and 2) a gradient system with two stable fixed points and an unstable fixed point.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I think in order for me to recommend this study for publication, there are several additional things I would need to see:\n\n1. There needs to be some actionable proscription. If a PINN is not converging well because of a fixed point, how should one simulate so as to improve training. \n\n2. There must be a better plausibility argument for why training fails near fixed points. The argument that stable fixed points lead to slow training dynamics could almost certainly be made more precise and illustrated in a more general setting. \n\n3. On a related note, some more foundational understanding of this phenomenon would be helpful to connect these observations to nontrivial systems.  ", "broader_impact_concerns": "No, none come to mind. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "75cOtgZBRs", "reviewer": "Reviewer_djNt", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n - The paper shows clear experimental evidence with good experimental setups, that the proposed problem in training PINNs has actually somethin to do with fixed points.\n - The experiments are support by physical and mathematical explanations, which ar eno proofs, but at least intuitively point to the empirical findings.\n\nWeaknesses\n - No proper solution to the problem is given, although some weakly mitigating effects are being demonstrated\n - Rather few test cases have been done, and mostly on ODEs", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper adresses a typical problem in PINNs: the existance of bad local minima in the loss functions, which correspond to overly simple solutions. The paper makes an empirical study using previously existing methods to collect evidence that these problems are connected to fixed points in the dynamical system. Section 3 show that the relevant problems exist for PDEs. Section 4 includes demonstrations on some examples, where fixed points probably induces bad learning behavior. The loss landscape on a toy example demonstrates that the proposed effect increases over long times, and hence makes PINN (as they are currently being used) unsuitable for learning of dynamical systems over long time periods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": " - Section 2:\n   - Not every dynamical system is of the form given in (1), and not even every dynamical system can be described by differential equations. Please be honest and state that you are considering dynmical systems as in (1), which is certainly an imporant class. (For example, the system in (10) is not of this form, although the system (10) can of course easily be modified to satisfy (1).)\n   - Please state necessary assumption on $\\Omega$, e.g. bounded, compact, simply connected, regularity assumptions on its boundary, ...\n   - Is your boundary operator necessarily linear?\n   - After Formula (5): you probably mean $\\times$ instead of $\\otimes$. (See also Appendix B)\n - Section 3:\n   - Please clearly state the assumptions made on the solutions of the NS equations, in particular on (in)compressibility, in the view of using the stream function.\n - Section 4:\n   - Plots do not really have colorblind people in mind. And they do not really work in grayscale.\n\nThis is more a suggestion than a requirement: the literature on PINNs is well-cited. One might (but not must) cite alternative approaches to differnetial equations using ML, e.g. (amonst many) the following arxiv links: 1406.2582, 1703.00787, 1801.09197, 2103.10153, 2103.12959, 2110.11812, 2202.01287, 2205.03185, 2208.01565, 2208.12515 ", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper analyses the training dynamics of physically-informed neural networks (PINNs), especially to study their failure cases. 
The main hypothesis in this work is that fixed points in differential equations can negatively impact the loss landscape when training PINNs, leading to nonphysical predictions. The authors conjecture that these local minima are responsible of the convergence issues usually observed with PINNs, especially for initial conditions close to fixed points or for long forecasting horizons.

The submission initially received mixed reviews, with two slightly positive and two slightly negative recommendations. The main concerns pointed out by reviewers related to the absence of theoretical analysis, the lack of solutions to overcome the problem, the experiments on simplistic tasks and contexts, and issues on paper presentation. During the discussion period, the authors provided a new set of experiments on the Allen-Cahn equation, clarified the focus of their work, and changed the paper's structure and presentation. The reviewers acknowledged the improvements made in the revised version. After the discussion period, there was on accept, one leaning accept and two leaning reject recommendations.


The AE carefully reads the submission and discussions. He considers that the claims made in the submission are overall supported by evidence. The experiments are composed of a fair spectrum of problems: although some of them are simple, the added experiments on the Allen-Cahn equation strengthen the empirical validation. The submission addresses issues in the training dynamics of PINNs, which is of great interest for the community, and this work can certainly inspire follow-up works on theoretical aspects or more complex tasks.
The AE thus recommends paper acceptance under the request of a minor change about the validity domain of submission's claims. This could be included in section 4, and echoes the remarks made by reviewer KDWh and jz2A, e.g. to discuss if there are any classes of ODEs/PDEs where the assumptions made in the submission do not hold, making the training dynamics behave differently as it is expected in this work. 
Decision: Accept (poster)