Mitigating Propagation Failures in PINNs using Evolutionary Sampling
Authors: Arka Daw
Jie Bu
Sifan Wang
Paris Perdikaris
Anuj Karpatne
Venue: Submitted to ICLR 2023 (2023)
Keywords: Physics-informed Neural Networks
Adaptive Sampling
Failure Modes of PINNs.

Abstract: Despite the success of physics-informed neural networks (PINNs) in approximating partial differential equations (PDEs), it is known that PINNs can sometimes fail to converge to the correct solution in problems involving complicated PDEs. This is reflected in several recent studies on characterizing and mitigating the ``failure modes'' of PINNs. While most of these studies have focused on balancing loss functions or adaptively tuning PDE coefficients, what is missing is a thorough understanding of the connection between failure modes of PINNs and sampling strategies used for training PINNs. In this paper, we provide a novel perspective of failure modes of PINNs by hypothesizing that the training of PINNs rely on successful ``propagation'' of solution from initial and/or boundary condition points to interior points. We show that PINNs with poor sampling strategies can get stuck at trivial solutions if there are propagation failures. We additionally demonstrate that propagation failures are characterized by highly imbalanced PDE residual fields where very high residuals are observed over very narrow regions. To mitigate propagation failures, we propose a novel evolutionary sampling (Evo) method that can incrementally accumulate collocation points in regions of high PDE residuals with little to no computational overhead. We provide an extension of Evo to respect the principle of causality while solving time-dependent PDEs. We theoretically analyze the behavior of Evo and empirically demonstrate its efficacy and efficiency in comparison with baselines on a variety of PDE problems.

Under review as a conference paper at ICLR 2023
MITIGATING PROPAGATION FAILURES IN PINNs USING EVOLUTIONARY SAMPLING
Anonymous authors Paper under double-blind review
ABSTRACT
Despite the success of chhysics-iffomaan neural networks (PINNs) in approximating partial differential equations (PDEs), it is known that PINNS can sometimes fail to converge to the correct solution in problems involving complicated PDEs. This is reflected in several recent studies on characterizing and mitigating the "failure modes" of PINNs. While most of these studies have focused on balancing loss functions or adaptively tuning PDE coefficients, what is missing a thorough understanding of the connection between failure modes of PINNs and sampling strategies used for training PINNs. In this paper, we provide novel perspective of failure modes of PINNs by hypothesizing that the training of PINNs rely on successful "propagation" of solution from initial and/or boundary condition points to interior points. We show that PINNs with poor sampling strategies get stuck at trivial solutions if there are propagation failures. We additionally demonstrate that propagation failures are characterized by highly imbalanced PDE residual fields where very high residuals are observed over very narrow regions. To mitigate propagation failures, we propose a novel evolutionary sampling (Evo) method that can incrementally accumulate collocation points in regions of high PDE residuals with little to no computational overhead. We provide extension of Evo to respect the principle of causality while solving time-dependent PDEs. We theoretically analyze the behavior of Evo and empirically demonstrate its efficacy and efficiency in comparison with baselines on a variety of PDE problems.
INTRODUCTION
inysice-informed neural networks (PINNs) (Raissi al. 2019) represent a seminal line of work in deep learning for solving partial differential equations (PDEs), which appear naturally in a number of domains. The basic idea of PINNs for solving a PDE is to train a neural network to minimize errors w.r.t. the solution provided at initial/boundary points of a spatio-temporal domain, as well as the PDE residuals observed over a sample of interior points, referred to as collocation points. Despite the success of PINNs, it is known that PINNs can sometimes fail to converge to the correct solution in problems involving complicated PDEs, as reflected in several recent studies on characterizing the "failure modes" of PINNs (Wang et al. 2020 2022c Krishnapriyan failure modes are related to the susceptibility of PINNs in getting stuck at trivial solutions acting as poor local minima, due the unique optimization challenges of PINNs. In particular, note that training PINNs is different from conventional deep learning problems as we only have access to the correct solution on the initial and/or boundary points, while for all interior points the domain, we can only compute PDE residuals. Also note that minimizing PDE residuals does not guarantee convergence a correct solution since there are many trivial solutions of commonly observed PDEs that show 0 residuals. While previous studies on understanding and preventing failure modes of PINNs have mainly focused on modifying network architectures or balancing loss functions during PINN training, the effect of sampling collocation points on avoiding failure modes of PINNs has been largely overlooked. Although some previous approaches have explored the effect of sampling strategies on PINN training (Wang al.) 2022a Lu et al.) 2021), they either suffer from large computation costs or fail to converge to correct solutions, empirically demonstrated in our results.
al. 2021). Many of these
In this work, we present a novel perspective of failure modes of PINNs by postulating the propagation hypothesis: "in order for PINNs to avoid converging to trivial solutions at interior points, the correct solution must be propagated from the initial/boundary points to the interior points." When this propagation is hindered, PINNs can get stuck at trivial solutions that are difficult to escape, referred to as the propagation failure mode. This hypothesis is motivated from similar behavior observed in

---

Under review as a conference paper at ICLR 2023
numerical methods where the solution of the PDE at initial/boundary points are iteratively propagated to interior points using finite differencing schemes (Le 2007).
We show that propagation failures in PINNs are characterized by highly imbalanced PDE residual fields, where very high residuals are observed in narrow regions of the domain. Such high residual regions are not adequately sampled in the set of collocation points (which generally is kept fixed across all training iterations), making it difficult to overcome the propagation failure mode. This motivates us to develop sampling strategies that focus on selecting more collocation points from high residual regions. This is related to the idea of local-adaptive mesh refinement used in FEM (Zienkiewicz et al. 2005) to selectively refine the computational mesh in regions with higher errors.
We propose a novel evolutionary sampling (Evo) strategy that can accumulate collocation points in high PDE residual regions, thereby dynamically emphasizing on these skewed regions as we progress in training iterations. We also provide causal extension of our proposed Evo algorithm (Causal Evo) that can explicitly encode the strong inductive bias of causality in propagating the solution from initial points to interior points over training iterations, when solving time-dependent PDEs. We theoretically prove the adaptive quality of Evo to accumulate points from high residual regions that persist over iterations. We empirically demonstrate the efficacy and efficiency of our proposed sampling methods in a variety of benchmark PDE problems previously studied in the PINN literature. We show the Evo and Causal Evo able to mitigate propagation failure modes and converge to the correct solution with significantly smaller sample sizes as compared to baseline methods, while incurring negligible computational overhead. We also demonstrate the ability of Evo solve a particularly hard PDE problem-solving 2D Eikonal equations for complex arbitrary surface geometries.
The novel contributions of our work are as follows: (1) We provide novel perspective for character- izing failure modes in PINNs by postulating the "Propagation Hypothesis." " (2) We propose a novel evolutionary algorithm Evo to adaptively sample collocation points in PINNs that shows superior prediction performance empirically with little to no computational overhead compared to existing methods for adaptive sampling. (3) We theoretically show that Evo can accumulate points from high residual regions if they persist over iterations and release points if they have been resolved by PINN training, while maintaining non-zero representation of points from other regions.
2 BACKGROUND AND RELATED WORK
Physics-Informet Neural Networks (PINNs). The basic formulation of PINN (Raissi et al. 2017) is to use a neural network (I, t) to infer the forward solution u of a non-linear PDE:
Ue + NN 0, E X,t t [0,T]; u(I, 0) h(x), I E X; u(I, t) g(x, t), E [0, x E Ox
where z and t are the space and time coordinates, respectively, X' is the spatial domain, OX is the boundary of spatial domain, and T is the time horizon. The PDE is enforced on the entire spatio- temporal domain (Ω X [0, T)) on a set of collocation points (x.' (at, ( by computing the PDE residual (R t)) and the corresponding PDE Loss (C,) as follows:
Ro 1) D1 fo(s, () NN
N. N, > ((Ro(a,, 1
(0) = - (Rx(x))(
(2)
where C, is the expectation of the squared PDE Residuals over collocation points sampled from uniform distribution 44 PINNs approximate the solution of the PDE by optimizing the following overall loss function L = Xt) Coc (0) + (B), where L,e and Lse are the mean squared loss on the initial and boundary data respectively, and Xr. Aie. Abe. are hyperparameters that control the interplay between the different loss terms. Although PINNS can be applied to inverse problems, i.., to estimate PDE parameters from observations, we only focus on forward problems in this paper.
Prior Work on Characterizing Failure Modes of PINNs. Despite the popularity of PINNs in approximating PDEs, several works have emphasized the presence of failure modes while training PINNs. One early work (Wang et al. 2020) demonstrated that imbalance in the gradients of multiple loss terms could lead to poor convergence of PINNs, motivating the development of Adaptive PINNs. Another recent development (Wang et al.) 2022c) made use of the Neural Tangent Kernel (NTK) theory indicate that the different convergence rates of the loss terms can lead to training instabilities.
2

---

Under review as a conference paper at ICLR 2023
Large values of PDE coefficients have also been connected to possible failure modes in PINNs (Krishnapriyan et al. 2021). ). In another line of work, the tendency of PINNs to get stuck at trivial solutions due to poor initializations has been demonstrated theoretically in Wong et al. 2022 and empirically in [Rohrhofer et al. 2022). In all these works, the effect of sampling collocation points on PINN failure modes has largely been overlooked. Although some recent works have explored strategies to grow the representation of collocation points with high residuals, either by modifying the sampling procedure Wu et al. (2022) Lu et al. (202T): Nabian et al. (202T) or choosing higher- order L" norms of PDE loss Wang et al. (2022a), they either require a prohibitively dense set of collocation points and hence are computationally expensive, or suffer from poor convergence to the correct solution as empirically demonstrated in Section [ In another recent line of work on Causal PINNs (Wang et al., 2022b), it was shown that traditional approaches for training PINNs can violate the principle of causality for time-dependent PDEs. Hence, they proposed an explicit way of incorporating the causal structure in the training procedure. However, this solution can only be applied to dime-dependent PDEs, and as we demonstrate empirically in Section also requires large sample sizes.
PROPAGATION HYPOTHESIS: A NEW PERSPECTIVE OF FAILURE MODES IN PINNS
ground truth
3000
Ber. 5000
Ite: 20100
Iser. 50010
0-
-1.5 -1.0
-05
0.0
Figure 1: PINN solutions for a simple ODE: Max + k2 0 (k: 20) with the analytical solution, w = A (k:r) + B cos().).. We can see smooth propagation of the correct solution from the boundary point at x = 0 to interior points (x >> 0) as we increase training iterations.
What is Unique About Training PINNS? Training PINNs presents fundamentally different opti- mization challenges than those encountered in conventional deep learning problems. In a conventional supervised learning problem, the correct solution for every training sample is known and the training samples are considered representative of the test samples such that the trained model can easily be extrapolated on closely situated test samples. However, in the context of PINNs, we only have access to the "correct" solution of the PDE on the initial and/or boundary points, while not having any labels for the interior points in the spatio-tempores domain f2 Note that the interior points in Ω can be quite far away from the initial/boundary points, making extrapolation difficult. Further, training PINNs involves minimizing the PDE residuals over a set of collocation points sampled from Ω. However, minimizing PDE residuals alone is not sufficient to ensure convergence to the correct solution, since there may exist many trivial solutions of a PDE showing very small residuals. For example, u(x,& i) 0 is a trivial solution for any homogeneous PDE, which a neural network is likely to get stuck in the absence of correct solution at initial/boundary points. Another unique aspect of training PINNs is that minimizing PDE residuals requires computing the gradients of the output w.r.t. (z, t) (c.g.. Ug and uz). Hence, the solution at collocation point is affected by the solutions at nearby points leading to local propagation of solutions.
Propagation Hypothesis. In light of the unique properties of PINNs, we postulate that in order for PINNs to converge to the "correct" solution, the correct solution must propagate from the initial and/or boundary points the interior points as we progress in training iterations. We draw inspiration for this hypothesis from a similar behavior observed in numerical methods for solving PDEs, where the solution of the PDE at initial/boundary points are iteratively propagated to interior points using finite differencing schemes Veque 2007). Figure [ demonstrates the propagation hypothesis of PINNs for a simple ordinary differential equation (ODE).
Propagation Failure: Why It Happens and How to Diagnose? As corollary of the propagation hypothesis, PINNs can fail to converge to the correct solution the solution at initial/boundary points is unable propagate to interior points during the training process. We call this phenomenon the "propagation failure" mode of PINNs. This is likely to happen if some collocation points start converging trivial solutions before the correct solution from initial/boundary points is able reach
3

---

Under review as a conference paper at ICLR 2023
them. Such collocation points would also propagate their trivial solutions to nearby interior points, leading to a cascading effect in the learning of trivial solutions over large regions of f2 and further hindering the propagation of the correct solution from initial/boundaryy points.
To diagnose propagation failures, note that the PDE residuals are expected to be low over both types of regions: regions that have converged to the correct solution and regions that have converged to trivial solutions. However, the boundaries of these two types of regions would show sharp discontinuity in solutions, leading to very high PDE residuals in very narrow regions. A similar phenomenon is observed in numerical methods where sharp high-error regions disrupt the evolution of the PDE solution at surrounding regions, leading to cascading of errors. We use the imbalance of high PDE residual regions as a diagnosis tool for characterizing propagation failure modes in PINNs.
To demonstrate propagation failure, let us consider an example PDE for the convection equation: due + + 0, u(x, 0) h((z), where 3 is the convection coefficient and h(x) is the initial condition de
(see Appendix G for details about this PDE). In a previous work (Krishnapriyan et al. (02I), has been shown that PINNs fail to converge for this PDE for B > 10. We experiment with two cases, 3 = 10 and 3 50, in Figure 2 We can see that the PDE loss steadily decreases with training iterations for both these cases, but the relative enror w.r.t. the ground-truth solution only decreases for 3 = 10, while for 3 = 50, remains flat. This suggests that for 3 = 50, PINN is likely getting stuck at trivial solution that shows low PDE residuals but high errors. To diagnose this failure mode, we plot two additional metrics in Figure 2 to measure the imbalance in high PDE residual regions: Kesher-Pearson' 's coefficient of Skewness (Kokoska & Zwillinger 2000) and Fisher's Kurtosis (Kokoska & Zwillinger, 2000) (see Appendix H for computation details). High Skewness indicates lack of symmetry in the distribution of PDE residuals while high Kurtosis indicates the presence of a heavy-tail. For 3 10, we can see that both Skewness and Kurtosis are relatively small across all iterations, indicating absence of imbalance in the residual field. However, for 50, both these metrics shoot up significantly as the training progresses, which indicates the formation of very high residuals in very narrow regions-a characteristic feature of the propagation failure mode. Figure confirms that this indeed is the case by visualizing the PINN solution and PDE residual maps. We see similar trends of propagation failure for other values of 3 > 10 (see Appendix J.1).
Relaiive Eroer
10P
Roskltua
40
2000
Kartock
1500
1000
10-2
20
100
500
0
1.0
1.0
1.0
0.5
1.0
lresation
Iteration
Iteration
Exeracionn
Figure 2: Demonstration of propagation failure while solving the convection equation with B 50.
Exact Solution
Predicted Solution
PDE Residuals
Exact Solution
Predicted Solution
PDE Residudds
[ 10)
[ = 10)
( 10)
(A = 50)
(3 50)
(9 = 50)
3.0g
1.6g
1000
500
0,0
1.0
6.4
Figure 3: Demonstration of the Exact PDE solution, the predicted PDE solution, and the PDE Residual Field for the convection equation with B = 10 and / = 50 respectively.
Potential Remedies for Mitigating Propagation Failures. Since propagation failures in PINNS are characterized by skewed regions with very high residuals acting as "barriers" in the flow of information, algorithms that can selectively focus on high residual regions during training can potentially break such barriers and thus offer some respite. Recently, number of algorithms have been proposed to increase the importance of collocation points from high residual regions in PINN training, which can be broadly grouped into two categories. The first category includes methods that alter the sampling strategy such that a point x, is picked up as a collocation point with probability proportional to its residual, i.e., p(x,) x [R(x+)]*, where k 2 1. This includes residual- -aaed
4

---

Under review as a conference paper at ICLR 2023
adaptive refinement (RAR) methods (Lu et al. 2021 and its variants Wu et al. (0022) Nabian et al. (2021). where a dense set of collocation points Paense is maintained to approximate the continuous residual field R(xx,, and points with high residuals are regularly added from Pdense to the set of collocation points every K iterations according to a sampling function. A second line of work was recently proposed in Wang al. (2022a), where higher-order T.P norms of the PDE loss (e.g.. I were advocated to be used in the training of PINNs in contrast to the standard practice of using L norms, to ensure stability of learned solutions in control problems involving high-dimensional PDEs. Note that by using higher-order L" norms, we are effectively increasing the importance of collocation points from high residual regions in the PDE loss, thereby having a similar effect as increasing their representation in the sample of collocation points (see Theorem A. for more details).
Challenges with Potential Remedies. There are two main challenges faced by the potential remedies described above that limit their effectiveness in mitigating propagation failures in PINNs. (1) High computational complexity: Sampling methods such as RAR and its variants require using dense set of collocation points Pdense (typically with 100k 1M points spread uniformly across the entire domain) to locate high residual regions, such that points from high residual regions can added to the training set every K iterations. This increases the computational cost in two ways. First, computing the PDE residuals on the entire dense set is very expensive. Second, the size of the training set keeps growing every K iterations, further increasing the training costs at later iterations. See Appendix for a detailed analysis of the computational complexity of RAR based methods (2) Poor Prediction Performance: As we empirically demonstrate Section 5| both sampling- bbsed methods such as RAR and its variants as well as - PDE loss-based methods suffer from poor performance in converging to the correct solution for complex PDE problems. This can be attributed to several reasons. First, while increasing the value of k in sampling- based methods and in L" norm-based methods influences a greater skew in the sample set towards points with higher residuals, the optimal values of l or p are generally unknown for an arbitrary PDE. Second, choosing & loss (or equivalently, only sampling collocation points with highest residuals) may not ideal for the training dynamics of PINNs, as it can lead to oscillatory behavior between different peaks of the PDE residual landscape, while forgetting to retain the solution over other regions. Instead, we need a way to gradually accumulate points from high residual regions in the collocation set while having non-zero representation of points from other regions.
PROPOSED APPROACH: EVOLUTIONARY SAMPLING (Evo)
Before we present our proposed sampling approach, in the following we first describe the four motivating properties that we focus on in this work to mitigate propagation failures: (1) Accumulation Property: To break propagation barriers, the set of collocation points should evolve over every iteration such that they start from uniform distribution and accumulate in regions with high residuals until the PINN training process eventually resolves them in later iterations. This is similar to starting with an L loss and increasing the order of L norm if high residual regions persist over iterations. (2) Sample Release Property: Upon sufficient minimization of a high residual region through PINN training, collocation points that were once accumulated from the region needs to be released, such that can focus on minimizing other high residual regions in later iterations. (3) Uniform Background Property: At every iteration, the set of collocation points should contain non-zero support of points sampled from a uniform distribution over the entire domain Ω, such that the collocation points do not collapse solely high residual regions. (4) Computational Efficiency: While satisfying the above properties, we should incur little to no computational overhead in sampling collocation points from high residual regions. Specifically, we should be able to add points from high residual regions without maintaining dense set of collocation points, Pdense, and by only observing the residuals over a small set of N,, points at every iteration.
To satisfy all of the above properties, we present a novel sampling strategy termed Eyolutionary Sampling (Evo) that inspired by algorithms used for modeling biological evolution (Eiben et al.. 2003). The key idea behind Evo is to gradually evolve the population of collocation points every iteration by retaining points with high residuals from the current population and re-sampling new points from a uniform distribution.
Algorithm T shows the pseudo-code of our proposed evolutionary sampling strategy. Analogous to evolutionary algorithms developed in optimization literature (Simon| 2013), we introduce notion of "fitness" " F(x,) for every collocation point x, such that points with higher fitness are allowed to survive in the next iteration. Specifically, we define F(x,) as the absolute value of PDE residual of
5

---

Under review as a conference paper at ICLR 2023
Algorithm 1 Proposed Evolutionary Sampling Algorithm For PINN
I: Sample the initial population Po of N, collocations point Po f- {xe Mi {x=]
from a uniform distribution
x-" ~ (()), where S2 is the input domain (st [0, T] X).
2: for i= O to max iterations 1 do
3: Compute the fitness of collocation points X+ E P. as F(x+) = R((x)].
4: Compute the threshold r. N. اド= F(x,')
5: Select the retained population P. such that Pf fxp' F(xx²) > Tiet
6: Generate the re samsedd population P. + (xe' :xx' ~ (11),. s.t. IP$ + IPTI = N,
7: Merge the two populations P.+1 + P. U
8: end for
Xr, i.e., F(x,) |x At iteration 0, we start with initial population Po of N, points sampled from a uniform distribution. At iteration i, in order to evolve the population to the next iteration, we first construct the "retained population" P. comprising of points from P; with fitness values greater than Tú, i.e., P < {x/ F(x(.) > T.). where Ti is equal to the expectation of fitness values over all points in P.. The remainder of collocation points in P; are re- sampled from a uniform distribution, thus constructing the "re-sampled population" P. < {x," x, ~ {.))). The retained population and the re-sampled population are then merged to generate the population for the next iteration, P:+1- Appendix Figure X schematically shows the accumulation of collocation points from high residual regions in Evo over training iterations.
Note that at every iteration of PINN training, Evo attempts to retain the set of collocation points in P with the highest fitness (corresponding to high residual regions). At the same time, the PINN optimizer is attempting to minimize the residuals by updating @ and thus in turn affecting the fitness function. We first show that when F(x) is fixed (e.g.. when B is kept constant), Evo maximizes F(x) in the retained population and thus accumulates points from high residual regions. In particular, Theorem 4.1 shows that for a fixed F(x), the expectation of the retained population in Evo becomes maximum (equal to 2 when the number of iterations approaches oc.
Analysis of Evo:
Theorem 4.1 (Accumulation Dynamies Theorem). Let Fo R" + B+ be fixed real-values k- Lipsschit continuous objective function optimized using the Evolutionary Sampling algorithm. Then, the expectation of the retained population ExEp= [F(x)] 2 maxx F(x) ke as iteration 0o, for any arbitrarily small e >> O.
The proof of Theorem 4.1 can be found in Appendix C.I This demonstrates the accumulation
property of Evo as points from high residual regions would keep accumulating in the retained population and make its expectation maximal if the fitness function is kept fixed. However, since the PINN optimizer is also minimizing the residuals at every iteration, we would not expect the fitness function to be fixed unless a high residual region persists over a long number of iterations. In fact, points from a high residual region would keep on accumulating until they are resolved by the PINN optimizer and thus eventually released from Pr. Also note that Evo always maintains some collocation points from a uniform distribution, i.c., the re-sampled population Pa is always non-empty. Theoretical proofs of the sample release property and the uniform background property of Evo are provided in Appendix a.2 and Lemma C.I 1, respectively. We also provide details of the computational complexity of Evo in comparison with baseline methods in Appendix F showing that Evo is computationally efficient. Table in the Appendix summarizes our ability to satisfy the motivating properties of our work comparison with baselines. Note that Evo shares a similar motivation as local-ade aacal-adative mesh refinement methods developed for Finite Element Methods (FEM) (Zienkiewizz et al.] 2005). where the goal is to preferentially refine the computational mesh used in numerical methods based localization of the errors. It is also related to the idea of boosting in ensemble learning where training samples with larger errors are assigned higher weights of being picked in the next epoch, to increasingly focus on high error regions (Schapire) 2003).
4.1 CAUSAL EXTENSION OF EVOLUTIONARY SAMPLING (CAUSAL Evo)
In problems with time-dependent PDEs, a strong prior dictating the propagation of solution is the principle of causality, where the solution of the PDE needs to be well-approximated at time t before moving to time t At. To incorporate this prior guidance,
present Causal Extension of Evo (Causal Evo) that includes two modifications: (1) we develop a causal formulation of the PDE loss Ly that pays attention to the temporal evolution of PDE solutions over iterations, and (2) we develop

---

Under review as a conference paper at ICLR 2023
10f
PS (lrwek) PMX (kol)
FISS pheeeely FVNN Dis 4f
Prx
Max PDE Rieshuul
PINS (eewe)]
Moan PDR Reskhhul
PXX shower) POaK (lna)
Fvo
PINC (Ber))
10P
Shann
300
10
10
k0m3
7.5
2.5
Iteration
Eteration
Rtenaion
Eseraciion
Figure 4: Comparison of Skewness, Kurtosis, Max PDE Residuals, and Mean PDE Residuals over training iterations for PINN-fixed, PINN-dynamic, and Evo for convection equation with 3 = 50.
causally biased sampling scheme that respects the causal structure while sampling collocation points. We describe both these modifications in the following.
Causal Formulation of PDE Loss. The key idea here is to utilize simple time-dependent gate function g() that can explicitly enforce causality by revealing only portion of the entire time-dommin to PINN training. Specifically, we introduce a continuous gate function g() = (1 (a where y is the scalar shift parameter that controls the fraction of time that is revealed to the model, a = is a constant scalar parameter that determines the steepness of the gate, and is the normalized time, i.e., 8 = t/T. Example causal gates for different settings of y are provided in Appendix D We use gt) to obtain a We initially start with small value of the shift parameter (7- = 0.5), which essentially only reveals a very small portion of the time domain, and then gradually increase during training to reveal more portions of the time domain. For y 2 1.5, the entire time domain is revealed.
PDE residual loss as (0) = N Σ [R(", f,)1 * gtt))
Causally Biased Sampling. We bias the sampling strategy in Evo such that it not only favors the selection of collocation points from high residual regions but also accounts for the causal gate values at every iteration. In particular, we modify the fitness function as F(x,). [R(x,)] * g(t,). A
schematic illustration of causally biased sampling provided in Appendix D
How to Update y Ideally, at some iteration i, we would like increase at the next iteration only if the PDE residuals at iteration are low. Otherwise, 7s should remain in its place until the PDE residuals under the current gate are minimized. To achieve this behaviour, we propose the following update scheme for ? 7++ = % Tye ())() where fg is the learning rate and denotes tolerance that controls how low the PDE loss needs to be before the gate shifts to the right. Since the update in is inversely proportional to the causally- PDE residuals are large. Also note that increasing y also increases the value of g(t) for all collocation points, thus increasing the convergence, y attains a large value such that the entire time domain is revealed.
PDE loss CJ. the gate will shift slowly if the
PDE loss and slowing down gate movement. Upon
Table Relative L2 errors (in %) of comparative methods over benchmark PDEs with N, = 1000.
Convection (B = 30)
Convection (3 = 50)
Allen Cahn 200k
Epochs.
100k
300k
150k
300k
PINN (fixed)
107.5 = 10.9%
107.5 + 10.7%
108.5 = 6.38%
108.7 - 6.59%
69.4 4.02% 0.77 + 0.06%
PINN (dynamic)
2.81 + 1.15%
1.35+ 0.59%
24.2 + 23.2%
56.9
9.08%
Curr Reg
63.2 + 9.89%
2.65+1.444
48.9 + 7.1%%
31.5 16.6%
CPINN (fixed)
138.8 + 11.0%
11.0%%
106.5 + 10.5%
106.5 10.5%
48.7 + 19.6%
CPINN (dynamic)
52.2 : 43.6%
23.8
79.0 - 5.11%
73.2
1.5 - 0.75%
RAR-G
10.5 = 5.67%
2.66
65.7 : 17.0%
28.9%
25. : 23.2%
RAD
3.35 = 2.02%
1.85
66.0 1 1.55%
1.98%
0.78 1 0.05%
RAR-D
67.1 + 4.28%
22.0%95.889
82.9 5.99%
9.58%
51.6 0.41%
L
66.6 + 2.35%
+
76.6 - 1.04%
75.8
1.65 + 1.36%
Evo. (ours)
1.51 = 0.26%
0.78 + 0.18%
6.03 = 6.99%
1.98 +.72%%
0.83 - 0.15%
Causal Evo. (ours)
2.12 1 0.67%
0.75 1 0.12%
5.99 1 5.25%
2.28 1 0.76%
0.71 1 0.007%
RESULTS
Experiment Setup. We perform experiments over three benchmark PDEs that have been used in existing literature study failure modes of PINNs. In particular, we consider two time-dependent PDEs: convection equation (with 3 = 30 and / = 50) and Allen Cahn equation, and one time- independent PDE: the Eikonal equation for solving signed distance fields for varying input geometries.
7

---

Under review as a conference paper at ICLR 2023
We consider the following baseline methods: (1) PINN-fixed (conventional PINN using a fixed set of uniformly sampled collocation points), (2) PINN-dynamic (a simple baseline where collocation points dynamically sampled from uniform distribution every iteration, see Appendix [B] for more details), (3) Curr. Reg. (curriculum regularization method proposed in (Krishnapriyan et al. 202T)), (4) cPINN-fixed (causal PINNs proposed in (Wang et al. (2022b) with fixed sampling), (5) cPINN-dynamic (cPINN with dynamic sampling). (6) RAR-G (Residual-based Adaptive Refinement strategy proposed in (Lu al. (02I)), ) (7) RAD (Residual-based Adaptive Distribution originally proposed in (Nabian et al.) (021)) and later generalized in Wu et al. (2022)), (8) RAR-D (RAR with a sampling Distribution proposed in Wu al. (2022). ). (9) L (Sampling top N, collocation points at every iteration from dense set Paense to approximate L norm). For every benchmark PDE, we use the same neural network architecture and
settings across all baselines and our proposed methods, wherever possible. Details about the PDEs, experiment setups, and hyper- parameter settings are provided in Appendix I available at Comparing Prediction Performance. Table I shows the rela- tive La errors (over 5 random seeds) of PDE solutions obtained by comparative methods w.r.t. ground-truth solutions for differ- ent dimendepnndnnt PDEs when N, is set IK. We particularly chose a small value of N, to study the effect of small sample size on PINN performance (note that the original formulations of base- line methods used very high N,). We can see that while PINN- fixed fails to converge for convection (33 = 30) and Allen Cahn equations (admitting very high errors), PINN-dynamic shows significantly lower errors. However, for complex PDEs such as convection (B- 50), PINN-dynamic is still not able converge to low errors. We also see that cPINN- fixed shows high errors across all PDEs when N,- 1000. This is likely because the small size of collocation samples are insufficient for ePINNs to converge to the correct solution. As we show later, <PINN indeed is able to converge to the correct solution when N, is large. Per- forming dynamic sampling with cPINN shows some reduction in errors, but it is still not sufficient for convection (B = 50) case. All other baseline methods including Curr Reg, RAR-based methods, and 0 fail to converge on most PDEs and show worse performance than even the simple baseline of PINN-dynamic. On the other hand, our proposed approaches (Evo and Causal Evo) consistently show the lowest errors across all PDEs. Figure 4 shows that Evo and PINN-dynamic are indeed able to mitigate propagation failures for convection (3 = 50), by maintaining low values of Skewness, Kurtosis, and Max PDE residuals across all iterations, in contrast to PINN-fixed. Additional visualizations of the evolution of samples in Evo, Caual Evo, and RAR-based methods across iterations are provided in Appendix P Sensitivity of RAR-based methods hyper-parameten is provided in Appendix J.3
All code and datasets used in this paper are
175
300
30 to 22
(a) Surface Plot of Auckley Function
10
35
Reteined 1
Norm
Resempee p
1 Norm
30
Noem
1 Nurm
25
10"
10'
10'
10a
10'
Epocks
(b) Dynamics of PDE loss for Evo
Figure 5: PDE loss of P going from L to L with iterations for a sample objective function.
To demonstrate the ability of Evo to accumulate high residual points in the retained population pr (or equivalently, focus on higher-order L' norms of PDE loss), we consider optimizing fixed objective function: the Auckley function in Figure S (see Appendix for details of this function). We can see that iteration 1, the expected loss over P is equal to the L norm of PDE loss over the entire domains. As training progresses, the expected loss over pr quickly reaches bigher-order L norms, and approaches L at very large iterations. This confirms the gradual accumulation property of Evo as theoretically stated in Theorem 4.1 Addition visualizations of the dynamics of Evo for a number of test optimization functions are provided in Appendix K
Adaptive Nature of Evo PDE loss:
Sampling Efficiency: Figure 6 shows the scalability of Evo to smaller sample sizes of collocation points, N,. Though all the baselines demonstrate similar performances when N, is large (> 10K). only Evo and Causal Evo manage to maintain low errors even for very small values of N,- = 100, showing two orders of magnitude improvement in sampling efficiency. Note that the sample size N, is directly related to the compute and memory requirements of training PINNs. We also show that Evo and Causal Evo show faster convergence speed than baseline methods for both convection
8

---

Under review as a conference paper at ICLR 2023
and Allen Cahn equations (see Appendix J.2 for details). Additional results three cases of Kuramoto-Shivee Shivashinsky (KS) Equations including chaotic behavior are provided in Appendix U.7
Solving Eikonal Equations. Given the equation of surface geometry in a 2D-space, u( Vs) = 0, the Eikonal equation is iime-independent PDE used solve for the signed distance field (SDF), u(x, y). which has negative values inside the sur- face and positive values outside the surface. See Appendix for details of the Eikonal equation. The primary difficulty in
Relative Eg Ervor
Eru
<de- 2000 (load)
Craal Bio
2008 (dywwwd))
10
CPEN (Eard)
10P
10-1
solving Eikonal equation comes from determining the sign of
the field (interior or exterior) in regions with rich details. We compare the performance of different baseline methods with re- spect to the ground-truth (GT) solution obtained from numerical methods for three complex surface geometries Figure 7 We also plot the reconstructed geometry of the predicted solutions to demonstrate the real-wor cal-world application solving this PDE, e.g., in downstream real-time graphics rendering. The quality of reconstructed geometries are quantitatively evaluated using the mean Intersection-O -ae---vve--nnn (mIOU) metric. The results show that PINN-fixed shows poor performance across all three geometries, while PINN-dynamic is able to capture most of the outline of the solutions with few details missing for difficult geometries like "sailboat" and "gear". On the other hand, Evo is able to capture even the fine details of the SDF for all three com- plex geometries and thus show better reconstruction quality. We can see that mIOU of Evo is significantly higher than baselines "sailboat" and "gear". See Appendix Section J.8 for more discussion and visualizations.
10-2
10a
10 Number of Collocation Points
10a
(a) Convection (B3
50, Iter: 300k)
10
P9NN (Bood)
Reledive C. Ecvot
101
Crnal
10"
10-1
10.2
10a1
10a
10'
10'
Number of Collocation Points
(b) Allen Cahn (Iter: 200k)
Figure 6: Sample Efficiency of Evo and Causal Evo low Ny.
CONCLUSIONS AND FUTURE WORK DIRECTIONS
We present a novel perspective for identifying failure modes in PINNs named "propagation failures." and develop a novel evolutionary sampling algorithm to mitigate propagation failures. From our experiments, demonstrate better performance variety of benchmark PDEs. Future work can focus on theoretically understanding the interplay between minimizing PDE loss and sampling from high residual regions PINN performance. Other directions of future work include exploring more sophisticated evolutionary algorithms involving mutation and crossover techniques.
SDF
SDF
Brocol
SDF
Racea.
kis
a
A
A
t
a
Sho (Cues)
A
A
PIEN (erannsk)
a
a
a
A
A
Eave " 23566m1
NESTI KETIS
BEOX In TEA19!
Ease 2.0uH
PINS (bood)
Figure Solving Eikonal equation for signed distance field (SDF). The color of the heatmap represents the values of the SDE. The gray region shows the negative values of SDF that represents the interior points in the reconstructed geometry from predicted SDF.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Agoston E Eiben, James E Smith, et al. Introduction to evolutionary computing, volume 53, Springer, 2003.
Stephen Kokoska and Daniel Zwillinger. CRC standard probability and statistics tables and formulae. Crc Press, 2000.
Aditi Krishnapriyan, Amir Gholami, Shandian Zhe, Robert Kirby, and Michael W Mahoney. Char- acterizing possible failure modes in physics informed neural networks. Advances in Neural Information Processing Systems, 34, 2021.
Randall LeQeque Finite difference methods for ordinary and partial differential equations: steady- state and time-dependent problems. SIAM, 2007.
Lu Lu, Xuhui Meng. Zhiping Mao, and George Em Karniadakis. Deepxde: A deep learning library for solving differential equations. SIAM Review, 63(1):208-222 2021.
Mohammad Amin Nabian, Rini Jasmine Gladstone, and Hadi Meidani. Efficient training of physics-
informed neural networks via importance sampling. Computer-Aided Civil and Infrastructure Engineering, 36(8):962-977, 2021.
Maziar Raissi, Paris Perdikaris, and George Em Karniadakis, Physics informed deep learning (part i): Data-driven solutions of nonlinear partial differential equations. arXiv preprint arXiv:171..1056. 2017.
Maziar Raissi, Paris Perdikaris, and George E Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378:686-707, 2019.
Franz M Rohrhofer, Stefan Posch, Clemens Göönitzer, and Bernhard C Geiger. Understanding the difficulty of training physics-informed neural networks on dynamical systems. arXiv preprint arXiv:2203.13648, 2022.
Robert E Schapire. The boosting approach to machine learning: An overview. Nonlinear estimation and classification, pp. 149-171, 2003.
Dan Simon. Evolutionary optimization algorithms. John Wiley & Sons, 2013.
Chuwei Wang, Shanda Li, Di He, and Liwei Wang. Is 12 physics-informed loss always suitable for training physics-iffrrmed neural network? arXiv preprint arXiv:2206.020/66 2022a.
Sifan Wang. Yujun Teng, and Paris Perdikaris. Understanding and mitigating gradient pathologies in physics-informed neural networks. arXiv preprint 1XXi::2011.04566, 2020.
Sifan Wang. Shyam Sankaran, and Paris Perdikaris. Respecting causality all you need for training physics-informed neural networks. arXiv preprini wrxXiv2203.004000 2022b.
Sifan Wang, Xinling Yu, and Paris Perdikaris. When and why pinns fail train: A neural tangent kernel perspective. Journal of Computational Physics, 449:110768 2022c.
Jian Cheng Wong, Chinchun Ooi, Abhishek Gupta, and Yew-Soon Ong. Learning in sinusoidal spaces with physics-informed neural networks. IEEE Transactions on Artificial Intelligence, 2022.
Chenxi Wu, Min Zhu, Qinyang Tan, Yadhu Kartha, and Lu Lu. A comprehensive study of non- adaptive and residual-based adaptive sampling for physics-informed neural networks. arXiv preprint arXiv:2207.1022 2022.
Olek C Zienkiewicz, Robert L Taylor, and Jian Z Zhu. The finite element method: its basis and fundamentals. Elsevier, 2005.
10

---

Under review as a conference paper at ICLR 2023
A CONNECTIONS BETWEEN L' NORM AND SAMPLING
In this section, we provide connections between adaptively sampling collocation points from a distribution ((x,) [Ro(x,)1* Wu et al. (2022 and using LP norm of the PDE loss Wang al. (202aa).
Theorem A.1. For p Z 2, let (U) denote the expected L PDE Loss computed on collocation points sampled from uniform distribution, U(2)). Similarly, for k z O, let C (2t)) denote the expected L PDE Loss computed on collocation points sampled from an alternate distribution (K+22/2
(²()) Xx q(3x), where q(xx) [Ro(x+)1". Then, L2(2") zV/aV-17z (()+2(4)
where Z is normalization constant, defined Z -x (Ro (R((x))) (x+) and V is the volume of the domain S2
Proof. The expectation of L PDE Loss for collocation points sampled from a uniform distribution ((t))= X, ~ p(x+) can be defined as follows
C2(2)= (Rx 1/8
= (ʃ p(xx)) (xx))
1/p
Note that for uniform distribution, p(x,) = +. where V the volume of the domain Ω i.e., V= I (supp(x;;) iin((z:)) with supp(-) and inf(.) being the supremum and infimum operators, and æ; the i-th dimension x, (e.g., the space dimension æ or the time dimension /).
Now, let us consider the case where we are interested in sampling from an alternate distribution (+())= x, ~ ((x)), where q(x,) x [R,( while using the I.² PDE Loss (the most standard loss formulation used in PINNs). The sampling function of (* can be defined as follows:
((x+) = Ro (x+))*
(4)
Z
where Z is the normalizing constant, i.e., Z Jx.en RR (xx) dxxr-
Hence, the 2 PDE Loss for collocation points sampled from ($2) can be defined as:
(² (Ro 1/2
(/ ((xx)))o(xx)] Poxx)
1/2
(ʃ (Ro(x)) 1 (Ro )
1/2
(From Equation 40
Z
(|IRO( 1 dx
1/2
21/2
(ʃ p(xx)
1/2
(R((x=) (=22 [x++dxx,)
z//
1
( ((xx))R((xx))1 k-+2 dxr)
1/2
X U())
p(x,)
21/2--14 1/2
1
1/2
(Ex (x(x))) (+22)
71/2
21/21-177 (c)+2(4)) (K+2)/2
1
(From Equation with p p= k + 22
(5)
We assume that the batch size/number of collocation points used to compute the PDE Loss tends to infinity,
i.c., N, oc. This allows analyze the behavior of the continuous PDE loss function (0) as N,
11

---

Under review as a conference paper at ICLR 2023
Roralion
Rocaliio
Lnenationn
1 Gen
2M Gen
god Guu
1
22
0
2
-1
Figure 8: Schematic to describe our proposed evolutionary sampling algorithm, where collocation points are incrementally accumulated in regions with high PDE residuals (shown contour lines).
Theorem A.1 suggests that sampling collocation points from distribution (xx,) xc [Rx(x))] and L norm of the PDE loss are related to each other by a scaling term 2 172 However, since this scaling term is variable nature as 7 depends on the neural network parameters 0, optimizing (U)) is not directly equivalent to optimizing C (*)) (or power thereof).
B ANALYZING PINN-DYNAMIN A STRONG BASELINE
very simple baseline for mitigating propagation failures is to dynamically sample a random set of N, collocation points from uniform distribution every iteration, independently of previous iterations. To see how this simple sampling strategy can help in sampling points from high PDE residual regions, let us consider partitioning the entire input domain Q into two subsets: regions with high PDE residuals, Rhigh, and regions with low PDE residuals Noww If the high PDE residual regions are imbalanced because of propagation failure, we would expect that the area of Rhigh (Anigh) is significantly smaller than the area of Qtww (Atow), i.e., Anigh < Alow- Let us compute the probability of sampling at least one collocation point from Rhigh across all iterations of PINN training. If we use a fixed set of collocation points sampled from uniform distribution at every iteration, this probability will be equal to Anigh/ (Anngh + Atow), which can be very small. Hence, are likely to under-represent points from Shigh in the training process and thus get stuck in trivial solutions with high residual regions around its boundaries, especially when N, is low.
On the other hand, if we perform dynamic sampling, the probability of picking at least point from Drigh across all iterations will be equal to - (1 Anigh/ (onn//(Ann (Anigh + Alow)) where N is the number of iterations. We that across all iterations of PINN training, we would have likely sampled a point from Shnigh in at least one iteration, and used it to minimize its PDE residual. As we empirically demonstrate later in Section 5S dynamic sampling is indeed able to control the skewness of PDE residuals compared to fixed sampling, and thus act as strong baseline for mitigating propagation failures.
that when N is large, this probability approaches indicating
However, note that even if we use dynamic sampling, the contribution of points from Lrigh in the overall PDE residual loss computed at any iteration is still low. In particular, since the probability of sampling points from Rhigh at any iteration is equal to Anigh / (Anigh + Atoo), the expected PDE residual loss computed over all collocation points will be equal to
Anigh
Alow
Eegnag (o)) X
+ ERO (0)] Ahigh + Atow
Anigh + Alow
Since Anigh < Atows the gradient update of 0 at every epoch will be dominated by the low PDE residuals observed over points from Room leading to slow propagation of information from initial/boundary points to interior points.
C ANALYSIS OF EVOLUTIONARY SAMPLING
In this section, we analyze the dynamic behavior (or evolution) of the collocation points for our proposed Evolutionary Sampling (Evo) approach over the iterations. A schematic representation of Evo is also provided in Figure g
C.I ACCUMULATION PROPERTY OF Evo.
In this section, we provide the proof of Theorem 4.1 presented in the main paper.
12

---

Under review as a conference paper at ICLR 2023
- be arbitrary positive real-valued k-Lipschitz continuous function, where 9 denotes the neural network parameters. When is fixed, the function F(x) does not vary with iterations, representing a fixed objective function.
Definition 1 (Objective Function). Let Fo(x)
Let X = {x] F(x;) maxx F(x) Vi E [n]] be the set of points where the objective function JF is maximal.
Now, Iet us define an e-neighborhood around each of point x; E as (x,)) such that ||x: -x S for any arbitrarily small > O and for all x, E (x,(x,) with i E [n].
Let us also assume that the objection function FF is k-Lipschitz continuous. Then the following is true:
[F()]) - F(x;)] S ke F(x;) - F(x,) S ke
vx E Nx()) & i [ F(x;) = max F(x)
(8)
Definition (e-maximal Neighborhood). Let J max F(x) be the maximal value of the objective function F(x) (Definition 7. Then c-maximal Neighborhood No can be defined as: No00 = (xxx (x1 U... U (x)) such that any point x sampled from No would have F F(x ke V x E N and for any arbitrarily small >00. 0.
Note that since the volume of N is greater than 0, the probability of sampling any No from uniform distribution ((x) is greater than 0.
Lemma C.1 (Population Properties). For any population P generated at some iteration of Evo optimizing given ohjective function F(x) (Definition [ the following properties are always true:
1. The re- re-sampled population is always non-empty, i.e. |P*|>0
2. The size of the retained population is always less than the total population size, i.e., |Pr| < PP
3. The size of the retained population is zero, i.e., |P'| 0, if and only F(x) = c, P.
Proof. The threshold r for the Evolutionary Sampling can be computed as r = 1
[P] ExEp F(x).
The retained population is defined as: P +- F(x) > Vx E PP).
Similarly, the non-retained population can be defined as: Pr + fx F(x) ST VX EXEP)
Proof of Property 1: For any arbitrary set of real numbers, there always exists some element in the set that is less than equal to the mean. Hence, the size of the non-retained population is always non-zero as there always exists some point x E such that F(x) S т. Thus, PP" O.
Now by definition, since the re-sampled population pd replaces the non-retained population at every iteration, |P*| = [Pr]. Hence, |P*| > 0, i.e., the size of the resampled population always non-zero.
Proof of Property By definition, P + |P*| = |P| (where |P] is the total size of the population and is always constant). Since, |r > O, can say that [P]] i.e.
i.e., the size of the retained
population can never be equal to the entire population size [P]-
Proof of Property Let us consider the case where F(x) = c, Vx P (where is some constant), i... the value of the function is constant at all of the points x E P. In this case, the mean of the population P, which is equal to the threshold, 7 will be equal to e. This condition would lead the entire population to be re-sampled as all element E P would satisfy the condition to belong in the non-retained population. Note that the constant function F(x) = elements are less than or equal to the mean. Otherwise, there would always be at least one element greater than the mean, resulting in a non-zero size of the retained population.
c is the only case where all of the
Lemma C.2 (Entry Condition). If point Xmm sampled from Noe any arbitrary iteration m. then it will always enter the retained population pr, unless Exep: [F(x)) > F ke.
Proof. The condition for any arbitrary point x. to enter the retained population P% at any arbitrary iteration m is given by the following:
F(xm) Ti ExeP... [F(x)].
(By definition of the threshold T.
13

---

Under review as a conference paper at ICLR 2023
Now, if the point X is sampled from N then F(xm) Ft - ke (from Definition 2). Hence, for Xm to enter the retained population Pm we need to ensure that F ke > Tm-
Let us consider the case where Xm is not able to enter the retained population. In such case, we will have the following inequality:
F
ke Tm-
(10)
It is also easy to show from the definition of retained population that the threshold T is always less than the expectation of the retained population:
Tm S EXEP= [F())]
(11)
From Equations 10 and we get,
F KE < Tm S ExEP: [F())
Ex EXEPE [(()) > F kc
(12)
We have thus proved that Xm will not be able to enter the retained population P only if
ExeP= [F()) > F ke, which suggests that the expectation of the retained population is al-
ready close to FF,, for any arbitrarily small c O. On the other hand, (eEP,,,[(xx) S J F ke, we would necessarily add Xm
O
Lemma C.3 (Exit Condition). A point Xm sampled from No that entered the retained population at any arbitrary iteration m, can exit the retained population P at an arbitrary iteration n (such that n > m) only if Exep; [xx) 2 F
ke.
Proof. The generic condition for any arbitrary point x to exit the retained population P at iteration is given by: F(x) S Tn Exep, [F(x)] (By definition of the threshold T).
Since a point Xm that was originally sampled from N will have F(xm) 2 F ke (from Definition 2). we use this inequality in the generic exit condition shown above get,
F kc S Tn I ExEP: [x())
(13)
Hence, the point Xm. exit the retained population at iteration n only if ExEP; [F(x) ke.
Theorem C.4 (Accumulation Dynamics Theorem). Let Fa(x): RR + R a fixed real-valued
k-Lipschitz continuous objective function optimized using the Evolutionary Sampling algorithm. Then, the expectation of the retained population ExEpr [F(x)] 2 max F(x) - ke as iteration + oc, for any arbitrarily small O.
Proof. We prove this theorem by contradiction. For the sake of contradiction, let us assume that as iterations i + oc, the expectation of the retained population Exep- [F(x)] < mmax F(x) kc, for any arbitrarily small c > 0. We can then make the following two remarks.
Entry of collocation points: Note that the probability of sampling x from N is non-zero because the size of the re-sampled population is non-zero, i.e., |P*| > O (proved in Lemma C.1 Also, since we have assumed Fx (F()) < Ft ke, we can use the Entry condition proved in Lemma C.2 to arrive at the conclusion that point from Noe will always be able to enter the retained population.
Exit of collocation points: Similarly, a point x that belongs the e-maximal neighborhood and is part of the retained population pr will not be able to escape the retained population we have asssumed ExEP- [Fx)) < F - ke (using the Exit condition proved in Lemmas C.3
From the above two remarks, retained population if our initial assumption (for the sake of contradiction) true. However, since the total size of the population [P] is bounded, the size of the retained population [P' cannot grow indefinitely. We have thus arrived at a contradiction suggesting our assumption incorrect. Hence, as iterations + oc, the expectation of the retained population Excp- [F(x)] 2 max, F(x) ke, for any arbitrarily small > 0.
that points would keep accumulating indefinitely in the
14

---

Under review as a conference paper at ICLR 2023
can prove continuous accumulation of collocation points from the e-maximal neighborhood until the expectation of the retained population is close to the maximum point (i.e., reaches )x), thus exhibiting the Accumulation Property described in Section 4 Although this theorem assumes that the objective function F(x) (or in the context of PINNs, the absolute residual values, R. (x+) is constant, this theorem is still valid when Ro (xx) is gradually changing with the highest error regions (defined using our e-maximal neighborhood) persisting over iterations. Under such conditions, the theorem states that the retained population p would always accumulate points from the e-maximal neighborhood, thereby adaptively increasing their contribution to the overall PDE residual loss and eventually resulting in their minimization.
From Theorem 4.1
C.2 RELEASE OF COLLOCATION POINTS FROM HIGH PDE RESIDUAL REGIONS.
Our definition of the Sample Release Property states that the distribution of collocation points should revert back to its original form by releasing the accumulated points in the high PDE residual regions once they are "sufficiently minimized". Let us define that for an arbitrary collocation point x, E P, "suffficiet minimization" of the PDE is achieved if Rx(x,") (x,,E((R.xx.)) threshold used by Evo). Then, by definition, such points will belong to the "non-retained population" and will be immediately replaced by the re-sampled population. Since we generate the re-sampled population Ts from uniform distribution, these "sufficiently minimized" collocation points are replaced with a uniform density. Thus, Evo satisfies the "Sample Release Property" an "ideal" sampling algorithm.
Ex,EP [Ra(x,)]
(where the
D ADDITIONAL DETAILS FOR CAUSAL EVOLUTIONARY SAMPLING
Figure 9b] represents a schematic describing the causally biased Evolutionary sampling described in and the causal gate g that is updated every iteration.
Section 4.
Iteratonn
Iteration N
15
0.90
1.5
0.00
3-
0.75
0.25
a 1.90
30
0:00
Cansel
5
0.0
4.0
112
3-4
Time (t)
Time
Time
(a) Causal Gate.
(b) Causally Biased Evo Sampling
Figure 9: Causal Evo uses a time-dependent causal gate for computing PDE loss and for sampling.
D.1 PREVENTING ABRUPT CAUSAL GATE MOVEMENT.
The shift parameter of the causal gate is updated every iteration using the following scheme: 7--1 Yi + (()),, where 7, is the learning rate that controls how fast the gate should propagate and denotes tolerance that controls how low the PDE loss needs to be before the gate shifts the right, and denotes the 5th iteration. Typically, in our experiments we set the learning rate to 1e-3. Thus, for example, if the expectation of e (0)) over 1000 iterations is 0.1, then would change by a value of 0.1 after 1000 iterations (since 7/+- R 7 + flg * N * E(e-,0))))), Additionally, note that, for a typical "tanh" causal gate, the operating range of y values vary from -0.5 to 1.5. However, if the loss is very small (C) 0), the magnitude of the update (0) 1, i.e., leads to abrupt change in the causal gate. Thus, to prevent an abrupt gate movement due to large magnitude update, we employ a magnitude clipping scheme (similar gradient clipping in conventional ML) as follows: Y++1 = ny min(e min(e-<2((0 (() Amax), where Amaz is the maximum allowed magnitude of update. Typically, for our experiments we keep Amar = 0.1. Note, that Amaz needs to be carefully chosen depending on the gate learning rate 7/g-
15

---

An alternate choice of causal gate is using composition of ReLU and tanh functions: g ReLU tanh(a(t -?)) (as shown in Figure 10 We can see that by using ReLU, this alternate gate function provides stricter thresholding of gate values to after cutoff value of time. The effect of this strict thresholding on the incorporation of causality in training PINNs can be studied in future Under review as a conference paper at ICLR 2023
D.2 CHOICE OF OTHER GATE FUNCTIONS.
The gate function to enforce the principle of causality is not limited to the "tanh" gate presented in the main paper. Any arbitrary function be used for a causal gate long obeys the following criteria:
Continuous Time Property: The function g should be continuous time, such that can be evaluated at any arbitrary time I.
2. Monotonic Property: The value of gate g at time + A should be less than the value of the gate at time i.e., g(t At) S g()). In other words, g should be monotonically decreasing function,
3. Shift Property:
The gate function should be parameterized using shift parameter y, such that 9, (t) < 9++o(t), where 5 > 0, i.c., by increasing the value of the shift parameter the gate value of any arbitrary time should increase.
analyses. our current analysis, we simply used the tanh gate function for all our experiments.
1.00
0.05
0.50
Causal Gate (g)
1.5
1.25
0.00
Y- 1.50
1.0
0.75
0.6
0.50
0.0
Time (1)
Figure 10: ReLU-tanh Causal Gate
E COMPARISON OF BASELINES
In this Section, we compare the different baseline methods w.r.t. the motivating properties. See Table
2
Table 2: Table comparing baseline methods in terms of their ability comply with the motivating properties of this work.
Uniform Background Computational Property
Accumulation Property
Release Property
Causality
Efficiency
RAR-G G Lu et al.
2021
RAD Nabian et a
202
RAR-D el al. 2022
Wane al 2222a
CausalPINN Wang
2022b
Evo (ours)
Causal Evo (ours)
F COMPUTATIONAL COMPLEXITY ANALYSIS OF EVOLUTIONARY SAMPLING VS ITS BASELINES
In this section, aim to provide comprehensive comparison of the computational complexity of Evolutionary Sampling and its baselines. is well-known that the cost of computing the PDE residuals using automatic-differentiati during training amounts is reasonably large, especially
16

---

Under review as a conference paper at ICLR 2023
when the PDE contains higher order gradients that require repeated backward passes through the computational graphs used by standard deep learning packages like Py ToTchhTensorrfow. Thus, in this section would mainly focus comparing the number of PDE residual computations that each algorithm makes during training. We can quantify the effect of this difference on computational costs as follows. Let us first define the Notations that are going
for the analysis:
Notations for Computational Analysis
Computational Cost to evaluate the PDE residual on a single collocation point.
N
Total Number Training Iterations
[P]
Total Number of Collocation Points/Population Size (Also re- ferred as the initial of collocation points for RAR based methods)
Pdense
Auxiliary Set of dense points [For RAR-G.RAD, and RAR-D], where [Pdeneel >> [P]
K
Resampling Period [For RAR-G.RAD, and RAR-D]
M
Number of additional collocation points added the initial set
Next, we present the computational cost to run each method separately.
PINN/Evo: The cost of computing the PDE residual at an arbitrary epoch i: CEvo C(P]. Thus, the overall computational cost for N iterations is C NCIP
RAR-G/RAD/RARRDD For RAR- -baedd methods, the initial set of collocation points P keeps growing as new test points are added every iterations. Thus, at an iteration i, the size the collocation point set [P; = iPul + m m KJ PP + M [i/K] (to simplify notations, let [Pol [P], i.e., all of the methods start with the same number of initial collocation points).
Thus, the cost to compute the PDE residuals this training set P is:
N
Ctrain = KCIP| + KC + MM + KC(PP + 2M) + KC M
K
M
N
K + +1)
=
KCPP
k
(14)
PP
K
There is also additional cost of evaluating the PDE residuals on the dense set Ceeeaee these M points from high PDE residual region.
RAR to select
N
C'denae
RAR
Cl
K
(15)
Therefore, the overall cost for RAR-based methods CRAR
Cidense + Cirain
RAR
Comparing the cost between Evo and RAR: Assuming that the total number of epochs N is divisible by the resampling period K, which true for most practical scenarios.
MNC 2
N
N
CRAR CRAR= + KCPP
+1) + ClP denee
K
K
MNC
N
CRAR CEvO + KCIP +
+1) + Cl
2
K
K
MNC
N
CRAR CEvO = KCIP|
+
+ CIP dense
(16)
2
K
K
Thus, we can see that the difference the computational cost can quickly grow depending on the choice of RAR setting. Also note that, since |Pannse >> IP), the additional cost of Cl K is significant, especially if we want to re-sample/re-evaa the adaptive sampling frequently (i.c., for small values of K).
17

---

Under review as a conference paper at ICLR 2023
G DETAILS OF PARTIAL DIFFERENTIAL EQUATIONS USED IN THIS WORK
G.1 CONVECTION EQUATION
We considered a ID-oovvctionn equation that is commonly used to model transport phenomenon, described as follows:
au 0u
0, E [0,
[0,
(17)
Öt
0r
u(z, 0) h(z)
(18)
w(0, t) = u( 2, t)
(19)
where 8 is the convection coefficient and h(x) is the initial condition. For our case studies, we used constant setting of (xx)) sin(x) with periodic boundary conditions in all experiments, while varying the value of 3 in different case studies.
G.2 ALLEN-CAHI EQUATION
We considered ID Allen Cahn equation that is used to describe the process of phase- separation in multi-component alloy systems as follows:
0u
I2u
0.0001
+ 5u
-5u = E [-1, 1],4 4 E [0.
(20)
at Or.
u(x, 0) x² cos (TT)
(21)
u(t, -1) = ult, 1)
(22)
du du
(23)
Øt 2=-1 Øt 1=11
G.3 EIKONAL EQUATION
We formulate the Eiknonal equation for signed distance function (SDF) calculation as:
V u| u(x,) 0,
x, [-1,1]
(24)
F, S
(25)
u(z, 1), w(x (r, 1), (1, u(1, >
(26)
where zero contour set the SDF. In training PINN, we use the zero contour constraint as initial condition loss and positive boundary constraint as boundary loss (see Table for details of loss balancing).
G.4 TURAMOTO-SIVASHINN EQUATION
We use 1-D Kuramoto-Sivashinsky equation from CausalPINN Wang et al. (2022b):
Ou du 8.u Øt
0Lu
auu
Ox
Ox
Dz
0,
(27)
subject to periodic boundary conditions and an initial condition
u(0, ..) uo
(28)
The parameter a, controls the dynamical behavior of the equation. We the same configu- rations as the CausalPINN: a 5, 3 0.3. = 0.005 for regular settings, and a = 100/16, 8 = 100/162, 7 100/164 164 for chaotic behaviors.
18

---

Under review as a conference paper at ICLR 2023
H DETAILS ON SKEWNESS AND KURTOSIS METRICS
Skewness and kurtosis are two basic metrics used statistics to characterize the properties of distribution of values {Y.),= A high value of Skewness indicates lack of symmetry in the distribution, i.e., the distribution of values to the left and to the right of the center point of the distribution are not identical. On the other hand, a high value of Kurtosis indicates the presence of a heavy-tail, i.e., there are more values farther away from the center of the distribution relative to Normal distribution. In our implementation using scipy, we used the adjusted coefficient of skewness and Fisher's s definition of kurtosis, as defined below.
Skewness: For univariate data Y1, Y,,,., YN, the formula of skewness is
skewness VN(N-1) - 1), EE (Y, P/ N 2
(29)
33
where Y the sample mean of the distribution and is the standard deviation. For any symmetric distribution (e.g.. Normal distribution), the skewness is equal to zero. A positive value of skewness means there are more points to the right of the center point of the distribution than there are to the left. Similarly, a negative value of skewness means there more points to the left of the center point than there are to the right. our use-case, a large positive value of skewness of the PDE residuals indicates that there some asymmetrically high PDE residual values to the right.
Kurtosis: Kurtosis is the fourth central moment divided by the square of the variance after subtracting 3, defined as follows:
(Y. -Y- 8
kurtosis
(30)
For Normal distribution, Kurtosis is equal to O. A positive value of Kurtosis indicates that there are more values in the tails of the distribution than what is expected from a Normal distribution. On the other hand, negative value of Kurtosis indicates that there are lesser values the tails of the distribution relative to a Normal distribution. In our use-case, large positive value of Kurtosis of the PDE residuals indicates that there are some high PDE residual values occurring in very narrow regions of the spatio-temporal domain, that are being picked up as the heavy-tails of the distribution.
I HYPER-PARAMETER SETTINGS AND IMPLEMENTATION DETAILS
The Hlper-paramenn settings for the different baseline methods for every benchmark PDE provided in Table 3 Note that we used the same network architecture and other hyper-parameter settings across all baseline method implementations for the same PDE. In this table, the column on 'r/ic/be' represents the setting of the Xr, Aic. Abe hyper-parameters that are used to weight the different loss terms in the overall learning objective of PINNs. Table also lists the type of Optimizer, learning rate (Ir), and learning rate scheduler (Ir.scheduler) used across all baselines for every PDE. For the Eikonal equation, we used the same modified lllti-layer perceptron (MLP) architecture as the one proposed in (Wang et al. 2020). Additionally, for the Causal Evolutionary Sampling method, we used the following rrper-parameter settings across all PDEs: a 5, learning rate of the gate "g = le 3, tolerance 20, initial value of B - 01.5, and Amar 0.1. The number of iterations (and the corresponding PDE coefficients for the Convection Equation) are provided in Section of the main paper.
Hardware Implementation Details: We trained each of our models on one Nvidia Titan RTX 24GB
GPU.
J ADDITIONAL DISCUSSION OF RESULTS
J.1 VISUALIZING PROPAGATION FAILURE FOR DIFFERENT SETTINGS OF
In Figure of of the main paper, we demonstrated the phenomenon of propagation failure for convection equation with = 50, which was characterized by large values of Skewness and Kurtosis in the
19

---

Under review as a conference paper at ICLR 2023
Table syyper-paameee settings for different baseline methods for every benchmark PDE
Periodic Encoding
Optimizer/
lr.
RAR Hyperparams
PDE
Method
Architecture
r/ie/bc
Ir
scheduler
k/m/N/jS
PINN,
StepLR
cPINN,
50 x
Adam/
No
1/100/100
rate=0.9
N/A
Convection
Evo,
(MLP)
le-3
steps=5000
Evo
50 4
Adam/
Curr. Reg.
1/1/1
No
N/A
(MLP)
le-4
RAR-G
StepLR
~11/100/100000
50
Adam/
RAD
1/100/100
rate=0.9
1/1/100/100000
(MLP)
le-3
RAR-D
steps=5000
1/1/100/100000
PINN,
StepLR
<PINN.
128x4
Adam/
Allen Cahn
1/100/100
rate=0.9
N/A
Evo,
(MLP)
le-3
steps=5000
CausalEvo
RAR-G
StepLR
~/1/100/100000
128x4
Adam/
RAD
1/100/100
rate=0.9
1/1/100/100000
(MLP)
1c-3
RAR-D
steps=5000
1/1/100/100000
128x4
StepI.R
PINN,
Adam/
Eikonal
Wang al.
2020
No
1/500/10
rate=0.9
N/A
Evo
le-3
(modified MLP)
steps=5000
PDE residual fields for a large number iterations (or epochs), and simultaneous stagnation in the relative error values even though the mean PDE residual kept on decreasing. Here, in Figure
11 we show that the same phenomenon can be observed for other large values of 10, namely, 30, 50, 70. We can that the relative errors for all these three cases remains high even though the PDE residual loss keeps on decreasing with iterations. We can also see that the absolute values of skewness and kurtosis increase as increase 3, indicating higher risks of propagation failure. In fact, for 3 = 30, we can even see that the epoch that marks an abrupt increase skewness and kurtosis (around 50K iterations) also shows sudden increase in the relative error the same epoch, highlighting the connection between imbalanced PDE residuals and the phenomenon of propagation failure.
TI
30
5
Reiatoe Roor
Kastosi
100
-00
Figure 11: Demonstration of Propagation Failure for Different Settings of B (3 = 10, 30, 50, 70)
J.2 CONVERGENCE SPEED OF EVOLUTIONARY SAMPLING
Figure 12 shows that Causal Evo is able to converge faster to low error solutions than all other baseline methods for both convection and Allen Cahn equations. This shows the importance of
20

---

Under review as a conference paper at ICLR 2023
Relatvve Cg Brocrr
10P
7033 (Bod)
Relative Eg Brvoe
FDS (taed)
10"
10
Relatioo Error
10
Gouad Evs
Rvi
2000 (Ber))
<PINN (Rand)
XAD
10P
10a
10P
10
10-1
10
10-2
0.0
0.5
1.0
1.5
2.0
2.5
3.0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
10-
war
0.0
0.2
0.4
0.6
0.8
1.0
(a) Convection
(3
50,
N,
(b) Convection (3
50,
Iteration
632
1k)
N, = 20k)
(c) Allen Cahn (N, 10k)
Figure 12: Comparing convergence speeds of baselines Evo and Causal Evo.
respecting causality along with focusing on high residual regions to ensure fast propagation of correct solution from initial/boundary points interior points. While cPINN- fixed and PINN-dynamic do not converge for convection (N, 1K), can see that they both converge to lower errors compared to PINN-fixed for Allen Cahn and for convection when N, is large (20K).
J.3 SENSITIVITY OF RAR-BASED METHOSS
Figures 13 shows the sensitivity of two RAR-based methods: RAR-G and RAR-D respectively different values of the resampling period K and the number of collocation points m added from the dense set Peense- Note that although the size of the initial set of collocation points [P] was same for each of these experiments, the final size of the collocation points vary depending on the choice of K (the final size of the collocation points increases collocation points increases with m). We essentially observe that adding more collocation points almost always improves the performance these RAR -bsedd methods.
decreases) and m (the final size of the
14
Resampling Purind (N)
100
0:70
1.53
0.89
0.56
1:2
Besssmy lng, Pertod (N)
100
2:15
1.51
138
130
300
9.01
5:38
1.00
0.:88
006
1000
55.74
6:32
2.44
1000
-84
0u4
15
10000
162.70
108.55
00.21
$0.80
10000
20 Number Addedd Points (M)
5
10
Number of Added Poinis (M)
(a) RAR-G
(b) RAR-D
Figure 13: Sensitivity of based Methods different values of K and M (for fixed |P Pdennse 100k on Convection Equation with B = 30.) The numbers in the heatmap denote the % Relative L2 Error.
J.4 VISUALIZING THE EVOLUTION OF COLLOCATION POINTS IN EVO
Figure 14 shows the evolution of collocation points and PDE residual maps of Evo progress in training iterations for the convection equation with 3 = 50. We can see that the retained population of Evo at every iteration (shown in red) selectively focuses high PDE residual regions, while the re-sampled population (shown in blue) are generated from uniform distribution. By increasing the contribution of high residual regions in the computation the PDE loss, we can see that Evo able to reduce the PDE loss over iterations without admitting high imbalance, thus mitigating the propagation failure mode, in contrast to conventional PINNs.
J.5 VISUALIZING THE EVOLUTION OF COLLOCATION POINTS IN CAUSAL EVO
Figure 15 shows the evolution of collocation points and PDE residuals of Causal Evo, along with the dynamics of the Causal Gate function. We can see that the retained population at every iteration (shown in red) strictly adheres to the principle of causality such that the collocation points are sampled
21

---

Under review as a conference paper at ICLR 2023
Rosmmpie pluto
Retainal points
Eeeration 0
Ireration 10000
Beration 20000
Tresation 40000
Tecration 100000
Collestion Points
PDR Roschalk
6.73
PINN Predlction
10.00
1.5
40.5
-05
-0.04
600
14
-40
0.30
8:35
01000
0.75
3..0
0.55
0.75
0..50
1.00
Mbalure Exvor
0..5
0000
005
1.55
0068
0.70
1.00
0.03
0.55
0.30
0.55
0000
0.55
6.3
0.75
1.00
3.99
6:55
0.00
0.25
1.03
6000
0.55
0.99
6:55
1.00
Figure 14: Demonstrating the propagation of information from the initial/boundary points to the interior points for Evo.Sample on Convection Equation( 50)
from later times only when the PDE residuals at earlier times have been minimized. This is also reflected in the movement of the causal gate function where the gate values are close to for only small portions of time domain at intermediate epochs. At 90K iterations, we can see that the causal gate values are close for all values of time, indicating that the entire time domain is now revealed for training PINN to converge to the correct solution.
J.6 VISUALIZING THE DYNAMICS OF COLLOCATION POINTS FOR RAR- -ASSD METHODS
Figures 16 17 annd shows the evolution of collocation points and PDE residuals of RAR-G G, RAD, and RAR-D, respectively. We can see that all three RAR-based methods are failing to converge to the correct solution even after 100K iterations, demonstrating their inabillity to mitigate propagation failures.
J.7 KURAMOTO-SHP KURAMOTO-SIVASHINKY (KS) EQUATIONS
We used three additional experiments on the Luramoto-Sivashinsky (KS) Equations (one for a regular relatively simple case and the remaining two exhibiting chaotic behavior). Please note that these equations are particularly more complex, especially the chaotic cases where small change in the state of the solution can result in very large errors downstream in time. Thus, for chaotic domains, the successful propagation of solution from the initial and boundary conditions is critical to guarantee convergence. We would also like highlight that the computational cost of these experiments are significantly higher. We used the exact same typper-prrmmttr settings as those provided in CPINN except the sample size, which was varied from 128 to 2048 in the KS-regular case, and the number of training iterations, which was kept 300k in proposed approaches while CPINN was allowed to use about M maximum number of iterations with early stopping. Our method on average takes 50-60% less time than CPINN because of the significantly smaller number of iterations.
Figure 19 compares the performance of CPINN, Evo, and Causal Evo on the KS equation (regular case) a function of the number of collocation points used in PINN training. We can that both Evo and Causal Evo show improvements over CPINN when the number collocation points is small (N, 128). As the number of collocation points is increased, Causal Evo shows better performance
22

---

Under review as a conference paper at ICLR 2023
Itemation
Iterations
Iteaaiinn
Eseration 40030
Iseration
Concel Cate
7.7
6.5
035
2.3
0.3
0S0
2.0
8.00
8.23
0.00
0.00
6.5
1.3
0.0
8.0
1.3
6.0
100
0.0
055
10
Collocation Poiate
PDE Doskhaale
6.03
0.55
0/75
1.00
100
PINN Prodiccon
-8.
6.23
0:25
0028
1.00
0.55
0.75
1.00
0.25
0..00
0.75
1.00
0000
0425
0.500
0.75
1,00
1000
01:25
01.00
0.755
Absouate Erose
3.00
Figure 15: Demonstrating the propagation of information from the initial/boundary points to the interior points for Causal Evo. on Convection Equation( = 50)
than Evo, as it incorporates additional prior of causality along with satisfying the four motivating properties of Evo. Overall, Causal Evo mostly performs better than both CPINN and Evo across different training set sizes. Note that these curves have been obtained using single run of every method due to the computational cost of training for the KS equations, and having multiple runs for every method will help to quantify the variance these results.
Table 4: Relative L2 errors (in %) of CausalPINN, Evo and CausalEvo over the three different KS-Equation Benchmarks Wang et al. (202bb)
Evo
CausalEvo (ours)
CausalPINN
(ours)
Regular
2.120%
3.740%
0.761%
Chaotic
3,272%
6.924%
7.630%
Chaotic Extended
52.66%
29.26%
33.50%
Table compares the performance CPINN, Evo, and Causal Evo on the three KS-Equations cases as was used in the original CPINN paper. Please note that for these experiments, we used the exact same hyper-pane prperpparameter settings as the original CPINN, thus a large number collocation points were used (2048 for the regular case and 8192 for the two chaotic regimes for each time-window). We can observe that on the regular case, Evo performs similarly to CPINN, while Causal Evo is significantly better than both. However, in the first chaotic case, CPINN is slightly better than both Evo and Causal Evo. Finally, in the much more chaotic regime for the KS ESSuatatio (extended case), we find that all of the methods struggle to obtain high fidelity solution of the field. However, Evo and Causal Evo are somewhat better than CPINNs. Hence, we can comment that Evo and CausalEvo have comparable performance CPINN on their benchmark settings. Additional visualizations of
23

---

Under review as a conference paper at ICLR 2023
Rosmmpie plito
Retainal points
Eeeration 0
Ireration 10000
Beration 20000
Treration 40000
Tecration 100000
Collectioo Points
PDR Roschalk
105
PINN Prediction
1.5
-035
-60
-004
-0.34
-10
6.43
0.30
8:35
0.00
0.75
3.00
1.00
Mbalure Exvoe
100
8.55
405
0.55
655
8.55
006
0.70
1.00
0.03
0.55
0.30
0.55
000
0.55
633
0.75
1.00
8.00
6.55
0.00
1.0
000
0.55
0.50
1.00
Figure 16: Demonstrating the dynamic changes in the collocation points for RAR-G on Convection Equation(3 50)
Iteration 0
Ireration 10000
Beeraton 20000
Trevation 40000
Eeraaion 100000
Oelbestion Points
PDE Reschunls
100
10
8.00
105
0.56
1.75
8.03
10:01
8.55
100
0.25
1.77
0.3
0000
0.75
3.03
01.00
1.00
PINN Prodiction
-0.12
-0.44
8.35
od6
0.75
1.00
0.03
0420
0.500
0.75
1.00
1000
0.25
8.53
0775
1.00
0..00
0.55
0000
0.755
3.53
CL00
01.55
0.59
6:15
1.00
3.00
Alooltte Farce
10755
675
0006
0.43
600
0175
655
1.0
0155
1.0
Figure 17: Demonstrating the dynamic changes in the collocation points for RAD Convection E = 50)
the solutions of comparative methods on the three KS equations cases are provided Figures 20 and 22
24

---

Under review as a conference paper at ICLR 2023
Rosmmpie plito
Retainal points
Berraion 0
Iteration 10000
Beration 20000
Treration 40000
Tecration 100000
Collectioo Points
PDR Roschalk
105 0
PINN Prediction
1.5
10.5
-035
-60
-0.34
6.0
-10
6.43
0.30
8:35
0100
0.75
0.00
0.75
1.00
Mbalure Exvoe
633
630
835
005
0.35
625
1.55
006
0.70
1.00
0.03
0.55
0.30
0.55
000
0.55
633
0.75
1.00
8.00
6.55
0.00
0.25
1.0
000
0.5
0.50
6.15
1.00
Figure 18: Demonstrating the dynamic changes in the collocation points for RAR-D on Convection Equation(3 50)
10
Relative Co Error
Evo
cPINN (finn)
10
10-2
10a
10
Number of Collocation Points
Figure 19: Comparison of Evo, CausalEvo and CausalPINNs on Simple KS Equation with varying
number of collocation points.
J.8 ADDITIONAL DISCUSSION AND VISUALIZATION FOR THE EIKONAL EQUATION
We chose to solve 2D Eikonal Equations for complex arbitrary surface geometries as they represent particularly hard PDE problems that susceptible to PINN failure modes. In these problems, are given the zero contours of the equation on the boundaries (representing the outline of the 2D object), which can take arbitrary shapes. The goal correctly propagate the boundary conditions to obtain the unique target solution where the interior is negative and the exterior is positive. Here, any small error in propagation from the boundaries can lead to cascading errors such that a large segment of the predicted field can have opposite signs compared to the ground-truth, even though their PDE residuals are close to Since Evo explicitly designed to break propagation barriers and thus enable easy transmission of the solution from the boundary to the interior/exterior points, can see that shows significantly better performance. On the other hand, PINN (fixed) and PINN (dynamic) struggle to converge to the correct solution especially for complex geometries (e.g., the "gear") because of the inherent challenge in sampling an adequate number of points from arbitrary shaped object boundaries exhibiting highly imbalanced residuals.
25

---

Under review as a conference paper at ICLR 2023
Predicted
Absclute
100
Cansel PINN
625
125
0.21
6475
4.90
134
-1.00
-108
-1.0
Exart
Predicted
Abexlute Erroc
1.0
1.00
0.0
6.0
0.00
600
13
BAD
620
-030
-37
0:00
104
010
-1.0
-108
-180
Exari
Predicted
Abexlute Erroc
0.0
134
Causal Evo
105
.090
0000
134
690
-100
-108
Figure 20: Visualization of the Evo, Causal Evo and CausalPINN on KS-Equation (Regular Case).
In Figure we show the evolution the solutions of comparative methods for the 'gear' case over iterations. We that Evo is able to resolve the high residual regions better than the baselines, and thus encounter less incorrect "sign flips" compared to the ground-truth, even in the early iterations of PINN training.
K OPTIMIZATION CHARACTERISTICS OF EVOLUTIONARY SAMPLING ON TEST OPTIMIZATION FUNCTIONS
In this section, we demonstrate the ability of our proposed Evolutionary Algorithm to find global minimas on various test optimization functions. We will also provide other characterizations of our proposed Evolutionary Sampling algorithm.
K.1 AUCKLEY FUNCTION
The wwo-dimensional form of the Auckley function has multiple local maximas in the near-fih flat region of the function and one large peak the center.
1
1
f(x) oexp
b
2
exp
Σ cos (czz)
+ + exp(1)
(31)
d
26

---

Under review as a conference paper at ICLR 2023
Causal PINN
Boat
Prodittd
Aboolete Extor
Evo
12
Esack
Prodictd
Aleckule Eitur
Caussa Exo
Figure 21: Visualization of the Evo, Causal Evo and CausalPINN on KS- KS-Equation (Chaotic Case).
K.2 BOHACHEVSKY FUNCTION
The two-dimensional form of the Bohachevsky function is a bowl shaped function having global maxima.
fxx,y) -22 2y² + 0.3 co) 0.4cos(4my) 0.7
(32)
K.3 DROP-WAVE FUNCTION
The two-dimensional form of the Drop-Worr Wave function which is multimodal and highly complex.
f(x, 1 coscc12 12 0.5(2² + +2)
(33)
K.4 EGG-HOLDER FUNCTION
The vro-dimensional form of the Egg-Holder function is highly complex function that is difficult to optimize because of the presence of multiple local maximas.
27

---

Under review as a conference paper at ICLR 2023
Bxart
Abwvlcte
Causal PINN
X
Boaet
Predicted
Aboolete Extor
Evo
24
1.0
Beack
Prodicted
Alocklle Eltur
Causal Exo
Figure 22: Visualization of the Evo, Causal Evo and CausalPINN on KS-Equation (Extended Chaotic Case).
((,,v) (y 47) sin( 477) + xxsnns((vvv (y 47)
(34)
K.5 WELDEE-TABBLE FUNCTION
The two-dimensional form the Holder- Table function has many local maximas, but has global maximas at the four corners.
f(x, y) = =)) exp |1- / x2 12
(35)
π
K.6
BUKIN FUNCTION
The two-dimensional form of the Bukin function has many local maximas, all of which lie ridge.
f(x. v) -100 0.01I 0.01 + 10
(36)
28

---

Under review as a conference paper at ICLR 2023
Epoch 5000
Epuch 10000
Epuch 20000
Ejoch 50030
Epoch 150000
da
08
08
0.5
-22
a2
-00
-00
.00
0.0
0.0
088
us
-06
Dan (Chos)
-0.
-00
0.0
0..
as
PTYN (dymemie))
06
000
.00
100
0.0
0.0
188
PINN (hood)
ILL
0L
-0.0
-00
Figure 23: training.
The predicted solutions of Eikonal equation (Figure at different iterations during
20.0
17.5
15
15.0
12.5
10.0
10
7.5
5.0
2.5
0.0
10203r140
40
30
20
Y
X
20
30
40
Figure 24: Surface Plot of the 2-D Auckley Function
29

---

Under review as a conference paper at ICLR 2023
Kyodo
Frech
Epedu 20
Fpoch 101
Epedl 1000
Kgnds 100)
Figure 25: Demonstrating the evolution of the randomly initialized points while optimizing the Auckley function. The red triangles represent the re-sampled population that epoch, and the blue dots represent the retained population at that epoch. The contour function of the objective function is shown in the background.
40
35
Retained p
I Norm
Resampled p
L Norm Norm
30
_ Norm
25
20
15
10
5
0
100
10
10a
10a
10
Epochs
Figure 26: Illustrating the dynamic behavior of the Evolutionary Sampling algorithm on Auckley Function using the 1 Physics-informed Loss computed on the retained and re-sampled populations. The horizontal lines represent the L Physics-informe Loss on dense set of uniformly sampled collocation points (where p 2, 4, 6, oo).
1.0
0.8
0.6
0.4
0.2
0.0
109
10
102 Epochs
103
10
Figure 27: Demonstrating the dynamic evolution of the retained population size over epochs on the Auckley Function.
30

---

Under review as a conference paper at ICLR 2023
4000
4000
3000
2000
2000
Rold 1020000
-00-00
20
Y
X
10
20
30
Figure 28: Surface Plot of the 2-D Bohachevsky Function
Epoilo
Upack
Kpod
Rpode 10
Epoci::
Booch: 5000
Figure 29: Demonstrating the evolution of the randomly initialized points while optimizing the Bohachevsky function. The red triangles represent the re-sampled population at that epoch, and the blue dots represent the retained population at that epoch. The contour function of the objective function is shown in the background.
7000
Retained Population
1 Norm
6000
Ressmpled Pogulation
1 Norm
Norm
5000
4000
3000
2000
1000
100
10
10a
10a
10a
Epochs
Figure 30: Illustrating the dynamic behavior of the Evolutionary Sampling algorithm on Bohachevsky informed
Function using the _ Physics-informed Loss computed the retained and re-sampled populations. The horizontal lines represent the LP Physics-informi Loss on dense set of uniformly sampled collocation points (where p 2, 4, 6, co).
31

---

Under review as a conference paper at ICLR 2023
1.0
0.8
0.6
0.4
0.2
0.0
10
10'
10a Epochs
10a
104
Figure 31: Demonstrating the dynamic evolution of the retained population size over epochs on the Bohachevsky Function.
1.0
0.8
0.75
0.6
0.4
0.2
0.25
0.0
2855 d.3.0
-2. 1.5 1.0 0.50.0
8
Y
X
0.5
1.0 1.5 2.0
2.
Figure 32: Surface Plot of the 2-D Drop- Wave Function
Bood
Figure 33: Demonstrating the evolution of the randomly initialized points while optimizing the Drop-Wav Wave function. The red triangles represent the re-sampled population at that epoch, and the blue dots represent the retained population at that epoch. The contour function of the objective function shown in the background.
32

---

Under review as a conference paper at ICLR 2023
1.4
Retained Population
Norm
Resampled Population
Norm
1.2
1.0
(2
0.8
0.6
0.4
0.2
0.0
10
10
102
10a
10a
Epochs
Figure 34: Illustrating the dynamic behavior of the Evolutionary Sampling algorithm on Drop-Wave Function using the L Physics-informed Loss computed on the retained and re-sampled populations. The horizontal lines represent the 1/" Physics-informee Loss on dense set of uniformly sampled collocation points (where p = 2, 4, 6, 20).
1.0
0.8
0.6
0.4
0.2
0.0
10
10a
102 Epochs
10
10a
Figure 35: Demonstrating the dynamic evolution of the retained population size over epochs on the Drop- Wave Function.
2000
2000
1750
1500
1250
1500
1000
750
1000
500
500
600
200
600
400
200
200
Y
200
400
X
600
600
Figure 36: Surface Plot of the 2-D Egg-Holder Function
33

---

Under review as a conference paper at ICLR 2023
Epocks
Spodis
Epods
Boock 10
Epedle
Epecl: .00
Epoch: 1900
Bpock: 2000
00
Figure 37: Demonstrating the evolution of the randomly initialized points while optimizing the Egg-Holder function. The red triangles represent the re-sampled population at that epoch, and the blue dots represent the retained population at that epoch. The contour function of the objective function is shown in the background.
3000
Retained Population
E Norm
Resampled Population
Norm
2500
Norm
1 Norm
2000
1500
1000
10
10a
102
10a
10a
Epochs
Figure 38: Illustrating the dynamic behavior of the Evolutionary Sampling algorithm on Egg-Holder Loss computed on the retained and re-sampled populations.
Function using the I Physics-informen The horizontal lines represent the L" Physics-ifformed Loss dense set of uniformly sampled collocation points (where = 2, 4, 6, oo).
1.0
0.8
0.6
P
0.4
0.2
0.0
100
101
102 Epochs
103
104
Figure 39: Demonstrating the dynamic evolution of the retained population size over epochs on the Egg-Holder Function.
34

---

Under review as a conference paper at ICLR 2023
17.5
15.0
12.5
10.0
7.5
5.0
2.5
5.000 26
10..0.5..000..5 2.5
000
Y
X
7.5 10.0
10.
Figure 40: Surface Plot of the 2-D Holder- Table Function
Eporic
Frocht
Epock
Bench
Rpodo
Epoda 100
Egndn 1000
Epocls 3000
Figure 41: Demonstrating the evolution of the randomly initialized points while optimizing the Holder-Table function. The red triangles represent the re-sampled population at that epoch, and the blue dots represent the retained population at that epoch. The contour function of the objective function is shown in the background.
30
Population
1.
25
Resampled Population
7.5 Norm
L. Norm
Z Norm
20
15
10
5
0
109
10
102
103
104
Epochs
Figure 42: Illustrating the dynamic behavior of the Evolutionary Sampling algorithm Holder-Table Loss computed on the retained and re-sampled populations.
Function using the L Physics-informe The horizontal lines represent the L Physics-ifformed Loss dense of uniformly sampled collocation points (where = 2, 4, 6, oo).
35

---

Under review as a conference paper at ICLR 2023
1.0
0.8
0.6
0.4
0.2
0.0
10
10a
10a Epochs
10a
10a
Figure 43: Demonstrating the dynamic evolution of the retained population size over epochs the Holder-Tabe Function.
600
F(x. y7
500
400
300
200
100
Pold d 12200000
40 30
X
10
20
30
40
Figure 44: Surface Plot of the 2-D Bukin Function
Trode
Oporke
Poodh
Foach
Epodl
Figure 45: Demonstrating the evolution the randomly initialized points while optimizing the Bukin function. The red triangles represent the re-sampled population that epoch, and the blue dots represent the retained population at that epoch. The contour function of the objective function is shown in the background.
36

---

Under review as a conference paper at ICLR 2023
1000
Retainer Popolation
1. Narm
Resampled Pogulation
Norm
Norm
1 Norm
800
(0)
600
400
200
109
10
10a
10a
10a
Epochs
Figure 46: Illustrating the dynamic behavior of the Evolutionary Sampling algorithm Bukin Function using the 2 Physics-informed Loss computed on the retained and re-sampled populations. The horizontal lines represent the L Physics-informet Loss on dense of uniformly sampled collocation points (where = 2, 4, 6, co).
1.0
0.8
0.6
0.4
0.2
0.0
109
101
102
103
104
Epochs
Figure 47: Demonstrating the dynamic evolution of the retained population size over epochs on the Bukin Function.
37

---

Under review as a conference paper at ICLR 2023
3.0
3
2.5
1.5
1.0
0.5
10 2
X
Figure 48: Surface Plot of the 2-D Michalewicz Function
Epedia
Epock:
Epocks
Riech 20
Porch 100
Poech 1000
Fposh 2000
Figure 49: Demonstrating the evolution of the randomly initialized points while optimizing the Michalewicz function. The red triangles represent the re-sampled population at that epoch, and the blue dots represent the retained population at that epoch. The contour function of the objective function is shown in the background.
K.7 MICHALEWICZ FUNCTION
The two-dimensional form of the Michalewicz function has multiple ridges and valleys which are very steep.
f(x) = Σ sin(x;; sin 2m
ir
(37)
38

---

Under review as a conference paper at ICLR 2023
4.0
Retained Population
Resempled Population
Norm
3.5
L Norm
1 Norm
2(0)
3.0
2.5
2.0
1.5
10
10
102
103
104
Epochs
Figure 50: Illustrating the dynamic behavior the Evolutionary Sampling algorithm on Michalewicz Loss computed the retained and re-sampled populations. The horizontal lines represent the L" Physics-inds -insrcss- Loss on dense set uniformly sampled collocation points (where p 2, 4, 6, 20).
Function using the L Physics-infonee
1.0
0.8
0.6
0.4
0.2
0.0
109
104
102
103
104
Epochs
Figure 51: Demonstrating the dynamic evolution of the retained population size over epochs on the Michalewicz Function.
39


Reviews:
[{"review_id": "wwB6e-9Ufg", "reviewer": "Reviewer_3fsB", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes to use evolutionary sampling to mitigate failures in training PINNs. The main idea is based on the premise that solution of a PINN propagates from boundaries towards the interior regions. In contrast to existing work, Evolutionary sampling is able to adapt to the changing PDE residual error landscape while being computationally efficient. Experiments are conducted on multiple challenging PDEs, and the results demonstrate the effectiveness of the proposed approach.", "summary_of_the_review": "I am positive about the paper, except for the probable impracticality of real-world adoption. The paper is very well written, and the experiments and analysis are thorough. Most importantly, the method seems to be effective across a range of PDEs.\n\nPost Rebuttal Update:\nI read the other reviews and the author's responses. I will maintain the initial rating for the paper. \n", "strength_and_weaknesses": "Strengths:\n1. Well written, proposed method is sound for the most part.\n2. Experimental validation is conducted on multiple problems that are challenging for standard PINN methods and the proposed method is effective in practice.\n\nWeaknesses:\n1. I am concerned about the practical viability of the idea of adaptive sampling. In real-world scenarios how realistic is to expect the PDE solution to be available at arbitrary space-time coordinates on demand? I imagine these are obtained through sensor measurements, which may be too constrained to obtain at arbitrary points or too expensive to obtain.\n2. While the proposed method is based on a population of samples, there are no mutation or crossover (interaction) operators as you would expect in a typical evolutionary algorithm.\n\nOther Comments:\n- Not a weakness per se, but all the references to the appendix in the main paper were missing. This should be an easy fix.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "lC_nJJ0FP7", "reviewer": "Reviewer_9MDS", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes \"the propagation hypothesis\" as an explanation of why some (collocation) sampling schemes might cause PINNs to fail.\nTo mitigate the issue, the authors propose an _evolutionary_ sampling strategy, that keeps points with high error in a pool and resamples others. They also incorporate a time-weighting scheme to propagate the importance of samples from $t=0$ onwards. The authors show that their method outperforms prior art in 3 datasets when the number of samples used is small.", "summary_of_the_review": "Given that the novelty is somehow limited and my concerns with the implications of the theory & the short experimental campaign, I recommend rejection.", "strength_and_weaknesses": "## Strengths\n* Authors formulate an intuitive concept to identify failure modes of PINNS;\n* Authors propose a simple strategy that improves significantly the sample efficiency of PINNS;\n* Ideas are clearly presented and the work is easy to follow.\n\n## Weaknesses\n* The idea of focusing on regions of high error is not exactly novel in PINNs;\n* While the \"propagation hypothesis\" is intuitive, there is no formal characterization for it in the work;\n* The experimental campaign is shorter than I would expect from a paper with a  very simple idea and no theoretical guarantees;\n* The theory is inconsistent (and perhaps wrong) --- see details below.\n* The experiments lack a more thorough discussion. For instance, do you have an intuition on why Evo is capable of solving the Eikonal equations in some cases (Figure 8) and other methods not? \n\n###More about the theory\n\nFor **theorem 3.1**, there are two issues: \n\n1. The step from lines 5 to 6 on page 11 of the appendix multiplies by $V^{-1/2}$ but should multiply by $V^{1/2}$ since $p(\\mathbf{x}_r)^{-1} = V$;\n\n 2. The paragraph before theorem 3.1 is misleading. It says that sampling proportional to a power of $\\mathcal{R}$ is equivalent to changing the $L^p$ loss in the norm. However, minimizing $\\mathcal{L}^2_r(\\mathcal{Q}^k)$ is not the same as minimizing $\\mathcal{L}^2_r(\\mathcal{Q}^k)$  (or a power thereof) os not the same since $Z= \\int_{\\mathbf{x}_r \\in \\Omega} |\\mathcal{R}(\\mathbf{x}_r) |^k d\\, \\mathbf{x}_r$ depends on network parameters (and therefore is not a constant). \n\nFor **theorem 4.1**, states that, as the number of iterations of Algorithm one approach $\\infty$, their loss reverts to an $L^\\infty$ loss. However, on page 5, the authors state that training PINNS with $L^\\infty$ is undesirable and can lead to $oscillatory behavior between different peaks of the PDE residual landscape\". Does this mean we should always use a small number of iterations in Algorithm 1? This aspect deserves further discussion.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "THJD98H5oYn", "reviewer": "Reviewer_rJ3m", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a novel sampling strategy for collocation points to improve the training performance of PINNs. The strategy is based on the propagation hypothesis, which suggests that information from the boundary of the computational domain has to \"propagate\" to its interior, and that this propagation sometimes fails during PINN training. The authors show that the approach indeed leads to more stable training.", "summary_of_the_review": "A clear paper with an intuitively appealing, simple, and yet effective method to mitigate training failures in PINNs. To judge if Evo is a major contribution towards PINN training requires more evidence.\n\n*EDIT:* Raised my score according to the discussion with the authors.", "strength_and_weaknesses": "The paper is excellently written, and the arguments for the Evo strategy are clear. The approach is simple, intuitive, and yet effective. The experimental evidence is good, although I would appreciate results with more benchmarks.\n\nThe two weaknesses I see are that 1) the benefit of Causal Evo is not shown or discussed in the experiments, and 2) that comparisons with some parts of the literature are missing entirely or difficult to evaluate.\n\n## Ad 1)\nI understand Evo as a method that mitigated training failures, rather than one that improves performance for a successfully trained PINN. Thus, from that perspective, Evo and Causal Evo perform equivalently in the shown experiments. It would be interesting to see an example where Evo fails because causality is not respected, but where Causal Evo successfully trains the PINN.\n\n## Ad 2)\nI am missing other approaches that have been proposed to mitigate training failures. These approaches include domain decomposition, loss weighting schemes, the architecture from arXiv:2109.09338, or the regularization term proposed in arXiv:2112.05620. It would further be interesting to see if and how the sampling scheme can be combined with the former methods. Further, the comparison is difficult to evaluate because the chosen benchmarks show little overlap with what has been proposed in the literature. For example, CPINN was evaluated in the original paper on a set of benchmarks that does not intersect with the benchmarks chosen in this paper. I would be interested to see how Evo performs on these benchmarks.\n\n## Minor\n- In the paragraph \"Propagation Failure\", the authors argue that trivial solutions can become attractive for PINNs. This has been made precise theoretically in arXiv:2109.09338 and experimentally in arXiv:2203.13648.\n- References to the appendix are broken.\n- The computational overhead of the comparison methods is not clear (\"Challenges with Potential Remedies\"). Similarly, for Evo it is not clear how $\\mathcal{P}_i^s$ changes throughout training (i.e., is it of fixed size, or is rather $\\mathcal{P}_i$ of fixed size?).\n- The four salient properties seem somewhat artificial and post-hoc. For example, the gradual accumulation property is not well justified. As an alternative property, it seems equally intuitive that initially the collocation points should concentrate around the initial and boundary conditions (ensuring successful learning locally, from which propagation can continue), while at later stages the collocation points should be uniform (ensuring uniform accuracy across the computational domain). Thus, a priori, the CPINN sampling strategy seems equally powerful to mitigate propagation failures as Evo. It is even more interesting to see that the CPINN fail for the systems for which Evo succeeds, and I would appreciate an evaluation of Evo on the benchmarks that were chosen by the original authors of CPINN.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject