Competitive Physics Informed Networks 
Authors: Qi Zeng
Yash Kothari
Spencer H Bryngelson
Florian Tobias Schaefer
Venue: ICLR 2023 poster (2023)
Keywords: Physics informed learning
multi-agent games
Lagrange multipliers
partial differential equations

Abstract: Neural networks can be trained to solve partial differential equations (PDEs) by using the PDE residual as the loss function. This strategy is called "physics-informed neural networks" (PINNs), but it currently cannot produce high-accuracy solutions, typically attaining about $0.1\%$ relative error. We present an adversarial approach that overcomes this limitation, which we call competitive PINNs (CPINNs). CPINNs train a discriminator that is rewarded for predicting mistakes the PINN makes. The discriminator and PINN participate in a zero-sum game with the exact PDE solution as an optimal strategy. This approach avoids squaring the large condition numbers of PDE discretizations, which is the likely reason for failures of previous attempts to decrease PINN errors even on benign problems. Numerical experiments on a Poisson problem show that CPINNs achieve errors four orders of magnitude smaller than the best-performing PINN. We observe relative errors on the order of single-precision accuracy, consistently decreasing with each epoch. To the authors' knowledge, this is the first time this level of accuracy and convergence behavior has been achieved. Additional experiments on the nonlinear Schr{\"o}dinger, Burgers', and Allen--Cahn equation show that the benefits of CPINNs are not limited to linear problems.

Published as a conference paper at ICLR 2023
COMPETITIVE PHYSICS INFORMED NETWORKS
Qi Zeng, Yash Kothari, Spencer H. Bryngelson & Florian Schäfer
School of Computational Science and Engineering Georgia Institute of Technology Atlanta, GA 30332, USA (qzeng376, ykot har shbê, sww.eeerrseeeeerec.coo.o /gatech.edu
ABSTRACT
Neural networks can be trained solve partial differential equations (PDEs) by using the PDE residual as the loss function. This strategy is called "physics- informed neural networks" (PINNs), but it currently cannot produce high-accuracy solutions, typically attaining about 0.1% relative error. We present an adversar- ial approach that overcomes this limitation, which we call competitive PINNs (CPINNs). CPINNs train a discriminator that is rewarded for predicting mistakes the PINN makes. The discriminator and PINN participate in a zero-sum game with the exact PDE solution the large condition numbers of PDE discretizations, which is the likely reason for failures of previous attempts to decrease PINN errors even on benign problems. Numerical experiments on Poisson problem show that CPINNs achieve errors four orders of magnitude smaller than the rest-performing PINN. We observe relative errors on the order of single-precision accuracy, consistently decreasing with each epoch. To the authors' knowledge, this is the first time this level of accuracy and convergence behavior has been achieved. Additional experiments on the nonlinear Schrödinger, Burgers', and Allen Cahn equation show that the benefits of CPINNs are not limited to linear problems.
optimal strategy. This approach avoids squaring
INTRODUCTION
PDE-constrained deep learning. Partial differential equations (PDEs) model physical phenomena like fluid dynamics, heat transfer, clectromagnetism. and more. The rising interest in scientific machine learning motivates the study of PDE-constrained neural network training (Lavin et al., 2021). Such methods can exploit physical structure for learning or serve PDE solvers in their own right.
Physics informed networks. Lagaris et al. (1998) represent PDE solutions as neural networks by including the square of the PDE residual in the loss function, resulting neural network- based PDE solver. Raissi et al. (2019) recently refined this approach further and called "physics informed neural networks (PINNs)." initiating a flurry of follow-up work. PINNs are far less efficient than classical methods for solving most PDEs but are promising tools for high- sigh--imenssonal or parametric PDEs (Xue al., 2020) and data assimilation problems. The training of PINNs also serves model problem for the general challenge of imposing physical constraints on neural networks, an area of fervent and increasing interest (Wang et al., 2021b; Li al., 2021; Donti al., 2021).
Training pathologies in PINNs. PINNs can, in principle, be applied to all PDEs, but their numerous failure modes are vell-documented (Wang al., 202 Liu al., 2021: Krishnapriyan et al., 2021). For example, they are often unable to achieve high-accuracy solutions. The first works on PINNs reported relative T.2 errors of about 10-3 (Raissi et al., 2019). The authors are unaware of PINNs achieving errors below 10-3 even in carefully crafted, favorable settings. Higher accuracy is required in many applications.
Existing remedies. A vast and growing body of work aims to improve the training of PINNs, often using wroblem-specifee insights. For example, curriculum learning can exploit causality in time- dependent PDEs (Krishnapriyan et al., 2021; Wang et al., 2022a; Wight & Zhao, 2020). Krishnapriyan et al. (2021) also design curricula by embedding the PDE in a parametric family of problems of varying difficulty. Other works propose adaptive methods for selecting the PINN collocation points (Lu et al.,

---

Published as a conference paper at ICLR 2023
2021; Nabian et al., 2021; Daw et al., 2022). Adaptive algorithms for weighing components of the PINN loss function have also been proposed (McClenny & Braga-Neto, 2020; Wang et al., 2022b; van der Meer et al., 2022). Despite these improvements, the squared residual penalty method used by such PINNs imposes a fundamental limitation associated with conditioning, which is discussed next.
The key problem: Squared residuals. Virtually all PINN-variants use the squared PDE residual as loss functions. For a linear PDE of order 8, this is no different than solving an equation of order 2s, akin using normal equations in linear algebra. The condition number x of the resulting problem is thus the square of the condition number of the original one. Solving discretized PDEs an ill-conditioned problem, inhibiting the convergence of iterative solvers and explaining the low accuracy of most previous PINNs. It is tempting to address this problem using penalties derived from p-norms with p + 2. However, choosing p > leads to worse condition numbers, whereas p < sacrifices the smoothness of the objective. The convergence rates of gradient descent on (non-)smooth convex problems suggest that this trade is unfavorable (Bubeck et al., 2015).
Weak formulations. Integration by parts allows the derivation of weak form of PDE, which for some PDEs can be turned into a minimization formulation that does not square the condition number. This procedure has been successfully applied by E & Yu (2017) to solve PDEs with neural networks (Deep Ritz). However, the derivation of such minimization principles is problem-dependemt limiting the generality of the formulation. Deep Ritz also employs penalty methods to enforce boundary values, though these preclude the minimization problem's solution from being the PDE's exact solution. Liao & Ming (2019) proposed partial solution to this problem. The work most closely related to ours is by Zang et al. (2020), who proposes game formulation based on the weak form.
Competitive PINNs. We propose Competitive Physics Informed Neural Networks (CPINNs) to address the above problems. CPINNs are trained using a minimax game between the PINN and a discriminator network. The discriminator learns to predict mistakes of the PINN and is rewarded for correct predictions, whereas the PINN is penalized. We train both players simultaneously on the resulting zero-sum game to reach a Nash equilibrium that matches the exact solution of the PDE.
Summary of contributions. A novel variant of PINNs, called CPINNs, is introduced, replacing the penalty employed by PINNs with a primal- approach. This simultaneously optimizes the PINN and a discriminator network that learns to identify PDE violations and boundary constraint violations. We optimize PINNs with competitive gradient (CGD) (Schäfer & Anandkumar, 2019) and compare their performance to regular PINNs trained with Adam. On a two-dimensional Poisson problem, CPINNs achieve relative accuracy of almost 10 88 improving over PINNs by four orders of magnitude. To the best of our knowledge, this is the first time a PINN-like network was trained to this level of accuracy. We compare PINNs with CPINNs on a nonlinear Schrödinger equation, a viscous Burgers' equation, and Allen-Cahh equation. In all but the last case, CPINNs improve over PINNs trained with comparable computational budget
2 COMPETITIVE PINN FORMULATION
We formulate CPINNs for PDE of the general form
A[u] = f. in 52 g, on 0n,
(1)
where Al-] is (possibly nonlinear) differential operator and 92 is a domain in IRd' with boundary 0.. To simplify notation, we assume that f. and u are real-valued functions on Ω, in, and 12 U an, respectively. One can extend both PINNS and CPINNs to vector-valued such functions if needed.
2.1 PHYSICS INFORMED NEURAL NETWORKS (PINNS)
PINNs approximate the PDE solution u by neural network mapping d- d-variate inputs to real numbers. The weights are chosen such as to satisfy equation and equation 2 on the points C Ω and के C a2. The loss function used to train P has the form
PINN (P, 2
C (P, (P) + L8n
(P, x),
(3)
2

---

Published as a conference paper at ICLR 2023
where CPNN measures the violation of the boundary conditions equation 2 and C measures the onn violation of the PDE of equation 1. They are defined as
No
PINN (P.
(A(P)(a)) -(x))))
(4)
No
Nan (P. (T.))²
PINN
(5)
Here, Ng and Nan are the number of points in the sets (interior) and (boundary), and x; and T; are the i-th such points in æ and Æ.
PINNS approximate the exact solution u of the PDE by minimizing the loss in equation 3. However, optimizing this loss using established methods such as gradient descent, ADAM, or LBFGS often leads to unacceptably large errors or an inability to train at all (Wang et al., 2021a; 2022b). This pathology has been attributed to the bad conditioning of the training problem (Krishnapriyan et al., 2021). Next, we introduce CPINNs, game- based formulation designed mitigate these problems.
2.2 COMPETITIVE PHYSICS INFORMED NEURAL NETWORKS (CPINNS)
CPINNs introduce more discriminator networks D with input e I and outputs (z and (z) P and D compete in a zero-sum game where P learns to solve the PDE, and D learns to predict the mistakes of P. This game is defined minimax problem
max min CCINN D
L
(P,D, + Lan (P,
(6)
P
where
Na Do(a;) (A[P](x;) - f(z,))
Cn
CPINN (D,P,
(7)
No
Naa
CPINN (D, P.
Dan [T, (P (T.) (I.)).
(8)
Nao
Here, (x:);) and (T)( can be interpreted as bets by the discriminator that the PINN will over- or under-shoot equation and equation Winning the bet results in reward for D and a penalty for P; a lost bet has the opposite effect. We emphasize that choosing bets on the residual is fundamentally different from choosing weights for the different components of a loss function, as done by McClenny & Braga-Neto (2020); Wang et al. (2022b); van der Meer et al. (2022)
The Nash equilibrium of this game is P u and D 0. Thus, iterative algorithms for computing Nash equilibria in such zero-sum games can be used to solve the PDE approximately. This work focuses on the CGD algorithm of Schäfer Anandkumar (2019). Still, CPINNs can be trained with other proposed methods for smooth game optimization (Korpelevich, 1977; Mescheder al., 2017; Balduzzi et al., 2018; 8;; Gemp & Mahadevan, 2018; Daskalakis et al., 2018; Letcher et al., 2019). In experiments, and D are fully connected networks with hyperbolic tangent and ReLU activation functions, respectively. Each network's 's number of layers and neurons depends on the PDE problem.
2.3 AVOIDING SQUARES OF DIFFERENTIAL OPERATORS
Multiagent methods for solving PDEs may seem unorthodox, yet they are motivated by observations in classical numerical analysis. Consider the particular case of linear PDE and networks P, D with outputs that depend linearly on their weight-vectors π and d resulting in the parametric form
dim((T)
dimu(8)
P(x)
Σ Tai(())
D(x)
Σ
for basis function sets (1: (gs)siid and { (ds)ssicdi(()) We focus our attention on the PDE constraint in equation 1, which is evaluated at set of Na points. Defining A Xddim(m) and f IR Na, this leads to
Aij Alvy)(ax),
fa = s(x))
(10)
3

---

Published as a conference paper at ICLR 2023
and the discretized PDE
AT= f.
(11)
PINNs solve equation 11 via a least squares problem
min ATT fll.,
(12)
trading the equality constraint for a minimization problem with solution π =
Since the matrix (A" (A A) is symmetric positive-definite, one can solve it with specialized algorithms such as the conjugate gradient method (CG) (Shewchuk, 1994). This approach is beneficial for well- conditioned nonsymmetric matrices but inappropriate for ill-conditioned A (Axelsson, 1977). This is because k( A A) ro( [A),, resulting in slow convergence of iterative solvers. Differential operators are unbounded. Thus, their discretization leads to ill-conditioned linear systems. Krishnapriyan et al. (2021) argue that the ill-conditioning of equation 12 causes the optimization difficulties they observe.
CPINNs turn the discretized PDE in equation 11 into the saddlepoint problem
min max 8 (A f)-
(13)
The solution to this problem is the same as that of the system of equations
0
A
T
π
0
with x ((2
0 A ) x(A).
(14)
8
f
By turning equation 11 into the saddle point problem of equation 13 instead of the minimization form of equation 12, CPINNs avoid squaring the condition number. Applying a Krylov subspace method directly to equation 14 will not improve the convergence compared to working with A A. However, gradients of equation 13 provide access to matrix-vector products with A and A which can not be obtained from gradients of the least-squares problem. For many matrices, including symmetric positive-definite ones, this allows the significantly faster solution of equation 11 (Greenbaum, 1997). We now demonstrate empirically the benefits of equation 13 in the nonlinear setting.
RESULTS
3.1 OVERVIEW
This section compares PINNs and CPINNs on series of model problems. The code used to produce the experiments described below can be found under gi ghu. com/ comp- comp-phys serrsic//oo Lcc//PPIIA INN. We study a -io-diiensional Poisson problem (section 3.2), a nonlinear Schrödinger equation (section 3.3). a viscous Burgers' equation (section 3.4), and the Allen-Cahn equation (section 3.5). Throughout the previous sections, we train PINNs using Adam and CPINN using adaptive competitive gradient descent (ACGD) (Schäfer et al., 2020b). The latter combines CGD of Schäfer & Anandkumar (2019) with Adam-like heuristic for choosing step sizes.
ACGD uses GMRES (Saad & Schultz, 1986) and the Hessian vector products obtained by automatic differentiation to solve the linear system defining the CGD update. Thus, iterations of (A)CGD are considerably more expensive than those of Adam. To account for this difference fairly, we also provide the number of forward passes through the neural network required by the two methods. An Adam iteration amounts to a single forward pass and an ACGD iteration to two forward passes, plus two times the number of GMRES iterations. We also tried other optimizers for both PINNs and CPINNs but found them inferior to Adam and ACGD. A comparison is presented in section 3.6.
3.2 POISSON EQUATION
We begin by considering a two-dimensional Poisson equation:
Au(x, v) = 2 sin(()) cos(y), x, I E 1-2, 2]
(15)
with Dirichlet boundary conditions
u(x, -2) = sin(x) cos( -2).
u(-2.y) = cos(y),
u(x, 2) sin (x) cos( 2): u( 2, sin( 2) cos(y): (v)-
4

---

Published as a conference paper at ICLR 2023
PINN + Adam
CPINN + ACGD
L2 Relative Error
10-1
10
10 5
10-7
10a
10a
10
10
10
107
10a
10
10
10"
Iterations
Forward Passes
Figure 1: Comparison of CPINN and PINN the Poisson problem of equation 15 in terms of relative error. CPINN has a faster convergence rate and reduces the L2 error to 1.7 x 10 whereas the PINN case has L2 error of 1.2 x 10 even with larger computational budget.
Dr (Discriminator)
(Exact)
PP
a (PINN)
P u (CPINN)
X10
10
10
1
-2
-1
0
1
-2
-1
2
-1
0
2
-2
-1
0
2
(b)
(d)
2
Figure 2: (a) Exact solution u to equation 1, absolute errors of (b) PINN + Adam after 3 x 10? training iterations and (c) CPINN ACGD after 48 000 training iterations, and (d) the discriminator.
This problem has the manufactured solution
u(x, v) sin( cos(y).
(16)
The PINN implementation has losses
1 Non
(P(I.T)) - (Z)).
(17)
Noo
No
PINN
(Pen Wi) + + Pay vi) + 2in(((; cos(un))2
(18)
Na
and the CPINN losses are
Non
CPINN
Dan(T.)
(P(T.,)) u(..,.)))
(19)
00
Non
1=1
No
CCPINN
Da(x;) (P (Zi, v.) Pay wi) + 2sin(x)) cos(y:))
(20)
No.
1=1
Figure 2 (a) shows the exact solution of the PDE and the absolute error of the best models trained using (b) PINN and (c) CPINN, as well as an example of the discriminator output (d). The CPINN achieves lower errors than the PINN by factor of about 106h throughout most of the domain.
5

---

Published as a conference paper at ICLR 2023
PINN
Adam
CPINN + ACGD
La Relative Error
10-1
10"
10 -3
10
10a
10a
10a
10a
10
10a
10
10a
10
10
Iterations
Forward Passes
Figure 3: Comparison of CPINN and PINN on the nonlinear Schrödinger eq. (21) in terms of relative errors. After 200 000 training iterations, PINN cannot reduce the L2 error further, plateauing about 4 x 10-3, whereas CPINN reduces the error x 10-4 under smaller computational budget.
PINN Adam
CPINN ACGD
La Relative Error
10-2
10-3
10-4
10-5
10a
10a
10"
10
107
10
10
10a
10"
107
Iterations
Forward Passes
Figure 4: The relative errors of CPINNs (current) and PINNs on the Burgers' equation.
Figure 1 shows the relative L2 error of a PINN and a CPINN on the Poisson problem. CPINN shows faster convergence rate regarding the number of forward passes and training epochs, and its accuracy is higher than that of PINN by orders of magnitude.
3.3 NONLINEAR SCHRÓDINGER EQUATION
In this subsection, we apply the competitive PINN methodology to solve the Schrödinger equation
1
ut + 2 jux + |u wu22u
= [-5,5),, te [T,]
(21)
where u(t, is the complex-valued solution and
u(0, x) = 2sech(x), ult, -) = u(t,
u, -5) = Un (t,
(22)
are the initial and boundary conditions.
The best results for CPINNs and PINNs presented in fig. 3. CPINN reached an Ly relative error of 6 10 after 4.2 x 10 training iterations with 4.1 x 10t forward passes. PINN reached an Lg relative error of 5 10-3 after 6.2 106 training iterations, equivalent 6.2 106 forward passes.
3.4 BURGERS EQUATION
We next consider the case of the viscous Burgers' equation. This nonlinear second-order equation is
U4 uur (0.01/))umz 0, I x [-111, tE [0,0].
(23)

---

Published as a conference paper at ICLR 2023
PINN + Adam
CPINN + ACGD
100
1m
La Relative Error
10-1
10
10-3
mnl
10a
10
10a
10
10a
10*
10
10
107
Iterations
Forward Passes
Figure 5: The relative for the Allen Cahn equation. A CPINN and PINN that uses the curriculum learning approach of Wight & Zhao (2020) are shown. CPINN does not outperform this particular PINN, which may be due the overall low accuracy of both methods.
which has parameters matching Raissi al. (2019). w(l, x) the solution of the PDE and
u(0, x) = sin(TT)) uu4, 1) u(t, 1)
(24)
are the initial and boundary conditions. A comparison to CPINNs is presented in fig. 4. CPINN exhibits an improved convergence rate and continues reduce the error, whereas the progress of PINN eventually stagnates.
3.5 ALLEN-CAHI EQUATION
We next consider the one-dimensional Allen Cahn equation with periodic boundary conditions, a cubically nonlinear equation given by
Ue 0.0001u
5u 5u 0, xE[-1,1], [0, 1],
(25)
where utt, the solution of the PDE and
u(0, x = x² cos(TT))
u(t, -(1,-1) = w(l, 1), Ua
un (1, 1) (1, -))
(26)
the initial and boundary conditions. This example follows Raissi et al. (2019), and their training and testing data.
Raissi et al. (2019) does report the performance of standard PINN and instead shows the performance of a problem-specific modification that exploits the temporal structure of the problem. Wight & Zhao (2020) observe that this problem is difficult solve directly for PINNs and propose curriculum learning approach to remedy this problem.
PINN + Adam CPINN ACGD CPINN XAAaa
WAN
Adam
PINN + SGD CPINN + BCGD
CPINN + XSGD WAN + ACGD
10a
(b)
La Relative Error
10-3
10
10a
10
10a
10c
107
10a
10a
10
107
Iterations
Forward Passes
Figure 6: Comparison of PINNs and CPINNs with different optimizers for the Poisson problem. Following Zang et al. (2020), WAN Adam uses Adagrad update the player.
7

---

Published as a conference paper at ICLR 2023
Table Performance of CPINNs and PINNS on the 2D Poisson problem of equation 15.
Optimizer
Iterations
Lg Rel. Error
CPNNN
CPNNN
CPNNN
WAN CPINN PINN
Adam
3 x 107
1.2 x 10-4
1.4 X 10-8
8.4 X 10-8
5.5 x 10-9
SGD
2.2 x 107
1.3 x 10-3
1.1 x 10-6
5.2 x 10-7
6.3 x 10-7
ACGD
4.8 x 104
1.7 x 10-8
2.1 10-14
2 10-14
x 10 16
CGD
6 x 105
3.5 x 10
5.1 X 10-7
3.5 x 10-7
1.7 x 10-7
XAdam
8.5 x 106
1.6
1.6 x 10
1.6 x 10t
0.2
XSGD
8.5 x 106
1.2 x 10-4
1.2 10-?
x 10-8
3 x 10-8
Adam + Adagrad
1.3 x 106
1.2
2.6
2.6
1.2 x 10-3
ACGD
1.8 x 10a 9 x 10-3
1.1 x 10-3
1.1 10-3
2.7 x 10-6
We use the approach of Wight & Zhao (2020) for PINNs and suitable adaptation for CPINN. On this problem, PINNs modestly outperform CPINNs. This may be due to the slow convergence that prevents reaching the regime where PINNs begin plateauing. Using curriculum learning could improve this (Wight& & Zhao, 2020).
3.6 OTHER OPTIMIZERS
We repeat experiments with the Poisson problem using different optimizers to distinguish the effects of CPINN and ACGD, We use Adam and SGD for PINNs, and basic CGD (BCGD, without adaptive step sizes), ACGD, ExtraSGD (XSGD), and ExtraAdam (XAdam) for CPINNs (Korpelevich, 1977; Gidel et al., 2019). We also show results for Adam and ACGD on the weak adversarial network (WAN) of Zang al. (2020). BCGD/ACGD applied to single agent reduce to SGD/Adam. Thus, is natural to compare CPINNs trained with BCGD/ACGI to PINNs trained with SGD/Adam.
We observe that combined with CPINN, the nonadaptive algorithms BCGD and extragradient improve over PINN trained with (the equally nonadaptive) SGD. The latter even improves over PINNs trained with Adam after sufficient iterations. We could not achieve training via ExtraAdam the CPINN strategy. We were also unable to achieve high accuracy with WAN, The ACGD optimizer achieves much better results than all others. We conclude that CPINN improves over PINN and WAN. Realization of this advantage is enabled by the robust and adaptive training afforded by ACGD.
CONNECTIONS TO EXISTING WORK
Saddle point formulations arise in Setrov-Galerte and mixed finite-element methods, which use different dinens-amniio function spaces to represent the solution (called trial functions) and measure the violation of the equations (called test functions) (Quarteroni & Valli, 2008; Fortin & Brezzi, 1991). The neural networks introduced here, P and D, are nonlinear analogs of trial and test spaces. Unlike our approach, Deep Sarro-GGlerkin of Shang et al. (2022) only parameterizes the trial space by a neural network and uses conventional discretization such as finite elements for the test space. They only train the last layer while keeping all other layers fixed after initialization.
Saddle point problems in PDEs can be interpreted geometrically and have game-theo interpreta- tion (Lemaire, 1973). In complementary game-theorethe approach, Owhadi (2017) cast computation as a game between a numerical analyst and an environment, using ideas from decision theory (Wald, 1945). Saddle point problems also arise from the introduction of Lagrange multipliers, which are used cast a constrained optimization problem as an unconstrained saddle point problem (Brezzi, wtw--nttww-wwwweettiieeee parametrized Lagrange multipliers that enforce distributional equality (in GANs) or satisfaction of the PDE (in CPINNs).
1974). These discriminators can be viewed as netral-network-
Our work is thus related to recent efforts that combine CGD with Lagrange multipliers to solve constrained optimization problems arising from reinforcement learning and computer graphics (Bacon al., 2019; Yu et al., 2021; Soliman al., 2021). The need for constrained training of neural networks also arises in other applications, resulting in an increasing body of work on this topic (Pathak et al., 2015; Donti et al., 2021; Lokhande al., 2020; Fioretto al., 2020).
8

---

Published as a conference paper at ICLR 2023
Following section 1, Deep Ritz exploits the fact that some PDEs be cast as minimization problems without squaring the condition number (E & Yu, 2017), expressing equation 11 as
π T Am
min
π f.
2
for symmetric positive definite matrix A. Such a formulation is not always available, and even if it exists, one still must enforce the PDE boundary conditions (Liao & Ming, 2019). Zang et al. (2020) uses weak PDE formulation to derive minimax formulation. In the notation of section 2.3, their approach is similar to using the zero-sum game
min max log (ATTA6) A6) log (1811").
In contrast to our work, this formulation encourages the max max- player only to learn where the min-playe violates the PDE, not how (the sign of the residual) the PDE is being violated.
Many works adapt the penalty weights in PINNS during training to improve accuracy, often using minimax formulations (Wang et al., 2021a; Xu et al., 2021: McClenny & Braga-Neto, 2020; van der Meer et al., 2022). Different from the D output of CPINNs, these weights are multiplied with the square violation of the equality constraint, which is always greater than zero. Therefore, they do not correspond to a meaningful zero-sum game because the optimal discriminator strategy. this case, drives all weights to infinity. Alternatively, Krishnapriyan et al. (2021) recommends the of curriculum learming. They train the initial PINN on better-condine contitroond variant of the original problem that slowly transforms into the target problem during training.
The discussion in section 2.3 seemingly contradicts the works of Soliman et al. (2021); Lu et al. (2022) on Sobolev acceleration, whereby training on higher order derivatives of the loss can lead improved accuracy. As discussed in appendix A.7 of the appendix, we believe that Sobolev acceleration can be reconciled with the arguments presented in section 2.3. However, a rigorous understanding of the relationship of these two perspectives is presently missing and will require additional research.
DISCUSSION
This work introduced CPINNs, an agent-based approach to the neural-network- based solution of PDEs. CPINNs crafted to avoid the ill-conditioning resulting from traditional PINNs least squares loss functions. Section 3 showed that CPINNs trained with ACGD improve upon the accuracy of PINNs trained with a comparable computational budget and can solve PDEs beyond even single- precision floating-poin accuracy, the first approach with this capability.
With CPINNs, one can now achieve required. This is due to the number of CG iterations performed for each ACGD step. Reducing this overhead is a direction for future work. Potential solutions this problem include cheaper approximate solutions of the matrix inverse, different iterative solvers, or the recycling of Krylov subspaces (Paige & Saunders, 1975; Parks al., 2006). It may also be worth investigating alternatives to ACGD with lower per-iteration cost, such as the methods proposed by Tan et al. (2018).
errors, but significant computation is still
The experiments used the same training points for each iteration, following Raissi et al. (2019). In the future, will investigate the effeets of batch stochasticity training accuracy. We also plan to investigate competitive mirror descent (CMD) for partial differential incqualities and contact problems (Schäfer et al., 2020a; Lions, 1972) and, more generally, CPINN-like approaches to other problems involving the constrained training of neural networks.
ACKNOWLEDGMENTS
This research was supported in part through research cyberinfrastructure resources and services provided by the Partnership for an Advanced Computing Environment (PACE) at the Georgia Institute of Technology, Atlanta, Georgia, USA. SHB acknowledges the NVIDIA Academic Hardware Grant Program for facilitating part this work. We thank Hongkai Zheng for maintaining the CGDs package used to conduct this work.

---

Published as a conference paper at ICLR 2023
REFERENCES
Owe Axelsson. Solution of linear systems of equations: Iterative methods. In Sparse matrix techniques, pp. 1-51. Springer, 1977.
Pierre-Luc Bacon, Florian Schäfer, Clement Gehring, Animashree Anandkumar, and Emma Brunskill. A Lagrangian method for inverse problems in reinforcement learning. In Optimization in RL workshop at NeurIPS, 2019.
David Balduzzi, Sebastien Racaniere, James Martens, Jakob Foerster, Karl Tuyls, and Thore Graepel. The mechanics of n-player differentiable games. In International Conference Machine Leurning, pp. 354-363. PMLR, 2018.
Franco Brezzi. On the existence, uniqueness and approximation of saddle-point problems arising from Lagrangian multipliers. Publications mathématiques et informatique de Rennes, (S4):1-26, 1974.
Sébastien Bubeck et al. Convex optimization: Algorithms and complexity. Foundations and Trends in Machine Learning, 8(3-4):231-355 -37,, 2015.
Constantinos Daskalakis, Andrew Hyas, Vasilis Syrgkanis, and Haoyang Zeng. Training GANs with optimism. In International Conference on Learning Representations, 2018.
Arka Daw, Jie Bu, Sifan Wang, Paris Perdikaris, and Anuj Karpatne. Rethinking the importance of sampling in physics-informed neural networks. arXiv:2207.02338. 2022.
Priya Donti, David Rolnick, and J Zico Kolter. DC3: A learning method for optimization with hard constraints. In International Conference on Learning Representations, 2021.
Weinan E and Bing Yu. The Deep Ritz method: A deep learning-basse numerical algorithm for solving variational problems. Communications Mathematics and Statistics, 6:1-12, 2017.
Ferdinando Fioretto, Pascal Van Hentenryck, Terrence WK Mak, Cuong Tran, Federico Baldo, and Michele Lombardi. Lagrangian duality for constrained deep learning. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 118-135. Springer, 2020.
Michel Fortin and Franco Brezzi. Mixed and hybrid finite element methods, volume 3. New York: Springer- Verlag, 1991.
Ian Gemp and Sridhar Mahadevan. Global convergence to the equilibrium of GANs using variational inequalities. wwxh::888.0153/ 2018
Gauthier Gidel, Hugo Berard, Pascal Vincent, and Simon Lacoste-Julien. A variational inequality
perspective on generative adversarial nets. 1802 2019.
Anne Greenbaum. Iterative methods for solving linear systems. SIAM, 1997.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv:1412.6980, 2014.
G. M. Korpelevich. Extragradient method for finding saddle points and other problems. Matekon, 13 (4):35 1977.
Aditi S. Krishnapriyan, Amir Gholami, Shandian Zhe, Robert M. Kirby, and Michael W. Mahoney. Characterizing possible failure modes in physics-informed neural networks. arXiv:2109.0000 1050, 2021.
Isaac E Lagaris, Aristidis Likas, and Dimitrios I Fotiadis. Artificial neural networks for solving ordinary and partial differential equations. IEEE transactions on neural networks, 9(5):987-1000, 1998.
Alexander Lavin, Hector Zenil, Brooks Paige, David Krakauer, Justin Gottschlich, Tim Mattson, Anima Anandkumar, Sanjay Choudry, Kamil Rocki, Atilım Güney Baydin, et al. Simulation intelligence: Towards new generation of scientific methods. arXiv:21 /2.02233, 2021.
10

---

Published as a conference paper at ICLR 2023
B. Lemaire. Saddle-point problems in partial differential equations and applications to linear quadratic differential games. Annali della Scuola Normale Superiore di Pisa-Clas AlasSClase di Scienze, 27 (1):105-160, 1973.
Alistair Letcher, David Balduzzi, Sébastien Racaniere, James Martens, Jakob Foerster, Karl Tuyls, and Thore Graepel. Differentiable game mechanics. The Journal of Machine Learning Research, 20(1):3032-3071 2019.
Zongyi Li, Hongkai Zheng, Nikola Kovachki, David Jin, Haoxuan Chen, Burigede Liu, Kamyar Azizzadenesheli, and Anima Anandkumar. Physics-ing mersics-inffrmed neural operator for learning partial differential equations. arXiv:2//1.0377 2021.
Yulei Liao and Pinghing Ming. Deep Nitsche method: Deep Ritz method with essential boundary conditions. arXiv:19/2.01309, 2019.
JL Lions. Partial differential inequalities. Russian Mathematical Surveys, 27(2):91, 1972.
Ziming Liu, Yunyue Chen, Yuanqi Du, and Max Tegmark. Augmene-Augmenedd Learning: A new paradigm beyond Physics-Iffrrmed Learning. arXiv:2109.33901 2021.
Vishnu Suresh Lokhande, Aditya Kumar Akash, Sathya N Ravi, and Vikas Singh. FairALM: Augmented Lagrangian method for training fair models with little regret. In European Conference on Computer Vision, pp. 365-381. Springer, 2020.
Lu Lu, Xuhui Meng, Zhiping Mao, and George Em Kariiadakis. DeepXDE: A deep learning library for solving differential equations. SIAM Review, 63(1):208-228, 2021.
Yiping Lu, Jose Blanchet, and Lexing Ying. Sobolev acceleration and statistical optimality for learning elliptic equations via gradient descent. arXiv preprint urXiv:2205.0733), 2022.
Levi McClenny and Ulisses Braga-Neto. Self-adaptive physics-informed neural networks using soft attention mechanism. arXiv preprint arXiv:2009.0444 044444, 2020.
Lars Mescheder, Sebastian Nowozin, and Andreas Geiger. The numerics of GANs. Advances in neural information processing systems, 30, 2017.
Mohammad Amin Nabian, Rini Jasmine Gladstone, and Hadi Meidani. Efficient training of physics- informed neural networks via importance sampling. Computer-Aided Civil and Infrastructure Engineering, 36(8):962-9) -977 2021.
Houman Owhadi. Multigrid with rough coefficients and multiresolution operator decomposition from hierarchical information games. SIAM Review, 59(1):99-144 2017.
Christopher C Paige and Michael A Saunders. Solution of sparse indefinite systems of linear equations. SIAM Journal on Numerical Analysis, 12(4):617-629, 1975.
Michael L Parks, Eric De Sturler, Greg Mackey, Duane Johnson, and Spandan Maiti. Recycling Krylov subspaces for sequences of linear systems. SIAM Journal on Scientific Computing, 28(5): 1651-1674, 2006.
Deepak Pathak, Philipp Krahenbuhl, and Trevor Darrell. Constrained convolutional neural networks for weakly supervised segmentation. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1796-1804, 2015.
Alfio Quarteroni and Alberto Valli. Numerical approximation of partial differential equations, volume 23, Springer Science & Business Media, 2008
Maziar Raissi, Paris Perdikaris, and George Em Karniadakis. Physics-informed informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378:686-707, 2019. ISSN 0021-9991.
Sebastian Ruder. An overview of gradient descent optimization algorithms. 1rXiv:1609.04747, 2016.

---

Published as a conference paper at ICLR 2023
Yousef Saad and Martin H Schultz. GMRES: A generalized minimal residual algorithm for solving nonsymmetric linear systems. SIAM Journal on Scientific and Statistical Computing, 7(3):856- 869, 1986.
Florian Schäfer and Anima Anandkumar. Competitive gradient descent. In NeurIPS, 2019.
Florian Schäfer, Anima Anandkumar, and Houman Owhadi. Competitive Mirror Descent. arXiv:2006. 10179, 2020a.
Florian Schäfer, Hongkai Zheng, and Anima Anandkumar. Implicit competitive regularization in GANs. arXiv:99/0.05852, 2020b.
Yong Shang, Fei Wang. and Jingbo Sun. Deep Petrov-Galerde method for solving partial differential equations. (rXXv:2201.12995, 2022.
Jonathan Richard Shewchuk. An introduction to the conjugate gradient method without the agonizing pain. Technical report, Carnegie Mellon University, 1994.
Yousuf Soliman, Albert Chern, Olga Diamanti, Felix Knöppel, Ulrich Pinkall, and Peter Schröder. Constrained Willmore surfaces. ACM Transactions on Graphics (TOG), 40(4):1-17, 2021.
Hwijae Son, Jin Woo Jang, Woo Jin Han, and Hyung Hwang. Sobolev training for the neural network solutions of pdes. arXiv preprint
2021.
Conghui Tan, Tong Zhang, Shiqian Ma, and Ji Liu. Stochastic primal-dual method for empirical risk minimization with (1) per- per-iteration complexity. Advances in Neural Information Processing Systems, 31, 2018.
Remco van der Meer, Cornelis W Oosterlee, and Anastasia Borovykh. Optimally weighted loss func- tions for solving PDEs with neural networks. Journal of Computational and Applied Mathematics, 405:1 13887, 2022.
Abraham Wald. Statistical decision functions which minimize the maximum risk. Annals of Mathematics, pp. 265-280, 1945.
Sifan Wang, Yujun Teng, and Paris Perdikaris. Understanding and mitigating gradient pathologies
in physics-informed neural networks. SIAM Journal of Scientific Computing, 43:A3055-A3081, 2021a.
Sifan Wang, Hanwen Wang, and Paris Perdikaris. Learning the solution operator of parametric partial differential equations with physics-informed DeepONets. Science advances, 7(40), 2021b.
Sifan Wang, Shyam Sankaran, and Paris Perdikaris. Respecting causality is all you need for training physics-informed neural networks. arXiv:2203.074000 2022a.
Sifan Wang. Xinling Yu, and Paris Perdikaris. When and why PINNs fail to train: A neural tangent kernel perspective. Journal of Compuational Physics, 449: 10768, 2022b.
Colby L Wight and Jia Zhao. Solving Allen and CChh-Hilliare equations using the adaptive physics informed neural networks. arXiv:2007.04542, 2020.
Rui Xu, Dongxiao Zhang, Miao Rong, and Nanzhe Wang. Weak form theory-guided neural network (TgNN-wf) for deep learning of subsurface single-and two-phase flow. Journal of Computational Physics, 436: 10318, 2021.
Tianju Xue, Alex Beatson, Sigrid Adriaenssens, and Ryan Adams. Amortized finite element analysis for fast PDE-constrained optimization. In International Conference on Machine Learning, pp. 10638-10647. PMLR, 2020.
Jing Yu, Clement Gehring, Florian Schäfer, and Animashree Anandkumar. Robust reinforcement learning: A constrained game-theoretic approach. In Learning for Dynamics and Control, pp.
1242-1254, 2021.
Yaohua Zang, Gang Bao, Xiaojing Ye, and Haomin Zhou. Weak adversarial networks for high- dimensional partial differential equations. Journal of Computational Physics, 411:109409, 2020.
Hongkai Zheng. CGDs. http://hhhttt..ceee/ee//////a/ap
2020.
12

---

Published as a conference paper at ICLR 2023
A APPENDIX
This appendix details the experiments mentioned in Section 3 and additional plots for the solutions. We also added a visualization of the CPINN method for a more clear understanding
All PINN models and CPINN generators use the hyperbolic tangent as the activation function. Discriminators in CPINNs contain ReLU as the activation function. The number of neurons and layers in the neural networks varies for each equation. We use the GMRES- ACGD implemen- tation of Zheng (2020), available under the MIT license at https: : /tgtthut.bb..///g/// com/ devzhk/ well the implementation of ExtraGradient methods available under the MIT license at https:/ /ttg//gttubbbb/// com/ Gauth rriidee /'Variia Paariationn aataioaalll Iww.innniitl--aeeeeeeeeeeaarelle For the experiments in section 3.3, 3.4, 3.5 use the training and testing data sets from https::// /ttp:://githhlee
cgds-package
com/ ma wwaaaaaaaraa iitarririssss.. /eriissiPiii /PPIINs / /teee//eeeeeee weee/mmaterrre messtermmai,, which are avail- able under the MIT license. For the Weak Adversarial Network experiments we used code from https: /gthhb... www.oo////////oo yaamm/yaahaa222..m wan, available under the MIT lience.
A.1 VISUALIZATION
In fig. 7 we provide visualization of PINN and CPINN example of Poisson problem
Au(x) in the domain Q u(x)= = g on the boundary 8Q.
(27)
(28)
In PINN, we feed training set x to PINN with weights a. PINN output u and we can calculate Au with respect to the input. The loss function CPNNN of PINN is equal to the sum of the mean square residual between g and on the boundary and the mean square residual between and use in the domain.
For CPINN, similarly, we feed training set to PINN with weights o. PINN outputs and can calculate Au with respective to the input. We also feed the training set æ to a discriminator network with weights 0. The discriminator predicts the residuals associated with PINN and outputs two vectors of bet coefficients, due and daw, to penalize the domain residual associated with eq. (27) and the boundary value residual eq. (28). The loss function [ CCPNN of CPINN is equal to the sum of the product of discriminator bet vectors and residuals of PINN.
Discriminator
PINN
Au
dea
da
Au
PINN
MSE(g Wa.) MSE(f Au)
CPINN
danlg us.) + da(s
Au)
(a) PINN
(b) CPINN
Figure 7: Illustration of (a) PINN and (b) CPINN.
A.2 POISSON EQUATION
For the Poisson equation, we use 5 000 training points within the domain [-2, [2, 50 training points on each side of the domain boundary. We randomly selected all training points with Latin Hypercube sampling. The PINN model contains 3 hidden layers with 50 neurons in each layer. The discriminator in the CPINN contains 4 hidden layers with 50 neurons in each layer.
13

---

Published as a conference paper at ICLR 2023
PINN
Adam
CPINN + ACGD
La Relative Error
10-1
10
10
10
10*
10
10
10
10
10
Forward Passes
Computational Time (in seconds)
Figure 8: Comparison of PINN and CPINN in terms of the number of forward passes and computa- tional time. Note that the two plots appear almost identical, establishing that forward passes are an implementaaioo-nndeeennnnt computational cost proxy.
The PINNs are trained with Adam (Kingma & Ba, 2014) and SGD (Ruder, 2016). Adam and ACGD both use a learning rate of 10 10-3, beta values 31 = 0.99 and β3 = 0.99. The e of Adam and ACGD are each set to 10-8 and 10-6, respectively (all parameters following the usual naming conventions).
We also included comparison of PINN and CPINN in fig. 8 in terms of both the forward passes and computational time compare the computational cost. We train both models on NVIDIA V100 GPU.
A.3 SCHRÖDINGER'S 'S EQUATION
We use Latin Hypercube sampling to randomly select 20 000 training points within the domain and 50 points on each boundary.
We test several different network configurations and hyperparameters, This includes combinations of the number of neurons per layer for each network, including 100 and 200 neurons per layer; learning rates of 10-5, 1004, x 10 10 10 and 2 10 2; and Adam and ACGD values of (0.99, 0.99) and (0.9, 0.999).
We chose 10 and 10 the learning rate, (0.9, 0.999) and (0.99, 0.99) the 3 values, 10-8 and 10 as the e for Adam and ACGD, respectively. The iterative lincar solve of the ACGD optimizer uses relative tolerance of 10-7 and absolute tolerance of 10-20
The PINN presented in fig. 3 contains hidden layers with 100 neurons per layer, and the discriminator in CPINN contains hidden layers with 200 neurons per layer.
Figure (a) shows the exact solution of the PDE and the absolute error of the best models trained using (b) PINN and (c) CPINN, well example of the discriminator output (d).
14

---

Published as a conference paper at ICLR 2023
Da (Discriminatorr
uia, f) (Exact)
PP a (PINN)
PP
ui (CPINN)
xli
10
10
-0.5
0.0
1.4
12
10
0.8
0.6
0.2
0.0
Figure 9: (a) Exact solution u to eq. (21), absolute errors of (b) PINN + Adam after 6.2 10 training iterations and (c) CPINN ACGD after 200 training iterations, and (d) the discriminator.
A.4 BURGERS' EQUATION
For the Burgers' equation, we use Latin Hypercube sampling to randomly select 000 training points within the domain, 100 points at the initial condition and 100 points for each boundary condition specified in eq. (24).
The PINN model contains hidden layers with 60 neurons per layer, and the discriminator in the CPINN contains 8 hidden layers with 60 neurons per layer.
The B of Adam and ACGD are (0.99, 0.99), and the learning rates of both optimizers are 10-3 The c are 10-8 and 10-6 for Adam and ACGD, respectively. The iterative linear solve of the ACGD optimizer had relative tolerance of 10-7 and an absolute tolerance of 10-20
Figure 10 (a) shows the exact solution of the PDE and the absolute error of the best models trained using (b) PINN and (c) CPINN, well example of the discriminator output (d).
w(x,6 () (Exact)
PP
(PINN) 10
IP (CPINN)
D: (Discriminator)
-0.5
0,9
0.5
10-1
10
10
10
10
100000
1.0
0.5
0.0
-1
0
Z
x
(a)
(d)
Figure 10: (a) Exact solution u to eq. (23), absolute errors of (b) PINN Adam after 1.4 x 106 training iterations and (c) CPINN ACGD after 28 000 training iterations, and (d) the discriminator:
A.5 ALLEN-CAHN EQUATION
For the Allen Cahn equation, we randomly select 10 000 training points within the domain for the PDE constraint in eq. (25), 100 and 256 points for the initial condition and boundary condition
15

---

Published as a conference paper at ICLR 2023
specified in eq. (26), respectively. The learning rates of Adam and ACGD are both 10-3, the B of Adam and ACGD are (0.99, 0.99), and the e are 10 100 respectively.
The PINN model contains 4 hidden layers with 128 neurons per layer for the adaptive sampling method experiments. The discriminator in the CPINN contains 4 hidden layers with 256 neurons per layer. GMRES was used as the inner iterative solver in ACGD, with relative tolerance of 10-² and an absolute tolerance of 10-20. 20
For the curriculum learning approach, divide the 10 000 points into 10 subsets based the time coordinate. Once the LPINN on the current training set is less than 10-?, we include the next subset of collocation points in the training.
Figure 11 (a) shows the exact solution of the PDE and the absolute error of the best models trained using (b) PINN and (c) CPINN, well an example of the discriminator output (d).
utar, f) (Exact)
|P a (PINN)
PP M (CPINN)
D3 (Discriminaturr)
10-3
10
5000
1.0
0.5-
0.0
(a)
(b)
(c)
(d)
Figure 11: (a) Exact solution eq. (25), absolute errors of (b) PINN Adam after 1.4 x 106 training iterations and (c) CPINN + ACGD after 13 250 training iterations, and (d) the discriminator.
A.6 OTHER OPTIMIZERS
In addition to ACGD and Adam, we also use SGD, CGD (Schäfer & Anandkumar, 2019), ExtraAdam and ExtraSGD (Gidel et al., 2019) to approximate the solution of equation 6 in section 3.6. We test several hyperparameter combinations for each optimizer. The learning rates vary included 10-1,5 x 10-4, 10-3 10-2, and 2 x 10-2. The ,3 of Adam and ACGD are (0.99, 0.99). We tested several 3 values for Adam and ACGD but did not observe meaningful changes to our results or conclusions when varying them. For the ExtraAdam optimizer, from 0.3, 0.5, 0.7.0.7.. 0.9, 0.99, though none provide accuracy competitive with the other optimizers. For CGD, SGD and ExtraSGD, the learning rates are 10-2 The CGD optimizer uses conjugate gradient as the iterative solver with relative tolerance 10 12 and absolute tolerance 10-20.
tested several pairs of 3 values,
We tried several different setups for the Weak Adversarial Network experiments, including using fixed set of training points or randomizing the training set per epoch, removing or preserving the logarithm in PINN loss, and different combinations of activation functions. We present the results of the best models optimized with either GACGD or the combination of Adam and AdaGrad in fig, 6.
In one WAN experiment we used Adam for the optimization of PINN with learning rate of 10-3 and (0.9, 0.999) as 3 values, AdaGrad for the optimization of the discriminator, with a learning rate of 1.5 x 10-3, We resample 5 000 interior points in the domain and 200 boundary points per training epoch. The PINN model contains 6 hidden layers with 40 neurons per layer, with softplus as the activation functions for layers 1, 2. and 4, sin as the activation function for layers 3 and 5, and no activation in the last layer. The discriminator, or network á, contains 8 hidden layers with 40 neurons per layer. The activation functions are set tanh for layers 1, softplus for layers 2, 4, and 6, sin for layers 3, 5, and 7, and no activation in the last layer. All model and optimizer settings in the first experiment are identical to the setup used in section 4.2.2 of Zang et al. (2020). We use the training procedure described in algorithm 3 of Zang et al. (2020) and removed the logarithm operation in the loss of PINN as mentioned in section 3.5 of Zang et al. (2020).
In the second WAN experiment, we used ACGD for the optimization, with learning rate of 10-3 and 3 of (0.99, 0.99). The iterative linear solve of the ACGD optimizer had a relative tolerance of 10-7 and an absolute tolerance of 10-20 We used fixed training with 000 interior points in
16

---

Published as a conference paper at ICLR 2023
the domain and 200 boundary points during the training. The PINN model contains hidden layers with 40 neurons per layer, with softplus as the activation functions for layers 1, 2, 4 and 6, sin as the activation function for layers 3, 5 and 7, and
activation in the last layer. The discriminator, or
network 6 contains 8 hidden layers with 40 neurons per layer. The activation functions are set to tanh for layers softplus for layers 2, 4, and 6, sin for layers 3, and 7, and no activation in the last layer. We use the training procedure as described by algorithm 1 of Zang et al. (2020) and removed the logarithm operation the loss of PINN mentioned in section 3.5 of Zang et al. (2020).
A.7 RELATIONSHIP WITH SOBOLEV ACCELERATION
We believe that the improvements due to Sobolev training observed by Son et al. (2021), as well as the theoretical analysis by Lu et al. (2022), can be reconciled with the perspective of section 2.3.
Lu et al. (2022) model PINN-training as kernel method with kernel that is jointly diagonalizable with the differential operator A, such that the former has polynomially increasing eigenvalues while the latter has polynomially decreasing eigenvalues. Thus, the kernel matrix and the associated feature map S behave like a power of the inverse of the differential operator A. After cancellation, the AA
Hessian S* AS behaves either like a power of A or power of its inverse. The reasoning of Lu et al. (2022) is that the additional differential operators due to Sobolev training cancel with excess powers of A 41 thus improving the conditioning. For instance. Sobolev training may improve the conditioning by replacing the Hessian S A' with S* A"AS In contrast, CPINNs replace the Hessian S*AA*AA with R AS, where R the feature map of the discriminator network. Under the assumption that R S A A-2/3, Sobolev training leads well-conditioned system while CPINNs leads to a worse condition number than PINNs. However, A ~ R. ~ S is a strong assumption that is unlikely hold exactly. Absent this assumption, or it only holds approximately, AS remains ill-conditioned in a way that can not be fully mitigated by multiplying with powers of A. While may also be ill- ill-conditioned, its large and small eigenvalues are not perfectly aligned with those of AS, leading to possibly significantly better conditioning of R' AS compared to S AS. For instance, if a single large eigenvalue of A is not matched by a small eigenvalue of S, it will appear squared in S A AS. However, it could be attenuated by a small eigenvalue of R. in R' AS.
Sobolev training furthermore assumes regularity of the PDE residual. Thus, it may bias the network to more regular solutions, thus improving generalization in sufficiently regular problems. This may explain the improved test errors observed by Son al. (2021) and Lu et al. (2022).
17


Reviews:
[{"review_id": "_HMPl_5FMF", "reviewer": "Reviewer_aueW", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents an adversarial approach that overcomes this limitation, called competitive PINNs (CPINNs). CPINNs train a discriminator that is rewarded for predicting mistakes the PINN makes. The discriminator and PINN participate in a zero-sum game with the exact PDE solution as an optimal strategy. This approach avoids squaring the large condition numbers of PDE discretizations, which is the likely reason for failures of previous attempts to decrease PINN errors even on benign problems. The numerical results show the accuracy. \n\n", "summary_of_the_review": "Based on the weakness, I reject the paper. ", "strength_and_weaknesses": "Strength:  By introducing another neural network as the point weight function and following with a min-max optimization, the solution of the PDEs can be more accurate. \n\nWeakness: Introducing a new neural network besides the solution neural network means much more complex for training and becomes much tricker on different tasks. And the weakness is summarized as follows.\n1. there is one similar paper in 2020 as McClenny, Levi, and Ulisses Braga-Neto. \"Self-adaptive physics-informed neural networks using a soft attention mechanism.\" arXiv preprint arXiv:2009.04544 (2020). In that work, they use mask functions for the point weight, and there is no significant difference using a neural network in this work, and the min-max procedure is the same; thus, the novelty is not much. The story is under the structure of the games but still can not change the reality of using a neural network for the point weight adversarial update. No intrinsic difference.\n2. The analysis in section 2.3 is not very firm and too simple, sometimes naive, and the analysis of the neural network is just for a linear mapping which should be at least a two-layer neural network if you really want to explain something. As far as I know, in traditional numerical PDEs, the linear combination of the basis is, of course, a very common and efficient way. And it works well even using the vanilla schemes, such as the finite element method. Further, what really makes the so-called neural PDE different is the approximation by nonlinear neural networks, and it is far not enough to use one linear combination to analyze the neural PDE. There is plenty of work for analyzing PINNs, a direct search may help your analysis.\n3. The results of the experiments are good but also tricky as a new neural network besides the solution neural network is introduced. More neural networks with a min-max optimization will increase the difficulties in training, and it is already complex compared with the traditional numerical solvers. If you can prove in high dimensional problem, the method still achieves good performance, just like Bao et al. in the weak adversarial neural network (Zang, Yaohua, et al. \"Weak adversarial networks for high-dimensional partial differential equations.\" Journal of Computational Physics 411 (2020): 109409.), it is much better. Further, in the experiments, such as nearly all of the l2 error figures, the L2 relative error curve with respect to the iteration is not stable, but the experiments are just stopped at best. See Figure 4; there is still a very large oscillation. And Figure 5 has a big shock and etc. Also, it is hard to determine which one enhanced the performance, the optimizer or the structure, see Table 1. \n4. The terms are a little confusing for those who are not familiar with neural PDEs and may lead misunderstandings. The work is very direct with adversarial updating point weight in the area of Neural PDE, but the terms, such as bets, game, etc., will confuse the readers. A direct statement would make things much easier.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "QRbjzpZvHr", "reviewer": "Reviewer_QL5E", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduce competitive physics-informed networks where two neural networks solve a partial differential equation by playing a zero-sum game. The formulation is built using a weak form. The authors claim that this weak form have smaller condition number. ", "summary_of_the_review": "The experiments look strong, and the writing is super clear to understand. Several claims are hand waving, and the reviewer is not convinced. I think if the paper is accepted at this stage, it will make confusion in the community. \n\nI still consider this method have made a huge contribution to the PINN family. I suggest authors to investigate paper [5] in detail to see the benefit of this method.", "strength_and_weaknesses": "Strength: The experiments look strong, and the writing is super clear to understand.\n\nWeakness: Several claims are hand waving, and the reviewer is not convinced. I think if the paper is accepted at this stage, it will make confusion in the community. \n\n**Regards Experiment** The reviewer's first major concern is the relationship of this paper with [1] which also uses the weak formulation to train a PDE solution.  I think an experiment comparing these works is essential (using the same optimization algorithm).\n\nThe paper only includes results comparison with respect to the number of iterations. Comparison with respect to the computation time is also needed. For CPINN introduces a new NN and CGD have a matrix inverse. Further computation cost have been included.\n\n**Regards Mni-max** min-max optimization is hard. The complexity lower bound proposed of minimax optimization has a term K_x K_y [2]. I guess using min-max has the same computational complexity as the original formulation.\n\nTo reduce the condition number of laplace squares, we can still use PINN to do this by considering\nmin_{u_1,u} ||u_1-\\nabla u||^2+||f-\\nabla dot u_1||^2\nto solve Delta u = f. I don't think the minimax formulation is essential.\n\nAt the same time, the paper uses ADAM, which is an adaptive method. The adaptive methods can cancel the condition number. I'm confused by this mismatch of the experiment with the theory claimed. \n\n**Regards the Squared residuals.** Changing the loss into the Sobolev norm also introduces a further matrix A. But [3,4] showed this would only accelerate the training both in experiments and theory for the machine learning algorithms. This paper have considered the theory as [4] includes a machine learning model into the learning objective. Can the authors comment on this?\n\n**Related work [5]** The reviewer thinks the paper is still valuable. [5] proposed a similar trick from the optimization community and figured out the improvement of this method.  I encourage the authors to think about combining the theory in [4] and [5] to figure out the true reason this method has an acceleration. \n\n[1] Yaohua Zang, Gang Bao, Xiaojing Ye, and Haomin Zhou. Weak adversarial networks for high-dimensional partial differential equations. Journal of Computational Physics, 411:109409, 2020\n\n[2] Zhang J, Hong M, Zhang S. On lower iteration complexity bounds for the convex-concave saddle point problems. Mathematical Programming, 2022, 194(1): 901-935.\n\n[3] Son H, Jang J W, Han W J, et al. Sobolev training for the neural network solutions of pdes[J]. arXiv preprint arXiv:2101.08932, 2021.\n\n[4] Lu Y, Blanchet J, Ying L. Sobolev Acceleration and Statistical Optimality for Learning Elliptic Equations via Gradient Descent[J]. arXiv preprint arXiv:2205.07331, 2022.\n\n[5] https://proceedings.neurips.cc/paper/2018/file/08048a9c5630ccb67789a198f35d30ec-Paper.pdf \n\n\n===\n\nThe author's response convinced me this is a worth pursuing direction, I've raised my score. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "CVFx99xcQk", "reviewer": "Reviewer_1E2V", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a new model called CPINN, which is an agent based approach to the machine learning solution of PDEs. CPINN is an adversarial approach where the discriminator and PINN, the existing approach to solve PDEs using neural networks, plays a game in order to improve the solution precision of PDEs. The authors motivate their work by the fact that the existing PINNs approaches fail to generate precise solution. The authors show that the proposed model, CPINN, is well-behaved compared to the original PINN model, from the optimization/training perspective. The authors support their claim using multiple PDE problems.", "summary_of_the_review": "I enjoyed reading this paper. Physics informed machine learning is a very important and hot problem and still in its infancy. The authors efforts in improving the performance of existing models for this problem needs to be highly acknowledged and rewarded. The authors were able to motivate their approach and support that using experimental study. Overall, I believe this paper has a high quality and ready for publication. I should only add that I am not fully familiar with the state of the arts in this topic and assumed that the authors reflected it well in the paper.  \n\nOne minor issue: I am curious to see similar plots to that of Figure 1 for other datasets.", "strength_and_weaknesses": "Strengths:\n1-\tNovelty \n2-\tWell-written\n3-\tExperimental results\n\nWeakness: \nCannot think of any\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "-sNAPTWPYh", "reviewer": "Reviewer_hCa9", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper produces more accurate physics-informed neural networks (PINNs) by including adversarial training through a discriminator which is rewarded for predicting PINN errors.", "summary_of_the_review": "This paper seems to be a solid step forward in the field of physics-based neural networks – a field that has high impact on a number of domains. The paper is generally well-written. The novelty of the work is middling (maybe too straightforward because of the recent popularity of GANs) but non-trivial. ", "strength_and_weaknesses": "Strengths:\n    • Claims are supported by reported results.\n    • Concept is sound and sensible.\n    • Making PINNs more accurate has good impact for the field.\n    • Paper is well-written and easy to read and navigate.\n    • Mathematics is well explained.\n    • Citations are thorough.\n    • Tradeoffs between this method and conventional PINNs are discussed.\n\nWeaknesses:\n    • Figures describing the method are not included, and they would make the method easier to understand in this case.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster