DEQGAN: Learning the Loss Function for PINNs with Generative Adversarial Networks
Authors: Blake Bullwinkel
Dylan Randle
Pavlos Protopapas
David Sondak
Venue: NeurIPS 2022 Submitted (2022)
Keywords: differential equations
generative adversarial networks
GANs
physics-informed neural networks
deep learning

Abstract: Solutions to differential equations are of significant scientific and engineering relevance. Physics-Informed Neural Networks (PINNs) have emerged as a promising method for solving differential equations, but they lack a theoretical justification for the use of any particular loss function. This work presents Differential Equation GAN (DEQGAN), a novel method for solving differential equations using generative adversarial networks to "learn the loss function" for optimizing the neural network. Presenting results on a suite of twelve ordinary and partial differential equations, including the nonlinear Burgers', Allen-Cahn, Hamilton, and modified Einstein's gravity equations, we show that DEQGAN can obtain multiple orders of magnitude lower mean squared errors than PINNs that use $L_2$, $L_1$, and Huber loss functions. We also show that DEQGAN achieves solution accuracies that are competitive with popular numerical methods. Finally, we present two methods to improve the robustness of DEQGAN to different hyperparameter settings.

DEQGAN: Learning the Loss Function for PINNs with Generative Adversarial Networks
Anonymous Author(s)
Affiliation
Address
email
Abstract
Solutions to differential equations are of significant scientific and engineering rele- vance. Doysics-Informed Neural Networks (PINNs) have emerged promising method for solving differential equations, but they lack a theoretical justification for the of any particular loss function. This work presents Differential Equation GAN (DEQGAN), a novel method for solving differential equations using gener- ative adversarial networks "learn the loss function"" for optimizing the neural network. Presenting results on suite of twelve ordinary and partial differential equations, including the nonlinear Burgers', Allen-Cahn, Cahn, Hamilton, and modified Einstein's 's gravity equations, we show that DEQGAN! can obtain multiple orders of magnitude lower mean squared errors than PINNS that use L2, L1, and Huber loss functions. We also show that DEQGAN achieves solution accuracies that are competitive with popular numerical methods. Finally, we present two methods to improve the robustness of DEQGAN to different hyperparameter settings.
10
11
12
Introduction
14
In fields such as physics, chemistry, biology, engineering, and economics, differential equations used to model important and complex phenomena. While numerical methods for solving differential equations perform well and the theory for their stability and convergence is well established, the has inspired rescarchers to apply neural networks to solving differential equations, which has given rise to the growing field of Physics-Infonn Informed Neural Networks (PINNs) [19 20
16
18 recent success of deep learning 3 10 17 29
40
47
52
53
20
35
36
50].
21 In contrast to traditional numerical methods. PINNs: provide solutions that are closed-form 30
48, provide a more accurate interpolation 23 scheme (30,, and can leverage transfer learning for fast discovery of new solutions [11 [3]. Further,
22 suffer less from the "curse of dimensionality" 16 20
24 PINNS do not require an underlying grid and offer a meshless approach to solving differential equations. This makes it possible to use trained neural networks, which typically have small memory
26
footprints, to generate solutions over arbitrary grids in single forward pass.
27 PINNs have been successfully applied to wide range of differential equations, but lack theoretical justification for the use of particular loss function from the standpoint of predictive performance. In domains outside of differential equations, data following known noise model (e.g. Gaussian) have clear justification for fitting models with specific loss functions (e.g. L2). the case of deterministic differential equations, however, there is no noise model and we lack an equivalent justification.
28
29
30
31
32
To address this gap in the theory, we propose generative adversarial networks (GANs) [14] for solving differential equations in a fully unsupervised manner. Recently, multiple works have shown that adaptively modifying the PINN loss function throughout training can lead to improved solution
33
34
We provide our PyTorch code at [link hidden preserve anonymity]
Submitted to 36th Conference on Neural Information Processing Systems (NeurIPS 2022). Do not distribute.

---

35 accuracies 37 577. The discriminator network of our GAN-based method, however, can be thought of as "learning the loss function" for optimizing the generator, thereby eliminating the need for an explicit loss function and providing even greater flexibility than an adaptive loss. Beyond the context of differential equations, it has also been shown that where classical loss functions struggle to capture complex spatio-temporal dependencies, GANs may be an effective alternative [3 26 31]
36
37
38
39
40
Our contributions in this work are summarized as follows:
41
We present Differential Equation GAN (DEQGAN), a novel method for solving differential equations in fully unsupervised manner using generative adversarial networks.
43
We highlight the advantage of "learning the loss function" with GAN rather than using pre-specified loss function by showing that PINNs trained using I22 I1, and Huber losses have variable performance and fail to solve the modified Einstein's 's gravity equations 77.
44
45
46
We present results on suite of twelve ordinary differential equations (ODEs) and partial differen- tial equations (PDEs), including highly nonlinear problems, showing that our method produces solutions with multiple orders of magnitude lower mean squared errors than PINNs that use
47
48
49
L2 L1, and Huber loss functions.
We show that DEQGAN achieves solution accuracies that are competitive with popular numerical methods, including the fourth-order Runge- Kutta and second-order finite difference methods.
51
52
We present two techniques to improve the training stability of DEQGAN that are applicable to other GAN-based methods and PINN approaches to solving differential equations.
53
54
2
Related Work
55 variety of neural network methods have been developed for solving differential equations. Some of these are supervised and learn the dynamics of real-world systems from data 4 9 15 44. Others are semi- supervisedd learning general solutions to a differential equation and extracting best fit solution based on observational data [41]. Our work falls under the category of unsupervised neural network methods, which are trained in a data-free manner that depends solely on the equation residuals.
56
57
58
59
Unsupervised neural networks have been applied to a wide range of ODEs [13 30 34 36 and PDEs
61
43 48 50], primarily use feed-forward architectures, and require the specification of a particular loss function computed over the equation residuals.
20
Goodfellow et al. [14 introduced the idea of learning generative models with neural networks and 63
64
an adversarial training algorithm, called generative adversarial networks (GANs). To solve issues of GAN training instability, Arjovsky et al. introduced a formulation of GANs based on the Wasserstein distance, and Gulrajani et al. [18) added a gradient penalty to approximately enforce a Lipschitz constraint on the discriminator. Miyato et al. [39] introduced an alternative method for enforcing the Lipschitz constraint with spectral normalization technique that outperforms the former method on some problems.
65
66
67
68
69
70
Further work has applied GANs to differential equations with solution data used for supervision. Yang et al. (56) apply GANs stochastic differential equations by using "snapshots" of ground-truth data for semi-supervised training. A project by students at Stanford [51] employed GANs to perform "turbulence enrichment" of solution data in manner akin to that of super-resolution for images proposed by Ledig et al. [32]. Our work distinguishes itself from other GAN-based approaches for
71
72
73
74
75 solving differential equations by being fully unsupervised, and removing the dependence on using
76
supervised training data (i.e. solutions of the equation).
77
3
Background
78
3.1
Unsupervised Neural Networks for Differential Equations
79
Early work by Dissanayake & Phan-Thien [2) proposed solving initial value problems in an unsuper- 80 vised manner with neural networks. In this work, we extend their approach to handle spatial domains and multidimensional problems. In particular, we consider general differential equations of the form
81
dv dv dt
F 1, (tt,
d12 AV. 2 A?W....
(I)
2

---

82 where (f, x) is the desired solution, dv/dt and / represent the first and second time 83 derivatives, A and u are the first and second spatial derivatives, and the system subject to
84 certain initial and boundary conditions. The learning problem can then be formulated as minimizing
85 the sum of squared residuals (i.e., the squared L2 loss) of the above equation
Vo(t, min Σ F (,,, Vo(,x) (Lx))D
dvo Vvo dt dt
AVO, A Ao.o....
86
where Ve is neural network parameterized by 0, D is the domain of the problem, and derivatives are computed with automatic differentiation. This allows backpropagation [22 to be used to train the neural network to satisfy the differential equation. We apply this formalism to both initial and
87
88
89
boundary value problems, including multidimensional problems, as detailed in Appendix A.2
90
3.2
Generative Adversarial Networks
91
Generative adversarial networks (GANs) 14] are generative models that use two neural networks to induce generative distribution p(z)) of the data by formulating the inference problem as a two-player, zero-sum game.
92
93
The generative model first samples latent random variable (O0 1), which is used as input into 95 the generator G (e.g., a neural network). A discriminator D is trained to classify whether its input was sampled from the generator (i.e. "fake") or from reference data set (i.e. "real").
97 Informally, the process of training GANs proceeds by optimizing a minimax objective over the
98 generator and discriminator such that the generator attempts to trick the discriminator to classify "fake" samples as "real". Formally, one optimizes
39
min max G) min max Ex-pon(s) [log D(x)] + (Ewp, log D(G()))
(3)
D
G
where I ~ Pdata (x) denotes samples from the empirical data distribution, and Ps ~ (0, 1) samples 101 latent space [14]. In practice, the optimization alternates between gradient ascent and descent steps 102 for D and G, respectively, Further details on training and architecture are provided in Appendix A.4
100
103
3.3
Guaranteeing Initial & Boundary Conditions
Lagaris al. (30 showed that it is possible to exactly satisfy initial and boundary conditions by adjusting the output of the neural network. For example, consider adjusting the neural network output To (t, x) satisfy the initial condition Va x Tg. We apply the re-parameterization
105
106
(t, 2o + (t,,))
(4)
which exactly satisfies the initial condition. Mattheakis al. parameterization
336 proposed an augmented re-
108
V (t, = (Me(t, x)) = xo x0 + (1 (1- (1- We (t,t,xx x)
(5)
that further improved training convergence. Intuitively, Equation 5 the output of the neural network (t, x) to be exactly when = to, and decays this constraint exponentially in t. Chen al. [8] provide re-pan parameterizations to satisfy a range of other conditions, including Dirichlet and Neumann boundary conditions, which we employ in our experiments and detail in Appendix A.2
109
110
111
4 Differential Equation GAN
118
114
In this section, we present our method, Differential Equation GAN (DEQGAN), which trains a GAN to solve differential equations in a fully unsupervised manner. To do this, rearrange the differential equation so that the left-hand side (LHS) contains all the terms which depend on the generator (e.g.
116
117 V. /i,, AV, etc.) and the right-hand side (RHS) contains only constants (e.g. zero).
118 During training, we sample points from the domain (1, x) D and use them as input to a generator
Gzr), which produces candidate solutions V.. We sample points from noisy grid that spans D,
3

---

RHS
V
V
LHS
Domain
G
S
F
D
(0, 1
V
00 m
log (1 D (LHSt)
1 V 9u
log D (RHS + log (1 - D (LHS
Figure I: Schematic representation of DEQGAN. We pass input points to generator G, which produces candidate solutions Va. Then we analytically adjust these solutions according and apply automatic differentiation to construct LHS from the differential equation F. RHS and LHS are passed discriminator D, which is trained to classify them as "real" and "fake," respectively.
120 which we found reduced interpolation error in comparison sampling points from fixed grid. We then adjust v for initial or boundary conditions obtain the re- re-parameterized output Vo construct the LHS from the differential equation F using automatic differentiation
121
122
di
d dt2
LHS F 1,x, No(,,,,
AF
A
Vo Vo....
dt
123 and set RHS to its appropriate value (in our examples, RHS 0). Training proceeds in manner
124 similar to traditional GANs. We update the weights of the generator G and the discriminator D
125
according to the gradients
log (1-D (LHS()))
9G
v 17
1=1
128
m
log D (RHS()) + log (1 (I-D(LHIS)))
9D
V 0a
m
where LHS()) the output of (z()) after adjusting for initial or boundary conditions and con- structing the LHS from F. Note that perform stochastic gradient descent for G' (gradient steps
127
129
x -G), and stochastic gradient ascent for (gradient steps & gD). We provide schematic
representation of DEQGAN in Figure I and detail the training steps in Algorithm
Algorithm DEQGAN
Input: Differential equation F, generator G(:;8,). discriminator (:: 0a). grid of m points with spacing Az, perturbation precision т. ce-parameterization function 4 total steps N, learning rates GG TJD Adam optimizer [27] parameters BG1, 8G2, 3D1, BD2
for i 1 to N do
for = to m do
Perturb j-th point in mesh a = xl1) + E,E ~ NoO, Az)
Forward pass V, = G(r)
Analytic re-parameterizatioo V = (VV)))
Compute 5H = F (1,x, lt x), di de
dwo
di.
A Awo AZ
Set RH =
end for
Compute gradients 9a.9n gn (Equation 7
Update generator 0g f- Adam(O,, 9G, TiG, 30r 9c22) Update discriminator 0a Adam (0, gn, un BDI 8m)) end for Output: G
131 Informally, our algorithm trains GAN by setting the "fake" component to the LHS (in our formulation, the residuals of the equation) and the "real" component to be the R.H. S of the equation.

---

133
This results in a GAN that learns to produce solutions that make LHS indistinguishable from RHS, thereby approximately solving the differential equation.
134
135
4.1
Instance Noise
136
While GANs have achieved state of the art results on a wide range of generative modeling tasks, they often difficult to train. As result, much recent work GANs has been dedicated to improving
their sensitivity to hyperparameters and training stability 2 18 25 28 38 39 44 [4]. In our experiments, found that DEQGAN could also be sensitive to hyperparameters, such as the Adam optimizer parameters shown in Algorithm T
138
139
140
Sønderby et al. [49] note that the convergence of GANs relies on the existence of unique optimal discriminator that separates the distribution of "fake" samples Plake produced by the generator, and the distribution of the "real" data Pdata- In practice, however, there may be many near-optimal discriminators that pass very different gradients to the generator, depending on their initialization. Arjovsky& & Bottou [1] proved that this problem will arise when there is insufficient overlap between the supports of Ptake and Päat- In the DEQGAN training algorithm, setting RHS = constrains Pda the Dirac delta function 8(0), and therefore the distribution of "real" data a zero-dimensional manifold. This makes it unlikely that Plake and pdam will share support in a high-dimensional space.
142
143
144
146
147
148
The solution proposed by [1 49 is to add "instance noise" to Prake and Pdata to encourage their overlap. This amounts to adding noise to the LHS and the RHS, respectively, at each iteration of Algorithm Because this makes the discriminator's job more difficult, we add Gaussian noise with standard deviation equal to the difference between the generator and discriminator losses, Lg and La, i.c.
150
T
152
E = OO), = ReLULLL La)
As the generator and discriminator reach equilibrium, Equation 9 will naturally converge to zero. We use the ReLU function because La Lg indicates that the discriminator is generally performing worse than the generator, suggesting that additional noise should not be used. In Section (5.2, we conduct an ablation study and find that this improves the ability of DEQGAN to produce accurate solutions across a range of hyperparameter settings.
153
155
157
158
4.2
Residual Monitoring
159
One of the attractive properties of Algorithm[]i iis that the "fake" LHS vector of equation residuals gives direct measure of solution quality at each training iteration. We observe that when DEQGAN training becomes unstable, the LHSS tends to oscillate wildly, while it decreases steadily throughout training for successful runs. By monitoring the Ln norm of the LHHS in the first 25% of training iterations, we are able to easily detect and terminate poor-performing runs if the variance of these values exceeds some threshold. We provide further details on this method Appendix A.. and experimentally demonstrate that it is able to distinguish between DEQGAN runs that end in high and low mean squared errors in Section 5.2
160
162
164
166
Experiments
167
168
We conducted experiments a suite of twelve differential equations (Table [], including highly nonlinear PDEs and systems of ODEs, comparing DEQGAN to classical unsupervised PINNs that use (squared) L2, L1. and Huber [24] loss functions. We also report results obtained by the fourth- order Kunge-Kuttaa (RK4) and second-order finite difference (FD) numerical methods for initial and boundary value problems, respectively. The numerical solutions were computed over meshes containing the same number of points that were used to train the neural network methods. Details for each experiment, including exact problem specifications and hyperparameters, are provided in Appendix A.2 and A.5
169
171
173
175
176
5.1
DEQGAN vs. Classical PINNS
177
We report the mean squared emrrr of the solution obtained by each method, computed against known solutions obtained either analytically or with high-quality numerical solvers 6. [54]. We added residual connections between neighboring layers of all models, applied spectral normalization
178
5

---

Table Summary of Experiments
Key
Equation
Class
Order
Linear
EXP
=(t) + x(1)
ODE
1st
Yes
SHO
H(t) + x(t) 0
ODE
2nd
Yes
NLO
(F)) + (()((t
+ xtt + (xx(t)² + tx((1)==
ODE
2nd
No
[a(t) -ty
COO
ODE
1st
Yes
(ú(() to
S(t) --I(C))(O)NN
SIR
I(t)
= -B(())(()/NN -
ODE
No
R(t)
= =T((t)
((()) Pæ
((t) Py
HAM
ODE
1st
No
Po(t) -V
(1) = =VV,
i(=)
(- 2v + 4y xu + x²)
241
5(2)
241
=1 (rzI(r) ry 4y 2yv)
EIN
ú(z)
=+1
(xI(r) + 2v)
-E
ODE
1st
No
((2)) -2(-1 2v x)
F(2)
-r((r)e
s+1
POS
Uaz + Byy
2r - (1 - 2x + xy + 2
PDE
2nd
Yes
HEA
dt Kllzx
PDE
2nd
Yes
WAV
Uut e Uzz
PDE
2nd
Yes
BUR
Me + Wtl Wize 0
PDE
2nd
No
ACA
Currr u u = 0
PDE
2nd
No
Table 2: Experimental Results
Mean Squared Error
Key
L1
L2
Huber
DEQGAN
Numerical
EXP
3-10-3
2- 10
1. 1-10-5
3- 10 16
2. 10 (RK4)
SHO
9--0
1010-10
6 6-10-1
4-10 4-10-13
1.10-1 (RK4)
NLO
6.10-2
1-10-9
90-0-10
1-10-12
4.10-11 (RK4)
COO
5-10-1
1.10-7
1.10-7
1.10-8
2-10-9 (RK4)
SIR
7-10-5
3-10-9
1.10-9
1-10-10
10 (RK4)
HAM
1.10-1
2.10-7
9.10-8
1.10-10
7. (RK4)
EIN
6 10-2
2010-2
1.10-2
3.10-1
4.10-7 (RK4)
POS
1.10-10
6.10-11
4.10-13
10-1 (FD)
HEA
6 10
3-10-
1.10-
6-10-10
4-10-7 (FD)
WAV
6. 10
4.10
6.10-
1.10
7.10 (FD)
BUR
40
2.10-
1.10
4.10
1.10-3 (FD)
ACA
6 10
9 10103
4.10-3
3-
2 2-10-4 (FD)
to the discriminator, added instance noise the Piake and Preal- and used residual monitoring to terminate poor-performing runs in the first 25% of training iterations. Results were obtained with hy perparameeeess tuned for DEQGAN, In Appendix A.6 tuned each classical PINN method for comparison, but did not observe a significant difference.
181
183
184
Trepotss the lowest mean squared error obtained by each method across ten different model weight initializations. We see that DEQGAN obtains lower mean squared errors than classical PINNs that use L2, L1, and Huber loss functions for all twelve problems, often by several orders of magnitude. DEQGAN also achieves solution accuracies that are competitive with the RK4 and FD) numerical methods.
185
187

---

10
10-1
Meen squared enror
10
10
Mean squared
10-4
10"6.
GAN
GAN
10-7
L
Hubse
10-9
Hubas
2000
4000
0000
8000
10000
2000
4000
60000 Iteration
8000
10000
12000
Iteration
(a) Nonlinear Oscillator (NLO)
(b) Hamilton System (HAM)
100
Mean squared error
10-2
10
Mean squuare
10-3
20
10
10-5
In
In
40000
2500
Iteration
Iteration
(c) Wave Equation (WAV)
(d) Burgers' Equation (BUR)
10
Mean squared ervor
10-1.
10-1
10
1g
10-2
10"9.
2000
4000
0000
8000
10000
10000
2000
3000
.00000
5000
(e) Allen-Cahn Equation (ACA)
(f) Modified Einstein's 's Gravity System (EIN)
Figure 2: Mean squared errors vs. iteration for DEQGAN, L2, L1, and Huber loss for six equations. We perform ten randomized trials and plot the median (bold) and (25, 75) percentile range (shaded). We smooth the values using simple moving average with window size 50.
Figure the mean squared error vs. training iteration for six challenging equations and highlights multiple advantages of using DEQGAN over pre-specified loss function (equivalent plots for the other six problems are provided in Appendix A.3). In particular, there is considerable variation in the quality of the solutions obtained by the classical PINNs. For example, while Huber performs better than T-2 on the Allen-Cahn PDE, it is outperformed by T.2 on the wave equation. Furthermore, Figure 2i shows that the L2, L1 and Huber losses all fail to converge to accurate solution to the modified Einstein's 's gravity equations. Although this system has previously been solved using PINNs, custom loss function that incorporated equation-specific parameters (7).
189
191
193
195
the networks relied DEQGAN, however, is able to automatically learn a loss function that optimizes the generator to produce accurate solutions. DEQGAN solutions to four example equations visualized in Figure which shows that the ODE solutions are indistinguishable from those obtained using a numerical integrator. Similar plots for the other experiments are provided in Appendix A.2
197
199
201
5.2
DEQGAN Training Stability: Ablation Study
202
In our experiments, we used instance noise to adaptively improve the training convergence of DEQGAN and employed residual monitoring terminate poor- nor-peefooning runs carly. To quantify
7

---

204
the increased robustness offered by these techniques, we performed an ablation study comparing the percentage of high MSE (2 10 10-5) runs obtained by 500 randomized DEQGAN runs on the exponential decay equation.
205
Figure plots the results of these 500 DEQGAN experiments with instance noise added. For each experiment, we uniformly selected random seed controlling model weight initialization as an integer from the range [0, 9), well as separate learning rates for the discriminator and generator in the range [0.01, 0.1]. We then recorded the final mean squared error after running DEQGAN training for 1000 iterations. The red lines represent runs which would be terminated early by our residual monitoring method, while the blue lines represent those which would be run to completion. We see
208
209
210
211
212
213
that the large majority of hyperparameter settings tested with the addition of instance noise resulted in low mean squared errors. Further, residual monitoring was able to detect all runs with MSE 2 10
214
Approximately half of the MSE runs in [10-8, 10-5 would be terminated, while 96% of runs with
215
MSE < 10-8 would be run to completion.
Model Seed
Learning Rate
Learning Rate
Log MSE
0.1
0.1
0/9
0.08
dos
.3
0.07
0.07
D:05
0.05
11755
1105
10:
11.4
1104
000
0.0
0.02
0.02
0.01
0.01
Figure 3: Parallel plot showing the results of 500 DEQGAN experiments on the exponential decay equation with instance noise. The red lines represent runs which would be terminated early by monitoring the variance of the equation residuals in the first 25% of training iterations. The mean squared error is plotted logio scale.
Table 3: Ablation Study Results
Runs with High MSE (2 10 10-5)
No Residual Monitoring
With Residual Monitoring
No Instance Noise
12.4
0.4
With Instance Noise
8.0
0.0
217
Table 3 compares the percentage of high MSE runs with and without instance noise and residual monitoring. We see that adding instance noise decreased the percentage of runs with high MSE and that residual monitoring is highly effective at filtering out poor performing runs. When used together, these techniques eliminated all runs with MSE > 10-5 These results agree with previous works, which have found that instance noise can improve the convergence of other GAN training algorithms 49]. Further, they suggest that residual monitoring provides a useful performance metric that
218
220
222
could be applied to other PINN methods for solving differential equations.

---

1.0
0.5
1.0
An
0.6
0.3
0.5
0.0
0.0
00.
00.5
-10
-0.50
0.25
0.00
0,25
0.50
(a) Damped Nonlinear Oscillator (NLO)
Coupled Oscillators (COO)
2.5
1e+000
1e-00
9e-01
9c-01
8e-01
6e-01
1.5
6c-01
3e-01
+0
1.0
5c-01
2
-3e-01
3c-01
-6e011
2c-01
9e:01
0.0
0e++00
1e++00
-5.0
-2.5
0.0
2.5
3.0
0
(c) Burgers' Equation (BUR)
(d) Allen-Call Clhn Equation (ACA)
Figure 4: Visualization DEQGAN solutions four equations. The top left figure plots the phase space of the DEQGAN solutions (solid color lines) obtained for three initial conditions the NL.O problem, which is solved as a second-order ODE, and known solutions computed by numerical integrator (dashed black lines). The figure to the right plots the DEQGAN solution to the COO problem, which solved as a system two first-order ODEs. The second row shows contour plots of the solutions obtained by DEQGAN on the BUR and ACA problems, both nonlinear PDEs.
Conclusion
224
225
PINNs offer a promising approach to solving differential equations and to applying deep learning methods to challenging problems in science and engineering. Classical PINNs, however, lack
226
227
theoretical justification for the of any particular loss function. In this work, we presented
228
Differential Equation GAN (DEQGAN), a novel method that leverages GAN-based adversarial training "learn" the loss function for solving differential equations with PINNs. We demonstrated the advantage of this approach in comparison to using classical PINNs with pre-specified loss functions, which showed varied performance and failed converge an accurate solution the modified Einstein's 's gravity equations. In general, we demonstrated that our method can obtain multiple orders of magnitude lower mean squared errors than PINNs that use I.2, L and Huber loss functions, including on highly nonlinear PDEs and systems of ODEs. Further, we showed that DEQGAN achieves solution accuracies that are competitive with the fourth-order Runge Kutta and second-order finite difference numerical methods. Finally, we found that instance noise improved training stability and that residual monitoring provides useful performance metric for PINNs. While good measure of solution quality, PINNs lack the error bounds enjoyed by numerical methods. Formalizing these bounds is an interesting avenue for future work and would enable PINNs to be more safcly deployed in real-world applications. Further, while results evidence the advantage "learning the loss function" with a GAN, understanding exactly what the discriminator learns is open problem. Post-hoc explainability methods, for example, might provide
229
231
233
235
237
the equation residuals
239
241
243
useful tools for characterizing the differences between classical losses and the loss functions learned by DEOGAN, which could deepen understanding of PINN optimization more generally.

---

References
245
246
[1] Arjovsky, M. & Bottou, L. (2017). Towards principled methods for training generative adversarial networks. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings: OpenReview.net.
248
249
[2] Arjovsky, M., Chintala, S., & Bottou, I., (2017). Wasserstein gan.
250
[3] Bahdanau, D., Cho, K., & Bengio, Y. (2015). Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA. May 7-9, 2015, Conference Track Proceedings.
251
252
[4] Bertalan, T., Dietrich, F., Mezić I., & Kevrekidis, I. G. (2019). On learning hamiltonian systems from data. Chaos: An Interdisciplinary Journal of Nonlineur Science, 29(12). 121107.
254
255
[]] Berthelot, D., Schumm, T., & Metz, L. (2017). BEGAN: boundary equilibrium generative adversarial networks. CoRR, abs/1 703.10717.
[6] Brunton, S. L., & Kutz, J. N. (2019). Data-Driven Science and Engineering: Machine Learning, Dynamical Systems, and Control. Cambridge University Press.
258
259
[7] Chantada, A. T., Landau, S. J., Protopapas, P., Scóccola, C. G., & Garraffo, C. (2022). Cosmo- logical informed neural networks to solve the background dynamics of the universe.
280
261
[8] Chen, F., Sondak, D., Protopapas, P., Mattheakis, M., Liu, S., Agarwal, D., & Di Giovanni, M. (2020). Neurodiffeq; A python package for solving differential equations with neural networks. Journal of Open Source Software, 5(46), 1931.
263
264
[9] Choudhary, A., Lindner, J., Holliday, E., Miller, S., Sinha, S., & Ditto, W. (2020). Physics- enhanced neural networks learn order and chaos. Physical Review E, 101.
[10] Dabney, W., Rowland, M., Bellemare, M. G., & Munos, R. (2018). Distributional reinforcement learning with quantile regression. In Thirty-Second AAAI Conference on Artificial Intelligence.
287
288
[1I] Desai, S., Mattheakis, M., Joy, H., Protopapas. P., & Roberts, S. (2021). One-shot transfer learning of tenysics-informed neural networks.
289
270
[12] Dissanayake, M. & Phan-Thien, N. (1994). Neural-networr-bbsse approximations for solving partial differential equations. Communications in Numerical Methods in Engineering, 10(3), 195-201.
272
273
[13] Flamant, C., Protopapas, P., & Sondak, D. (2020). Solving differential equations using neural network solution bundles.
275
[14] Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., & Bengio, Y. (2014). Generative adversarial networks.
278
277
[15] Greydanus, S., Dzamba, M., & Yosinski, J. (2019). Hamiltonian neural networks.
[16] Grohs, P., Hornung, F., Jentzen, A., & von Wurstemberger, P. (2018). A proof that artificial neural networks overcome the curse of dimensionality in the numerical approximation of black- scholes partial differential equations.
279
[17] Gu, S., Holly, E., Lillicrap. T., & Levine, S. (2017). Deep reinforcement learning for robotic manipulation with asynchronous off- polipy updates. In 2017 IEEE international conference on robotics and automation (ICRA) (pp. (3899-3396). IEEE.
282
284
[18] Gulrajani, I., Ahmed, F,, Arjovsky, M., Dumoulin, V. Courville, A. (2017). Improved training of wasserstein gans.
285
286
[19] Hagge, T., Stinis, P., Yeung, E., & Tartakovsky, A. M. (2017). Solving differential equations with unknown constitutive relations as recurrent neural networks.
287
288
[20] Han, J., Jentzen, A., & F, W. (2018). Solving high-dimensional partial differential equations using deep learning. Proceedings of the National Academy of Sciences, 115(34), 8505-8510.
10

---

290
[21] He, K., Zhang, X., Ren, S., & Sun, J. (2015). Deep residual learning for image recognition. CoRR, 1b2./333..
291
292
[22] Hecht-Nielsen, R. (1992). Theory of the backpropagation neural network. In Neural networks for perception (pp. 65 93). Elsevier.
293
284
[23] Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Klambauer, G., & Hochreiter, S. (2017). Gans trained by time-scale update rule converge nash equilibrium. CoRR,
285
296
[24] Huber, P. J. (1964). Robust estimation of location parameter. Ann. Math. Statist,, 35(1),
298
73-101.
[25] Karnewar, A., Wang, O., & Iyengar, R. S. (2019). MSG-GAN: multi-scale gradient GAN for stable image synthesis. CoRR, abss/1903.0644..
300
[26] Karras, T., Laine, S., & Aila, T. (2018). A style-based generator architecture for generative adversarial networks. CoRR, Abs./18...0494..
302
303
[27] Kingma, D. P. & Ba, J. (2014). Adam: A method for stochastic optimization. cite arxiv: 1...14442.9000mmment Published as a conference paper at the 3rd International Conference for Learning Representations, San Diego, 2015.
304
306
[28] Kodali, N., Abernethy, J. D., Hays, J., & Kira, (2017). How train your DRAGAN. CoRR, abs//705.nn 155500055...
307
308
[29] Krizhevsky, A., Sutskever,
Hinton, G. E. (2012). Imagenet classification with deep convolutional neural networks. In F. Pereira, J. C. Burges, L. Bottou, & K. Q. Weinberger (Eds.), Advances in Neural Information Processing Systems 25 (pp. 1097-1105). Curran Associates, Inc.
309
310
312
[30] Lagaris, I., Likas, A., & Fotiadis, D. (1998). Artificial neural networks for solving ordinary and partial differential equations. IEEE Transactions on Neural Networks, 9(5), 987-1000.
313
[31] Larsen, A. B. L.,, Sønderby, S. K., & Winther, O. (2015). Autoencoding beyond pixels using learned similarity metric. CoRR, abs/1512.093000.
315
[32] Ledig, C., Theis, L., Huszar, F., Caballero, J., Aitken, A. P., Tejani, A., Totz, J., Wang, Z., & Shi, W. (2016). Photo-realistic single image super-resolution using a generative adversarial network. CoRR, Eet1116090000000
317
319
[33] Liaw, R., Liang, E., Nishihara, R., Moritz, P., Gonzalez, E., Stoica, (2018). Tune: A research platform for distributed model selection and training. CoRR, abs/1807.0511
321
[34] Mattheakis, M., Joy, H., Protopapas, P. (2021). Unsupervised reservoir computing for solving ordinary differential equations.
323
[35] Mattheakis, M., Protopapas, P., Sondak, D., Giovanni, D., & Kaxiras, (2019). Physical symmetries embedded in neural networks.
324
325
[36] Mattheakis, M., Sondak, D., Dogra, A. S., & Protopapas, P. (2020). Hamiltonian neural networks for solving differential equations.
326
327
[37] McClenny, & Braga-Neto, U. (2020). Self-adaptive physics-informed neural networks using a soft attention mechanism.
328
[38] Mirza, M. & Osindero, S. (2014). Conditional generative adversarial nets.
330
[39] Miyato, T., Kataoka, T., Koyama, M., & Yoshida, Y. (2018). Spectral normalization for generative adversarial networks. CoRR, 852/1802.05957.
332
[40] Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., Riedmiller,
M. (2013). Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.560..

---

334
[41] Paticchio, A., Scarlatti, T., Mattheakis, M.. Protopapas, P., & Brambilla, M. (2020). Semi- supervised neural networks solve an inverse problem for modeling covid-19 spread.
335
336
[42] Piscopo, M. L., Spannowsky, M., & Waite, (2019). Solving differential equations with neural networks: Applications to the calculation of cosmological phase transitions. Phys. Rev. D, 100, 016002.
338
339
[43] Raissi, M. (2018). Forward-backward stochastic neural networks: Deep learning of high- dimensional partial differential equations. arXiv preprint
341
[44] Raissi, M.. Perdikaris. P., & Karniadakis, G. (2019). Physics-informi neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378, 686 707.
342
343
344 [45] Riess, A. G., Filippenko, A. V., Challis, P., Clocchiatti, A., Diereks, A., Garnavich, P. M., Gilliland, R. L.,, Hogan, C. J., Jha, S., Kirshner, R. P., Leibundgut, B., Phillips, M. M., Reiss, D., Schmidt, B. P., Schommer, R. A., Smith, R. C., Spyromilio, J., Stubbs, C., Suntzeff, N. B., & Tonry, J. (1998). Observational evidence from supernovae for accelerating universe and cosmological constant. The Astronomical Journal, 1 116(3), 1009-1038.
346
348
349
[46] Salimans, T., Goodfellow, I. J., Zaremba, W., Cheung. V., Radford, A., & Chen, X. (2016). Improved techniques for training gans. CoRR, abs/1606.034988.
350
[47] Silver, D., Hubert, T., Schritwieser,, J., Antonoglou, I., Lai, M., Guez, A., Lanetot, M., Sifre, L., Kumaran, D., Graepel, T., et al. (2018). A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362(6419), 1140-1144.
352
353
354
[88] Sirignano, & Spiliopoulos, K. (2018). Dgm: A deep learning algorithm for solving partial differential equations. Journal of Compulational Physics, 375, 1339-1364.
[49] Sønderby, C. K., Caballero, J., Theis, L., Shi, W., & Huszár, F, (2016). Amortised MAP inference for image super-resolution. CoRR, abs/1610.044490.
357
358
[50] Stevens, B. & Colonius, T. (2020). Finitenet: A fully convolutional Istm network architecture for time-dependent partial differential equations.
360
[51] Subramanian, A., Wong, M.-I.., Borker, R., & Nimmagadda, S. (2018). Turbulence enrichment using generative adversarial networks.
381
362
[52] Sutskever, I., Vinyals, O., & Le, Q. V. (2014). Sequence to sequence learning with neural networks. CoRR, abs/1409.3215.
383
364 [53] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017). Attention is all you need. CoRR, abs/1706.0376
[54] Virtanen, P., Gommers, R., Oliphant, T. E.,, Haberland, M., Reddy, T., Cournaeea,, D., Burovski, E.,, Peterson, P., Weckesser, W., Bright, J., van der Walt, S. J., Brett, M., Wilson, J., Jarrod Millman, K., Mayorov, N., Nelson, A. R. J., Jones, E., Kern, R., Larson, E., Carey, C.,, Polat, i., Feng, Y., Moore, E. W., Vand erPlas, J., Laxalde, D.. Perktold, J., Cimrman, R., Henriksen, I., Quintero, E. A., Harris, C. R., Archibald, A. M., Ribeiro, A. H., Pedregosa, F., van Mulbregt, P., & Contributors, S. (2020). SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods, 17, 261-272.
367
389
370
371
372
373
[55] Wang, C., Horby, P. W., Hayden, F. G., & Gao, G. F. (2020). novel coronavirus outbreak of global health concern. The Lancet, 395(10223), 470-473
374
[56] Yang, L., Zhang, D., & Karniadakis, G. E. (2018). Physics-informed generative adversarial networks for stochastic differential equations.
376
377
[57] Zeng. S., Zhang, Z., & Zou, Q. (2022). Adaptive deep neural networks methods for high- dimensional partial differential equations. Journal of Computational Physics, (pp. 111232)
378
12

---

Checklist
379
380
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's s contributions and scope? [Yes] Our claims are evidenced by the experimental results in
382
Section [
384
(b) Did you describe the limitations of your work? [Yes] We discussed limitations and directions for future work in Section
385
(c) Did you discuss any potential negative societal impacts of your work? [Yes] While our research is focused on the study of differential equations and does not hold particularly poignant ethical consequences, we discussed future research directions for ensuring that our method can safely be deployed in world applications in Section 6
387
389
390
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
381
2. If you are including theoretical results...
393
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
395
3. If you ran experiments..
396
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material as URL)? [Yes] See the footnote on page 1 (link is currently hidden to preserve anonymity).
398
389
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Appendix A.2 and [A.4
400
401
(c) Did you report error bars (c.g., with respect to the random seed after running experi-
ments multiple times)? [Yes] We conducted an ablation study that includes a sensitivity analysis of our method. See Appendix A.7.
403
404
(d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix A..4
405
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
407
(a) If your work uses existing assets, did you cite the creators? [N/A]
(b) Did you mention the license of the assets? [N/A]
409
(c) Did you include any new assets either in the supplemental material or as URL? [Yes]
410
See the footnote on page I (link is currently hidden to preserve anonymity).
(d) Did you discuss whether and how consent was obtained from people whose data you're
using/curating? [N/A]
412
413
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
416
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
417
418
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
421
13

---

A Appendix
422
423
A.1
Classical Loss Functions
A plot of the various classical loss functions is provided in Figure
2.0
In
1.5
Huber
1.0
0.5
0.0
Figure 5: Comparison of L2, L1, and Huber loss functions. The Huber loss is equal to L2 for e S
and to L1 for e >> 1.
425
A.2
Description of Experiments
426
A.2.1
Exponential Decay (EXP)
427
Consider a model for population decay x(t) given by the exponential differential equation
a(t) + xx()== =
(10)
428
with z(0) = and E [0. 10]. The ground truth solution xtt) = e- can be obtained analytically, which we use to calculate the mean squared error of the predicted solution.
430
To set up the problem DEQGAN, we define LHS + and RHS Figure presents the results from training DEQGAN this equation.
10
1.0
Mean Squared Errer
2.0
10-1
Loos
0.6
10-1-
0.5-
02-
10"
0
500
1000
500
1000
10
0
10
Iteration
Iteration
Figure 6: Visualization of DEQGAN training for the exponential decay problem. The left-most figure plots the mean squared error vs. iteration. To the right, we plot the value of the generator (G) and discriminator (D) losses at each iteration. Right of this we plot the prediction of the generator å and the true analytic solution functions of time The right-most figure plots the absolute value of the residual of the predicted solution P.
432 A.2.2
Simple Harmonic Oscillator (SHO)
Consider the motion of an oscillating body 2(t), which can be modeled by the simple harmonic oscillator differential equation
434
7(1) + x(()= 0,
(11)
435
with (0) = 0, .()) = 1, and t E [0, 2T. This differential equation can be solved analytically and has an exact solution s=() sin I.
437 Here we set T.HSS 3 + x and RHS 0. Figure plots the results of training DEQGAN on this problem.
14

---

toin
Moan Squared Error
10
10-1
1011_
Loss
0.0
10
10-1
-0.5
0.6
-1.
3000
10000
5000
10000
0.5
2.5
6.0
0.0
2.5
5.0
Iteration
Iteration
Figure 7: Visualization of DEQGAN training for the simple harmonic oscillator problem.
439
A.2.3
Damped Nonlinear Oscillator (NLO)
440
Further increasing the complexity of the differential equations being considered, consider a less idealized oscillating body subject to additional forces, whose motion x(t) we can described by the nonlinear oscillator differential equation
((() + 23²(1) + =1((()))))))))))))) dzz)) + ex(()3 =
(12)
with 3 = 0.1, = 1, -, x(1, = 0, =() 0.5, and t [0, 4x]. This equation does not admit an analytical solution. Instead, we use the high-quality solver provided by SciPy's solle_ivp 54
444
446
We set LHS # + 23r + wax + ox² + cx and RHS from training DEQGAN on this equation.
Figure plots the results obtained
447
0.1
10
Mcan Squared Error
vol
1.2
0.2
10
1.11
0.5
10.1
(U6
10-9-
11
5000
10000
01
5000
1000
01
10
10
lteration
Iteration
Figure 8: Visualization of DEQGAN training for the nonlinear oscillator problem.
A.2.4 Coupled Oscillators (COO)
449
Consider the system of ordinary differential equations given by
(+(())
ty L.
(13)
((())
450
with (0) = 1, y(0) = 0, and / E [0, 2m). This equation has an exact analytical solution given by
t
2 cos
2
(14)
12
y = sin
2
451 Here we set
dx
dy
T
LHS = +
ty,
Ty
(15)
dt
dt
and RHS [0, Figure 9 plots the result of training DEQGAN on this problem.
15

---

10
2.5
1.0
107
Moan Squared Erorr
10-1
val
2.0
0.5
1.5
4 10-1
0.0
1.0
10"1
-0.5
10"1
0.5
300000
50000)
0.0
2.5
6.0
0.0
2.5
5.0
Iteration
Iteration
Figure 9: Visualization of DEQGAN training for the coupled oscillators system of equations. In the third figure, plot the predictions of the generator é; ÿ and the analytic solutions æ, y as functions of time t. The right-most figure plots the absolute value of the residuals of the predicted solution Fj for each equation j.
453
A.2.5
SIR Epidemiological Model (SIR)
454
Given the ongoing pandemic of novel coronavirus (COVID-19) (55], we consider an epidemiological
model of infectious disease spread given by system of ordinary differential equations. Specifically,
456
consider the Susceptible Stt), Infected ((t), Recovered R(t) model for the spread an infectious
disease over time The model is defined by system of three ordinary differential equations
IS S(1) B
N i(() 3 IS N 71
(16)
R(() 7/
where 3 = 3, 7 1 given constants related to the infectiousness of the disease, N S+I+
459 the (constant) total population, -(0) = 0.99, I(0) = 0.01, R(0) = 0, and E [0, 10). As this system has analytical solution,
SciPy's sslllelll_____ ivp solver [54 to obtain ground truth solutions.
461
set LHS to the vector
ds dt
IS
IS N
dR dt
T
LHS =
+
+
TT
(17)
N
dt
462 and RHS = [0, 0, OT.
We present the results of training DEQGAN to solve this system of differential
483 equations in Figure 10
1.0
Moca Squared Eror
1.5
10
Loss
1.0
0.6
16
0-
0.4
-0-7
0.--
0.2
.0-5
10-
0.0
20000
Iteration
Iteration
Figure 10: Visualization of DEQGAN training for the SIR system of equations.
A.2.6 Hamiltonian System (HAM)
465
Consider particle moving through potential the trajectory of which is described by the system of ordinary differential equations
16

---

a(t) = Pr
i(()) = Py
(18)
(I) = - V
(t) = --V V
467 with z00) - O, v(0) -
pz - P. (0) - 0, and E [0, 1]. V2 and V, are the æ and derivatives of the potential V. which construct by summing ten random bivariate Gaussians
A
10
1
exp
11x(1 - mil1 20
(19)
2T
469 where x(t) (x(t),(()]] := 0.1, 0.1, and each p is sampled from [0, 1] [0, 1] uniformly at random. As before, we use SciPy to obtain ground-truth solutions.
We set LHS to be the vector
dr dt
dy
dps
dp + V
T
LHS
Ps:
Pyr
+ Vn
(20)
dt
472 and RHS [0, 0, 0, We present the results of training DEQGAN to solve this system of differential equations in Figure
10
0.40
1.10
0.15
Mon Squred Enorr
0.-
0.5
0.55
1.15
0.30
60110
32
0.33
0.55
0.06
12-4
0.1
0.30
0.00
01.00
5000
10000
0.0
1.0
00
0.5
0.0
0.5
1.0
0.0
0.5
1.0
Iberation
3.5
10-1
8
10
017
117
Figure 11: Visualization of DEQGAN training for the Hamiltonian system of equations. For ease of visualization, we plot the predictions and residuals for each equation separately.
474
A.2.7
Modified Einstein's Gravity System (EIN)
475
The most challenging system of ODEs we consider comes from Einstein's theory of general relativity. Following observations from type la supernovae in 1998 [45) several cosmological models have been proposed to explain the accelerated expansion the universe. Some of these rely on the existence of unobserved forms such dark energy and dark matter, while others directly modify Einstein's theory.
477
479
Hu-Sawicky (R) gravity is one model that falls under this category. Chantada et al. 7 show how the following system of five ODEs can be derived from the modified field equations implied by this model.
481
17

---

1
x(2)
(-S2 2u + x: + 4y + x²)
2 +
-1
((2) 2 +
(vxc((r) xy + 4y 2ye)
i(2) -
(xI((r + +4- 22))
(21)
Ω
((=)) 2 2+1
(-1+ + 2u xx)
-r)
f(z) = + 1
483
where
(r + b) [rr + 6)2 - 26]
[(r)
(22)
4br
The initial conditions are given by
I 0
om (
20) + 2(1 Qm.o)
yo 2m2 (1 + 2n)² + (1 2m,
am + 200 + 4(1 Pm,0))
4p = 2m 20)3 + (1 1mm
(23)
Sm + =0)3 Po = 0m, 20) 33 + (1 1m,
Dm.u(1 (1 + 20) ) + 4(1 Qm.o)
2m,0)
485 where 20 = 10, 2m = 0.15,6 and we solve the system for E [0, zol. While the physical interpretation of the various parameters is beyond the scope of this paper, we note that Equations 21
487
22d exhibii a high degree of non-linearity. Ground truth solutions are again obtained using SciPy.. Py, and the results obtained by DEQGAN are shown in Figure 12
8.2
-011
01
100
-0.1
N000
A1
x-"
15
10"
65
177
075
3/a
107
3101
Figure 12: Visualization of DEQGAN training the modified Einstein's gravity system of equations. For ease of visualization, we plot the predictions and residuals for each equation separately.
A.2.8 Poisson Equation (POS)
Consider the Poisson partial differential equation (PDE) given by
p.u
82u
+
2x(y - () 2r + ry + 2) "-
(24)
0yz
18

---

where (r, y) [0, 1] x [0, 1]. The equation is subject to Dirichlet boundary conditions on the edges 492 of the unit square
u(x, 5)
2=0
u(z, y)
(25)
uxx v)
0
y=
u(x, v) y=1
O.
The analytical solution is
u(t,,)) y) = xx1 - x)y((111111 - y)e""
(26)
484 We use the two-dimensional Dirichlet boundary adjustment formulae provided in Chen et al.
g To
495 set up the problem for DEQGAN we let
8 ax
8cu 2x(y 1) - 2x + ay + 2 Yy²
LHS
+
(27)
and RHS = O. We present the results of training DEQGAN this problem in Figure 13
10
tran
Moan Squared Error
10-4
0.75
0.6
1D--
4e-02
0.4
3e-02
0.4
2e-02
0.25
0.2
Ie-02
0.2
Se
0c+-00
0.0
0c+00
2000
2000
0.5
1.0
0.0
0.5
Iteration
Figure 13: Visualization of DEQGAN training for the Poisson equation. In the third figure, we plot the prediction of the generator û as a function of position (z, y). The right-most figure plots the absolute value of the residual F, as a function of 2).
487
A.2.9
Heat Equation (HEA)
We consider the time-dependent heat (diffusion) equation given by
Ou 22u Oz?
(28)
Øt
499 where x = and (x,4) [0, 0.2). The equation subject to an initial condition and Dirichlet boundary conditions given by
u( y)
sin (TT)
1=
u(x, v)
(29)
w=0
u(x, v) #=1
501 and has an analytical solution
u( y)
Nontt (sn((T))
(30)
The results obtained by DEQGAN on this problem shown in Figure 14
19

---

0.20
0.20
46-02
Mcan Squared Error
10-2
tra
9c-01
02
0.16
0.10
Se 02
Loss
01.8
0.10
2e-02
10-6
2w-02
0.06
0.6
0e=00
6e 03
0e+00
0
1000 2000
1000
2000
0.0
0.5
1.0
0.0
0.5
1.0
Iteration
Iteration
Figure 14: Visualization of DEQGAN training for the heat equation. In the third figure, we plot the prediction the generator à as function of position (xr,). The right-most figure plots the absolute value the residual F, as function of (ar, L)-
503
A.2.10
Wave Equation (WAV)
504
Consider the time-dependent wave equation given by
00u
(31)
012
Or²
where 1 and (z. [0. x 10. This formulation is very similar to the heat equation but involves a second order derivative with respect to time. We subject the equation to the same initial condition and boundary conditions as 29 but require added Neumann condition due to the equation's ' second time derivative.
506
508
u(x, v)
sin
1=0
ue (e 8)
1=U
(32)
u(x, y)
l==
u(x, v)
0
a==1
509
This yields the analytical solution
u(r y) = cos(ctt) sin(xx)..
(33)
510
The results of training DEQGAN on this problem are shown in Figure 14
10
1.11
1e:00
1.01
7e-02
Moan Squared Error
train
2,5
G
8--01
02
2.0
5e-01
Se 02
2-01
Lo05
1.5
0e+00
4e002
-2-01
0.4
3--02
Se- 01
2x00
02-
0.2
8e 01
0.
1e-00
0.
0c+00
10'0
2000 Iteration
Ineration
Figure 15: Visualization of DEQGAN training for the wave equation.
A.2.11 Bugers' Equation (BUR)
Moving to non-linear PDEs, consider the viscous Burgers' equation given by
Ou
Ou
82
(34)
0t Ox
20

---

518 where // = 0.001 and (z,t) E [-5,,]] x [0,.. To specify the equation, we use the following initial condition and Dirichlet boundary conditions:
uxx v)
cosh(2)
1==
ul (z, y)
0
(35)
u(3;, y)
m==
515 As this equation has no analytical solution, we use the fast Fourier transform (FFT) method 6 to obtain ground truth solutions. The results obtained by DEQGAN are summarized by Figure T6 As time progresses, we see the formation of a "shock wave" that becomes increasingly steep but remains smooth due the regularizing diffusive term Vugga
517
518
2.5
1e+00
2.5
6c--00
Moan Squared Error
2.0
9--01
-0-2
11.9
8s-01
4e:
Lo55
0.8
6e:01
3e+00
1.0
5e-01
2e-00
0.7
3e-01
2e: 80
0.5
0.5
0.6
0.0
De++00
0.0
0e-00 00
2003
Iteration 2 0,000
Iteration
22
- 0.833
1.007
-- 2.500
1.00
1,00
0.75
0.75
0.75
0.77
0.25
0.25
0.25
0.25
-3
-3
Figure 16: Visualization of DEQGAN training for Bugers' equation. The plots the second row show "snapshots" of the ID wave at different points along the time domain.
A.2.12
Allen-Cahn Equation (ACA)
520
Finally, we consider the Allen-Cahn PDE, well-known reaction-diiftsioon equation given by
Ou
82u u
(36)
Øt
dz
where e = 0.001 and (x. [0, 2m]
x [0, 5]. We subject the equation to an initial condition and
522
Dirichlet boundary conditions given by
u(x y)
sin(x)
(=U
u(z, y)
0
(37)
l==0
u(x 3)
0
523 The results are shown in Figure We see that time progresses, the sinusoidal initial condition transforms into square wave, becoming very steep at the turning points of the solution.
21

---

10P
0.8
Mean Squared
3
10m1
Los
0.0
10-2
0.4
3000
10000
3000
10000
0
0
Iteration
Iteration
(.000)
1.0
1.0
0.2
0.5
0.5-
1.0-
0.01
0.0
0.0
-0.5
-0.5
-1.0
2.5
5.0
5.0
0.0
0.0
2.5
Figure 17: Visualization of DEQGAN training for the Allen-Cahn equation. The plots in the second row show "snapshots" of the ID wave at different points along the time domain.
22

---

525
A.3
Method Comparison for Other Experiments
Figure 18
visualizes the training results achieved by DEQGAN and the alternative unsupervised neural networks that use L, L1 and Huber loss functions for the remaining six problems.
527
10
Moan squared error
6.
Mian squard
10
10-9
GAN
10
GAN
Le
In
10-15
10-12
Huber
Huber
3.0
400
(20) Iteration
8.0
1000
1200
0
2000
4000
(XXI) Iteration
NI00
10000
12000
(a) Exponential Decay (EXP)
(b) Simple Harmonic Oscillator (SHO)
10
10-1
10-2
Mean squared error
10-3
Mean squared
10-1
10-3
GAN
10-6
GAN
In
10-7
10
Huber
Huber
20000
40600
0000
5000
10003
15000
20000
lteration
Iteration
(c) Coupled Oscillators (COO)
(d) SIR Disease Model (SIR)
10"2.
Moan squared erter
10-0.
Mean squared
10-1
10-7
10-9
GAN
10-6.
GAN
Lo
10-11
10-4
Huber
Hubee
300
1000
1500
2003
2500
3000
500
1000
1500
2000
Iteration
Iteration
(e) Poisson Equation (POS)
(0) Heat Equation (HEA)
Figure 18: Mean squared errors vs. iteration for DEQGAN, L2. L1, and Huber loss for various equations. We perform ten randomized trials and plot the median (bold) and (25, 75) percentile range (shaded). We smooth the values using a simple moving average with window size 50.
528
A.4 DEQGAN Training and Architecture
A.4.1
Two Time-Scale Update Rule
530
Heusel et al. 23 proposed the two simelscale update rule (TTUR) for training GANs, method in which the discriminator and generator are trained with separate learning rates. They showed that their method led to improved performance and proved that, in some cases, TTUR ensures convergence to stable local Nash equilibrium. One intuition for TTUR comes from the potentially different loss surfaces of the discriminator and generator. Allowing learning rates to be tuned to particular loss surface can enable more efficient gradient-based optimization. We make use of TTUR throughout this paper instrumental lever when tuning GANs to reach desired performance.
531
532
535
535
23

---

537
A.4.2
Spectral Normalization
538
Proposed by Miyato et al. [39] Spectrally Normalized GAN (SN-GAN) is method for control- ling exploding discriminator gradients when optimizing Equation B3 that leverages a novel weight normalization technique. The key idea is to control the Lipschitz constant of the discriminator by
539
541
constraining spectral norm of each layer in the discriminator. Specifically, the authors propose dividing the weight matrices W; of each layer by their spectral norm oWW)
W WsN, = o(W.)
(38)
543 where
(W,)
max [A,12211
[Wh,
(39)
and h; denotes the input to layer i. The authors prove that this normalization technique bounds the Lipschitz constant of the discriminator above by thus strictly enforcing the 1-Lipshcitz constraint on the discriminator. In our experiments, adopting the SN-GAN formulation led to even better performance than WGAN-GP 18
545
547
A.4.3 Residual Connections
He al. [1 showed that the addition of residual connections improves deep neural network training. We employ residual connections in our networks, as they allow gradients to flow more easily through the models and thereby reduce numerical instability. Residual connections augment typical activation with the identity operation.
550
552
y= F(x, W.) +
(40)
where F is the activation function, is the input the unit, W. are the weights and y is the output of the unit. This acts as a "skip connection", allowing inputs and gradients to forego the nonlinear component.
554
A.5 DEQGAN Hyperparameters
557
We used Ray Tune 33 to tune DEQGAN hyperparameters for each differential equation. Tables and 5 summarize these hyperparameter values for the ODE and PDE problems, respectively. The experiments and hyperparameter tuning conducted for this research totaled 13,272 hours of compute performed Intel Cascade Lake CPU cores belonging internal cluster.
559
Table 4: Hyperparameter Settings for DEQGAN (ODEs)
HYPERPARAMETER
EXP
SHO
NLO
COO
SIR
HAM
EIN
NUM. ITERATIONS
1200
12000
12000
70000
20000
12500
50000
NUM. GRID POINTS
100
400
400
800
800
400
1000
G UNITS/LAYER
40
40
40
40
50
40
40
G NUM. LAYERS
2
3
4
5
4
4
D UNITS/LAYER
50
20
40
50
30
D NUM. LAYERS
4
3
2
2
4
2
ACTIVATIONS
tanh
tanh
tanh
tanh
tanh
tanh
tanh
G LEARNING RATE
0.094
0.005
0.010
0.004
0.006
0.017
0.011
D LEARNING RATE
0.012
0.0004
0.021
0.082
0.012
0.019
0.006
3, (ADAM)
0.491
0.363
0.225
0.603
0.278
0.252
0.202
G 3, (ADAM)
0.319
0.752
0.331
0.614
0.777
0.931
0.975
D 31 (ADAM)
0.542
0.584
0.362
0.412
0.018
0.105
0.154
D 32 (ADAM)
0.264
0.453
0.551
0.110
0.908
0.869
0.797
EXPONENTIAL LR DECAY (t)
0.978
0.980
0,999
0.992
0.9996
0.985
0.996
DECAY STEP SIZE
3
19
15
16
11
13
17
24

---

Table 5: Hyperparameter Settings for DEQGAN (PDEs)
HYPERPARAMETER
POS
HEA
WAV
BUR
ACA
NUM. ITERATIONS
2000
5000
3000
NUM. GRID POINTS
32 x 32
32 X
32
32 X 32
64 x 64
64 x 64
G UNITS/LAYER
50
40
50
50
50
G NUM. LAYERS
4
3
UNITS/LAYER
30
20
NUM. LAYERS
2
ACTIVATIONS
tanh
tanh
tanh
tanh
tanh
G' LEARNING RATE
0.019
0.010
0.012
0.012
0.020
D LEARNING RATE
0.021
0.001
0.088
0.005
0.013
G 9, (ADAM)
0.139
0.230
0.295
0.185
0.436
G 32 (ADAM)
0.369
0.657
0.358
0.594
0.910
D 81 (ADAM)
0.745
0.120
0.575
0.093
0.484
D 32 (ADAM)
0.759
0.251
0.133
0.184
0.297
EXPONENTIAL DECAY (n)
0.957
0.950
0.953
0.954
0.983
DECAY STEP
10
20
581
A.6 Non-GAN Hyperparameter Tuning
Table presents the minimum mean squared errors obtained after tuning hyperparameters for the alternative unsupervised neural network methods that use L,, Lg and Huber loss functions.
563
Table 6: Experimental Results With Non-GAN Hyperparameter Tuning
Mean Squared Error
Key
L
L2
Huber
DEQGAN
Traditional
EXP
1 101
4 10-8
2 10-
3 10 16
2 10-14 (RK4)
SHO
10
10-
10-1
4. 1010-13
1.10-11 (RK4)
NLO
1-10-
3-10-10
1.10-10
1-10-12
4.10-11 (RK4)
COO
5-10-1
2.10-7
3-10-7
1.10-8
2-10-9 (RK4)
SIR
9-10-6
1.10-10
1.10-10
1-10-10
5-10-1
(RK4)
HAM
4.10-5
1.10-8
6-10-9
1-10-10
7.10-14 (RK4)
EIN
50
2.10-2
1.10-2
4.
4-10-7(RK4)
POS
90
1.10-10
1.10-10
4.10-13
3.10-10 (FD)
HEA
10
4.10-8
2.10-8
6-10-10
4.10-7 (FD)
WAV
4.
6-10-7
2.10-7
1.10
7-10-5 (FD)
BUR
10
1-
9.10-5
4.10-6
1.10-3(FD)
ACA
50-0
1.10-2
3.10-3
5.10-3
2-10-1 (FD)
25

---

584
A.7
Residual Monitoring
565
Figure shows several examples how we detect bad training runs by monitoring the variance of the L1 norm of the LHS (vector of equation residuals) in the first 25% of training iterations. Because the LHS may oscillate initially even for successful runs, we use patience window in the first 15% of iterations. In all three equations below, we terminate runs if the variance of the residual L1 norm
586
587
569
over iterations exceeds 0.01.
NLO
10
10a
Residaals (Ia norm)
10a
10a
10-4
10
سله الله اللهة
10
8
10-1
3000
10000
Step
Step
Step
Figure 19: Equation residuals in the first 25% of training runs that ended with high (red) and low (blue) mean squared error for the exponential decay (EXP), non-linear oscillator (NLO) and coupled oscillators (COO) problems. The black crosses show the point at which the high MSE runs were terminated early.
26


Reviews:
[{"review_id": "rOGwO5FbR9", "reviewer": "Reviewer_rL1x", "summary": "The paper proposes a method to solve ordinary and partial differential equations using GANs. Following in the tradition of physics-informed neural networks (PINNs), the flow equation is parametrized by a neural network. The parameters of the neural network are optimised by replacing a hand-tuned loss function such as the mean-squared error, with a discriminator and an adversarial loss. This method appears to provide improved performance over PINNs trained with hand-tuned loss functions and numerical solvers, on several well-known differential equations.", "questions": "As outlined in the previous section, I believe the paper makes a strong case for using GANs to solve differential equations. Some concrete suggestions to address the weaknesses:\n1) It would be good to move the details of sections 3.2.1, 3.2.2 and 3.4 to the appendix, and move details of the ablation study to the main paper. Furthermore, sections 3.3 and 4.1 describe training / engineering choices to a level of detail that hinders readability, and can certainly be condensed.\n2) It would be good to have a more extensive discussion of Figure 3. Moreover, Figure 3 appears to show only simulations from the GAN -- it would be good to have some examples in of the simulations from other methods as well, in order to contextualize the GAN performance beyond the reported MSE numbers.\n3) While the methods residual monitoring and instance noise are certainly important details for reproducibility, it is not convincing that they would work without the other hyperparameter or architectural choices. It would therefore be good to modify the discussion of these two \"methods\", and to present them as potential avenues for stable training rather than a contribution of the paper.\n\nQuestions:\n1) The performance of the different methods for solving differential equations are compared solely on the basis of MSE. Is there a particular reason for choosing this metric? -- it is particularly surprising that there is no justification provided for this in the paper, since the introduction makes a strong point that the choice of similarity metric typically used to optimize parameters for the PINNs are arbitrary. It would also be good to see whether the GAN performance is equally good under other similarity metrics e.g. L1, or MMD \n2) In line 181-182, \"We use the ReLU function...noise should not be used.\" -- this logic is not clear. Why should $L_g < L_d$ on any particular iteration indicate a better generator? -- this could also be due to initialisation, especially at the beginning of training.\n3) How does the coarseness of the grid from which $t, x$ are sampled affect the performance of the GANs over other methods?\n\nMinor comments:\n1) The notation in section 4 is confusing in places: $G(x)$ and $\\Psi_{\\theta}(x)$ are used somewhat interchangeably; in line 179-181 $L_g$ and $L_d$ are not defined.\n2) In Figure 2, a legend is not required in every panel\n3) In Figure 3, the dotted lines in (a) and (b) are note identified in the legend; the colourbars have no label in (c) and (d)  ", "limitations": "It would be good if the discussion could address the following issues:\n1) While the discriminator is not interpretable, the generator has potentially the same issues. What scientific insight is gained from an (effectively) black box generator modeling a flow equation?\n2) While the GANs do provide an improvement over existing methods in terms of performance, it would be good to have an idea of the associated computational costs, sample efficiency and training times compared to the other methods.\n3) How does this method scale with the number of dynamical variables, the size of the coarseness of the $t, x$ grid?", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1) The paper is very well-written, well-structured and easy to follow. The method and results, for the most part, are described in clear detail.\n2) The ablation study, and discussion of methods to ensure stable training were extremely interesting and relevant, given that GANs are known to be difficult to train.\n2) The extensive results in the paper make a convincing case for using GANs over other methods for solving differential equations -- in particular, with respect to the variability of the solutions from other methods across different problems.\n\nWeaknesses:\n1) Many of the details of training, hyperparameters etc., included in the methods section (sections 3.2.1, 3.2.2, 3.4), while important for reproducibility, distract from the main message of the paper, and hinder readability.\n2) Some of the results require a lot more elucidation: e.g. the details of the ablation study in section 5.2 are mostly relegated to the appendix, and thus make the import of the results hard to parse; the plots in Figure 3 are barely mentioned or described in the text. \n3) The paper could improve with a more extensive discussion of the limitations of the method (the conclusion mentions non-interpretability of the discriminator, but not much more).\n4) The paper claims to present two \"methods\" to improve robustness in training: residual monitoring and addition of instance noise. While it is important to document the techniques used to stabilised training, neither of these are nescessarily novel contributions in the context of GANs. Furthermore, it is not clear whether these measures would be equally effective without spectral normalisation, skip connections in the generator, or with different choice of variance for the instance noise, or a change in the 25% training iteration threshold for residual monitoring.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "IaZwF7t__ax", "reviewer": "Reviewer_mpYu", "summary": "The paper presents a new way to solve differentiable equations by leveraging GAN-based adversarial training to “learn” the loss function for PINNs.", "questions": "-It would be good to see an ablation study where they see the effectiveness of the “tricks” that they used to train the GAN effectively because some of them may not be needed.\n-Can you make some comments on mode collapsing as a I mentioned on the weaknesses above?", "limitations": "The authors have addressed the limitations on the Conclusions section. The work is not applicable for negative societal impact in my opinion.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n-Provides a novel method to solve Differential Equations with GANs in an unsupervised training setup. They do not use the solutions to the equations.\n-As can be seen from the results it improves the performance on the Differential Equations task by multiple orders of magnitudes.\n-They authors also integrate different effective methods in Section 3 for stable training of the GAN.\n-Good paper presentation and a lot of details for the experiments in the Appendix. I also like the \n\n\nWeaknesses\n-No discussion for mode collapsing. I understand that this is a different application of GANs but it would be good to have a discussion on mode collapsing. Maybe the proposed GAN can provide different solutions for some equations.\n-No theoretical analysis or guarantees are provided in the work. GANs in general they do not have a lot of guarantees but a theoretical analysis would be much appreciated.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "HQuIij8JYUa", "reviewer": "Reviewer_v1ox", "summary": "This work proposes Differential Equation GAN (DEQGAN) for solving differential equations using generative adversarial networks (GAN) where the generator learns to output the solution and the discriminator tries to learn a good loss function. The authors compare DEQGAN against classical PINNs over twelve ordinary differential equations (ODEs) and partial differential equations (PDEs). The experimental results show that DEQGAN can achieve multiple orders of magnitude lower L2 error than PINNs.", "questions": "1. The authors motivate the need of learning loss function by saying that L2,L1 loss functions lack theoretical justification. However, the author challenges the standard practice in physics-informed learning without explaining why and when L1, L2 loss functions can be bad choices. If L2 loss is problematic, how is the learned loss function better than L2 loss? Can the authors provide theoretical justification for it?\n2. How fast is DEQGAN compared to PINNs and numerical solvers? Can authors provide time complexity comparison against PINNs and its improved variants such as SA-PINNs?\n3. Only one specific artificial initial condition is used for each equation. How will DEQGAN perform if the initial condition is a random function sampled from a random field?", "limitations": "1. Lack of theoretical justification of the loss function and how the disciminator can learn a better objective function.\n2. DEQGAN has training instability issue. To solve one equation, it needs to repeat runs multiple times and filters out the ones with poor performances.\n3. The equations considered in the experiments are artificial. Only one specific initial condition is used for each equation, which raise the concern that the authors may pick the initial conditions to put DEQGAN in advantage in the experimental results.", "rating": 3, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: the paper is well-written and easy to follow. DEQGAN demonstrates superior empirical results compared to classical PINNs. The fact that DEQGAN doesn't depend on predefined distance such as L1, L2 can be useful in some problems.\n\nWeakness:\n1. The Related Work section misses some important discussion [1, 2]. The authors should discuss the relationship to works and place the contribution of DEQGAN in the context of prior works. For example, SA-PINNs [1] can also be viewed as learning a loss function.\n2. The authors state that the L2 loss lacks theoretical justification but miss the important explanation on why and when L2 loss can be problematic. Furthermore, DEQGAN do not have a theoretical justification of the learned loss function either.\n3. The loss function in DEQGAN is borrowed from generative adversarial network (GAN). The original GAN work designs its loss function such that generator is minimizing Jensen-Shannon divergence between the generated distribution and target distribution given optimal discriminator dynamics. However, the Jensen-Shannon divergence doesn't make sense in DEQGAN because the problem is to solve a differential equation with unique solution. Even with Gaussian instance noise, minimizing the divergence between two Gaussians will just be minimizing the L2 loss. I do not see why a generative model is needed here.\n4. The equations considered in the experiments are artificial. Only one specific initial condition is used for each equation, which raise the concern that the authors may pick the initial conditions to put DEQGAN in advantage in the experimental results.\n5. DEQGAN has training instability issue. To solve one equation, it needs to repeat runs multiple times and filters out the ones with poor performances.\n\n[1] McClenny, Levi, and Ulisses Braga-Neto. \"Self-adaptive physics-informed neural networks using a soft attention mechanism.\" arXiv preprint arXiv:2009.04544 (2020).\n[2] Daw, Arka, M. Maruf, and Anuj Karpatne. \"PID-GAN: A GAN Framework based on a Physics-informed Discriminator for Uncertainty Quantification with Physics.\" Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining. 2021.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Reject