Neural Network Differential Equation Solvers allow unsupervised error estimation and correction
Authors: Akshunna S. Dogra
Jeffrey B. Lai
Min Chul Lee
Martin Peev
Venue: Submitted to ICLR 2023 (2023)
Keywords: Deep learning
Numerical Methods
Differential Equations

Abstract: Neural Network Differential Equation (NN DE) solvers have surged in popularity due to a combination of factors: computational advances making their optimization more tractable, their capacity to handle high dimensional problems, easy interpretability, etc. However, most NN DE solvers suffer from a fundamental limitation: their loss functions are not explicitly dependent on the errors associated with the solution estimates. As such, validation and error estimation usually requires knowledge of the true solution. Indeed, when the true solution is unknown, we are often reduced to simply hoping that a ``\textit{low enough}'' loss implies ``\textit{small enough}'' errors, since explicit relationships between the two are not available. In this work, we describe a general strategy for efficiently constructing error estimates and corrections for Neural Network Differential Equation solvers. Our methods do not require \textit{a priori} knowledge of the true solutions and obtain explicit relationships between loss functions and the errors, given certain assumptions on the DE. In turn, these explicit relationships directly allow us to estimate and correct for the errors.

Under review as a conference paper at ICLR 2023
NEURAL NETWORK DIFFERENTIAL EQUATION SOLVERS ALLOW UNSUPERVISED ERROR ANALYSIS AND CORRECTION
Anonymous authors Paper under double-blind review
ABSTRACT
Neural Network Differential Equation (NN DE) solvers have surged in popularity due to a combination of factors: computational advances making their optimiza- tion more tractable, their capacity to handle high dimensional problems, easy interpretability, etc. However, most NN DE solvers suffer from fundamental limitation: their loss functions are not explicitly dependent on the errors associated with the solution estimates. As such, validation and error estimation usually re- quires knowledge of the true solution. Indeed, when the true solution unknown, we are often reduced to simply hoping that "low enough" loss implies "small enough" errors, since explicit relationships between the two are not available. this work, we describe general strategy for efficiently constructing error estimates and corrections for Neural Network Differential Equation solvers. Our methods do not require a priori knowledge of the solutions and obtain explicit relationships between loss functions and the errors. In turn, these explicit relationships allow for the unsupervised estimation and correction of the model errors.
INTRODUCTION
Deep learning has heralded new methods for many scientific disciplines - the field of numerical methods for differential equations has been no exception 222 3 4. Deep neural network based differential equations (NN DE) solvers have been proposed under a variety of different names (PINNs (), DGM (3), etc) each catering various classes of problems. However, they all share certain common features: the use of known differential equation (DF) as the central component of an appropriate loss function, the use of existing knowledge (boundary conditions, experimentaal/ynntetii data, etc) to constrain the search for solutions, randomized optimization methods that sample from the domain of interest requisite resolution, etc.
We investigate another common facet of many NN DE solvers: the lack of unsupervised error quantifiationn/orrrction methods to estimate model errors without prior knowledge of the solution. Most solvers use the equation based loss functions as a surrogate measure for the error. However, while such measure is intuitively related to the error in the solution model, an explicit description of that connection is mandatory, if error quantification is to be done without knowing the solution.
We achieve these goals by explicitly relating the loss terms and the model error. We showcase how these connections allow results on the model eтror that don't 't rely on prior knowledge of solution. We propose techniques by which these results can be used to build significantly more efficient NN DE solvers, with only marginal increases in computational complexity. We formalize our ideas into four theorems, two inequalities, and two algorithms. We validate our claims with a collection of numerical experiments several non-trivial DEs (including nonlinear PDEs).
For the sake of readability and simplicity, all proofs have been rigorously presented in the appendices, while the main text simply reports the results and discusses their significance.
An associated codebase is also provided, with inbuilt options for the DEs already studied as part of this work. However, the codebase has been designed so that the users may easily add their DEs of interest (the assumptions under which this work is valid should be general enough for a wide variety of DEs from many different scientific disciplines).

---

Under review as a conference paper at ICLR 2023
2 NEURAL NETWORK DIFFERENTIAL EQUATION SOLVERS
For case of discussion, we let D C Rd denote some closed, bounded, path connected domain of interest for some differential equation (DE). Let 0D D be the portion of the domain over which some constraint conditions on the solution exist (usually obtained as boundary conditions, empirical data, etc). Assume that our chosen DE, when given unique constraints over 0D, admits unique solution Φ D PRP
We wish consider (possibly non-linear) equation operator F G + H, where G and H are some suitable spaces of functions over D (4b E G). We decompose F as
F L N[o C
where represents the term(s) which depend linearly on 4, N the represents the term(s) which depend non-linearly on L and C are the terms independent of Φ. This additive decomposition into linear, nonlinear, and constant terms is always possible: for linear DEs, N
C. We have
L
N + C
Let us assume we wish to construct an NN based approximation N D + RD. Let us also assume the NN uses analytic activation functions that N C Let W be its width (neurons per hidden layer) and Dx be its depth (number of hidden layers). Finally, let w = (b), who w by, wit wf.....) M C R.M. be the M weights and biases of the NN.
2.1 EXISTENCE AND COMPLEXITY
Feedforward NNs with ReLU activation and W > d can arbitrarily well approximate any Lebesgue integrable function & D RD w.r.t. L norm, provided that D is some compact subset of TRd and enough depth Dy is provided to the NN (5) The same holds for NNs with ReLU activation and Dv 2 log,( (d + 1) provided that they are wide enough (6). Hence, there exist w such that V arbitrarily approximate any T E C² over D, given large enough (W, Dx)
These theoretical guarantees may be practically realized by leveraging the minimum regularity expected from $ Recall w,k is the Sobolev space of order ki on compact D Rd w.r.t. the 0 norm. Let us assume that + E Wk.00 (D). There exists some A with (W, Dx) that can s approximate D + R, if W = + 1, DN = ((diamDD)/u (c))
①. where
6= x1 X21. X1, E. There also exist N that approximate Φ D R. if M (lm(2)) d/K), DN O(I(()) (
w 1 (e) su (8) S E),
2.2 OPTIMIZATION
The loss function to train / usually takes the following form
3
L = E2,eD
[[[( ErrEoD [b(2) ( (
(2)
where || Ilp is the usual p-norm on R., Variants of Stochastic Gradient Descent (SGD) are almost always capable of eventually reaching adequately small loss values for large enough (W, D2) under such loss (9). Thus as C > 0 during optimization, the uniqueness of & implies A - $ over 'D
(formally shown Theorem [.
2.3 ALTERNATIVE PARAMETERIZATION OF THE CONSTRAINT CONDITIONS ON ØD
Sometimes, the constraint conditions (such initial boundary conditions) may be enforced by using the following parametrization as the model N for क., at some arbitrary #1 E D:
M(=) = (xx)) + dist(211,22) No
(3)
where x2 E WD is some appropriate nearest constraint point 21- Here, Ao represents the NN and dist(21, 222) is some metric that enforces that the model A is always exact over OD, while allowing the NN flexibility to learn elsewhere (ex: 1 - e [[]] ==2ll, as used in (10). This parametrization eliminate the need for the EyyeaD term in Eq. 2 Our work is applicable in such scenarios too.
2

---

Under review as a conference paper at ICLR 2023
3 ERROR BOUNDS AND CONVERGENCE FOR NN DE SOLVERS
Define 2., := 2 N as the error in our solution model. Three simple observations about W, Per. the convergence of L to 0, and their inter-plays underpin the entirety of this work:
1. F[M] is not explicitly dependent on Φ there is no explicit relation between L and t
over D - 0D, where o is unknown. We don't know if/how
|| -> 0 as + 0.
2. Thus, Φ,, associated with N not estimable over D OD standard NN DE solvers.
3. Optimization performance saturates when / settles around a local minima(s) of L.
In its most general form, the first problem can be immensely intractable and is often handled separately for different kinds of DEs (3 TO:
II 12, 13). In this section, we present generic results (proofs in Appendix B) which strongly combat this problem for a large collection of DEs originating in scientific domains, including many nonlinear PDEs. Our results will be valid for any F[V], s.t.
F[A] = = N == L, thus allowing larger class of loss function designs as well.
Theorem 1. Let G and H be two Banach spaces. Suppose that
3 - is Fréchet differentiable in U, that the derivative of F tb E U is invertible, and F / 0 uniquely. Then, there exists neighbourhood V C H of small enough such that
F[M] - - - Φ.
Note that the existence of the Frechet derivative of F (denoted DF) is necessary if gradient descent or related methods are to work. As such, only the assumption that DF exists within some open neighbourhood of ④ is imposing additional structure. Theorem | is powerful result backing the wisdom of using NN DE solvers. However, the same assumptions allow us to go much further:
Theorem 2. Under the same assumptions Theorem 7
have
1 e
O ([[F[]]]I).
We codify that as theorem over relaxed class of assumptions on F as Theorem 2 validating the intuitive wisdom of using the C as a surrogate for error: especially when the associated constant of proportionality for the Lipschitz continuous F is of order unity. Some famous examples where Theorems [ and 2 are valid are non-degenerate Hamiltonian systems, nonlinear Poisson's 's equation with homogeneous boundary conditions, etc. Our assumptions exponentially convergent optimization phase somewhere at some rate in G (see Lemma 3 in Appendix B). However, we can go even further and explicitly estimate that rate with one additional assumption:
common observation made for NN DE solvers is that F[I]] & E |
-oasson-oolzzaan equation, Heat and
guarantee an
Theorem 3. Let G, H be Hilbert spaces, U C G an open subset, and (t) be the NN model after t iterations. Assume F E (U; H). and that DF () is invertible 4. Then for every > there exists an R > 0, s.t. for all initial conditions (0) E BR( C U, the gradient descent equation
((t) VtN((t))
has a solution that satisfies
(W(()) - T1| < (- a-d)omin ([ +b1
where
|DF |
0 min
inf NEG
inf Spec (DF()))(DF()) 0.
mmin > O implies exponential convergence at least at that rate is always possible. However, while we mitigate the concerns raised in the first two observations, [ describe &., over D. We can obtain stronger estimates on || by assuming structural information on the & dependent terms L, N and on the interactions they might have with each other.
only weakly/globally
In particular, assume that DN is a positive (or negative) definite operator (such an assumption is often taken to avoid degenerate systems). Define Nº + 2 Further, let:
Hmin == inf inf [[A|] A E Spec (DN())))
sE(0,]]
3

---

Under review as a conference paper at ICLR 2023
Let Fmax be the maximum of [|[]]] over D for some model A that has finished optimizing. We then have the following inequality on Pe,, (see Appendix C for assumptions and proof):
Fmax
| (x)
X E D
(4)
Hmin
A variation of Inequality in Appendix C showcases how NN DE solver design can encode other knowledge/guesses on the system's behavior. Those can be used fine-tune the assumptions that go into obtaining the presented inequality and lead us to modifications needed on the bound.
However, even with the stronger inequalities on (e. || (which might still give severe overestimates, if min is not properly controlled for), we don't It yet have information regarding pointwise behavior of 4e over D åD. There is scareity of work on algorithms that can explicitly estimate $ over D - DD using only the information available to the model N. More often than not, precise error estimations rely on actual knowledge & over entirety of D. the next section, we shall provide an unsupervised algorithm to estimate &e, over D - 0D, without knowledge of Φb
This endeavor, combined with third observation will directly lead to another useful application. For any nontrivial DE, it is improbable that N over D will be the exact solution once stable minima(s) of optimization is reached (stability is meant here in the sense of a converged model that is unlikely find a significantly better minima in reasonable amount of iterations). Indeed, it is demonstrably impossible if the DE does not admit closed form solutions, since A always a closed form expression. Any further training would not provide any meaningful gains in accuracy.
However, in that regime, even a moderately good estimate NV, for De, would be immediately useful: could simply use N + NN, as better model (albeit, additional optimization/moder memory costs). Finally, just like Theorems 3 remove the ambiguity associated with standard NN DE solvers, we will find similar results reliably eliminate such ambiguities vis vis the error model N
ERROR ESTIMATION AND CORRECTION
We begin by noticing the following:
F[A]+ = L[W] + N[W] + C= - N'W, [.,]
(5)
where N'W, Tb == N(W + ob,,1 NWW] Thus, we have a new equation and its operator F₁ Fi Fi F[N] +[0,, L[
N'W, 2., = 0
(6)
with o. the only unknown quantity. The uniqueness of & implies that A is fixed, then & is unique solution to the DE given by Eq. 6 and we have
F (.. F[W + =
Note that N' is obtained such that terms explicitly dependent on & don't appear Eq. that is what allows tight bounds on 4 in (10), without depending upon explicit knowledge of & (Inequality is a strong generalization of that result). As a motivating example, consider the cases when the nonlinearities in F are degree 2 polynomials: N[1 = [1- In that case, we have:
F[J] [ + N'W,, [ 0 N'W, Φ (Ø., & +N Dr, R .N)
The transformation technique itself (algebraic manipulations, Taylor expansions, etc) is not important: only need it to be such that F[N]. D., appear in relation each other over all of D. Indeed, are always able to directly leverage the unsimplified N[W] NN + P.,] form of N' (or even the equation) P for optimization purposes in the proposed Algorithm 1.
F[N T Pe.]
Note that F, [W,]] F[W .N,,] for any appropriate mapping NN. and not just 4ey
Hooweerr it is sometimes profitablc to obtain an explicit, separable expression for N'W, $.,). For example, whenever Taylor expansions can be used.
N' o., TI(A) Φ1, T2(A)
2.
where Ti(N)) are the th order Taylor terms. These can be useful for analysis, since N,, 2, appear in multi- plicatively separable terms in the new expressions (10) used these forms to bound & However, we obtained Inequality Tin way that renders such transformations superfluous: bounds stand under weaker assumptions.
4

---

Under review as a conference paper at ICLR 2023
Algorithm Unsupervised Estimation and Correction of d.,,
1: procedure ERRORCORRECT
2: initialize NN DE solver N - R.
3: while N not converged do
Initial model A over D
4: Optimize N for step using L (Eq.
5: end while
6: Freeze the parameters of N
Fix N to start error analysis
7: initialize Error estimation model Ne.
8: while Ne, not converged do
>> Error estimation over D
9: Optimize Ne, for step using L. (Eq.)
10: end while
return N +N
Error correction over D
12:
end procedure
Eq. immediately hints at method to estimate Deg we use Eq. with another network model, Ne the manner we used V with Eq. 2 to obtain an approximation for Pe, More precisely, once we have a converged model N that has saturated its capacity to model Φ, we use:
Fi [N,,
F[A] + L[Ve,] +N
to define a new loss function for the error model No,
EDED
F [We, (11)] 2 (x, Ne, (12))lp)
(8)
Alternatively, since the parameters of N are kept unchanged, we can use the equivalent form: C, = ED
[[]((;) + +N, [IIp] Ex [1 [M(zz) WW, (z2))ll)
In problems with homogeneous constraint conditions, ((D = 0, we may replace the second term in Eq. 8 by [[N,, (12) + +Z22)] since 2. (ÖD) = - NaD) = -0D)
The same reasoning underlying L analogs of Theorems the next subsection. In particular, knowledge of F[W] allows substantial control over how to estimate P.., whereas the knowledge of F [NV,, (or alternatively F[W "[W++,,,]) allows us bounds on || .N,, ||. The ability internally and reliably estimate 4. associated with A is not only useful for error analysis purposes, but also for refining N. Algorithm describes the procedure.
. 4. allows that L.; = - No. D... Indeed, and Inequality 4 are obtained directly, as stated a generalized form in
Note that the difficulty of estimating and 4. should be roughly equivalent since b. has least the same regularity as Φ. As such, the optimization costs of Ne, per iteration will not be substantially larger than those of N (see discussion/cxperiments in the next sections).
Finally, note that Alg. provides relative corrections to some model V. We do not claim that only eтror correction can provide improvements performance the extent shown in Fig. We only claim that еттоr analysis and correction are useful and available tools for any model N that might have been obtained via some technique based on NN solvers. It should always be possible to error correct.N / using the strategy we are describing, as long as the associated assumptions are satisfied.
N
M1
N₂
N3
Le
Lez
Les
Les
N..
N₂
Nº.
N
Figure I: Graphical Model the 7hh order error correction scheme. Each vertical pair of blue and red nodes denotes specific order in the estimation and correction process.
5

---

Under review as a conference paper at ICLR 2023
4.1 HIGHER ORDER ERROR ESTIMATIONS AND CORRECTIONS
each modeling the errors Bez Prgi..., & Cep associated with error corrected solution models N NN Here, Nj N + Ne, is the 7th order corrected model. Assume the i order emror correction model Ne.- has saturated its capacity to model 4 44-1- We can obtain a higher order corrected model NG by setting up new network Ne, estimate D. The new network Ne; optimized using:
Consider some sequence of networks Nea.Ne
F. F.[Ne.] == F[W2 + W = 1i + LWV.1 N + Na.1 N(W-1]]
(10)
Le. EDDDD || [NC. [11p]] + EZLEaD | (12) No. (x2))lp)
Figure provides a visual representation of the recursive process involved in building 5t order corrected models. Appendix A demonstrates how to arrive at these results.
We have direct analogs of Theorems 1, 2, and 3 for the 7th order errors and error models as: Theorem 4. Under the assumptions above on F, the analogous result holds for for all i E N, ie. we can choose neighbourhood U; G of small enough such that if if; U, then
F:[We.] 0 = No.
Dc.
Furthermore,
cc+1 || O (NN.I
Lastly, under gradient descent, initial condition Ne, (0) has solution that satisfies
(1 tm-
(1) &.,1l Se
We, (0) +.,.
Inequality 4 becomes (under the same structural assumptions Inequality 4
Fi-lmax
(1I)
Hmin
Algorithm in the appendix provides the pseudo-code to setup higher order corrections.
WHY ERROR CORRECTION WORKS AND HOW WE CAN MAKE IT BETTER
NN DE solvers transform the problem of estimating & into one of finding an appropriate A some suitably chosen functional space of NN models. The use of gradient descent type methods turns this into a stochastic dynamical process in that space.
The probability that real network N will end up exactly modeling & within finite number of iterations is 0 (assuming F is even moderately non-trivial). Note this is true without even considering that N is constrained to be closed form model, whereas most non-trivial DEs don't ton't have such solutions. As such, a real network N is doomed to languish in some inadequate minima(s) of optimization, matter how much optimize it. Its capacity approximate & is necessarily limited and once that capacity is saturated, further training of N is waste of computational resources.
Error correction expands that capacity for approximating 4 by introducing new trainable parameters (as components of Ne,) which exist exclusively for modeling the error 2., = $ - M. Since the parameters of the base network V are frozen, have locked the amount of performance already achieved on the base model. Further, since we assume eгтоr correction is being turned on when the base model's approximation capacity is effectively saturated (which is certain happen), the gains we make with Ne, are truly unachievable without it.
We have put forth a claim that error correction not significantly more complex or resource hungry than the standard NN DE solver method: the reason being that &b and Φ- -V are in the same functional space, have the same regularity, etc. Essentially, we meant that 2,, is no more complicated than & However, it bears noting that modeling L., using Me, can be a very different kind of problem than modeling using N. For example, l presumably lies in a different scale of magnitude than t and might present itself as different kind of structure over D (for example, in one sense, it might be significantly more or less "oscillatory" than Φ). Its optimization might also require its own set of hyper-parameters, algorithm choices, activations, etc.

---

Under review as a conference paper at ICLR 2023
This is where control over the sampling of F (and F1) comes into play. Theorems and Inequality 4 allow us to estimate both the scale and complexity of Pe., relative to N. The correlation expected between &. and F F[N] means even crude analysis of F[W] is enough to point in the correct direction, vis a vis the scale and complexity we expect Ne, to model.
For example, we can bound the expected scale of &e, by simply analysing the max of [|[[]]] over D: the linear correlation result already gives us one way to weakly bound T.,- The strong inequality may allow for more precise control, when its assumptions are satisfied.
Similarly, over domain D, we could perform a FFT analysis of F[N] and gauge how oscillatory Nr, should be relative to A (since have control over both NV and F[V] and expect 4., to be linearly correlated to F[N]). Alternatively, we could also analyse the distribution of [[[]]] over D> to understand where the equation is being badly solved, and switch from uniform to targeted sampling over those subdomains. Other methods of estimating relative complexity of features should also be possible, since we have complete access to the behavior of F[W] and F [.V.
The code we provide allows the user to supply their own estimates for relative scale and complexity of Ne, wrt while keeping our preferred defaults. We note again, that in no way is prior knowledge of & assumed: only the information already known before or computed using the NN DE solver is being used. Domain knowledge of an expert building an NN DE solver for system of their interest is thus rewarded during error analysis and correction.
Finally, note that error correction inherits one major issue prevalent within machine learning: the ambiguity in determining if real model N has reached (or nearly reached) its capacity for effectively modelling Φ. The issue is caused because of lack of work on a priori or in situ convergence guarantees. It is extremely tough gauge whether loss trajectory has flattened due to saturation in modeling capacity or it is simply in a non-trivial region of dynamics from which it could escape at any time. If error correction is attempted while the base model itself is fully capable of providing significant increases in accuracy, the additional resources will simply be wasted (and may even prove to be counter-prodductive) Indeed, in our numerical experiments, we found that convergence of the base model N was a practical pre-requisite for significant gains via eтrоr correction.
However, in our proposed approach, the issue still somewhat mitigated. Error correction can not cost significantly more than the standard methods (it can at worst be 2X expensive). Since it can lead very significant improvements in accuracy when used at the appropriate time, it can always be used as an unsupervised, vost-proeessing/eeinemenn routine without significant risks.
6 NUMERICAL EXPERIMENTS
Let us exemplify the above discussion with an example DE that contains terms with linear, non- linear, and no dependence on the solution. In particular, we investigate a variant of the nonlinear Poisson-Boltzmane Boltzmann equation (nPBE), which serves as an important tool for the study of electro- static interactions with widespread applications biology and medicine (14). We will choose L[] = -[], N[] sinh]], C g. where g is some independent function/term. Thus, we define:
F = [] + sinh[] +
(12)
For some solution attempt N _ IR and associated error 4. -
N. we get
F[N]= A 1 sinh sinh + $.,1
(13)
Thus, replacing 4e, with an error model Ne,, we get the residual equation for the new loss as:
F FiW.,,] +N, = F[N] + SiiNi+++ssh([[[Cos|N,i..I.. 1)
(14)
Now, to setup Ne, as the error NN DE solver (after halting the optimization of A), we can use Eq. to obtain an approximation of .. We not only get an estimate for Q we also get to use Nr, as correction term, while retaining the advantages that make A a lucrative option.
6.1 ERROR CORRECTION RESULTS
We choose D = [-T, . C Rd, and randomly sample points from this domain as training data. The exact form of our chosen DE, considered over D with solution & D - R and homogeneous
7

---

Under review as a conference paper at ICLR 2023
boundary conditions ((DD) = 0, is:
F F( [ + sinh [+} + = 0
(15)
g(x) = -w- [sinnwwr))...............>>..) sin (war)) sinh (sin (wzr )... sin (wira)
x ( Fa}
The solution this DE, which showcases significant scale and feature variation over 'D (Fig.
3), is:
& (xx) = sin(waz)....sn(urr)))
x X
(16)
Note that the solvers never this solution, and only have access to their own parameters and the operator F during optimization. We only use the exact solution to verify that our claims hold true.
Given network N to model the true solution + optimize N using the mean squared loss C:
C(N) = EzI-D || (x)())) sinh + s(x1)]l² + Ezy~oD [|
(17)
The error correction model N₁.
D effectively optimized using the following loss:
L E
Fi [V (x1))l ExzwaD [12()))) + Ne, (12))l²)
(18)
As discussed before, L = F[](x)] - 0, over D, which implies N - +. Similarly, fe; + = Fi[N, (x)] - O, which implies No, - Pe., We use Eq. [6]t benchmark efficacy of our proposed algorithm for (w, d) choices of (5, 2) and (1,4). 4). The goal of the experiments was to figure out what impact error correction would have, used lieu of the standard approach, while running for equivalent number of training iterations. As such, choices width, depth, architecture, etc were kept the same between N and N The experiments also demonstrate the effects of invoking error correction different stages of training (Fig. 2)
10
100
10-1
10-1
Relative Error
Relative Error
10
10-2
10-3
100%LAV
1009
10-4
50% 30RLAG
NCC -N,, NN, 1196 NN. SHOPE NG, 1198 Nr,
10
75%6 IN 2596
9076 NT 1036.C
10-5
96% AT, JA,
9009 N., 898 M.,
98% AC, 19%.
999 N., 15
10 4
0.5 1.0 1.5 2.0
2.5
3.0
3.5
4.0 x12
1
Iterations
Iterations
K207
(a) 2 dimensions, ==
(b) 4 dimensions, =1
Figure 2: Relative errors for a single order correction on nPBE, Legend labels indicate duration of training N, and duration of training Nr, for correction respectively.
To present and analyse our results, we define computational cost metric, 7: (time per iteration), and computational performance metric, Relative Error (RE):
K
Relative Error
(xxx) (]x
+ (xx))
where N₁ D + R is the tth order corrected, NN DE based approximation for & over D. K represents the number sampled points D for the calculation. In Fig. _ we plot the relative error dynamics of the various choices made. Notice the very immediate gains made by error correction.
8

---

Under review as a conference paper at ICLR 2023
Table I: Performance comparison across different systems and optimization strategies
System (Adapted
T (10-2 s)
Rel. Error (10-1)
ROI
ROI (Size Ablation)
ROI (Time
Codebase)
(Std, EC)
(Standard, EC)
(EC)
Ablation)
HH (exact IC HNN (00)
(1.37, 1.63)
(3.08, 0.18)
14.4
1.38
0.81
nOsc (exact IC HNN (00)
(1.39, 1.68)
(1.73, 0.02)
71.6
1.12
0.83
nPBE (5, 2) (PINN 22)
(8.6, 11.8)
(436, 0.37)
861
1.65
0.77
nPBE (1, 4) (PINN 2
(72.1, 89.8)
(87.3, 1.87)
37.5
1.23
0.79
We also investigate alternative scenarios, where the slightly higher consumption of resources is used in other ways, quantify ablation possibilities. We conduct two sets of examples. For size ablation experiments, we allow N to be proportionally larger, while training it for the same number of training iterations (say T). For time ablation experiments, we allow the standard algorithms a proportionally higher number of iterations. We define the Return Investment ROI of each choice (error correction, size, or time ablation) as:
RE(C)
Tsu
ROI(c)
X
T(c) REsid
T(c)
As Table I shows, error correction consistently provides the best ROI.
The associated codebase with this work provides numerical experiments some other nonlin- ear/chaotic DEs (it designed to compatible with any DE operator with permissible Frechet derivatives and allow any finite order of conrection). The results those systems are also tabulated in Table (first order error correction was injected at 50% of the run-time, for all other examples). All results in line with our expectations. The additional systems are described in appendix D.
7 CONCLUSIONS
The surging popularity of NN DE solvers presents exciting possibilities in many scientific fields: their capacity to sidestep the curse of dimensionality (IS), general advances in computing/GPU power, and their easy interpret-abilityy make for powerful combo. As such, it is important that these solution models be capable of validation over domains where true solutions are not available. Our work proposes theorems/methods that fix this deficiency for many NN DE solvers. Summarily, NN model errors are unambiguously estimable, and profitably so, if the assumptions of Theorem i hold.
For systems where the assumptions of Theorems 3 do not hold, existing work still leads to expect that NN DE solvers (and thus, eлтоr correction) will work. However, for those systems, error correction converts the ambiguity associated with the models A into the ambiguity associated with the estimate N These unreliable estimates of & can lead to more accurate models, but our work does not rigorously predict when how this happens. However, since this ambiguity is present for all NN DE solvers prior to our work, unsupervised етrоr analysis and correction can only be an improvement upon the existing situation (even when it is unreliable). Future work will focus on extending our results rigorously to more systems, so that even larger classes of DEs may come within the remit of reliable error estimation and correction.
Lastly, note that the suggested ideas not radically different than those that already exist for classical numerical methods: higher order corrections are about as old as the field itself. By pairing the many significant advantages of modern NNs with those old ideas, we simply hope to have presented a blueprint that will be useful for a wide class of scientific problems that are being tackled using NN DE solvers.
REFERENCES
[1] I. E. Lagaris, A. Likas, and D. I. Fotiadis. Artificial neural networks for solving ordinary and partial differential equations. IEEE Transactions on Neural Networks and Learning Systems, 9-987-1000, 1998.

---

Under review as a conference paper at ICLR 2023
[2] M. Raissi, P. Perdikaris, and G.E. Karniadakis. Physics-infommn neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378:686-707, 2019.
[3] J. Sirignano and K. Spiliopoulos. Dgm: A deep learning algorithm for solving partial differential equations. Journal of Computational Physics, 375:1339-136,, 2018.
[4] Han, A. Jentzen, and Weinan E. Solving high-dimensional partial differential equations using deep learning. Proceedings of the National Academy of Sciences, 2018.
[5] Z. Lu, H. Pu, F. Wang, Z. Hu, and L. Wang. The expressive power of neural networks: A view from the width. Advances in Neural Information Processing Systems (NIPS), 30, 2017.
[6] R. Arora, A. Basu, P. Mianjy, and A. Mukherjee. Understanding deep neural networks with rectified linear units. arXiv:1611.014
2018.
[7] B. Hanin and M. Sellke. Approximating continuous functions by relu nets of minimal width. arXiv: 1710.1 1278, 2018.
[8] D. Yarotsky. Error bounds for approximations with deep relu networks. Neural Networks, 94:103-114, 2017.
[9] Simon S. Du, Jason D. Lee, Haochuan Li, Liwei Wang 0001, and Xiyu Zhai Gradient Descent Finds Global Minima of Deep Neural Networks. In Proceedings of the 36th International Conference on Machine Learning, pages 1675-1685. PMLR, 2019.
[10] M. Mattheakis, D. Sondak, A. S. Dogra, and Protopapas. Hamiltonian neural networks for solving differential equations. Physical Review E 105, 065305, 2022.
[1]] Yeonjong Shin, Jérôme Darbon, and George Em Karniadakis. On the convergence of physics informed neural networks for linear second-order elliptic and parabolic type pdes. Communica- tions in Computational Physics, 28(5):2042 2074, 2020.
[12] Philipp Grohs, Fabian Horuung, Arnulf Jentzen, and Philipp Zimmermann. Space-time error estimates for deep neural network approximations for differential equations. wrXiv:1908.03833 2019.
[13] Fabian Hornung. Arnulf Jentzen, and Diyora Salimova. Space-time deep neural network approximations for high-dimensional partial differential equations. arXiv:2006.00 02/0 2020.
[14] José A Iglesias and Svetoslav Nakov. Weak formulations of the nonlinear poisson-boltzmane equation in biomolecular electrostatics. Journal of Mathematical Analysis and Applications, page 126065, 2022.
[15] P. Grohs, Fabian Hornung, Arnulf Jentzen, and Philippe Von Wurstemberger. A proof that artificial neural networks overcome the curse of dimensionality in the numerical approximation of black-scholes partial differential equations. Mem. Amer. Math. Soc. (to appear), 2018.
[16] Serge Lang. Differential and Riemannian Manifolds. Graduate texts in mathematics, 160. 160. Springer, New York, 1995.
[17] Klaus Deimling. Ordinary differential equations in Banach spaces. Lecture Notes in Mathemat- ics, Vol. 596. Springer- Verlag, Berlin-New York, 1977.
A SETTING UP HIGHER ORDER CORRECTIONS
Using Eq. equation
7 we are able to define new loss Le, using F1 [N,,]] (or equivalently, FW++N,, D).
Define Φr,, := $ No, = +- (N + Ne,,, and use the same trick used to derive Eq. 5 to realize
that.
Fi + NN,1 = F[W] L &.,1 + NW + +W,,1 - -N[1 + (NW + &.,1 - NNW + D.,.) = -L( + N(W + +W,,1 -,,1-N,, N/V + +
+ N[W =-L(D,,)
+N,] 1 -NW + +NG, + Tc2)
(19)
10

---

Under review as a conference paper at ICLR 2023
Let us define the 1s order corrected model N₁ == N +
Ne, and with slight abuse of notation,
Fi[Mi] = F[W + Ne,] Thus, we may rewrite Eq. as:
Fi(Mi] [( NWV p.2] N[N1] =
Note the functional similarity to Eq. Further, to setup a 2nd order correction estimate Nez for Pez- we need to minimize over the following residual, similar to Eq. 7
F2[No]] F, [M]] + [[ + NW + -
In general, given an (f-1)th order corrected model N using V = N + +.... +N we can define 4z, ;= + N and - [ the corresponding residual for the preceding equations. Then, b. is the unique solution to the following equation
F--1[W-1l + L . N t.,. N 0
(20)
The error can be estimated using a differential equation solver (c.g. NN) with residual:
F,[We,] F[W,, + WG-1] F, F; L[N2,] N + N[W-1]]
(21)
Figure D depicts the extension of the error estimation and correction framework as a graphical model. Notice that the initial solver estimator and the error estimators may be pre-fetched before carrying out the optimization steps. Since 4b 4 - -N is always known exactly over 0D for all i, we can optimize the th order eπror model No, using C.,:
Lep E ER
F, (m)
Buz~aD
[122 (
NN., (x2))))))
(22)
Algorithm 2 demonstrates the procedure for higher order estimation and correction. The uniqueness of + as solution to F[] also implies the uniqueness of 4c, as solution to F.[]. Unpacking the definition of F; and M we find that
Fi + F[ - NG-1 + W F
and therefore the solution this equation is still unique and convergence of L. + implies F; + O, and thus also No, + P.,. Theorem follows naturally from Theorems [ and Inequality [T follows naturally from Incquality Algorithm 2 describes the proposed implementation.
Algorithm 2 Internal Error Estimation and Correction to Order m
1: procedure ERROR CORRECCOR)))((m)
2: initialize NN DE solver : R
3: initialize Error estimators (We, )
4: Train N, and define N := N
N serves as a dummy variable
5: for = 1,...,,7. do Train NV, until loss converges 7: Save parameter states Ne;
6:
8:
N +N
Ne
9: Freeze parameters of A
end for
11: return + Em
Equivalently, return /
12: end procedure
B THEOREMS ON THE RELATIONSHIPS BETWEEN PE,, F[M], AND L
This work is centered around three theorems. We begin this section by summarizing and contextualiz- ing those results.
Theorem verifies the intuitive expectation that F[N] + implies the fitness of N as an approxima- tion of 4. We do this by taking advantage of the uniqueness of Φ solution, alongside the fact that any gradient descent method can only be successful, iff the gradient itself exists well defined sense.
11

---

Under review as a conference paper at ICLR 2023
Theorem 2 naturally follows consequence of Theorem 1 It adds strong priori expectation we should have from NN DE solvers that satisfy the assumptions of Theorem and tells us that their errors should be quantifiable in sense, if information about F F[W] is sampled (which is always possible up to the desired resolution). Additional structure leads us to stronger quantification capabilities (surmised in Inequalities 4 and 11 and proved in appendix C).
Finally, assume a mapping F G C U + H between two Hilbert spaces, e.g. representing a non-linear PDE mapping between function spaces. Further, assume it twice differentiable and the solution o, satisfying F O, DF[$]. is an invertible linear map. Then, the gradient descent
procedure guarantees convergence rate
A-elemin
(IV(() - 2 S e
(IW - +1
where > O can be chosen sufficiently small if (V( + - R. where R depends on c. miin here given by the minimum of the spectrum of ( y DF 4 which is strictly positive DF is invertible. Theorem 3 guarantees that exponentially converging NN models for & exist somewhere in G, and that gradient descent will allow that level of performance. We need only find some initial model from that region.
Together, the three theorems give iterative optimization. Real NN DE solvers differ from this idealized framework in two ways: their optimization is a discrete process and the subspace within which real NN models lie (say Gm. where M is number of parameters in our model), while being capable of coming arbitrarily close to 4, seldom contains The real optimization a discrete approximation of the ideal trajectory: further, the empirical trajectory is actually a projection of that discrete trajectory onto G.-
idealized framework for describing NN DE solvers under
Note that for finite dimensional approximation of the problem on a subspace G C of dimension M. the corresponding constant o Gf greater or equal m Furthermore, if Gm Gm then GM 2 0 G.. Thus, one can expect the error correction procedure to allow for an exponential improvement each step, iff miin > 0, i.e. exactly in the cases when DF [ is invertible.
Finally, Theorem follows trivially from the definitions of F; NG-1, 1,,,,,,,2,0 Pe,, and Theorems now move to rigorously prove Theorems below.
We
B. INVERSE FUNCTION THEOREM AND THEOREMS 1 AND
We quickly summarize the Banach space version of the inverse function theorem that allows to establish that F[N] - 0 implies N - & (for further details see for example (16).. We also establish another lemma will use estimate the convergence rate of NN DE solvers
Let G and H be two Banach spaces, and U C G an open subset. A continuous map between the two Banach spaces F :: G U is said to be Fréchet differentiable point æ E U iff there exists linear bounded operator La : G + H such that
((y) F(x) - (u --)x = ollv - x(la)-
If the map DF(x] Lg is continuous, then says that F 66 (U; Analogously (U; H) denote the functions which are twice differentiable, and if H = IR we drop H from the notation.
be a '6'- Suppose that there exists a point zo U such that DF (zo) is an isomorphism (i.e. it has a continuous inverse). Then there exists a neighbourhood c H of F(Fo) and 8¹ function F V G that is local inverse of F.
Lemma 1. F
Proof. See Theorem 5.2 of (66)
Lemma 2. Assume L E & L is strongly convex at N E U. for all N' G
D²C 2 P|
Proof. This follows by a simple application of Taylor's theorem as in the finite dimensional case. For Taylor's 's theorem for functions on Banach spaces, ef. (16).
12

---

Under review as a conference paper at ICLR 2023
Theorem 1. Suppose that F (U; (U:H), H). that the derivative of F at o U is invertible, and F(Ø)= = 0. There a neighbourhood C H of small enough such that
F(N) - - ~ +.
Proof. By Lemma I we can choose neighbourhoods
C U and 0 E c H such that F:U V is diffeomorphism. Then, if N E U" the continuity of F-! iimliies
F[ -
which is the assertion.
Theorem 2. Under the same assumptions as above we
|| - 2 (II[M]]
Proof. Since by Lemma 1
F-1 1 is differentiable at 0, it follows that it locally Lipschitz continuous
around 0, implying that
| - 2 [
The constant of proportionality is approximately given by X-1 where
Amin inf (DF
[X]
eigenvalue with minimal absolute value in the spectrum of DF The nPBE example from section 6 fits nicely into this paradigm the map
F 22000 (RR')
W0,00 (RR)
f Af sinh(f) +
continuous and continuously Fréchet differentiable with Fréchet derivative
DF()) = - cosh(/)
which is everywhere continuously invertible as a linear map. The constant of proportionality is approximately 1.
Theorems and 2 guarantee that there exist adequate models for 4, if the loss is going down sufficiently enough. However, we still have no clue about how fast or slow this convergence is going to be and how to convert these existence results into actual estimates on e., (and higher order D..). We develop those results below, with minimal additional assumptions.
B.2 GRADIENT DESCENT HILBERT SPACES
We assume that G is Hilbert space, i.e. in addition to being Banach space, it is also equipped with an inner product (: such that (v, v) = ||u|| for e E Assume that we are given a loss function L G U IR that is element of E6' (U). We denote by VL(N) the unique element of G, s.t. for all V G
(V) (VL(N), V)
V is a locally Lipschitz continuous or equivalently if L is a locally L-smooth function, then the ODE
((1) - VV(((())
(23)
has unique local solution, cf. (17). By locally L-smooth, we mean that around minimum 4 there exists R > O such that for all N. ..N" E BR(Φ)
[VC()) VC(N")| S TIW -IM--11 N'
Furthermore, for μ > O, we call the function L locally μ-strongly convex around a minimum 4, there exists R such that for all A, BR(D)
L(N") 2 L(N) + (VL(N), N)+ -
We need more lemma to have all the tools will use get Theorem
13

---

Under review as a conference paper at ICLR 2023
Lemma 3. Let L (U) be locally L-smooth, locally μ-strongly convex function around minimum Q, s.t. L 2 0 and (( = 0.
Then for any initial condition N E (R( ball where C is both L-smooth and μ-sttongly convex,
the Gradient Descent equation 23 converges exponentially at rate with rate &/ towards D i.e.
[[((t) - 21 S $== -
Proof. From the strong ju-conveitty it follows that
2 dt [W((t) +11 = (W()) Φ, VL(N(()) - - C(N())
2 bl
x1 -
2 y
x=
20
Solving this differential inequality, we find
[[V(()) 2 S (
+01
Lemma guarantees the existence an exponentially convergent regime of optimization, as long as assume that DF exists and C is μ--ttrongyy convex. However, it is non-constructive statement: we have no information on what behavior should be expected from μ. We can make it constructive with additional assumption the existence of D
B.3 CONSTRUCTIVELY ESTIMATING THE RATE OF CONVERGENCE
Assume that F is in ? definition of Fréchet differentiability we can write F as
(U: H). Furthermore, require that H is also Hilbert space. From the
F[W] = ( - +) + D - +,N - b) (W),
where [W) - oII.N' 3).. Then our loss function C(N) = (F[A], F[W]) is strictly convex
in neighbourhood of Φ
Dd D² 4, 4) (DF[ D), DF -2))
= =(N - - (DF DF Ø)) 2 - 2112
where we used that C(D) = 0 and defined
|DF inf Spec T)) )F(()) 0,
Omin Omin
inf NEG
[|
as DF is invertible. Furthermore, our loss function can written as
L(N) = (DF [ - DF (N - +)) + (2) (N- - 4
4) + R2(N)
where [R2( o[I (1)) and Rg also 2f2-ffuctioo and its second derivative is oII-V ().
Thus, for every > 0 there exists R > 0 such that for all V in the ball of radius R around $b
|D R2
min
||W
11
min.
Then, have that for all A BR(Φ)
Omin (N) 2 (1 emin IV oli)) 2 c)omin
follows that is j-strongly convex in the ball of radius R around & with constant μ = (1 (1-66) e)omin+ and thus the gradient descent flow (L) converges exponentially at rate (1-e]amin
We have thus proven Theorem Theorem 4 follows immediately from the results in appendix B.
14

---

Under review as a conference paper at ICLR 2023
C STRICT UPPER BOUNDS ON ||
We will begin our attempt to ascertain bounds on | with classical Hamiltonian systems, to exemplify the intuition and the technique involved in the generic bound wish to develop. Once that is achieved, we will show how a generalization follows in a straightforward manner.
C.I HAMILTONIAN SYSTEMS
To start, let remember $ := - (Ni .1...., ND &D) for a D-dimensional dynamical system to be solved on domain of interest [0, T Note that D is necessarily even for a Hamiltonian dynamical system. Let us construct a worst case scenario for the norm of Pey We write the equation in terms of the Hamiltonian formulation:
di (VH(
(24)
where the appropriate Hamiltonian, and J is the symplectic matrix
D//2
J Ip/2
(25)
and D/2 is the identity matrix. The NN DE solver is trained using F[N] given by
dN di
F F[M] = JVH(N)
(26)
The equation above represents D separate differential equations in vector form. Since [()](() = 0 for all we can write F[N] follows, suppressing the time dependence
DFW, o. do de. - ! (D²H) (NG)ds) 8 =: RD,,
F[M]
(27)
where NG, : s and D2H(()) is the Hessian matrix of H with components (NV).. In order to be able to extract a meaningful error bound from Eq. equation 27 we need to make some structural assumption: We assume that at a time t4., [0, 2] which [ obtains its maximum
[((to,,) ) (t..,)) )] E., (tv.,) )
Note that the assumption of focusing all error in only component of क. done in (10), already implies the above, but not vice versa. As such, we are using a much weaker assumption than (10).
Then, for time læ,, E [0, 7' we have
1R4,,113 < |R 112
+ 1 112
FIMI
(28)
We next need an estimate on the matrix norm of the inverse R for every E [0, T). We can ignore J as it is an orthogonal matrix and only consider
(D'H) (W.)ds.
We can do this we know that D²H((x) is a strictly positive or strictly negative definite matrix for all E D C IRP in our domain of interest, which is standard assumption for the motion to be non-degenerate. W.l.o.g,, assuming that D2H is positive definite, and setting
Hmin min
min Amin (D²H(x)) FED
zED
where Amii is the smallest eigenvalue of the matrix D²H(r), we have for all / E [0,
R S H-1 min
Unfortunately, need this assumption separate the two terms appearing on the right hand side of Eq. equation 27
15

---

Under review as a conference paper at ICLR 2023
Finally, letting
Fmax = max
[F)
e0
we reach the an estimate on [ as:
Fmax
(29)
Hmin
Additionally, if we assume that the time td. at which &ee reachee its maxima,
[i(t. ) (t+t. ) d (t )=0
we have the following bound on d., as well
| S Fmax
(30)
C.2 BOUNDS ON | IN MORE GENERAL SETTINGS
Eq. 27 generalizes beyond Hamiltonian systems in a straightforward manner for F[] = [[]+N[[
DF[..]0.,d do ! DNNNGjdss &r - L( =: -RO,, -L(
F[J]
(31)
The generalized version of the R &. 0 assumption is that at the X maxima in, we have:
D that | takes its
[ )d, (xx.,) )] ((,,,,(X..,))
X E DD
The next assumption we generalize is the one made to have non-degeneracy in the solutions we were trying to model. As such, we assume that DN is positive definite (or negative definitive) operator. From then on, we define Hmin in similar manner. More precisely, we say:
Huin: = inf inf { E Spec (DNN()JII)
sE(0,1]
(32)
Hming = inf inf X X E Spec ]1)} =E(0.1]
where
DN = - (DN(N)))DDNNA)
Finally, let Hmin = max (Hain, Hming for [ ||:
With that place, we obtain the following inequalities
Fmax
1
S
(33)
Mmin
The generalized version of the assumption on X4. where | attains its maxima similarly gives:
| S Fmax
(34)
The assumption that [[xx,,, 2d, (x..,) )} L fe 1 ) = 0 is critical in allowing us strong
estimates on | Superficially, it might seem like too strong of an assumption (and even somewhat of non-sequitur).
However, note that L[] in a DE is often the domain derivative term (we denote it as Vx in this discussion, where x E D represents derivatives for spatial DEs, time derivatives for ODEs, a spatio-temporal one for space-time DEs, etc). When | achieves its maxima, o, Vx(+,)) 0. When L Vx. indeed have
arbitrary element in the domain. As such. Vx the spatial
16

---

Under review as a conference paper at ICLR 2023
Per [(( 0. As such, what really assuming is that the R term effectively has tes as its eigenvector on X (since that means R$., Vx( is O). For example, such is the case for Hamiltonian systems without mixed position-mmmntie terms. What we are really constraining, is the behavior exhibited by the operator in a system of interest.
If the problem of interest comes from scientific domain with which the user has familiarity, they may supply a different sort of assumption based on those considerations. For example, we might be able to say:
R
V.
at the x0,, where | takes its maxima. The generalized bound simply changes to:
Frax +
| S
(35)
Hmiu
D ADDITIONAL NUMERICAL EXPERIMENTS
As part of validating our claims, we also performed numerical experiments several scientific ODEs and PDEs. We chose our experiments such that the set of examples showcases non-trivial spatial and temporal phenomena amongst the combined examples (chaos, high dimensional domains, etc).
Table presents the median results from the collection of randomized experiments conducted for each system. All numerical experiments were done using 2019 MacBook Pro with 2.6 GHz 6-Core Intel Core i7 processor and 16 GBs of 2667 MHz DDR4 RAM.
We describe the chosen systems below:
D.1 HENON HEILES
Henon Heiles is represented by the following ODE governing the dynamics of Φ
{x, Px. Pylt:
$ -VVH, J 0): 2²+ 2 + P + P
2 y(30²
1²) y²)
(36)
2
3
where I the 2 identity matrix. We picked [0, 6T] as the time domain of interest. All 4 (O),))))))........)) {z(0), v(), (z())) P;))),,;()) (0)} were picked s.t. H()( < - r0) [ v3 /3 and y(0) E [-0.5, 1
2
v3/21], which corresponds to the subset of the phase space that has bounded orbits.
We implement the NN DE solver prescribed in (T). The models each had 2 hidden layers, with 50 sine activation functions per layer. The base models were trained for 50000 iterations using ADAM, with an error correction made at 25000 iterations. We sampled 200 time points were iteration.
D.2 NONLINEAR POISSON BOLTZMANN EQUATION
We have already given details of the equation in the main paper, alongside the associated choices of dimensionality and frequency parameter. Below, plot what the (w, d) (5, 2) system should look like when modeled without (a) and with (b) enror correction, alongside the true solution (c). To quantify the visual, also quote the mean average value of |[A]], ||W +NG ||*|| respectively.
17

---

Under review as a conference paper at ICLR 2023
N (MAV 0.424)
N4. M. (MAV 0.405)
+ (MAV 0.404)
1.0
4/2
3/2
x72
0.0
6
-0.5
-2/2
-/2
-7/2
(a) N
(b) N + Na.
(c)
Figure 3: Heatmaps of non-error corrected N & emror corrected solver with Ne, at 50% of the total training iterations (left middle), and (right) for the (w, d) = (5, 2) setting.
N and,V, Me, were fully connected Neural Networks, with 4 hidden layers, each with 50 sine activation functions. The base model was trained via ADAM for 400000 iterations, while error models were trained for varying number of iterations, depending when the error correction models were activated (see Fig. 2). We sampled 1024 points per iteration.
The (ue, d) (1, 4) model was trained for 50000 iterations, with 8192 points sampled per iteration.
Both NN DE solvers are implemented using the methods prescribed in 2
D.3 NONLINEAR QUARTIC OSCILLATOR
The nonlinear quartic oscillator is represented by the following ODE governing the dynamics of = Pa.pa
t -JVH J= (- 1). # x² p
0
x1
(37)
2
4
We sample (0) such that {a(0), (()) 1-, X 1-1, each time.
We implement the NN DE solver prescribed in (TO). The models had 2 hidden layers each, with 50 sine activation functions per layer. The base models were trained for 50000 iterations using ADAM, with an error correction made 25000 iterations. We sampled 200 time points were iteration.
The associated codebase allows the user to add other systems as per their choice.
18


Reviews:
[{"review_id": "gqjvQFh4T7", "reviewer": "Reviewer_cgGX", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This article proposes a novel method for training neural networks to solve partial differential equations. It begins by pointing out the necessity to infer from the vanishing PDE residual the vanishing of the error of the solution of the PDE. \n\nIt begins by providing theorems that, under certain conditions, allow bounding the error as a function of the residual. \nIt then proceeds to attempt to obtain better estimates for the error by plugging it into the PDE, and thus obtaining a PDE (closely related to the original PDE) that has to be satisfied by the mismatch between the true solution and the PDE. \n\nBy introducing a *second* neural network that is trained to solve for this mismatch and adding its results to the first neural network, the authors obtain remarkable improvements in accuracy.", "summary_of_the_review": "I am very much on the fence about this work. On the one hand, the proposed method is showing surprisingly clear gains, despite being very simple. On the other hand, the present work contains too many spurious claims or handwavy claims and too little insight into why the method works. I therefore believe that it is not ready for publication in the present state.\n\nMore concretely, my main points of criticism are summarized below:\n\n1. Not a single example is given for the application of Theorem 1. In particular, I was not able to discern what the norm is denoted by $\\|\\cdot\\|$ throughout the paper. When using the norms from elsewhere in the paper (p-norms for the residuals, L^2 error for the solution error in the final results), not even a linear differential operator will satisfy the assumptions of the theorem. \n\n2. The paper leads with the promise of improving the error estimation in NN-based PDE solvers. However, as the authors themselves remark, this problem is merely hidden in the \"error estimate of the error estimate\". I was not able to observe any argument as to why estimating the error of the error should be any easier than that of the original error. To this end, the authors seem to only produce the unsubstantiated claim that \"... this is still substantially better than the situation that existed before: even an ambiguous estimate of $\\Phi_{\\epsilon_1}$ has clear practical benefits\". \n\n3. It is quite surprising that the proposed method achieves the improvements reported by the authors. It would therefore be very valuable to provide additional insights. For instance, it would be helpful to provide the derivation of the PDEs for the error corrections in the case of one particular PDE. Also, additional ablation studies would be welcomed, for instance understanding how much of the improvement simply arises from the architectural change of representing the solution as the sum of the outputs of two neural networks.\n\nOverall, I believe that the work shows promising results but is not yet ready for publication in ICLR. \n\n==============================================================\nThe discussion with the authors has further confirmed my concerns about the presentation of the paper. I therefore changed my recommendation to 3 (reject) ", "strength_and_weaknesses": "### Strengths\n\n- To the best of my knowledge, the methodology proposed by the authors is novel. \n- The empirical performance of the proposed method is very promising.\n\n### Weaknesses\n\n- It seems unclear that the theory presented in section 3 is applicable to any PDE\n- The claim of an improved error estimation does not seem to hold up. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "S1Ut13wLEN", "reviewer": "Reviewer_3Hkr", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "NNDE solvers are popular these days, e.g., PINN, etc. This paper presents a method to estimate the error of those NNDE solvers and an iterative method to improve their accuracy. The authors introduce two fundamental theorems, from which their algorithms are designed. They also present some experimental results to show the correctness of their findings and algorithms.", "summary_of_the_review": "I think this paper has non-trivial value in the PINN community. I like their overall idea to estimate the error level and iteratively correct in an unsupervised manner. However, the biggest problem is their experiments. To be accepted, I think they need to test with many PINN variants. Ever since the first introduction of PINN, there have been many other variants. The authors should show the efficacy of their method to them. If not, their nice findings cannot be appreciated. Please answer the weak points above to help my understanding. During the rebuttal period, I recommend that the authors work hard to test with real PINN models. I will reconsider my score after the rebuttal since this paper has non-trivial value in my opinions with one critical problem in their experiments.", "strength_and_weaknesses": "- S1) As I know, this paper is the first to measure the error level of NNDE solvers in an unsupervised manner.\n- S2) Their findings are novel, and their error estimation method is easy to follow and seems effective.\n\n- W1) Experimental results are weak. They need to test with PINN models.\n- W2) Some PINNs use additional loss functions other than F. In such a case, I am unsure if the proposed method can still be applied.\n- W3) Do you consider boundary conditions?\n- W4) The correction method is iterative, which requires large resources.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "OPgqXRCYbdS", "reviewer": "Reviewer_25yz", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This is primarily a theoretical paper addressing the problem of solving differential equations with neural networks. Existing approaches do not provide any guarantees about their error relative to the true solution. Specifically, optimising the loss function doesn’t bound the error of the solution.\n\nSection 2 of the paper sets up a formalism and notation used throughout the rest of the paper. \n\nSection 3 describes some basic results about bounding the error of a NN solution. Specifically, some asymptotic results are given about the loss going down with the solution error of the neural net, and a bound is given for the solution error dependent on Spec of the non-linear portion of the DE.\n\nSection 4 describes the main contribution of the paper. It describes a way to give more useful bounds on the solution error, without knowing the solution already. Essentially, they describe a simple procedure to define a new DE whose unique solution is the error of the NN solver on the original problem. We can extend this to create a chain of networks, each estimating the error of the solution defined so far (i.e. the sum of the original solution, and subsequent error estimates).\n\nSection 5 describes some potential extensions of the work.\n\nSection 6 shows numerical experiments.", "summary_of_the_review": "Overall, the paper is very well-written. Despite having a lot of dense theorems, overall the explanation is clear, and it is very well-connected to practical implications.\n\nI have a few concerns regarding some additional ablations (see Strengths and Weaknesses).\n\nI have not checked the proofs of any of the theorems, and I do not have an extensive theory background, hence the low confidence in my score.\n", "strength_and_weaknesses": "Overall, the paper is very well-written. Despite having a lot of dense theorems, overall the explanation is clear, and it is very well-connected to practical implications.\n\nI have a few questions:\n- Can you bound how much the solution error goes down at each stage of estimating the error of the previous network? How do we know we won’t get some degenerate behaviour, e.g. where the networks in the chain just alternate between overcorrecting each other.\n- Are there ablations to other simpler methods for training NN solvers? For example, your approach requires K models to perform K - 1 steps of error correction. If we just take one model that is (roughly) K times as big and train it once, how does the solution error compare?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "EXaOrltH7m", "reviewer": "Reviewer_6cUn", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose techniques for building more efficient neural network differential equation solvers based on a novel analysis of the optimization objective used by existing solvers. The authors study and derive new bounds, equations, and convergence rate for existing solvers and use these to propose a new algorithm. They provide numerical experiments that show improvements over existing baselines.", "summary_of_the_review": "I can imagine this paper being a useful addition to this line of literature, but this is outside my scope of expertise, and I would mostly defer to the other reviewers.", "strength_and_weaknesses": "STRENGTHS:\n* The algorithmic ideas proposed in the work are novel, technically sound, and non-trivial. The depth of the technical results and their derivations is significant.\n* The experiments show promising results against existing baselines.\n\nWEAKNESSES:\n* This work could benefit from a better contextualization of the results relative to a real-world problem.\n* The experiments section could benefit from additional analysis on downstream applications of the proposed solvers.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject