Sobolev Training for the Neural Network Solutions of PDEs
Authors: Hwijae Son
Jin Woo Jang
Woo Jin Han
Hyung Ju Hwang
Venue:  (2021)
Keywords: Sobolev Training
Partial Differential Equations
Neural Networks
Convergence

Abstract: Approximating the numerical solutions of partial differential equations (PDEs) using neural networks is a promising application of deep learning. The smooth architecture of a fully connected neural network is appropriate for finding the solutions of PDEs; the corresponding loss function can also be intuitively designed and guarantees the convergence for various kinds of PDEs. However, the rate of convergence has been considered as a weakness of this approach. This paper introduces a novel loss function for the training of neural networks to find the solutions of PDEs, making the training substantially efficient. Inspired by the recent studies that incorporate derivative information for the training of neural networks, we develop a loss function that guides a neural network to reduce the error in the corresponding Sobolev space. Surprisingly, a simple modification of the loss function can make the training process similar to Sobolev Training although solving PDEs with neural networks is not a fully supervised learning task. We provide several theoretical justifications for such an approach for the viscous Burgers equation and the kinetic Fokker--Planck equation. We also present several simulation results, which show that compared with the traditional $L^2$ loss function, the proposed loss function guides the neural network to a significantly faster convergence. Moreover, we provide the empirical evidence that shows that the proposed loss function, together with the iterative sampling techniques, performs better in solving high dimensional PDEs.

Under review as a conference paper at ICLR 2021
SOBOLEV TRAINING FOR THE NEURAL NETWORK SO- LUTIONS OF PDEs
Anonymous authors Paper under double-blind review
ABSTRACT
Approximating the numerical solutions of partial differential equations (PDEs) using neural networks is a promising application of deep learning. The smooth architecture of fully connected neural network is appropriate for finding the so- lutions of PDEs; the corresponding loss function can also be intuitively designed and guarantees the convergence for various kinds of PDEs. However, the rate convergence has been considered as a weakness of this approach. This paper introduces novel loss function for the training of neural networks to find the solutions of PDEs, making the training substantially efficient. Inspired by the recent studies that incorporate derivative information for the training of neural networks, we develop a loss function that guides neural network to reduce the error in the corresponding Sobolev space. Surprisingly, simple modification of the loss function can make the training process similar to Sobolev Training al- though solving PDEs with neural networks is not fully supervised learning task. We provide several theoretical justifications for such an approach for the viscous Burgers equation and the kinetic Fokker-Planck equation. We also present several simulation results, which show that compared with the traditional L loss func- tion, the proposed loss function guides the neural network to significantly faster convergence. Moreover, we provide the empirical evidence that shows that the proposed loss function, together with the iterative sampling techniques, performs better in solving high dimensional PDEs.
INTRODUCTION
Deep learning has achieved remarkable success in many scientific fields, including computer vision and natural language processing. In addition to engineering, deep learning has been successfully applied to the field of scientific computing. Particularly, the use of neural networks for the numerical integration of partial differential equations (PDEs) has emerged as a new important application of the deep learning.
1996). a neural network Being a universal approximator (Cybenko) 1989; Hornik et al., 1989
approximate solutions of complex PDES. To find the neural network solution of a PDE, neural network is trained on a domain wherein the PDE is defined. Training a neural network comprises the following: feeding the input data through forward pass and minimizing predefined loss function with respect to the network parameters through backward pass. In the traditional supervised learning setting, the loss function is designed guide the neural network to generate the same output as the target data for the given input data. However, while solving PDEs using neural networks, the target values that correspond to the analytic solution are not available. One possible way to guide the neural network to produce the same output as the solution of the PDE is to penalize the neural network to satisfy the PDE itself (Sirignano Spiliopoulos) 2018 Berg & Nyström 2018 Raissi et 2019
Hwang et al. 2020).
Unlike the traditional mesh-based schemes including the finite difference method (FDM) and the finite element method (FEM), neural networks are inherently mesh-free function-approxxiattrs. Advantageously, as mesh-free aapetionnapproximator.... neural networks can avoid the curse di- mensionality (Sirignano & Spiliopoulos/ 2018) and approximate the solutions of PDEs complex geometries (Berg & Nyström 20188). Recently, Hwang et al. (2020) showed that neural networks could approximate the solutions of kinetic Fokker-Plane equations under not only various kinds

---

Under review as a conference paper at ICLR 2021
of kinetic boundary conditions but also several irregular initial conditions. Moreover, they showed that the neural networks automatically approximate the macroscopic physical quantities including the kinetic energy, the entropy, the free energy, and the asymptotic behavior of the solutions. Further issues including the inverse problem were investigated by Raissi et al. (2019): Jo et al. (2020).
Although the neural network approach can be used solve several complex PDEs in various kinds of settings, it requires relatively high computational cost compared to the traditional mesh-based schemes in general. To resolve this issue, we propose novel loss function using Sobolev norms in this paper. Inspired by a recent study that incorporated derivative information for the training of neural networks (Czarnecki al. 2017). we develop a loss function that efficiently guides neural networks to find the solutions of PDEs. We prove that the H' and H² norms of the approximation errors converge to zero as our loss functions tend to zero for the 1-D Heat equation, the 1-D viscous Burgers equation, and the 1-D kinetic Fokker-Planck equation. Moreover, we show via several simulation results that the number of epochs to achieve certain accuracy is significantly reduced as the order of derivatives in the loss function gets higher, provided that the solution is smooth. This study might pave the way for overcoming the issue of high computational cost when solving PDEs using neural networks.
The main contributions of this work are threefold: 1) We introduce novel loss functions that en- able the Sobolev Training of neural networks for solving PDEs. 2) We prove that the proposed loss functions guarantee the convergence of neuarl networks in the corresponding Sobolev spaces although it is not supervised learning task. 3) We empirically demonstrate the effect of Sobolev Training for several regression problems and the improved performances of our loss functions in solving several PDEs including the heat equation, Burgers' equation, the Fokker-Planck equation, and high-dimensional Poisson equation.
RELATED WORKS
Training neural networks to approximate the solutions of PDEs has been intensively studied over the past decades. For example, Lagaris et al. (1998] 2000) used neural networks to solve Ordinary Differential Equations (ODEs) and PDEs on a predeñned set of grid points. Subsequently, D & Spiliopoulos (2018) proposed method to solve high-dimensional PDEs by approximating the solution using a neural network. They focused on the fact that the traditional finite mesh-based scheme becomes computationally intractable when the dimension becomes high. However, because neural networks are mesh-free funetion-aprooximatrrr, they can solve high-dimensional PDEs by incorporating mini-batch sampling. Furthermore, the authors showed the convergence of the neural network to the solution of quasilinear parabolic PDEs under certain conditions.
Recently, Raissi et al. (019)) reported that one can use observed data to solve PDEs using physics- informed neural networks (PINNs). Notably, PINNs can solve a supervised regression problem on observed data while satisfying any physical properties given by nonlinear PDEs. A significant advantage of PINNS is that the data-driven discovery of PDEs, also called the inverse problem, is possible with a small change in the code. The authors provided several numerical simulations for various types of nonlinear PDEs including the Navier-Stokes equation and Burgers' equation. The first theoretical justification for PINNs was provided by Shin al. (2020), who showed that sequence of neural networks converges to the solutions of linear elliptic and parabolic PDEs in L sense as the number of observed data increases. There also exists study aiming to enhance the convergence of PINNs (van der Meer et al., (020).
Additionally, several works related deep neural networks with PDEs but not by the direct approx- imation of the solutions of PDEs. For instance, [ong et al. (2018) attempted to discover the hid- den physics model from data by learning differential operators. A fast, iterative PDE-solive was proposed by learning modify each iteration of the existing solver (Hsich et al. 2019). A deep backward stochastic differential equation (BSDE) solver was proposed and investigated in Weinan et al. (2017) Han et al. (2018) for solving high-dimensional parabolic PDEs by reformulating them using BSDE.
The main strategy of the present study is to leverage derivative information while solving PDEs via neural networks. The authors of Czamnecki et al. (2017) first proposed Sobolev Training that uses derivative information of the target function when training neural network by slightly modifying
2

---

Under review as a conference paper at ICLR 2021
the loss function. They showed that Sobolev Training had lower sample complexity than regular training, and therefore it is highly efficient in many applicable fields, such as regression and policy distillation problems. We appropriate the concept of Sobolev Training to develop a loss function for the efficient training of a neural network for solving PDEs.
LOSs FUNCTION
We consider the following Cauchy problem of PDEs:
Pu f. (t, E [0,T] S2
(3.1)
Iu = (f,z) E {0} x Ω
(3.2)
Bu h, (t,z) E [0,7] 852
(3.3)
where P denotes a differential operator; I and B denote the initial and boundary operators, respec- tively; f. 9. and h denote the inhomogeneous term, and initial and boundary data, respectively. In most studies that reported the neural network solutions of PDEs, a neural network was trained on No.,, uniformly sampled grid points {(1,, xr)) jij== E [0, x Ω which were completely determined
before training. One of the most intuitive ways make the neural network satisty PDEs 3.1 is to minimize the following loss functional:
3.3
Loss( p) Punn- - flLL. (1.))xx) + ||/unn - (0l)=(0) + Bunn - h 1 ([0,11x000)
where thun denotes the neural network and p or 2, they have been the most commonly used exponents in regression problems in previous studies. Evidently, an analytic solution u satisfies Loss(u) O, and thus one can conceptualize neural network that makes Loss(unn) 0 possible solution of PDEs (3.1 (3.3). This statement is in fact proved for second-order parabolic equations with the Dirichlet boundary condition in jo et al. (2020). and for the Fokker-Planck equa- tion with inflow and specular reflective boundary conditions in Hwang et al. (2020). Both the proof's are based on the following inequality:
||u unn (0,7;;LP()) S Coss 2),
for some constant C, which states that minimizing the loss functional implies minimizing the ap- proximation eтror.
The main concept behind Sobolev Training is to minimize the error between the output and the target function, and that between the derivatives of the output and those of the target function. However, unlike the traditional supervised regression problem, neither the target function nor its derivative is provided while solving PDEs via neural networks. Thus, special treatment is required to apply Sobolev Training for solving PDEs using neural networks. In this and the following sections, we propose several loss functions and prove that they guarantee the convergence of the neural network to the solution of a given PDE in the corresponding Sobolev space. Therefore, the proposed loss functions play similar roles to those in Sobolev Training.
We define the loss function that depends on the Sobolev norm Whp as follows:
LossgE bossce(umnis k p, q) = [P (t,an(t,))) :)) f(t, M1 (l
(3.4)
Lossic (unni L ④) = /1un x) - g(x))
(3.5)
Loss (unn' k. p. 1. q) = (1,,anttt))) h(t, )1i (1,T)))
(3.6)
Remark 3.1. Here, LoSSTOT (Wnx) (unx)
(0)
LossGE 0, 0, Lossio (uni 2) +
Lossnc(was 2, 0, 2) coincides with the traditional L,2 loss function employed by Sirignano & Spiliopoulos 2018); Berg & Nyström (2018); Raissi et al. (011 Hwang el al. (2020).
3.6 computed by Monte- ap- proximation. Because the grid points are uniformly sampled, the loss functions are approximated as follows:
When we train neural network, the loss functions
T
N
ds
LossGE (unn k. k,p,
Σ
D ((unn(ts,, D ((t,,2,))
N.
|
==1
|
1=1
3

---

Under review as a conference paper at ICLR 2021
N
Lossic(unnil.g)
D Una (0, D g(x;))
N
T
N
ds
Loss (unni 4)
Σ
u (42,)) )))(4,,3))
N
dt
i=1
lalsta, 60N
where a and 3 denote the conventional multi-indexes, and D denotes the spatial derivatives.
4 THEORETICAL RESULTS
In this section, we theoretically validate our claim that our loss functions guarantee the convergence of the neural network to the solution of given PDE in the corresponding Sobolev spaces, and that they play similar role to those in Sobolev Training while solving PDEs via neural networks. Throughout this section, we will denote the strong solution of each equation by u, neural network solution by Uyn- and Sobolev spaces W1 and W22,2 by H' and H² respectively. All the proofs are provided the Appendix.
4.1 THE HEAT EQUATION AND BURGERS' EQUATION
We define the following three total loss functions for the heat equation and Burgers' equation:
LOSSTOTAL (Wan) LOSSGE Unn 0, + Lossic (Wani 0, 2) + Loss umn 0, 2),
(4.1)
Loss (Unn) = LossGE
Unn (umn: 0, + Lossie (unn 1,2) + Loss BC (mn' 0,2,0, 2),
(4.2)
LossTOTAL (wnn) = LOSSGE 1, 1, 2) + Lossic (Unn; 2, + Lossno(Umn 0, 2). (4.3)
We then obtain the following convergence theorem:
Theorem 4.1. (Proofs are provided in AA5) for the heat equation, and (A.)) for Burgers' equation) For the following I-D heat and Burgers' equations:
The heat equation
Burgers' equation
= 0 in (0, 7] S2, u(0, x) un un() on Ω
+ ut, Vulrrr = 0 in (0, 7] x S2
u(0, x) = un (x) on 9 u(t, 0 on [0, [0 80,
u(t, x) 0 on [0, T] x 8n,
there hold, provided that Unn is smooth,
max [|u(t) unn (f)llz2(0) - as Loss 0,
(0)
ess sup ||u(t) unn (t) (t) (aggg - 0 as Loss + 0,
05t5T
sup [|u(t) Unn (1) (1))) + 0 as LOSSTOTAL 0.
(2)
051ST
4.2 THE FORKER-PLAACK EQUATION
For the Fokker-Planck equation, we need additional parameters for a new input variable e, We define the following two total loss functions for the Pokker-Plance equation:
Loss TOTAL (Wnn) = LOSSGE Wann 0, 2,0,2,0,22) + Lossic(umn 0,2,0,
(0:FP)
+ Lossnc (nn; 0,2,0, 0,2),
(4.4)

---

Under review as a conference paper at ICLR 2021
(L-FP) LOSSTOTAN (Wnn) (Ussern(Unn) 0,2,,,,,1111 + Lussio(umnt 1,2,1,2)
+ Loss
(nnn (Unn; 0,2,0, 2, 0, 2).
(4.5)
We then have the following convergence theorem:
Theorem 4.2. (Proofs are provided (A.10 and A.12 For the I-D Folker-PPanck equation with the periodic boundary condition:
Ut vt, - B(uu) quve = 0, for (t, v) E [0, T] x x m(0, v. 1) (0 v), for (ar, v) E [0, 1] x R. Re u(1, 1, - p. ul1, 0, v) = O, for (1, E [0. T] x IR,
there hold, under assumptions (A.50 and MA.57
(0:FP) [l((t) unn (t) unn(t)lleeaxxx-w.)) + 0 LOSSTOTAL 0,
sup
0S1ST
sup OS1ST
[u(() Unn (n (0
LOSSTOTAL O.
(1:PP)
Remark 4.3. The theorems this section imply that the proposed loss functions guarantee the convergence of neural networks in the corresponding Sobolev spaces, thereby coinciding with the main idea of Sobolev Training.
Remark 4.4. The theorems in this section cannot be directly generalized the dime--aimenionnal cases because even the 2-dimensional case starts involving the convexity of the boundary. Though has also been shown that the Fokker-Plan Planck operator has strong kypoellipticity and the solutions to the boundary problems are smooth even in the higher dimensional case, the proof requires long rigorous mathematical analysis. For more information, see Iwang et al. (2018) 2019).
Remark 4.5. Because we cannol access the label (which corresponds to the analytic solution) on the interior grid, solving PDEs using neural network is not fully supervised problem. Interestingly, by incorporating derivative information in the loss function, the proposed approach enables Sobolev Training even if neither the labels nor the derivutives of the target function are provided.
5 EXPERIMENTAL RESULTS
In this section, we provide experimental results for toy examples that comprise several regression problems and various kinds of differential equations, including the heat equation, Burgers' equa- tion, the kinetic Fokker-Planck equation, and high-dimensional Poisson's 's equation. We employ fully connected neural network, which is a natural choice for function approximation. We use the hyperbolic tangent function nonlinear activation function. Although R=LU(x)) = max(0, x) is a frequent choice in modern machine learning, it is not appropriate for solving PDEs because the second derivatives of the neural network vanish.
In appreciation of Automatic Differentiation. we can casily compute derivatives of any order of neural network with respect to input data despite the compositional structure; see |Baydin et al. 2017) and references therein. We implemented our neural network using PyTorch, a widely used deep learning library (Paszke et al.) 2019). For the numerical experiments, we used neural net- work with three hidden layers each of which had 1-256-256-2566 neurons, where d denotes the input dimension. We used the ADAM optimizer (Kingma & Ba 2014), a popular gradient-based optimizer.
To see whether our loss functions performed more efficiently than the traditional L loss function introduced in Remark 3.1 we made everything maintain the same except the loss function. We compared the loss functions on the basis of (22( test error for the toy examples, absolute relative Poisson equation, and I (0, T: (.2)) test error for the other
test error for the high-dimension PDEs. For each loss function, we recorded the number of epochs required to meet a certain error threshold and the test eтrоr. Considering the randomness due network initialization, we repeated the training a hundred times. Conversely, we initialized hundred different neural networks with uniform initialization and trained them in the same manner. To compute the test error, used lytic solutions for the Heat equation, Burgers' equation, and the high-dimensionall Poisson equation, and numerical solution from Wollman & Ozizmir (2008) for the kinetic Bokker-Plancl equation.
5

---

Under review as a conference paper at ICLR 2021
5.1 TOY EXAMPLES
First, we consider two simple regression problems with target functions sin(x) and ReLU(x), re- spectively. For these toy examples, we define the loss functions as follows:
L2 loss Unn (z) y(x))
HI loss = |n (x) - y(a) + (1 - ((())
- H2 loss = I|unn (2) y 112 (l - (x) ""())11;
where y(z) denotes either sin(z), or (x). We uniformly sampled hundred grid points from [0, 2T] for training sin(2:). Similarly, we uniformly sampled a hundred grid points from [- 1, 1] for training (x) We expected the training to become fast using higher order derivatives as many as possible when training sin(x) and (x) Figure I confirms our assumption be true. Interestingly, although ReLU(x) (x) is not twice weakly differentiable at only one point 0, the H2 loss does not facilitate the training.
(Mumber Coodre
Loo
mm
Humber Cocch
Lcsr
Figure 1: First row: results for sin(x), Second row: results for (x) First column: Histograms generated from the repeated training of neural networks for training sin(x), and (x). Second column: Test L errors. Third column: Average training time for each loss function to achieve certain error threshold. Error bars are for standard deviations. The thresholds for the error are set to 10-4
In order to explore the nature of Sobolev Training, we design more complicated toy examples. Consider the target functions sin(kæ), for k = 2,,,... 5, and ReLU(kx) = max(0, kr), for = 1, 3..,, 10. As increases, the target functions and their derivatives contain drastic changes in their values, so it is difficult to learn those functions. We hypothesize that in Sobolev Training, the training becomes faster since we give explicit label for the derivatives and becomes easier to capture the drastic changes in the derivatives. This is empirically shown to be true in Figure 2 We train neural networks to approximate sin (kn)), and Re (kx) for different k and record the number of training epochs to achieve certain error threshold which problem. As one can see in Figure 2 the difficulty changes little to no when we train with H1 and H2 losses while the difficulty increases with & when L2 loss is used. This implies that the difficulty of training barely changes in Sobolev Training even the target function has stiff changes. The same observations are made when solving PDEs. The improvement of our loss functions compare L2 loss function are more dramatic for Burgers' equation (which has stiff solution (Raissi et al.) (019)) than for the heat equation, with the initial condition of f2 (which has a higher frequency) than with the initial condition of fi initial condition in the Fokker-Planck equation, and as k increases for the high-dimensional Poisson equation
be regarded
a difficulty of the
5.2
THE HEAT EQUATION & BURGERS' EQUATION
We now demonstrate the results of the Sobolev Training of the neural networks for solving PDEs. We begin with the I-D heat equation, and Burgers' equation, which is the simplest PDE that combines both the nonlinear propagation effect and diffusive effect. Burgers' equation often appears

---

Under review as a conference paper at ICLR 2021
12000
Figure 2: Average number of epochs to make error less than 10-3 increases in 1.2 loss k increases. However, when we use H1, and H2 losses, required number of epochs increases much more slowly or stays the same increases.
simplification of a more complex and sophisticated model, such as the Navier-Stokes equation. The equations with the homogeneous Dirichlet boundary condition read as follows:
The Heat equation
Burgers' equation
U - Ugr = 0 in (0, 10] x [0, T.. u(0, x²) sin (z) on [0, T).
Ut uu,, 0.2ugz = 0 in (0, 0.011 x [0, I],
u(0, x)) sin(mx) on [0. 1]. u(t,x) = 0 on [0, 0.01] x {0,
u(t, x)) = 0 on [0, 10 x x)
The heat equation attains unique analytic solution u(t, = sin(x:) exp(-t); an analytic solution of Burgers' equation is provided in Basdevant al. (1986).
Although Sirignano & Spiliopoulos/ (2018) indicated that iterative random sampling reduces the computational cost, we fixed the grid points before training because we aimed to compare the effi- ciency of our loss function with that of the traditional one. For the heat equation and the Burgers'
equation, we uniformly sampled the grid points {t,, 12, -ijj=1 Ne.N= from (0, T) x Ω where N₂ and N₂ denote the number of samples for interior and æ; respectively. For the initial and boundary condi- tions, we sampled the grid points from {( = 0, 0, E (0) Z and {4, xy f.j== [0, 0 respectively, where NB denotes the number of grid points in 0Ω. Here, set No, N NB 31. The testing data were also uniformly sampled from the domain of the PDEs.
Nr,No
The L2, H1, and H2 losses are the Monte-Carke approximations of tively, for the heat equation and Burgers' equation. Working on achieving a smooth solution, we observed that the H2 loss performed the best, followed by the H1 loss and then the 1.2 loss in both accuracy, and computation time. We show the corresponding results in Figure
and (4.3) respec-
5.3 THE -IEKER-PILANC EQUATION
The kinetic Pokker-Planck equation describes the dynamics of particle whose behavior is similar to that of the Brownian particle. The Fokker-Planck operator has strong regularizing effect not just in the velocity variable but also in the temporal and the spatial variables by the hypoellipticity. The Fokker-Planck equation has been considered in numerous physical cireumstances including the Brownian motion described by the Uhisten processes.
We provide two simulation results for different initial conditions for the 1-D Fokker-Plande equation with the periodic boundary condition. For the Pokker-Plande equation, sampling from Hwang et al. (2020). Because it is practically difficult to consider the entire space for the R variable, truncated the space for v as 15,, 5). We then uniformly sampled the grid N..NN. from (0, T] [-5, where N, denotes the number of samples for v. The grid points for the initial and periodic boundary conditions were accordingly sampled. The truncated equation reads as follows:
adopted the idea of
points (t,, Zy, Ut ) fij,k=1
Uug B(vu), quler
0. for (L, v) E (0, x 00 x [-5,5]. 5], a0, I v) = f(a, v), for (x, v) E [0, x [ 5],
8 ut 1, 1, 80 2 ut, O, v) O. for (t, e) E [0.3] x [-5,5].

---

Under review as a conference paper at ICLR 2021
The Hunber trace
The heat
Surraar Spocts
Hunde
Lest Vaidion
Figure 3: First row: results for the heat equation. Second row: results for Burgers' equation. First column: Histograms for the heat and Burgers' equation generated from hundred neural networks for each loss function. Second column: Test L T; (2())) errors. Third column: Average training time for each loss function to achieve certain error threshold. Error bars are for standard deviations. The thresholds for the error are set to 10-5.
(1+cos(2nz)) exp( where f(x, v) is either fi(, PG exp(-19)do" or fa(2, u) L2, (1(eo()()) expl - dduds
exp(-o*)
and B 0.1, 0.1.
A numerical solution on the test data was computed by method shown by Wollman & Ozizmir (2008) and used for computing the test error. L2 loss and loss denote the Monte-Carlo approx- imations of (4.4) and (.5),, respectively. The values of Nr, Nu, and N, were set to be 31, and the grid points were uniformly sampled. Expectedly, a solution of the Fokker-Planee equation could be estimated substantially faster using our loss function in both cases. We have provided the detailed results in Figure 4
Figure 4: First row: results for f1 initial condition. Second row: results for f2 initial condition. First column: Histograms generated from hundred neural networks for each loss function. Second column: Test (0, T; (.(()) errors. Third column: Average training time for each loss function achieve certain error threshold. Error bars for standard deviations. The thresholds for the errors for the initial conditions fi (z, v), and f2 v) are set to 10 and 10-3, respectively.
5.4
THE HIGH-DING DIHH-IIIENSIIAL POISSON EQUATION
The Poisson equation serves as an example problem in the recent literature; see Weinan & Yu (011);;
Hsieh et al. (2019) Zang et al. (2020). In this section, provide empirical results demonstrate
8

---

Under review as a conference paper at ICLR 2021
that the proposed loss functions perform satisfactorily when equipped with iterative sampling for solving high-dimensional PDEs; see Sinignano & Spiliopoulos (2018) for more information. Con-
vergence result similar to those of in section 4 for the Poisson equation is given in section [A.4] We consider the following high-dimensional Poisson equation with the Dirichlet boundary condition:
d
π
π sin Ti), for x E Ω = (0, 1),,
Au
4
2
iml
π sin Ui), for x E 82,
2
where x = (r1, "e,,,,,,...................... Ta) E Q. One can readily prove that u(x:) = Ei= sin (x;)) ) is strong
solution. We compare the following three loss functions with each other:
LOSSTOTAL
Poisson))
(Wan) = LOSSGE (Unn: 0,2) + Loss BC (unn 0, 2),
(5.1)
Poisson)) (unn) LossaE unn 1, 2) + Loss (Unn 0, 2),
Loss TOTAL AL
(5.2)
(2.poi))))) (unn) LOSSGE ( 1, 2) + Loss (Unn; 1,
LosSTOTAL AL
(5.3)
Notably, the aforementioned loss functions have the variable æ only. Table [ presents the relative errors on a predefined test set for d 10, 50, and 100. Evidently, in all cases, the proposed loss functions outperform the traditional L loss function. More detailed experimental results are given in section
Average of the relative errors of hundred neural networks for the high-dimensional Pois- son's 's equations. We uniformly sampled 500 data points from Ω for each epoch and trained the neural networks in 10000 epochs at learning rate 10
Table 1:
Dimension
LoSSTOTAL
(U: Poisson)
(1:Poisson) LoSSTOTAL
LosSTOTAL
(2:Poisaon)
10
0.38%
0.22%
0.22%
50
2.00%
1.74%
1.52%
100
3.15%
3.06%
2.89%
DISCUSSION AND CONCLUSION
Inspired by Sobolev Training, we proposed novel loss functions, which efficiently guided the training of neural networks for solving PDEs. We theoretically justified that the proposed loss functions guaranteed the convergence of a neural network to solution PDEs in the corresponding Sobolev spaces. We also discussed that the proposed theorems imply that the training becomes Sobolev Training by slightly modifying the loss function, although the process of estimating neural network solutions of PDEs is not fully supervised.
In addition to the toy examples, which showed the exceptional speed of Sobolev Training, we pro- vided empirical evidences demonstrate that loss functions expedited the training more than the traditional T loss function. We believe that this can solve the problem associated with the high costs involved in estimating the neural network solutions of PDEs. Moreover, our experiments on high-dimensional problems showed that the proposed loss function performed better when equipped with iterative grid sampling. The histograms in Figure 1 indicate that our loss function provided more stable training in that it reduced the variance in the distribution of the number of epochs (e.g., for the Burgers' equation, L2 loss: 3651=812, H1 loss: 995171, and H2 loss: 331+15). Thus, the training, when governed by our loss function, became robust to the random initialization of the weights.
REFERENCES
Cea Basdevant, M Deville, P Haldenwang, JM Lacroix, J Ouazzani, R Peyret, Paolo Orlandi, and AT Patera. Spectral and finite difference solutions of the Burgers equation. Computers Fluids, 14(1):23-41, 1986.

---

Under review as a conference paper at ICLR 2021
Atlim Günes Baydin, Barak A Pearlmutter, Alexey Andreyevich Radul, and Jeffrey Mark Siskind. Automatic differentiation in machine learning: a survey. The Journal of Machine Learning Re- search, (1:555555-5677, 2017,
Yassine Benia and Boubaker- Khaled Sadallah. Existence of solutions to Burgers equations in do- mains that can be transformed into rectangles. Electron. J. Differential Equations, pp. Paper No. 157, 13, 2016.
Jens Berg and Kaj Nyström. A unified deep artificial neural network approach to partial differential equations in complex geometries. Neurocomputingg 317:28 41, 2018.
George Cybenko. Approximation by superpositions of a sigmoidal function. Mathematics of Con- trol, Signals and Systems, 2(4):303-314, 1989.
Wojeiech M Czarnecki, Simon Osindero, Max Jaderberg, Grzegorz Swirszez, and Razvan Pascanu. Sobolev training for neural networks. In Advances in Neural Information Processing Systems, pp. 4278 2017.
2017.
Lawrence C. Evans. Partial differential equations. American Mathematical Society, Providence, R.I., 2010. ISBN 11880221849753 0821849743.
Jiequn Han, Arnulf Jentzen, and E Weinan, Solving high-dimensional partial differential equations using deep learning. Proceedings of the National Academy of Sciences, 115(34))8505-8510, 2018.
Kurt Hornik, Maxwell Stinchcombe, Halbert White, et al. Multilayer feedforward networks universal approximators. Neural Networks, 2(5):359-366, 1989.
Jun-Ting Hsieh, Shengjia Zhao, Stephan Eismann, Lucia Mirabella, and Stefano Ermon. Learning neural PDE solvers with convergence guarantees. arXiv preprint 1906.0:90000///0 2019.
Hyung Ju Hwang, Juhi Jang, and Jaewoo Jung. The Fokker-Planck equation with absorbing bound- ary conditions in bounded domains. SIAM J. Math, Anal., 50(2))2194-22233 2018. ISSN 0036- 1410. doi: 10.1137/16M1 109928. URL
Hyung Ju Hwang. Juhi Jang, and Juan J. L. Velázquez. On the structure of the singular set for the kinetic Fokker-Planck equations in domains with boundaries. Quart. Appl. Math., 77(1):1--70, 2019. ISSN 0033-569X. doi: 10.1090/gam/1507. (RL/ttps:://oo..org 100 1090/qaml 1507
Hyung Ju Hwang, Jin Woo Jang, Hyeontae Jo, and Jae Yong Lee. Trend to equilibrium for the kinetic Fokker-Pii Planck equation via the neural network approach. Journal of Computational Physics, pp. 109665, 2020.
Hyeontae Jo, Hwijae Son, Hyung Ju Hwang, and Eun Heui Kim. Deep neural network approach to norward-inverse problems. Networks & Heterogeneous Media, 15(2):247-25) 2599, 2020.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Isaac E Lagaris, Aristidis Likas, and Dimitrios I Fotiadis. Artificial neural networks for solving ordinary and partial differential equations. IEEE Transactions on Neural Networks, 9(5)9887- 1000, 1998.
Isaac Lagaris, Aristidis C Likas, and Dimitris Papageorgiou. Neural-network methods for boundary value problems with irregular boundaries. IEEE Transactions on Neural Networks, 11 15):1041-1049 2000.
Xin Li. Simultaneous approximations of multivariate functions and their derivatives by neural net- works with one hidden layer. Neurocomputing, 12(4):3273343, 1996.
Zichao Long, Yiping Lu, Xianzhong Ma, and Bin Dong. Pde-net: Learning pdes from data. In International Conference Machine Learning, pp. 3208-3216, 2018.
10

---

Under review as a conference paper at ICLR 2021
Adam Paszke, Sam Gross, Francisco Massa, Adam Lever, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. In Advances in Neural Information Processing Systems, pp- 8024 8035, 2019.
Maziar Raissi, Paris Perdikaris, and George Karniadakis. Physics-inffrmed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378:686-707, 2019.
Yeonjong Shin, Jerome Darbon, and George Em Karniadakis. On the convergence and generaliza- tion of physics informed neural networks. arXiv preprint arXiv:2004.01886 2020.
Justin Sirignano and Konstantinos Spiliopoulos. Dgm: deep learning algorithm for solving partial differential equations. Journal of Computational Physics, 375: 2018.
Remco van der Meer, Cornelis Oosterlee, and Anastasia Borovykh. Optimally weighted loss func- tions for solving pdes with neural networks. arXiv preprint arXiv:2002.06269, 2020.
Weinan and Bing Yu. The deep ritz method: a deep learning- numerical algorithm for solving variational problems. Communications in Mathematics and Statistics, 6(1):1-12, 2018.
Weinan, Jiequn Han, and Amuuf Jentzen. Deep learning- beaneng-bbased numerical methods for high- dimensional parabolic partial differential equations and backward stochastic differential equa- tions. Communications Mathematics and Statistics, 5(4):349 380, 2017.
Stephen Wollman and Ercument Ozizmir. A deterministic particle method for the vlasov-fokker- planck equation in one dimension. Journal of Computational and Applied Mathematics, 213(2): 316-365, 2008.
Yaohua Zang, Gang Bao, Xiaojing Ye, and Haomin Zhou. Weak adversarial networks for high- dimensional partial differential equations. Journal of Computational Physics, pp. 109409, 2020.
A PROOFS FOR THE THEOREMS IN SECTION
We begin with the basic definitions of Sobolev spaces. We excerpt the definitions from Evans 2010). Definition A.1. Suppose u. E (U) and a multindex. We say that the a weak derivative of written
Ddu = v.
provided
uDa ødx = (-1)lal [ vodr,
for all test functions & E (U).
Now define the Sobolev space. Fix S S so and let k be nonnegative integer. Definition A.2. The Sobolev space w consists of all locally integrable functions u U R such that for each multiidex a with la| < k. D exists in the weak sense and belongs to (U)... Note that if p 2, usually write H (U)) ?
(U) (k 0, 0,1,
The Sobolev norms are defined as follow: Definition A.3. If E WU.((U), we define its norm to
[ Einsk So (dx) (1 (1<p oo),
[ulllw+>(0)
Σ Iasss Css supr [Daul (p=
Finally, we define notion of convergence in Sobolev spaces. Definition A.4. Let ( (um 100=1 E Wkkp(U) We say um converges to u in Wk.P(U) provided
lim Imm ullwwwrry
m m0000
11

---

Under review as a conference paper at ICLR 2021
A.1 THE HEAT EQUATION
We denote the strong solution of the heat equation
Uj Uzz = 0 in (0, T x Q
u(0, x) 14p (1) f2
u(t, x) = O on [0, T) x an,
by and the neural network solution by Van- Then, t = u Uun satisfies:
U Ves = [(1, a) in (0, 7) x Ω
v00 x ((x) on S2
(A.1)
vt, x) = 0 on [0, T) x an,
for some f, and g. Here, we can set the boundary to be zero by multiplying B(x), where B(x) is smooth function satisfying (xx)
0,
3r E On
Then the following holds:
+ O,
Ω2
Theorem A.S. (Theorem 7.1.5 Evans 2010 If'g E (" L2)))), then,
0stST [|e((t (((1)f111220) Il9ll=>(0))
max
(A.2)
ess sup | (=(11f1le200: + +1olla
(A.3)
0SigT
sup [|]((()))) s
(A.4)
0S1ST
for some C1, C22 C3-
By applying above theorem to (A.I). we get the results of Theorem 4.1
Remark A.6. The left hand sides (A.2) (A.4) are the ervors of neural networks in corresponding norms, and the right hand sides are the losses (4.1) (4.3) for the heat equation, respectively. This implies that the proposed loss functions are the upper bounds of the errors the Sobolev spaces, and
by minimizing them, we can expect the effect of Sobolev Training when solving PDEs with neural networks.
In the rest of this section, we will show the similar results for Burgers' equation and the Fokker- Planck equation.
A.2 BURGERS' EQUATION
We consider the strong solution u of the following Burgers equation in a bounded interval Ω [a, b).
O.u udzu 82 = in Ω
(A.5)
0 on 82
(A.6)
and the corresponding neural network solution Man satisfying
i Unn + Unn Og Unn - 02 Unn f in 82
(A.7)
Unn on 8Ω
(A.8)
with the inital data u(O,-) and unn (0, respectively.
The following proposition ensures the existence of a strong solution to the initial boundary value problem (A.5)- (A.6) (see Benia & Sadallah (2016)). Here, we multiply ((x) to (n, in order to meet the boundary condition. We use the notation X where the relation A S B stands for S CB, where C denotes generic constant.
Proposition A.7. (Theorem 1.2 in Benia & Sadallah (0166)) Let uo E H3. Then there exists time T* =T* > such that the problem A.5 (A66) with intiill data ug has a unique solution of u satisfying
(2))) (1)
"4 Furthermore, ifT < 00, then |||||| H4())
T
12

---

Under review as a conference paper at ICLR 2021
We will show that the following theorem holds.
Theorem A.8. Let and thn be strong solutions of [A.5) A.6) and {A.7 A.8) respectively, on the time interval [o, T. For w == Unn u. following statements are valid.
(1) There exists a continuous function
Fo Fo (Iu(())) )BB. ! I1f1ade, f( 110.ulBat
such that
sup 1w012 + S 110,w|ldt Fo + 0S1ST
as
||| -12 /" 11 - O.
(2)
There exists a continuous function
F Fi (1oo0. [-" I1sIBel, S'" 1
such that
sup | + S'"d 110.0lim 11: - F1 +0, 051ST
(A.9)
as
(10(0,))1B3,,, /-" |15 O.
33)
There exists a continuous function
F2 = F2 (wu(0,))fa /-" 11.112 + 110df sup |10,ul13 + S ||dru (2dt
0515T
such that
(Ilw11322 + [1wxl()) [' 1 10,, < F2 -
sup 0S1ST
(A.10)
as
(w. { II + 10.f1ldt
Remark A.9. By the Morrey's s embedding theorem and the Poincare's 's inequality, for J E H2))), we have the following inequality,
1 S 115122 1 1/
(A.11)
Throughout the proof, we widely use A.17
Proof. Subtracting A.5 from
get equations of w follows.
ax + ww + wuy + uwl, f in Ω w on M,
(A.12)
(A.13)
w(0,)) -) in Ω.
(A.14)
By multiplying w to (A.12) and integrating by parts in Ω. we have
w 11 + I|w, 12
(A.15)
2 dt
13

---

Under review as a conference paper at ICLR 2021
fw
w² W2
52
w²
5p
wwwe
Jen
ww2:
K-
k=1
Now estimate the terms the right hand side of (A.15). Applying the Young's inequality, the Hölder's inequality, the Sobolev inequality, and thd Poincare inequality, we have
1 - || w la H
|| + e||w|l x I + ellwal12.
(A.16)
1 -- -
12
3 3
" (b)
(A.17)
00
13 < /
(A.18)
2 | 1112 <<wl=>
14 <
S | |1 + e/
(A.19)
5 = wb) - w(a)ws(aa) = O,
(A.20)
for any small c <> 0. Applying estimates 16
A.20 AA.15). we have the following inequality
d
|l + I 12
1 + 1l.
(A.21)
dt
A.21 and the Grönwall inequality imply that
|| 11 [1.112 + L/" [1S1BBt
sup 0515T
||ll| S elo"
(A.22)
Let us denote
I|u Edtt 33
Now we integrate (A.21 between 0 and T and drop the term [Tw on the left-hand side to obtain
I 12 S 11913 /
||u |1 1
(A.23)
Ilall + L. [1r11att)
S (Be +1)
This completes the proof of (1) of Theorem A.8
Next, by multiplying War to (2.12) and integrating by parts in Ω obtain
1 112 + || l12
(A.24)
2 dt
/ fwaz f www / uwwees + / uw Wan + f WeWer
1=1
Similarly to A.
20 we estimate the terms on the right hand side of A.24).
1 < [
113
$
[[
1
1|
(A.25)
7 < ||0 [0 1 ll -
|| || + e|| el| 113-
(A.26)
r} < ||l]| | 1
2A
||
|| [/w 1 + <l Wre 11
(A.27)
1] < ||u] 1l
||
e||
(A.28)
r (.))
(A.29)
14

---

Under review as a conference paper at ICLR 2021
for any small 0. Applying estimates A.25
A.29
A.244)
have the following inequality
12 +
||
12 2
(1
w/y
1 +
11 u 112 1l.
(A.30)
dt
follows from (A.23).
A.33 and the Grönwall inequality that
U/ 11 S edo 1|w wwwll3+1llu. lādt [1aml13 + L/¹ 11f1120t
sup 055T
(A.31)
S eF 11am112++ /-" 1151BBu))
In similar way (A.23). there exists function F -i(lifllla Illll,, such that
W [d S Fi.
(A.32)
(2) of Theorem A.8 follows from AA.31) A..22) and the fact that
W;
+ Wan
WMy Www-
(A.33)
Finally, we differentiate A.112
with respect to t. then we obtain
Wht Wrart
we Wew wwat Welt, Wilzt Wg + uwxl in f2
(A.34)
Wi = 0 on aQ,
(A.35)
wy (0) f + 9ux 99x guau Uo99 in Ω2
(A.36)
By multiplying wa to (A.34) and integrating by parts in Ω2 we have
d dt
wwell + ||wwell2
(A.37)
fewe / wfww= ww/W2t
- / wfuz
Ja Wwllltt / u4 ue Wr
www
Wi Wart
80
8
Terms on the right hand side of (A.37) are estimated by
1 < | [ellll|wella
x f 3 <||wael
1
(A.38)
13 < |rx] 1/..s//wll0 ||
|xx ||
||w
11
M2z 1
(A.39)
13 < ||w]| [wwll Mus l2 S
1
<
1
(A.40)
< //e/l=c//w:lll///w//
[|wwee
(A.41)
13 + 12 ww S
||m|| /1
-
[ 11 w1
(A.42)
I < || w/ |1 wella ||v,
113
||aa 12.
(A.43)
13
(A.44)
for any small e > 0. Applying estimates A.38
A.44
to
A.37
we have the following inequality
w, 12 War
[
I| 12) wl 11 + 11.a.11..
(A.45)
It follows from A.36).
A.45 and the Grönwall inequality that
[/y] S eo [| ua ll [1xx + /-' f
sup
(A.46)
0S1ST
15

---

Under review as a conference paper at ICLR 2021
[160133 + 11 + 119x 11 + [| 19 12 /
where
L
| + || [2d..
In similar way to the proof of (2) of Theorem
A.8 (3) of Theorem A.8 follows from AA.3 and This completes the proof of the Theorem.
A.46).
A.3
THE FOKKER-PLANCE EQUATION
A.3.1
BOUNDARY LOSS DESIGN
Define the loss function for the periodic boundary condition as
Loss BC
Σ
dt
[ du
00 f 1, m; w. T m, ue b){
|a|=1
18 (n 1, U&; m. w, P O, 22; 2aim, w.b) u. (A.47)
~ Ni
|a|=1,i,k
A.3.2 THE PLANCK EQUATION A PERIODIC INTERVAL
In this section, we introduce an T,2 energy method for the Tokker-Planc's equation and introduce regularity inequality for the solutions to the equation. Throughout the section, we will abuse the notation and use both notations 0,u and u, for the same derivative of with respect to z.
We consider the Fokker-Planck equation in a periodic interval [0. L]:
Ut + vug -((w)) - qUee = 0, for (t, x, v) E [0, I] x [0. x R, w00, x. uu(2,, u), for (x. [0, 1] R, and
(A.48)
ut, 1, 8i. u (t, v) = O, for (t, v) E [0,2] x R,
for any 3-dimensional multi-index a such that la I and a given initial distribution uo = un( v). Now consider the Fokker-Planck equation that the corresponding neural network solution uan would satisfy:
(unn)e + ((Unn)z B(vuan)o ((unn) ve f for (t, u) [0, [0, 1] x [-5,5], Unn (0, x, v) = for (x, v) E [0. x [-5,5],
Σ E dt S3 du (0p (. una (t, 1, (a unn (t, 0) S. la|=1"
2
(A.49)
for any 3-dimensional multi-index such that |a|| < 1 and given f f(t, g(x, v). and a constant L 0. Suppose that f. g and / are C functions. Also, we suppose that the priori solutions u and Myn are sufficiently smooth; indeed, we require them to be in 1..1.22
For the a priori solution u and Uan to equation A.48 and equation A.49 assume that |o| suffi- ciently large, then we have that for some sufficiently small > 0,
| utt, +5) 0. Unn (t, (5) (2((0.1) Se, te[o,]]
sup
(A.50)
for |a| < 1 and 0, Also, suppose that
10 u(t, x, +5)], 18 Unn (t, +5)l S C, for some C < oo for |a| < 1 and a = 00, Now we introduce the following theorem on the energy estimates:
(A.51)
16

---

Under review as a conference paper at ICLR 2021
Theorem A.10. Let and unn be the classical solutions equation A.48 and equation A.49
respectively. Then we have
Unn (t) u(t)l1 2(1 +((g-E) //=²'0 [[0.un((s) (s) -0,u(s))
sup OSTST
1. 2 exp (1+
25/32
|T]
[T]
[199 uoll2
[If(s))ldds + 2gcCT,
2z
for any E (0,g), where L, uo, f. 9, m, and C. are given in equation A.48 equation A.51
Proof.
Define def Unn
Then by equation A.48 and equation A.49 w satisfies
Wt + ww2 Sow, qu f for (t, x, v) E [0, 7] w00 x, u) = wi, for (z, v) E [0. 1] x [-5,5]. 5].
[0, 1] [-5,5),
(A.52)
where wo By multiplying to equation A.52 and integrating with respect to dæde, we have
2 dt // /0..xx1-5555
1 d
|w]2drdo + HOLIH wwswdrde
MOUAIT-SS gw wdddd
- -1 (o.1xx(-0001
f fwdrdd + I wow,wdzdu.
Then we take the integration by parts and obtain that
1 d
///
|wI dxdd + 1 S du v(w(t, 1, 1, 2 wit, 0, u)2) 2
2 dt
[0.11xx1-555
[1 1...I11-660
Pdodd
- fddado +
S1 J...iii--ss5
woddwdodu
+ I
wo(t, 5)w(t, 5)dz S
Wolt 2o -5)(t,, -5)dz
1o.1
[0.1]
def
I1 + 12
I4-
We first define
[ de 1,)2 wot,, (,) v(tt,,,0))))
A(t)
2
We now estimate 71-14 on the right-hand side. By the Hölder inequality and Young's 's inequality, we have
41 S |fll2/|w|12 - 2 11
||
where we denote
||all2 def
Mooux drdd...
Similarly, we observe that
25,32
1 < 5,8 www.l///wl - el|we|12
4s 1|01
for sufficiently small
0 as |u|
5. By equation A.50 have
1/3 + 44 s allw, (4,,5)ll22g [|lll,///((tt... 5) w(t, -5)1
+ allua, (t, 5, w. (t, 1 5)llza [| pautt,,-5))ll.. -5)lle3 s 2geC.
17

---

Under review as a conference paper at ICLR 2021
Altogether, have
|wl12 + 2(q e) & 11f + (1+
25/32
11 + A(t) + 2qcC.
dt
25
We integrate with respect to the temporal variable on [0, (] and obtain
[|w(t)] 2(q - E) S. [1),
(11(5)ll++ (1+ 2
(1+
25,3²
1lw(0) +
[1 + A(ss) + 2qec) ds.
By equation A.49 we have
S A(s)ds S
Thus, by the Grönwall inequality, we have
[ 2(q e) S' [s))(s)]
(11obl1
L
exp [1
(1+
25
/ [[f())]dds + 2qeCt,
t
2:
where wo(2, v) g(. u) - (o v). This completes the proof for the theorem.
Regarding the derivatives 0,w and O, w we can obtain the similar estimates as follows. Corollary A.11. Let and Uan be the classical solutions to equation A.48 and equation A.49 respectively. Assume that equation A.57 holds. Then for z = or x. we have
0m Unn (t) 0,u(t)l1 2(g E) [ 110,0, unn (n))s) 0,),u(s)
sup OSEST
$ <1100-00 02u0l
exp [ (1+
25,32
T
[0. + 2qeCT,
2e
for any E E (0.9). q). where L, uo, f. B, 3.q. m, e, and are given in equation A.48 equation A.51
Proof. For both d₂ Ô₁ and On, we take 0 onto equation A.52 and obtain
(0.w), + v(0,w), - Bu(0, w) - w)0,w). = 0 for (1, v) E [0, [0,7]
x 10. 1 x [-5,5],
(0.w) x, v) (0.w)o. for (z,v) E [0, x 5,
(A.53)
where (8.w)o 0.g 0g Then the proof is the same as the one for Theorem A.10 for d_w replacing the role of w. This completes the proof.
Finally, also obtain the regularity estimates for the derivative w, as follows:
Theorem A.12. Let and unn be the classical solutions to equation A.48 and equation A.49 respectively. Assume that equation A.50l holds. Then we have
sup OS1ST 1 (t) 0,u(t)11 + 2(g - =) 1/" |10cr Unn(s) - O.v u(s)l{ds
exp (2+ 25,32
$ (L+ 11ag 0. 122 110.9 u.ul) exp
T
2s
. (10, + [ + 4qeCT,
for any E (0, where L., un, J. B, 4, m, e, and C given in equation A.48 equation A.51
18

---

Under review as a conference paper at ICLR 2021
Proof. we take 0g onto equation A.52 and obtain
(e.w) + wa + v(O,w), Bu(O,w), () on S. for (t, v) E [0. x [0, 1] x [-5,5],
(A.54)
where (D, de 0.g uo By multiplying d, to equation A.54 and integrating with respect to dædv, we have
2 dt 11 [/..1xx(-5.5]
d
1 + S1 S (O,
f1 0.11)x15,33]
[0.1]x[-5.5] (-w + O, 0. wddddr +
Then we take the integration by parts and obtain that
2 di //
1wnl2dade
- - (t,,))))) du
+ 2 |-5,
1
/
(1,2,w))(t,1,,))-
2
[0.1]x1-5.5]
On wwnddov 11 10.iixi-s,51
Bowno
worlt -5),, wenlt,,,-6)m,(tt.a -5)dx Wee (t, 5)w, 5)dz - S.
+ q
[.11]
10.11
-Jooixt-a dowddrdu de 11 + 12 + 11 + 11 Is.
We first define
2 f du (0,, d,,,(t,,,
B(t) del
We now estimate /1-4 on the right-hand side. We estimate Ii-Is on the right-hand side. By the Hölder inequality and Young's 's inequality, we have
11 s |10.flllllwlll - 11 + 11
where we denote
|Al|| def ffo.ii--ss drdd.
Similarly, we observe that
25,32
U2l 5,8 Wee 1elllllw.ll S e|| 11
4s
for sufficiently small & >>> 0 as |u| < 5. By equation A.50 and equation
and
A.51
we have
14g 41 s gllw 5) I., (t, 5) - + (t,, wor (t, -5)1l (t,. - S 2geC.
Finally, we have
[/| < wwsllelwalllz - J + 1,,
Altogether, have
d dt
|/c,112 + 2(g w ||0./lla + || + (2 2
25,5
5L
||w,1122 +
+ 2geC.
2
2
19

---

Under review as a conference paper at ICLR 2021
Then take the integration with respect to the temporal variable [0, t] and obtain that
[lw,(()11+ J, ds 2(1 e))
$ |l,,(0)12 + f(as ds (10.f(6)la+ + (s) ( 22 25,82
||w.(s)l B(s) 200C).
By equation A.49 3. we have
I. ds B(8) S L
2
Thus, by the Grönwall inequality, we have (1)112 + 2(1 - e) L(" (s))gs
2/ 10.3 exp [[2
25,32
$1
+
(1x
110.f + 2qcCt, (A.55)
2:
where (,, v) = g(x,, v) uo( v). (0,u((s) and obtain that
Then we use Corollary A.11 for an upper-bound of
Lo (s)]Bas
exp ( (1+
25,32 2E
T + 1-'
2 + +1 0,20
110xf())lBdds + 2qcCT.
Then by equation A.55 we obtain that
(t) + 2(g - E) / [s)o
sup
0S1ST
25,32
(L 10 (10.w00l)) (2+
T
2=
I.
(1))) + ||.. (+) + 4qeCT,
where 0, (, v) 0.g(x, u) (z v). This completes the proof for the theorem.
A.4 THE POISSON EQUATION We consider the Poisson equation equation with Dirichlet boundary condition:
Au u = f in Ω
g 02
Suppose there exists
g H ( s.t. glon =
(A.56)
Then, the equation can be written by:
Ar in Ω
on an,
where u- f-AffAj Therefore, assume the homogeneous Dirichlet boundary condition provided (A.56).
Now, let be strong solution of
Au f in Ω 0 8n,
(A.57)
20

---

Under review as a conference paper at ICLR 2021
and let uhn be neural network such that
! Unn fro in S2,
(A.58)
Uyn
Here, set the boundary to be zero by multiplying Bxx), where B(x) is smooth function satisfying B(x)
00 { O, x By subtracting (A.58) from (A.57). we get
0,
-uu - Unn) = (f - fnn) in S2
(A.59)
Unn on an.
Then apply below theorem (A.59 to get the convergence results.
Theorem A.13. (Theorem 6.3.5 in Evans (2010)) Let be nonnegative integer. Suppose that u E H)) is a weak solution of the boundary-value problem (A.57). Assume dn Cm++2 Then,
(m+)0))) I C(Ilflllmm + ||l|
(A.60)
Furthermore, if u is the unique solution A.57 then
| + (2) S Cllf ((m)))
(A.61)
By applying (A.61) to (A.59). we obtain
||
Una unall (0) S CII - Smallam(())
where the right hand side corresponds to LOSSGE (Unn; 2,
21

---

Under review as a conference paper at ICLR 2021
B ADDITIONAL EXPERIMENTAL RESULTS
B.1 DEPENDENCY ON LEARNING RATES
In this subsection, we show several experiments that shows the proposed loss functions generally performs better in different learning rates. We first show the results for Burgers' equation. In Figure 5 we show the test errors versus training epochs plot for different learning rates. We used 10-3 10 100 learning rates and we observe that H2 loss performs best followed by HI and L2 loss functions.
Pargeit
Nunder Sppccs
ipacks
Vander lppcts
Figure 5: Test errors as training goes for different learning rates.
We next present the similar experiments for the high-dimension Poisson equation. We trained 30 neural networks with different initializations with different learning rates. The average errors are presented in Figure 6 As the same in the Burgers' equation, our loss functions performs better than the traditional one in all learning rates.
upetmn/e-11
Lour
Loaf
Figure 6: Test errors as training goes for different learning rates.
B.2 HIGH- DIMENSIONAL POISSON EQUATION
In this subsection, consider the high-dimensional Poisson equation with different boundary con- dition. In subsection 5.1, we pointed out that the "difficulty" learning sin(kx) increases as increases. As generalization of the argument, consider the following PDEs:
Au (kT)) 2
sin KT zx;), x;), for S2 (0,
KT xy), for x e an,
sin
for d=100 1,3, and 5. As one can see in Figure the improvement of Sobolev training gets bigger as k increases. This observation coincides with the one in section present the comparison of training time to meet a certain error value for different loss functions in Figure The result shows that it advantageous to use the proposed loss functions in time, even in high-dimensional case.
expected. Moreover, we
22

---

Under review as a conference paper at ICLR 2021
10-9 Polsson equation [
R- 1 Time Plor
15"
Last Losstina Loeetiine
Averape
900
Number Epecha
14580
20046
21000
13382
11 PrAwn
Agenny
Losutaggumn LosaHai2 son Loos
Talk otter
Acerape
Number Epochs
LOAS
Loss
10 Poisson eguation/k=3
Kw Time Plot
Lonatliga Pareen
2429
12P
Antrage
3000
10000
11500
20000
31000
33500
10 Aquare
LOSSTOTAL
Figure 7: Left column: Test errors as training goes for different values of k. Right column: Required Training Time to achieve certain test error.
23


Reviews:
[{"review_id": "kQ7Y3RDaLv", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Overview: \nThe paper proposes a novel loss function using Sobolev norms to decrease the computational costs when solving PDEs using neural networks. I think the idea of the work is very interesting and relevant and in a useful direction, however I think the paper in its current form is not yet suitable for publication and it should be strengthened by incorporating more theoretical and numerical aspects. This will make the concept a lot more convincing. I present some ideas and comments for improvement below. \n\nComments and ideas for improvement as well as clarification questions:\n- You state “it requires relatively high computational cost compared to traditional mesh-based schemes in general”. Would you have a reference for this? In part I agree with the notion that neural network training could be computationally-heavy, but on the other hand, as you also mention, it should not suffer from the curse of dimensionality compared to mesh-based methods which would seem that it is computationally efficient?\n- The main claim of this work is to introduce Sobolev training to speed up convergence, or as you mention in the introduction “overcoming the issue of high computational cost when solving PDEs using neural networks”. Theoretically the results in Section 4 are not showing this. I know that in the original Sobolev training paper there is a result on how Sobolev training has a lower sample complexity than regular training. Extending such a result to this setting would be necessary to make the claims in the introduction rigorous. \n- The results in Thm 4.1 and 4.2 are only for 1D equations. I understand that higher order could be more complex, and perhaps the 1D equations are sufficient to convey the intuition, however in that case at least a comment is needed on how these results could be extended to higher orders. \n- In Figure 1, what is the reason for H2 loss not speeding up convergence with the ReLU? Is it the differentiability? \nThe results in 5.2 are again only for 1D equations. I think that if theoretically you do not prove the results for high-dimensional PDEs, the value of the proposed methodology for high-dimensional PDEs should at least be shown in extensive numerical experiments. I do think the example in 5.4 is in the right direction, but a more rigorous analysis would be needed. \n- I would like to have more information on how the “true” (PDE) values of the gradients of the boundary and interior differential operators are computed, and whether this is always possible. \n- My last comment is a general one, but given that the research in this area is growing rapidly with various approaches to improve convergence, a stronger literature review would be necessary. I give two examples of papers which could be of interest below. \n\nSome references which may be of interest:\n- Ito, Kazufumi, Christoph Reisinger, and Yufei Zhang. \"A Neural Network-Based Policy Iteration Algorithm with Global H^ 2 H 2-Superlinear Convergence for Stochastic Games on Domains.\" Specifically, Remark 4.2 could be of interest. The authors also discuss how certain norms cannot guarantee convergence of the derivatives of the numerical solutions. \n- van der Meer, Remco, Cornelis Oosterlee, and Anastasia Borovykh. \"Optimally weighted loss functions for solving PDEs with Neural Networks.”. The authors discuss the choice of loss functions to also speed up / improve convergence and the solution accuracy. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "IVyktT5pH4_", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The idea of using neural networks to approximate the solutions of the pdes is very interesting, specially in high-dimensional setting where classical approaches fail to scale. Although there has been many efforts in this direction, there are still open venues to explore. One of the most important aspect is the choice of loss function to guide the training of the neural network. And the paper's aim is to address this issue by proposing Sobolev norm as the loss function instead of the commonly used $L^2$-norm. The Sobolev norm includes additional term about derivatives of the error. The main claim is that with the inclusion of the additional term, the convergence of the neural network training becomes faster. This is the basic promise of the paper.  \n\nAlthough an interesting proposal, I think the paper did not address the full aspects of it. In particular,  \n\n1) training with derivatives in the loss function should be costly compared to $L^2$ loss. This becomes worse as the dimension increases. If the dimension is $d$, the first derivative scales with $d$ and the second derivative scales with $d^2$, .... It seems that at most, one can try only the first derivative.   \n2) No comparison is provided in terms of computational time. \n3) Comparing the convergence speed may not be fair, because for $L^2$ loss, one might be able to use larger learning rate which would yield faster convergence, while large learning rate might make Sobolev training unstable. \n4)  Including the derivative requires strong smoothness assumption about the pde data (the forcinig term, the BC, IC) which are not standard in the pde literature. \n\nGiven this, I think the contributions of the paper are incremental. Moreover, the theoretical results do not really support the claim that Sobolev loss function is better. It would be interesting to have a negative result about the L^2 loss function that motivates the application of the Sobolev norm. \n\nMoreover, it seems that the appendix is not written with care. \n\nSubsection A1 does not really include a proof. If the result is already known, it seems better to cite the reference (with exact pointer to the result) in the main body of the paper and do not include it as the contribution. \n\nSubsection A2 also includes the result Porp A.3 without proof or reference. The proof of Thm. A4 is not written with care. The bounds in A19 and A20 are obtained without explaining the steps. What type of Poincare inequality is used? \n\nIt will be good to include a definition of the norms and Poincare inequality for the reader unfamiliar with pde analysis.  ", "extra_scores": {}, "extra_text": {}}, {"review_id": "84Nxj8qxoYm", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Sobolev training of neural networks, which augments the standard loss function with terms that penalize discrepancies between the derivatives of the network and target functions, has been shown empirically to improve data-efficiency. Intuitively, one would expect that it also aids generalization in settings where the target function is sufficiently smooth. This manuscript proposes augmenting the loss functions used to represent the solutions of partial differential equations with terms penalizing the Sobolev norm of the solution, its initial condition, and the boundary condition. The motivation for this approach is clear because data -efficiency is of the utmost importance in PDE learning problems where the data could be very difficult to access. \n\nThe experiments in this paper clearly show that a target accuracy can be achieved with fewer overall training points when using Sobolev training, very much consistent with the established understanding of the effect of penalizing the Sobolev norms in typical supervised machine learning problems.  Some of the examples are high-dimensional and non-trivial. \n\nThe theoretical results are not particularly compelling, but they serve a reasonable justification for the proposed scheme. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper solves a PDE using an additional penalty function between the derivatives of the function. On toy examples and two PDEs it is shown that these additional terms help.

Pros: - The motivation is to include derivatives in the computationa
          - Implementation and testing on several examples, including high-dimensional ones
          - Timing is included in the latest version


Cons: -The loss is Sobolev norm of the residuals of the equation. 
           - The usage of the norm of the residual is not 100% consistent with the smoothness properties of the corresponding equation. For example, for the Poisson equation, the problem is selected in such a way the solution is analytic. However, for example, if the zero boundary conditions are enforced, and right hand side is all ones, the solution will have singularities. Thus, the main challenge would be the case when solution does have the singularities (and it will have it in many practical cases). The L2-norm then is not the right functional for the solution to exist, not to say about the higher-order derivatives. So, these functionals are not motivated by the theory of the solution of PDEs, but are rather focused on much smoother solution.
    - Convergence. There are quite a few papers on the convergence of DNN approximations to solution of PDEs. The presented methods might have converged to a local minimum. An important reference is the paper by Yarotsky D. Error bounds for approximations with deep ReLU networks. Neural Networks. 2017 Oct 1;94:103-14.
  
Decision: Reject