Title: MESH-FREE UNSUPERVISED LEARNING-BASED PDE SOLVER OF FORWARD AND INVERSE PROBLEMS

Abstract: We introduce a novel neural network-based partial differential equations solver for forward and inverse problems. The solver is grid free, mesh free and shape free, and the solution is approximated by a neural network. We employ an unsupervised approach such that the input to the network is a points set in an arbitrary domain, and the output is the set of the corresponding function values. The network is trained to minimize deviations of the learned function from the strong PDE solution and satisfy the boundary conditions. The resulting solution in turn is an explicit smooth differentiable function with a known analytical form. Unlike other numerical methods such as finite differences and finite elements, the derivatives of the desired function can be analytically calculated to any order. This framework therefore, enables the solution of high order non-linear PDEs. The proposed algorithm is a unified formulation of both forward and inverse problems where the optimized loss function consists of few elements: fidelity terms of L 2 and L ∞ norms that unlike previous methods promote a strong solution. Robust boundary conditions constraints and additional regularizers are included as well. This setting is flexible in the sense that regularizers can be tailored to specific problems. We demonstrate our method on several free shape 2D second order systems with application to Electrical Impedance Tomography (EIT), diffusion and wave equations.

1. INTRODUCTION:

Partial differential equations are fundamental in science and mathematics with wide applications in medical imaging, signal processing, computer vision, remote sensing, electromagnetism, economics and more. Classical methods such as finite differences, finite volume and finite elements are numerical discretization-based methods where the domain is divided into a uniform grid or polygon mesh. The differential equation is then reduced to a system of algebraic equations. These methods may have some limitations: the solution is numeric and may suffer from high condition number, highly dependent on the discretization and even the second derivative is sensitive to noise, see for example Garcia (2017) ; LeVeque (2007) ; Thomas (1995) ; Reddy (2005) and references therein.

In the last few years, deep learning and neural network-based algorithms are extensively used in pattern recognition, image processing, computer vision and more. Recently, the deep learning approach had been adopted to the field of PDEs as well by converting the problem into a machine learning one. In Supervised learning, the network maps an input to an output based on example input-output pairs. This strategy is used in inverse problems, where the input to the network is a set of observations/measurements (e.g. x-ray tomography, ultrasound) and the output is the set of parameters of interest (tissue density etc.) Feigin et al. (2018) ; Lucas et al. (2018); McCann et al. (2017) ; Seo et al. (2019) . Unsupervised learning on the other hand is a self-learning mechanism where the natural structure present within a set of data points is inferred.

Algorithms for forward and inverse problems in partial differential equations via unsupervised learning were recently introduced. The indirect approach utilizes a neural network as a component in the solution. Li et al. (2018) for example, proposed the NETT (Network Tikhonov) approach to inverse problems. NETT considers regularized solutions having a small value of a regularizer defined by a trained neural network. Khoo & Ying (2018) introduced a novel neural network architecture, Switch-Net, for solving the wave equation based inverse scattering problems via providing maps between the scatterers and the scattered field. Han et al. (2018) developed a deep learning-based approach that can handle general high-dimensional parabolic PDEs. To this end, the PDEs are reformulated using backward stochastic differential equations. The latter is solved by a temporal discretization and the gradient of the unknown solution at each time step is approximated by a neural network. Li et al. (2019) approximate the inverse solution map of linear and nonlinear problems directly by a deep network.

Direct algorithms solve the forward and inverse problem PDEs by directly approximating the solution with a deep neural network. The network parameters are determined by the optimization of a cost function such that the optimal solution satisfies the PDE, boundary conditions and initial conditions. Chiaramonte & Kiener (2017) addressed the forward problem by constructing a one layer network which satisfies the PDE within the domain. The boundary conditions were analytically integrated in the cost function. They demonstrated their algorithm on the Laplace and hyperbolic conservation law PDEs. Sirignano & Spiliopoulos (2017) proposed a deep learning forward problem solver for high dimensional PDEs. Their algorithm was demonstrated on the American option freeboundary equation. Raissi et al. (2017) focused on continuous time models and solved the Burgers and Schrödinger equations, and Xu & Darve (2019) introduced a novel direct method for the inverse problem and demonstrated their algorithm on various PDEs.

In this work we address the forward and inverse PDE problems via a direct unsupervised method.

Our key contributions are four fold: (1) inverse problems can be solved in the same framework as the forward problems. (2) In the both forward and inverse parts we extend the standard L 2 -based fidelity term in the cost function by adding L ∞ -like norm. Moreover, (3) some regularization terms which impose a-priori knowledge on the solution can be easily incorporated. (4) Our construction exemplifies the ability to handle free-form domain in a mesh free manner.

Point (1) is essential for full tomography-like solutions as will be explained in the sequel. The extension of the loss function by the L ∞ -like norm is fundamental. This term promotes a strong solution of the PDE. The L 2 term, used in recent studies, aims only for weak solutions of the PDE. Weak solutions may differ from the strong solutions by a set of isolated points where the function is not continuous. This is not a merely theoretical issue but strongly affects the quality of the result as we empirically demonstrate. In unsupervised learning of ill-posed problems regularization is crucial. Choosing the right regularizer and the ability to incorporate it in the formulation is of prime importance. Our formalism integrates such regularizations in a natural way. We demonstrate our algorithm by a second order elliptic equation, in particular the Electrical Impedance Tomography (EIT) application on a circular and three other arbitrary domains. We additionally solve the inverse problem of diffusion and wave equations.

2. MATHEMATICAL FORMULATION:

Let Ω be a bounded open and connected subset of R d where d is the dimension. A differential operator L acting on a function u(x) : R d → R is defined as

EQUATION

where D n is the nth order d-dimensional derivative and a 0 (x), . . . , a n (x) are the coefficients. Consider the partial differential problem with Dirichlet boundary conditions

EQUATION

where f (x) : R d → R is a given function. The forward problem solves u(x) given the coefficients θ := {a 0 (x), . . . , a n (x)} while the inverse problem determines the coefficients set θ given u(x).

The proposed algorithm approximates the solutions in both problems by neural networks u(x; w u ), {a j (x; w aj )} such that the networks are parameterized by w u , {w aj }, and the input to the network is x ∈ R d . Figure 1 depicts a network architecture of u in R 2 . The network consists of few fully connected layers with tanh activation and linear sum in the last layer. The network is trained to satisfy the PDE with boundary conditions by minimizing a cost function. In the forward problem and in the inverse problem

EQUATION

EQUATION

)

The first two terms enforce the solution to satisfy the equation. The first term minimizes the error in the L 2 sense while the second term minimizes the maximal error. Ideally, the solution to the PDE has to be satisfied at every point x. This is considered as a strong solution. In the deep learning framework the optimization is carried out on mini batches of points. The first term minimizes the equation in the integral or average sense and is therefore insensitive to a point jump in the value of the integrand. It therefore yields a weak solution. This manifests itself by point discontinuities of u(x) that are empirically seen as well. The L ∞ term promotes a strong solution since it handles possible discontinuous points where the value of the week solution is different than the strong one. We consider this discrepancy as an error. Practically, we use a relaxed version of the L ∞ norm where we take the top-K errors. Then, at every mini batch the training procedure minimizes the equation at K different points having the largest errors. The third term imposes boundary conditions where u 0 and θ 0 are the boundary values of u(x) and θ(x). We used the L 1 norm on the boundary so that the solution is robust to measurements noise. The last term is a regularizer which can be tailored to the application. There are few advantages of this setting. First, the solution is a network and therefore an explicit function. It is a smooth analytic function and is therefore analytically differentiable. It was recently proven that for such functions, e.g. Lipchitz with constant ν, the approximation of the function by a fully connected network Γ is given by

u(x) -Γ L p ([0,1] d ) ≤ 40ν √ dN -2/d L -2/d ,

where p ∈ [0, ∞), N number of neurons in a layer, and L number of layers Shen et al. (2019) .

In addition, the proposed framework enables setting a prior physical knowledge on the solution by designing the regularizers R F and R I . Moreover, the training procedure is mesh free. Unlike finite differences or finite elements methods, we use random points in the domain and its boundary in the course of the optimization of equation 3 and equation 4, see Figure 2 . This means that the solution does not depend upon a coordinate mesh and we can also define an arbitrary regular domain Ω.

3. APPLICATION TO ELECTRICAL IMPEDANCE TOMOGRAPHY:

Consider the following elliptic equation which is a special case of equation 1,

EQUATION

where ∇ := (∂/∂ x , ∂/∂ y ). We assume that 0 < σ(x) ∈ C 1 (Ω), which guarantees existence and uniqueness of a solution u ∈ C 2 (Ω) Evans (2010) . The elliptical system equation 5 was addressed by Siltanen et al. (2000) in the context of Electrical Impedance Tomography (EIT) which is a reconstruction method for the inverse conductivity problem. The function σ indicates the electrical conductivity density, and u is the electrical potential.

An electrical current

ψ n,ϕ = σ ∂u n ∂ν ∂Ω = 1 √ 2π cos(nκ + ϕ), n ∈ Z (6)

is applied on electrodes on the surface ∂Ω, where κ is the angle in polar coordinate system along the domain boundary, n is the current frequency, ϕ is the phase and ν is the normal unit. The resulting voltage u| ∂Ω = u 0 is measured through the electrodes. The conductivity σ is determined from the knowledge of the Dirichlet-to-Neumann map or voltage-to-current map

Λ γ : u| ∂Ω → σ ∂u n ∂ν ∂Ω ,

Mueller & Siltanen (2012); Alsaker & Mueller (2018) ; Fan & Ying (2019) .

We demonstrate our framework by solving the forward and inverse problem of equation 5 which is a first step towards a full tomography. Following Mueller & Siltanen (2012) , we simulate the voltage measurement u| ∂Ω by the Finite Elements Method (FEM) given three variants of a conductivity phantom σ(x) depicted in Figure 3 . We calculate the FEM solution with different triangle mesh densities and select as ground truth the one such that finer meshes do not improve the numerical solution. With our suggested method, the forward problem determines the electrical potential u in the whole domain Ω given σ, while the inverse problem uses the approximated u and calculates the conductivity σ given that σ| ∂Ω = σ 0 .

4. FORWARD PROBLEM:

In the forward problem the conductivity σ(x i ) and boundary conditions u 0 (x b ) are given for random points set

{x i } ∈ Ω ⊂ R 2 , {x b } ∈ ∂Ω ⊂ R 2

with sets size of N s and N b respectively. A neural network having the architecture shown in Figure 1 approximates u(x). Let

EQUATION

The cost function equation 3 is then rewritten as

F u(x; w u ) = λ N s Ns i=1 |L i | 2 + µ K k∈top K (|Li|) |L k | + 1 N b N b b=1 u(x b ) -u 0 (x b ) + α w u 2 2 . (8)

The first term is the L 2 norm of the differential operator, the second term is a relaxed version of the infinity norm where we take the mean value of the top-K values of |L i |. There is some balance between small K which accounts for few error points and can have negligible effect in the training procedure, and a large K which resembles the effect of the L 2 norm. In all our experiments we empirically selected K = 40 as an intermediate value. The third term imposes the boundary conditions and the last term serves as a regularizer of the network parameters. The first phantom is shown in Figure 3 left. The impedance values associated with the background ellipses and circle were 1, 5 and 2 respectively. The original piecewise constant function σ was slightly smoothed by a Gaussian kernel.

Figure 10 in the Appendix shows the forward problem results for current ψ with n = 1 and ϕ = π/8. The left column is the FEM solution which is referred to as ground truth, where the top row indicates the solution u(x) and the bottom row the derivative of u(x) with respect to the first coordinate x calculated as the finite difference approximation of the FEM result. The middle column depicts the outcome of the proposed method where ∂u/∂x is an analytical derivative of our result. The right column shows the outcome of the DGM method Sirignano & Spiliopoulos (2017) which is a special case of equation 3 with λ = 1, µ = 0, L 2 norm of the boundary conditions constraint with no regularizers. Quantitative results of the mean square error (MSE) and peak signal-to-noise ratio (PSNR) are summarized in Table 1 . Clearly, the proposed method outperforms the DGM method since the weighting parameters, the L ∞ norm and the network weights regularization play a significant role in the loss function. 3 . The forward problem was repeated using phantom 2 where the background and circle conductivities were 1 and 0.2 respectively (Figure 3 , middle). Four different current combinations were applied.

Quantitative results are summarized in rows 1-4 of Table 2 . Figure 11 in the Appendix shows the results for both u and ∂u/∂x for n = 2 and ϕ = π/2. The right column presents the relative error defined as e(x, y) = (u fem (x, y) -u(x, y))/max(u fem ).

5. INVERSE PROBLEM FOR EIT:

In the inverse problem, the electrical potential u(x) is known while σ(x) is unknown. Since we have a network which approximates u(x), we can evaluate it at any point x. The objective function equation 4 then takes the form

EQUATION

As in the forward problem, the first two terms enforce σ to satisfy the PDE, where L i is defined in equation 7. The third term imposes the boundary conditions, and the fourth regularizes the network parameters. A physical a priori knowledge regarding σ is exploited in this application. The conductivity is assumed to have well defined sub-regions. We therefore design the fifth term to have sparse edges via the total variation regularization (p = 1).

Additional inverse problem generalization may exploit multiple u approximations for several currents ψ j . The σ calculation thus, simultaneously relies on all {u j }, resulting a more stable solution.

Let

EQUATION

)

Then equation 9 is generalized to 3 and

EQUATION

I σ(x; w σ = j J j σ(x; w σ ) + 1 N b N b b=1 σ(x b ) -σ 0 (x b ) + α w σ 2 2 + β N s Ns i=1 |∇σ(x i )| p .

(12) Adequate reconstruction results by the generalized inverse problem are shown in Figure 4 and rows 1-2 in Table 3 . In both phantoms we used four current combinations ψ n,ϕ : (1, 0), (1, π/4), (2, 0), (2, π/4), and σ 0 = 1.

6. FREE SHAPE GEOMETRY:

We applied the proposed method to arbitrary domains Ω 1 , Ω 2 and Ω 3 , see Figure 5 . The random sample points within the domain and along its boundary can be easily obtained as can be seen in Figure 5 right. Forward problem results applied on phantom 3 are presented in rows 5-10 of Table 2 . Figure 12 in the Appendix shows the results for n = 2 and ϕ = π/4 for the three domains. The outcome of the generalized inverse problem equation 12 is shown in Figure 6 and rows 3-5 of Table 3 .

7. ADDITIONAL INVERSE PROBLEM EXAMPLES:

In this section we exemplify the proposed algorithm with additional inverse problem setting introduced by Xu & Darve (2019) . Unknown parameters which are approximated by neural networks are inferred out of few measurements. The differential operators are approximated in finite differences schemes discretized by ∆t and ∆h, the temporal and spatial grid spacings. In the two following examples we applied the cost function equation 4 with total variation and network parameters regularizations.

7.1. DIFFUSION EQUATION:

Diffusion PDE is given by

∂u(x, t) ∂t = C(x)∇ 2 u(x, t) + f (x, t) (x, t) ∈ [-1, 1] 2 × [0, T ],

where t denotes the time and the conductivity C(x) is the unknown parameter. In the following we solve the equation in 2D where x := (x, y). Given two measurements at adjacent times u(x, y, t 1 ) and u(x, y, t 1 +∆t), the conductivity C(x, y) is approximated as a neural network and is determined as the minimizer of equation 4 with

L(u)-f = u(x, y, t 1 + ∆t) -u(x, y, t 1 ) ∆t -C(x, y) D 2 u(x, y, t 1 ) + D 2 u(x, y, t 1 + ∆t) 2(∆h) 2 -f (x, y, t 1 ),

where D 2 is the central scheme approximation for the Laplacian on 3x3 stencil. The analytical 2D conductivity function is given by ) 2 , and u(x, y, t) = e -π 2 t sin πx cos(πy + π/4), f (x, y, t) = π 2 u(x, t)(2C(x, y) -1).

C(x, y) = 1 + e -(12xy

The measurements were taken at t 1 = 0.1 with ∆t = 0.001 and ∆h = 0.002. The result is shown in Figure 7 . As can be seen the smoothness regularizer and the L ∞ terms increase the reconstruction accuracy. In addition, we added few Gaussian noise levels to the measurements. Quantitative results are summarized in rows 1 -4 of Table 4 . 4 7.2 WAVE EQUATION

The second example is the wave equation given by

∂ 2 u(x, y, t) ∂t 2 = η 2 (x, y)∇ 2 u(x, y, t) ((x, y), t) ∈ [-1, 1] 2 × [0, T ].

The unknown parameter is the velocity field η(x, y). Its analytical value is given by η(x, y) = 0.1 + e -((x-0.5) 2 )+(y-0.1) 2 ) , and the two initial conditions are u(x, y, 0) = e -10x 2 -3y 2 , and u(x, y, ∆t) = e -10(x-0.001) 2 -3(y-0.001) 2 .

Solutions for all time steps are recursively calculated using the finite differences scheme using the correct η(x, y). Given three consecutive values of u(x, y, t) the velocity field η(x, y) is extracted by the minimization of equation 4 where

L(u) = u(x, y, t 1 + ∆t) -2u(x, y, t 1 ) + u(x, yt 1 -∆t) (∆t) 2 -η 2 (x, y) D 2 u(x, y, t 1 ) (∆h) 2 .

Reconstruction result is depicted in Figure 8 showing the advantage of the proposed method. Additional quantitave results of noisy measurements are given in rows 5-8 of Table 4 . 

8. IMPLEMENTATION DETAILS:

The network architecture had 4 layers having 26, 26, 26 and 10 neurons. The algorithm was implemented by TensorFlow Ten (2015) using the ADAM optimizer which is a variant of the SGD algorithm. We used almost the same hyper parameters set in our experiments. Batch size=1000, decaying learning rate starting at 5e -3. The learning rate was factored by 0.8 every 200 epochs, N s = 80000, N b = 1200, λ = 0.01, α = 1e -8, K = 40, µ = 0.01, and β = 0.01. In the diffusion and wave equation case we set λ = 1, β = 1e -4, µ = 1e -4 and α = 1e -5. Running time, for the EIT, on Intel i7-8650u CPU was about 15 minutes for the forward problem and 13 minutes for the generalized inverse problem. For the diffusion and wave equation the running time was about 5 minutes.

Finally, we performed a sensitivity analysis of the algorithm with respect to the number of sample points N s . Figure 9 shows the sensitivity of the forward problem algorithm. As expected, with large number of points we obtain a plateau while with small and intermediate number (up to 50000) the error is not monotonic due to local minima.

9. DISCUSSION:

Deep networks by their nature use compositions of simple functions such as matrix multiplication and non-linear activations like sigmoid or tanh. This structure (i) enables the approximation of an arbitrary function Hornik (1989) and (ii) is inherently differentiable. The network architecture dictates the number of degrees of freedom which in turn enables the expressibility of complex functions. In this work we present a unified framework for the solution of forward and inverse problems in partial differential equations in an arbitrary domain. The algorithm relies on direct approximation of the unknown function by a neural network which yields an analytical smooth solution. The network is trained to satisfy the PDE and boundary conditions in an unsupervised fashion by the minimization of a cost function. The optimization procedure depends on random points set within the domain and its boundary. The problem is therefore mesh free with free-form domain. While recent works formulate the cost function in both forward and inverse problems in the L 2 sense, we introduce a broadened framework. The relaxed L ∞ fidelity term was designed to approximate a strong solution and control possible discrepancies between the weak and the strong solutions.

The L 1 norm on the boundary conditions is more robust to measurement noise than the L 2 norm. The additional regularizer (total variation in our examples) has a significant role in the optimization process where known physical properties like smoothness are imposed on the solution. Lastly, the network regularizer controls the network weights and prevents over fitting. These additional terms make the proposed algorithm more accurate and robust compared to other learning-based methods. We also stress the robustness of our approach exemplified by having almost the same hyper parameters set in our experiments.

This framework alleviates several problems of the finite differences and finite elements methods. In particular meshing, discretization and derivatives approximation are solved in a simple and natural way. Numerical solutions of PDEs in an arbitrary domain are of extreme importance, in particular in medical imaging applications. The framework is demonstrated by an elliptic system in R 2 applied to Electrical Impedance Tomography for both forward and inverse problems, diffusion and wave equations. Promising results were achieved for complex and non monotonic functions. Rigorous analysis of the approximation error and its relation to the network architecture and design are under current study. Future research includes also full tomography solution, higher dimensional problems and other classes of PDEs such as dynamic non-linear equations (Burgers, Navier-Stokes etc.) 

:



Figures and Tables:

Figure 1: Network architecture: the point (x, y) ∈ R 2 serves as an input and the value u(x, y) as the output.

Figure 2: Left to right: finite differences grid, finite elements mesh and random points samples used in the proposed algorithm.

Figure 3: Synthetic conductivities σ(x) referred to as phantoms. Left to right: phantom 1, phantom 2 and phantom 3

Figure 4: Reconstructed conductivity σ(x) using the generalized inverse problem. Left: phantom 1, Right: phantom 2. MSE and PSNR are summed up in rows 1-2 of Table3.

Figure 5: Free form shapes. Left to right: Ω 1 , Ω 2 , Ω 3 and sample points of Ω 3 . The red points indicate the boundary.

Figure 6: Reconstructed conductivity σ(x) using the generalized inverse problem given phantom 3 applied on different domains. Left to right: Ω 1 , Ω 2 , Ω 3 . MSE and PSNR are given in rows 3-5 of Table 3

Figure 7: Reconstructed conductivity C(x, y). Left: ground truth, Middle: proposed method, Right: Xu & Darve (2019). MSE and PSNR are given in rows 1-4 of Table 4

Figure 8: Reconstructed velocity field η(x, y). Left: ground truth, Middle: proposed method, Right: Xu & Darve (2019). MSE and PSNR are given in rows 5-8 of Table 4

Figure 9: Sensitivity analysis of the algorithm with respect to the number of sample points.

Figure 11: Forward problem results of u(x) and ∂u(x)/∂x for current frequency n = 2 and phase ϕ = π/2 given phantom 2. Left column: ground truth (FEM). Middle column: proposed method. Right column: relative error. MSE and PSNR are reported in row 4 of Table 2

Figure 12: Forward problem results of u(x) and ∂u(x)/∂x for current frequency n = 2 and phase ϕ = π/4 given phantom 3 applied on different domains. Left to right: Ω 1 , Ω 2 , Ω 3 . MSE and PSNR are reported in rows 6, 8 and 10 of Table 2.

Forward problem results for phantom 1 compared with the DGM methodSirignano &

Table content: <table><tr><td colspan="2">Spiliopoulos (2017)</td><td/><td/><td/></tr><tr><td>n ϕ</td><td colspan="4">u: DGM MSE PSNR MSE u: Proposed PSNR MSE u x : DGM PSNR MSE u x : Proposed PSNR</td></tr><tr><td>1 1 0</td><td>2.50e-1 9.02</td><td>2.38e-4 39.23</td><td>6.01e-6 11.17</td><td>7.11e-9 40.43</td></tr><tr><td colspan="2">2 1 π/8 4.20e-2 14.90</td><td>1.13e-4 40.60</td><td>7.95e-7 19.27</td><td>5.92e-9 40.55</td></tr><tr><td colspan="2">3 1 π/4 5.46e-2 13.54</td><td>1.23e-4 40.03</td><td>1.87e-6 13.36</td><td>5.62e-9 38.58</td></tr><tr><td>4 2 0</td><td>5.12e-2 16.90</td><td>4.50e-5 47.46</td><td>1.64e-6 14.28</td><td>2.81e-9 41.94</td></tr><tr><td colspan="2">5 2 π/4 6.11e-2 6.51</td><td>8.31e-5 35.18</td><td>1.99e-6 11.08</td><td>2.74e-9 39.69</td></tr></table>

Forward problem results of phantom 2 given a circular domain Ω, and phantom 3 with domains Ω 1 , Ω 2 and Ω 3 as defined in Figure5

Table content: <table><tr><td/><td colspan="2">phantom n ϕ</td><td>MSE</td><td>u</td><td colspan="2">PSNR MSE</td><td>u x</td><td>PSNR</td></tr><tr><td>1</td><td>2, Ω</td><td>1 0</td><td colspan="3">2.86e-4 47.43</td><td>1.70e-8 40.61</td></tr><tr><td>2</td><td>2, Ω</td><td colspan="4">1 π/2 1.74e-3 38.90</td><td>1.03e-8 33.79</td></tr><tr><td>3</td><td>2, Ω</td><td>2 0</td><td colspan="3">1.29e-4 45.52</td><td>3.26e-9 41.18</td></tr><tr><td>4</td><td>2, Ω</td><td colspan="4">2 π/2 1.30e-4 45.49</td><td>8.63e-4 37.07</td></tr><tr><td>5</td><td>3, Ω 1</td><td colspan="4">1 π/4 6.42e-5 47.16</td><td>5.64e-9 39.43</td></tr><tr><td>6</td><td>3, Ω 1</td><td colspan="4">2 π/4 1.08e-4 34.03</td><td>2.61e-9 41.32</td></tr><tr><td>7</td><td>3, Ω 2</td><td colspan="4">1 π/4 1.08e-4 44.91</td><td>4.51e-9 40.39</td></tr><tr><td>8</td><td>3, Ω 2</td><td colspan="4">2 π/4 5.93e-5 36.64</td><td>2.74e-9 41.11</td></tr><tr><td>9</td><td>3, Ω 3</td><td colspan="4">1 π/4 1.22e-4 44.37</td><td>9.17e-9 37.31</td></tr><tr><td colspan="2">10 3, Ω 3</td><td colspan="4">2 π/4 1.74e-4 31.96</td><td>2.24e-9 41.99</td></tr></table>

Reconstruction of σ(x) using the generalized inverse problem

Table content: <table><tr><td colspan="2">phantom MSE</td><td>PSNR</td></tr><tr><td>1 1, Ω</td><td colspan="2">4.06e-2 27.90</td></tr><tr><td>2 2, Ω</td><td colspan="2">9.04e-5 40.44</td></tr><tr><td>3 3, Ω 1</td><td colspan="2">1.07e-2 33.67</td></tr><tr><td>4 3, Ω 2</td><td>5.9e-3</td><td>36.27</td></tr><tr><td>5 3, Ω 3</td><td colspan="2">1.12e-2 33.51</td></tr></table>

Inverse problem results for diffusion and wave equations

Table content: <table><tr><td colspan="2">Equation σ noise</td><td colspan="2">Xu &amp; Darve (2019) MSE PSNR</td><td colspan="2">Proposed MSE PSNR</td></tr><tr><td colspan="2">1 Diffusion 0</td><td colspan="2">4.90e-4 39.12</td><td colspan="2">2.04e-4 42.92</td></tr><tr><td colspan="2">2 Diffusion 1e-7</td><td>0.259</td><td>11.89</td><td>0.249</td><td>12.06</td></tr><tr><td colspan="2">3 Diffusion 5e-7</td><td>0.335</td><td>10.77</td><td>0.316</td><td>11.02</td></tr><tr><td colspan="2">4 Diffusion 1e-6</td><td>0.448</td><td>9.50</td><td>0.438</td><td>9.60</td></tr><tr><td>5 Wave</td><td>0</td><td>0.015</td><td>19.01</td><td>0.013</td><td>19.53</td></tr><tr><td>6 Wave</td><td>3e-9</td><td>0.049</td><td>13.92</td><td>0.015</td><td>19.16</td></tr><tr><td>7 Wave</td><td colspan="2">3.5e-9 0.056</td><td>13.34</td><td>0.018</td><td>18.32</td></tr><tr><td>8 Wave</td><td>4e-9</td><td>0.059</td><td>13.11</td><td>0.029</td><td>16.19</td></tr></table>

Peer review (iclr2020):
In this work, the authors propose an approach to solve forward and inverse problems in partial differential equations (PDEs) using neural networks. In particular, they consider a specific type of second-order differential operator combined with Dirichlet boundary conditions, and suggest how neural networks with suitable training objectives can be used to solve both types of problems. For a specific elliptic system relevant for Electrical Impedance Tomography (EIT), they then present numerical results obtained with their method and compare those to high-resolution finite element solutions and previous work. The paper addresses a very general and important problem with wide ranging applications. People have solved PDEs using spectral, finite difference, finite volume or finite element methods for decades and there is a huge body of literate on this subject. The neural network based approach proposed in this paper seems general and simple with encouraging experimental results. However, there are several important points missing in the current paper based on which I would recommend rejecting it. However, I would be willing to increase the score if these points were addressed in sufficient detail. Major comments: 1) Choice of problems There are many important problems in the literature that belong to the general type of PDE considered in this work (e.g. in electrostatics). To appreciate and understand the merits and limitations of the proposed approach better, it would be highly necessary to apply it to a wider range of problems, including ones with known analytical solution. 2) Convergence tests Convergence tests are an important part that is missing in the current paper. In particular, how does the error in the neural-network solution decay with respect to the number of random points used to train the network? Plots comparing a suitable error norm versus the training dataset size would be very informative. Furthermore, what are the conditions, if any, to guarantee convergence to the exact solution? The authors mention in the last paragraph of the Discussion that a more rigorous analysis will be published elsewhere. However, as error analysis is such an integral part of the study, I think it needs to be addressed, at least to some extent, already in the current paper. 3) Related work I don’t think the current work is put in sufficient contrast with existing work. Several related papers are mentioned in the introduction and the experimental section includes one other method (GDM method of Sirignano & Spiliopoulos) for comparison. However, a more thorough discussion on how the current approach complements existing literature on neural network based PDE solvers would be in order. Minor comments: i) There are quite a few typos and grammar mistakes in the paper that need to be fixed. To give a few examples: ‘the natural structure presents’ -> ‘the natural structure present’ ‘Shrödinger’ -> ‘Schrödinger’ ‘approximated by neural network’ -> ‘approximated by a neural network’ ‘on circular and’ -> ‘on a circular and’ ‘forces the equation’ -> ‘enforces the equation’ etc... ii) Eq. (7): There is a parenthesis missing on the LHS. iii) The authors show many plots in Figs 3-9 but don’t comment much on the results in the main text. A lot of the results presented could therefore go into an appendix. I would find it better to present fewer results/plots and discuss those in more detail. iv) It would be good to define the Nabla operator in Eq. 5 for completeness. v) Please put references to equations and papers in parentheses or, at least, separate them otherwise from the sentence, to avoid confusion. For example: - the reference to ‘Evans (2010)’ below Eq. 5, or - the list of dangling references before the last paragraph on page 3, - the reference to Sirignano & Spiliopoulos in the caption of Table 1, etc... vi) Please expand the captions in Fig. 3 to make the figure meaningful as a stand-alone, without having to read the entire text to understand what *phantom* means. The same applies to all other captions. vii) Perhaps it would be clearer to use ‘row’ instead of ‘line’ when referring to the results in the table. viii) In Sec. 7 ‘s’ in ‘Ns’ should be a subscript. ix) Explain what the acronym PSNR means. x) Please elaborate on why you take the top k values in Eq. (7). What happens if you take more/less? xi) First paragraph in Discussion: I am not sure the robustness w.r.t. the hyperparameters is really so surprising given that you apply the solver to very similar problems. If you could show that the same hyperparameters worked on a completely different problem, that would be much more interesting. xii) Please add references for the first paragraph of the introduction. ********************************************************* I increased my rating based on the revisions. *********************************************************