Mesh-Free Unsupervised Learning-Based PDE Solver of Forward and Inverse problems
Authors: Leah Bar
Nir Sochen
Venue:  (2020)
Keywords: PDEs
forward problems
inverse problems
unsupervised learning
deep networks
EIT

Abstract: We introduce a novel neural network-based partial differential equations solver for forward and inverse problems. The solver is grid free, mesh free and shape free, and the solution is approximated by a neural network. 
We employ an unsupervised approach such that the input to the network is a points set in an arbitrary domain, and the output is the
set of the corresponding function values.  The network is trained to minimize deviations of the learned function from the PDE solution and 
satisfy the boundary conditions. 
The resulting solution in turn is an explicit smooth differentiable function with a known analytical form. 
 
Unlike other numerical methods such as finite differences and finite elements, the derivatives of the desired function can be analytically calculated to any order. This framework therefore, enables the solution of high order non-linear PDEs. The proposed algorithm is a unified formulation of both forward and inverse problems
where the optimized loss function consists of few elements: fidelity terms of L2 and L infinity norms, boundary conditions constraints and additional regularizers. This setting is flexible in the sense that regularizers can be tailored to specific 
problems. We demonstrate our method on a free shape 2D second order elliptical system with application to Electrical Impedance Tomography (EIT). 

Under review as a conference paper at ICLR 2020
MESH-FREE UNSUPERVISED LEARNING-BASED PDE SOLVER OF FORWARD AND INVERSE PROBLEMS
Anonymous authors Paper under double-blind review
ABSTRACT
We introduce a novel neural network-bawe partial differential equations solver for forward and inverse problems. The solver is grid free, mesh free and shape free, and the solution is approximated by a neural network. We employ an unsu- pervised approach such that the input to the network is a points set in arbitrary domain, and the output is the set of the corresponding function values. The net- work is trained minimize deviations of the learned function from the strong PDE solution and satisfy the boundary conditions. The resulting solution in turn is an explicit smooth differentiable function with known analytical form.
Unlike other numerical methods such as finite differences and finite elements, the derivatives of the desired function can be analytically calculated any order. This framework therefore, enables the solution of high order non-linear PDEs. The proposed algorithm is unified formulation of both forward and inverse problems where the optimized loss function consists of few elements: fidelity terms of Le and Loe norms that unlike previous methods promote a strong solution. Robust boundary conditions constraints and additional regularizers are included as well. This setting is flexible in the sense that regularizers can be tailored specific problems. We demonstrate our method on several free shape 2D second order systems with application to Electrical Impedance Tomography (EIT), diffusion and wave equations.
INTRODUCTION
Partial differential equations are fundamental in science and mathematics with wide applications in medical imaging, signal processing, computer vision, remote sensing, electromagnetism, economics and more. Classical methods such as finite differences, finite volume and finite elements are numer- ical discretization-basse methods where the domain is divided into uniform grid polygon mesh. The differential equation is then reduced to a system of algebraic equations. These methods may have some limitations: the solution is numeric and may suffer from high condition number, highly dependent on the discretization and even the second derivative is sensitive to noise, see for example Garcia (207)) Veque (2007); Thomas (1995): Reddy (2005) and references therein.
In the last few years, deep learning and neural network-hased algorithms are extensively used in pattern recognition, image processing, computer vision and more. Recently, the deep learning ap- proach had been adopted to the field of PDEs as well by converting the problem into machine learning one. In Supervised learning, the network maps an input to an output based on example onput- pairs. This strategy is used in inverse problems, where the input to the network is set of dbservvatoossmmeasueemnntt (e.g. x-ray tomography. ultrasound) and the output is the set of parameters of interest (tissue density etc.) |eiginn et al. (2018); Lucas et al. (2018); McCann et al. (207)); Seo et al.] (2019). Unsupervised learning on the other hand is self-learning mechanism where the natural structure present within set of data points is inferred.
Algorithms for forward and inverse problems in partial differential equations via unsupervised learn- ing were recently introduced. The indirect approach utilizes neural network as component in the solution. Li et al. (2018) for example, proposed the NETT (Network Tikhonov) approach to inverse problems. NETT considers regularized solutions having small value of regularizer defined by trained neural network. Khoo & (2018) introduced a novel neural network architecture, Switch- Net, for solving the wave equation based inverse scattering problems via providing maps between

---

Under review as a conference paper at ICLR 2020
the scatterers and the scattered field. Han et al. (0188) developed deep learning-based approach that can handle general high- high-dimensional parabolic PDEs. To this end. the PDEs are reformulated using backward stochastic differential equations. The latter is solved by temporal discretization and the gradient of the unknown solution at each time step is approximated by a neural network. [L] et al. (2019) approximate the inverse solution map of linear and nonlinear problems directly by deep network.
Direct algorithms solve the forward and inverse problem PDEs by directly approximating the solu- tion with a deep neural network. The network parameters are determined by the optimization of a cost function such that the optimal solution satisfies the PDE, boundary conditions and initial con- ditions. Chiaramonte & Kiener (2017) addressed the forward problem by constructing a one layer network which satishes the PDE within the domain. The boundary conditions were analytically integrated in the cost function. They demonstrated their algorithm on the Laplace and hyperbolic conservation law PDEs. Sirignano & Spiliopoulos (2017) proposed deep learning forward problem solver for high dimensional PDEs. Their algorithm was demonstrated the American option free- boundary equation. Raissi et al. (2017) focused on continuous time models and solved the Burgers and Schrödinger equations, and Xu & Darve (2019) introduced novel direct method for the inverse problem and demonstrated their algorithm on various PDEs.
In this work we address the forward and inverse PDE problems via a direct unsupervised method. Our key contributions are four fold: (1) inverse problems can be solved in the same framework as the forward problems.
(2) In the both forward and inverse parts we extend the standard I-2-based fidelity term in the cost function by adding -like norm. Moreover, (3) some regularization terms which impose a-priori knowledge on the solution can be easily incorporated. (4) Our construction exemplifies the ability to handle free-form domain in a mesh free manner.
Point (1) is essential for full tomography-like solutions as will be explained in the sequel. The extension of the loss function by the - norm is fundamental. This term promotes strong solution of the PDE. The L2 term, used recent studies, aims only for weak solutions of the PDE. Weak solutions may differ from the strong solutions by set of isolated points where the function not continuous. This is not a merely theoretical issue but strongly affects the quality of the result as we empirically demonstrate. In unsupervised learning of ill-posed problems regularization is crucial. Choosing the right regularizer and the ability to incorporate it in the formulation of prime importance. Our formalism integrates such regularizations in natural way. We demonstrate algorithm by a second order elliptic equation, in particular the Electrical Impedance Tomography (EIT) application on circular and three other arbitrary domains. We additionally solve the inverse problem of diffusion and wave equations.
2
MATHEMATICAL FORMULATION
Let Ω be a bounded open and connected subset of IR where d is the dimension. A diffcrential operator L acting on function u(x) I2 + IR is defined as
Lu(x) = (x) D + an 1 D +... + ai (x) D + an ax(x)) u(x),
where D is the nth order d-dimensional derivative and (x(x),..... a. (x) are the coefficients. Con- sider the partial differential problem with Dirichlet boundary conditions
Lu(x) f(x), u(x) = uo(x),
XEΩ C Tdd
x 82,
where f(x) : _2d + IR is a given function. The forward problem solves u(x) given the coefficients 0 = {ao(x) ... a, (x)) while the inverse problem determines the coefficients set @ given u(x).
The proposed algorithm approximates the solutions in both problems by neural networks u(x; w.), {ay(x; Wa,)] }} such that the networks are parameterized by Wa [Wa, }. and the input to the network E Rdd Figure I depicts a network architecture of u in I22 The network consists of few fully connected layers with tanh activation and linear sum in the last layer. The network is trained to satisfy the PDE with boundary conditions by minimizing cost function. In the forward problem
F(u) AllLa f112 Lu flloo + ||ue uolli,00 + R (u),
(3)
2

---

Under review as a conference paper at ICLR 2020
x
y
Figure I: Network architecture: the point (z,, v) IR? serves as an input and the value ux y) as the output.
and in the inverse problem
I(0) = A1|Cu- fll2 + ul|Cu flloo + 110 00111,a0 + R (0).
(4)
The first two terms enforce the solution to satisfy the equation. The first term minimizes the error in the L2 sense while the second term minimizes the maximal error. Ideally, the solution to the PDE has to be satisfied at every point This is considered as a strong solution. In the deep learning framework the optimization carried out on mini batches of points. The first term minimizes the equation in the integral or average sense and is therefore insensitive to point jump in the value of integrand. It therefore yields a weak solution. This manifests itself by point discontinuities of u(x) that empirically seen well. The Loo term promotes strong solution since it handles possible discontinuous points where the value of the week solution is different than the strong one. We consider this discrepancy
error. Practically, we relaxed version of the Too we take the top- K errors. Then, at every mini batch the training procedure minimizes the equation at K different points having the largest errors. The third term imposes boundary conditions where uo and Og are the boundary values of u(x) and (xx). We used the L1 norm on the boundary so that the solution is robust measurements noise. The last term regularizer which be tailored the application. There are few advantages of this setting. First, the solution is a network and therefore an explicit function. It is smooth analytic function and is therefore analytically differentiable. was recently proven that for such functions, e.g. Lipchitz with constant the approximation of the function by fully connected network T is given by
where
[z) I'll c=(.,1()))))) S 2 2/d
where p E [0, oo), N number of neurons in a layer, and L number of layers Shen et al. 20199) In addition, the proposed framework enables setting prior physical knowledge the solution by designing the regularizers R and R'. Moreover, the training procedure is mesh free. Unlike finite differences or finite elements methods, course of the optimization of equation 3 and equation 4 see Figure 2 This means that the solution does not depend upon coordinate mesh and we also define arbitrary regular domain Q.
use random points in the domain and its boundary in the
3 APPLICATION TO ELECTRICAL IMPEDANCE TOMOGRAPHY
Consider the following elliptic equation which is special case of equation
V (α(x)Vu(x)) O,
Ω C IR2
(S)
u(x) (x()),
E 89,
where = (0/0,, 0/0,). We assume that 0 (xx) C² (2), which guarantees existence and uniqueness of solution (2) Evans 2010
3

---

Under review as a conference paper at ICLR 2020
Figure 2: Left to right: finite differences grid, finite elements mesh and random points samples used the proposed algorithm.
The elliptical system equation 5 was addressed by Siltanen et al. (2000) in the context of Electrical Impedance Tomography (EIT) which reconstruction method for the inverse conductivity prob- lem. The function indicates the electrical conductivity density, and is the electrical potential. An electrical current
Oun dv
Wn.g
cos(nn v), n E 22T
(6)
1822
applied on electrodes on the surface ön, where s: is the angle in polar coordinate system along the domain boundary, is the current frequency, is the phase and is the normal unit. The resulting voltage u ulan No is measured through the electrodes. The conductivity a is determined from the knowledge of the Dirichler-to-Neumann map voltage-to-current map
Jun
Ay ulon +
dv
On
Mueller Siltanen )2022; Alsaker & Mueller (2018) Fan Ying 2019).
We demonstrate our framework by solving the forward and inverse problem of equation 5 which is first step towards a full tomography. Following Mueller & Siltanen (2012). we simulate the voltage measurement ulan by the Finite Elements Method (FEM) given three variants of conductivity phantom (xx depicted densities and select ground truth the such that finer meshes do not improve the numerical solution. With our suggested method, the forward problem determines the electrical potential u in the whole domain given a, while the inverse problem uses the approximated and calculates the conductivity a given that alan = do.
Figure 3 We calculate the FEM solution with different triangle mesh
4 FORWARD PROBLEM
In the forward problem the conductivity a (2;) and boundary conditions ua(ss) are given for random points set { E C R², { E 0s C IR? with sets size of N, and N, respectively. A neural network having the architecture shown in Figure approximates u(x). Let
(o(a))Vu(a.))
C,
The cost function equation 3 is then rewritten as
N.
No No Σ ((xx)) uo( - allwull (8)
Σ 1. F F(u(x;)w)) wa) N₂
X
+ 1
Σ kEWo
K
The first term the Le norm of the differential operator, the second term is a relaxed version of the infinity norm where we take the mean value of the top-K values of [Ca]. There is some balance between small which accounts for few error points and can have negligible effect the training procedure, and large K which resembles the effect of the L2 norm. In all experiments pirically selected = 40 as an intermediate value. The third term imposes the boundary conditions and the last term serves regularizer of the network parameters.
4

---

Under review as a conference paper at ICLR 2020
Figure 3: Synthetic conductivities (x) referred to as phantoms. Left to right: phantom 1, phantom 2 and phantom 3
Table I: Forward problem results for phantom 1 compared with the DGM method
Sirignano &
Spiliopoulos (2017)
u: DGM
u: Proposed
Ug:
DGM
Uy: Proposed
MSE
PSNR
MSE
PSNR
MSE
PSNR
MSE
PSNR
0
2.50e-1
9.02
2.38e-4
39.23
6.01e-6
11.17
7.1le-9
40.43
2
n/8
4.20e-2
14.90
1.13e-4
40.60
7.95e-7
19.27
5.92e-9
40.55
3
4/4
5.46e-2
13.54
1.23e-4
40.03
1.87e-6
13.36
5.62e-9
38,58
4
0
5.12e-2
16.90
4.50e-5
47.46
1.64e-6
14.28
2.81e-9
41.94
5
27 7/4
6.11e-2
6.51
8.31e-5
35.18
1.99e-6
11.08
2.74c-9
39.69
The first phantom is shown in Figure left. The impedance values associated with the background ellipses and circle were 1, 5 and 2 respectively. The original piecewise constant function a was slightly smoothed by a Gaussian kernel.
Figure in the Appendix shows the forward problem results for current & with n = 1 and yp 7/8. The left column the FEM solution which is referred as ground truth, where the top row indicates the solution u(x) and the bottom row the derivative of u(x) with respect to the first coordinate calculated as the finite difference approximation of the FEM result. The middle column depicts the outcome of the proposed method where 0u/ô Du/Or is an analytical derivative of our result. The right column shows the outcome of the DGM method [Sirignano & Spiliopoulos (2017) which special case of equation 3 with À 1, with no regularizers. Quantitative results of the mean square error (MSE) and peak signal-to-noise ratio (PSNR) are summarized in Table [ Clearly, the proposed method outperforms the DGM method since the weighting parameters, the Loo norm and the network weights regularization play significant role in the loss function.
0, L2 norm of the boundary conditions constraint
Table 2: Forward problem results phantom 2 given a circular domain Q, and phantom with domains Q4, Sh and f2 defined in Figure
phantom
U
U
MSE
PSNR
MSE
PSNR
I
2,
2.86c-4
47.43
1.70c-8
40.61
2
2,Ω
7/2
1.74e-3
38.90
1.03e-8
33.79
3
2.
2
1.29e-4
45.52
3.26e-9
41.18
4
2,
2
4/2
1.30e-4
45.49
8.63e-4
37.07
5
3, 9
1
4/4
6.42e-5
47.16
5.64e-9
39.43
6
3
2
7/4
1.08e-4
34.03
2.61e-9
41.32
7
3
4/4
1.08e-4
44.91
4.51e-9
40.39
8
3
2
7/4
5.93e-5
36.64
2.74c-9
41.11
9
3, 3
1.22c44 7/4
44.37
9.17e-9
37.31
10
3. 22
4/4
1.74e-4
31.96
2.24e-9
41.99
5

---

Under review as a conference paper at ICLR 2020
06
02
Figure 4: Reconstructed conductivity a (x) using the generalized inverse problem. Left: phantom 1,
Right: phantom MSE and PSNR are summed up rows 1-2 of T
Table 3: Reconstruction of (x)
using the generalized inverse problem
phantom
MSE
PSNR
1,52
4.06e-2
27.90
2
2,02 Ω
9.04e-5
40.44
3
3,
1.07e-2
33.67
4
3, 522
5.9e-3
36.27
5
3,52g
1.12e-2
33.51
The forward problem was repeated using phantom where the background and circle conductivities and 0.2 respectively (Figure 3] middle). Four different current combinations were applied. Quantitative results are summarized in rows 1-4 of Table [ Figure 11 in the Appendix shows the results for both u and 0u/ /3x for 2 and o π 7/2. The right column presents the relative error defined as e(x, 3) (Urem (I, 3) u(r, 8)) max utem).
5 INVERSE PROBLEM FOR EIT
In the inverse problem, the electrical potential u(x) is known while a(x) is unknown. Since we have a network which approximates u( uxx), we can evaluate at any point x. The objective function equa- tion then takes the form
N.
I(0(xxwo))
A
C
Σ
N
K
1=1
(EE.)((1..))
(9)
1 No
B N Σ Vo(x;)).
-
+ +allwol12
As in the forward problem, the first two terms enforce satisfy the PDE, where L, is defined in equation 7 The third term imposes the boundary conditions, and the fourth regularizes the net- work parameters. A physical priori knowledge regarding exploited in this application. The conductivity is assumed to have well defined sub-regions. We therefore design the fifth term to have sparse edges via the total variation regularization (p 1).
Additional inverse problem generalization may exploit multiple approximations for several cur- rents vi. The calculation thus, simultancously relies on all {u,}. resulting a more stable solution. Let
Lij v (a(x:)))))
(10)
Then equation generalized to
A N.
[,(0(x;w,))
Cig 2
Σ (e
(11)
N
K
1=1

---

Under review as a conference paper at ICLR 2020
Figure 5: Free form shapes. Left to right: Q1, f22, S2 and sample points of Z2 The red points indicate the boundary.
Figure 6: Reconstructed conductivity (x) using the generalized inverse problem given phantom applicd on different domains. Left to right: Q1, 92 92z. MSE and PSNR are given in rows 3-5 of Table 3
and
No
N₂ N Vo (x:)
I( - +
1 No
Σ
(ar)) oo(x)) + allluall +
(12)
Adequate reconstruction results by the generalized inverse problem are shown in Figure 4 and rows 1-2 in Table
3 In both phantoms we used four current combinations d'n.pi (1, (1, 4), (2, 0),
(2, n /4), and dg
FREE SHAPE GEOMETRY
We applied the proposed method to arbitrary domains 22, S2 and Ωgg see Figure 5 The random sample points within the domain and along its boundary can be easily obtained as can be seen in Figure right. Forward problem results applied on phantom 3 are presented in rows 5-10 of Table 2 Figure 12 in the Appendix shows the results for = 2 and & π /4 for the three domains. The outcome of the generalized inverse problem equation 12 shown in Figure 6 and rows 3-5 of Table 3
7 ADDITIONAL INVERSE PROBLEM EXAMPLES
In this section we exemplify the proposed algorithm with additional inverse problem setting intro- duced by Xu & Darve (2119). Unknown parameters which are approximated by neural networks are inferred out of few measurements. The differential operators are approximated in finite differences schemes discretized by At and Ah, the temporal and spatial grid spacings. In the two following examples we applied the cost function equation 4 with total variation and network parameters regu- larizations.
7.1 DIFFUSION EQUATION
Diffusion PDE is given by
Ou(x,
C(x) + f(x,t) x [0,
Øt

---

Under review as a conference paper at ICLR 2020
where t denotes the time and the conductivity C(x) is the unknown parameter. In the following solve the equation in 2D where x := (z, y). Given two measurements at adjacent times u(z, y, t1)
and u( y, + At), the conductivity Czx,y) is approximated as neural network and is determined as the minimizer of equation with
[(u) - v, + Ar) w(x, v. C(r, v)
Drulæ, y, (1) + Dau(a, At)
f(x, y 4),,
At
2(Ah)²
where D2 is the central scheme approximation for the Laplacian on 3x3 stencil. The analytical 2D conductivity function is given by
C(x,y) = 1 + e -12xg))
and
u(x, v, -- -2 sin cos((ry + 17 /744,
fzz v) = = (2C((z))) 1). The measurements were taken at 1 0.1 with At = 0.001 and Ah 0.002. The result is shown in Figure 7 As can be seen the smoothness regularizer and the Loo terms increase the reconstruction accuracy. In addition, added few Gaussian noise levels to the measurements. Quantitative results are summarized in rows 1 of Table
05
Figure 7:
Reconstructed conductivity C(a,v) v). Left: ground truth, Middle: proposed method,
Right: Xu Darve (011). MSE and PSNR are given in rows 1-4 of Table
7.2
WAVE EQUATION
The second example the wave equation given by
8 Pu(a,y.t
ni V)) ((x, =|-1, [].
dt
The unknown parameter the velocity field (( v). analytical value is given by
7) v) 0.1 +e ((--05))+
and the two initial conditions
u(x, y, 0) = -10n2-ap?
and
u(r, y, At) - 10(00000010= 0000000)
Solutions for all time steps recursively calculated using the finite differences scheme using the correct 1722, y). Given three consecutive values of u(x, y, t) the velocity field 1/(x, y) is extracted by the minimization of equation E where
((u) u(x,, 4 + At) 2u(x, y. 41) + u(ær, 44 - A1) 12 v) (At)²
(,,,,,,44. (Ah)²
Reconstruction result is depicted Figure showing the advantage of the proposed method. Addi- tional quantitave results of noisy measurements given in rows 5-8 of Table
8

---

Under review as a conference paper at ICLR 2020
08
88
0.
08
04
D6
04
Figure 8:
Reconstructed velocity field 17(2, y). Left: ground truth, Middle: proposed method,
Right: Xu Darve (019). MSE and PSNR are given in rows 5-8 of Table
Table 4: Inverse problem results for diffusion and wave equations
Xu & Darve (2019)
Proposed
Equation
noise
MSE
PSNK
MSE
PSNR
1
Diffusion
4.90e-4
39.12
2.04e-4
42.92
2
Diffusion
le-7
0.259
11.89
0.249
12.06
3
Diffusion
5e-7
0.335
10.77
0.316
11.02
4
Diffusion
le-6
0.448
9.50
0.438
9.60
5
Wave
0.015
19.01
0.013
19.53
6
Wave
3e-9
0.049
13.92
0.015
19.16
7
Wave
3,5e-9
0.056
13.34
0.018
18.32
8
Wave
4c-9
0.059
13.11
0.029
16.19
IMPLEMENTATION DETAILS
The network architecture had layers having 26, 26, 26 and 10 neurons. The algorithm was im- plemented by TensorFlow Ten (2015) using the ADAM optimizer which is variant of the SGD algorithm. We used almost the same hyper parameters set in our experiments. Batch size=1000, decaying learning rate starting at 5e 3. The learning rate was factored by 0.8 every 200 epochs, ., 80000, NN, = 1200, 0.01, a = le 8, and wave equation the EIT, on Intel i7-8650u CPU was about 15 minutes for the forward problem and 13 minutes for the generalized inverse problem. For the diffusion and wave equation the running time about minutes.
40, μ = 0.01, and 3 = 0.01. In the diffusion set À = 1, 3 le 4 le 4 and le 5. Running time, for
Finally, we performed sensitivity analysis of the algorithm with respect the number of sample points Ng. Figure shows the sensitivity of the forward problem algorithm. As expected, with large number of points obtain a plateau while with small and intermediate number (up 50000) the error is not monotonic due to local minima.
DISCUSSION
Deep networks by their nature use compositions of simple functions such matrix multiplication and non-lineat activations like sigmoid or tanh. This structure (i) enables the approximation of an ar- bitrary function Hornik 1989) and (ii) is inherently differentiable. The network architecture dictates the number of degrees of freedom which in turn enables the expressibility of complex functions. In this work we present unified framework for the solution of forward and inverse problems in partial differential equations in an arbitrary domain. The algorithm relies on direct approximation of the unknown function by neural network which yields an analytical smooth solution. The network is trained to satisfy the PDE and boundary conditions in unsupervised fashion by the minimization of cost function. The optimization procedure depends on random points set within the domain and its boundary. The problem therefore mesh free with free-form domain.

---

Under review as a conference paper at ICLR 2020
100
Mean Square Error
10
10
10-3
10-4
0
2
4
6
8
Number of points
x x0
Figure 9: Sensitivity analysis of the algorithm with respect the number of sample points.
While recent works formulate the cost function in both forward and inverse problems in the Le sense, introduce a broadened framework. The relaxed Loo fidelity a strong solution and control possible discrepancies between the weak and the strong solutions. The L1 norm on the boundary conditions more robust measurement noise than the L2 norm. The additional regularizer (total variation in our examples) has significant role in the optimization process where known physical properties like smoothness are imposed on the solution. Lastly, the network regularizer controls the network weights and prevents over fitting. These additional terms make the proposed algorithm more accurate and robust compared to other sarming-basee methods. We also stress the robustness of our approach exemplified by having almost the same hyper parameters set in our experiments.
was designed to approximate
This framework alleviates several problems of the finite differences and finite elements methods. In particular meshing, discretization and derivatives approximation are solved in a simple and natural way. Numerical solutions of PDEs in an arbitrary domain of extreme importance, in particular in medical imaging applications. The framework is demonstrated by an elliptic system in I2 applied Electrical Impedance Tomography for both forward and inverse problems, diffusion and wave equations. Promising results were achieved for complex and non monotonic functions. Rigorous analysis of the approximation error and its relation the network architecture and design are under current study. Future research includes also full tomography solution, higher dimensional problems and other classes of PDEs such as dynamic non-linear equations (Burgers, Navier-Stoker etc.)
REFERENCES
TensorFlow: Large-scale machine learning on heterogeneous systems, 2015. URL http:// tensorflow org/ Software available from tensorflow.org.
M. Alsaker and J. L. Mueller. Use of an optimized spatial prior in D-bar reconstructions of EIT tank data. Inverse Problems and Imaging. 12:883-99 901, 2018.
T. K. Bera, S. K. Biswas, K. Rajan, and J. Nagaraju. Improving image quality in electrical impedance tomography (eit) using projection error propagation- based regularization (pepr) technique: A sim- ulation stud. Journal of Electrical Bioimpedance, 2:2-12, 2011.
M. M. Chiaramonte and M. Kiener. Solving differential equations using neural networks, 2017. edu/pro
L.C. Evans. Patial Differential Equations. the American Mathematical Society, 2010.
10

---

Under review as a conference paper at ICLR 2020
Y. Fan and L. Ying. 1906.03944 03944v1, 2019.
Solving electrical impedance tomography with deep learning. arXiv
M. Feigin, D. Freedman, and B. W. Anthony. A deep learning framework for single-sided sound speed inversion in medical ultrasound. arXiv 1810.00322v3, 2018.
A. L.. Garcia, Numerical Methods for Physics (Python). CreateSpace Independent Publishing Plat- form, 2017.
J. Han, A. Jentzen, and E. Weinan. Solving high-dimensional partial differential equations using deep learning. arXiv 1707.02568, 2018.
K. Hornik. Multilayer feedforward networs are universal approximators. Neural Networks, 2:359- 366, 1989.
Y. Khoo and L. Ying. Switchnet: neural network model for forward and inverse scattering prob- lems. arXiv 33/0.09676v1,20111
R. J. LeVeque. Finite Difference Methods for Ordinary and Partial Differential Equations: Steady- State and Time-dependent Problems. SIAM, 2007.
H. J. Schwab, S. Antholzer, and M. Haltmeier. NETT: Solving inverse problems with deep neural networks. arXiv 1803.00092 2018.
Y. Li. Lu, and Mao. Variational training of neural network approximations of solution maps for physical models. arXiv 0005..2789v1, 2019.
A. Lucas, M. Iliadis, R. Molina, and A. K. Katsaggelos. Using deep neural networks for inverse problems imaging. IEEE Signal Processing Magazine, 2018.
M. McCann, K. H. Jin, and M. Unser. A review convolutional neural networks for inverse problems in imaging. arXiv 1710.04011, 2017.
Mueller and S. Siltanen. Linear and Nonlinear Inverse Problems with Practical Applications. SIAM, 2012.
M. Raissi, P. Perdikaris, and G. E. Karniadakis. Physics informed deep learning (part i): Data-driven solutions nonlinear partial differential equations. arXiv 1711. 2017.
N. Reddy. An Introduction to the Finite Element Method. McGraw Hill Education, 2005.
J.K. Seo, K. C. Kim, A. Jargal, K. Lee, and B. Harrach. A tearning-based method for solving ill- posed nonlinear inverse problems: A simulation study of lung EIT. SIAM journal on Imaging Sciences, 12:1275-1295, 2019.
Z. Shen, H. Yang, and S. Zhang. Nonlinear approximation via compositions. arXiv: 192. 2019.
Siltanen, Mueller, and D. Isaacson. An implementation of the reconstruction algorithm of nachman for the 2d inverse conductivity problem. Inverse Problems, 16(3):681-699, 2000.
Sirignano and K. Spiliopoulos. DGM: A deep learning algorithm for solving partial differential equations. arXiv 1708.07469v3, 2017.
J. W. Thomas. Numerical Partial Differential Equations: Finite Difference Methods. Springer, 1995.
K. Xu and E. Darve. The neural network approach to inverse problems in differential equations. arXiv 1901 077551,, 2019.
11

---

Under review as a conference paper at ICLR 2020
A
APPENDIX
(a) FEM
(b) proposed
(c) DGM
(d) 0u/Os: FEM
(e) 0u/ proposed
(f) 0u/Ozz DGM
Figure 10: Forward problem results of u(x) and 0u(x) /0r for current frequency 1 and phase = T/ /8 given phantom 1. Left column: ground truth (FEM). Middle column: proposed method. Right column: DGM method. MSE and PSNR reported in of Table
12

---

Under review as a conference paper at ICLR 2020
2005
u:
(b) u: proposed
u: relative error
<10?
D2
D.1
(d) 0u/Or: FEM
(e) /r: proposed
(I) 0u/0u:: relative error
Figure 11: Forward problem results of u(x) and (u(x) for current frequency = 2 and phase j 7/2 given phantom 2. Left column: ground truth (FEM). Middle column: proposed method.
Right column: relative error. MSE and PSNR are reported in row 4 of Table
0.
0.6
405
-55
08
45
-55
-15
25
24
2.5
(a) u: proposed,
(b) u: proposed, Q2
(c) u: proposed, Pa
10
(d) Ou/Ox: proposed, Ωi
(e) 0u/Ow: proposed, U2
(f) du/Ox: proposed, Za
Figure 12: Forward problem results of u(x) and 0u(x)/Oc for current frequency = 2 and phase s n /4 given phantom applied on different domains. Left to right: Ωn Ω2 2g. MSE and PSNR are reported in rows 6, 8 and 10 of Table 2
13


Reviews:
[{"review_id": "Hyg9zSZCYr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents an unsupervised learning approach to solve forward and inverse problems represented by partial differential equations. A framework is proposed based on the minimisation of loss functions that enforce the boundary conditions of the PDE solution and promote its smoothness. The method is then applied to solve forward and inverse problems in electrical impedance tomography.\n\nFlexible machine learning approaches to solving partial differential equations is a subject of ongoing research. While the paper presents an elegant solution folding forward and inverse problems into a single framework, the presentation is missing a few important details which difficult the assessment of the contribution and favour a rejection of the paper. The main issues are insufficient experimental comparisons and a lack of theoretical support for the method.\n\nMajor issues:\n\n1. When compared to DGM (Sirignano & Spiliopoulos, 2017), for the forward problem, the method only differs in the form of the loss function, which is almost identical, with the exception of the additional L-infinity term and the optional user-defined regularisers. The argument for the inclusion of the L-infinity loss is its high sensitivity to outliers, enforcing a smooth solution. (a) Why PDE solutions learned using the original loss from DGM, which also yields continuous functions, should present outliers in the first place? Moreover, the possibility of adding a user-defined regularizer seems to be a relatively simple extension. (b) What should the theoretical or practical implications for the extra user-defined regularisation term be?\n\n2. The loss function for the inverse problem, which seems to be one of the paper’s contributions, misses a dedicated discussion. An important detail in this loss is the third term, which enforces boundary conditions for the coefficients at the boundary of the domain. In Equation 2, however, the coefficients only affect the PDE through the Lu term over the domain, not its boundary. So what does \\theta_0 mean in Equation 4 then?\n\n3. The paper proposes a general framework, but experimental results are presented for only one specific problem, the electrical impedance tomography. The generalisability of the method to more complex problems, such as PDEs with time components and a high-dimensional spatial domain, cannot be inferred. Adding experimental comparisons on higher-dimensional domains would strengthen the paper.\n\n4. Experiments only present comparisons to relevant state-of-the-art methods (DGM) in the forward problem. There are no comparisons against other methods for the inverse and the free-shape geometry problems. For example, have the authors considered the method in [A]?\n[A] Xu, Kailai, and Eric Darve. \"The neural network approach to inverse problems in differential equations.\" arXiv preprint arXiv:1901.07758 (2019).\n\nMinor issues:\n\n1. The background on PDEs is relatively short for a machine learning conference. (a) There lacks an explanation on what the operator \\mathcal{L} means. (b) Equation 1 lacks an explicit use of “u(x)”, instead of simply “u”, causing confusion with the dependence of the coefficients on “x”. (c) The meaning of the index subscripts on the partial derivatives is also not made clear, especially if “u” could be interpreted as a vector-valued function for someone unfamiliar with PDEs. Replacing “some u” by “some u:R^d\\to\\R” would already help.\n\n2. What does “n” mean in the electrical current equations in Sec. 3?\n\n3. The derivative of a scalar “u(x)” with respect to a vector “x” should be a vector. So what are the plots in figures 4, 5 and 8 showing when referring to du/dx? Is that the magnitude of the vector or the partial derivative with respect to a single spatial component?\n\n4. What does “PSNR” stand for?\n\n5. Indirect citations in the text should be enclosed by brackets using something like the “\\citep” command from the package “natbib”.\n\n6. In Table 1, there is a typo: “GDM”->”DGM”.\n\n7. The context contains a few minor grammatical issues that can be distracting at times, but should be solvable by revision.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Hkem-N2DFH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this work, the authors propose an approach to solve forward and inverse problems in partial differential equations (PDEs) using neural networks. In particular, they consider a specific type of second-order differential operator combined with Dirichlet boundary conditions, and suggest how neural networks with suitable training objectives can be used to solve both types of problems. For a specific elliptic system relevant for Electrical Impedance Tomography (EIT), they then present numerical results obtained with their method and compare those to high-resolution finite element solutions and previous work.\n\nThe paper addresses a very general and important problem with wide ranging applications. People have solved PDEs using spectral, finite difference, finite volume or finite element methods for decades and there is a huge body of literate on this subject. The neural network based approach proposed in this paper seems general and simple with encouraging experimental results. However, there are several important points missing in the current paper based on which I would recommend rejecting it. However, I would be willing to increase the score if these points were addressed in sufficient detail.\n\nMajor comments:\n\n1) Choice of problems\n\nThere are many important problems in the literature that belong to the general type of PDE considered in this work (e.g. in electrostatics). To appreciate and understand the merits and limitations of the proposed approach better, it would be highly necessary to apply it to a wider range of problems, including ones with known analytical solution. \n\n2) Convergence tests\n\nConvergence tests are an important part that is missing in the current paper. In particular, how does the error in the neural-network solution decay with respect to the number of random points used to train the network? Plots comparing a suitable error norm versus the training dataset size would be very informative. Furthermore, what are the conditions, if any, to guarantee convergence to the exact solution? The authors mention in the last paragraph of the Discussion that a more rigorous analysis will be published elsewhere. However, as error analysis is such an integral part of the study, I think it needs to be addressed, at least to some extent, already in the current paper.\n\n3) Related work\n\nI don’t think the current work is put in sufficient contrast with existing work. Several related papers are mentioned in the introduction and the experimental section includes one other method (GDM method of Sirignano & Spiliopoulos) for comparison. However, a more thorough discussion on how the current approach complements existing literature on neural network based PDE solvers would be in order. \n\nMinor comments:\n\ni) There are quite a few typos and grammar mistakes in the paper that need to be fixed. To give a few examples:\n‘the natural structure presents’ -> ‘the natural structure present’\n‘Shrödinger’ -> ‘Schrödinger’\n‘approximated by neural network’ -> ‘approximated by a neural network’\n‘on circular and’  -> ‘on a circular and’\n‘forces the equation’ -> ‘enforces the equation’\netc...\n\nii) Eq. (7): There is a parenthesis missing on the LHS.\n\niii) The authors show many plots in Figs 3-9 but don’t comment much on the results in the main text. A lot of the results presented could therefore go into an appendix. I would find it better to present fewer results/plots and discuss those in more detail.\n\niv) It would be good to define the Nabla operator in Eq. 5 for completeness.\n\nv) Please put references to equations and papers in parentheses or, at least, separate them otherwise from the sentence, to avoid confusion. For example:\n- the reference to ‘Evans (2010)’ below Eq. 5, or\n- the list of dangling references before the last paragraph on page 3,\n- the reference to Sirignano & Spiliopoulos in the caption of Table 1,\netc...\n\nvi) Please expand the captions in Fig. 3 to make the figure meaningful as a stand-alone, without having to read the entire text to understand what 'phantom' means. The same applies to all other captions.\n\nvii) Perhaps it would be clearer to use ‘row’ instead of ‘line’ when referring to the results in the table.\n\nviii) In Sec. 7 ‘s’ in ‘Ns’ should be a subscript.\n\nix) Explain what the acronym PSNR means.\n\nx) Please elaborate on why you take the top k values in Eq. (7). What happens if you take more/less?\n\nxi) First paragraph in Discussion: I am not sure the robustness w.r.t. the hyperparameters is really so surprising given that you apply the solver to very similar problems. If you could show that the same hyperparameters worked on a completely different problem, that would be much more interesting.\n\nxii) Please add references for the first paragraph of the introduction.\n\n\n*********************************************************\nI increased my rating based on the revisions. \n*********************************************************", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BJeTZ3a0tH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: In the paper, the authors purpose to use neural networks to model both the function $u$ and the parameters and in a sense, unify the forward and inverse problems. The authors demonstrate the work on Electrical Impedance Tomography problem. The authors also purpose a new regularizer, the sum of L2 and L_inf norm of the differential operator.\n\nConcerns: there have been plenty of works that use neural networks to model the function $u$ for forward problems and another bunch of works that use neural networks to model parameters to do inverse problems.\n\nIt is not clear to me if combining the two will really give us benefits, since we are still doing these two problems separately. If we are doing some alternating training, unifying them could be useful.\n\nThe mesh free part is less interesting in my opinion. Works using feed forward neural networks are mesh free in general. And when you try to use the solution, or to compare with some ground truth generated by traditional methods, usually we still need to make the solution discrete to use it. \n\nThe experiments in the paper is limited. It compares with only one work in the forward task, but no comparison in the inverse problem. It is hard to evaluate its performance.\n\nThe theory is also needed. It is not very clear why L2 + L_inf regulation terms will help us. \nAfter all, in computer science, conference papers are considered as final publications. So a more extensive studied is expected. I would suggest submit this work for a workshop. \n\nDecision: This work need further experiments and theoretical analyze. I suggest weakly reject this paper.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes modifying the training loss for neural net-based PDE solvers, by adding an L_infty (max) term to the standard L_2 loss.  The motivation for this loss is sensible in that it matches the definition of a strong solution, but this is only a heuristic motivation, and is missing a theoretical analysis.

This paper's lack of novelty and polish, as well as the lack of clarity in the implementation details, makes this a narrow reject.
Decision: Reject