Improved Training of Physics-Informed Neural Networks Using Energy-Based Priors: a Study on Electrical Impedance Tomography
Authors: Akarsh Pokkunuru
Pedram Rooshenas
Thilo Strauss
Anuj Abhishek
Taufiquar Khan
Venue: ICLR 2023 poster (2023)
Keywords: Physics-informed neural networks
electrical impedance tomography
energy-based models

Abstract: Physics-informed neural networks (PINNs) are attracting significant attention for solving partial differential equation (PDE) based inverse problems, including electrical impedance tomography (EIT). EIT is non-linear and especially its inverse problem is highly ill-posed. Therefore, successful training of PINN is extremely sensitive to interplay between different loss terms and hyper-parameters, including the learning rate. In this work, we propose a Bayesian approach through data-driven energy-based model (EBM) as a prior, to improve the overall accuracy and quality of tomographic reconstruction. In particular, the EBM is trained over the possible solutions of the PDEs with different boundary conditions. By imparting such prior onto physics-based training, PINN convergence is expedited by more than ten times faster to the PDE’s solution. Evaluation outcome shows that our proposed method is more robust for solving the EIT problem. Our code is available at: https://rooshenasgroup.github.io/eit_ebprior.

Published as a conference paper at ICLR 2023
IMPROVED TRAINING OF PHYSICS-INFORMED NEU- RAL NETWORKS USING ENERGY-BASED PRIORS: A STUDY ON ELECTRICAL IMPEDANCE TOMOGRAPHY
Akarsh Pokkunuru'
Amirmohammad Rooshenas' pedramêuic. edu
apokkunu@unce. edu
Thilo Strauss thi 1 ume gmx. de
Anuj Abhishek anuj. abhishhkeeee..... unco. edu
Taufiquar Khan tkaan1 3@unce. edu
ABSTRACT
Physics-informed neural networks (PINNs) are attracting significant attention for solving partial differential equation (PDE) based inverse problems, including elec- trical impedance tomography (EIT). EIT is non-linear and especially its inverse problem is highly ill ill-posed. Therefore, successful training of PINN is extremely sensitive to interplay between different loss terms and arper-parameters includ- ing the learning rate. In this work, propose Bayesian approach through data- driven energy-base based model (EBM) as a prior, to improve the overall accuracy and quality of tomographic reconstruction. In particular, the EBM trained over the possible solutions of the PDEs with different boundary conditions. By imparting such prior onto physics- based training. PINN convergence expedited by more than ten times faster to the PDE's s solution. Evaluation outcome shows that our proposed method more robust for solving the EIT problem. Our code avail- able at: htttps://rooseenagggte
gi thub. io/eit iit. ebprior/
INTRODUCTION
Physics-informen neural networks (PINNs) (Raissi et al. 2019) parameterize the solution of a partial differential equation (PDE) using a neural network and trains the neural network to predict the solu- tion's 's scalar value for any given point inside the problem's 's domain by minimizing the residual PDE and associated boundary conditions (BCs). Various other factors such as gradient stiffness (Wang cet al. 202T) and complex parameter settings, lack of constraints and regularization (Krishnapriyan et al. 2021). often cause major issues in training PINNs and makes them very sensitive to hyper- parameters and regularization. For example, depending
the PDE, the interplay between BCs residuals and PDES residuals can result in invalid solutions that mostly satisfy one type of con- straints over the others. In general, lack of joint representation in PINNs, as these models are assumed to implicitly learn that two points close vicinity possibly have similar solution. However, this representation is hard-coded in the numerical methods as they update the solution value of each mesh point based on the neighboring mesh points.
believe that some of these problems
attributable to the
In this work, we augment PINNs with joint representation via a Bayesian approach. We train data-driven prior over joint solutions of PDEs on the entire domain and boundary points. This prior relates the predictions of PINNs via explicit joint representation and encourage learning a coherent and valid solution. Priors have been studied extensively statistical approach to solve inverse
problems (aipio al. 2000; Ahmad et al.) 2019: Abhishek et al., 2022. Strauss & Khan 2015)
and also recently have been used in data-driven approaches as well (Ramzi et al. 20220). More importantly, focused on the non-linear and ill- posep electrical impedance tomography (EIT)
DDeaarmeet of Computer Science, University of North Carolina at Charlotte, NC, USA
1 "eppartment of Computer Science, University of Illinois Chicago. IL, USA
1 Bosch ETAS Research, Germany, EU 6 Department of Mathematies and Statisties, University of North Carolina at Charlotte, NC, USA

---

Published as a conference paper at ICLR 2023
inverse problem and experimentally show that using data-driven Bayesian approach results in accurate, fast, and more robust training algorithms. We have the following contributions:
We present various experimental studies which cireumvent instability training PINNs for the EIT problem.
We introduce a dlta-drivenn prior in the form of an EBM for improving the training conver- gence and accuracy of PINNs.
We propose robust training framework for the EIT semi-inverse problem.
2
AN INVERSE PROBLEM WITH APPLICATION TO ELECTRICAL IMPEDANCE TOMOGRAPHY (EIT)
The inverse problem that we are interested is inspired by applications imaging paradigms such as electrical impedance tomography (EIT) and geophysical imaging for ground water flow. The EIT inverse problem reconstructs the unknown electrical conductivities a of a body f2 C Rd with d (2, 3} from measurements of finite electrical potential differences of neighboring surface electrodes. The differential equation
oV in Ω
governs the distribution of electric potential in the body. Additionally, its accompanying BCs are given as following:
Ou Ön
on 02 Neumann BC,
on 02 Dirichlet BC
An EIT experiment involves applying an electrical current on the surface ön of the region Ω to be imaged, which produces a current density (u 8n lan = 9 (Neumann data) where n is unit normal vector w.r.t associated with Ω at its boundary. The current also induces electric potential in the body, whose surface value u 8o = ʃ can be measured. Thus by repeating several such experiments in which surface current is given and the corresponding surface voltage (Dirichlet data) is measured, we obtain the information on the Neumann -ee--tto-Diiiie Dirichlet (NtD) operator which can be denoted by:
Ag : ggg
(3)
In the full EIT version, the problem is to reconstruct from just the surface current and voltage measurements. The estimate the unknown conductivity a can reconstructed from a set of EIT experiments (Somersalo et al.) 1992, Borcea 2002 Hanke & Brühl (003). In simplified version that we attempt to solve here using our novel method, we will be interested in recovering a from the measurements of in the interior of the medium. This problem formulation is close to the groundwater flow problem, wherein the source term on the right hand side of Eq. Tis non-zero. should also be noted, that similar formulation
studied in Bar & Sochen (2021).
In our training paradigm, we need to simulate u for the interior of the medium for any underlying a. We achieve that by first solving the forward problem by training neural network that predict the value of the function u(x) at any given point æ E Ω for any given a. Once we have access to this pre-trained network that we will call u-Net, will subsequently train another network predict the value of ar (xx) ) for any æ E Ω from point-wise measurements of function u(z) that satisfies Eq. aand agrees with the Neumann and Dirichlet boundary data given from Eq.
2.1 USING PINNS FOR EIT
Following the construction of physics-informed neural networks (PINNs), a and u using neural networks, called o-Net and u-Net, respectively, and train them such that the values of a -eNet and u-Net for any provided point in the interior boundary of the medium respect the PDE in Eq or its boundary condition depending on the position of the point. The forward
parameterize both
'To the best of our knowledge only Bar & Sochen (022)) claim to train the EIT inverse problem using PINNs by jointly training u-Net and -Net hut unfortunately. their implementation is not open-source and we unable reproduce their results based the details in the paper. See Appendix A.5 details.
2

---

Published as a conference paper at ICLR 2023
problem assumes that distribution is known everywhere inside the domain Q. The w-Net takes mesh points in Cartesian space (z, y) E IR² as input and correspondingly learns to recover correct potential values u at the points inside the domain S2 given the voltages only at electrodes ue.. To train w-Net, we utilize Eq. 1 and to write them in a functional form as follows:
LPDE = V. (oa Vua)
Ad Ω
(4)
dub
oe Owe One ge] Ve E d2..
C/c ob
wb a
CBc
(5)
Ons
Here, LIDE governs the potential in forward and conductivity a distributions in inverse problem respectively. CBc are the combined BCs where enforce Neumann BC twice, once on all boundary points HZ, except electrodes and separately on electrodes 22,. Thus, a-Net and u-Net represent their respective distributions parameterized functions o (xr, y) and u(x,y). We then write common objective function which is used to train both problems as follows:
Co
Σ (E(
(CEDE) +
3 M
Σ
[PDEE
+
7 [a22
|C
+
[anc
6
LEc-
(6)
mEoOPar Cree
bEaN,
eEa0.
where a; B, 7 and control the contribution of each terms to the overall loss. The required deriva- tives and a w.r.t coordinates x and are calculated via reverse mode differentiation provided by autodiff frameworks such as PyTorch or Tensorflow. Following Bar & Sochen (202I), we penalize violations in LYDE twice using L2 and Loe norm variants in order to provide stronger approximate solution. Finally, order train u-Net more efficiently, we enforce additional regularizers:
Lousa La
Σ
Iue - Sol 1ol+6 + |/w/|]
Jan
eE (
The second term of Eq. enforces Dirichlet BC only on electrodes while the last term controls the weights of u-Net using weighed L2 regularization on O, with temperature Ç.
We train o-Net (parameters are denoted as 0,) such that it learns the conductivity distribution over all given mesh points solve the semi-inverse problem. Similar to the forward problem training, o-Net incorporates the main PDE and the Nuemann BCs seen in Eq. 6 as part of its training objective order learn the conductivity inside Ω. However, the problem is known to be illposed and thus needs strong regularizers to improve the quality of reconstructions in conjunction with Co- For instance, we regulate the norm of gradients Va.y inside to promote sparse edges in predictions, and penalize any conductivity prediction of less the one (= conductivity of vacuum) using Chinge (0, 1 ox)) vh E { URUU.
The combined oNNe training objective assimilating the elliptical PDE, its BC, and all the aforemen- tioned mentioned regularizers is given as follows:
Loasa = La +
lo - Jen,
Σ
Cinge + ( [| (8)
[an)
[
bEan,
DEO
he(u)eg)
where oan, is the known conductivity on the boundary points.
3 ENERGY BASED PRIORS
C from Fq. 6 can be interpreted residual of violation (of Eq. noted as for simplicity. Assume r (residual entire domain) follow a multi-variate Gaussian distribution with zero mean and diagonal covariance matrix. Therefore, maximizing the likelihood of p(rlor; 0,) results in the similar optimization as minimizing Eaea (C with respect to 9 (parameters of o-Net). Now we can define our Bayesian approach but assuming ar follows the prior distribution p(o)):
mox logp(ar),
0,
where is parameterized by 6 via -ee. We interpret p(ar) possible solutions to the PDE defined by Eq. | for different boundary conditions. We then define the prior distribution using energy-based models (EBMs) (LeCun al. 2006]: (x(ar x exp(-Fiç(a)) Several techniques

---

Published as a conference paper at ICLR 2023
have been proposed for training EBMs, including contrastive divergence (Hinton 2002). noise-
contrastive estimation (Gutmann& & Hyvärinen 2010), score matching (Hyvärinen & Dayan 2005
and denoising score matching (Vincent 2011). We found that denoising score matching (DSM) is
more stable, less compute intensive as it avoids expensive second order partial derivatives involved in score matching, and ultimately generates more realistic solutions in our setting. DSM trains the energy function such that its vector field (Vo.. log log matches the vector field of the
underlying distribution p(( which is approximated by perturbing the empirical data distribution with Gaussian noise of different intensities. See/Song& & Ermon (2019), Song & Ermon (0200) for
more details.
We jointly estimate a noise conditional energy function Eg 14) for all noise-perturbed data dis- tributions conditioned on various noise scales μ4 for E [1... ..., which are chosen such that Nn > Hl > ... fLz, In our work, we chose I. = 20 linearly spaced noise scales: lá E [2, 0.01]. Essentially, the training objective is to minimize the following:
L(o; (4) T. Σ X(u) Eptor BEwNYO.D Vo
z _ s4 2
(10)
where A(44) > 0 is a coefficient function chosen as A(u) = 11² and finally, ô = a + 142 is the noise perturbed version of conductivity distribution ō No IQ.). In order use E.p as a prior, in contrast to the standard DSM training that trains a score network (S(.) -V E)), we directly train the energy network. See Appendix A.33 for the details on training Es and also for generated samples from Eg and their closest training data points (Fig.
Upon successful training of the Es using Eq. 10 the energy function (o HL) will assign low energy values to that more likely to present the real world (valid solution of Eq. [ and high energy value to unlikely assignments (invalid assignments that violates Eq. greatly..
Now we can rewrite the final training objective of -Net (using Eq. 8 as follows:
Conne = Co +
lo0
Tan, + [01
V Va.
[nn]
bEgn
den
(II)
Chinge + < [|lwlll 112
xE (or (I.)
Aequan
where x is the weight of the prior in the overall loss.
EXPERIMENTS
Our EIT data simulation setup primarily consists of phantom generation and forward solution construction via finite element solver. Initially. construct various discretized solutions of on a 2D mesh-grid of size 128 x 128 obtain phantoms Il....2 In particular, we randomly choose the anomaly location, radius, shape deformity, quantity and their conductivity on a circular mesh. The target a values are chosen randomly be- tween E [3. 15) for either 1, 1 or 3 anomalies per mesh with restrictions on their Iocations, so as to not let the anomalies touch each other and the boundary. We also assume a uniform background conductivity of (simulating vacuum) on locations devoid of any anomaly and phantom configuration followed by discretization, we smooth our solutions following Evans (2010) using a Gaussian low-pass filter of size 200 and standard deviation 3 to obtain smoother solutions. We then gen- crate 6512 such smoothed phantoms for EB prior training and 1628 for testing. Majority of these datasets contain circular anomalies with varying radii and few thousand examples with minor shape deformity (el- lipsoids). Furthermore, maximum conductivity a value obtained from training set. We additionally generate a few hand-crafted phan- toms to train the u-Net and o-Net for forward and inverse problems, seen in the first row of Fig. 2 These configurations are specifically designed to be more challenging by setting higher shape deformity, larger vari- ance in a values for each anomaly, and placing them in challenging locations inside the mesh. This is purposefully make the physics-basee based training and inference more challenging to ensure extreme robustness. Note that there are no phantoms in the FBM training set which resemble these hand-crafted samples. With regards to our forward problem simulation setup, we assume that Ncumann condition is imposed on the entire boundary where the function g in Eq. is given a trigonometric pattern (Siltanen et al. 2000) as follows: g V25 cos (mw v), E Z, where is the angle along dl2, and zj and vi the current frequency and phase, respectively. We use current pattern where = 1 and 1 0 for all our experiments. Thus, to obtain the forward solution, use finite clement method (FEM) solvers by utilizing discretized hand-crafted
1 0 outside the circle mesh. Upon selection
standardize both train, test sets to [0. 1] interval by dividing all samples using the
4

---

Published as a conference paper at ICLR 2023
phantoms profile (a value, anomaly location) and current y as input. We then solve for everywhere and obtain the solver based solution on circular mesh.
4.1 EXPEDITED SEMI-INVERS, PROBLEM EVALUATION
We now present results of our proposed framework. As discussed prior sections, each step of a-Net training procedure, current prediction fed to the trained prior ( ) to obtain scalar energy value. This energy value provides useful supervision to O-Net and essentially expedites the convergence of PlNNs. This phenomenon can be viewed in Fig. Here. each sub-figure represents the learning curve of or-Ntt with the choice of metric with and without inclusion of prior. Although the model is trained 3000 epochs, brevity, we only showcase the first 500 epochs of the curve due to more interesting properties here. Evidently, the PINNs with EB prior converge much faster within the first few 100 epochs than the training without prior. In case of phantom 4, the convergence more than ten times faster while also aiding the PINN to avoid getting stuck in local minima. Corresponding these learning curves, we present Fig. showing the quality of reconstruction for each phantom.
squared error (MSE), separately trained
conductivity of various phantoms
Phantom
2.0
wio prior
wio prior
we prior
1.5
Pricr
with Prioe
0.6
with Prir
1.0
0.5
0.2
0.0
0.0
100
200
300
400
500
0
1000
20.0
300
400
500
100
200
300
400
500
Epochs
Epochs
Epochs
wio prior
1.0
with Prior
0.5
0.0
1.0
200
100
400
500
1.00
20.0
100
400
5.0
Eports
Epochs
Figure 1: PDE solution convergence and accuracy analysis via a MSE for various phantoms.
Ground truihs
Ground sruth
Ground truth
Ground trush
Ground bruhh
Figure Comparison of ground truth and predictions from PINN with EB prior.
4.2 COMPARISON WITH BASELINES
framework with baselines Bar&Sochen Bar Sochen (021). Deep Galerkin Methods (DGM) Sirignano & Spiliopoulos (2018) by using the hand-crafted phantoms. In order to introduce these works, there are a few differences that set our works apart. Firstly, both Bar & Sochen (2021). Sirignano & Spiliopoulos (2018) utilized high resolution mesh which naturally enable PINNs to learn the problem efficienlly due availability of large input data points solve the PDE and BC. Additionally,
now compare performance
5

---

Published as a conference paper at ICLR 2023
Bar & Sochen (021) utilizes large number of randomil selected boundary voltages enforce the Dirichlet BC Eq. 2 during forward solution learning. In contrast, we train our u-Net and o-NNet a much more challenging settings while using a low-resolution mesh and only 16 boundary voltages during the forward problem training. The driving point here is that, one can view Dirichlet BC as a L1 regression loss over given boundary voltages. When more boundary points and their corresponding voltages are revealed for learning u, the more stronger this supervision signal, which enables learning the forward map and anomaly location more accurately. However from practical EIT system standpoint, is unrealistic to collect such close proximity fine- fine-grained information over many boundary points. Apart from these difference, we treat |arr & Sochen (2021) and Siiiinnano& & Spiliopoulos (2018) as special cases of our work. Our work primarily differs in the incorporation of the energy scalar from EBM and the new hinge loss term in Eq. Eq.8to improve the performance. Given the high non-lincarity and ill-posed nature of the ETT problem, even slight change in one parameter can lead to highly varying solutions. Hence, we tunc our loss penaltics differently to obtain higher quality reconstructions. We show a list of common parameters among the considered baselines. We now present the
Table 1: List of common hyperparameters among various baselines
Method
Hyper-parameters
0r
0
M
P
DGM
Bar&en
0.01
0.01
40
16-8
0.01
0
Ours
0.05
0.05
40
0.1
100
16-6
0.01
10
0.0001
Table 2: Evaluation of our method, DGM (Sirignano & Spiliopoulos 2018), and Bar & Sochen (2021) with and without proposed EB Priors.
DGM w/ EB Prior
&ach w/ EB Prior
Our Method
Phantom
Metrie
DGM
Bar&Sochen
Our Method
w/ EB Prior
MSE 1 PSNR
3.26
0.13 0.002
2.57 + 0.041
0.16 0.026
0.13 + 0.001
0.03
0.001
0
8.84 + 0.038
22.77 + 0.079
9.89 + 0.068
22.02 0.645
22,79 + 0.034
29.94
0.146
MDE
2.29 0.024
0.36 0.003
0.028
0.23 0.002
0.35 0.002
0.014
MSE 1
6.19
0,3 1.3=0.00 0.004
5.18 + 0.039
0,2 0.015
0.4 + 0.002
0.06 0.001
0g
PSNR f
10.15
23.3 0.054
10,92 0.033
22.04 0.023
30.58
0.079
MDE
2,85 1 0.015
0.39 1 0.003
2.2 1 0.023
0.25 0.002
0.43 1 0.003
0.1 0.009
MSE L
1.56 0.032
0.03 0.001
1.37 + 0.017
0.11 0.013
0.04 + 0.001
0.01 0.0001
0g
PSNR
12.05 1 0.089
29.03 : 0,193
12.62 1 0.056
23.45 1. 0.47
28.2 1 0.061
35.46
:: 0.088
MDE
0.033
0.25 0.002
0.021
0.002
0.002
0.009
MSE L
0.00+
0.08 + 0.002
1.64 + 0.019
0.2 1 0.036
0.07 + 0.001
0.01 + 0.0001
PSNR f
15.04
29.07 0.009
15.92 + 0.052
25.23 1123=00 0.779
29.68 + 0.034
38.45 = 0.052
MDE
0.013
0.26 = 0.004
1.1 + 0.02
0.19 - 0.001
0.25 + 0.003
0.12 0.011
MSE 1
7.39::0.046
1.38 + 1.25
6.68 : 0.018
0,5 0.084
0.57 : 0.003
0.22 0.004
PSNR T
14,83 0.027
24.03 + 1.99
15.27 + 0.012
26.67
0.694
25.95 + 0.025
30.17 0.082
MDF
2.97+00.013
0.73 0.439
0.017
0.28 0.004
0.44 0.002
0.016
semi-inverse problem evaluation results for considered baselines in Table all five phantoms. We primarily use three metrics namely, mean squared error (MSE), peak signal-to-noise ratio (PSNR) and mean difference error (MDE) to evaluate the quality of reconstruction of a solution while learning the objective Eq. 8 Note that MDE is computed as the difference in means of a over all (ar y) E Ω and df2 We then utilize the hyper- parameters shown in Table [ to train -NNet for our framework, while the neural network presented in Bar & Sochen (2021) is used on both Bar Sochen (2021) and [Sirignano & Spiliopoulos (2018) baselines. All these models are trained separately on all five phantoms while using the same settings, once without the EB prior and once under the presence of EB prior. These experiments are repeated for 10 runs and the averaged metrics along with their standard deviation error is presented. The main observation from Table 2 is, that by incorporating the EB prior in PINN training enables lower error rates and higher quality reconstructions for the semi-inverse problem. Additionally, we can also see that the EB prior can easily plugged into any PINN-based framework and vastly improve the underlying training, showcasing our methods strong generalizability.
4.3 LEARNING RATE CONVERGENCE
We aim to show through this study that the learning rate vastly affects PINN's solution accuracy and this inherent instability be easily alleviated by introducing EB priors, even while using large learning rates. For this study, we keep all the parameters of our proposed method shown in Table [ fixed and vary only the learning rate during a NeN training with Eq. g We choose 0.0001, 0.001, 0.01 and 0.1 learning rates to study their effects on the enni-inverse PINN training with and without the EB prior Note that, we still decay all these chosen learning rates using exponential decay with rate 0.9 over every 200 epochs. The results of learning rate studies are presented in Fig. F As seen this figure, the true a value for the ellipsoid anomalies is 5 while the

---

Published as a conference paper at ICLR 2023
LA
15
20
16
12
30
05
52
600
With Prirr
Wis Prtor
With Price
Wih Prio"
Nis. vSE Prior: 3.1805
Min Prion: 0.33094
Min Prter: 0.23263
Mis HSE Pric: 1.47544
Hin. With Prinr: 3.52626
MSE Fram 777
vSS W 0.02558
Min. MSC With Pr 0.3007
kile
3.5-
2.0
1.5
1.2
1000
19:30
20:00
Epochs
Spects
Epacts
Eoods
Figure 3: Training robustness of our proposed work with EBM prior for various learning rates Phantom 1.
smaller circular anomaly in the bottom-middle has 2. We evidently see that the PINN without EB prior fails converge an accurate solution with regards for row one and two's color-bars indicate the failure to reach an accurate conductivity prediction. Clearly, the PINN without EB prior never matches the solution accurately, under all learning rates. More adversely, the PINN produces trivial solutions while using learning rate that is either too low too high. On the other hand, PINNs augmented with EB prior always produce accurate solutions of slightly varying degree of MSE as seen in row two, three of Fig. More importantly, the training session with extremely large learning rate LR = 0.1 still manages to produce reconstruction although the accuracy might be lower. This proves the robustness of our framework and instigates clear evidence that PINN training is being improved by very large margin while incorporating EB priors. Other advantages such burden of choosing the optimal learning rate also alleviated.
three. More importantly, the maximum values
4.4 SENSITIVITY ANALYSIS VIA RANDOM PARAMETER SEARCH
As discussed these earlier sections, PINNS extremely sensitive the balance interplay among different hyperparameters. The training procedure essentially becomes an optimization problem where one has to tune the right settings in order to converge to a highly accurate solution obtain failed trivial solution. The problem of choosing parameters is more challenging thus study the effect of changing the strength of various loss weighting penalties mentioned in Table while simultancously observing the sensitivity of the EB prior trained with various data quantities. As a preliminary, we first bifurcate the 6512 training samples into a smaller training. validation sets (20% set aside for cross- validation). After splitting, we separately train five EB priors (labeled 100% data priors) on the whole training In addition, train another five EB priors (labeled as 1% data priors) separately. by randomly selecting only 1% the data that was used for training the 100% data priors. Note that the model architecture and all parameerssssrrss are fixed for consistency, while only the batch sizes are different; 64 and 16 for 100% and 1 1% data priors respectively (see more training details and synthetic generated samples in Appendix A.4).
EIT as it highly non-linear and ill-posed. We
The final step is to evaluate all the trained EBMs and perform sensitivity analysis of PINN training. For this, restrict the search space a, 8, 7 and 6 parameters in Eq. 6 as they the main terms influencing the the search configuration set and start training the PINN: a E [0.01, 0.05, 0.1, 0.5, 1), B E [0.01, 0.05, 0. 0.5, 1]. r E [0.01, 0.1, 0.5,10 1.0, 2.0) and finally 8 [0.001, 0.01, 0.1, 0.5, 1.0, 2.0]. We use Phantoms 1 and 5 to conduct 100 random search runs for PINN with each of the ten FBMs as priors, and 500 runs for PINN without EB prior. We then average the results of every training category (w/o prior, 1% and 100% prior) and display them
solution quality of the EIT inverse problem. We randomly choose single number from

---

Published as a conference paper at ICLR 2023
Fig. Here, the y-axis in Fig. 4a indicates that large number of random runs successful solve Phantom and reach set upper bound 0.3 MSE and 3000 epochs. Fig. 44balso corresponds to Phantom however we set strict MSE threshold of 0.3 and check for number experiments that reach this threshold by the end of all 3000 epochs. Similar observations can be said about Figs. 44 and [4d which are studies related to Phantom 5. In conclusion, we have substantial evidence that the PINN with EB prior always has higher success rates producing highly accurate reconstructions. More importantly, the PINN with the EB prior converges much faster towards near-zero MSEs regardless of quantity of the EBM training sample sizes. Thus, showcasing the strong robustness of our proposed framework.
PISERTI
souch
Harter Mio Thorsed GJ
101
(h)
Preriom
Paaton 3 US Thensd 3.0
72
12
nel pror
(c)
(d)
Figure 4: Sensitivity analysis of aNet while randomly searching the optimal weights in Eq.
4.5 SEMI-INVERSE WITH NOISY DATA
the final set experiments, we test the robustness our framework while learning from noisy data. Initially, we add uniform noise of various standard deviation levels such as 0.1, 0.25, 0.5, to corrupt the 16 boundary measurements. The u- w-Net learns the forward problem under these noisy settings and approximates noisy- forward map (see appendix[A.2|tt for evaluation of w-Net under noisy settings). While moving to the semi-inverse problem, we data originating from the noisy u-Net predictions and check o-Net's ahility to withstand measurement noise. These results can be seen in Table 3 The performance degradation is visible when noise std increases, more noticeable in some phantoms that others. Despite these noisy conditions, incorporating the EB prior during --Net training greatly improves the predictions and robustness.
Table 3: Performance evaluation of -NNet using noisy measurements
Metric
Phantom
Phantom
Phantom
Phantom
Phantom
w/o Price
w/ Prior
w/o Prior
w/ Price
wlo Prjor
w/ Prior
wio Prior
w/ Prior
w/o Prior
w/ Prior
MSE
0.13
0.113
0.4
0.06
01.04
11.01
0.07
0.01
0.57
0.22
PSNR
30.58
28.20
35.46
68
38,45
MDE
0.35
0.12
0.43
0.1
0.25
0.11
0.25
0.12
0.44
0.08
MSE
0.094
0
3.38
1.64
0.02
0.03
2.84
0.01
0.49
0.11
PSNR
24.25
29.91
12.77
15.91
31,50
29.72
13.53
36.78
26.67
33,09
MDE
0.29
0.16
0.52
0.21
0.21
0.02
0.17
0.42
0.23
MSE
0.17
0.02
0.58
0.08
0.03
0.03
0.09
0.03
0.67
0.31
0.25
PSNR
21.55
31,20
20:40
29.10
28.95
29.29
28.46
33,86
25.24
28.56
MDE
0.37
0.23
0.50
0.29
0.24
0.16
0.26
0.17
0.49
0.37
MSE
0.13
0.06
0.37
0.13
0.61
0.57
0.09
0.02
5.22
4.39
PSNR
22.96
26.05
22.42
26.84
16.15
16.41
28.69
35,43
16.35
17.09
MDEL
0.34
0.14
0.22
0.25
5 RELATED WORK
Inverse problems for elliptic PDEs are ill-posed and highly nonlinear, which makes it imperative to include regularization methods for obtaining reasonable reconstruction Jin al. (2012); Din & Maass (2012). Many
8

---

Published as a conference paper at ICLR 2023
approaches exit for solving the EIT inverse problem with rigorous theoretical justifications. For e.g. variational methods for least-square fitting (Jin et al. (2012); [ & Maass (2012) Ahmad ct al. (2019)), the factorization
method (Kirsch & Grinberg (2008), ), the d-bar method (Mueller et al. 2002] Isaacson et al. 2004 Knudsen
ct al. 2009). the monotonicity shape estimate Harrach & Ullrich| (2013), the level set method Liu ct al. [2017]
2018 and series inversion methods [Ariige et al. [2012] Abhishek et al.] (2020) to name few. Besides,
regularized Newton type methods such as Haber (2005) Lechleiter & Rieder (2008) also popular. For the
groundwater flow model, some important references can be found in Frühauf et al. (2005) ; Russell & Whheeler
Oliver et al. (2008).
Driven by the need to quantify uncertainties. many works proposed to solve such inverse problems in a para- metric Bayesian setting (Kaipio et al. (2000 2004); [Strauss & Khan [2015)[[Ahadd [Ahmad et al. (2019); ( (Strauss et al.]
(2015) ). However, the error these methods rely discretization schemes hence, alternative non- non-parametric Bayesian inversion mesh-indeppndent methods have heen proposed. Following Stuart (2010), a Bayesian level set method was proposed for reconstruction of piecewise smooth conductivity in Dunlop & Stuart (2016), for reconstruction of piecewise continuous permeability in ground water flow model in Iglesias et al. (2014) and for reconstruction of conductivity in certain Sobolev spaces in Abraham & Nickl (019). Recall that in Bayesian setting, the posterior p(0(a) x p())p( depends on the space of conductivities a and a prior. The prior p(0), in addition to encapsulating our subjective belief about the functional space; also introduces regulariza- tion in the problem. Great strides have been made in using efficient MCMC sampling methods which increased computational efficiency of such Bayesian inversion methods (Cotter et al. 2013).
With the rise of machine learning methods, many solutions for solving the EIT with neural networks were proposed. In supervised setting, Hammlton & Hauptmann (2018); |aan Ying 2020); Hamilton al. (2019);
Agnelli et al. (2020) primarily use convolutional neural network (CNN) architectures to predict the conduc- tivity a based on boundary mcasurements. Deviating to other inverse problems, Ulyanov et al. (2018); Lucas ct al. (2018) describe the use of various generative models in this context. Generative Adversarial Networks
(s) [Goodfellow al. (020) and variants
great example. Patel & Oberai (202)) Patel al.] 2020
2022 | Molin (2022) solve various image and physics-inspreed inverse problems by training sampling algo- nihmms that jointly learn measurements y and latent vectors = with generator (G) as a prior. This allows for generation of new samples given the posterior, G and unseen y. In contrast. Marinescu et al. (2020) use pre- trained generators as priors for maximum a-posteriori (MAP) inference over a given z, known corruption model and recover a corrupted input image such that G(z") arg max, ((())((()) If o G(z)), where G(z*) is the clean image. Lastly, (Adler& & Öktem 2018 2019) trained conditional WGANs in supervised setting (pair-wisc data and y) and sample a posterior given conditional G, z, and unseon y. Their second approach allows for trainable point-wise inference networks similar to our work, but these rely on supervised data. In comparison, work learns both EBMs and PINNS in an unsupervised manner directly in data space, resulting in more direct and faster inference.
Instead of using implicit GAN priors via latent spaces, score and energy models are more dircet due to stronger
a- a-priori beliefs. In this work, we share connection with score matching (SM) Hyvärinen & Dayan 2005
through DSM. Many score-based works solved linear inverse problems utilizing DSM (Ramzi et al. (2020):
SSng al. 202T: : Kawar al. 202T Chung ). ). For MRI/CT reconstruction, |aazi al. 2020
score-prior with posterior sampling using annealed HMC: [ong et al.] (20TT): Chung & Ye (202) score-prior
with posterior sampling using reverse-ting stochastic differential equations: Kawar et al. (202 score-pri prior
with posterior sampling using LD and singular value decomposition (SVD): Kadkhodaie & Simoncelli 1 1222
generalize DSM and blug-and-plas
priors (Venkatakrishnan al. 2013) and sample with implicit CNN
denoising prior. While, |Zach et al.] (2022) use EB priors trained with contrastive divergence and solve linear inverse problems by sampling with proximal gradient descent. Clearly, the similarities between our work and aforementioned are purely prior design choices; (1) DSM is used for training the prior; (2) all our works use discretized inverse map solutions (MRI/CT images or EIT phantoms in our case) as input data; (3) there is no dependency on measurement data (labels) during prior training. The important difference is that solve non-linear, ill ilo-posed physics-based inverse problem of EIT using DSM priors without sampling the posterior.
6 CONCLUSION
Physics-ifformed neural networks are an important category of data-driven PDE solvers. However, PINNs are not stable and robust for more complicated problems. In this work, we look at the EIT problem and show that we can improve the stability and robustness of PINNs training (w.r. noisy measurement and hyper-parameter selection) via Bayesian approach. We describe data fata-driven prior using energy-based methods, which can easily be used with the other loss terms and robustly train PINNs. In the EIT setting, our experimental results also show that PINNs converge faster and also to more accurate solution when trained with prior. In our future work, we will study using priors for potential in order to the improve the training of PINN for full EIT inverse problem and other PDE problems.

---

Published as a conference paper at ICLR 2023
REFERENCES
Anuj Abhishek, Mare Bonnet, Shari Moskow. Modified forward inverse Born series for the Calderon and wife problems. Inverse Problems, (6611):111001 19, 2020. ISSN 0266-5611. doi: 10.1088/
1661-6420abac11. Wttphhtp:: org/ 1088/18883668 361 (000/-644///bbba
Anuj Abhishek, Thilo Strauss, and Taufiquar Khan. An optimal bayesian estimator for absorption coefficient in diffuse optical tomography. SIAM Journal on Imaging Sciences, 15(2):797-821, 2022. doi: 10.1137/
21M1462842. URL httthtttp:///// 1ttp///.1137/2MM1553333
Kweku Abraham and Richard Nickl, On statistical Calderón problems. Math. Stat, Learn,, 2(2):165-216, 2019. ISSN 2520-2316.
Jonas Adler and Ozan Öktem. Deep bayesian inversion. arXiv preprint
2018.
Jonas Adler and Ozan Öktem. Deep posterior sampling: Uncertainty quantification for large scale inverse problems. International Conference on Medical Imaging with Deep Learning-Extenderr Extended Abstract Track, 2019.
J. P. Agnelli, A. Çöl, M. Lassas, R. Murthy, M. Santacesaria, and S. Siltanen. Classification of stroke using neu- ral networks in electrical impedance tomography. Inverse Problems, 11:110)11500... 26, 2020. ISSN 0266- 5611. doi: 1011110//1116621100 6420/abbbdd. URLLhttss:.
org 101101081110 1361- 6420/ abbded
Sanwar Ahmad, Thilo Strauss, Shyla Kupis, and Taufiquar Khan. Comparison of statistical inversion with iteratively regularized gauss newton method for image reconstruction in electrical impedance tomography. Applied Mathematics and Computation, 358:436-449, 2019.
Simon Arridge, Shari Moskow, and John C. Schotland. Inverse Born series for the Calderon problem. Inverse Problems, 2833)035003,, 16, 2012. ISSN 0266-5611, doi: 10. 7/3o1 org/ 1088 0266-6002266222 5111//666661111/////38888
hRLhtt
Leah Bar and Sochen. Strong solutions for pde-based tomography by unsupervised learning. SIAM Journal on Imaging Sciences, 14( (44(1):12-115:) 2021.
Liliana Borcea. Electrical impedance tomography. Inverse problems, 18(6):R99, 2002.
Hyungjin Chung and Jong Chul Ye. Score-based diffusion models for accelerated mri. Medical Image Anulysis, pp. 102479, 2022.
L. Cotter, G. Roberts, A. M. Stuart, and White. MCMC methods for functions: modifying algo- rithms to make them faster. Statist. Sei., 28(3):424 446. 2013. ISSN 0883-4237, doi: 10.1214/13-STS421. URL http:://dc.../gg///e////// 100 00 1214/
STS421
Matthew M. Dunlop and Andrew M. Stuart. The Bayesian formulation of EIT: analysis and algorithms. Inverse
Probl. Imaging, 10(4): 1007-1166 2016. ISSN 1930-8337. doi: 10.39344ipi.2016630. URL https:
1/col.orge 10. 3934/111.20110333333333333333333
Lawrence Evans. Partial differential equations, volume 19. American Mathematical Soc., 2010.
Yuwei Fan and Lexing Ying. Solving electrical impedance tomography with deep learning. Journal of Compu- lational Physics, 404:109) 1091109119, 2020.
F. Frühauf, O. Scherzer and A. Leitão. Analysis of regularization methods for the solution of ill posed problems involving discontinuous operators. SIAM Journal Numerical Analywis, 43(2):667 2005. doi: 10. 1137755000611429933300060 URL htt.hhtta///// /ttppa://ddi.... org. http///0...111///343333000333330
Ian Goodfellow, Jean Abbdet-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11): 139-144, 2020.
Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A estimation principle for unnor- malized statistical models. In Proceedings of thirteenth international conference on artificial intelligence and statistics, pp. 297-304. JMLR Workshop and Conference Proceedings, 2010.
E. Haber. Quasi Newton methods for large-scale electromagnetic inverse problems. Inverse Problems, 21(1): 305-333, 2005. ISSN 0266-5611. doi: 10. (0.100088226-66112//111119 URI. hntpp:: //doi boo..... oorg/ 1C.
1088/0266-55 561555661111 /
10

---

Published as a conference paper at ICLR 2023
Sarah J Hamilton, Asko Hänninen, Andreas Hauptmann, and Ville Kolehmainen. Beltrami-net: domain- independent deep d-bar learning for absolute imaging with electrical impedance tomography (a-cit). Physi- ological measurement, 40(7):074002, 2019.
Sarah Jane Hamilton and Andreas Hauptmann. Deep d-bar: Real-time electrical impedance tomography imag- ing with deep neural networks. IEEE transactions on medical imaging, 37(10):237723777, 2018.
Martin Hanke and Martin Brühl. Recent progress in electrical impedance tomography. Inverse Problems, 19 (6):565, 2003.
Bastian Harrach and Marcel Ullrich. Monotonicity- based shape reconstruction in electrical impedance tomog- raphy. SIAM Journal on Mathematical Analysis, 45(6):3382 3403, 2013.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, 2015. //t:://aaxiiv www. abs/1 03385
Geoffrey Hinton. Training products of experts by minimizing contrastive divergence. Neural computation, 1408):1771-1800, 2002.
Lior Horesh, M Schweiger, SR Arridge, and DS Holder. Large-scale non-linear 3d reconstruction algorithms for electrical impedance tomography of the human head. In World Congress on Medical Physics and Biomedical Engineering 2006, pp. 3862-3865. Springer, 2007.
Aapo Hyvärinen and Peter Dayan. Estimation of non-norman non-normalized statistical models by score matching. Journal of Machine Learning Research, 6(4), 2005.
Marco A. Iglesias, Kui Lin, and Andrew Stuart. Well-posed Bayesian geometric inverse problems arising in subsurface flow. Inverse Problems, 30(11): 39, 2014. ISSN 0266-5611. doi: 10, 1088/0266-56111 30/1 1/011. htLhttp:: http:///iiooggg/// 10. 1088 0200002666-22
David Isaacson, Jennifer I. Mueller, Jonathan C Newell, and Samuli Siltanen. Reconstructions of chest phan- toms by the d-bar method for electrical impedance tomography. IEEE Transactions on Medical Imaging, 23 (7):821-82) 2004.
Bangti Jin and Peter Maass. An analysis of electrical impedance tomography with applications to tikhonov regularization. ESAIM: Control, Optimisation and Calculus of Variations, 18(4): 1048, 2012.
Bangti Jin, Taufiquar Khan, and Peter Maass. reconstruction algorithu electrical impedance tomography based on sparsity regularization. International Journal for Numerical Methods in Engineering, 89(3):337-
353, 2012.
Zahra Kadkhodaic Eero Simoncelli, Stochastic solutions for linear inverse problems using the prior implicit in denoiser. Advances in Neural Information Processing Systems, 34:13242-13254, 2021.
Jari Kaipio, Ville Kolehmainen, Erkki Somersalo, and Marko Vauhkonen. Statistical inversion and monte 2000. carlo sampling methods in electrical impedance tomography. Inverse problems, 16(5):1487, 2000.
JP Kaipio, Seppänen, E Somersalo, and H Haario. Posterior covariance related optimal current patterns in electrical impedance tomography. Inverse Problems, 20(3):919, 2004.
Bahjat Kawar, Gregory Vaksman, and Michael Elad. Snips: Solving noisy inverse problems stochastically. Advances Neural Information Processing Systems, 34:21757-21769, 2021.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint
arXiv:1412.6944 2014.
Andreas Kirsch and Natalia Grinberg, The factorization method for inverse problems. Number 36. Oxford University Press, 2008
Kim Knudsen, Matti Lassas, Jennifer L. Mueller, and Samuli Siltanen. Regularized D-bar method for the inverse conductivity problem. Inverse Probl. Imaging. 3(4):599-624, 2009. ISSN 1930-8337. doi: 10.3934/ ipi.2000.3.599. /RRL:hctts///oi.orrrr
3934 39t.//3334//11.... 3.599
Aditi Krishnapriyan, Amir Gholami, Shandian Zhe, Robert Kirby, and Michael W Mahoney. Characterizing possible failure modes in physies-informed neural networks. Advances in Neural Information Processing Systems, 34:26548-2653 265660 2021.
Armin Lechleiter and Andreas Rieder. Newton regularizations for impedance tomography: convergence by local injectivity. Inverse Problems, 24(6):655099, 18, 2008. ISSN 0266-5611. doi: 10. 1088/0266-5611//0 1/24/ 6/065009. wtt..tttss://///te.. :/tp:::/://...../ http://///oo..oo11.1113.0 0/2000000000000011111666666000

---

Published as a conference paper at ICLR 2023
Yann LeCun, Sumit Chopra, Raia Hadsell, M Ranzato, and F Huang. tutorial energy-based learning. Predicting structured data, 1(0), 2006.
Dong Liu, Anil Kumar Khambampati, and Jiangfeng Du. parametric level set method for electrical impedance tomography. IEEE transactions on medical imaging, 37(2)451-446) 2017.
Dong Liu, Danny Smyl, and Jiangfeng Du A parametric level set-based approach to difference imaging in electrical impedance tomography. IEEE transactions on medical imaging, 38(1):145 155, 2018
Alice Lucas, Michael Iliadis, Rafael Molina, and Aggelos K Katsaggelos. Using deep neural networks for inverse problems imaging: beyond analytical methods. IEEE Signal Procexsing Magazine, 35(1)200-36, 2018.
Razvan Marinescu, Daniel Moyer, Polina Golland. Bayesian image reconstruction using deep generative models. arXiv preprint arXiv:2012.04567, 2020.
Vincent Molin, Bayesian inverse problems with neural generative priors. 2022.
J. Mueller, S. Siltanen, and D. Isaacson. A direct reconstruction algorithm for electrical impedance tomogra- phy. TEEE Transactions on Medical Imaging, 21(6):555-559, 2002. doi: 10. 109/TMM.2002 800574..
Dean Oliver, Albert C. Reynolds, and Ning Liu. Inverse Theory for Petroleum Reservoir Characterization and History Matching. Cambridge University Press, 2008, doi: 10.1017/CBB97700111533 1533644
Dhruv Patel and Assad A Oberai. Gan-based priors for quantifying uncertainty. arXiv:2003. 12597, 2020.
arXiv preprint
Dhruv V Patel, Deep Ray, Harisankar Ramaswamy, and Assad Oberai. Bayesian inference in physics-driven prohlems with adversarial priors. NeurlPS 2020 Workshop on Deep Learning and Inverse Problems, 2020.
Dhruv Patel, Deep Ray, and Assad Oberai. Solution of physics- based bayesian inverse problems with deep generative priors. Computer Methods in Applied Mechanics and Engineering, 400: 15428, 2022.
Maziar Raissi, Paris Perdikaris, and George F. Kariiadakss. Physics-informed neural networks: deep learn- ing framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics, 378:686-707, 2019.
Zaccharie Ramzi, Benjamin Rémy, Francois Lanusse, Jean-Luc Starck, and Philippe Ciuciu. Denoising score-
matching for uncertainty quantification in inverse problems. arXiv preprint arXiv:201 1,8698, 2020.
Thomas F. Russell and Mary Fanett Wheeler. 2. Finite Element and Finite Difference Methods for Continuous Flows in Porous Media pp. 35- 106. doi: 1711137719781111999999 org doi/ abs /wwd/bsi//si//ooo 1137/11 9781 11181611971171... ch2
siam.
Samuli Siltanen, Jennifer Mueller, and David Isaacson. An implementation of the reconstruction algorithm of nachman for the 2d inverse conductivity problem. Inverse Problems, 16(3))681, 2000.
Justin Sirignano and Konstantinos Spiliopoulos. Dgm: A deep learning algorithm for solving partial differential
equations. Journal Computational Physics, 375:1339-1364, 2018. ISSN 0021-9991. doi: hups://doi. jep. p 9991 8.355277
URL httpp::/ htttps::/www.... wwt.::/wwweenccee.. rect http://ccc/ccccccceeear......
Erkki Somersalo, Margaret Cheney, and David Isaacson. Existence and uniqueness for electrode models for electric current computed tomography, SIAM J. Appl. Math., 52(4):1023-144) 1992. ISSN 0036-1399, doi: 10.1137/0152060. [ttLllttts://dooo
111.//01..1111111111111 520660
Yang Song and Stefano Ermon. Generative modeling by estimating gradicnts of the data distribution. Advances in Neural Information Processing Systems, 32, 2019.
Yang Song and Stefano Ermon. Improved techniques for training score-based generative models. Advances in neural information processing systems, 33:1 13:12438-12448, 2020.
Yang Song, Liyue Shen, Lei Xing, and Stefano Ermon. Solving inverse problems in medical imaging with bosre-bascd generative models. arXiv preprint wxXiv:211.00000 2021.
Thilo Strauss and Taufiquar Khan. Statistical inversion in electrical impedance tomography using mixed total variation and non-convex lp regularization prior. Journal of Inverse and Ill-posed Problems, 2355)529-542, 2015.
12

---

Published as a conference paper at ICLR 2023
Thilo Strauss, Xiaolin Fan, Shuyu Sun, and Taufiquar Khan. Statistical inversion of absolute permeability in single-phase darcy flow. Procedia Computer Science, 51, 2015, doi: htt.:r//1..1111..prroes.... 2015. 09.
Stuart. Inverse problems: a Bayesian perspective. Acta Numer, 19:451-552 2010. ISSN 0962-4929. doi: 00.0017/0099224991000000.. htt.tttts:///b..rrg//// 0tt.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Deep image prior. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 9446 0454,9944,,222 2018.
Singanallur V Venkatakrishuan, Charles A Bouman, Brendt Wohlberg. Plug-and-play priors model based reconstruction. 2013 IEEE Global Conference on Signal and Information Processing, pp. 945-948.
IEEE, 2013.
Pascal Vincent. A connection between score matching and denoising autoencoders. Neural computation, 23 (7):1661-1674, 2011.
Sifan Wang. Yujun Teng, and Paris Perdikaris. Understanding and mitigating gradient flow pathologics in physics-informed neural networks. SIAM Journal on Scientific Computing, 4365543005--330000 2021.
Yuxin Wu and Kaiming He. Group normalization, 2018. wRt.::pp:://xx....rog///o/////// 08494
Martin Zach, Erich Kobler, Thomas Pock. Computed tomography reconstruction using generative energy- based priors, arXiv preprint arXiv:2203.1266 2022.
13

---

Published as a conference paper at ICLR 2023
A APPENDIX
The model architectures and training implementation details for u-Net, a-Net and EBM are shared here.
A.1 SEMI-INVERSE IMPLEMENTATION DETAILS
The common loss weighting penaltics used by both the u-Net, a-Net models are: 0.05, B 0.05,
M = 40, = 0.1 and Ç = Ie- 6 The Nuemann loss penalty 7 = 0.1 and Dirichlet loss penalty c 100 are specific to the forward problem only. While. 7 = 1, T = 0.01. V = 10 and lastly K = 0.0001 are specific inverse problem. Additionally, both models use the same multi-layer perceptron (MLP) architecture with residual connections (Wang et al. 2021) and consist of 4 hidden layers with tanh activation and 64 neurons each, single output neuron with no activation aids in predicting the solutions for given mesh points. We train both these models with a batch size of 1000 using ADAM (Kingma & Ba 2014) optimizer for 3000 epochs with an initial learning rate of 0.005 and decay it exponentially with a rate of 0.9 over intervals of every 200 epochs. We primarily use NVIDIA Titan RTX GPU for all our training purposes and the training time for forward and semi-inverse problems separately take around takes minutes for 3000 epochs.
A.2 FORWARD PROBLEM IMPLEMENTATION AND RESULTS
Table 4 shows the evaluation results of w-Net for the forward problem the considered phantoms. We study the quality of forward problem solutions from w-Net while adding random uniform noise the 16 electrode mcasurements of various levels. As seen these experiments, the quality degradation is obvious due increas- ing levels of noisc. To counter this effect, we tune the penalty weights o, 8, 7, 5 on individual loss terms in Eq. for increasing noise levels to improve the quality of reconstructions. The weights on Dirichlet BC = 100, L2 model parameter regularization Ç <= Le 6 are fixed for all experiments and phantoms. Additionally, we the same parameter for all phantoms while only tuning them for given noise level, which showcases the robustness of our method.
Table 4:
Performance evaluation of w-Net for solving Forward problem with various noise levels
Noise Std
Metric
Phantom
Phantom 2
Phantom
Phantom 4
Phantom
Parameters: a = 0.05, 3 0.05, 7 0.1, = 0.1
MSE 1 PSNR
0,00046
0.00091
0.00027
0.00058
0.00036 40.51
39.41
33.40
46.90
47.02
Parameters: a 0.1, 8 = 0.1, = 0.1, 8 = 0.1
0.1
MSE 1
0.0090
0,070
0.02076
0,0208
0.026
PSNR
26.51
14.55
28.
31.48
21.91
Parameters: = 0,05, 8 0.05, 7 1, 5,5 = 0.1
0.25
MSE 4
0.0426
0.1665
2.667
1.722
0.039
PSNR
16.70
13.854
7.04
12.30
20.22
Parameters:
0.1, B = 0.05,
1, =0.1
0.5
MSE 1 PSNR
1.041
2.25
10.59
2.46
4.70
5.89
0,52
1.05
3.01
0.587
A.3 EB PRIOR IMPLEMENTATION AND EVALUATION
Implementation We train our EB prior solutions (as described in Section 4 by perturbing them with
Gaussian noise of 20 noise scales μr [2. 0.01] in order learn by DSM training. We visualized some of the training samples in Fig. 7 and the compared the nearest samples in EBM training data and inference set of
PINN in Fig. 5 We then create a deep convolutional neural network with multiple residual He et (2015) connections inspired by the architectures in [Song & Ermon (2019) and [Song & Ermon (202)) In particular, our model consists of a single 3 x 3 convolution layer tollowed by series of 16 convolutional residual and residual- -ownaamplnnnn blocks in the form of an encoder block of typical convolution encoders. Each of these blocks consist of convolutional and Group normalization Wu & He (2018) layers with group size 32. We then learn the input features by gradually down-sampling them to smaller resolutions. Upon feature map reduction the end of the last convolution layer, we introduce single hidden dense layer of 256 units to jointly learn the compressed a features along with the noise scales, followed by a final dense layer which outputs scalar energy values. We use ELU non- linearity for all our EBM layers except the last energy layer which has no activation. We then train this EBM model to denoise the noise perturbed o solutions via the objective seen in Eq. [0 for 2000 epochs, with a hatch size of 64, using ADAM optimizer with fixed learning rate of 0.0001. The training time takes around 15 hours while using NVIDIA Titan RTX GPU.
14

---

Published as conference paper at ICLR 2023
Ground truch
Ground suuth
Ground truch
Ground truth
Ground srthh
12.5
160
255
Nearest
Nearest
Nearest
Nearest
Nearest
13.0
25.0
11.5
11.
12.
22.5
15.0
20.2
203
16
5.
50
2.
255
2.5
25
Figure 5: Comparison ground truth a and nearest data in EBM training set
Evaluation After the EBM (a (4) is trained, we can generate samples by using annealed Langevin dy- namics (LD) Song & Ermon (209)) sampling. We start from fixed prior distribution such as uniform noise and initially run L.D for 100 steps with step size of 8 using first μa noise scale and draw samples from E- (1)))) by adding Gaussian noise. Next draw samples from E=(-; (2) by reducing the step size and refine the samples. We continue running LD until the noise scales are used to sample and we finally arrive at final step size 8n,, = apt/pá, where a = 0.0002 in our inference run. During the final step of last noise scale Jr,, we perform 1 step of gradient descent instead of LD to obtain an better denoised version of the of these curated solutions along with their nearest neighbors in the
generations. In Fig. 6 display training set which are obtained by calculating the La distance between all training set examples.
Nearest ground truth
Nearest gro truth
Nearest ground truth
Nearest ground truth
Nearest ground truth
Prediction
Predition
10
Figure 6: Comparison of training set samples to nearest generated a using annealed Langevin dy- namics.
A.4 ENERGY-BASED PRIOR SENSITIVITY ANALYSIS
In this section, provide ing data. For fairesss in comparison w.r.t original experiments, we fix the EBM architecture and all training hyperparameters which were originally used for the main results mentioned in Appendix [A.3] except for the batch size so as to cater for small training sample sizes and better generalization. We additionally employ early stopping with patience of 100 epochs by monitoring the validation loss in order to avoid problems such as over-fitting and memorization of data. After training with DSM all the EBMs, sample from the hest and worst energy models trained on the 1% subset data using annealed Langevin dynamics and generate synthetic data shown in Figs. respecciiiivve,,
additional details understand EB prior sensitivity the quantity of train-
The ground truth training set samples are provided in Fig. 7 for visual comparison. As seen in these figures, despite being trained on such small sample sizes, the FBMSs are capable of understanding salient features such anomaly shape, anomaly conductivity values. Moreover, these models display the ability infer
15

---

Published as a conference paper at ICLR 2023
Figure 7: Ground truth data from training set
essential rules for our mathematical model such as, anomalics should not touch cach or the circular boundaries. In case of the best 1% data EBM in Fig. 8 it even shows capabilities generate unscen modes which contains than
three anomalies modes in training set and of different shapes and conductivities.
A.5 FULL-INVERSE EIT
share some implementation details and evaluations much harder and highly ill posed inverse problem ETT. By problem definition, given a set of boundary voltages at N₂ electrode locations and corresponding current the goal is to recover the target and inside ? jointly while all the networks are initiated from scratch. We train all PINN models jointly.
LU
Lo Ne
(12)
K
Lo Lo +
lob
T
Σ Va.
[802
[
beon,
den
(13)
+ UU 89
Chings
< -
AEQUBR
Eq. 13 we use 8 which correspond multiple current patterns g from Eq. in our joint training scheme. Here, each current pattern is leamt a separate u-Net like model and one o-Net model learns the conductivity distribution inside ?. Essentially, we obtain A loss values from K current patterns which are then averaged and used to update every u-Net and deNet model simultancously. We present some of the phantoms along with their a-Net reconstructions used evaluation of this joint training strategy in Fig.
As seen Eq. 12
16

---

Published as a conference paper at ICLR 2023
23
Figure 8: Uncurated synthetic samples generated from the best EBM trained on 1% of the training set.
10 As seen in this figure, we able recover accurately for simple anomalies like row However, when the phantom configuration becomes complex, the joint training fails. reserve this for our future work in scope of improving this method further.
A.6 FINITE ELEMENT METHOD BASELINE
We provide an additional baseline which is based on Finite Element Method (FEM) reconstructions. More particularly. we utilize Levenberg-Marquar (LM) (Horesh et al., (007) which is an iterative second order gradient-based optimization algorithm and compare its performance against both our proposed semi-inverse and full-inverse EIT setups. The problem formulation according LM shown below:
L. = arg min ||F(a) BV,11 + v||w||
(14)
Here in Eq. 14| F(a) is forward operator that maps the conductivity distribution into the data space, BV, the given boundary measurements at designated electrode locations. And lastly, w is regularization operator with temperature e which allows for inclusion of any prior information and generally encourages goodness of data fitting properties through the data residual. We then use an iterative approach to compute the gradient and Hessian H optimize the final objective function
follows:
V H g
where;
H +J++A++VV?
(15)
J (F66 - 2 - v.w
The term J in Eq. [5 refers the Jacobian matrix which is computed by DBV, /do is the identity matrix and A hyper-parameter control the strength of Hessian. The LM method explicitly employs second
17

---

Published as a conference paper at ICLR 2023
Figure 9: Uncurated synthetic samples generated from the worst EBM trained on 1% of the training set.
order derivatives inside the objective while using only the measurements the electrodes to reconstruct the images. Hence, it is fair when LM is compared against full-inverse EIT PINN both methods similar data compute the final solution. We provide Fig. 10 and Table [S] in order to showcase the performance of our proposed approach. As seen here, our method outperforms LM on all metrics and achieves higher accuracy while pinpointing the location of anomaly. Lastly, although it is unfair to compare LM to semi- -I'- as the semi- EIT problem has access to the data from the internal domain, provide results the phantoms displayed in Fig. 2 for the sake of completeness. The corresponding results shown in Table and Fig. 12
Table 5: Comparison of Full-EIT augmented with EB Prior and structions from phantoms seen in Fig. 10
FEM recon-
Full-EIT with EB Prior
Phantom
Metrie
Levenberg-Marquaat
MSE
1.63
1.30
0
PSNR
16.04
16.90
MDE
0.10
0.022
0.36
PSNR
18.46
23.53
MDE
0.31
0.20
MSE
1.01
0.12
Q4
PSNR
14.02
23.26
0.26
A.7 COMPUTATION COST AND MEMORY USAGE ANALYSIS
We present the last set of results by comparing the computation cost of PINN training with and without inclusion of the EB prior. The histograms in Figs. 13 show the computation time (in seconds) using various
18

---

Published as a conference paper at ICLR 2023
Molto CCrund Duts
Porten Pedictee
Phettm Enpr
Mootum soround man
#orten Prcictiie
Mastion MRd Eror
Figure 10: Unsupervised Full Inverse EIT evaluation considered phantoms.
Phantom
Prantom
Phartom
Phantom
Phantom
Ground truth
Ground truth
Ground trush
Ground truth
Ground truth
15.0
15.0
FEM Recotrraaatoonn
FEM Reconstruction
FEM Recorstruction
FEM Recor struction
FEM Reconstruction
150
Figure 11: Comparison of ground truth phantoms used for Full-EIT in Fig. and predictions from FEM-based iterative gradient Levenberg-Marquaat
Nvidia GPUs along with the peak memory consumption (in MB) for 500 of PINN without prior and 500 PINN with EB prior. It straightforward to infer that the PINN without EB prior faster putation time and uses less memory however, the numbers One on an average, the compute for PINN with EB priors is slightly less than twice that of normal PINN, Given the massive performance and stability guarantees, the meagre increase in time and compute cost is admissible.
our proposed framework are quite comparable.
19

---

Published as a conference paper at ICLR 2023
Phantom 1
Phantom
Phantom 3
Ground
truth
2
TEM Reconstruction
FEM Reconstruction
FEM Reconstruction
2
Figure 12: Comparison of ground truth phantoms used for Semi-EIT in Fig. 2 and predictions from FEM-based iterative gradient Levenberg-Marquane
Table 6: Comparison of Semi-EIT augmented with EB Prior and Maaanbeerr-Maquurddd FEM recon-
structions from phantoms seen in Fig.
Semi ETT
Phantom
Metric
Levenberg-Marquaee
with Prior
MSE
1.59
0.03 0.000
0
PSNR
12.07
29.94 1: 0,146
MDE
0.07
0.12 0.014
MSE
1,63
0.06
Q
PSNR 1
16.04
30,58 = 0.079
MSE
0,39
0.01 = 0:0001
0g
PSNR
18.18
35.46 1: 0.088
MDE
0.001
0.11 0.009
MSE
0.99
0.01 = 0.0001
94
PSNR
18.14
38.45 = 0.052
MDE
0.02
0.12 + 0.011
MSE
6.66
0.22 + 0.004
PSNR
15.39
30.17 0.082
MDE
0.40
0.08 0.016
PINN compute times with various Nvidia GPUs
PINN pesk memory usage
A100
Expermment trequens
Experimett Trequency
25-
V100
A100
250
500
750
10000
1250
1500
1750
2000
2250
30
40
60
70
Time (o0cs)
Nemory (MB)
Figure 13: Histogram analysis of computation cost and memory usage for our proposed EB prior with PINN framework on 1000 random runs.
20


Reviews:
[{"review_id": "vEe9avPUbE7", "reviewer": "Reviewer_bAau", "summary": "", "questions": "", "limitations": "", "rating": 10, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The manuscript is using PINN to solve the problem of electrical inverse tomography. It provides examples where PINNs circumvent the instability of the EIT problem, it creates a data-driven energy-based model to improve the accuracy of the EIT reconstruction. The problem is not only mathematically interesting but has a significant practical use in multiple domains, not least in medicine.\n\nCareful analysis is done on the different regularisation methods. These  show that viable solutions are found with regularisation. The traditional solutions are computed for comparison with different degrees of noise. ", "summary_of_the_review": "Excellent and informative paper.", "strength_and_weaknesses": "This is an excellent paper. Introduction of the energy based criteria that quantifies the non-physical domain for the solutions is innovative and effective. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "l_79cQw-kOk", "reviewer": "Reviewer_WoWz", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces the use of Bayesian approach as prior for stabilizing and helping the training of Physics Inspired Neural Network (PINN) for solving the semi-inverse Electrical Impedance Tomography. ", "summary_of_the_review": "The authors present an interesting approach for stabilizing the training of PINN for EIT problems. The paper is well presented, and the results are convincing. It is a pity that the authors have only focused on the EIT semi inverse problem, and not the full inverse problem.", "strength_and_weaknesses": "Strengths:\n1.\tThe proposed approach was demonstrated to be outperforming the strong solution proposed by Bar and Sochen, with faster convergence and more robustness toward the learning rate.\n2.\tThe authors demonstrated the added value of EBM priors for different EIT approach, and also highlight the added value of added hinge loss.\n3.\t\nWeaknesses:\n\n1.\tThe authors have only focused on the semi invers problem, whereas in real EIT experiments, one will only have access to current measurement on the body torso. Could the authors explain why they restricted themselves to the semi-inverse problem? \n2.\tWhy did the authors not implement the classical FEM approach for solving the EIT problem, in order to compare their approach to a standard technique? \n3.\tAlso, it would be interesting for the authors to report computational times and also memory usage. As this would help for comparison purpose.\n4.\tCould the authors also discuss how their proposed approach could extended to the 3D case?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Tk9qSV7YeC", "reviewer": "Reviewer_FzKV", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to incorporate energy-based priors to improve the training of physics-informed neural networks (PINN), for solving partial differential equation (PDE) based inverse problems. The method is mainly validated in the application of electrical impedance tomography (EIT).", "summary_of_the_review": "The paper studies an interesting problem and proposes an interesting idea to incorporate energy-based priors to improve the training of physics-informed neural networks (PINN). But there may still be some concerns as aforementioned. ", "strength_and_weaknesses": "Strength:\n- The paper is well written and organized.\n- The studied problem and the proposed idea is interesting intersecting two areas of PINN and energy-based model, particularly denoising score matching (DSM)\n- The results of simulation experiments and ablative study support the improvement of incorporating energy-based priors in the application of EIT\n\nWeakness:\n- How to train and obtain a EBM prior is an important and interesting part in the proposed method. For example, DSM always may require a large-scale dataset for training, but the paper “generates 6512 such smoothed phantoms for EBM prior training and 1628 for testing.” How can the author decide this is a sufficient dataset to train the EBM prior. Moreover, the distribution captured by EBM prior is always dependent on the training data. How could the trained EBM prior generalize to other cases or how easy is it to train an adaptive EBM prior for different application cases?\n- The paper mentions that “ training of PINN is extremely sensitive to interplay between different loss terms and hyper-parameters, including the learning rate.” But for the proposed method, as shown in Eq. (11) as well as the training of EBM prior, there are still many hyper-parameters that need to be decided. Although Sec. 4.4 shows that with EBM prior, the model seems to have a higher training success rate, the training of EBM prior also depends on the different hyper-parameters choices and may also raise other problems during the training. How practical it could be to choose a reasonable set of hyper-parameters when generalized to different application cases?\n- The paper conducts all the experiments on the simulation data. Is there any real dataset that could be used to validate the effectiveness of the proposed method in real applications? Especially, for the real data, is it still possible to get a decent scale of dataset to train the EBM prior and how efficient is it?\n-For comparison among different baseline methods, beyond the quantitative results shown in Table 2, can the paper also provide the qualitative results comparison? It would be interesting how these methods may distinguish in the real application?\n- The idea of combining PINN and DSM prior for solving inverse problems is interesting. The author introduces this mainly from the perspective of PINN and  solving the EIT inverse problem as in Sec. 5. There are also a series of researches recently for using DSM models for inverse problem solving such as medical image reconstruction by incorporating physics priors. How could this paper be relevant to those line of research also for inverse problem solving?\n- In Table 2, the paper citations of titles are all missing?\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster