Transformer Meets Boundary Value Inverse Problems
Authors: Ruchi Guo
Shuhao Cao
Long Chen
Venue: ICLR 2023 poster (2023)
Keywords: inverse problems
attention
operator learning
Transformer
partial differential equations

Abstract: A Transformer-based deep direct sampling method is proposed for electrical impedance tomography, a well-known severely ill-posed nonlinear boundary value inverse problem. A real-time reconstruction is achieved by evaluating the learned inverse operator between carefully designed data and the reconstructed images. An effort is made to give a specific example to a fundamental question: whether and how one can benefit from the theoretical structure of a mathematical problem to develop task-oriented and structure-conforming deep neural networks? Specifically, inspired by direct sampling methods for inverse problems, the 1D boundary data in different frequencies are preprocessed by a partial differential equation-based feature map to yield 2D harmonic extensions as different input channels. Then, by introducing learnable non-local kernels, the direct sampling is recast to a modified attention mechanism. The new method achieves superior accuracy over its predecessors and contemporary operator learners and shows robustness to noises in benchmarks. 
This research shall strengthen the insights that, despite being invented for natural language processing tasks, the attention mechanism offers great flexibility to be modified in conformity with the a priori mathematical knowledge, which ultimately leads to the design of more physics-compatible neural architectures. 

Published as a conference paper at ICLR 2023
TRANSFORMER MEETS
BOUNDARY VALUE INVERSE PROBLEMS
Ruchi Guo
Shuhao Cao
Department of Mathematics University of California, Irvine
Division of Computing, Analytics, and Mathematics
School of Science and Engineering
University of Missouri-Kansar City
Long Chen
Department of Mathematics University of California, Irvine
ABSTRACT
A Transformer-basee deep direct sampling method is proposed for electrical impedance tomography, a well-known severely ill-posed nonlinear boundary value inverse problem. A real-time reconstruction achieved by evaluating the learned inverse operator between carefully designed data and the reconstructed images. An effort is made to give a specific example to a fundamental question: whether and how one can benefit from the theoretical structure of mathematical problem deep neural networks? Specifically,
develop task-oriented and cructure-conformin inspired by direct sampling methods for inverse problems, the 1D boundary data in different frequencies are preprocessed by a partial differential equation-based feature map to yield 2D harmonic extensions as different input channels. Then, by introducing learnable non-local kernels, the direct sampling is recast to mod- ified attention mechanism. The new method achieves superior accuracy over its predecessors and contemporary operator learners and shows robustness to noises in benchmarks. This research shall strengthen the insights that, despite being invented for natural language processing tasks, the attention mechanism offers great flexibil- ity to be modified in conformity with the a priori mathematical knowledge, which ultimately leads the design of more physics-compatible neural architectures.
INTRODUCTION
Boundary value inverse problems aim recover the internal structure distribution of multiple media inside an object (2D reconstruction) based on only the data available on the boundary (1D) signal input), which arise from many imaging techniques, e.g., electrical impedance tomography (EIT) (Holder, 2004), diffuse optical tomography (DOT) (Culver et al., 2003), magnetic induction tomography (MIT) (Griffiths al., 1999). Not needing any internal data renders these techniques generally non-invasive, safe, cheap, and thus quite suitable for monitoring applications.
In this work, we shall take ETT an example illustrate how a more cennctre-cooformiin neural network architecture leads to better results in certain physics-base based tasks. Given a 2D bounded domain Ω and an inclusion D, the forward model the following partial differential equation (PDE)
Vu)
in 2 where a = 01 D, and a = ao in D,
(1)
where a a piecewise constant function defined on 52 with known function values do and σı, but the shape of the inclusion D buried in Ω is unknown. The goal is to recover the shape of D using only the boundary data on dΩ (Figure 1). Specifically, by exerting a current on the boundary, one solves (I) with the Neumann boundary condition Vu n|an g. where is the outwards unit normal direction of öZ, to get a unique u on the whole domain Ω. practice, only the Dirichlet boundary value representing the voltages Neumann-to-Dirichh (NtD) mapping:
ulaa the boundary can be measured. This procedure is called
Ag H 1/22 172/2(00), with g Vv a0 f ulag-

---

Published as a conference paper at ICLR 2023
F: V (oVu) = 0, Vv.. nl ト f = u 80
H
To
fg
0, (tensor2tensor DNN) (PDE-based feature map) x FLD L.
Figure I: Schematics of the pipeline: the approximation of the inverse operator decomposed into (i) a PDE-based feature map H: the ID boundary data is extended to 2D features by harmonic extension; (ii) a tensor2tensor neural network To that outputs the reconstruction.
For various notation and the Sobolev space formalism, we refer readers Appendix A; for brief review of the theoretical background of EIT we refer readers to Appendix B. The NtD map above in (2) can expressed as
f Arg,
where g and are (infnite-dimensionaal) vector representations of functions g and f relative chosen basis, and A, is the matrix representation of A, (see Appendix B for an example).
The original mathematical setup of EIT is to the NtD map Ar in (2) to recover o, referred to as the case of full measurement (Calderón, 2006). In this case, the forward and inverse operators associated with EIT can be formulated as
Ar. and F Ag
Fix set of basis {g1) of the corresponding Hilbert space containing all admissible currents. Then, mathematically speaking, "knowing the operator Ag means that one can measure all current-to-volle -ooltaee pairs {gi, == A. and construct the infiiie-dimmensoonal matrix Ag.
However, as infinitely many boundary data pairs are not attainable in practice, the problem of more practical interest is to use only a few data pairs {(g,, fi) for reconstruction. In this case, the forward and inverse problems can be formulated as
a. (5) For limited data pairs, the inverse operator Fr. is extremely ill-posed or even not well-defined (Isakov & Powell, 1990; Barceló et al., 1994; Kang & Seo, 2001; Lionheart, 2004); namely, the same boundary measurements may correspond to different In view of the matrix representation A,, for gi = e₁, = 1, ..., L, with e being unit vectors of a chosen basis, (f,,.., fz) only gives the first L columns of A, It is possible that two matrices A, and A, have similar first L columns but ||o 5 large.
FL + (G),, Aagi),.... (gr, Ag9t)) and Fr.1
(91,))
How to deal with this "ill-posedness" is central theme in boundary value inverse problem theories. The operator learning approach has the potential to tame the ill-posedness by restricting F,.1 at a set of sampled data ID == ( Jk=1 with different shapes and locations following certain distribution. Then the problem becomes approximate
FIb: : {(11, Aumn))),.))) (gL, (...m)mg))))))) + (A)) k=
The fundamental assumption here is that this map is "well-defined" enough to be regarded as high-dimensional interpolation (learning) problem on compact data submanifold (Seo al., 2019; Ghattas & Willcox, 2021), and the learned approximate mapping can be evaluated at newly incoming a's. The incomplete information of A, due small L for one single a is compensated by large N > 1 sampling of different o
2 BACKGROUND, RELATED WORK, AND CONTRIBUTIONS
Classical iterative methods. There are in general two types of methodology to solve inverse problems. The first one is large family of iterative or optimization-based methods (Dobson &
2

---

Published as a conference paper at ICLR 2023
Santosa, 1994; Martin & Idier, 1997; Chan & Tai, 2003; Vauhkonen et al., 1999; Guo et al., 2019; Rondi & Santosa, 2001; Chen et al., 2020; Bao et al., 2020; Gu et al., 2021). One usually looks for an approximated a by solving minimization problem with regularization R(o)) to alleviate the ill-posedness, say
inf, { 11 fi fillan + R(o)).
(7)
The design of regularization R(o) plays a critical role in successful reconstruction (Tarvainen et al., 2008; Tehrani et al., 2012; Wang et al., 2012). Due to the ill ill-posedness, the computation for almost all iterative methods usually takes numerous iterations to converge, and the reconstruction is highly sensitive to noise. Besides, the forward operator F(-) needs to be evaluated at each iteration, which itself expensive as requires solving forward PDE models.
Classical direct methods. The second methodology is develop well-defined mapping Go parametrized by 8, empirically constructed to approximate the inverse map itself. say Go = F-1. These methods are referred to non non-iterative or direct methods in the literature. Distinguished from iterative approaches, direct methods are typically highly problem-specific, as they are designed based on specific mathematical structures of their respective inverse operators. For instance, methods in EIT and DOT include factorization methods (Kirsch & Grinberg. 2007; Azzouz et al., 2007: Brühl, 2001; Hanke & Brühl, 2003), MUSIC-type algorithms (Cheney, 2001: Ammari & Kang, 2004; 2007; Lee et al., 201 and the D-bar methods ( (Knudsen et al., 2007: 2009) based on a Fredholm integral equation (Nachman, 1996), among which are the direct sampling methods (DSM) being our focus in this work (Chow et al., 2014; 2015; Kang et al., 2018; Ito et al., 2013; Ji et al., 2019; Harris & Kleefeld, 2019; Ahn et al., 2020; Chow et al., 2021; Harris et al., 2022). These methods generally have closed-form Go for approximation, and the parameters 0 represent model-specific mathematical objects. For each fixed A. this procedure is usually much more stable than iterative approaches with respect to the input data. Furthermore, the evaluation for cach boundary data pair is distinctly fast, as no optimization is needed. However, a simple closed-form Go admitting efficient execution may not be available in practice since some mathematical assumptions and derivation may not hold. For instance, MUSIC-type and D-bar methods generally require an accurate approximation to Ag, while DSM poses restrictions on the boundary data, domain geometry, etc., see Appendix D for details.
Boundary value inverse problems. For most cases of boundary value inverse problems in 2D) the major difference, e.g., with an inverse problem in computer vision (Marroquin et al., 1987). that data are only available on ID manifolds, which are used to reconstruct 2D targets. When comparing (4) with a linear inverse problem in signal processing y Ax + C, to recover signal x from measurement y with noise c, the difference is more fundamental in that F(.) itself is highly nonlinear and involves boundary value PDEs. Moreover, the boundary data themselves generally involve certain input-output structures (NtD maps), which adds more complexity. In Adler & Guardo (1994); Frianndez-Fuentee et al. (2018); Feng et al. (2018), boundary measurements are collected and directly input into feedforward fully connected networks. As the data reside on different manifolds, special treatments are made to the input data, such as employing pre- rrerrcconstruction stages to generate rough 2D input to CNNs (Ben Yedder et al., 2018; Ren et al., 2020; Pakravan et al., 2021).
Deep neural network and inverse problems. Solving an inverse problem is essentially to give a satisfactory approximation to F-1 but based finitely many measurements. The emerging deep learning (DL. based on Deep Neural Networks (DNN) to directly emulate operators significantly resembles those classical direct methods mentioned above. However, operator learners by DNNs are commonly considered black boxes. A natural question is how the priori mathematical knowledge can be exploited to design more thyyics--coppttin DNN architectures. In pursuing the answer to this question, we aim to provide a supportive example that bridges deep learning techniques and classical direct methods, which improves the reconstruction of EIT.
Operator learners. Operator learning has become an active research field for inverse problems in recent years, especially related to image reconstruction where CNN plays a central role, see, e.g., Klosowski & Rymarczyk (2017); Nguyen et al. (2018); Tan et al. (2018); Jin et al. (2017); Kang et al. (2017); Barbastathis et al. (2019); Latif et al. (2019); Zhu et al. (2018); Chen et al. (2021): Coxson et al. (2022): Zhu et al. (2023b). Notable examples of efforts to couple classical reconstruction methods and CNN include Hamilton et al. (2019); Hamilton & Hauptmann (2018), where a CNN post-processes images obtained by the classical D-bar methods, and Fan et al. (2019); Fan & Ying (2020) where BCR-Net developed to mimic pseudo-differental operators appearing in many inverse
3

---

Published as a conference paper at ICLR 2023
problems. A deep direct sampling method is proposed in Guo & Jiang (2020); Guo al. (2021) that learns local convolutional kernels mimicking the gradient operator of DSM. Another example is radial basis function neural networks in Hrabuska al. (2018); Michalikova et al. (2014); Wang et al. (2021a). Nevertheless, convolutions in CNNs use kernels whose receptive fields involve only a small neighborhood of a pixel. Thus, layer-wise, CNN does not align well with the non-local nature of inverse problems. More recently, the learning of PDE-related forward problems using global kernel has gained attraction, most notably the Fourier Neural Operator (FNO) (Nelsen & Stuart, 2021; Li et al., 202 la; Kovachki et al., 2021: Guibas et al., 2022; Zhao et al., 2022; Wen et al., 2022; Li et al., 2022b). FNO takes advantage of the low-rank nature of certain problems and learns local kernel in the frequency domain yet global in the spatial-temporal domain, mimicking the solution's is kernel integral form. Concurrent studies include DeepONets (Lu et al., 2021; Wang et al., 20; Jin et al., 2022b), Transformers (Cao, 2021; Kissas et al., 2022; Li et al., 2022a; Liu et al., 2022; Fonseca et al., 2023), Integral Autoencoder (Ong ct al., 2022), Multiwavelet Neural Operators (Gupta et al., 2021; 2022), and others (Lütjens et al., 2022; Hu et al., 2022; Boussif et al., 2022; de Hoop et al., 2022a;;; Ryck & Mishra, 2022; Seidman et al., 2022; Zhang et al., 2023; Lee, 2023; Zhu et al., 2023a).
Related studies on Transformers. The attention mechanism based models have become state of the art in many areas since Vaswani et al. (2017). One of the most important and attractive aspects of the attention mechanism is its unparalleled capability to efficiently model non-local long-range interactions (Katharopoulos al., 2020; Choromanski et al., 2021; Nguyen al., 2021). The relation of the attention with kernel learning is first studied in Tsai et al. (2019) and later connected with random feature (Peng et al., 2021). Connecting the non-PDE-based integral kernels and the attention mechanism has been seen in Hutchinson et al. (2021); Guibas et al. (2022); Nguyen et al. (2022); Han et al. (2022). Among inverse problems, Transformers have been applied in medical imaging applications, including segmentation (Zhou et al., 2021: Hatamizadeh et al., 2022: Petit et al., 2021), X-Ray (Tanzi et al., 2022), magnetic resonance imaging (MRI) (He et al., 2022), ultrasound (Perera et al., 2021), optical coherence tomography (OCT) (Song et al., 2021). To our best knowledge, no work in the literature establishes an architectural connection between the attention mechanism in Transformer and the mathematical structure of PDE-based inverse problems.
2.1 CONTRIBUTIONS
struecture-oofforming network architecture. Inspired by the EIT theory and classic DSM, we decompose the approximation of the inverse operator into a harmonic extension and an integral operator with learnable non-local kernels that has an attention-like structure. Additionally, the attention architecture is reinterpreted through Fredholm integral operator to rationalize the application of the Transformer to the boundary value inverse problem.
Theoretical and experimental justification for the advantage of Transformer. We have proved that, in Transformers, modified attention can represent target functions exhibiting higher frequency natures from lower frequency input features. A comparative study in the experiments demonstrates favorable match between the Transformer and the benchmark problem.
3 INTERPLAY BETWEEN MATHEMATICS AND NEURAL ARCHITECTURES
In this section, we try to articulate that the triple tensor product in the attention mechanism matches exceptionally well with representing solution in the inverse operator theory of EIT. In pursuing this end goal, this study tries to answer the following motivating questions:
(Q1) What is an appropriate finite-dimensionl data format as inputs to the neural network?
(Q2) Is there suitable neural network matching the mathematical structure?
3.1 FROM EIT TO OPERATOR LEARNING
In the case of full measurement, the operator F-1 can be well approximated through a large number of (a, A,) data pairs. This mechanism essentially results in a tensor2tensor mapping/operrator from A, to the imagery data representing σ. In particular, the BCR-Ne Net (Fan & Ying, 2020) is a DNN approximation falling into this category. However, when there are very limited boundary data pairs accessible, the task of learning the full matrix A, becomes obscure, which complicates the development of tensor2tensor pipeline for operator learning.
4

---

Published as a conference paper at ICLR 2023
Operator learning problems for EIT. We first introduce several attainable approximations of ciinttedimmnnsionll spaces by miiee-iimensional counterparts for the proposed method.
(1) Spatial discretization. Let Ω2 be a mesh of Ω with the mesh spacing and let {2y) == M be the set of grid points to represent the 2D discretization of continuous signals. Then function u defined almost everywhere in can be approximated by a vector 4n E R
(2) Sampling of D. We generate N samples of D with different shapes and locations following certain distributions. For example, elliptical inclusions with random semi-axes and centers are generated as benchmark (see Appendix C.1 for details). With the known do and Ø1, set the corresponding data set D = {1(1) ((() (..,,(N)), N is usually large enough to represent field applications of interest.
(3) Sampling of NiD maps. For the k-th sample of D, we generate L pairs of boundary data [(9) (8) ))"") by solving PDE (1), which can be thought of sampling of columns of the infinite matrix A, representing the NtD map. By the proposed method, L can be chosen to be very small (S 3) to yield satisfactory results.
Our task is to find a parameterized mapping Go to approximate FT (6) by minimizing
J(0)= (G f/ -
for suitable norm || Il. Hyper-parameee N, h, L will affect the finite sitte-diimsiionall approximation to the infiniteness in the following way: determines the resolution to approximate D; N affects the representativity of the training data set; L decides how much of a finite portion of the infinite spectral information of Ag can be accessed.
3.2 FROM HARMONIC EXTENSION TO TENSOR-TO-TENT
To establish the connection between the problem of interest with the attention used in the Transformers, first construct higher-dimen dimeeeridinsii tensors from the ID boundary data. The key is harmonic extension of the boundary data that can be viewed as a PDE-based feature map. We begin with a theorem to motivate it.
Let D be the characteristic function of D named index function, i.e., [x) = 1 E D and (x) 0f x & D. Thus, can be directly identified by the shape of D through the formula - D)). In this setup, reconstructing o is equivalent to reconstructing I"
σ = -= + +
Without loss of generality, whole domain, i.e., it is taken as the known background conductivity (no inclusion), and thus A 9 be readily computed. Then f A
let 0 > on- A., is understood as the NtD map with do on the
(Ag Aao )g measures the difference between the Ntl mappings and encodes the information of a. The operator A, Aoo is positive definite, and it has cigenvalues { with A A2 >..>> 0 (Cheng et al., 1989).
Theorem 1. Suppose that the ID boundary data gi is the eigenfunction of Ao Aoo corresponding to the 1-th eigenvalue A and let the 2D data functions Φr be obtained by solving
-d 0 Ω n n Voh = (Ji - -oogt) on an, Son di ds =0 0,
for 1 1,2,... Define the space Sr. = Span(Or, di Or dh I. I.I....IJ.. and the dictionary Sr. = fai + an arctan(ago) : u e S,, 01 a2 ag E R). Then, for any e > O, we con construct an index function IP, E St. s.l. supzen [xD I (+)) < provided L is large enough.
The full proof of Theorem I can be found in Appendix E. This theorem gives constructive approach for approximating F and justifies the practice of approximating I when L is large enough. The function 6i is called the harmonic extension of fi Aa,
On the other hand, it relies on "knowing" the entire NtD map Ag to construct ZíP. explicitly. Namely, the coefficients of 0z, &i Ōz depend big chunk of spectral information (eigenvalues and eigenfunctions) of Ag, which may not be available in practice. Thus, the mathematics itself this theorem does not provide architectural hint on building
cooffurr-conforming DNN.
To further dig out the hidden structure, we focus on the case of a single measurement, i.c., L = 1. With this setting, it is possible derive explicit and simple formula to approximate DD which
5

---

Published as a conference paper at ICLR 2023
is achieved by the classical direct sampling methods (DSM) (Chow et al., 2014; 2015; Kang et al., 2018; Ito et al., 2013; Ji et al., 2019; Harris & Kleefeld, 2019; Ahn et al., 2020). For EIT.
I (x) 2 IP(x)) == R(x) (d(x)-
Vo(z)) a E Ω d(x) IR?,
(10)
derived in Chow et al. (2014), where (see a much more detailed formulation in Appendix D)
Φ is the harmonic extension of f Aoog with certain noise f A,,g + 5; ((x) is called probing direction and can be chosen empirically as d(z) Vo(=)/
R(z) (Ilf (oongllllnnllll where ?he is a function of d(x) and measured in -l
semi-norm on boundary 02.
Both and 7/2 can be computed effectively by traditional fast PDE solvers, such as finite difference or finite element methods based on S2, in Section 3.1. However, the reconstruction accuracy is much limited by a single measurement, the nonparametric ansatz, and empirical choices of d(x) and - These restrictions leave room for DL methodology. See Appendix D for detailed discussion.
Constructing harmonic extension (2D features) from boundary data (ID signal input with limited depth) contribute the desired high-qualij, reconstruction. First, harmonic functions highly smooth away from the boundary, of which the solution automatically smooths out the noise on the boundary due to PDE theory (Gilbarg & Trudinger, 2001, Chapter 8), and thus make the reconstruction highly robust with respect to the noise (e.g., see Figure in Appendix C.1). Second, in terms of using certain backbone networks to generate features for downstream tasks, harmonic extensions can be understood as a problem-specific way to design higher dimensional feature maps (Álvarez et al., 2012). which renders samples more separable in a higher dimensional data manifold than the one with merely boundary data. See Figure to illustrate this procedure.
The information of a is deeply hidden in Φ. As shown in Figure 1 (see also Appendix C), one cannot observe any pattern of a directly from 6. It is different from and more challenging than the inverse problems studied in (Bhattacharya et al., 2021; Khoo et al., 2021) that aim to reconstruct 2D targets from the much more informative 2D internal data of
In summary, both Theorem and the formula of DSM (10) offer inspiration to give a potential answer (QI): the harmonic extension & (2D features) of f- Aoog (ID measurements) naturally encodes the information of the true characteristic function Ip (2D targets). As there is pointwise correspondence between the harmonic extensions and the targets at 2D grids, a tensor representation of V at these grid points can then be used as the input to rensor2tensor-typp DNN to learn Ip. Naturally, the grids are set as the positional embedding explicitly. In comparison, the positional information is buried more deeply in ID measurements. As shown in Figure 1, Go can be nicely decoupled into composition of learnable neural network operator To and a non-learnable PDE-based feature map H. i.e., Gg To H. The architecture of To shall be our interest henceforth.
3.3 FROM CHANNELS ATTENTION TO BASIS IN INTEGRAL TRANSFORM
In this subsection, modified attention mechanism is proposed as the basic block in the tensor2tensor- type mapping introduced in the next two subsections. Its reformulation conforms with one of the most used tools in applied mathematics: the integral transform. In many applications such as inverse problems, the interaction (kernel) does not have any explicit form, which meshes well with DL methodology philosophically. In fact, this is precisely the situation of the EIT problem considered.
Let the input of an encoder attention block be XΛ2 E IR Mxe with c channels, then the query Q, key K, value V are generated by three learnable projection matrices 0 == (w W WV} C Rexe, o = x1, Wo, o E {Q, K. V). Here e > L is the number of expanded channels for the latent representations. A modified dot-product attention is proposed as follows:
= Attn(x))= = a
=a(kk =
E
(1))
where nlg(.) and nlx(-) are two learnable normalizations, Different from Nguyen & Salazar (2019); Xiong et al. (2020), this pre-inner-produs normalization is applied right before the matrix multiplication of query and key. This practice takes inspiration from the normalization in the index function kernel integral (10) and (20), see also Boyd (2001) where the normalization for orthogonal bases essentially uses the (pseudo)inverse, of the Gram matrices. In practice, layer normalization (Ba

---

Published as a conference paper at ICLR 2023
et al., 2016) or batch normalization (loffe & Szegedy, 2015) is used as a cheap altermative. Constant a = h2 is a mesh- based weight such that the summation becomes an approximation to an integral.
To elaborate these rationales, the j-th column of the i-th row U, of U is (U,)² a A.. V in which the i-th row A,. (QK i and the j-th column V := V.j. Thus, applying this every column 1 S j S c, attention (11) becomes a basis expansion representation for the i-th row U;
Vi
U. a Ai.
:
Σ
M m=1 ar Aim =: > m=1
M
A(Qi, Km) Vm-
(12)
VM
Here, a,, contains the coefficients for the linear combination of (Vm ) This set (Vm m J forms the V's row space, and it further forms each row of the output U by multiplying with A. A(:-) in (12) stands for the attention kernel, which aggregates the pixel-wise feature maps to measure how the projected latent representations interact. Moreover, the latent representation U, in an encoder layer is spanned by the row space of V and is being nonlinearly updated cross-layer-wise,
m=
For X U, Q. K, V. set of feature maps assumed to exist: for example u(-) maps IR - Ixxe (2)),,, e.g., see Choromanski et al. (2021), then an instance- dependent kernel KO (o R R22 - R can be defined by
ie.. Ug = u(z)) = [uy (2i)
A(Q K2) =- a (Q,,, K
(00(z))(())))))) =: a (o 2z).
(13)
Now the discrete kemnel A(..)) with tensorial input is rewritten to this kernel ke(-,),, thus the dot-product attention expressed as a nonlinear integral transform for the (-th channel:
(2)) a ExEM (9(z) (z))) 6₂ ~ Sq (0(2, ())((z) du(z), 1s (14)
Through certain minimization such as (8), the backpropagation updates which further leads a new set of latent representations. This procedure can be viewed as an iterative method to update the basis residing in each channel by solving the Fredholm integral equation of the first kind in (14).
To connect attention with inverse problems, the multiplicative structure in a kernel integral form for attention (14) is particularly useful. (14) type of secherle-Goursal (degenerate) kernels (Kress, 1999, Chapter 11) and approximates the full kernel using only finite number of bases. The number of learned basis functions in expansion (12) depends on the number of channels n. Here we show the following theorem; heuristically, it says that: given enough but finite channels of latent representations, the attention kernel integral can "bootstrap" in the frequency domain, that is, generating output representation with higher frequencies than the input. Similar approximation results are impossible for layer-wise propagation in CNN if one opts for the usual framelet/ wavelet interpretation (Ye et al., 2018). For example, if there no edge-like local features in the input (see for empirical evidence in Figure 9 and Figure 10), a single layer of CNN filters without nonlinearity cannot learn weights to extract edges. The full proof with a more rigorous setting is in Appendix F. Theorem 2 (Frequency bootstrapping). Suppose there exists a channel in V such that (V:), = san() for some E Σt, the current finite-channnn sum kernel A(,, -) approximates non-separable kernel to an error of Oee) under certain norm || ||x- Then, there exists a set of weights such that certain channel k in the output of (12) approximates sin(a' 2), Z+ d' > with an error of O(e) under the same norm.
The considered inverse problem is essentially to recover figher-frequeni eigenpairs of A, based on lower-frea frere-fre data, see, e.g., Figure 1. A, together with all its spectral information can be determined by the recovered inclusion shape. Thus, the existence in Theorem partially justifies the advantages of adopting the attention mechanism for the considered problem.
3.4 FROM INDEX FUNCTION INTEGRAL TO TRANSFORMER
In (10), the probing direction (()), the inner product (x)) V and the norm | II are used as ingredients to form certain non-l instance-based based learnable kernel integration. This non-localness is a fundamental trait for many inverse problems, in that I (z) depends on the entire data function. Then, the discretization of the modified index function is shown to match the multiplicative structure of the modified attention mechanism in (11).
In the forthcoming derivations, K(r,y) y), Q(x, and self-adjoint positive definite linear operator V : (a92 + L' (21)), are shown to yield the emblematic Q Q-K-V structure of attention. To this end, make the following modifications and assumptions to the original index function in (10).
7

---

Published as a conference paper at ICLR 2023
The reformulation of the index function is motivated by the heuristics that the agglomerated global information of & could be used as "keys" to locate a point 2.
(x()) R(x) / d(x) K(x, v) Voly) dy.
(15)
If an ansatz K(x, 1/) dz (v) is adopted, then (15) reverts the original one in (10).
The probing direction d ) as "query" is reasonably assumed to have global dependence ø
( = / Q(x, Voly) dy.
(16)
which is the choice of the probing direction (Ikehata, 2000; Ikehata & Siltanen, 2000; Ikehata, 2007).
If (z,,v) = 5z (z)] then d(x)
In the quantity R(x) in (10), the key is |- ly which is assumed to have the following form:
1/2 y = (V'lz The) LP(0)))
(17)
In Chow et al. (2014), it is shown that if V induces a kernel with sharply peaked Gaussian-like distribution, the index function (10) can achieve maximum values for points inside D.
Based on the assumptions from (15) to (17), derive matrix representation approximating the new index function on grid, which accords well with an attention-like architecture. Denote by Φ,, the vector that interpolates dz., & the grid points {zy]. n = 1,2.
Here, we sketch the outline of the derivation and present the detailed derivation in Appendix D. We shall discretize the variable x: by grid points 2; in (15) and obtain an approximation the integral:
J. K(zz, u)0.., (ly) dy Σ WiK(Zz, 2z) o(z)) = ",.
(18)
where {w,} are some integration quadrature weights. We then consider (16) and focus on one component du (x) d(x). With suitable approximated integral, it can be rewritten as
(z,),) ~ Σ, W/ 2(z+, 2z)0 d(zz) =: q. Φ..
(19)
Note that the self-adjoint positive definite operator V in (17) can be parameterized by symmet- ric positive definite (SPD) matrix denoted by V. There exist vectors Vn,* such that Ma, & s
On dh Vn,i vo. &.. Then, the modified indicator function can be written
(z.))) {115- Aoo9l 12 Σ Vn. (,) -1/2) Eoakko on.
(20)
Now, using the notation from Section 3.3, we denote the learnable kernel matrices and an input vector: for (Q, K, V). and (q. v)
W =
u1.1
u1.M U2.M
ER2MXM
Xh. 1 Φ @ RX2M
(21)
12
Then, rewrite (20)
| (M = Crg(xx x, ) 12/22
(22)
where Cy,y
= llf A on is normalization weight, and both and / are element-wing wise. Here, may define Q x,, @ K x, W and V xx, W as the query, keys, and values. We can see that the right matrix multiplications (11) in the attention mechanism are low-rank approximations of the ones above. Hence, based on (22), essentially we need to find function I resulting in vector approximation to the true characteristic function (2,))
I(Q, K) ~ (T))
(23)
Moreover, when there are T. data pairs, the data functions Φi are generated by computing their harmonic extensions as (9). Then, each dy is then treated as a channel of the input data X₁.-
In summary, the expressions in (22) and (23) reveal that Transformer may be able to generalize the classical non-parametrized DSM formula further in (10) to non-local learnable kernels. Thus, may have an intrinsic architectural advantage that handles multiple data pairs. In the subsequent EIT benchmarks, we provide a potential answer to the question (Q2); namely, the attention architecture is better suited for the tasks of reconstruction, it conforms better with the underlying mathematical structure. The ability to learn global interactions by attention, supported by a non-local kernel interpretation, matches the long-range dependence nature of inverse problems.
8

---

Published as a conference paper at ICLR 2023
4 EXPERIMENTS
In this section we present some experimental results to show the quality of the reconstruction. The benchmark contains sampling of inclusions of random ellipses (targets), and the input data has single channel (L = 1) of the 2D harmonic extension feature from the ID boundary measurements. The training uses Icycle and mini-batch ADAM for 50 epochs. The evaluated model is taken from the epoch with the best validation metric on a reserved subset. There are several baseline models to compare: the CNN-based U-Nets (Ronneberger et al., 2015; Guo & Jiang, 2020); the state-of-the-ar operator learner Fourier Neural Operator (FNO) (Li et al., 2021a) and its variant with token- mikkng layer (Guibas al.. 2022); MultiWavelet Neural Operator (MWO) (Gupta al., 2021). The Transformer model of interest is drop-in replacement of the baseline U-Net, and it is named by U-Integral Transformer (UIT). UIT uses the kernel integral inspired attention (II), and we also compare UIT with the linear attention- based Hybrid U-Transformer in Gao et al. (2021), as well as Hadamard product- based cross attsenttttnionn U-Transformer in Wang et al. (2022). An ablation study is also conducted by replacing the convolution layers in the U-Net with attention (11) on the coarsest level. For more details of the hyperparameters'' setup in the data generation, training, evaluation, network architectures please refer to Section 3.1, Appendix C.I. and Appendix C.2.
The comparison result can be found in Table 1. Because FNO (AFNO, MWO) keeps only the lower modes in spectra, it performs relatively poor in this EIT benchmark where one needs to recover traits that consist of higher modes (sharp boundary edges of inclusion) from lower modes (smooth harmonic extension). Attention-based models are capable recover thighffreueency target from low-frequency data", and generally outperform the CNN-based U-Nets despite having only 1/3 of the parameters. Another highlight is that the proposed models are highly robust to noise thanks the unique PDE-based feature map through harmonic extension. The proposed models can recover the buried domain under moderately large noise (5%) and an extreme amount of noise (20%) which can be disastrous for many classical methods.
Table Evaluation metrics of the EIT benchmark tests. T: the normalized relative strength of noises added in the boundary data before the harmonic extension; see Appendix C for details. 7.2-error and cross entropy: the closer to 0 the better; Dice coefficient: the closer to I the better.
Relative error
Position wise cross entropy
Dice coefficient
0.05
7 0.2
r 0.05
0).2
01.55
0.2
params
U-Net baseline
0.200
0.311
0.366
0,0836
0.132
0.143
0.815
0.810
0.799
7.7m
U-Net+Coasse Attn
0.184
0.313
0.360
0.0801
0.136
0.147
0.852
0.807
0.804
8.4m
U-Net big
0.195
0.338
0.350
0.0791
0.133
0.138
0.850
0.812
0.805
31.0m
FNO2d haseline
0.318
0.492
0.502
0.396
0.467
0.508
0.650
0.592
0.582
10.4m
Adaptive FNO2d
0.323
0.497
0.499
0.391
0.466
0.471
0.635
0.595
0.592
10.9m
FNO2d hig
0.386
0.482
0.501
0.310
0.499
0.638
0.601
0.580
33.6m
Multiwavllet NO
0.275
0.390
0.407
0.152
0.178
0.192
0.715
0.694
0.688
9,8m
Hybrid UT
0.185
0.320
0.333
0.0785
0.112
0.116
0.877
0.829
0.821
11.9m
CrossAttention UT
0.171
0.305
0.311
0.0619
0.105
0.109
0.887
0.840
0.829
11.4m
UTT+Softmax (ours)
0.159
0.261
0.269
0.0551
0.0969
0.0977
0.903
0.862
0.848
11.1m
UIT (purs) 0.163
0,261
0.272
0.0564
0.0967
0.0981
0.897
0.858
0.845
11.4m
UTT+(L=3) (ours)
0.147
0,250
0.254
0.0471
0.0882
0.0900
0.914
0.891
0,880
11.4m
CONCLUSION
For boundary value inverse problem, we propose a novel operator learner based on the mathe- matical structure of the inverse operator and Transformer. The proposed architecture consists of two components: the first one harmonic extension of boundary data (a PDE-based feature map), and the second one is a modified attention mechanism derived from the classical DSM by introducing learnable noc-l integral kernels. The evaluation accuracy on the benchmark problems surpasses the current widely-used CNN-based U-Net and the best operator learner FNO. This research strengthens the insights that the attention is an adaptable neural architecture that can incorporate a priori mathematical knowledge to design more physics-compatible DNN architectures. However, we acknowledge some limitations: in this study, a to be recovered relies on a piecewise constant assumption. For many EIT applications in medical imaging and industrial monitoring, a may involve non-sharp transitions or even contain highly anisotropic/multiscau behaviors; see Appendix G for more discussion on limitations and possible approaches.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
L. Chen is supported in part by National Science Foundation grants DMS-1913080 and DMS- 2012465, and DMS-2132710. S. Cao is supported in part by National Science Foundation grants DMS-1913080 and DMS-2136075. The hardware to perform the experiments are sponsored by and UMKC School of Science and Engineering computing facilities. No additional revenues are related to this work. The authors would like to thank Ms. Jinrong Wei (University of California Irvine) for the proofreading and various suggestions on the manuscript. The authors would like to thank Dr. Jun Zou (The Chinese University of Hong Kong) and Dr. Bangti Jin (University College London & The Chinese University of Hong Kong) for their comments on inverse problems. The authors also greatly appreciate the valuable suggestions and comments by the anonymous reviewers.
NSF grants
REPRODUCIBILITY STATEMENT
This paper is reproducible. Experimental details about all empirical results described in this paper provided in Appendix C. Additionally, we provide the PyTorch (Paszke et al., 2019) code for reproducing our results at The dataset used in this paper available at https://www. http://wwwwkgglllle.e eletrical impedance- tomography dataset. Formal proofs under a rigorous setting of all our theoretical results are provided in Appendices E-F.
10

---

Published as a conference paper at ICLR 2023
REFERENCES
Andy Adler and Robert Guardo. A neural network image reconstruction technique for electrical impedance tomography. IEEE Trans Med Imaging, 13(4):594-66 600594 1994.
Chi Young Ahn, Taeyoung Ha, Won- WwanK Park. Direct sampling method for identifying magnetic
inhomogeneities in limited-aperture inverse scattering problem. Computers & Mathematics with Applications. 80(1 80(12):811-2222) 2020. ISSN 0898-1221. URL pS1/$$8888:22120044888
Mauricio A. Álvarez, Lorenzo Rosasco, and Neil D. Lawrence. Kernels for Vector- Valued Functions: A Review. 2012.
Habib Ammari and Hyeonbae Kang. Reconstruction of Small Inhumogeneities from Boundary Measurements. Berlin: Springer, 2004.
Habib Ammari and Hyeonbae Kang. Polarization and Moment Tensors: With Applications to Inverse Problems and Effective Medium Theory. New York: Springer, 2007.
Nachman Aronszajn. Theory of reproducing kernels. Transactions of the American mathematical society, 68(3): 337-404, 1950.
Kari Astala and Lassi Päivärinta. Calderón's 's inverse conductivity prohlem in the plane. Annals Mathematics, 163(1):265-29) 2006.
Mustapha Azzouz, Martin Hanke, Chantal Ocsterlein, and Karl Schilcher. The factorization method for electrical impedance tomography data from new planar device. International journal of biomedical imaging, 2007: 83016 83016, 2007. URL httpp::/pubbed: ncbi. sicbi.nlm wim.mmm.m...mmmmmmmmmmmmmmm.. nothhhovv/83351222
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization, 2016. URL https://arxiv.. org/abs/ /a//////////..../// /611//6666666... 06450.
Guillaume Bal. Hybrid inverse problems and internal functionals. Inverse problems and applications: inside out. 11, 60:300032522 368, 2013.
Gang Bao. Xiaojing Ye, Yaohua Zang, and Haomin Zhou. Numerical solution inverse problems by weak ad- versarial networks. Inverse Probl., 36(11): 15003, nov 2020. URL ottps:///oi.orr/1e 1888/1361 2121136666666//1/ abb447.
George Barbastathis, Aydogan Ozcan, and Guohai Situ. On the use af deep learning for computational imaging. Optica, 6(8)92119943, Aug 2019. URI http:://wwtt optica-6 6-8-921.
Bartolomé Barceló, Eugene Fabes, and Jin K. Seo. The inverse conductivity problem with one measurement: Uniqueness for convex polyhedra. Proc. Amer. Math. Soc., 122(():183311) 1994.
Hanene Ben Yedder, Aicha BenTaich, Majid Shokoufi, Amir Zahiremami, Farid Golnaraghi, and Ghassan Hamarnch. Deep learning based image reconstruction for diffuse optical tomography. Florian Knoll, Andreas Maier, and Daniel Rueckert (cds.), Machine Learning for Medical Image Reconstruction, pp- 112-119, Cham, 2018. Springer International Publishing. ISBN 978-3-030-00129-22
Jose Antonio I Lara Benitez, Takashi Furuya, Florian Faucher, Xavier Tricoche, and Maarten V Hoop. Fine- tuning ncural-operator architectures for training and generalization. arXiv preprint arXiv:2301. 11509, 2023.
Kaushik Bhattacharya, Bamdad Hosseini, Nikola B. Kovachki, and Andrew M. Stuart. Model reduction and neural networks for parametric pdes. The SMAI journal of computational mathematics, 7, 2021.
Oussama Boussif, Yoshua Bengio, Loubna Benabbou, and Dan Assouline. MAgnet: Mesh agnostic neural PDE solver. In Alice H. Oh, Alekh Agarwal. Danielle Belgrave, and Kyunghyun Cho (eds.). Advances in Neural Information Processing Systems, 2022. URL https: / hwtp//ppprreee.eeeeeeeeee net wwweee//oooridieeeeeeeeeeerr /tr/forrmmnd-baaoohiiaaaaaa
John P Boyd. Chebyshev and Fourier spectral methods. Courier Corporation, 2001.
Johannes Brandstetter, Rianne van den Berg, Max Welling, and Jayesh K Gupta. Clifford neural layers for PDE modeling. In International Conference Learning Representations, 2023. URL https://openree
het/forru?id-okwwau
Martin Brühl. Explicit characterization of inclusions in electrical impedance tomography. SIAM Journal on Mathematical Analysis, 1327:1122,1341, 2001. URL http:///....rooo..o.o////////aa.
11

---

Published as a conference paper at ICLR 2023
Tan Bui-Thanh, Qin Li, and Leonardo Zepeda-Núñtez. Bridging and improving theoretical and computational electrical impedance tomography via data completion. SIAM Journal on Scientific Computing, 4463):6668 B693, 2022.
Alberto P. Calderón. On an inverse boundary value problem. Comput. Appl. Math, 25(2-3), 2006.
Shuhao Cao. Choose a Transformer: Fourier or Galerkin. In Advances in Neural Information Processing Systems (NeurIPS 2021), volume 34, pp. 24924-249444 2021. URL http:s////peeneeeeeeen///rrrrrre
ssohLcmn4-1z
Tony F. Chan and Che-Cheng Tai. Identification discontinuous coefficients elliptic problems using total variation regularization. SIAM Sei. Comput, 25(3):881-904, 2003.
Dongdong Chen, Julián Tachella, and Mike E Davies. Equivariant imaging: Learning beyond the range space.
In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.
2021.
Junqing Chen, Ying Liang, and Jun Zou. Mathematical and numerical study of three-dimensional inverse eddy current problem. SIAM J. Appl. Math., 1677)11467- 2020.
Margaret Cheney. The linear sampling method and the MUSIC algorithm. Inverse Probl., 17(5):591-555 jul 2001. URL https://doi.org: org/10, 1888/0266- /611/17/4/3333
Kuo-Sheng Cheng, David Isaacson, JC Newell, and David G Gisser. Electrode models for electric current computed tomography. JEEE. Trans. Biomed. Eng., 36(9):118 999-994, 1989.
Krzysztof Marcin Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser, David Benjamin Belanger, Lucy J Colwell, and Adrian Weller. Rethinking attention with Performers. In International Conference on Learning Representations (ICLR), 2021. URL https://openrevieee
Yat Tin Chow, Kazufumi Ito, and Jun Zou. A direct sampling method for electrical impedance tomography. Inverse Probl., 30(9)3095003, 2014.
Yat Tin Chow, Kazufumi Ito, Keji Liu, and Jun Zou, Direct sampling method for diffusive optical tomography. SIAM J. Sci. Comput., (7()): 1658-A1684, 2015.
Yat Tin Chow, Fuqun Han, and Jun Zou. direct sampling method for simultancously recovering inhomogencous inclusions different nature. SIAM Journal on Scientific Computing, 43(3):A2161-A218,, 2021.
Thomas M Cover. Elements of information theory. John Wiley & Sons, 1999.
Adam Coxson, Ivo Mihov, Ziwei Wang, Vasil Avramov, Frederik Brooke Barnes, Sergey Slizovskiy, Ciaran Mullan, Ivan Timokhin, David Sanderson, Andrey Kretinin, et al. Machine learning enhanced electrical impedance tomography for 2d materials. Inverse Problems, 3888000500 2022.
P. Culver, R. Choe, M. Holboke, L. Zubkov, T. Durduran, A. Slemp, V. Ntziachristos, Chance, and A. G. Yodh. Three-dimmnnional diffuse optical tomography in the parallel plane transmission geometry: Evaluation of hybrid frequency domain/continuous wave clinical system for breast imaging. Medical Physics, 30(2): 235-247, 2003. URL /nttts::/aaappe anlinelibbare wiley. www/doo/mbb//oooooooo/oooo//a aaa/aa/s//// /tt///...////////......../111//1 15319.,
Maarten de Hoop, Daniel Zhengyu Huang, Elizabeth Qian, and Andrew M Stuart. The cost-accuracy trade-off' operator learning with neural networks. Journal of Machine Learning, 1(3):299-341, June 2022a. URL.
https::/doi.. http://ddoooooo/////// 10.g///..oo 4288/226///0/ jml.
4208/jml.2205510 220589.
Maarten V de Hoop, Matti Lassas, and Christopher A Wong. Deep learning architectures for nonlinear operator functions and nonlinear inverse problems. Mathematical Statistics and Learning, 4(1):1 86, February 2022b.
URL https://doi. org/ 4172/ms1/28.
David C Dobson and Fadil Santosa. An image-enhancement technique for electrical impedance tomography. Inverse Probl., 10(2):317, 1994.
Yuwei Fan and Lexing Ying. Solving electrical impedance tomography with deep learning. J. Comput. Phys., 104:109119, 2020.
Yuwei Fan, Cindy Orozco Bohorquez, and Lexing Ying. BCR-Net: A neural network based on the nonstandard wavelet form. J. Comput. Phys., 384:1-15, 2019. ISSN 0021-9991. URL
12

---

Published as a conference paper at ICLR 2023
Jinchao Feng. Qiuwan Sun, Zhe Li. Zhonghua Sun, and Kebin Jia. Back-propagee neural network-based reconstruction algorithm for diffuse optical tomography. Journal of Biomedical Optics, 24(5):1 12, 2018. URL httpps://ooiiooooo org/ wwtp//...///........///././ JBO. 24. 5. 851487.
Xosé Fernández-Fuentts, David Mera, Andrés Gómez, and Ignacio Vidal-Franco. Towards fast and accurate eit inverse problem solver; machine learning approach. Electronics, 7(12). 2018, ISSN 2079-9292. URL
wttp://99//79999////770 0292/7/12/422 422...
Antonio H de O Fonseca, Emanuele 7appala, Josue Ortega Caro, and David Dijk. Continuous spatiotemporal transformers, arXiv preprint 2rXiv:2301.1333. 2023.
Yunhe Gao, Mu Zhou, and Dimitris N Mctaxas. Utnet: hybrid transformer architecture for medical image segmentation. International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 61-71. Springer. 2021.
Omar Ghattas and Karen Willeox. Leaming physics-based models from data: perspectives from inverse problems and model reduction. Acta Numerica, 30:445-554, 2021.
David Gilbarg and Neil S. Trudinger. Elliptic partial differential equations of second order, volume 224. Springer, New York, 2 edition, 2001.
H Griffiths, WR Stewart, and W Gough. Magnetic induction tomography. a measuring system for biological tissues. Ann N Y Acad Sci., 20(873). 1999.
Danping Gu, Dong Liu, Danny Smyl. Jiansong Deng, and Jiangfeng Du. Supershape recovery from electrical impedance tomography data. IEEE Transactions Instrumentation and Measurement, 70:1-11, 2021.
John Guibas, Morteza Mardani, Zongyi Li, Andrew Tao, Anima Anandkumar, and Bryan Catanzaro. Adaptive
Fourier neural operators: Efficient token mixers for Transformers. In International Conference on Learning Representations, 2022. URL httpp:://operrevieeeeettt
AM,
Ruchi Guo and Jiahua Jiang. Construct deep neural networks hased on direct sampling methods for solving electrical impedance tomography. SIAM J. Sei. Comput., 43(3):B678-B777 2020.
Ruchi Guo and Tao Lin. A group of immersed of Numerical Analysis, 3982):482- 111-2-1111 511, 2019.
spaces for elliptic interface problems. IMA Journal
Ruchi Guo, Tao Lin, and Yanping Lin. fixed mesh method with immersed finite elements for solving interface inverse problems. J. Sci. Comput., 79(1): (0()):448-755 2019.
Ruchi Guo, Jiahua Jiang, and Yi Li. Learn index operator by CNN for solving diffusive optical tomography: a deep direct sampling method. J. Sci. Comput., 2021.
Gaurav Gupta, Xiongye Xiao, and Paul Bogdan. Multiwavelet based operator learning for differential equations.
In Thirty-Fii Fifth Conference on Neural Information Processing Systems (NeurIPS 2021). 2021. URL https: // /wwppneeeiew...o net wwwp://ue/fotttteeeeee Lid.:iiiaa-dGGGL.....
Gaurav Gupta, Xiongye Xiao, Radu Balan, and Paul Bogdan. Non-linear operator approximations for initial value problems. International Conference on Learning Representations, 2022. URL htpp:ttt////opeereeiie.. net/ set.forummid-ddTTTgkkee
Sarah J Hamilton, Asko Hämninen, Andreas Hauptmann, and Ville Kolehmainen. Beltrami-net: domain- independent deep bar learning for absolute imaging with electrical impedance tomography (a-cit), Physiol Meas., 4077):07 2019.
Sarah Jane Hamilton and Andreas Hauptmann. Deep d-bar: Real-time electrical impedance tomography imaging with deep neural networks. IEEE Trans Med Imaging, 37(10)23377-377, 2018.
Xing Han, Tongzheng Ren, Tan Minh Nguyen, Khai Nguyen, Joydeep Ghosh, and Nhat Ho. Robustify transformers with robust kernel density estimation. arXiv preprint arX/4:2210.05794, 2022.
Martin Hanke and Martin Brühl. Recent progress in electrical impedance tomography. 19(6) S90, nov 2003. URL https:://oi..oo orgooo..rr/...o. 18811/266880000 (111/666611/1////1 5611/19/6/555
Isaac Harris and Andreas Kleefeld. Analysis of new direct sampling indicators for far-field measurements. Inverse Problems, (5550055 apr 2019. URL httpps://oii org/ 1081/136111111 6420/abe8be.
Isaac Harris, Dinh- Liem Nguyen, and Thi-Phon Nguyen. Direct sampling methods for isotropic and anisotropic scatterers with point source measurements. Inverse Problems and Imaging, 16(5): 137-116,, 2022. ISSN 1930-8337.
13

---

Published as a conference paper at ICLR 2023
Ali Hatamizadeh, Yucheng Tang. Vishwesh Nath, Dong Yang, Andriy Myronenko, Bennetl Landman, Holger Roth, and Daguang Xu. Unetr: Transformers for 3d medical image segmentation. 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 1748 1758, Los Alamitos, CA, USA, jan 2022. IEEE Computer Society. URL https://oooiccccoptttsoooii www.cooputeeoooceety......oo org/ 20. 119g/AAV524...8.. 2622. 80181.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778. 2016.
Sheng He, P. Ellen Grant, and Yangming Ou. Global-local transformer for brain age estimation. IEEE
Transactions on Medical Imaging, 41(1):213 224, 2022.
David S Holder. Electrical impedance tomography: methods, history and applications. CRC Press, 2004.
Radek Hrabuska, Michal Prauzek, Marketa Venclikova, and Jaromir Konecny. Image reconstruction for electrical impedance tomography: Experimental comparison of radial basis neural network and gauss newton method. PapCrssepennniiinee 51(6):438 443, 2018. ISSN 2405-8963, URL https://www. wwt.ws.siieeeee sciencediree 15th IFAC Conference on Programmable Devices and
com/sciemn
Embedded Systems PDeS 2018.
Peiyan Hu, Qi Meng, Bingguang Chen, Shiqi Gong, Yue Wang. Wei Chen, Rongchan Zhu, Zhi-Ming Ma, and Tie-Yan Liu. Neural operator with regularity structure for modeling dynamics driven by spdes. arXiv preprint arXiv:2204.00 0,55:22204,,6255, 2022.
Michael Hutchinson, Charline Le Lan, Sheheryar Zaidi, Emilien Dupont, Yee Whye Teh, and Hyunjik Kim. Lietransformer: Equivariant self-attention for lie groups In International Conference on Machine Learning. pp. 4533-4543 PMLR, 2021.
Masaru Ikehata. Reconstruction of the support function for inclusion from boundary measurements. Journal of Inverse and Ill-posed Problems, 8(4):367-378, 2000. URL http::////.i...rg..../////////// 2008.8 8.
367.
Masaru Ikehata. Probe method and carleman function. Inverse Problems, 1375:1887-1194,, aug 2007. URL https://doi.. org/10. 10. 1088
Masaru Ikehata and Samuli Siltanen. Numerical method for finding the convex hull of inclusion in conductivity from boundary measurements. Inverse Problems, 16(4): 1043- aug 2000. URL https:::tt:iiorrggeeeeo http///.......///..//e//. 1888/e266 20/21/16/43311
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448 456. PMLR, 2015.
Haris Iqbal. Harisigbaal&//loonnerabbb v1.0.0, 2018. URL
Victor Isakov and Jeffrey Powell. On the inverse conductivity problem with one measurement. Inverse Probl., 6: 311, 1990.
Kazufumi Ito, Bangti Jin, and Jun Zou, A direct sampling method for inverse electromagnetic medium scattering. Inverse Probl., 095011,, sep 2013. URL http::://od..org//o..068// //t////226666111//////9//333
Xia Ji, Xiaodong Liu, and Bo Zhang. Phaseless inverse source scatteriigg problem: Phase retrieval, uniqueness and direct sampling methods. Journal of Computational Physics: X, 1:100003, 2019. ISSN 2590-0552. URL
https://wwee www. http://wwwwciecccciiecee wwweenceeeeeet............. cttm:///aaaiccaarraaeeeee///eeee
Bangti Jin and Yifeng Xu. Adaptive reconstruction for electrical impedance tomography with a piecewise constant conductivity. Inverse Problems, (6( 1)0014033 2019.
Bangti Jin, Xiyao Li, and Xiliang Lu. Imaging conductivity from current density magnitude using neural networks. Inverse Problems, 38(7):075003, 2022a.
Kyong Hwan Jin, Michael T McCann, Emmanuel Froustey, and Michael Unser. Deep convolutional neural network for inverse problems in imaging, IEEE Transactions on Image Processing, (509)4445--45222 2017.
Pengzhan Jin, Shuai Meng, and Lu Lu. Mionet: Learning multiple-npput operators via tensor product. arXiv preprint arXiv:2202.00 220.:2200000133... 06137. 2022b.
Hachem Kadri, Emmanuel Duflos, Philippe Preux, Stéphane Canu, Alain Rakotomamonjy, and Julien Audiffren. Operator-valued kemelss for learning from functional response data. 2016.
14

---

Published as a conference paper at ICLR 2023
Eunhee Kang, Junhong Min, and Jong Chul Ye. A deep convolutional neural network using directional wavelets for low-dose x-ray et reconstruction. Med Phys, 44(10))360-6375, Oct 2017. ISSN 2473-4209 (Electronic); 0094-2405 (Linking).
Hyeonbae Kang and Keun Seo. Note on uniqueness and stability inverse conductivity problems with one measurement. Korean Math. Soc, pp. 781-792, 2001.
Sangwoo Kang. Mare Lambert, Won- WwanK Park. Direct sampling method imaging small diclectric inhomageneities: analysis and improvement. Inverse Probl., 34(9)0955005, jul 2018. URL https://doi. org/18 1888///.188/1100 18611136111111111111111111111111 642e/aacfid.
Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are RNNs: Fast autoregressive transformers with linear attention. In International Conference on Machine Learning, pp. 5156-5165, PMLR, 2020.
Yuchaw Khoo, Jianfeng Lu, and Lexing Ying. Solving parametric pde problems with artificial ncural networks. European Journal of Applied Mathematics, 32())421-435, 2021.
Andreas Kirsch and Natalia Grinberg. The factorization method for inverse problems, volume 36. OUP Oxford, 2007.
Georgios Kissas, Jacob Seidman, Leonardo Ferreira Guilhoto, Victor M Preciado, George Pappas, and Paris Perdikaris. Learning operators with coupled attention. arXiv preprint 2tti::2001010322 2022.
Grzegorz Klosowski and Tomasz Rymarczyk. Using neural networks and deep learning algorithms in electrical impedance tomography. Informatyka, Automatyka, Pomiary w Gospodarce i Ochronie Srodowiska, 7(3), 2017.
Kim Knudsen, Matti Lassas, Jennifer L. Mueller, and Samuli Siltanen. D-bar method for electrical impedance tomography with discontinuous conductivities. SIAM Journal Applied Mathematics, 67(3):893 913, 2007. URL htwpt:httoso...oooo org/ 10. 1111 /66655633..
Kim Knudsen, Matti Lassas, Jennifer L. Mueller, and Samuli Siltanen. Regularized d-bar method for the inverse conductivity problem. Inverse Probl. Imaging, 3(4):599 624, 2009.
Robert Kohn and Michael Vogelius. Determining conductivity by boundary measurements. Communications Pure and Applied Mathematics, 37(3):289-298. 1984.
Nikola Kovachki, Samuel Lanthaler, and Siddhartha Mishra. On universal approximation and error bounds for Fourier neural operators. The Journal of Machine Learning Research, 22(1):13337-13312 2021.
Rainer Kress. Linear Integral Equations. Springer New York, 1999. URL https://doi.org//// 1007/
118-1-4612-0559-11
Jahanzaib Latif, Chuanghai Xiao, Azhar Imran, and Shanshan Tu. Medical imaging using machine learning and deep learning algorithms: A review. In 2019 2nd International Conference on Computing, Mathematics and Engineering Technologies (iCoMET), pp. 1-5, 2019.
Thu Le, Dinh-Liem Nguyen, Vu Nguyen, and Trung Truong. Sampling type method combined with deep learning inverse scattering with one incident wave, arXiv preprint arXiv:2207.1000 2022.
Okkyun Lee, Jongmin Kim, Yoram Bresler, and Jong Chul Ye. Diffuse optical tomography using generalized music algorithm. 2011 IEEE International Symposiam on Biomedical Imaging: From Nano Macro, pp- 1142-1145, 2011.
Seungjun Lee. Mesh indeeennttn operator learning for PDEs using set representations, 2023. URL https: doBF26Hf.
Zijie Li, Kazem Meidani, Amir Barati Farimani. Transformer for partial differential equations' operator learning. arXiv preprint Xiv::2205 13671, 2022a.
Zongyi Li, Nikola Borislavov Kovachki, Kamyar Azizzadenesheli, Burigede liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. In International Conference on Learning Representations, 2021a. URL httpp:://opereeiiiiiie net /fotrrror c8P9NQVtmnO.
Zongyi Li, Hongkai Zheng, Nikola Kovachki, David Jin, Haoxuan Chen, Burigede Liu, Kamyar Azizzadenesheli, and Anima Anandkumar. Physics-informed neural operator for learning partial differential equations. arXiv preprint arXiv:2/11.0377 2021b.
15

---

Published as a conference paper at ICLR 2023
Zongyi Li. Miguel Liu-Schiaffini, Nikola Borislavov Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Learning chaotic dynamics dissipative systems. In Alice H. Oh, Alckh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances Neural Information Processing Systems, 2022b. URL https:://pperreviww www.nnt/forummid::cooo fo
William RB Lionheart. Eit reconstruction algorithms: pitfalls, challenges and recent developments. Physiological measurement, 25(1): 125, 2004.
Xinliang Liu, Bo Xu, and Lei Zhang. Ht-net: Hierarchical transformer based operator learning model for multiscale pdes. arXiv preprint vxXiv22210.1089) 2022.
Fei Lu, Qingei An, and Yue Yu. Nonparametric learning of kernels nonlocal operators. arXiv preprint arXiv:2205.11000 2022.
Lu La, Pengzhan Jin, Guofei Pang, Zhonggiang Zhang, and George Karniadakis. Learning nonlinear operators via deeponet based on the universal approximation thcorem of operators. Nature Machine Intelligence, 3(3): 218 229, 2021. URL https://doi.ors org/10. 0038/s42256-120 021 0011-00002-5
Björn Lütjens, Catherine H Crawford, Camphell D Watson, Christopher Hill, and Dava Newman. Multiscale neural operator: Learning fast and grid-independent pde solvers. arXiv preprint arXiv:2207.11417 2022.
Jose Marroquin, Sanjoy Mitter, and Tomaso Poggio. Probabilistic solution of ill- pll-posed problems in computational vision. Journal of the american statistical association, 82(397):76-899 1987.
Thierry Martin and Jérôme Idier. A based nonlinear map estimator in electrical impedance tomography. In Proceedings of ICIP, volume 2. pp. 684-688 687. IEEE, 1997.
James Mercer. Xvi. functions of positive and negative type, and their connection the theory of integral equations. Philosophical transactions of the royal society of London. Series A, containing papers of a mathematical or physical character, (09(441-458):444 446, 1909.
Marketa Michalikova, Rawia Abed, Michal Prauzek, and Jiri Koziorek. Image reconstruction in electrical impedance tomography using neural network. In 2014 Cairo International Biomedical Engineering Confer- ence (CIBEC), pp. 42. IEEE, 2014.
Ha Quang Minh, Partha Niyogi, and Yuan Yao. Mercer's theorem, feature maps, and smoothing. In Learning Theory: 19th Annual Conference on Learning Theory, COLT 2006, Pittsburgh, PA, USA, June 22-25, 2006. Proceedings 19, pp. 154-168. Springer, 2006.
Jeffrey S Morris. Functional regression. Annual Review of Statistics and Its Application, 2:321-359, 2015.
Adrian Nachman, Alexandru Tamasan, and Alexandre Timonov. Conductivity imaging with single measurement of boundary and interior data. Inverse Problems, 23(6):2551, 2007.
Adrian I. Nachman. Global uniqueness for two-dimensional inverse boundary value problem. Annals of Mathematics, 143())771-6,, 1996.
Nicholas H Nelsen and Andrew M Stuart. The random feature model for input output maps between banach spaces. SIAM Journal Scientific Computing, 43(5):A3212-A3223, 2021.
Tan M. Nguyen, Vai Suliafu, Stanley J. Osher, Long Chen, and Bao Wang. FMMformer: Efficient and Flexible Transformer via Decomposed Near-field and Far-field Attention. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
Tan Minh Nguyen, Minh Pham, Tam Minh Nguyen, Khai Nguyen, Stanley Osher, and Nhat Ho. Fourierformer: Transformer meets generalized fourier integral theorem. Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https: //openreviews www.tnruunteeeee ie-rumndd- _i__
Thanh C. Nguyen, Vy Bui, and George Nehmetallah, Computational optical tomography using 3-D deep convolutional neural networks. Optical Engineering, 57(4): - 11, 2018. URL https://doi.org/ 1. OE. 57 043111
Toan Q Nguyen and Julian Salazar. Transformers without tears: Improving the normalization of self-attention. In Proceedings of the 16th International Conference Spoken Language Tronslation, Hong Kong, November 2-3 2019, Association for Computational Linguisties. URL https: http::::://allaaahloogo.ooo whwolnyyooooooooooooooooooooooot www.//19...../... iwslt-i..1.
Yong Zheng Ong, Zuowei Shen, and Haizhao Yang. Integral autoencoder network for discretization-inaarii learning- Journal of Machine Learning Research, 23(286): 2022.
16

---

Published as a conference paper at ICLR 2023
Samira Pakravan, Pouria A Mistani, Miguel A Aragon-Calver and Frederic Gibou, Solving inverse-pde problems with physics-aware neural networks. Journal of Computational Physics, 440:1 10414, 2021.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf. Edward Yang. Zachary De- Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32 (NeurIPS 2019), pp. 8024-8035. 2019. URL http:://ppppppe neurips. pc/ppppr/9915 pytorch- an- www--iiiieeeeeeeeeeee serriiieessylllee style- www.gg---pffrraneeeeee learning wawtannnnnn-bbrryyoorooaaaaaaaar .dd.
Hao Peng. Nikolaos Pappas, Dani Yogatama, Roy Schwartz. Noah Smith, and Lingpeng Kong. Random feature attention. International Conference on Learning Representations, 2021. URL httpps://openreview.ettt forum?id=Qt TKddVrFFF..
Shehan Perera, Srikar Adhikari, and Alper Yilmaz. Pocformer: A lightweight transformer architecture for dctection of covid- 19 using point of care ultrasound. In ICIP, 2021.
Olivier Petit, Nicolas Thome, Clement Rambour, Loic Themyr, Toby Collins, and Luc Soler. U-net transformer: Self and cross attention for medical image segmentation. In Machine Learning in Medical Imaging: 12th International Workshop, MLMI 2021, Held in Conjunction with MICCAI 2021, Strusbourg, France, September 27, 2021, Proceedings, pp. 267-276, Berlin, Heidelberg, 2021. Springer-Vertee ISBN 978-3-030-88588-6 URL wttpt:tttt/:oooooo org/ 0000/97--3---000 0-0- 87589-35288
M. K. Pidcock, M. Kuzuoglu, and K. Leblebicioglu. Analytic and semi-analytic solutions in electrical impedance tomography. i. two-dimensional problems. Physiological Measurement, 16(2):77-99) may 1995. URL
https://doi. org/ 1088% 3334%2 F2622
Shangjie Ren, Kai Sun, Chao Tan, and Feng Dong. A two-stage deep learning method for robust shape recon- struction with electrical impedance tomography. IEEE Transactions on Instrumentation and Measurement, 69(7):4887 4897478777477, 2020.
Luca Rondi and Fadil Santosa. Enhanced electrical impedance tomography via the mumford-shah functional. ESAMM: COCV, 6:517-538, 2001. URL https://doi. org/ 10. 1051/c0cv:2001121.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention, pp. 234-241. Springer, 2015.
Tim De Ryck and Siddhartha Mishra. Generie bounds on the approximation error for physics-informed (and) operator learning. In Alice Oh, Alckh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL http::::///ppervvieee http////wwweiieeeee/orrrrrr
net/forummiiii id Beeddd-beeyyyyyyyy TR9.
Jacob H Seidman, Georgios Kissas, Paris Perdikaris, and George J. Pappas. NOMAD: Nonlinear manifold decoders for operator learning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (cds.), Advances in Neural Information Processing Systems, 2022. URL //opprrvve///eeeeeeeeeeeeeeeeeee wwwpenrwwwwwwwewweeeeeeeeeeeeeee net/forum? id=50wv- sZvMl.
Jin Keun Seo, Kang Cheol Kim, Ariungerel Jargal, Kyounghun Lee, and Bastian Harrach. A learning- based method for solving ill-posed nonlinear inverse problems: simulation study of lung eit. SIAM journal on Imaging Sciences, 12(3):1275 2019.
Leslie N Smith and Nicholay Topin. Super-convergence:: Very fast training of neural networks using large learning rates. Artificial Intelligence and Machine Learning for Multi-Domate Operations Applications, volume 1 11006, pp. 1100612. International Society for Optics and Photonics, 2019.
Diping Song, Bin Fu, Fei Li, Jian Xiong, Junjun He, Xiulan Zhang, and Yu Qiao. Deep relation transformer for diagnosing glaucoma with optical coherence tomography and visual field function. IEEE Transactions on Medical Imaging, 4009):2392-2409 2021.
John Sylvester and Gunther Uhlmann. A global uniqueness theorem for an inverse boundary value problem. Annals of Mathematics, 125(1):153 -699, 1987.
Chao Tan, Shuhua Lv, Feng Dong, and Masahiro Takei. Image reconstruction based on convolutional neural network for electrical resistance tomography. IEEE Senxory Journal, 19(1):19-2004 2018
Leonardo Tanzi, Andrea Audisio, Giansalvo Cirrincione, Alessandro Aprato, and Enrico Vezzetti. Vision trans- former for femur fracture classification. Injury, 2022. ISSN 0000--3833 URL httpps://ww.sscinneeiiiii
17

---

Published as a conference paper at ICLR 2023
T. Tarvainen, M. Vauhkomen, and S.R. Arridge. Gauss-newton reconstruction method for optical tomography using the finite element solution the radiative transfer equation. Journal of Quantitariie Spectroscopy and Radiative Transfer, 109(17):2767 2778, 2008, ISSN 0022-4073. URL https///wwwwssccnncccrettccccc
J Nasehi Tehrani, Alistair McEwan, Craig Jin, and A Van Schaik. LI regularization method electrical impedance tomography by using the 11-curve (pareto frontier curve). Applied Mathematical Modelling, 36 (3):1095-1105, 2012. ISSN 0307-904X. URL https://www.scieneeeeeeee... S0307904X11894537.
Yao-Hung Hubert Tsai, Shaojie Bai, Makoto Yamada, Louis-Philippe Morency, and Ruslan Salakhutdinov. Transformer dissection: An unified understanding for transformer's attention via the lens of kernel. In Proceedings the 2019 Conference on Empirical Methods Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLL pp. 4344-4353, Hong Kong. China, November 2019. URL https:/ www. ac wwwb....b..o http///ntoloogyyaaaa Dt////19991100 1443.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems (NIPS 20/7), volume 30, 2017.
Plivi J Vauhkomen, Marko Vauhkonen, Tuomo Savolainen, and Jari Kaipio. Three-dimensional electrical impedance tomography based on the complete electrode model. IEEE Trans. Biomedical Engrg. 46(9): 1150-11600 1999.
Hongyi Wang. Shiao Xie, Lanfen Lin, Yutaro Iwamoto, Xian-Hua Han, Yen- Wei Chen, and Ruofeng Tong. Mixed transformer u-net for medical image segmentation. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 2390-2394,, IEEE, 2022.
Huan Wang, Kai Liu, Yang Wu, Song Wang, Zheng Zhang, Fang Li, and Jiafeng Yao. Image reconstruction for electrical impedance tomography using radial basis function neural network based on hybrid particle swarm optimization algorithm. IEEE Sensory Journal, 21(2): 1924-194.. 202
Qi Wang, Huaxiang Wang. Ronghua Zhang. Jinhai Wang. Yu Zheng. Ziqiang Cui, and Chengyi Yang. Image reconstruction based on 11 regularization and projection methods for electrical impedance tomography. Review of Scientific Instruments, 83(10))004707, 2012. URL https://doi. http:://doooo/oood/o/ org/1e. arg/19.1633//111.1.. 1863/1.4760255... .4760253.
Sifan Wang, Hanwen Wang, and Paris Perdikaris. Learning the solution operator of parametric partial differential equations with ahysics-informed deeponets. Science advances, 7(40):eabi8605, 2021b.
Gege Wen, Zongyi Li, Kamyar Azizzadenesheli, Anima Anandkumar. and Sally M Benson. U-fno--nn an enhanced fourier neural operator- based deep-learning model for multiphase flow. Advances in Water Resources, 163: 104180, 2022.
Ruihin Xiong, Yunchang Yang. Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tieyan Liu. On layer normalization in the transformer architecture. In International Conference on Muchine Learning, pp. 10524-10533 PMLR, 2020.
Jong Chul Ye, Yoseob Han, and Eunju Cha. Deep convolutional framelets: general deep learning framework for inverse problems. SIAM Journal on Imaging Sciences, 11(2).911-0448 2018
Rui Zhang, Qi Meng, Rongchan Zhu, Yue Wang, Wenlei Shi, Shihua Zhang. Zhi-Ming Ma, and Tie-Yan Liu. Monte carlo neural operator for learning pdes probabilistic representation. arXiv preprint arXiv:2302.05104 2023.
Jiawei Zhao, Robert Joseph George. Yifei Zhang, Zongyi Li, and Anima Anandkumar. Incremental fourier neural operator. arXiv preprint urXiv:2211.1511 2022.
Hong- Yu Zhou, Jiansen Guo, Yinghao Zhang, Lequan Yu, Liansheng Wang, and Yizhou Yu. nnformer: Volumetric medical image segmentation via 3d transformer. arXiv:2109.05201v6 2021.
Bo Zhu, Jeremiah Z Liu, Stephen F Cauley, Bruce R Rosen, and Matthew S Rosen. Image reconstruction by domain-transform manifold learning. Nature, 555(7697) Mar 2018. ISSN 1476-4687 (Electronic); 0028-0836 (Linking).
Jianqing Zhu, Juneai He, and Qiumei Huang. An euhanced v-cycle mgnet model for operator learning in numerical partial differential equations, arXiv preprint arXiv:2302.009:: 2023a.
Jianqing 7hu, Juncai He, Lian Zhang, and Jinchao Xu. Fv-mgnet: Fully connected v-cyele mgnet for interpretable time series forecasting, arXiv preprint 2xXiv:2302.00962, 2023b.
18

---

Published as a conference paper at ICLR 2023
A
TABLE OF NOTATIONS
Table Notations used in an approximate chronological order and their meaning in this work.
Notation
Meaning
Ω
an underlying spacial domain in R2
D
subdomain in S2 (not necessarily copolagically-ccomeeeed) D's and f2's boundary, 1-dimensional manifolds
aD, ăn
u
the gradient vector of a function, Vu(x) u(x), O22 u(x))
M
the L,"nnorm on region w
1l
the -orm on whole domain ? the delta function such that Sn2 (1) f(x) the unit outer normal vector on the boundary ON.
n
u
normal derivative of u, measures the rate of change along the direction of NtD map from Neumann data g Vu n (how fast the solution changes toward the outward normal direction) to Dirichlet data f w/ ala (the solution's value along the tangential direction) the Sobolev space of functions
Aa
( $2 0
H ( <0
the bounded linear functional defined on H*2) all u E HS2) such that u's integral on Ω vanishes the seminorm defined for functions Y
( 1
B BACKGROUND OF EIT
For EIT, an immediate question is whether J=1 and Fi.1 in (4) and (5) are well-defined, namely whether can be uniquely determined. In fact, for the case of full measurements (L oo), the uniqueness for F-1 has been well established, (Brühl, 2001; Hanke & Brühl, 2003; Astala & Päivärinta, 2006; Nachman, 1996; Kohn & Vogelius, 1984; Sylvester & Uhlmann, 1987). It is worthwhile to point out that, this case, a is not necessarily a piecewise constant function. In (1), present simplified case for purposes of illustrating as well as benchmarking. In general, with the full spectral information of the NtD map, a can be uniquely determined as a general positive function.
infinitely many eigenpairs are known, then the operator itself can be precisely characterized using infinitely many feature channels by Reproducing Kernel Hilbert Space (RKHS) theory, e.g., Mercer (1909); Aronszajn (1950); Minh et al. (2006); Morris (2015); Kadri et al. (2016); Lu et al. (2022). In the context of EIT, this known as the "full measurement". more challenging and practical problem is to recover a from only finitely many boundary data pairs. A common practice for the theoretical study reconstruction using finite measurements is the assumption a being a piecewise constant function. The task is usually set to recover the shape and location of the inclusion D. Otherwise, the problem is too ill-posed. With finite measurements, the uniqueness of the inclusion remains long-standing theoretical open problem, and it can be only established for several special classes of the inclusion shape, such as the convex cylinders in Isakov & Powell (1990) or convex polyhedrons in Barceló et al. (1994). We refer readers some counter-examples in Kang & Seo (2001) where a two- or three-dimensional ball may not be identified uniquely by one single measurement the values of ao and i are unknown.
Furthermore, here we provide one example to illustrate the difficulty in the reconstruction procedure (Pidcock et al., 1995). Let Q be unit circle, let D be circle with the radius p < and define
| 2 p. if [||l<<<>>
(xx)
(24)
with 01 < 1 being an arbitrary constant. In this case, the eigenpairs A, can be explicitly calculated
1 11- =
A 71+ 2
cos(10) or 2 2
sin() [ = 1,2,...
(25)
19

---

Published as a conference paper at ICLR 2023
with j === =1 (1
(1- + vo),, which exactly the Fourier modes in unit circle. In this the matrix
case, if the set of basis (=r is just chosen as {cos(?), sin(f),
representation A, in (3) can be written as infinite diagonal matrix
(1 ,² p2)(1(1 + p4))
(1 - p²v)((1 - x²1))
0
Aa
(1- p1-p²m) 11111 + + p²m)
0
(26)
0
0
Thanks to this special geometry, eigenvalues (25) can elearly determine and 01 as follows:
- (1)+ and (1 (1- (1 (1+
(27)
(1+
However, in practice, Ar does not have such simple structure. Approximating A, itself requires a large number of data pairs that are not available in the considered case. Besides, an accurate approximation of the eigenvalues of A is also very expensive. Furthermore, for complex inclusion shapes, two eigenvalues are not sufficient to exactly recover the shape and conductivity values.
C EXPERIMENT SET-UP
C.I DATA GENERATION AND TRAINING
In the numerical examples, the data generation mainly follows standard practice in theoretical prototyping for solving EIT problems, see e.g.,Chow al. (2014); Michalikova al. (2014); Hamilton & Hauptmann (2018); Guo & Jiang (2020); Fan & Ying (2020). For examples, please refer to Figure 2. The computational domain is set to be
(-1, 1) and the two media with the different conductivities are with 01 = 10 (inclusion) and = 1 (background). The inclusions are four random ellipses. The lengths of the semi-major axis and semi-minor axis of these ellipses sampled from (( 1, 0.2) and (0.2, 0.4), respectively. The rotation angles are sampled from (0, 2T)). There are 10800 samples the training set, from which 20% are reserved as validation. There are 2000 in the testing set for evaluation.
Figure 2: Randomly selected samples of elliptic inclusion represent the coefficient a (left 1-4). A Cartesian mesh 2n with grid point {2,} (right). In computation, discretization of I consists of values taken at the mesh points of f2, inside D and 0 at others.
The noise (xx below (10) is assumed to be
{xx) = (f(x) (z))(=)))((
(28)
where 7 specifies the relative strength of noise, and G(x) normal Gaussian distribution indepen- dent with respect to As {xz) is merely pointwise imposed, the boundary data can be highly rough, even if the ground truth f()) - A-) is chosen be smooth. Nevertheless, the harmonic extension makes the noise from boundary data have minimal impact on the overall reconstruction. thanks to the smoothing property of the inverse of the Laplacian operator ( ) for example, please refer to Figure 3. In data generation, the harmonic extension feature map is approximated by finite element methods incorporating stencil modification near the inclusion interfaces (Guo & Lin, 2019; Guo et al., 2019). Similar data augmentation practices using internal data be found in Nachman al. (2007); Bal (2013); Jin et al. (2022a).
Thanks to the position-wis binary nature of ID, another choice of the loss function during training can be the binary cross entropy -(-,),, applied for function in P., to measure the distance between the ground truth and the network's prediction
C(p,, un) == - -=---e ESSM (n) + (1 - pn())) (=)))
(29)
20

---

Published as a conference paper at ICLR 2023
0.06
-0.0031
-04
-04
0002441
Figure The harmonic extensions di with zero noise on boundary (left); the harmonic extensions di with 20% Gaussian noise on boundary (middle). Their pointwise difference (right). || Φz 1//l// 0.0203.
Thanks to the Pinsker inequality (e.g., see (Cover, 1999, Section 11.6),, C(P), u,) serves as good upper bound for the square of the total variation, which can be further bounded below by the L -rror given the boundedness of the position-wise value.
The training 1cycle (Smith & Topin, 2019) learning rate strategy with warm-up phase. A mini-batch ADAM iterations are run for 50 epochs with no extra regularization, such as weight decay. The evaluated model taken from the epoch that has the best validation metric. The learning rate starts and ends with 10-3 ITmaax and reaches the maximum of maaa at the end of the 10-th epoch. The lT max 10-3 The result demonstrated is obtained from fixing the random number generator seed. Figure shows the testing results for randomly chosen sample. All models are trained on an RTX 3090 or an A4000. The codes to replicate the experiments are open-source and publicly available.
U Net
U Integral
Transformer
Train Evaluation
Troin
Evaluation
10-1
10
30
50
40
Epoch
Epoch
Figure 4: The left: the training- pixel-wise binary cross entropy convergence for the CaNeb U-Net with 31 million parameters, clear overfitting pattern is shown. The right: the thining-testing convergence for the attention- based U- J-Transformer with 11.4 million parameters.
C.2 NETWORK ARCHITECTURE
The difference in architectural hyperparameters, together with training and evaluation costs compari- son for all the models compared in the task of EIT reconstruction can be found in Table 3.
U-Integral Transformer
Overall architecture. The U-Integral-Trrnssoore architecture is a drop-in replacement of the standard CNN- based U-Net baseline model (7.7m) in Table 1. The CNN-based U-Net is used in ba-based approaches for boundary value inverse problem in Guo & Jiang (2020); Guo et al. (2021); Le et al. (2022). One of the novelties is that the input a tensor that concatenates different measurement matrices as different channels, and similar practice can be found in Brandstetter et al. (2023). Same with the baseline U-Net, the UIT has three downsampling layers as the encoder (feature extractor). The downsampling layers map
latent representations to m 2 m m/2, 2, and
21

---

Published as a conference paper at ICLR 2023
Figure 5: The harmonic extension feature map Φr (left 1-3 as different channels' inputs to the neural network) corresponding
randomly chosen sample's inclusion map (right). No visible relevance shown with the ground truth. The layered heatmap appearance is adopted in plotly contour only for aesthetics purposes, no edge-like nor layer-like features can be observed from the actual harmonic extension feature map input.
(a)
(b)
(d)
(c)
(f)
(h)
Figure 6: The neural network evaluation result for the inclusion in Figure using various models. model's input is the left most function in Figure 5 if this model uses single channel, all left three in Figure 5 if a model uses 3-channel input. (a) Ground truth inclusion in Figure 5; (b) U-Net baseline (7.7m) prediction; (c) U-Net big (31m) prediction with 3 channels; (d) Fourier Neural Operator (10.4m) prediction with channel; (e) Fourier Neural Operator big (33m) prediction with channel; (f) Adaptive Fourier Neural Operator (10.7m) prediction with channel; (g) Multiwavelet Neural Operator (9.8)) prediction with channel; (h) Hybrid UT with linear attention (10.13)) prediction with channel; UIT (11.4m) prediction with channel; (j) UIT (11.4m) prediction with
3 channels.
expand the number of channels from C' to 2C. To leverage the "basis + channel" interpretation and the basis update nature the attention mechanism in Section 3.3, the proposed attention block is first added on the coarsest grid, which has the most number of channels. UIT has three upsampling layers as the decoder (feature selector), which map m/2 x m/2 latent representations m, and shrink the number of channels from 2C C. In these upsampling layers, attention blocks are applied on each cross-layer propagation to compute the interaction between the latent representations on both coarse and fine grids (see below). Please refer Figure 8 for a high-level incoder-decoder schematic.
Double convolution block. The double convolution block is modified from that commonly in Computer Vision (CV) models, such as ResNet (He et al., 2016). We modify this block such that upon being used in an attention block, the batch normalization (Toffe & Szegedy, 2015) can be replaced by the layer normalization (Ba et al., 2016), which can be understood as a learnable approximation the Gram matrices' inverse by a diagonal matrix.
Positional embedding. At each resolution, the 2D Euclidean coordinates of an m X m regular Cartesian grid the input of channel expansion through fixed learnable linear layer and
22

---

Published as a conference paper at ICLR 2023
then added to each latent representation. This choice of positional embedding enables a bilinear interpolation between the coarse and fine grids or vice versa (see below).
Mesh-normalized attention. The scaled dot-product attention in the network is chosen be the integral kernel attention in (11) with a mesh-based normalization. Please refer to Figure for diagram in a single attention head.
Norm
1:84
2
Cuer
Attn
1AA
Coos
Corry
M
V
X
1:1 Der
x
Cant
Noe
VM
Figure 7: Detailed flow of the modified 2D attention- based encoder layer using (11). C: the number of channels in the input, N: the number of expanded channels (for the basis expansion interpretation in Theorem 2).
Interpolation. Instead of max pooling used in the standard CNN-based U-Net, opt for bilinear interpolation on the Cartesian grid to map latent representation from the fine grid to the coarse grid vice versa. Note that, in the upsampling layers, the interpolation's 's outputs directly inputted into an attention block that computes the interaction of latent representations between coarse and fine grids (see below).
Coarse-fine attention in up blocks. A modified attention in Section 3.3 with a pre-inner-product normalization replaces the convolution layer on the coarsest level. The skip connection from the encoder latent representations to the ones in the decoder are generated using architecture similar to the cross attention used in Petit al. (2021). Q and K are generated from the latent representation functions the same coarser grids. As such, the attention kernel to measure the interaction between different channels is built from the coarse grid. V is associated with a finer grid. Compared with the one in Petit et al. (2021), the modified attention in our method is inspired by the kernel integral for a PDE problem. Thus, it has (1) no softmax normalization or (2) no Hadamard product-type skip connection.
Other operator learners compared
Fourier Neural Operator (FNO) and variants. Fourier Neural Operator (FNO2d) learns con- volutional filters in the frequency domain for some pre-selected modes, efficiently capturing glohally-supported spatial interactions for these modes. The weight filter in the frequency domain multiplies with the lowest modes in the latent representations (four corners in the FFT). The Adaptive Fourier Neural Operator (AFNO2d) adds a token-mixing layer, seen in Figure 2 in Guibas et al. (2022), appending every spectral convolution layer in the baseline FNO2d model.
Multi- Wavelet Neural Operator (MWO). The MultiWavele Neural Operator (MWO) is proposed in Gupta et al. (2021), which introduces multilevel structure into the FNO architecture. MWO still follows FNO's practice on each level by pre-selecting the lowest modes.
D FROM DSM TO TRANSFORMER
This section gives more detailed presentation of how the attention-like operator derived from the DSM ansatz (15) with learnable kernels.
We begin with recalling the original indicator function from (10):
= Cí. d(xx) VΦ(Φ)
(30)
1/ (a0)
where Cf.g Illf (200))) constant, and the equations of the functions ò and 7
AΦ 0 in Q Vo (f - + on 0N, Son ds 0,
(31)
23

---

Published as a conference paper at ICLR 2023
Table 3:
The detailed comparison of the networks used in this study. For U-Net-based neural networks, the channel/width is the number of the base channels on the finest grid after the initial channel expansion.
cfoat type parameter entry counts as two parameters. GFLOPs: Giga FLOPs for I backpropagation (BP) performed for batch of samples recorded the Py PyTorch autograd profiler for BP averaging from 100 BPs. Eval: number of instances per second.
Architectures
Training/Evaluation
layers
channel/width
modes
norm
GFLOPs
eval
params
U-Net
7
64
N/A
batch
140.6
298.6
7.70m
U-Net big
9
64
N/A
batch
184.1
273.4
31,04m
FNO2d
6
48
14
FFT
196.5
235.4
10.86m
AFNO2d
6
48
14
FFT
198.4
151.0
10.88m
MWO2d
4
64
12
Legendre
1059
59.6
9.81m
Hybrid UT
7
32
N/A
batch
427.5
114.0
10.
Cross-Attn UT
7
64
N/A
layer
658.9
103.3
11,42m
UIT (ours)
7
64
N/A
laycr
658.3
104.8
11.43m
Figure 8: A simplified schematic of the Transfor-T that follows the standard U-Net. The input is tensor concatenating the discretizations of á and V The output is the approximation to the index function _ C: 3 convolution ReLU; C: layer normalization batch normalization; : bilinear interpolations from the fine grid to the coarse grid; D cross attention Ae that uses the latent representations on coarse grid compute interactions produce the latent representations on finer grid; I input and output discretized functions in certain Hilbert spaces. The TikZ source code produce this figure modified from the examples Iqbal (2018).
-An d(z) V8 in Ω n VTl on a2, Son The ds 0.
(32)
Chow et al. (2014) shows that I (x) can be written as a sum of Gaussian-lin distributions that attain maximum values for E D close to D. However, the accuracy is much limited by empirical choices of quantities in (30), for example, d(x) vø(x)/ [o))) and 3/2. In addition, the frequency of the boundary data plays an important role in reconstruction; the derivation in Chow et al. (2014, Section 5) for circular Ω. What is more, such simple formula can be derived only for single data pair.
Henceforth, these "empirical chosen quantities" are made to be learnable from data by introducing two undetermined kernels K(x,, y), Q(x, y), and self-adjoint positive definite linear operator V. the modified indicator function is written as
Sa d(x) K(x,, V Vo(y) dy
(x) = Cf.g
(33)
1
with
d(x) ((z,))
(34)
24

---

Published as a conference paper at ICLR 2023
1 (Vnz, M2) 12200))
(35)
Applying certain quadrature rule to (33) with the quadrature points 24, i.e., the grid points of a2 and weights {wyJ. we obtain an approximation to the integral:
Ja K(zz, y)O.., (yy dy Σ, 25) ((2)) of
(36)
i... qf is the vector of (z; zy)lE- For (34), we consider one component dn(xx) of d(x)). With the same rule compute the integral, (16) can be written as
(x)))) E, W Q(z (2(22,2))00, 2,)02, =: of Φ...
(37)
ie., of is the vector of w Q(2,, 2/1lg- Next, we proceed to express MM, by discretizing the variational form in (32) using linear finite element method (FEM). Applying integration by parts, the weak form of (32) at x = 2; is
/ VIz dy / -(())) Voz, (v))b())))) dy = d(z)) Vo(=),,
(38)
for any test function E H. (1).. Here, we let (3b0)) J be the collection of the finite element
basis functions, and for the fixed z; we further let ,n,, the vector approximating [O., v/) (z, }
Denote 77 as the vector approximating {7=, 3 Introduce the matrices
the finite element/fiiite difference discretization of - on S4 discrete Laplacian) coupled with the Neumann boundary condition and the zero integral normalization condition in (9).
R: the matrix that projects vector defined at interior grids to the one defined on ÖΩ.
Then, the finite element discretization of (38) yields the linear system:
Bn: Σ, da da(z)) 2 E, th Nn Φn.
(39)
where we have used (37). Then, the trace of 1)., on 02 admits the following approximation
ni == 1/2 72.000
Σ, RB- RB-BBBBB.... "n, Φ...
(40)
Now, we can discretize (35). Note that the trace of linear finite element space on in is still continuous piecewise linear space, defined as S, (aQ)) Then, the self-adjoint positive definite operator V can be parameterized by symmetric positive definite (SPD) matrix denoted by V operating on the space (892). We can approximate 172,13 as
1n
n. Vn Σ,
(41)
where VT, RT VRB 2b "nii 2 V is SPD. Define
Vn,i = (bb R q.
(42)
This can considered another learnable vector since the coefficient of q; comes from the learnable matrix V. Then, (41) reduces to
M ? Σ
(43)
Putting (36), (37) and (43) into (15), we have
1P(z.) z {Il- - Alo
Vhh Σ
2h qike Φ..
(44)
n
Now, using the notation in (21), we get the desired representation (22).
25

---

Published as a conference paper at ICLR 2023
E PROOF OF THEOREM 1
Lemma 1. Suppose the boundary data gi is the eigenfunction of Do Nog corresponding to the 1-th eigenvalue A and let or be the data data functions generated by harmonic extensions
J
-di =0 Ω
Voi -Ji- -ao9i) -Aoo on an,
ds (45)
Jan
where 1 1,2, Let d be arbitrary unit vector in R??, define function
(d Voi(z))²
OL(z)=
(46)
1=1 Xi
Then, there holds
00, a finite constant,
ix $ D
lim O2(x) L-0
(47)
if x E D.
Proof. See Theorem 4.1 in Guo & Jiang (2020), and also Brühl (2001); Hanke & Brühl (2003).
Theorem 1 (A finite-dimensional approximation of the index function), Suppose the boundary data gi is the eigenfunction of A, Ago corresponding to the 1-th eigenvalue N and let or be the data functions generated by harmonic extensions given in (9). Define the space:
S Span di 0z201: 1 = L...,
(48)
and the dictionary:
Sr. {ar 02 arctan(agv) v E St, a1 02 03 E RJ.
(49)
Then, for any c > 0, we con construct an index function IP, E gy s.t.
sup uPn [x) [Z) se
(50)
provided large enough.
Proof. Consider the function θ (x) from Lemma 1. As 0z.x)) > O, it increasing with respect to L. Then, there is constant such that p > (z(x), Vo D. Given any >> there is integer such that (x) > 4pe 2 2/72, vx g D. Define
If(x) = arctan
Tt O (x)
(51)
π
2p
Note the fundamental inequality 2 > arctan(2)
2 x z Vz > 0. Then, if E D, there holds
((x))IZ(00 = arctan
TE 2p
< [0 <e
x & D, there holds
TC (L(D))
4p <C. x200)(x)
arctan
2p
Therefore, the function in (51) fulfills (50).
26

---

Published as a conference paper at ICLR 2023
F PROOF OF THEOREM 2
In presenting Theorem 2 in Section 3.3, we use the term "multiplicative" to describe the fact that multiplied in the attention mechanism. In contrast, no such operation
two latent representations exists in, c.g., pointwise FFN or a convolution layer. Heuristically speaking, the main result in Theorem states that the output latent representations can be of a higher "frequency" than the input if the neural network architecture has "multiplicative" layers in it. The input latent representations are discretizations of certain functions, and they are combined using matrix dot product as the one used in attention. Suppose that this discretization can represent functions of such frequency with a certain approximation error, the resulting matrix/tensor can be an approximation of function with higher frequency than the existing latent representation under the same discretization. Please see Figure 9 and Figure 10 for empirical evidence of this phenomenon for the latent representations: with completely smooth input (harmonic extensions), e.g., see Figure 5, attention-based learner can generate latent representations with multiple peaks and valleys.
Theorem (Frequency-bootsse bootstrapping for multiplicative neural architectures). Consider $2 (0, which has a uniform discretization of (zi \ of size h, and v(z) sin(az) for some E Z= Let N: a N 1 be the number of channels in the attention layer of interest, assume that (i) the current latent representation Ps. E R. MxN consists of the discretization of the first N Legendre polynomials {)){-) such that (Da)is = PP(Zs), (ii) Pn is normalized and the normalization weights in (14), (iii) the discretization satisfies that EA h)((z)) Sa (x) dx| S hh Then, there exists a set of attention weights (WQ, W.K WV3 such that for u(x) = sin sin(a'r) with Z+ 3 >
|| - ull S C maxx(h h, [e]]
(52)
where à and R((,-) are defined as the output of and the kernel of the attention formulation in (14), respectively; s(x) = [|x( R(z, 200 is the error function for the kernel approximation.
Proof. Without loss of generality, it is assumed that a' a + 1. The essential technical tools used suggest the validity for any a' > a > 0 (a', a 2+). Consider a simple non-separable smooth kernel function
<(x, 2) == sin((a + 1)(x - 2)), it is straightforward to verify that for v(x) =- sin(ax), we have c₁ 2aa((2 + 1)
(53)
1 n(x,, 2))(x) dr = =/o sin ((a + 1)(x= - =)) sin(ar) dx = ci sin ((a 1)=) =: cu()). (54)
As is shown, it suffices to show that the matrix multiplication (a separable kernel) in the attention mechanism approximates this non-separable kernel with an error related to the number of channels. To this end, taking the Taylor expansion, centered at 20 Ω of <(x,-) with respect the second variable at each x E Ω we have
(2 20) g
RN(2,2)
Σ
(1 1)! dzl-1 (x, 20).
(55)
It is straightforward to check that
-1 (T 20) 2N + 222
1
[|<(2,-) - (x, (22 S c1
OzN (z,
(56)
N! 2N +I
(0)
By the assumptions on n(;; -), and a straightforward computation we have
(T)
2 K HzN
|n(,, (N, (LL=(0)))))))))))) S 02
(x,
(57)
N
(2)
Next, let (1) (2- 20) 1). and ka(x) == - 1x/00/11(p1/ (s,, 20) 1 N, i.e. they form a Pincherle- Goursat (degenerate) kernels (Kress, 1999, Chapter 11)
N
KN (z, 2) = Σ (()))(2))
(58)
27

---

Published as a conference paper at ICLR 2023
By this choice of the latent representation space being the first N Legendre polynomials, Y := (pj) thus there exists set of weights {wf R') corresponding to each channel, such that
N
(((+) = Σ w
1 (p)p)()) qr(-).
(59)
This is to say, Q pA W. E MXN with the /-th column of Q being the discretization of gr(-).
For the key matrix, by standard polynomial approximation since V = PN((),, there exists set of weights {wf IR J such that
k+)) Σ wt)
(60)
ie., K = py Pk E R MxN with the 1-th column of K being the discretization of ky(.). Moreover, approximate k-))) with the following estimate
NN
(+x (-1L20))) S c3 (N + 1) (B(()
(61)
Similarly, without loss of generality, we choose v1-) == v1).). which is concatenated to such that occupies the first channel of V defined earlier, we have {wf IR such that
N
v(-) ~ Σ
K
(,) =: ü.)
(62)
and
N
||((-) - 6())112(0)) I C1 2 (N + 1) 1o() 10()
(63)
Now, to approximate the frequency bootstrapping in (54), define
ū(=) = / (N(x, dr, with &x,, = Σ (x)))(()).
(64)
Then, we have for any
u(2) ū(z) / n(x, 2)0(z) dx // nN (x, dr
(65)
= / (m((, RN(2, =))u(t) + (x((e, =)((x)) - -(x)) dr.
Thus, we have
| (K(x,, 2)- - (N =) =))e(z}} dr + $
||u 7 (10) S max
-((x)] da
SEll
max
[k(2,)) RN(2 )1 ||l||=(0))) + ( 5| +1
sel2
(66)
Now, by triangle inequality, q: q. the definitions above (53) implying (k()) 2 and || (2))) S ex N ((NNNI), and the estimate in (61)
(+) S ||<[ -) (,,))),,,)))ll.....> )22 + (I, (2,(2x)))22e
< [|(2z -) xx +
(2
N
(67)
65
C6 2N (N + 1)
[ (z (1)()))))))))))
NVV N
(2T)
5 Co
N!vN
28

---

Published as a conference paper at ICLR 2023
Notice this is the order with the estimate of [|<(x,,) (2, For the term || (2, (2))) a simple triangle inequality trick can be used:
[ (1, || ) + [1,x Rx(a, (l
(68)
| (o(x,))]l<2(0)) + ([,, FN )122)
which can be further estimated by reusing the argument in (67).
Lastly, using the following argument and the estimate for || أا (n) yield the desired result:
I| ú I| ( (0) max |u] || - ū alli=(0)
(69)
n
(a)
(b)
(c)
(d)
(e)
(g)
(h)
(k)
Figure The latent representations from CNN-based UNet when evaluating for the sample in Figure 5, in their respective layers, 4 latent representations are extracted from 4 randomly selected channels: (d) are from the feature extracting layer (layer 1, 128 x 128 grid), (e)-(h) are from the middle layer acting on the coarsest level (32 32 grid), (i)-(I) are from the next level (64 x 64 grid).
(a)
(c)
(d)
(c)
(g)
(h)
kk)
Figure 10: The latent representations from UTT when evaluating for the sample in Figure 5, in their corresponding layers with respect to those in Figure 4 latent representations are extracted from randomly selected channels: (a)-d) are from the feature extracting layer (layer 1, 128 x 128 grid), (e)-h) are from the middle layer acting on the coarsest level (32 32 grid), (i)-(1) are from the next level (64 64 grid).
29

---

Published as a conference paper at ICLR 2023
G LIMITATIONS, EXTENSIONS, AND FUTURE WORK
In this study, the a to be recovered relies on a piecewise constant assumption. This assumption is commonly seen in the theoretical study of the original DSM. For many EIT applications in medical imaging and industrial monitoring, may involve non-sharp transitions or even contain highly anisotropic/multicaaee behaviors making it merely an L function. If the boundary data pairs are still quite limited, i.e., only few electric modes are placed on the boundary 2n, the proposed model alone not expected to perform as well as in benchmark problems. Nevertheless, it can still contribute to achieving reconstruction with satisfactory accuracy. if certain priori knowledge of the problem accessible. End2end- wise, our proposed method has limitations like other operator learners: the data manifold which the operator is learned is assumed to exhibit Nov-dimenssiona//ow-rrra attributes. The behavior of the operator of interest on a compact subset assumed to be reasonably well approximated by finite number of bases. Therefore, for non-piecewiss constant conductivities, the modification can be employ suitable data set, in which the sampling of (a(*) represents true a's distribution a posteriori to certain degree. However, to reconstruct non- con-piecewise constant conductivities. more boundary data pairs even the entire NtD map is demanded from theoretical perspective (Astala & Päivärinta, 2006; Nachman, 1996; Kohn Vogelius, 1984; Sylvester & Uhlmann, 1987). For fewer data pairs and more complicated conductivity set-up, there have been efforts in this direction hierarchically using matrix completion (Bui-Thanh et al., 2022) to recover Ag. When A, is indeed available, a can be described by Fredholm integral equation, see Nachman (1996, Theorem 4.1), which itself is strongly related to the modified attention mechanism (14) of the proposed Transformer. The architectural resemblance may lead to future explorations in this direction. Optimization with regularization can be applied for the instance of interest (fine-tune) from the perspective of improving the reconstruction for a single instance. This approach dates back to the classical iterative methods involving adaptivity (Jin & Xu, 2019). Recent novel DI-inspired adaptions (Li al., 2021b; bb; Benitez al., 2023) re-introduce this type of method. In fine-tuning, the initial guess is the reconstruction by the operator learner trained in the end2end pipeline (pre-train).
30


Reviews:
[{"review_id": "VcXd9e5Wea", "reviewer": "Reviewer_CBcr", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper works on the solving boundary value inverse problem by designing the transformer network layers inspired by the mathematics foundations in the boundary value problem. Specifically, it proposed attention-based transformer by learnable non-local kernel, with application to electrical impedance tomography.  The proposed network architecture is based on learning inverse operator by a harmonic extension and an integral operator with the non-local attention kernel. Experiments show better results than the compared U-Nets and inverse operator, including FNO.", "summary_of_the_review": "The good point of this paper is the proposed attention modules for designing deep network for learning inverse operators, with better performance than the FNO and other baselines with U-net as backbone. However, the paper is hard to read, and has limitations in experiments. Please refer to the weakness for details.", "strength_and_weaknesses": "1. Strength\n\n(1) The paper tackles an important inverse problem (boundary value problem) by designing attention-based neural network architecture.\n\n(2) The proposed attention-like learnable operators have novelty and mathematical foundation. \n\n(3) The proposed network for inverse operator of electrical impedance tomography shows good results.\n\n2. Weakness\n\nThe major weakness  this paper are on the evaluation of effectiveness of the proposed  transformer, the unclear presentation of the proposed method, and the missing training and experimental details and comparisons.\n\n(1)  The paper designed the attention mechanism in Eqn. (9) deriving the non-local kernel with theoretical foundation in Theorem 2. In the section 3.4, the index map integral is also formulated as attention. In these subsections, the modeling of ingredients in electrical impedance tomography are formulated as attentions. What are the advantages of the proposed  attention? These attention might be also possible to further extended with learnable parameters, might further increase the learning power. It is also unclear to me, how these proposed attentions construct a deep network? The network training loss and details should be also presented in the major part of the paper. \n\n(2) In the experiments, the proposed method is compared with the CNN-based U-Net, FNO, and also variants of U-Net with different attentions. The proposed attention shows better performance. However, in the experimental sections, their is no detailed introduction to the experimental datasets, training details, network parameter sizes, etc.  The comparisons with more neural operator learning methods (refer to the related works) should be also given. \n\n(3) Overall, the organization of the paper can be improved. It is hard to read to me in the current version, mainly because of missing explanations to the global network architecture, the rule of the proposed modulus in the network architecture, and training/experimental details. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "VWXbDwS5tv", "reviewer": "Reviewer_5fLN", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to use the transformer architecture to solve boundary value inverse problems. \n\nTheoretically, the authors justify the desirable properties of Transformers in boundary-value inverse problems in two ways: (i) an attention-based mechanism can generate an output signal with high frequencies than its input (Theorem 2), and (ii) the index map integral of boundary-value problems is similar to an attention mechanism in form (Equation (25)). \n\nEmpirically, the authors demonstrated the strong performance of transformer-based architecture in an electrical impedance tomography problem.", "summary_of_the_review": "I feel that the clarity of the paper (especially Section 3.1) can be improved in a few ways. As mentioned above, I think that the current formulation can be a bit confusing; there may exist a few glitches. Other than that, I think the work is solid.", "strength_and_weaknesses": "# Strength\nThe proposed transformer-based operator model is novel and achieves strong performance. The \"frequency bootstrapping\" property demonstrated for the attention mechanism in Theorem 2 is revealing -- It may be of independent interest to general applications of transformers beyond inverse problems.\n\n# Weakness\nI was confused by the writing of the authors in quite a few places. Below I list a few of them.\n \n- Is the sum $\\sum_{m \\in \\mathbb{M}_L}$ in Equation (2) extraneous? First, the letter $m$ in the sum is not defined. Second, if $m$ is meant to be $\\mathbf{m}$, then one should probably write an integral instead of a sum -- the space $\\mathbb{M}_L$ is not discrete in general. Last, even if the integral is used to replace the sum,  I still do not see why one wants to find a ground-truth signal $\\mathbf{p}$ so that its forward process is close to all **all** possible measurements $\\mathbf{m} \\in \\mathbb{M}$. The curly bracket in Equation (2) was also placed in a confusing way -- I assume that the regularization term should be included in $\\inf$.\n\n- What confuses me the most is Section 3.1. There seem to be a couple of glitches in the current formulation. At the bottom of page 3, the authors wrote \"Then, the coefficient of (4) to be recovered can be described by a characteristic function $\\mathbf{p} = \\mathcal{I}^D(x)$ defined for ...\" Later, in the same sentence at the beginning of page 4, the authors wrote \"or equivalently, $\\mathbf{p}=\\sigma_1 \\mathcal{I}^D+\\sigma_0\\left(1-\\mathcal{I}^D\\right)$.\" The definition of $\\mathbf{p}$ at these two different places is inconsistent. \n\n- If I understand it correctly, the object $\\mathbf{p}=\\sigma_1 \\mathcal{I}^D+\\sigma_0\\left(1-\\mathcal{I}^D\\right)$ is what we aim to recover in EIT.  However, in the paragraph before the paragraph of Equation (6), it was mentioned that $\\mathcal{F}^{-1}: \\mathbb{M} \\to \\mathbb{P}$ is \"essentially a map $\\Lambda_{\\sigma} \\mapsto \\sigma$\".  I am baffled by this sentence. Indeed, $\\Lambda_{\\sigma}$ is not an element of $\\mathbb{M}$, and $\\sigma$ is not an element of $\\mathbb{P}$. In addition, I thought that $\\sigma$ is known -- in lines after Equation (4), the authors mentioned \"The values $\\sigma_0$ and $\\sigma_1$ are two (approximately) known constants...\". So my understanding is that the unknown in  $\\mathbf{p}=\\sigma_1 \\mathcal{I}^D+\\sigma_0\\left(1-\\mathcal{I}^D\\right)$ is $D$, not $\\sigma$. Could the authors correct me if this is wrong? If $\\sigma$ is the unknown, then the authors may want to remove the confusing sentence \"The values $\\sigma_0$ and $\\sigma_1$ are two (approximately) known constants...\".  If $D$ is the unknown, how does one recover $\\mathbf{p}=\\sigma_1 \\mathcal{I}^D+\\sigma_0\\left(1-\\mathcal{I}^D\\right)$ from $\\sigma = (\\sigma_0, \\sigma_1)$ yielded from $\\Lambda_{\\sigma} \\mapsto \\sigma$, when $D$ is not given?\n\n- In general, to improve clarity in Section 3.1, I think it would be great if the authors could explicitly relate the forward operator in EIT to Equation (1) -- what do $\\mathbf{m}$, $\\mathbf{p}$, and $\\mathcal{F}$ stands for in the EIT problem. \n\n- In section 3.2, the authors cast the attention mechanism as a kernel integral operator. A similar observation has been previously shown in [1]\n\n- In the experiment section, it would be great if the authors could comment/demonstrate the mesh-invariance of the proposed operator model. For instance, the authors could train their operator model on one mesh discretization and evaluate it on another discretization (\"zero-shot superresolution\"), and see if the performance degrades substantially. Similar experiments have been reported by many earlier neural operator papers -- the authors can use them as benchmarks.\n\n- The experiment section focuses on an EIT problem. Have the authors considered other generic operator-learning problems, e.g., generic parametric PDE datasets? Since EIT is the focus of this paper, it would be unfair to ask the authors to train their models on these irrelevant datasets -- but I would be curious to learn if the authors have tried before. \n\n- Typo: In \"2.1 Contributions\", second bullet point, Transformeris -> Transformers\n\n[1] Guibas J, Mardani M, Li Z, Tao A, Anandkumar A, Catanzaro B. Adaptive Fourier neural operators: Efficient token mixers for transformers. ICLR 2022\n\n\n ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "N81yXiVw22I", "reviewer": "Reviewer_ne3Z", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new Transformer based architecture to certain inverse problems in which attempt is to reconstruct the structure of inner domain from boundary measurements. The problems are important as several practical applications fall into this class (e.g. EIT, optical tomography, seismic tomography). The study attempts to answer two questions: 1) how the boundary data should be fed to the model, and 2) what is structure of the neural network\n\n\n", "summary_of_the_review": "Overall, I am positive about this study and feel that it deserves publications. But I would have some discussion about practical e.g. in future directions.\n\n(Note: I have also previously reviewed this paper for another conference.)", "strength_and_weaknesses": "Strengths:\nI feel that the paper has potential as it proposes a new concept to solve the boundary value inverse problems (at least in theory). They combine attention mechanism from transformers to solution and provide theoretical background/justification for that why it is done. This is interesting new approach. The paper is also written relatively well and derivations are understandable.\n\nWeaknesses:\nI feel that the study is quite theoretical as it considers a very simplified setup: the unknown is assumed to be an inclusion with known conductivity with also known background conductivity. This setup is quite commonly used assumption in more theoretical studies related to EIT and therefore would not high weight on this. But I would still like to also see some discussion about possible extensions for more practical setups. For example, how the approach could be extended to conductivity distributions which have non-sharp transitions. There are several examples of such cases in medical imaging and monitoring of industrial processes.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "8LcrXWPL7AL", "reviewer": "Reviewer_4qwm", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose to use transformers for electric impedance tomography, with ideas that may apply to a broade range of boundary value inverse problems. EIT is known to be very ill-posed (only log stable) so it is a challenging test for any method. The authors frame the problem as inverting the samples of the NtD map. They draw parallels between integral operators / kernels and the architecture of attention layers. They consider the design of proper spatial inputs to the reconstruction map from the boundary data, and propose harmonic extensions as a solution.\n", "summary_of_the_review": "I think this paper contains some nice ideas and I very much appreciate the proper \"inverse problems culture\" in respecting the structure of the problem. On the other hand, the presentation needs a lot of work, both in the direction of improving the prose and clarity, and in the direction of adapting the style to the ICLR audience, providing the requisite background, and abstracting the most important messages without over-entangling them with the specifics of EIT. \n", "strength_and_weaknesses": "The authors design a transformer-based architecture to solve EIT (and more generally boundary value inverse problems). Their design is inspired by the mathematical / computational structure of the problem which is in a certain sense analogous to that of the attention layer. This is related to but different from physics-driven deep learning where usually the forward operator is somehow embedded in the network design. The present contribution is rather about shaping the inductive bias of the network for a specific class of problems.\n\nI very much like that the authors address the problem by looking at operator-valued data (namely, the NtD map) and its discretization. This is the \"right way\" in the inverse problems community but earlier approaches mostly employ ad hoc solutions. Another nice thing in the manuscript is a principled proposal to \"continue\" the boundary data into the interior via harmonic extensions. These extensions can then be used as inputs to the transformer network. Numerical results show that their proposed method outperforms several strong baselines.\n\nOn the critical side, I find the paper very hard to follow. The prose could use a great deal of work---there are numerous misprints, non-idiomatic constructions, and broken and confusing sentences. Since this is very technical and specialist material that cannot be easily picked up \"on the go\", this will present a huge challenge to most ICLR readers. Just as importantly, many explanations are non-intuitive and  require background in EIT to follow. This is unfortunate because a typical ICLR reader will not have this background even if they are familiar with inverse problems in a different context. For comparison, earlier operator-learning papers like the FNOs are much easier to follow.\n\nI'll give some examples to support my claims but they are far from exhaustive:\n\n- \"... the problem of seeking the approximated operator F_L^{-1} is usually highly-ill-posed (not having a well-defined unique output) and poses great challenges to the reconstruction algorithms\" -> what is ill posed? computing an operator approximation or the inverse problem itself? what poses great challenges? the problem or the ill-posedness? and what is meant by output?\n\n- in equation (2), what is the summation over? should the norm be replaced by some pointwise discrepancy? or the sum removed?\n\n- \"In this regard, the proposed study provides a positive example to a hopefully definitive answer to this question, which bridges deep learning and conventional tasks in physical sciences.\" -> I don't undersatnd this sentence\n\n- \"the constructional proof of the existence of ILD still relies on the entire NtD mapping Λσ , which again resorts to infiniteness, thus inaccessible in real applications.\" -> this is just very hard to read and broken\n\n- \"If we further assume that there exist a set of feature maps for query, key, and value, e.g., see Choromanski et al. (2021).\" -> this is again a broken sentence\n\n- I find the descriptions in 3.3 and 3.4 very difficult to follow. I understand the high-level idea in 3.3 but the intuitive reasoning, the motivation for the different choices, and the interpretation of the harmonic extension would greatly help my understanding. \n\n- I am not listing the numerous misprints and non-idiomatic constructions that further complicate things\n\n\n### Additional comments and questions\n\n- just before Theorem 2 you state that a CNN cannot generate an output with higher frequencies than the input but this is not true; CNNs such as a U-Net are routinely uses for tasks like image super-resolution. Your statement would be true for pure convolutions but CNNs have nonlinear activations and other architectural details which allow them to generate high frequencies.\n\n- I find it unfair to say that methods like MUSIC or D-bar are ad hoc in the EIT context; one could then similarly state that harmonic extensions as input to a transformer is also ad hoc.\n\n- \"However, such a simple closed form of Gθ admitting efficient execution may not be available in practice since some mathematical assumptions and derivation may not hold.\" which assumptions and derivations are those? \n\n- Could you show the ground truth in Figure 5? (appendix)\n\n- Out of curiosity: is there any relation between harmonic extensions and positional encodings?\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster