Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors propose to use transformers for electric impedance tomography, with ideas that may apply to a broade range of boundary value inverse problems. EIT is known to be very ill-posed (only log stable) so it is a challenging test for any method. The authors frame the problem as inverting the samples of the NtD map. They draw parallels between integral operators / kernels and the architecture of attention layers. They consider the design of proper spatial inputs to the reconstruction map from the boundary data, and propose harmonic extensions as a solution.

strength_and_weaknesses: The authors design a transformer-based architecture to solve EIT (and more generally boundary value inverse problems). Their design is inspired by the mathematical / computational structure of the problem which is in a certain sense analogous to that of the attention layer. This is related to but different from physics-driven deep learning where usually the forward operator is somehow embedded in the network design. The present contribution is rather about shaping the inductive bias of the network for a specific class of problems.

I very much like that the authors address the problem by looking at operator-valued data (namely, the NtD map) and its discretization. This is the "right way" in the inverse problems community but earlier approaches mostly employ ad hoc solutions. Another nice thing in the manuscript is a principled proposal to "continue" the boundary data into the interior via harmonic extensions. These extensions can then be used as inputs to the transformer network. Numerical results show that their proposed method outperforms several strong baselines.

On the critical side, I find the paper very hard to follow. The prose could use a great deal of work---there are numerous misprints, non-idiomatic constructions, and broken and confusing sentences. Since this is very technical and specialist material that cannot be easily picked up "on the go", this will present a huge challenge to most ICLR readers. Just as importantly, many explanations are non-intuitive and  require background in EIT to follow. This is unfortunate because a typical ICLR reader will not have this background even if they are familiar with inverse problems in a different context. For comparison, earlier operator-learning papers like the FNOs are much easier to follow.

I'll give some examples to support my claims but they are far from exhaustive:

- "... the problem of seeking the approximated operator F_L^{-1} is usually highly-ill-posed (not having a well-defined unique output) and poses great challenges to the reconstruction algorithms" -> what is ill posed? computing an operator approximation or the inverse problem itself? what poses great challenges? the problem or the ill-posedness? and what is meant by output?

- in equation (2), what is the summation over? should the norm be replaced by some pointwise discrepancy? or the sum removed?

- "In this regard, the proposed study provides a positive example to a hopefully definitive answer to this question, which bridges deep learning and conventional tasks in physical sciences." -> I don't undersatnd this sentence

- "the constructional proof of the existence of ILD still relies on the entire NtD mapping Λσ , which again resorts to infiniteness, thus inaccessible in real applications." -> this is just very hard to read and broken

- "If we further assume that there exist a set of feature maps for query, key, and value, e.g., see Choromanski et al. (2021)." -> this is again a broken sentence

- I find the descriptions in 3.3 and 3.4 very difficult to follow. I understand the high-level idea in 3.3 but the intuitive reasoning, the motivation for the different choices, and the interpretation of the harmonic extension would greatly help my understanding. 

- I am not listing the numerous misprints and non-idiomatic constructions that further complicate things


### Additional comments and questions

- just before Theorem 2 you state that a CNN cannot generate an output with higher frequencies than the input but this is not true; CNNs such as a U-Net are routinely uses for tasks like image super-resolution. Your statement would be true for pure convolutions but CNNs have nonlinear activations and other architectural details which allow them to generate high frequencies.

- I find it unfair to say that methods like MUSIC or D-bar are ad hoc in the EIT context; one could then similarly state that harmonic extensions as input to a transformer is also ad hoc.

- "However, such a simple closed form of Gθ admitting efficient execution may not be available in practice since some mathematical assumptions and derivation may not hold." which assumptions and derivations are those? 

- Could you show the ground truth in Figure 5? (appendix)

- Out of curiosity: is there any relation between harmonic extensions and positional encodings?



clarity,_quality,_novelty_and_reproducibility: I find the paper quite unclear. As far as I can tell the work is novel and reproducible.
summary_of_the_review: I think this paper contains some nice ideas and I very much appreciate the proper "inverse problems culture" in respecting the structure of the problem. On the other hand, the presentation needs a lot of work, both in the direction of improving the prose and clarity, and in the direction of adapting the style to the ICLR audience, providing the requisite background, and abstracting the most important messages without over-entangling them with the specifics of EIT.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes to use the transformer architecture to solve boundary value inverse problems. 

Theoretically, the authors justify the desirable properties of Transformers in boundary-value inverse problems in two ways: (i) an attention-based mechanism can generate an output signal with high frequencies than its input (Theorem 2), and (ii) the index map integral of boundary-value problems is similar to an attention mechanism in form (Equation (25)). 

Empirically, the authors demonstrated the strong performance of transformer-based architecture in an electrical impedance tomography problem.
strength_and_weaknesses: # Strength
The proposed transformer-based operator model is novel and achieves strong performance. The "frequency bootstrapping" property demonstrated for the attention mechanism in Theorem 2 is revealing -- It may be of independent interest to general applications of transformers beyond inverse problems.

# Weakness
I was confused by the writing of the authors in quite a few places. Below I list a few of them.
 
- Is the sum $\sum_{m \in \mathbb{M}_L}$ in Equation (2) extraneous? First, the letter $m$ in the sum is not defined. Second, if $m$ is meant to be $\mathbf{m}$, then one should probably write an integral instead of a sum -- the space $\mathbb{M}_L$ is not discrete in general. Last, even if the integral is used to replace the sum,  I still do not see why one wants to find a ground-truth signal $\mathbf{p}$ so that its forward process is close to all **all** possible measurements $\mathbf{m} \in \mathbb{M}$. The curly bracket in Equation (2) was also placed in a confusing way -- I assume that the regularization term should be included in $\inf$.

- What confuses me the most is Section 3.1. There seem to be a couple of glitches in the current formulation. At the bottom of page 3, the authors wrote "Then, the coefficient of (4) to be recovered can be described by a characteristic function $\mathbf{p} = \mathcal{I}^D(x)$ defined for ..." Later, in the same sentence at the beginning of page 4, the authors wrote "or equivalently, $\mathbf{p}=\sigma_1 \mathcal{I}^D+\sigma_0\left(1-\mathcal{I}^D\right)$." The definition of $\mathbf{p}$ at these two different places is inconsistent. 

- If I understand it correctly, the object $\mathbf{p}=\sigma_1 \mathcal{I}^D+\sigma_0\left(1-\mathcal{I}^D\right)$ is what we aim to recover in EIT.  However, in the paragraph before the paragraph of Equation (6), it was mentioned that $\mathcal{F}^{-1}: \mathbb{M} \to \mathbb{P}$ is "essentially a map $\Lambda_{\sigma} \mapsto \sigma$".  I am baffled by this sentence. Indeed, $\Lambda_{\sigma}$ is not an element of $\mathbb{M}$, and $\sigma$ is not an element of $\mathbb{P}$. In addition, I thought that $\sigma$ is known -- in lines after Equation (4), the authors mentioned "The values $\sigma_0$ and $\sigma_1$ are two (approximately) known constants...". So my understanding is that the unknown in  $\mathbf{p}=\sigma_1 \mathcal{I}^D+\sigma_0\left(1-\mathcal{I}^D\right)$ is $D$, not $\sigma$. Could the authors correct me if this is wrong? If $\sigma$ is the unknown, then the authors may want to remove the confusing sentence "The values $\sigma_0$ and $\sigma_1$ are two (approximately) known constants...".  If $D$ is the unknown, how does one recover $\mathbf{p}=\sigma_1 \mathcal{I}^D+\sigma_0\left(1-\mathcal{I}^D\right)$ from $\sigma = (\sigma_0, \sigma_1)$ yielded from $\Lambda_{\sigma} \mapsto \sigma$, when $D$ is not given?

- In general, to improve clarity in Section 3.1, I think it would be great if the authors could explicitly relate the forward operator in EIT to Equation (1) -- what do $\mathbf{m}$, $\mathbf{p}$, and $\mathcal{F}$ stands for in the EIT problem. 

- In section 3.2, the authors cast the attention mechanism as a kernel integral operator. A similar observation has been previously shown in [1]

- In the experiment section, it would be great if the authors could comment/demonstrate the mesh-invariance of the proposed operator model. For instance, the authors could train their operator model on one mesh discretization and evaluate it on another discretization ("zero-shot superresolution"), and see if the performance degrades substantially. Similar experiments have been reported by many earlier neural operator papers -- the authors can use them as benchmarks.

- The experiment section focuses on an EIT problem. Have the authors considered other generic operator-learning problems, e.g., generic parametric PDE datasets? Since EIT is the focus of this paper, it would be unfair to ask the authors to train their models on these irrelevant datasets -- but I would be curious to learn if the authors have tried before. 

- Typo: In "2.1 Contributions", second bullet point, Transformeris -> Transformers

[1] Guibas J, Mardani M, Li Z, Tao A, Anandkumar A, Catanzaro B. Adaptive Fourier neural operators: Efficient token mixers for transformers. ICLR 2022


 
clarity,_quality,_novelty_and_reproducibility: The clarity of the paper (especially Section 3.1) can be improved in a few ways -- see my comments above. The quality and novelty of the work are high in my opinion. The proposed operator learning model, described in detail in Appendix B.2, seems reproducible.

summary_of_the_review: I feel that the clarity of the paper (especially Section 3.1) can be improved in a few ways. As mentioned above, I think that the current formulation can be a bit confusing; there may exist a few glitches. Other than that, I think the work is solid.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper works on the solving boundary value inverse problem by designing the transformer network layers inspired by the mathematics foundations in the boundary value problem. Specifically, it proposed attention-based transformer by learnable non-local kernel, with application to electrical impedance tomography.  The proposed network architecture is based on learning inverse operator by a harmonic extension and an integral operator with the non-local attention kernel. Experiments show better results than the compared U-Nets and inverse operator, including FNO.
strength_and_weaknesses: 1. Strength

(1) The paper tackles an important inverse problem (boundary value problem) by designing attention-based neural network architecture.

(2) The proposed attention-like learnable operators have novelty and mathematical foundation. 

(3) The proposed network for inverse operator of electrical impedance tomography shows good results.

2. Weakness

The major weakness  this paper are on the evaluation of effectiveness of the proposed  transformer, the unclear presentation of the proposed method, and the missing training and experimental details and comparisons.

(1)  The paper designed the attention mechanism in Eqn. (9) deriving the non-local kernel with theoretical foundation in Theorem 2. In the section 3.4, the index map integral is also formulated as attention. In these subsections, the modeling of ingredients in electrical impedance tomography are formulated as attentions. What are the advantages of the proposed  attention? These attention might be also possible to further extended with learnable parameters, might further increase the learning power. It is also unclear to me, how these proposed attentions construct a deep network? The network training loss and details should be also presented in the major part of the paper. 

(2) In the experiments, the proposed method is compared with the CNN-based U-Net, FNO, and also variants of U-Net with different attentions. The proposed attention shows better performance. However, in the experimental sections, their is no detailed introduction to the experimental datasets, training details, network parameter sizes, etc.  The comparisons with more neural operator learning methods (refer to the related works) should be also given. 

(3) Overall, the organization of the paper can be improved. It is hard to read to me in the current version, mainly because of missing explanations to the global network architecture, the rule of the proposed modulus in the network architecture, and training/experimental details. 
clarity,_quality,_novelty_and_reproducibility: The paper clarity is not satisfactory. The datasets and codes are suggested to be released for reproducibility. 
summary_of_the_review: The good point of this paper is the proposed attention modules for designing deep network for learning inverse operators, with better performance than the FNO and other baselines with U-net as backbone. However, the paper is hard to read, and has limitations in experiments. Please refer to the weakness for details.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a new Transformer based architecture to certain inverse problems in which attempt is to reconstruct the structure of inner domain from boundary measurements. The problems are important as several practical applications fall into this class (e.g. EIT, optical tomography, seismic tomography). The study attempts to answer two questions: 1) how the boundary data should be fed to the model, and 2) what is structure of the neural network



strength_and_weaknesses: Strengths:
I feel that the paper has potential as it proposes a new concept to solve the boundary value inverse problems (at least in theory). They combine attention mechanism from transformers to solution and provide theoretical background/justification for that why it is done. This is interesting new approach. The paper is also written relatively well and derivations are understandable.

Weaknesses:
I feel that the study is quite theoretical as it considers a very simplified setup: the unknown is assumed to be an inclusion with known conductivity with also known background conductivity. This setup is quite commonly used assumption in more theoretical studies related to EIT and therefore would not high weight on this. But I would still like to also see some discussion about possible extensions for more practical setups. For example, how the approach could be extended to conductivity distributions which have non-sharp transitions. There are several examples of such cases in medical imaging and monitoring of industrial processes.


clarity,_quality,_novelty_and_reproducibility: The approach is novel and well presented. Quality is good. 
summary_of_the_review: Overall, I am positive about this study and feel that it deserves publications. But I would have some discussion about practical e.g. in future directions.

(Note: I have also previously reviewed this paper for another conference.)
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper presents a new method for solving a class of boundary value inverse problems, electrical impedance tomography (EIT), based on Transformer. This paper makes contributions in both theoretical and experimental sides. Theoretically, the authors prove that a modified attention module can represent target functions with higher frequencies than the input, and show the connections between the index function integral and the attention mechanism in Transformer. Experimentally, the authors obtain the strong empirical results on EIT benchmark tests with the proposed U-Integral Transformer architecture under various settings.
 
The reviewers agree that this work is novel and interesting (4qwm, 5fLN, CBcr, ne3Z), and the strong empirical performances look promising (4qwm, 5fLN, CBcr). Some concerns are raised, particularly on the presentation quality of the paper (4qwm, 5fLN, CBcr), and most reviewers feel the rebuttal revision addresses the concerns. In conclusion, we recommend acceptance.
justification_for_why_not_higher_score: This work only study a very specific class of boundary value inverse problem and requires several assumptions. The quality of the paper could be further improved by investigating more general cases.
justification_for_why_not_lower_score: The paper is well-motivated. The method is solid and interesting. During the rebuttal, they addressed most reviewers' concerns and largely improve the paper.
Decision: Accept: poster