Latent Neural Operator for Solving Forward and Inverse PDE Problems
Authors: Tian Wang
Chuang Wang
Venue: NeurIPS 2024 poster (2024)
Keywords: latent neural operator
PDE
forward and inverse problems
Primary area: machine_learning_for_physical_sciences
TL;DR: We propose Latent Neural Operator to solve forward and inverse PDE problems in latent space.

Abstract: Neural operators effectively solve PDE problems from data without knowing the explicit equations, which learn the map from the input sequences of observed samples to the predicted values. Most existing works build the model in the original geometric space, leading to high computational costs when the number of sample points is large. We present the Latent Neural Operator (LNO) solving PDEs in the latent space. In particular, we first propose Physics-Cross-Attention (PhCA) transforming representation from the geometric space to the latent space, then learn the operator in the latent space, and finally recover the real-world geometric space via the inverse PhCA map. Our model retains flexibility that can decode values in any position not limited to locations defined in the training set, and therefore can naturally perform interpolation and extrapolation tasks particularly useful for inverse problems. Moreover, the proposed LNO improves both prediction accuracy and computational efficiency. Experiments show that LNO reduces the GPU memory by 50%, speeds up training 1.8 times, and reaches state-of-the-art accuracy on four out of six benchmarks for forward problems and a benchmark for inverse problem. Code is available at https://github.com/L-I-M-I-T/LatentNeuralOperator.

Latent Neural Operator for Solving Forward and Inverse PDE Problems
Tian Wang
Chuang Wang"
State Key Laboratory of
State Key Laboratory of
Multimodal Artificial Intelligence Systems
Multimodal Artificial Intelligence Systems
Institute of Automation, Chinese Academy of Sciences School of Artificial Intelligence,
Institute of Automation, Chinese Academy of Sciences
School of Artificial Intelligence,
University of Chinese Academy of Sciences wangtian20220ia..ee an
University of Chinese Academy of Sciences wangchuang@ia.c an.
Abstract
Neural operators effectively solve PDE problems from data without knowing the explicit equations, which learn the map from the input sequences of observed samples to the predicted values. Most existing works build the model in the original geometric space, leading to high computational costs when the number sample points is large. We present the Latent Neural Operator (LNO) solving PDEs in the latent space. In particular, we first propose Physics-Cross transforming representation from the geometric space the latent space, then learn the operator in the latent space, and finally recover the real-world geometric space via the inverse PhCA map. Our model retains flexibility that can decode values in any position not limited to locations defined in the training set, and therefore naturally perform interpolation and extrapolation tasks particularly useful for inverse problems. Moreover, the proposed LNO improves both prediction accuracy and computational efficiency. Experiments show that LNO reduces the GPU memory by 50%, speeds up training 1.8 times, and reaches state-of-the-art accuracy four out of six benchmarks for forward problems and benchmark for inverse problem. Code is available at
(PhCA)
Introduction
Neural network approaches for partial differential equations (PDEs) attract increasing attention in diverse scientific fields, for instance, meteorological prediction industrial design 2), geological sensing[3] and environmental monitoring(4) to methods, e.g., finite element [e which demands a large number of computational resources and requires meticulous discretization involving lot of specialized knowledge, neural networks provide a new paradigm for solving PDE-driven problems with better trade-off between accuracy and flexibility.
few. Compared with traditional numerical
Neural PDE solvers can be divided into two main categories, physics-informet and operator-learning models. The physics-inie innfrre- [1 enforce the inductive bias of explicit PDE instances into loss function or model architecture, usually yielding solutions with high accuracy, but are less instance. In contrast, the
flexible they require fresh training to find the solution for each operator learning methods 12 adopt data-driven paradigm to learn the implicit PDE constraint
*Corresponding Author, work is supported by Beijing Municipal Science and Technology Project (No. Z231 100010323005), the 2035 Innovation Mission (Grants No. E4J10102) and Pioneer Hundred Talents Program of CAS under Grant Y9S9MS08 and E2S40101.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

from pair of samples representing imput-to-uuipin functional mappings, providing better flexibility that can generalize different initial/bunddrry conditions beyond the scope of the training data, but have yet large room to improve in the prediction accuracy.
Recently, Transformers T6] prevail in neural operator structures. Theoretically, the attention mecha- nism was proved to be a special form of the integral kernel155 in neural operator, which naturally conforms
-ooseueee--- characterization. Moreover, the fully-connected attention structure enables the model to characterize long-distance and quadratic relationships among sample points, which yields more accurate prediction than the vanilla MLP structure [I3] but the expense of increasing the computational complexity drastically, as the complexity is quadratic respect to the length of imput sequence.
To alleviate the complexity, existing works employed linear attention mechanism [8] speeding up the computation but sacrificing precision due their limited expressive power. Another line of works attempted to solve the PDE in the latent space with a small number of samples (2) which get rid of abundant sample points in the original geometric space and capture physical correlations in the tighter latent space.
In this work, we present the Latent Neural Operator (LNO) with particular effort designing the Physics-Crov Cross-Attensian (PhCA) module to learn the latent space to optimize the trade-off between accuracy and flexibility, and reduce the computational complexity as well. Specifically, LNO first encodes the input sample sequence into a learnable latent space by PhCA. Then, the model learns the PDE operator in the latent space via stack of Transformer layers. Finally, the predicted output function decoded by the inverse PhCA, given any query locations. Unlike previous works that either pre-define the latent space as frequency domain [14]. switch back-and-forth between latent space and geometric space at each layer [23), the latent space of the LNO is learnable from data and the data are only transformed at the first and last layer.
The proposed PhCA module decouples the locations of input samples fed in the encoder and output samples fed in the decoder, which can predict values in any position not limited by the observation. Therefore, it can naturally perform interpolation and extrapolation tasks, which fundamental for solving inverse problems. The proposed LNO improves both prediction accuracy and computational efficiency. Experiments show that LNO reduces the GPU memory by 50% and speeds up training 1.8 times. Moreover, it reaches oftte-of-theaar accuracy four (Pipe, Elasticity, Darey, Havier-Stokes out of six benchmarks for forward problems and a benchmark for inverse problem on Burgers' equation.
The main contributions our LNO framework are summarized as follows.
Flexibility:
We propose the Physics-Crosse Cross-Attention (PhCA) module which decouples the locations of input and output samples and learns the latent space from data. We also build the Latent Neural Operator (LNO) model for solving both forward and inverse PDE problems.
Efficiency: The LNO model transforms the data to the latent space only once. Compared with other approaches that switch the latent space and geometric space in each Transformer layer, the proposed model reduces complexity drastically in terms of GPU memory usage, training time and number of model parameters.
Accuracy: We obtain state-of-the-an results the forward problems of Pipe, Elasticity, Darcy, Navier-Stokes, and the inverse problem of Burgers' equation.
2
Related works
We first review two major classes of neural PDE approaches, physics-informed and operator-learning methods. Then, we briefly discuss the works using neural networks for solving inverse PDE problems.
2.1 Physics-Informee Machine Learning
Physics-informed machine learning methods incorporate prior knowledge of PDEs into loss func- tion or model architecture. The Physics-Informen Neural Network (PINN) and its variants are among the most classic methods. PINN integrates the differential equation, initial conditions and boundary conditions into the loss function, guiding the neural network to approximate the solution

---

by ensuring its derivatives satisfies the equation and the output aligns with the initial and boundary conditions. Other methods attempt encode physical priors into the model architecture, e.g., the Flnite volume Neural Network (FINN) g and Physics-encoded Recurrent Convolutional Neural Network (PeRCNN) [O] These methods achieve high precision but require solving optimization to train the networks for each instance. In addition, in the case, where concrete PDE is unknown or not exact, it is challenging to adopt those methods, limiting the generalization ability and flexibility in practical data-rich situations.
2.2 Operator Learning
Operator learning methods aim to find functional map from coefficients, forcing terms and initial conditions to the solution function by learning an approximating operator. These methods train neural networks to capture the correspondence between input and output functions from data without needing to know the underlying PDE explicitly. DeepONet 13 27 first introduces approximating operators using neural networks.
series of kernel integral operator forms were proposed to model the mappings between functions neural operators 28 14 [5]. Specifically, Galerkin Transformer(20) utilizes the Galerkin-ty eeree attention mechanism as kernel integral operators. OFormer (29] improves Galerkin-type attention by decoupling observation positions and query positions through cross-attention. GNOT[11] introduces heterogeneous normalized cross-attention to modulate various input conditions for query positions and alleviates computational pressure of the large grid by thear-time-complees attention mecha- nism. FactFormer (30) proposes axial factorized kernel integral, decomposing input functions into multiple one-dimensional sub-functions, effectively reducing computation. ONO[3]] enhances the generalization of neural operators by adding orthogonality regularization in attention mechanisms.
Researchers also explored the idea of modeling in the latent space. Transolver 23 employs Physics- Attention in each layer to map back-and-fort between geometric features and physical features. LSM encodes the input function into the latent space using cross-attenss attenta,,nt,on, constructs a set of orthogonal bases, and then decodes the solution back to the geometric space through cross- attention again. UPT224 introduces encoding and decoding losses, and compresses geometric point information into super-node information the latent space via graph neural network (32]
Compared with existing methods, we autonomously learn mappings between functions in the real- world space and the latent space an end-to-end fashion with the cross-altention mechanism, without manually constructed orthogonal basis nor additionally introduced loss.
2.3 Inverse Problem
Inverse problems of PDEs have extensive applications in fields involving sensing and reconstruction such as medical imaging and geological sensing. For instance, Transformer-based deep direct sampling methods (33] have been proposed for electrical impedance tomography, and convolutional network methods 36 are widely used in full waveform inversion.
Theoretical works on solving inverse problems have also been proposed. The reconstruction of diffusion fields with localized sources has been studied (37] from the perspective of sampling matrix
in the spatiotemporal domain. PINN 38 40) approximates the solution based partially observed
38
data to compute the unknown data. NIO 41 combines DeepONet [3] and FNO 14) to construct a framework for solving inverse problems. FINN 42 sets boundary conditions as learnable param- eters, and infers boundary values through backpropagation. The generative method[43] generates the initial conditions through latent codes and refines them based on the results of time evolution. We aim to unify the solution of forward and inverse problems in the latent space through the PhCA module, which decouples observation and prediction positions.
Method
We formally define both the forward and the inverse problems of PDEs. Then, we introduce our Latent Neural Operator (LNO) model. Finally, we discuss the core module that learns the transformation between the real-world geometric space and the latent space via the cross-attention mechanism.
3

---

Latent Space
Input Function
[N D]
[M D]
[M
[Want D]
((poss vut 1m
Branch Projecter
PaCA Eneeder
Transformer Block
P&CA Decoder
MLP
(eat sut
New D]
[N mur. D]
Trank Projecter
Trank Projecter
Sharing Parameters
(pas in li=
(D Nos [pus ouc jel
(ppos 20
vul " f..
Observation Position
Prediction Position
Output Function
Figure I: The overall architecture of Latent Neural Operator.
3.1 Problem Setup
We consider partial differential equation (PDE) with boundary and/or initial conditions defined ? C R or gd x [0,
La o O, u(x) b(x), x E 0n
(1)
where La is an operator containing partial differentials parameterized by a set of coefficients a, and u(x), Ω and b(x) represent the solution the PDE and the boundary/initial conditions respectively.
A conventional forward problem aims to find the solution u(x)), given the differential operator La in I with the explicit parameters a and the boundary/initaal conditions b(z). For an inverse problem, given an observed set of partial data {(x, u(x)) x E C Ω), the task is to infer the PDE that generates the observed data along with possible boundary/initial conditions the parameters in the differential operator C..
Both forward and inverse problems can be unified operator learning task fitting into sequence- to-sequence translation framework. An operator model estimates the mapping F f from an input function f to output function g by the data-driven approach using paired training data of input-output sequences ( {g(2))}:). where ys and žj correspond positions in the domain of the input and output function respectively.
Specifically, for the forward problem, the inputs are samples of either boundary/initial function b or PDE coefficient function a, and the output is the physical quantity u evaluated at a grid of points. Conversely, for the inverse problem, the input is few partially observation samples { in va( (
in } Ni. = {(r, u(x;))); and the outputs are boundary/initial function (e.g., infer initial state) or the PDE coefficient function (also coined as system identification problem).
The challenge in solving forward and inverse problems of PDEs lies in the large lengths of input and output sequences. In terms of operator approximating. complex kernel integrals lead high computational cost while simple kernel integrals lack accuracy. this work, we build an accurate and computationally efficient model for both forward and inverse problems by learning the operator in the learnable latent space.
3.2 Latent Neural Operator
The proposed model Latent Neural Operator (LNO) is applicable for both forward and inverse PDE problems. It consists of four modules: an embedding layer to lift the dimension of the input data, an encoder to transform the input data to a learnable latent space, a series self-attention layers for modeling operator in the latent space, and decoder to recover the latent representation to the real-world geometric space. The overall architecture of LNO shown in Figure
Overview
The geometric space the original space of the PDE input or output which contains No. or N samples, each with d-dimensional position coordinates (which may additionally
Embedding inputs

---

include a 1-dimensional time coordinate for time-dependent PDEs) and n-dimensional physical quantity values.
First, we embed the positions and physics quantity values of the input sequence in the geometric space to higher D-dimensional representations. This process is implemented by two MLPs, where the first only embeds the position information and the second embeds both the location and physics quantity information
( trumk-projector(ppo..
+00 E R g D
branch-projector (conctttcccoc , eral
Analogous to DeepONet [3], we name these MLPs trunk-prow mooje-rooeetor and wranch-projeece respec- tively. The embedding operation helps map the locations and physics quantities the embedding space where their relationships are easier to capture.
Removing the physies quantity values from the trunk-projector input grants our model the decoupling property. In this way, we can predict values at positions not included in the sampled input function from the latent space, thus achieving the decoupling of observation and prediction positions. Since the sampled input function always includes position-value pairs, using both position and value information in the branch projector will not affect the decoupling objective.
Encoding to the latent space We attempt to represent the input function using the representation tokens of hypothetical sampling positions [ (((k) E RDx1 which exist in the latent space, where M can be much smaller than the number Nin of samples of raw inputs.
We use the Physics- Croiss--rsss-AA Attent-nttention to model the transformation from the geometric space to the latent space. The inputs are query
[1(") (,) ((A))T, key X = [a ,()), (J"))
and value Y = [ g())) j()),... ÿf ) which corresponding to embedding of hypothetical locations in the target latent space, embedding of locations in the source real-world geometric space, and embedding of the concatenation of locations and physics quantities in the source real-world geometric space respectively. The output can be calculated through
z Phhysccs-rrosseee Prnsics-CrossAre Clyssis-Cross-AAteniii Attention H X
In contrast to FNO[14, where the latent space pre-defined as the frequency domain, we learn the latent space from data. Therefore, the embeddings of sample locations H queries in the cross-attention are also learnable parameters.
After encoding the input function into the latent space, the length of the sequence to be processed is reduced from Nin to M. With < Nin, extracting and converting the feature of the input function in the latent space will be much more efficient than in the real-world geometric space.
Learning operator in the latent space We model the operator of the forward/inverse PDE problem in the latent space to convert the feature of the input function to that of the output function, using stack of Transformer blocks with the self-attention mechanism
Z (Tyeerayerormm Z + z' with 2' = AAtennion(Tayerroomi (2))m)((())) 7
where E 21,2,... I.} is the index of Transformer block with I. being the total number of the blocks. We experiment with several implementations of attention mechanisms the ablation study and find the classical scaled dot-product attention [6] performs consistently well various tasks.
Decoding to the geometric space through Physics-Crowe Cross-Attention according to the query pos out of the output sampling locations and use another MLP to map the embeddings the values of the output function
Finally, we decode the output sequence from the latent sequence
p trank-projcctor(pos(..)
pb))
E
Seeicss-rross-Atte P. H 2-)
vat (I))
n6) E DxI
where P = P= (p pp()) .p (Nott)) and U = (, ( u (N. During the inferring
phase, the trunk projecto can accept inputs from any position, enabling LNO generalize the unseen region where the positions are not presented in the training phase.
5

---

PhCAA Encoder
PhCA Decoder
[N ma
[Nout P]
[No. M]
[Nout M]
X
Attention
Attention Projecter
softmax
Projecter
axis
[N D]
[M.D]
[M.
[Nouc
Linear Projection
Linear Projection
V
z
z
U
Figure 2: The working mechanism of Physics-Crone Cross--ttenttiii in encoder and decoder respectively.
3.3 Physics-Cross-Atennte
The Physics- Cross-rttennn (PhCA) is the core module in the aforementioned latent neural operator (LNO) model, acting the encoder and decoder which transform the data representation back-and- forth between the real-world geometric space and the latent space. Motivated by the work [22) that learns the operator mapping across domains using the cross-attention,
use the same cross-attention to model the transformation to/from the latent space but set the latent space learnable rather than pre-defined, e.g., triangular space or frequency domain [14]
In particular, the rows of the query H and key X matrices in (2) represent the embedded positions of samples in the target and source space respectively. A row of the value matrix Y is the embedding of both the position and physics quantity of a sample the source space. We design the latent space to be learned from data. Thus, the positions H of samples in the latent space should be set learnable parameters as well. Then, the cross-attention of the encoder in (2) is simplified as
2 softmax VD HW W X
YW, = softmax (W, x YW
where we merge the learnable matrices of query H, query weight W, and key weight W single matrix W1- 1.
Correspondingly, the cross- attes-att in the decoder has learnable key matrix that represents positions of samples in the latent space. Therefore, the cross- atossttttnn of the decoder in (3) is written as
U softmax JD PW w Z z'w
CestmaxxPP''' ) w'
(5)
In addition, utilizing the relationship of the encoder and decoder as mutually inverse transformations, we set W, W2 reduce the number of parameters, which also improves model performance experimentally. Practically, we generalize the linear projection W, i. W₂ with an MLP for further improvement, which is named attention-projeettr
shown in Figure 2
In the special case where the input only contains sample positions without physics quantities, our PhCA module is similar to the transformation of physics-aware token in Transolver [23] (only differing in normalization).
general cases when the inputs include both positions and physics quantities, the two approaches are different. Specifically, in Transolver, the key X in (4) and query P in (5) contain both positions and physics quantities information, whereas in PhCA, the key matrix X in the encoder and the query matrix P in the decoder only depend the positions of the input and output respectively. The physics quantities are used only to produce the value matrix V in attention mechanism. Such decoupling property of values and locations query and key enables LNO to predict values at different positions from input positions.
4 Experiments
We first evaluate the accuracy for the forward problems on the six popular public benchmarks and for the inverse problem using Burger's 'ss equation. Then, we measure the efficiency in terms of the number of model parameters, the GPU memory usage and the training time. Next, we assess the generalization of our model concerning the number N,, of observation samples and the number Nout of prediction positions. Finally, we establish a series of ablation studies including choices of different attention mechanisms, strategy of sharing weights and sampling numbers in the latent space. Model developed on the PyTorch and the PaddlePaddle deep learning framework. Additional experiments on scaling, sampling rate, and significance are presented in the appendix.

---

4.1 Accuracy for the forward problems
We conduct experiments for the forward problem on six benchmarks including Darcy and NS2d on regular grids (refer to section 5.2, 5.3 in [14] for details), and Airfoil, Elasticity, Plasticity and Pipe problems on irregular grids (refer section 4.1-4.5 4.5 in 44 for details).
Table shows a comprehensive comparison with previous works that reported their performance on the same benchmarks. Our LNO achieves the best accuracy four benchmarks: Darcy, NS2d, Elasticity and Pipe. On the Airfoil and Plasticity benchmarks, our LNO achieves performance close the SOTA with only half computational cost discussed Section 4.3 These results demonstrate the effectiveness of approximating the kernel integral operator in the latent space after transformation from the real-world geometric space through the PhCA module.
Table Prediction error on the six forward problems. The best result is in bold. "*" means that the results of the method are reproduced by ourselves. "/" means that the method not handle this benchmark. The column labeled D.C. indicates whether the observation positions and prediction positions are decoupled.
Relative L2(x10-2)
Model
D.C.
Darey
NS2d
Airfoil
Elasticity
Plasticity
Pipe
4
N
1.08
15.56
Geo-FNO
47
N
1.08
15.56
1.38
2.29
0.74
0.67
F-FNO" 45
N
0.75
11.51
0.60
1.85
0.27
0.68
U-FNO" 46
N
1.28
17.15
1.19
2.13
0.41
0.62
LSM(2
N
0.65
15.35
0.59
2.18
0.25
0.50
Galerkin 20
N
0.84
14.01
1.18
2,40
1.20
0.98
OFormer 29
Y
1.24
17.05
1.83
1.83
0.17
1.68
GNOT 21
Y
1.04
13.40
0.75
0.88
3.19
0.45
FactFormer
N 1.09
12.14
0.71
3.12
0.60
ONO 31
Y
0.76
11.95
0.61
1.18
0.48
0.52
Transolver"
N
0.58
8.79
0.47
0.62
0.12
0.31
LNO(Ours)
Y
0.49
8.45
0.51
0.52
0.29
0.26
4.2 Accuracy for the inverse problem
To demonstrate the flexibility of LNO, we design an inverse problem. Given the partially observed solution u(x), the aim is to recover the complete uxx) in larger domain. Specifically, we test on 1D Burgers' equation
@ ăr u(z = 0.01 - u(z,t) u(x, t), E [0, tE [0.
u(x: 0) GaussianProcess(0.. exp pla
sin (n|l( - r'1),, w(0,t) u1,
The ground-truth data is generated on a 128 x 128 grid with the periodic boundary condition. Initial conditions are sampled from the Gaussian process with the periodic length scaling factor 1 1
The objective of this inverse problem is to reconstruct the complete solution u(x) in the whole spatiotemporal domain (æ, () E [o, 1] [0, 1] based on sparsely random-sampled or fixed-sampled observation in the sub-spatiotemporal domain (z, () E [0, 1] x (0.25, 0.75), as illustrated in Figure in the appendix.
Instead of the naive approach which directly learns the mapping from partially observed samples in the subdomain the complete solution the whole domain, we propose two-stage strategy inspired by inpainting 47 48 and outpainting 49 [0). First, we train the model completer to interpolate the sparsely sampled points the subdomain [0, 1] [0.25, 0.75] (left sub-figure in Figure 6 to predict all densely and regularly sampled points in the same subdomain (middle sub-figure in Figure (). Then, we train the model as a propagator to extrapolate the results of the completer from the subdomain to the whole domain [0. 1] 00. 1] (right sub-figure in Figure Since the observation and the prediction samples are located in different positions, only the models with decoupling properties can be used the completer and propagator.
7

---

We compare the performance of our LNO with T3 [3] and 2N in two stages respec- tively. The results in Table Table Table iindcatee that i) in the first stage, LNO performs significantly better than DeepONet and GNOT at different random observation ratios; ii) in the second stage, LNO performs better when the complete solution in the subdomain approaches the ground truth, and the performance degrades as the reconstruction error of the complete solution in the subdomain increases. This is reasonable since we train the propagator based on the ground truth in the subdomain. When there is significant discrepancy between the ground truth and the prediction of the completer, the model faces challenging distribution shifts. Nevertheless, when LNO is used both as the completer and propagator, it achieves the most accurate results for solving this inverse problem.
We also investigate the impact of different temporal and spatial sampling intervals on the accuracy of LNO in the fixed grid observation situation. See appendix B for details.
Table 2:
Relative MAE of different completers in the subdomain in the 1st stage of the inverse problem with different settings of observation ratio.
Completer
Observation ratio
20%
10%
5%
0.5% 1 1%
DeepONet 13
2.51%
2.59%
2.82%
3.25%
4.82%
GNOT
1.12%
1.39%
1.62%
1.63%
2.56%
LNO(OurS)
0.60 %
0.74%
0.77% %
1.18% %
2.05%
Table 3: The reconstruction error of different propagators in the 2nd stage of the inverse problem. Propagators are trained to reconstruct the solution in the whole domain based the ground truth (G.T.) the subdomain and are tested using the output of different completers. Relative MAE of = 0 and = 1 is reported.
LNO
GNOT
DeepONet
Propagator Completer
G.T.
10%
1%
10%
1%
10%
1%
DeepONet 13
7.34%
8.01%
9.38%
9.09%
10.80%
11.14%
13.87%
21
5.45%
6.50%
8.07%
8.04 %
9.91 %
10.41 %
13.45 %
LNO(Ours)
3.73%
5.69 %
7.72%
9.03%
10.98%
13.11%
15.50%
Table 4: Comparison of efficiency between LNO and Transolver on the six forward problem bench- marks. Three metries are measured: the number of parameters, the cost of GPU memory and the cost of training time per epoch.
Metric
Model
Darcy
NS2d
Airfoil
Elasticity
Plasticity
Pipe
Transolver
1.91
5.33
1.91
1.91
1.91
1.91
Paras Count(M)
LNO
0.76
5.08
1.36
1.42
1.36
1.36
Transolver
17.11
17.17
4.49
1.48
18.41
5.94
Memory(GB)
LNO
5.75
7.58
2.47
1.39
7.16
2.89
Transolver
88.68
107.62
19.49
5,66
83.43
25.56
Time(s/epoch)
LNO
38.98
57.83
9.35
5.33
41.62
14.13
4.3 Efficiency
We demonstrate improvement the efficiency of our proposed LNO compared Transolver, which was considered the most efficient Transformer-based PDE solvers[23] Results in Table Abbleaiidiictee that our method has fewer parameters, lower memory consumption, and faster training speed. Roughly speaking, LNO reduces the parameter count by average of 30% and memory consumption by 50% compared to Transolver across all benchmarks, also with a 1.8x training acceleration.
The improvement of LNO stems from the idea of solving PDEs in the latent space. Given N input points in the real-world geometric space, both Transolver and LNO consist T. Transformer blocks, the Transolver, which applies Physics-Attention Attention in each block, has the time complexity

---

of O(LMN + LM²), where M is the number of slices. Our LNO, which applies Prysscs-Crosss Attention only the encoder and decoder, has the time complexity of O(MN -M*), where M is the sampling numbers in latent space.
4.4 Generalization
To validate the generalization of model concerning the number of discretization points in the input and output, we train the model in a uniform low-resolution scenario and perform zero-shot inference in other higher-resolution scenarios. Specifically, we downsample the Darey dataset with resolution of 421 x 421 to 241 x 241, 211 x 211, 141 x 141, 111,60 85 x 85, 61 x 61 and 43 x 43. Models are trained on the 43 43 dataset and tested on the other datasets. The comparison results of our model and other methods shown in Table 5 show that our model exhibits great generalization capability regarding the number of sampling points, consistently outperforming other methods accuracy on test sets with different number of discretization points from the training set.
Table 5: Prediction errors on Darcy with different resolutions. Models are trained on 43 43 and tested on higher resolutions. Relative L2 errors are reported. The best results are in bold.
Model
61 x 61
85 x 85
141 x 141
211 x 211
241 x 241
FNO
0.1164
0.1797
0.2679
0.3160
0.3631
ONO
0.0204
0.0259
0.0315
0.0349
0,0386
GNOT
0.0256
0.0275
0.0294
0.0305
0.0317
Transolver
0.0239
0.0240
0.0258
0.0279
0.0290
LNO(Ours)
0.0179
0.0192
0.0214
0.0229
0.0244
4.5 Ablation Study
Decoupling the values and locations as well as the observation positions and prediction positions not only guarantees flexibility but also improves accuracy. To study the effect of decoupling property, we replace the PhCA in LNO with Physics-Attention proposed in Transolver 223] In this way, the query and keys are calculated from the same embedding of both pos,, and val.n. of the input sequence. The pos out which is not present in the input sequence can not generate queries and keys. So the observation positions and prediction positions must be kept the same. We compare the performance of LNO with PhCA or Physics- Attention on the forward problem benchmarks, and the results in the first and last rows of Table verify the necessity of decoupling.
Necessity decoupling
Expressiveness of quadratic attention We study the impact on accuracy using different attention mechanisms in the latent space. We replace the scaled dot-product attention in LNO with Galerkin- type attention used in Galerkin Transformer 20]. efficient attention [18] used in GNOT[21), and Nystrom attention 19 used in ONO The comparison results in Table 6 show that quadratic attention, i.e., the scaled dot-product attention [16], more expressive than linear attention, which helps improve the accuracy. Many previous methods have employed different linear attentions (2)) 21 31 to reduce the computational costs at the expense of reducing the accuracy. Since the length of the latent sequence is much smaller than the inputs and outputs in real-world geometric space, applying quadratic attention latent space does not incur significant overhead.
Sharing parameters We explore the performance of LNO when W1 * W2 in 4 and 5). In this way, we replace the linear projection W, and W2 with two individual MLPs. Results of the comparison between LNO using shared MIPs and non-shared MLPs in the PhCA encoder and decoder are shown in T can be found that LNO performs better on most benchmarks (except Plasticity) under the shared situation.
Sampling numbers in latent space We investigate the impact of hypothetical sampling position number in the latent space on LNO for solving both the forward and inverse problems. Results are shown in Figure 3 the forward problems, as the sampling number M in the latent space increases, the performance of LNO gradually improves to a saturation level, followed by gentle decline on the benchmarks of Airfoil, Flasticity, Plasticity, and Pipe. However, on benchmarks of Darcy and NS2d, the performance of LNO continues to improve till M 512. This may be attributed to that

---

the former set of benchmarks containing only location information. while the latter set involves both location and physics quantity information, which requires more sampling positions to represent. In the inverse problem, the performance of the completer and propagator are similar to the trends in the forward problems.
Table 6: The ablation results using different attention implementations. P.A. stands for Physics- Attention, E.A. stands for Efficient Attention, N.A. stands for Nystrom Attention and G.A. stands for Galerkin-type type Attention.
Relative L2(x 10
Model
Darey
NS2d
Airfoil
Elasticity
Plasticity
Pipe
LNO with P.A.
0.53
22.65
0.74
0.88
0.49
0.40
LNO with E.A. [8
21
0.63
20.18
0.59
0.54
0,44
0.36
LNO with N.A.
0.67
23.03
0.63
0.79
0.30
0.48
LNO with G.A. 20
0.71
22.44
0.89
0.93
0.43
0.39
LNO
0.49
8.45
0.51
0.52
0.29
0.26
Table 7: The ablation results of strategy of sharing weights six forward problem benchmarks. Relative L2 is recorded. Smaller values represent better performance. The better result is in bold.
Relative 12(x10-2)
Model
Darcy
NS2d
Airfoil
Elasticity
Plasticity
Pipe
(NO(non-shared)
0.54
10.67
0.53
1.09
0.26
0.36
LNO
0.49
8.45
0.51
0.52
0.29
0.26
rL2(c-2)
rL2((e2) 0.85
rMAE(%)
25
1.10
55
Airfoll
DarcyFlow
Completer 1%
0.30
22
52
Elasticity
NS2d
Completer 10%
Plasticity
3.90
Propagator
Pipe
0.70
17
0.65
0.60
15
0.79
43
12
0.5
0.55
0.50
0.4
0.50
0.3
0.45
0.40
38
0.40
32
255
512
12
64
125
235
Sample Size in Latent Space
Sample Size in Latent Space
Sample Size in Latent Space
(a)
(b)
(c)
Figure Sampling numbers in latent space v.s. Accuracy. (a) Forward problems on Airfoil, Elasticity, Plasticity and Pipe. (b) Forward problems on Darcy and NS2d. (c) Inverse Problem.
Conclusions
The paper explores the feasibility of characterizing PDFs in learnable latent space. We present Physics-Cal Cross-Attension (PhCA) for encoding inputs into the latent space and for decoding outputs. We build the Latent Neural Operator (LNO) based PhCA and validate its flexibility, efficiency and accuracy for solving both forward and inverse problems. We leave the generalization ability and re-usability of PhCA across multiple types of PDEs as another open research topic.
Our work has also some limitations. For time- depeedenddd equations, we decode the states at intermedi- ate time steps into geometric space to compute the loss. Implementing additional training strategies may allow us to conduct all operations of the dynamic evolution process directly in the latent space, which potentially enhances efficiency. Also, in our work, no prior is imposed on the latent space. However, in the future, it is worth exploring whether physical priors, e.g. symmetry, can help gain further precision improvement.
10

---

References
[1] Jaideep Pathak, Shashank Subramanian, Peter Harrington, Sanjeev Raja, Ashesh Chattopadhyay, Morteza Mardani, Thorsten Kurth, David Hall, Zongyi Li. Kamyar Azizzadenesheli. et al. FourCastNet: global data-driven high-resolution weather model using adaptive Fourier neural operators. arXiv preprint urXiv: 11214, 2022.
[2] Zhenze Yang, Chi-Hua Yu, and Markus J Buehler. Deep learning model to predict complex stress and strain fields in hierarchical composites. Science Advances, 7(15)cabd77416, 2021.
[3] Yifan Mei. Yijie Zhang. Xueyu Zhu, and Rongxi Gou, Forward and inverse problems for eikonal equation based on DeepONet. arXiv preprint (XXx:2336600754 2023.
[4] Timothy Praditia, Matthias Karlbauer, Sebastian Otte, Sergey Oladyshkin, Martin V Butz, and Wolfgang Nowak. Learning groundwater contaminant diffusion-sorption processes with finite volume neural network. Water Resources Research, 58 (5(1)):2022WR0333444 2022.
[5] Junuthula Narasimha Reddy. An introduction the finite element method. New York, 27:14, 1993.
[6] Zhongkai Hao, Songming Liu, Yichi Zhang, Chengyang Ying, Yao Feng, Hang Su, and Jun Zhu. Physics-informed machine learning: a survey on problems, methods and applications. arXiv preprint 10Xiv:22111080644 2022.
[7] Maziar Raissi, Paris Perdikaris, and George E Karniadakis. Physics-informed neural networks: deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378:686- 2019.
[8] Matthias Karlbauer, Timothy Praditia, Sebastian Otte, Sergey Oladyshkin, Wolfgang Nowak, and Martin Butz. Composing partial differential equations with physics-aware neural networks. In Proceedings of the International Conference on Machine Learning (ICML), 2022.
[9] Craig R Gin, Daniel E Shea, Steven L Brunton, and Nathan Kutz. DeepGreen: deep learning of Green's functions for nonlinear boundary value problems. Scientific Reports, 11(1):21614, 2021.
[10] Chengping Rao, Pu Ren, Qi Wang, Oral Buyukozturk, Hao Sun, and Yang Liu. Encoding physics to learn reaction-diffusion processes. Nature Machine Intelligence, 5(7):765-779, 2023.
[11] Sifan Wang, Hanwen Wang, and Paris Perdikaris. Learning the solution operator of parametric partial differential equations with physics-infomen DeepONets. Science Advances, 7(40))aa 2021.
[12] Tianping Chen and Hong Chen. Universal approximation nonlinear operators by neural networks with arbitrary activation functions and its application to dynamical systems. IEEE Transactions on Neural Networks, ((4):911-917, 1995.
[13] Lu Lu, Pengzhan Jin, and George Em Kamiadakis. DeepONct: lcarning nonlinear operators for identi- fying differential equations based on the universal approximation theorem of operators. arXiv preprint arXiv:1910.03:
2019.
[14] Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. arXiv preprint 100.2000,,8895,, 2020.
[15] Nikola Kovachki. Zongyi Li, Burigede Liu, Kamyar Azizzadenesheli, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Neural operator: learning maps between function spaces with applications to PDEs. Journal of Machine Learning Research, 24(89);1 97, 2023.
[16] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
[17] Georgios Kissas, Jacob H. Seidman, Leonardo Ferreira Guilhoto, Victor M Preciado, George J Pappas, and Paris Perdikaris. Learning operators with coupled attention. Journal of Machine Learning Research, 23(215):1663, 2022.
[18] Zhuoran Shen, Mingyuan Zhang, Haiyu Zhao, Shuai Yi, and Hongsheng Li. Efficient attention: attention with linear complexities. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2021.
11

---

[19] Yunyang Xiong, Zhanpeng Zeng, Rudrasis Chakraborty. Mingxing Tan, Glenn Fung, Yin Li, and Vikas Singh. Nyströmfomner: Nystrrmmbasee algorithm for approximating self-attention. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI). 2021.
[20] Shuhao Cao. Choose a transformer: Fourier Galerkin. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
[21] Zhongkai Hao, Zhengyi Wang. Hang Su, Chengyang Ying. Yinpeng Dong. Songming Liu, Ze Cheng, Jian Song, and Jun Zhu. GNOT: general neural operator Transfomer for operator learning. Proceedings of the International Conference on Machine Learning (ICML), 2023.
[22] Haixu Wu, Tengge Hu, Huakun Luo, Jianmin Wang, and Mingsheng Long. Solving high-dimensional PDEs with latent spectral models. arXiv preprint arXiv:2301 22223311.26600 /2664, 2023.
[23] Haixu Wu, Huakun Luo, Haowen Wang, Jianmin Wang, Mingsheng Long. Transolver: fast Transformer solver for PDEs on general geometries. arXiv preprint arXiv:440..02366. 2024.
[24] Benedikt Alkin, Andreas Fürst, Simon Schmid, Lukas Gruber, Markus Holzleitmer, and Johannes Brand- stetter. Universal physics Transformers. arXiv preprint arXiv:2402. 12365, 2024.
[25] Valerii lakovlev, Markus Heinonen, and Harri Lähdesmäki. Learning space-time continuous latent neural PDEs from partially observed states. In Advances in Neural Information Processing Systems (NeurIPS), 2024.
[26] Tailin Wu, Takashi Maruyama, and Jure Leskovee. Learning to accelerate partial differential equations via latent global evolution. In Advances in Neural Information Processing Systems (NeurlPS), 2022.
[27] Somdatta Goswami, Katiana Kontolati, Michael Shields, and George Em Karniadakis. Deep transfer operator learning for partial differential equations under conditional shift. Nature Machine Intelligence, 4(12): 155-1115,,, 2022.
[28] Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar Neural operator: graph kemel network for partial differential equations. arXiv preprint arXiv:2003.00 033485 2020.
[29] Zijie Li. Kazem Meidani, and Amir Barati Farimani. Transformer for partial differential equations' operator learning. arXiv preprint arXiv:2205.1 13671, 2022.
[30] Zijie Li, Dule Shu, and Amir Barati Farimani. Scalable Transformer for PDE surrogate modeling. In Advances in Neural Information Processing Systems (NeurIPS), 2024.
[31] Zipeng Xiao, Zhongkai Hao, Bokai Lin. Zhijie Deng, and Hang Su. Improved operator learning by orthogonal attention. arXiv preprint arXiv:2310. 12487, 2023.
[32] Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. The graph neural network model. IEEE Transactions Neural Networks, 20(1):61 80, 2008
[33] Ruchi Guo, Shuhao Cao, and Long Chen. Transformer meets boundary value inverse problems. In Proceedings of the International Conference Learning Representations (ICLR), 2022.
[34] Guanchao Liu, Lei Zhang, Qingzhen Wang, and Jianhua Xu. Data-driven seismic prestack velocity inversion via combining residual network with convolutional autoencoder. Journal of Applied Geophysics, 207:104846, 2022.
[35] Bin Liu, Yuxiao Ren, Hanchi Liu, Hui Xu, Zhengfang Wang, Anthony G Cohn, and Peng Jiang. Get:nyNet: deep learning- based ground-penetrating radar data inversion for tunnel linings. IEEE Transactions on Geoscience and Remote Sensing, 50(10):8305-8325 2021.
[36] Zhongping Zhang, Yue Wu, Zheng Zhou, and Youzuo Lin. VelocityGAN: subsurface velocity image estimation using conditional adversarial networks. Proceedings of the IEEE Winter Conference on Applicutions of Computer Vision (WACV), 2019.
[37] Juri Ranieri, Amina Chebira, Yue M Lu, and Martin Vetterli. Sampling and reconstructing diffuusin fields with localized sources. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2011.
[38] Hassan Bararnia and Mehdi Esmaeilpour. On the application of physics informed neural networks (PINN) to solve boundary layer thermal-flud problems. International Communications in Heat and Mass Transfer. 132(8): 105890, 2022.
12

---

[39] Shengze Cai. Zhicheng Wang, Chryssostomos Chryssostomidiis, and George Em Karniadakis. Heat transfer prediction with unknown thermal boundary conditions using Proceedings of the Fluids Engineering Division Summer Meeting (FEDSM). 2020.
neural networks. In
[40] Hongping Wang, Yi Liu, and Shizhao Wang. Dense velocity reconstruction from particle image velocime- tryparticle tracking velocimetry using physics-informed neural network. Physics of Fluids, 34(1)001 2022.
[41] Roberto Molinaro, Yunan Yang. Bjöm Engquist, and Siddhartha Mishra. Neural inverse operators for solving PDE inverse problems, arXiv preprint 1xXi::3301.11161 2023.
[42] Coşku Can Horuz, Matthias Karlbauer, Timothy Praditia, Martin Butz, Sergey Oladyshkin, Wolfgang Nowak, and Sebastian Otte. Infering boundary conditions in finite volume neural networks. In Proceedings of the International Conference on Artificial Neural Networks (ICANN), 2022.
[43] Qingqing Zhao, David B Lindell, and Gordon Wetzstein. Learning to solve pde-constrained inverse problems with graph networks. In Proceedings of the International Conference on Machine Learning (ICML.), 2022.
[44] Zongyi Li, Daniel Zhengyu Huang. Burigede Liu, and Anima Anandkumar. Fourier neural operator with learned deformations for PDEs general geometries. Journal of Machine Learning Research, 24(388):1 22, 2023.
[45] Alasdair Tran, Alexander Mathews, Lexing Xie, and Cheng Soon Ong. Factorized Fourier neural operators. arXiv preprint arXiv:2111. 13802, 2021.
[46] Gege Wen, Zongyi Li, Kamyar Azizzadenesheli, Anima Anandkumar, and Sally M Benson. U-FNO an enhanced Fourier neural operator- Resources, 163:104180 2022.
gpesetor-based based deep-learning model for multiphase flow, Advances Water
[47] Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei Efros. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
[48] Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas Huang. Generative image inpainting with contextual attention. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
[49] Zongxin Yang, Jian Dong, Ping Liu, Yi Yang, and Shuicheng Yan. Very long natural scenery image prediction by outpainting. In Proceedings of the IEFF/CVF International Conference on Computer Vision (ICCV), 2019.
[50] Mark Sabini and Gili Rusak. Painting outside the box: Image outpainting with gans. arXiv preprint /tti::/808,088400 2018
[51] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In Proceedings of the Interna- tional Conference on Learning Representations (ICLR), 2018
[52] Leslie N Smith and Nicholay Topin. coperrconveeeecc:: very fast training of neural networks using large learning rates. In Artificial Intelligence and Machine Learning for Multi-Doman Operations Applications, 2019.
13

---

A Scaling
We conduct scaling experiments to investigate the impact of model depth (number of stacked Transformer blocks) and model width (dimension of tokens in Transformer blocks) on the performance of LNO.
As shown in Figure 4 with model depth increasing, the performance of LNO initially improves and then declines, reaching its optimal performance at 4 or 8. This suggests that deeper LNO encounters optimization challenges in solving PDEs in the latent space. Simply stacking more Transformer blocks does not necessarily lead to better accuracy.
As shown in Figure although the performance of LNO continues benefiting from increasing model width, it gradually exhibits a saturation trend when the dimension of tokens reaches as large as 256.
Darty
N524
Artol
L2
3080
Depth Eastity
Depch Fastidiy
Depth
Piec
5000
5030
3,700
Reaiive
Relitbve
3000
Dupth
Dapch
Desch
Figure 4: The influence of model depth on the performance of LNO across six forward problem benchmarks.
A000
Derry
NS24
Artoil
AX0A
Reaitbe L2
Dastiiity
Pliiiitty
Pioe
3000
Neattee 12 67075
4.00
Litir L2 Storr
5050
1000
1200
Witth
Figure 5: The influence of model width on the performance of LNO across six forward problem benchmarks.
14

---

B Observation Ratio and Locations for the Inverse PDE Task
We investigate the impact of different temporal and spatial sampling intervals the accuracy of LNO when solving inverse problem in the fixed observation situation. As shown by Figure [ can be found that i) the inverse problem spatial sampling intervals are both as large as 16 (approximately only 0.4% observation ratio); ii) the increase of the spatial sampling interval significantly compromises the solution accuracy compared to that of the temporal sampling interval; iii) spatial undersampling can hardly be compensated for through temporal oversampling.
still be solved accurately
when the temporal and
Random Observation
Complete Solution Sub Domain
Complete Solution Whole Domain
Sub Domain
10
10
10
0.6
006
0.
0.
56-
0.2
0.4
044
08
0.0
0.6
0.2
06
144
Figure 6: Visualization of Burgers' equation with different observation situations in different regions. We propose two-stage strategy. First, we interpolate the solution in the subdomain. Then, we extrapolate from the subdomain to the whole domain.
Fixed Observation (Intervals=4)
Fixed Observation (intervais=8)
Fixed Observation (Intervais=16)
10
LU
10
00.
0.6
08-
0.
-00
02
0.2
0.2
0.0
0.4
0.8
0.0
0.6
3.2
02
0.6
3.
Figure 7: Visualization of Burgers' equation in the fixed observation situation with different temporal and spatial sampling intervals.
Relative MAE (%6
Relative MAE (%)
Temporal Sampling Intervals
8-
Temporal Sampling Intervals
3
40
0.22
0.25
0.74
4.3
14
50
5.1
7.5
16
22
30
8.
3.29
0.1
5.2
20
10
Spatial Sampling Intervals
Spatial Sampling Intervals
(a)
(b)
Figure 8: Accuracy of solving the inverse problem at different temporal and spatial sampling intervals. (a) Accuracy of LNO as completer. (b) Accuracy LNO as propagator based on the results of the corresponding completer.
15

---

C Significance
We conduct repeated experiments on both the forward and inverse problems. initializing the weights of our model and the reproduced models with different random seeds each time. We report the mean and standard deviation of these experiments to demonstrate the significance of our experimental results, as shown in Figure 8 and Figure
Table 8:
The mean and standard deviations (computed based on 5 independent trials) of different models in solving the forward problem.
Relative L2(x10-3)
Model
Darcy
NS2d
Airfoil
Elasticity
Plasticity
Pipe
F-FNO 45
0.75(=0.01)
11.51(#:0255)
0.60(+0.033)
1.85(+00.03)
0.27(+0.02)
0.68(+-0022
U-FNO 46
1.28(=0.02) 17.15(++029)
1.19(+0.05)
2.13(+-0.33
0.41(+0.03)
0.62(+0.02)
GNOT 21
1.04(=0.02)
13.40(10.28)
0.75(10.02)
0.88(10.03)
3.19(=0.07)
0,45(=0.02)
Transolver LNO(Ours)
23
0.58(+0.01) 8.79(+0.17)
0.47(+0.02)
0.62(+0.02)
0.12(+0.02)
0.31(+0.01)
0.49(10.01)
8.45(11.22)
0.51(10.05)
0.52(10.03)
0.29(10.03)
0.26(10.03)
Table 9: The mean and standard deviations (computed based on 3 independent trials) of different models in solving the inverse problem. Relative MAEs are expressed percentages. The second to the second-to-Bee columns correspond to the situation where models act as completers under different observation rates, while the last column corresponds the situation where models act as propagators.
Model
20%
10%
5%
1%
0.5%
propagator
DeepONet 13
2.37(=0.18)
2.52(10.04)
2.77(11011)
3.18(10.03)
7.42(10.37) 4.70(10.31)
21
1.23(=0.07)
1.33(=+0.06)
1.63(+0.04)
1.75(+0.02)
2.53(+0.08)
5,43(=0.10)
LNO(OurS)
0.60(-0.03)
0.73(11.111)
0.78(10.04)
1.26(10.09)
1.97(10.04)
3.46(10.21)
D Full Implementation Details
The full implementation details LNO in both forward and inverse problems are shown in Table T0 All our experiments conducted single two NVIDIA RTX 3090 GPUs.
For the forward problems, we train models using relative L2 error for 500 epochs. AdamW optimizer [5] and OneCycleLr scheduler[52] are applied with initial learning rate at 10-3, We reproduce the results Transolver following implementations in appendix B.3 (23 and also excerpt the results of other methods from Table 2 in [23] The data splits are the same in Appendix B. in (23].
For the inverse problem, we train the models using mean squared error and StepLr scheduler. For DeepONet we use 4 Transformer blocks with 228-dimensional tokens as the branch net and fully-com collecconeeted layers as the trunk net. For GNOT[11, we set the number of Transformer blocks as the dimension of tokens 96, and the number of experts as We use 4096 samples for training, 128 samples for validating and 128 samples for testing.
Table 10: The hyperparameters and training configuration of LNO in solving the forward and inverse
problems in Section 4.1 and 4.2
Benchmark
Layers
Dim
Size
Heads
Batch
Epuch
Loss
Optimizer
Scheduler
Darcy
4
128
NS2d
8
256
Airfoil
8
128
256
500
rL.2
AdamW
OneCycleL
Elasticity
8
128
Plasticity
8
128
Pipe
8
128
Completer
256
500
MSF
Adam
Stepl
Propagator
16

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the
paper's contributions and scope?
Answer: [Yes]
Justification: We have elearly claimed the main contributions and scope of our work in the abstract and introduction, which be reflected by the demonstration of methods in Section and experimental results in Section
3.2
3.3
4.5
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We have discussed the limitations of our proposed LNO and potential ap- proaches overcome them in Section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
17

---

Answer:
[NA]
Justification:
Our work does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide detailed descriptions of our methods in Section 3.2 and Section 3.3 and full implementation details regarding the experiments in Appendix D Additionally, we also provide GitHub repository which contains the source code and running instructions.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
18

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We have provided our source code along with detailed instructions the runtime environment, data access, data processing, and experiment conducting in our GitHub repository.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
mublic/guides PoodeSubmissiooooliiee
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips
(odeSubmissionn iwribbmmssiooooool.
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: We have provided the full implementation details in Appendix D including data splits, hyperparameters, optimizer, scheduler, epoch, etc. Also, the ablation studies in Section 4.5 and scaling experiments in Appendix have shown the reasonableness of our choice for the hyperparameters.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We have reported the experiment statistical significance in Appendix shown in Table and Table
C
as
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
19

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We have provided comprehensive summary of computational resources required for all the experiments in Appendix D along with detailed records of memory, execution time, etc., in Section 4.3
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: We have reviewed the NeurlPS Code of Ethics carefully and made sure research conform with every respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our work is foundational research about solving forward and inverse problems PDEs, which is not tied to particular applications deployments. And there no direct path to any negative applications.
20

---

Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considcrations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning conrectly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work primarily focuses solving the forward and inverse problems of PDEs, which does not pose such risks of misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have utilized publicly available datasets provided in [14] and 44 as
mentioned in Section 4.1 The URLs the assets are also provided our GitHub repository.
Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
21

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thoode com comddatasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We have provided the source code of our proposed LNO in our GitHub repository. Detailed documentation also included explain how code.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our work does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
22

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
23


Reviews:
[{"review_id": "MhFesppGc2", "reviewer": "Reviewer_WmcY", "summary": "The paper presents the Latent Neural Operator (LNO), a novel approach for solving forward and inverse partial differential equations (PDEs) by operating in a latent space. The LNO introduces a Physics-Cross-Attention (PhCA) module for transforming data from the geometric space to a learnable latent space and demonstrates improved prediction accuracy and computational efficiency, reducing GPU memory usage by 50%, accelerating training by 1.8 times and achieving state-of-the-art accuracy on several benchmarks.", "questions": "1. It is a good idea to separate the position and the corresponding value. Still, for the physics-cross-attention proposed in the paper, only the position is used to generate the attention map. Will this operation cause a lack of information? Their correlations may not be the same for the same points in different initial fields.\n2. Can all operations be done in the latent space? The decoder decodes the latent information only when it needs to generate an output and is not involved in the autoregressive process.\n3. Does the model in Table 4 share the same parameters as the one in Table 1? It needs to be a fair comparison.\n4. I think the term decouple is not very rigorous because, in a branch projector, the positions and corresponding values are used simultaneously, so they are not completely separated.", "limitations": "Lack of real-world benchmarks and some theoretical proofs.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. **Clean model design**: The model design looks very clean, achieving the separation of positions and corresponding values so that they can be input into different model modules.\n2. **Excellent result**: The proposed model surpasses nearly all models currently in this field, achieving SOTA.", "weaknesses": "1. **Lack of theoretical proof**: Although the results are fantastic, can a set of theories support the current structure design?\n2. **Lack of real-world data examination:** Many 3D datasets exist in this area, such as smoke and turbulence[1]. Adding these benchmarks would strengthen the model's efficacy claim. Industrial car simulation data can also be used to test the proposed model's performance in this paper[2].\n\n[1] Li, Z., Shu, D., & Barati Farimani, A. (2024). Scalable transformer for pde surrogate modeling. *Advances in Neural Information Processing Systems*, *36*.\n\n[2] Wu, H., Luo, H., Wang, H., Wang, J., & Long, M. (2024). Transolver: A fast transformer solver for pdes on general geometries. *arXiv preprint arXiv:2402.02366*.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Reviewer pointed out that our work lacks theoretical proof and expected us to provide a set of theoretical support for the current structure design.** \n\nOur current structure design is mainly inspired by the kernel integral operator with position-only kernel function. Please refer to the \"Theoretical Justification and Motivation\" part of our author rebuttal for more explanation.\n\n**Q2: Reviewer pointed out that our model lacks real-world data examination and expected us to add benchmarks in [28] to strengthen our model's efficacy claim or add car simulation experiment in [23] to test our model's performance.**\n\nAs requested, we provide preliminary experimental results of our model on the car simulation dataset in Table 1 of author rebuttal attachment, and compare the precision with other approaches according to Table 3 of [23].\n\nTraining Transolver requires 6.6 hours while training our model only takes 3.8 hours. In terms of precision, our model is second to Transolver. Due to the limited time constraint, we did not fine-tune hyper-parameters, and can expect further improvement in the future. Anyway, the current results indicate that our model can also achieve good performance on real-world data. \n\nThe smoke and turbulence datasets in [28] have large scale and the experiments involve different temporal training strategies. We run out of time before the rebuttal deadline. We will report both these experiments in the final version of our paper. \n\n**Q3: Reviewer asked will the separation operation of position and the corresponding value causes a lack of information since their correlations may not be the same for the same points in different initial fields.**\n\nPractically, we have established the experiment in our ablation study and showed that the solution precision of the strategy using the concatenation of position and value (the first row of Table 5) is not as good as only using position (last row of Table 5). \n\nTheoretically, since the branch projecter contains position-value pair, the trunk projector, which only contains position, can still match right value via position information. Please refer to the \"Theoretical Justification and Motivation\" part of our author rebuttal for more explanation from a theoretical perspective.\n\n**Q4: Reviewer asked whether all operations can be done in the latent space so that the decoder decodes the latent information only when it needs to generate an output and is not involved in the autoregressive process.**\n\nThis suggestion is very valuable and inspiring for future research. \n\nOur current method primarily focuses on the mapping between functions (such as from equation coefficients to solution in steady-state PDE). Our encoder and decoder are used only at the start and end of the model to establish the connection between the real geometric space and the latent space. For time-dependent PDE, we employ the same straightforward approach as [14], treating the prediction at each time step as a function mapping problem, and thus decoding the latent information at each time step.\n\nIt is entirely feasible to continuously auto-regress from the initial time step to the final time step within the latent space without decoding the intermediate information. This requires designing specific mechanisms such as the latent march in [28] to ensure the solution precision for our future research.\n\n**Q5: Reviewer asked whether the model in Table 4 shares the same parameters as the one in Table 1 to ensure a fair comparison.**\n\nYes, the model parameters in Table 1 and Table 4 are exactly the same. These comparison results are fair and can be reproduced using the code we have provided in the supplementary material.\n\n**Q6: Reviewer thought the term \"decouple\" is not very rigorous because the positions and corresponding values are used simultaneously and not completely separated in the branch projector.**\n\nThe term \"decouple\" majorly means that we remove the value from the trunk projector. In this way, we can predict values at positions not included in the sampled input function from the latent space, thus achieving the decoupling of observation and prediction positions. \n\nWe admit that branch projector contains both position and values, which will be clarified in the revised manuscript. On the other hand, since the sampled input function always includes position-value pairs, using both position and value information in branch projector is reasonable and does not affect our decoupling goal.\n\nPlease also refer to the \"Theoretical Justification and Motivation\" part of our author rebuttal for more explanation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1tK28NTGLK", "reviewer": "Reviewer_RkUb", "summary": "The paper introduces Latent Neural Operator for solving forward and inverse problems in latent space. It uses a Physics-Cross-Attention to encode the input to the latent space, uses self-attention to evolve the state in the latent space, and uses another Physics-Cross-Attention to decode back to the real-world space. Experiments in forward and inverse problems demonstrates the effectiveness of the method.", "questions": "1. Why does in Table 1, FNO and GeoFNO gets the exact same result in Darcy and NS2d? \n\n2. Why does F-FNO achieves worse performance compared to FNO in NS2d? In the original paper, it achieves much better performance than FNO in NS and etc.", "limitations": "The paper can benefit from a more in-depth discussion on its limitations.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper addresses the important problem of learning PDEs and improving its efficiency. The paper is mostly written clearly.", "weaknesses": "1. In terms of novelty, the paper is somewhat limited. The usage of cross-attention to embed the input into latent tokens is much like the one in LSM [1], with the main difference being that the paper uses self-attention in latent evolution. In addition, the main modules of the paper are all based on attention, which is a standard neural architecture. \n\n2. The paper lacks comparisons to important past works in latent space, including forward problems of [2,3], and inverse problem of [4].\n\n3. There are some questions that needs to be resolved in the Questions section.\n\n\nReferences:\n\n[1] Wu, Haixu, et al. \"Solving high-dimensional pdes with latent spectral models.\" ICML 2023.\n\n[2] Iakovlev, Valerii, Markus Heinonen, and Harri Lähdesmäki. \"Learning Space-Time Continuous Latent Neural PDEs from Partially Observed States.\" Advances in Neural Information Processing Systems 36 (2023).\n\n[3] Wu, Tailin, Takashi Maruyama, and Jure Leskovec. \"Learning to accelerate partial differential equations via latent global evolution.\" Advances in Neural Information Processing Systems 35 (2022): 2240-2253.\n\n[4] Zhao, Qingqing, David B. Lindell, and Gordon Wetzstein. \"Learning to solve pde-constrained inverse problems with graph networks.\" ICML 2022", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Reviewer thought our paper is somewhat limited in terms of novelty because the usage of cross-attention to embed the input into latent tokens is much like that in [22] and our main modules are based on attention which is a standard neural architecture.**\n\nWe have discussed the difference between LSM[22] and our method in Section 2.2 of our paper. LSM manually constructs orthogonal triangular bases while we employ attention to find learnable bases (as explained in [20]). \n\nAdditionally, our Physics-Cross-Attention provides an encoding and decoding mechanism that decouples observation and prediction positions to allow the prediction on positions not included in observation, thus offering greater flexibility.\n\nPlease also refer to the \"Theoretical Justification and Motivation\" part and \"Novelty\" part of our author rebuttal for more explanation and clarification.\n\n**Q2: Reviewer thought our paper lacks comparisons to important past works in latent space, including forward problems of [r.1][r.2], and inverse problem of [r.3].**\n\nThese papers are excellent works, but they have different focuses and settings from our work.  We will add discussion about them in our final version. \n\nFor [r.3], it mainly focuses on solving initial conditions. The generator generates estimated initial condition based on the latent code, and the GNN simulates temporal evolution from the estimated initial condition. The initial condition is obtained through the reverse optimization of the loss between the temporal evolution result and the ground truth. In our inverse problem experiments, we use partial observation as input and compute the initial condition as output, solving the inverse problem through a forward process. So the method in [r.3] is also different from ours in terms of methodology. \n\nFor [r.2], it mainly focuses on solving time-dependent PDEs. It has components primarily designed for temporal evolution such as multi-step. Their method for solving inverse problem is also mainly based on temporal evolution and inverse optimization. The experimental result in the last row and last column in Table 2 of [r.2] is obtained under the same setting as our experiment on the Navier-Stokes equation. The relative L2 error of [r.2] is $0.1862$, whereas ours is $0.0845$. \n\nFor [r.1], it mainly focuses on solving time-dependent PDEs. It has components designed for temporal evolution such as multi-shooting. In this method, the spatial state is completed through interpolation method and the temporal evolution is predicted by combining the collocation method and method of lines under a probabilistic framework. Our model directly learns implicit constraints between spatiotemporal positions and the solution based on neural operator with attention. Thus, the method in [r.1] is different from ours in terms of methodology. \n\nOur experiments in the paper mainly compare our method with other neural operator methods, intending to demonstrate that our method achieves good precision, efficiency, and flexibility across a wide range of scenarios, which aim to create a general neural network-based PDE solver. Therefore, we have not delved deeply into specific techniques for scenarios such as time evaluation, initial condition or equation coefficient generation, and reverse optimization.\n\n[r.1] Learning space-time continuous latent neural PDEs from partially observed states. In Advances in Neural Information Processing Systems (NeurIPS), 2023.\n\n[r.2] Learning to accelerate partial differential equations via latent global evolution. In Advances in Neural Information Processing Systems (NeurIPS), 2022.\n\n[r.3] Learning to solve PDE-constrained inverse problems with graph networks. In Proceedings of the International Conference\non Machine Learning (ICML), 2022.\n\n**Q3: Reviewer asked why FNO and Geo-FNO get the exact same results on Darcy and NS2d in Table 1.**\n\nThe main contribution of Geo-FNO[41] lies in learning to map irregular grids to regular grids. The data of Darcy and NS2d problems is on the regular grids. In such scenarios, Geo-FNO is equivalent to FNO, and their experimental results are the same. In [21][22][23][29], their reported experimental results of FNO and Geo-FNO on datasets with regular grids are also always the same.\n\n**Q4: Reviewer asked why F-FNO achieves worse performance compared to FNO on NS2d in our paper and pointed out that F-FNO achieves much better performance than FNO on NS In the original paper.**\n\nThe setting for F-FNO in the original paper [43] is different from ours. The experimental results for F-FNO in Table 1 of our paper are sourced from Table 2 of [23]. \n\nWe then conduct experiments on our forward problem dataset using the open source code provided by the original authors. The re-conducted experimental results are consistent with those in Table 1 of [29] and the relative L2 error on Navier-Stokes equation is $11.51\\%$. \n\nWe will make corrections to the data in Table 1 of our paper, and reproduce all experimental results based on the official code of corresponding methods as well.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2EbcJU9fNc", "reviewer": "Reviewer_ny2e", "summary": "This paper suggests a neural operator architecture for learning solutions of forward and inverse PDE problems using a sequence of transformer layers. The novelty of the architecture is that after a carefully designed initial transformation step, named “Physics-Cross-Attention”, the inputs to subsequent transformer blocks have dimensions completely independent of the number or dimension of the samples of the input function. This design is meant to allow for efficient processing of data at very fine resolutions. The paper evaluates the efficacy of their method on standard PDE forward problems discretized on regular and irregular grids; they report superior performance compared to 11 other neural operator methods. The paper also considers two inverse problems dealing with Burgers' equation. One problem is an interpolation of randomly-sampled data, and another is extrapolation to recover initial conditions given sparsely-sampled data.", "questions": "Questions about the introduction:\n - The introduction alludes to the connection between the attention mechanism and kernel integral operators. Can the Latent Neural Operator (LNO) architecture be interpreted as a new way to parameterize kernel integral operators? If so, is the new parameterization motivated by the physics or numerics of the problems considered? I believe this type of discussion would better motivate the architecture choices made, and the paper would have more impact if the architecture choices were justified in this way. \n- The introduction discusses the improved computational complexity of this method over other methods. Presumably, there are specific problems that the LNO method could solve but are intractable for other methods. Could the authors give examples of such problems? This could help to better motivate the paper.\n\nQuestions about the architecture and notation:\n - What are the input and output dimensions of the branch and trunk projectors? This confusion is caused by conflicting notation. In section 3.1, $x$ is a spatial location, but in section 3.2, $x$ is the output of the trunk projector. Similarly, $D$ is the spatial domain in section 3.1 but is a shape hyperparameter in Figure 1.\n - In Figure 2, what is the white box labeled V?\n - What is the intuition guiding the softmax being applied across different dimensions in Figure 2?\n - The figures make it seem like the \"Trunk Projector\" is separate from the \"Attention Projector\", but these are both MLPs, and they are always used in sequence. Why make the distinction between these two networks? Could one think about them as one single MLP in the \"Trunk Projector\" module?\n\nQuestions about the experiments:\n - In Table 1, what is the meaning of the two groups of comparison models separated by a horizontal line?\n - In Table 1, what is the randomness in the 5 independent trials? Different train/test splits? Different initializations of the model weights?\n - In Section 4.1, for each benchmark task, what is different between the train and test set?\n - In Section 4.2, what exactly are the inputs and outputs of the model for the completion experiment? Similarly, what exactly are the inputs and outputs for the propagation experiment? Are different initial conditions used?\n - In Figure 6, it appears like the solution of Burgers’ equation gets smoother as it progresses through time; Burgers' equation is known to form shocks as waves propagate through time. If this plot is correct, can the authors comment on why there are no shocks forming in this example?", "limitations": "The authors identify one limitation, that the model does not scale well to deeper architectures (i.e. more transformer blocks), in the NeurIPS Paper Checklist, but I can not find this limitation discussed in the text or appendix.", "rating": 7, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "Overall, the paper is relatively strong in terms of originality, quality, and significance. I believe this is a technically strong paper that requires editing to become a strong submission. I will list individual strengths below:\n - The proposed “Physics-Cross-Attention” module is a simple innovation of the popular cross-attention mechanism. I think it is likely others will build off this idea due to its easy implementation.\n - The experimental results on the forward PDE problems are impressive. When compared to 11 other methods, this method outperforms all others on 4 out of the 6 problems considered.\n - The solution method for the inverse problem is interesting and novel. The solution method breaks down the problem into one of first interpolating randomly-located samples to a regular grid and then extrapolating the reconstruction on a regular grid to earlier and later time points. If this is a novel solution method, I think it is valid to claim this as another contribution, and if this is inspired by other works, readers would be interested to know where this solution method comes from.", "weaknesses": "The clarity of the paper could be greatly improved. I will list individual weaknesses below:\n - The paper claims to solve PDEs in a novel way by transforming from a \"geometric space\" to a \"latent space\" and solving the PDE in the \"latent space\". These terms are never defined, which makes it difficult to distinguish this paper's innovation from past work. The “Physics-Cross Attention” module transforms an input of shape $(N_{in}, d+1)$, where $N_{in}$ is the number of sample points and $d$ is the dimension of the spatial variable, to an array of shape $(M, D)$, where both $M$ and $D$ are hyperparameters completely divorced from $N_{in}$ or $d$. However, other neural operators have an encoding step that transforms inputs of shape $(N_{in}, d+1)$ to shape $(N_{in}, p)$. In this case, one could say the samples are projected to some latent space in $\\mathbb{R}^p$. Clearly defining the meaning of “latent space” considered in this paper would better define the novelty of this work.\n - The paper claims that the hyperparameters $M$ and $D$ are independent of the number of discretization points in the input $N_{in}$ and the number of discretization points in the output $N_{out}$. This is not supported by the experiments, which do not vary $N_{in}$ or  $N_{out}$ at test time. Without an evaluation of the model’s robustness to varying $N_{in}$ or $N_{out}$ at test time, it could be the case that $M$ and $D$ were tuned to have good performance on datasets with the specific settings of $N_{in}$ and $N_{out}$.\n - The description of the model architecture is confusing due to notational choices. I have added specific questions about the architecture below.\n - The experiments in sections 4.1, 4.2, and 4.3 are missing some important details. I have added specific questions about experimental details below.\n - The paper claims this method speeds up training time by 1.8x when compared to the state-of-the-art Transolver architecture. This is not supported by the relevant table (Table 4), which only reports time per epoch and not the total number of epochs needed to train each model.\n - There are grammatical mistakes throughout the text. These mistakes did not obscure the meaning of the text but were quite distracting. These mistakes could be addressed by software such as Grammarly or an AI chatbot; using these tools for grammar checking is allowed by NeurIPS 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Reviewer thought our inverse problem solution method interesting and wondered whether it is a novel solution method.**\n\nThis two-stage method is inspired by the inpainting and outpainting problem in the computer vision field.\n\n**Q2: Reviewer thought our definitions of \"geometric space\" and \"latent space\" were not clear enough.**\n\nPlease refer to the  \"Latent Space\" part in author rebuttal.\n\n**Q3: Reviewer pointed out we lack evaluation of the model’s robustness to varying $Nin$ or $Nout$ at test time.**\n\nAs requested, we present a comparison with other methods in Table 2 of author rebuttal attachment, following the setting in Section 4.2 of [29]  to varying $N_{in}$ and $N_{out}$ at test time.\n\nSpecifically, we downsample the Darcy dataset with resolution $421^2$ to $211^2,141^2,85^2,61^2,43^2$. We train model on the $43^2$ dataset and test it on the others. Compared with the results in Table 2 of [29], our method performs best.\n\n**Q4: Reviewer asked about the total number of epochs needed to train each model in our efficiency experiments.**\n\nWe train models for the same $500$ epochs same as [23]. Details are in Appendix D of our paper.\n\n**Q5: Reviewer pointed out our grammatical mistakes.**\n\nWe will check and eliminate these grammatical errors to ensure the readability in revised manuscript.\n\n**Q6: Reviewer asked whether the LNO architecture can be interpreted as a new way to parameterize kernel integral operators and wished us to discuss about the motivation of architecture choice.**\n\nYes, please refer to the \"Theoretical Justification and Motivation\" in the author rebuttal.\n\n**Q7: Reviewer asked about specific problems the LNO could solve but are intractable for other methods since we have improved the computational complexity.**\n\nAlthough our method reduces the complexity by about half, compared to previous SOTA, this is not sufficient to solve specific problems which are intractable for other methods in terms of complexity. \n                    \nOn the other hand, compared to coupled methods, our method can solve problems where the prediction and observation positions are not fully corresponding, enabling us to do interpolation and extrapolation, as shown by inverse problem experiments in Section 4.2 of our paper, which previous SOTA can not.\n\n**Q8: Reviewer asked about the input/output dimensions of the branch/trunk projectors and pointed out conflicting notations $x$ and $D$.**\n\nWe appreciate pointing out the conflicting notations. We will correct them by using $\\Omega$ to denote the PDE spatial domain and $\\hat{x}$ to denote the output of the trunk projector. \n\nBranch projector takes input with shape of $(N_{in},d+n)$ and outputs with shape of $(N_{in},D)$. Trunk projector takes input with shape of $(N_{in},d)/(N_{out},d)$ and takes outputs with shape of $(N_{in},D)/(N_{out},D)$ in encoding/decoding phase.\n\n**Q9: Reviewer asked about the meaning of white box labeled \"V\" in Figure 2**\n\nThe white box labeled \"V\" in the left sub-figure is $YW_v$ in Equation 4, and in the right sub-figure is $Z^LW_v^{'}$ in Equation 5.\n\n**Q10: Reviewer asked about the intuition guiding of applying softmax across different dimensions in Figure 2.**\n\nWe follow the standard practice in attention mechanism of computing attention scores for each query with respect to all keys, ensuring the sum of them is 1. When encoding, the queries are the $M$ hypothetical positions in the latent space, so we apply softmax to each column of the $N_{in} \\times M$ score matrix. When decoding, the queries are the $N_{out}$ predicted positions in the geometric space, so we apply softmax to each row of the $N_{out} \\times M$ score matrix.\n\n**Q11: Reviewer asked whether the trunk projector and the attention projector can be merged.**\n\nMerging them is mathematically valid but separating them facilitates intuitive understanding. The input should first be embedded by the branch/trunk projector, and the connection between the geometric space and the latent space should then be built by the attention projector through PhCA.\n\n**Q12: Reviewer asked about the meaning of horizontal line in Table 1.**\n\nThe methods above the horizontal line are not based on Transformer architectures while the methods below the horizontal line are based on Transformer architectures.\n\n**Q13: Reviewer asked about the randomness in the independent trials in Table 1.**\nThe independent trials are conducted with 5 different random seeds for weight initialization under the same train/test splitting strategy as in Section B.1 of [23] and in source code of [14,29] for fair comparison.\n\n**Q14: Reviewer asked about the difference between the train and test set for each benchmark in Section 4.1.**\n\nFor each benchmark, the data in train/test set are generated from the same distribution whose variable is equation coefficients (Darcy), initial conditions (NS2d, Plasticity) or geometric shapes (Elasticity, Airfoil, Pipe).\n\n**Q15: Reviewer asked about the inputs/outputs for the completion/propagation experiment and whether different initial conditions are used.**\n\nPlease refer to the \"Inverse Problem\" part in author rebuttal.\n\n**Q16: Reviewer asked about the shock wave in our plotting example of Burgers' equation in Figure 6.**\n\nThe shock wave is more pronounced at smaller diffusion coefficient. Its visual effect can be affected by the complex initial conditions and force terms as well. The diffusion coefficient is $0.01$ in our experiment (it is $0.1$ in [14], and $0.01/\\pi$ in [8]). Our shock wave exists but is not evident (bright yellow diagonal stripes at the color boundary). For clearer demonstration, please refer to Figure 1 in author rebuttal attachment.\n\n**Q17: Reviewer pointed out that we have not discussed the limitation we identified in checklist.**\n\nFigure 4 in Appendix A shows that our model's precision on the forward problem is not necessarily better at greater depths. We will discuss it more explicitly in revised manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NDoEjTw0Ml", "reviewer": "Reviewer_LtWn", "summary": "The present paper introduces a novel architecture, coined Latent Neural Operator (LNO), made of a sequence of self-attention layers intertwined between two Physics-Cross-Attension (PhCA) encoder and decoders. Experiments conducted on multiple PDE datasets evaluate the efficacy of the proposed approach across various metrics and highlight its potential advantages.", "questions": "- The learning tasks in Section 4.2 are not entirely clear to me. Could you formalize each task and specify the number of samples available for each?\n- I have some concerns about the scaling experiments of Section A: are the width and depth selected based on the L2 test errors? How do you perform neural architecture search for state-of-the-art methods?\n- Why does the number of parameters change from dataset to dataset in Table 4?", "limitations": "The paper does not adequately provide the statistical significance of the experiments. I recommend that the authors include a thorough statistical analysis to support the validity and reliability of their experimental results.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "The incorporation of self-attention layers in neural operators for solving PDEs has recently received significant attention. This paper contributes to this evolving field by introducing a novel architecture that integrates self-attention mechanisms within a stack of transformer blocks applied in the latent space. This design choice appears to enhance computational efficiency. A notable strength of the paper is its comprehensive evaluation, with the inclusion of an ablation study.", "weaknesses": "- *Originality*: The incorporation of self-attention layers in neural operators is not new and is well-documented. To this regard, multiple references are missing. The primary novelty of this work appears to lie in the integration of a stack of transformer blocks with self-attention mechanisms within the latent space.\n- *Quality*: The paper's main contribution is the introduction of a novel architecture evaluated through numerical experiments. However, the absence of theoretical guarantees represents a notable weakness. It is also very concerning that the authors have copied and pasted Table 1 from \"Transolver: A Fast Transformer Solver for PDEs on General Geometries,\" and merged it with their results. This practice raises questions about the originality and integrity of the numerical experiments presented. See also my concerns in the \"Limitations\" review section.\n- *Clarity*: The paper is difficult to read and lacks a coherent structure, especially in the numerical experiments section. Note that, since the contribution is mainly about the proposed architecture, a detailed description of the selection of the hyperparameters would have been appreciated. Overall, it is not very clear how this work differs from related works and what it borrows. Understanding the content requires multiple back-and-forth readings. The authors should focus on organizing their content more logically and clearly differentiating their contributions from existing research to enhance the readability and impact of their paper. In the same line, the paper contains numerous grammatical errors, which hinder its readability. I highly recommend that the authors thoroughly review and correct these errors to improve the clarity and overall quality of the paper.\n- *Significance*: While the paper suggests there may be some performance gains, it lacks clear demonstration of the significance of its findings in advancing the state of the art. A more rigorous comparison, such as cross-validation of the architecture and ensuring fairness in number of learning parameters with existing methods, would mitigate its somewhat limited contribution. Please also refer to my comments under the \"Quality\" and \"Limitations\" sections.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Reviewer thought our primary novelty lies in the integration of transformer blocks within the latent space and pointed out the incorporation of self-attention layers in neural operators is not new and well-documented but we have missed multiple references.**\n\nOur main contribution lies in the Physics-Cross-Attention which parameterizes the kernel integral operators with position-only kernel function and builds the connection between the real geometric space and the latent space, rather than simply using self-attention layers in the latent space. Please also refer to the \"Theoretical Justification and Motivation\" and \"Novelty\" parts of author rebuttal for more explanation and summarized novelty of our work. \n\nIn the second paragraph of Section 2.2, we have reviewed relevant neural operator methods, e.g., Galerkin Transformer, GNOT, FactFormer, ONO, which use self-attention and have provided ample citations. If we missed out any other important literature, we would appreciate it if the reviewer could point it out.\n\n**Q2: Reviewer pointed out we lack theoretical guarantees and have copied the Table 1 from [23].**\n\nPlease refer to the \"Theoretical Justification and Motivation\" part of our author rebuttal for the theoretical guarantees of our work.\n\nThe accusation of copied Table 1 is biased and unprofessional. \n\nFirst, for Transolver, we not only provided their claimed results, but also reported results of our independent implementation of Transolver (at time of submission of this manuscript, no official codes of Transolver were released). Moreover, our implementation of Transolver are even better than the claimed performance on NS2d, Airfoil, Elasticity and Pipe.\n\nSecond, for other methods, we do directly reference the claimed results, not only same as those in Transolver, but also identical to those in other papers. In fact, the Table 2 in [23] has also `copied` (or better to say, referenced)   evaluation results from previous papers. We list some examples:\n* In Table 2 of [23] and Table 1 of [21] , Galerkin Transformer on Darcy, OFormer on Darcy, GNOT on Darcy, OFormer on Elasticity, and GNOT on NS2d are exactly the same, with values $0.0084$, $0.0124$, $0.0105$, $0.0183$, $0.1380$, respectively.\n* In Table 2 of [23] and Table 1 of [21], FNO on Darcy, FNO on NS2d, Galerkin Transformer on NS2d, and OFormer on NS2d are also very close (same when rounded to three decimal places), with values $0.0108/0.0109$, $0.1556/0.156$, $0.1401/0.140$, $0.1705/0.171$, respectively.\n* In Table 2 of [23] and Table 1 of [29], ONO on NS2d, ONO on Darcy, ONO on Elasticity, ONO on Plasticity, Geo-FNO on Airfoil, Geo-FNO on Pipe, Geo-FNO on Elasticity, and Geo-FNO on Plasticity are also exactly the same, with values $0.1195$, $0.0076$, $0.0118$, $0.0048$, $0.0138$, $0.0067$, $0.0229$, $0.0074$, respectively. \n\nAs criticized, we will reproduce all experimental results based on the their official code of corresponding methods if available.  \n\n**Q3: Reviewer pointed out our paper is difficult to read, lacks a coherent structure and contains numerous grammatical errors.**\n\nWe will correct the grammatical errors and improve the presentation to ensure the readability in the revised manuscript.\n\n**Q4: Reviewer pointed out we lack clear demonstration of the significance of our findings in advancing the state of the art and recommended thorough statistical analysis to support the validity and reliability of our experimental results**\n\nWe have provided the standard deviations for the forward problems in the last row of  Table 1 using  5 independent trials, which reflected the significance of improvement over the state of the art. \n\nIn addition, we complete the standard deviations for the inverse problems here, shown in Table 3 of author rebuttal attachment using 3 independent trials. \n\nIn ablation study and scaling experiments, we do not provide significance results, as the primary goal is to study the effect and trend of model structure/hyper-parameter rather than to seek higher absolute accuracy (and also due to limitations of our computational resources).\n\n**Q5: Reviewer wanted us to formalize each task and specify the number of samples available in Section 4.2 to help on entirely understanding of the learning tasks.**\n\nPlease refer to the \"Inverse Problem\" part in author rebuttal for details of solving inverse problem.\n\n**Q6: Reviewer asked whether the width/depth of model are selected based on the L2 test errors in scaling experiments of Appendix A and how we perform neural architecture search for state-of-the-art methods.**\n\nThe major objective of Appendix A is to investigate the scaling property rather than neural architecture search. We show that the proposed model scales well in width (not yet saturates up to 256 channels) but saturates in depth at 8 layers, with one exception in Darcy, which saturates at 4 layers and 192 channels.\n\nFor fair comparison with other peer methods, we do not use the best hyper-parameters.\nInstead, we use 128 channels (for all problems but NS2d) keeping the same setting as Transolver, and set 256 channels for NS2d, which is identical to Transolver's setting as well.\n\n**Q7: Reviewer asked why the number of parameters changes from dataset to dataset in Table 4.** \n\nThe hyper-parameters are in Table 7 of Appendix D, and the number of model parameters are calculated accordingly. Specifically, in most experiments we use 8 layers and 128 (or 256 for NS2d) channels same as the Transolver's setting presented in Table 6 of [23]. \n\nThere is a single exception in Darcy, where we cut the number of layers by half to 4 as it yields an even better accuracy. We note that both the settings of 4 layers and 8 layers yield smaller L2 error (shown in top left figure in Figure 4, 0.49 for 4 layers, and 0.53 for 8 layers, smaller than 0.57 claimed in Transolver) with a smaller model and faster training speed.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely appreciate the valuable feedback and suggestions from the reviewers. We are delighted to see that the reviewers recognize the overall design of our model as clean and efficient. We are particularly grateful for the acknowledgment of the innovation in our core module, Physics-Cross-Attention, as well as the novelty of our inverse problem solution strategy. Additionally, we are honored by the reviewers' recognition of the value and effectiveness of our method in addressing the important problem of learning PDEs.

We address the comments point by point, responding the common questions below and specific questions to each reviewer individually. The paper will also be revised accordingly, with additional details.

**Theoretical Justification and Motivation**

Reviewer (WmcY, LtWn) asked if we can provide theory to support the current structure design, and Reviewer (ny2e) asked if the proposed LNO can be interpreted as a new way to parameterize kernel integral operators.

Indeed, the theoretical connection between the kernel integral operator and the cross-attention did motivate us to design Physics-Cross-Attention (PhCA) module, which separates position information from values. 

Specifically, previous works, e.g. Proposition 6 in [15] and Appendix A of [23], showed that the attention mechanism can be used to parameterize kernel integral operators. Roughly speaking, the integral operator 
$$
\mathcal{K}[f] (h) = \int_{\Omega} k(h,x) f(x) d x,
$$
can be considered as the convolution of the value $f(x)$  and the kernel $k(h,x)$ calculated by the product of the query and key
$k(h,x)=\frac{\exp(W_q h (W_k x)^T W_v) }{\int \exp(W_q h (W_k x)^T W_v) dx}$.

Conventional implementation of the attention in operator learning takes the concatenation of position $x$ and value $f(x)$ as input for all key, query, and value parts. We observed an insight that the kernel $k(h,x)$ is actually independent of the value $f(x)$, and therefore designed the PhCA module, where the input of key and query only depend on position $x$, and the input of value depends on the position-value pair $(x, f(x))$. 

We validate the efficacy of this idea in the ablation study shown in Table 5, where **LNO with P.A** in the first row uses both position and value as input to compute query and key yielding less accurate prediction than ours.

**Novelty**

Reviewer (LtWn) commented that self-attention is not new, and Reviewer (RkUb) is concerned that the novelty of the structure is limited, as the usage of cross-attention is similar to LSM with the main difference is using self-attention in latent evolution.

First,  the main contribution of this work is the PhCA module used as encoder and decoder transforming between geometric space and latent space (blue and green in Figure 1), rather than a stack of self-attention in the latent space (grey in Figure 1), which is standard. 

Second, LSM is notably different from ours, where the former focuses on constructing orthogonal bases, and ours learns the transform without extra constraint. In addition, as discussed in the above section of motivation, our cross-attention PhCA module separates the position and value, which not only improves precision but also increases flexibility making it possible to predict new locations not specified at the training stage.

Finally, on the overall structure, compared with the recent state-of-art, Transolver, which transforms between geometric space and latent space back-and-fourthly, we only transform once, reducing the computational complexity, training time, and GPU memory usage.

**Inverse Problem**

Reviewer (ny2e and LtWn) asked us to further explain the completion and propagation experiments for solving the inverse problem in Section 4.2.

The inverse problem designed in this work is to reconstruct the complete solution in the whole spatiotemporal domain, given only a set of partially observed (randomly or regularly distributed) samples in a limited space and time region. Completion and propagation are two sequential steps for solving a single inverse problem, which can be analogous to inpainting and outpainting in imaging processing. The two steps are visualized in Figure 6 in Appendix B.

In particular, for the completion experiment, the input, (left sub-figure in Figure 6), consists of sparsely sampled positions and corresponding values in $\Omega \times [0.25,0.75]$, where $\Omega=[0,1]$ denotes the spatial domain, and $[0.25,0.75]$ is a time interval. The completion task predicts all (dense, regularly sampled) positions and corresponding values in $\Omega \times [0.25,0.75]$ (middle sub-figure in Figure 6). The propagation step takes the output of the completion step as input and predicts the solution in the whole region $\Omega \times [0,1]$, visualized as the right sub-figure in Figure 6. 

We set the spatiotemporal resolution to be $128 \times 128$ for the discretized representation of $\Omega \times [0,1]$. Thus, the total number of samples is 16384. When the observation rate is 1%, the sparse sampling in $\Omega \times [0.25,0.75]$ will contain $81$ sample points. The completion step predicts all the  $8192$ sample points in  $\Omega \times [0.25,0.75]$ from these $81$ samples, and the propagation step further predicts all 16384 points in $\Omega \times [0,1]$.

**Latent Space**

Reviewer (ny2e) asked to clarify the definition of "latent space". 

The geometric space is the original space of the PDE input with shape of $({N_{in},d+n})$, containing $N_{in}$ samples, of which corresponding to $d$-dimensional position and $n$-dimensional value.

The transformation of the encoder from geometric space to latent space actually plays two operations: 1) lift the input value to a higher dimension $D$, 2) change the $N_{in}$ samples in geometric space to $M$ samples in the latent space (one can analogous this to Fourier transform that changes $N_{in}$ samples in geometric space to $M$ samples in frequency domain).

Meta-review: The paper proposes a novel neural operator architecture for solving PDEs with a cross-attention block, called 'physics cross attention'. The idea of cross-attention is well-established in classical generative models (for example, in vision language models), but before it was never applied in the context of neural operators. It makes perfect sense, and the results are quite reasonable. I think, this paper can be accepted.
Decision: Accept (poster)