Transformer for Partial Differential Equations’ Operator Learning
Authors: Zijie Li
Kazem Meidani
Amir Barati Farimani
Venue: Accepted by TMLR ()

Abstract: Data-driven learning of partial differential equations' solution operators has recently emerged as a promising paradigm for approximating the underlying solutions. The solution operators are usually parameterized by deep learning models that are built upon problem-specific inductive biases. An example is a convolutional or a graph neural network that exploits the local grid structure where functions' values are sampled. The attention mechanism, on the other hand, provides a flexible way to implicitly exploit the patterns within inputs, and furthermore, relationship between arbitrary query locations and inputs. In this work, we present an attention-based framework for data-driven operator learning, which we term Operator Transformer (OFormer). Our framework is built upon self-attention, cross-attention, and a set of point-wise multilayer perceptrons (MLPs), and thus it makes few assumptions on the sampling pattern of the input function or query locations. We show that the proposed framework is competitive on standard PDE benchmark problems and can flexibly be adapted to different types of grids.

Published in Transactions on Machine Learning Research (April/2023)
Transformer for Partial Differential Equations' Operator Learning
Zijie Li
zjjeli@@nddrew.cmu.ch
Deportment of Mechanical Engineering
Carnegie Mellon Universily
Kazem Mcidani
mmmeidans@anddrew.u cmaa, edu
Department of Mechanical Engineering Cornegie Mellon University
Amir Barati Farimani
berati@oma..cm edu
Department of Mechanical Engineering Carnegie Mellon University
Reviewed on OpenReview: /ttps://openreeet net/ forum? 9 EPPgt3uRRT
Abstract
Data-driven learning of partial diffcrential equations' solution operators has recently emerged as a promising paradigm for approximating the underlying solutions. The solution operators are usually parameterized by deep learning models that are built upon problem-specifie inductive biases. An example is convolutional
graph neural network that exploits the local grid structure where functions' values are sampled. The attention mechanism, on the other hand, provides flexible way to implicitly exploit the patterns within inputs, and furthermore, relationship between arbitrary query locations and inputs. In this work, we present an attention- framework for data-driven operator learning, which we term Operator Transformer (OFormer). Our framework is built upon self-at tention, cross-attention, and set of point-wise multilayer perceptrons (MLPs), and thus makes few assumptions on the sampling pattern of the input function or query locations. We show that the proposed framework is competitive standard PDE benchmark problems and can flexibly be adapted to different types of grids.
1 Introduction
Many the phenomena different fields of science and engineering are modeled by Partial Differential Equations (PDEs). Formulating and solving the governing PDEs help us understand, predict, and control complex systems in physics, engineering, etc. Various numerical schemes approach the problem of solving the PDEs via discretizing the solution space and solving finite-dimensional problem. However, solving complex spatiotemporal PDEs demands high-resolution discretization in both time and space which is computationally expensive.
A group of data-driven PDE solvers aim to directly learn the solution from the data observations without any prior knowledge of the underlying PDE. Those methods usually rely on a supervised deep lcarning architecture which imposes an inductive bias that can suit the problem in hand. The use of convolutional
layers for structured grids (Khoo al., 2021 Zhu & Zabaras 2018 Bhatnagar et al. 2019 Pant al., 2021 and graph layers for learning unstructured local relations in system (de Avila Belbute- Peres et al. 2020
lakovlev et al. 2021 Li & Farimani 2022 Li et al. 2022 Ogoke et al. 2021 are just two instances of these
1 Codee available https://github http:///gthhoo...m///aaaaaaatttt

---

Published in Transactions on Machine Learning Research (April/2023)
models. Learning such function mappings, however, is restricted to the specific resolution and the geometry observed in the training phase.
2021a) introduces a new data-driven modeling paradigm that aims to learn the solution operator of PDEs and provides a practical realization of the general universal nonlinear operator approximation theorem (Chen & Chen 1995). Concurrent work Graph Neural Operator (Anandkumar et al. 2020) proposes another operator learning architecture that leverages learnable iterative kernel integration. The promising features of data-driven operator learning, in particular the generalization capability within family of PDE and the potential adaptivity to different discretizations, have given rise to
The pioneering work DeepONet (u et al.
wide array of recent research works (Li et al.
2020b;
2021a Wen et al.
2022
Rahman et al. 2022 Cao 2021
Kissas et al. 2022 Lu et al. 2022 Cai et al.
2021 Bhattacharya et al.
2021 Nelsen & Stuart 2021 Gupla
et al. 2021). The training of data-driven operator can also be augmented with
physics prior to regularize the
search space of optimal model, bridging the bhysics-informed machine learning
[Raisii et al. 2019: Sun et al.
2020) with data-driven operator learning (Wang et al., 2021a Li et al.] 2021b Wang et al.
2021b).
In practice, the data-driven operators are trained based on the finite-dimensional approximation of the input and output functions, where functions sampled some discretization grids. Many of the learned operators, despite showing promising performance on solving PDFs, are limited to have the same discretizations for the input and output (Anandkumar et al. 2020 Li et al. 2020b fixed input grids (Lu et al. 2021a) or to have equi-spaced grid points (Li et al. 2021a Gupta et al., 2021 Cao 2021). As stated in Cao (2021). attention (Vaswani et al., 2017) can be viewed as certain types of learnable integral operators and its inference is equivalent to calculating the integral using input points as the quadrature points, which poses few constraints on the underlying discretization grid. Leveraging this property, we propose a flexible atlention-based operator learning framework. Concretely, our main contribution includes: (i) A
module that enables query of output function; (ii) A latent time-marching scheme that reduces time- dependent PDE into latent ODE (ordinary differential equation) with a fixed time interval; (iii) A flexible Transformer architecture that can take a varying number of input points and can handle both uniform and non-uniform discretization grids.
2 Related works
Learned PDE Solvers In the past decade, there have been a number of studies on applying deep learning
architectures for solving PDEs (Lu et al. 2021b Karniadakis al.] [002] Brunton al. 2020). Physics- informed methods (Raissi et al., 2019 Sun et al.) 2020 Sirignano & Spiliopoulos 2018 E & Yu 2018)
exploit the PDE supervision to approximate the solution (usually parameterized by a neural network) in an aggregated space-time domain. They are accurate and generally mesh-agnostic, yet require retraining across different instances of a PDE (e.g. change of coefficient). Without knowledge of the PDE, the solutions can also be learned purcly from the data. A common strategy for such task is to first spatially encode the data and then use various schemes to evolve in time. Convolutional layers (Stachenfeld et al. 2022 Wang et al. 2020; Bhatnagar et al. 2019 Kochkov et al. 2021 kernels (Saha & Mukhopadhyay 2021 1. and graph-basod layers (de Avila Belbute-Peres some of the common architectures for learning the spatial relations in PDE. In this study, we show that transformers can be pliable yet effective spatial encoders.
al. 2020: lakovlev al.] 2021 Li & Farimani, 2022 Li et al. 2022) are
Timo-marching in the latent space Learning the temporal component of the spatio-temporal PDEs can be challenging. A common and straightforward approach is to follow an Encoder-
(EPD)
scheme to map the input solution at time t to the solution at next time step (Brandstetter et al.
2022
Stachenfeld et al. 2022 Pfaff et al. 2020 Sanchez-Gonzalle et al. 2020 Pant et al. 2021 Hsieh et al.
2019
An alternative approach which can significantly reduce the computational complexity and memory usage is to propagate the dynamics in the latent space (Lee & Carlberg 2021 Wiewel et al. 2019). Therefore, once the encoding from observation space to the latent space is done, the system can evolve in time using Recurrent Neural Networks (RNNs) such as LSTM (Wiewel et al. 2019) or even lincar propagators basod
on the assumptions of learning Koopman operator (Morton et al. 2018 Li et al. 2020a Lusch et al. 2018
Takeishi et al.) 2017 Pan & Duraisamy 2020). In this work, we design latent time-marching architecture
2

---

Published in Transactions on Machine Learning Research (April/2023)
that brings about scalability of the model's attention encoder and allows the model to unroll for the future time steps during training.
2021a) introduces the first practical realization of universal operator approximation theorem Chen & Chen 1995), which can further be integrated with prior knowledge of the system (Wang et al., 2021a bj. In another group of works, the infiniteddimcnsional solution operators are approximated by iterative learnable integral kernels Anandkumar et al. (2220). Such
Neural Operators The pioneering work DeepONet (Lu et al.)
kernel can be parameterized by message- message-passing (Gilmer et al. 2017) neural networks (Anandkumar et al.,
2020: Li et al. 2020b) or convolution in the Fourier domain (Li al. 2021a). Kovachki al. (2021) further provide theoretical analysis of the approximation capacity of kernel-integral-based learning framework.
Fourier Neural Operator (FNO) (Li et al.] 2021a) and its variants (Li et al.] 2021b Guibas et. al. 2022
Tran et al., 2021) have shown promising results in various applications (Pathak et al.) 2022 Wen al. 2022 These integral kernels can also be learned in the multiwavelet domain (Gupta et al. 2021 Tripura & Chakraborty 2022). Furthermore, other than Fourier bases or multiwavelet bases, the bases needed for approximating the solution operator can be learned and composed via attention (Cao 2021
Transformer for physical system After the groundbreaking success in natural language processing Vaswani et al. 2017 Devlin et al. 2018), attention has also been demonstrated as a promising tool for
various other machine learning tasks Dosovitskiy ct al. 2021 Tunyasuvunakool et al.) 2021 Veličković et al,
2017 Following the success in these fields, several previous works (Geneva & Zabaras 2022 Han al.,
2022
2021 Kissas et al., 2022) have explored using attention to model and simulate physical system, which can generally be divided into two orthogonal lines. In the first group, attention layers are used to capture the structures and patterns lie in the PDEs' spatial domain (Cao) 2021 Kissas et al.
Shao et al., 2022 Cao
Shao et al. 2022) On the contrary, in the second group, the attention is used to model the temporal evolution of the system while the spatial encoding is done by other mechanisms like CNNs or GNNs (Geneva & Zabaras 2022 Han et al., (002)). Our model lies in the former group where attention-based layers are used for encoding the spatial information of the input and query points, while the time marching is performed in the latent space using recurrent MLPs.
2022
3 Method
3.1 Attention mechanism
Standard attention The standard altention Vaswanni el al. 2017 Bahdanau el al. 2014 Graves et al.
2014 Luong et al., 2015) is an operation defined upon three sets of vectors, namely the query vectors {q,
key vectors { and value vectors {vy}. Concretely, for cach query q6 the corresponding output z; is caleulated weighted sum (assuming number of vectors in each set is n):
exp (p(qq;
2; Σ Vi
aij Σn -==1 exp (h(q k,))
(1)
The most common choice for the weight function k(-) the scaled dot-product (Vaswani et al. 2017): :
h(q;, k,) (q: k,)/vd, with d being the channel of vectors q, and ky. The scaled dot-product can be written in matrix representation as: = softmax [QK3/v/] V, where 2i, qi ki, Vi are the i-th row vectors of matrices Z, Q, K, V respectively. Due to the presence of the softmax, the multiplication QK" must be evaluated explicitly, resulting in overall complexity of O which is prohibitively expensive when applying to long sequences.
Attention without softmax In natural language processing and computer vision tasks, the i-th row vector of query/key /vluee matrices, q q//k/v,, are usually viewed as the latent embedding of a Loken (e.g. a word or an image patch). Other than the "row-wise" interpretation of attention, Cao (2021) proposes that each column in the query/key /vay/va matrices, can potentially be interpreted as the evaluation of a lcarned basis function at each point. For instance, Vg, (also the i-th element of the j-th column vector: (v'),) can be viewed as the evaluation of the j-th basis function on the i-th grid point Ii, i.e. Vo = vy (z;), and similarly
3

---

Published in Transactions on Machine Learning Research (April/2023)
for the matrices Q. K. (with each column represent the sampling of basis function (,()), Facilitated by the learnable bases interpretation, Cao (2021) proposes two types of attention that can be viewed as the numerical quadrature of different forms of integrals:
13 (q k.)(v', 2 (x())))
Fourier type: (zi);
(2)
n
d Galerkin type: (z)); (k' v²)
(q) ( (L1 (L((E)s((E)) (((z)). 1=1
Σ
(3)
where Ω denotes the spatial domain that is discretized by n points fai)--1. and the dot product k, in equation 2 approximates the learnable kernel function K (:..) at Ty). The approximation capability
of Fourier type attention and Galerkin type attention are studied theoretically at (Kovachki et al. 2021
Choromanski et al. 2020) and (Cao 2021 respoctively. Inspired by the Gram-Schmidt process, Cao (0021
heuristically chooses layer normalization (Ba et al., 2016) to normalize q, k, in equation 2 and k' v in equation But if we further exploit the basis function property of these column vectors, it is also reasonable to impose non-learnable instance normalization (Ulyanov et al. 2016) on each column vector such that their L2 norm equals to one, i.e. ||>' = 1. In practice, we find this slightly improves the model's performance, and thus we adopt instance normalization throughout the model. Equipped with the normalization schemes, equation 2 and equation in matrix multiplication form write as:
1 == V; Galerkin type: Z= ==-Q(KTV)
Fourier type: Z =
(4)
where denotes a column-wise normalized (via instance normalization) matrix. The above integral-based attention mechanisms serve simple but powerful building blocks for PDE operator learning model. The softmax-free property allows both Fourier lype and Galerkin lype to be computed efficiently in the form (omitting the normalization scheme): Z V)/n, since matrix multiplication is associative.
Cross-attention While the aforementioned altention mechanisms are flexible with respect to the dis- cretization of input domain, the fact that Q K, V are essentially different linear projections of the same imput feature embedding H makes the input H and output Z = ((K"V))n locked to the same fixed grid {a:}== To decouple the output domain from the imput domain and allow for arbitrary query locations, we extend the abve aftention mechanisms from self-attention to cross-attention, where the query matrix Q is encoded from different imput. To be more specific, the i-th row vector qi in the query matrix Q is the latent encoding of query point y;, where {n, ) denotes the discretization of the output domain and is not necessarily same as the imput grid points {- Hence, leveraging the cross-attention mechanism, we can query at arbitrary locations which are independent. of the input grid points. If we adopt the learned bases interpretation from previous section, the cross-attention mechanism can also be viewed as a weighted sum of three sets of basis functions: (kx()), vz()), v1))). (-)){= with (+()), vx(-) defined on the input discretization grid D(x) (mx)²= and q((-) defined the query discretization grid D(v)= : (yim which serves the finite-dimensional approximation space's basis for the final target function:
E (2 (1y))) n
(5)
If we drop the point-wise feed forward network (FFN) after the attention layer, which is a MLP with two layers (omitting the bias term): x <- a (xW() W where W(l), w e Raxd are learnable weight matrices and (-) is a non-linear activation function, and then directly make (x;)) as output, the cross attention in the equation 5 is a special case of the universal operator approximator proposed in DeepONet (Lu et al. 2021a) with Ei (x:z)) (zz)/n the output of branch net and (ı,) as the output of trunk net. But in practico, we find that appending FFN after attention improves the model's performance and a propagating MLP is necessary for time-dependent PDEs (see Figure 2). Another notable difference is that the number of parameters in the attention layer does not depend on the input's 's discretization and thus it is applicable to samples with varying grids without re-training. More broadly speaking, the query matrix Q whose column space is spanned by the learnable query basis functions {q(-) loosely resembles the induced points in Set Transformer (Lee et al.] 2018) and the latent array in Perceiver (Jaegle et al. 2021
4

---

Published in Transactions on Machine Learning Research (April/2023)
MLP
QIKV Linear
MLP
MLP
Q/KV Linear
( 0(x,)
Attention
MLP
X
MLP
Q/IV Linear
MLP
Observations of input function
Input eacoding
KIV Linear
K/V Linear
MLP
Linear
e ely,), ((x;)
MLPP
X
MLP
Linear
Attentian
MIPP
X
X
MLP
LLneer
MLP
Coordinates
Latent encoding on query locations
Query encoding
query locations
Figure 1: Atlention-based encoder architeclure. Top row: Input encoder that encodes inpul function information based on locations ({=+)==) where input function is sampled and the sampled function value. Bottom row: Query encoder that encodes the coordinates of query locations (fv)) and uses encoded coordinates to aggregate information from input encoding via cross-attention. Rotary positional encodings (())qquaiio 9 are used in both self-attention and cross-attention.
3.2 Model architecture
The proposed model has an Encoder-Decoder architecture that is similar to the original Transformer (Vaswani et al. 2017 where the input is first processed by several self-attention blocks and then attended to the output. However, only use cross-attention to derive the latent embeddings sampled query locations and then use recurrent MLPs to propagate dynamics instead of masked attention as in typical Transformer decoder.
Encoder The encoder comprises three components (Figure 1. an input encoder &f (.) that takes the input function's sampling a(x;) and coordinates of sampling positions {z,)i= as input features, a query encoder (-) that takes the coordinates of query locations { as input features, and a cross-attention module pass system information from input locations to the query locations.
The imput encoder first uses point-wise MLP that is shared across all locations to lift input features into high-dimensional encodings (()) and the encodings are then fed into stacked self-attention blocks. The update protocol inside each self-attention block is similar to the standard Transformer (Vaswani et al. 2017
f(') = LayerNorm (F() + Attu(P())) p((+1) LayerNorm (FO) + FFN((()))
(6)
where Atta(-) denotes the linear attention mechanisms discussed in the previous sub-section, Layer.orm(.) denotes the layer normalization [Ba et. al. 2016 When input and output data are not normalized, we drop the LayerNorm to allow the scaling to propagate through layers.
The query encoder consists of shared wise MLP, whose first layer is random Fourier projection layer (Tancik et al. 2020 Rahimi & Recht 2007). The random Fourier projection 7(-) using Gaussian mapping is defined as:
YY) [cos (2x YB), sin (BB))),
(7)
where Y = Y2,..,, yv is the Cartesian coordinates of the i-th query point, B E xgdx (with di the dimension of input coordinates and d2 the output dimension) is a matrix with elements sampled from Gaussian distribution (00 22) with predefined a. The random Fourier projection 7()) alleviate the spectral bias of coordinate-basse neural networks (Tancik al.] 2020 Mildenhall al.) 2020), which has also been
introduced into physics-informed machine learning in Wang et al. (2021c).
5

---

Published in Transactions on Machine Learning Research (April/2023)
Steady state system
Time-dependent system
t
t = t1
Shared decoder
MLP
MLP
MLP
MLP
MLP
MLP
MLP
MLP
XT
:
MLP
MLP
Latent encoding
Shared propugutor
Figure 2: Decoder and propagator architecture. Given the latent encoding, for steady state system, a point-wise MLP is used to map the latent encoding z to the output function value. For time-dependent system, a point-wise MLP with skip connection is used to march states temporally in the latent space.
At the cross-attention module, by attending the latent encoding of the last (L-th) attention layer's output f(1)) of input function with the learned encoding z (0)) of query locations, we pass the information of input function to the query points. The cross-attention process is defined as:
z' (0) + Cross-Autn(2 p(()), z= x' (FN(2').
(8)
Compared to equation 6 we drop the LayerNorm at each sub-layer to preserve the scale (variance) of z (z)
Latent propagator for time-dependent system A direct way to approach the time-dependent problem would be by extending the degrees of freedom and introducing temporal coordinates for cach grid point as Raissi et al. (2019), which requires the model learn the entire space-time solution once. As analyzed in Krishnapriyan et al. (2021), for certain problems such reaction-diffusion equation, this could lead to poor performance. In addition, this usually requires the model to have larger number of parameters in order to model the extra degree of freedom (Li et al., 2021a Gupta et al. 2021).
Another dircction would be using a timo-marching schome to autoregressively solve the entire state space with a fixed time discretization (resembles the Method of lines (Schiesser 1991)) ), which is adopted by many neural-network based solvers as is easier to train. common architecture learned time-marching solvers is the Encoder- Seecoder-Processee Prroeeeoo--ecooeeee (EPD) architecture (Sanchez- Gonzalez al., 2020 Pfaff al. 2020 Brandstetter et al. 2022 Stachenfeld et al. 2022). which takes state variable u, input, project it into latent space, then processes the encoding with neural networks and finally decodes it back to the observable space: attAr, Such model is trained by minimizing the per-step prediction error, with input u' and target + sampled from ground truth data. In fact, if we unroll the learned per-step solver fNN(-) to predict the full trajectory (a² = (wN(""), _² = JNN (a")) the unrolled solver can be viewed as a recurrent neural network (RNN) with no hidden states, and thus the standard per-step training pipeline is essentially a recast of the Teacher Forcing (Williams & Zipser [989). Therefore, the learned per-step solvers face similar problems RNNs trained with Teacher Forcing, the accumulated prediction error leads to the shift of the input data distribution and thus makes model perform poorly. Such data distribution shift can be alleviated by injecting noise during training Stanchez-Gozzalle et al., 2020 Pfaff et al. 2020 Stachenfeld et al. 2022). Alternatively, Brandstetter el al. (2022) propose to unroll the solver for several steps and let the gradient îlow through the
last step to train the model on the shifted data. For operator learning problem where time horizon is fixed,
Li et al. (2021a) push this further by fully unrolling the trajectory during training. However, fully unrolled training has a memory complexity of Ottn) (with t the length of time horizon and n the number of model parameters), which can be prohibitively expensive if the model is large.

---

Published in Transactions on Machine Learning Research (April/2023)
Here approach the space-time modeling as sequence-to-sequence (Sutskever et al. 2014 learning task and propose a recurrent architecture to propagate the state in the time dimension. Different from the standard EPD model, we forward the dynamics in the latent space instead of the observable space, and thus encoder only needs to be called once throughout the whole process, which reduces the memory usage and allows scalable fully unrolled training. Given the latent encoding z from equation g we set it as the initial condition 2 and recurrently forward the latent state with a propagator N(-) that predicts the residuals (He et al. 2015) between each step: z N(z2) + +z². There are many suitable choices for parametrizing the propagator (e.g. self-attention, message- message-passing layer), but in practice, we found point-wise MLP shared across all the query locations and all the time steps suffice, which indicates that the our model can effectively approximate the original timendependent PDE with a fixed-interval ODE in the latent space.
After reaching target time step in the latent space, we use another point wise MLP to decode the latent encodings 2 2x) back to the output function values u(x,)) t).
Relative positional encoding The altention mechanism is generally position-agnostic if no positional information is present in the input features. Galerkin/ Foluriii/Foouier /Fuurier Transformer (Cao 2021) approaches this problem by concatenating the coordinates into input features and latent embedding inside every attention head, and adds spectral convolutional decoder (Li et al. 2021a) on top of the attention layers. Instead of using convolutional architecture which is usually tied to the structured grid, we use Rotary Position Embedding (RoPE) (Su et al.) 2021) to encode the positional information, which is originally proposed for encoding word's relative position.
Given the 1D Cartesian coordinate 2y of point and its d-dimensional embedding vector qi E Rdx1, the RoPE (+) is defined as:
vb 2z) θ (ax) qo
(a;) = Diag(Daa R2, Re...., Ra/2),
(9)
coos (Ax,01)
sin (Ax;01)
where: R
(10)
sin (Av;00) cos (Ax;0t))
where Diag(-) denotes stacking sub-matrices along the diagonal, À is wavelength of the spatial domain (e.g. A = 2048 for a spatial domain discretized by 2048 equi-distant points), O2 is set to 10000-2(-11)4 (1,2...., d/2} following Vaswani et al.] [2017]: [uu ct al. (2021). Leveraging the property of the rotation matrix, the above positional encoding injects the relative position information when atlending a query vector qi with key vector ky: + (z) & 2z)
q O - zy)ky. RoPE can be extended to arbitrary degroes of freedom as proposed in (Biderman al.) 2021) by splitting the query and key vectors. Suppose the spatial domain is 2D, the attention of query q, with coordinate x; (a,, B,) and key kj with x, = (3) write as:
(z())).)) (O(=)),)
a (q, x;) w xj), = wi ((q)) os) wi (())) + vhe ((1))a/25) B:) 22 ((k))4/2::33))
(11)
where (q:).d/p denotes the first d/2 dimensions of vector qe and (q:) 4/2. denotes the rest d/2 dimensions, similarly for kj, and dh (-) (+) is the 1D RoPE defined in equation 9
Training setting The general training framework of this work similar to the previous data-driven operator learning models (Gupta et al.) 2021 Li et al., 2021a; Cao 2021 Anandkumar et al. 2020 Lu et al, 2021a). Given the input function
A, and target function u E U, our goal is to learn an operator The model parameter 0 is optimized by minimizing the empirical loss Eu- [C(Go(a), u)], where ju is a measure supported on A, and L(.) is the appropriate loss function. In practice, we evaluate this loss numerically based on the sampling of {a ((a)) "D(y 1,y2 on discrete grid points D(x)= : {z, (e. D(y): {vi Hm and choose L(-) as
A + U.
(i)
(i)
R |]a(i] 210 given B samples with (() being the model prediction. Different from several previous works (Li et al. 2021a 2020b] Cao 2021 Gupta et al.] 2021), Dar), D(y) does not need to be the same discretization of the PDE spatial domain ? and can vary across different training samples
relative L2 norm: 1 +E=
{a()) (ap() (3) "6) )-
7

---

Published in Transactions on Machine Learning Research (April/2023)
4 Experiment
In this section, first present the benchmark results the standard neural operator benchmark problems from Li et al. [021aa) and compare model with several other state-of-the-art operator learning frameworks, including Galerkin/Pouriss Transformer (G.T. F.T.) from Cao (2021) and Multiwavelet-basee Operator (MWT) Gupta et al, 2021). Then we showcase the model's application to irregular grids where above frameworks cannot be directly applied to, and compare model to graph ncural network bascline (Lötzsch et al. 2022). We also perform an analysis on the model's latent encoding. Finally, we present an ablation study of key architectural choices. (The full details of model architecture on different problems and training procedure are provided in the Appendix A More ablation studies of hyperparameter choices are provided in Appendix
4.1 Benchmark problems
Below we provide a brief description of investigated PDEs. Full details of these equations' definition and dataset generation are provided in the Appendix D
Problems on uniform grid The first kind of problems considered include 1D viscous Burgers' equation, 2D Darcy flow, and 2D incompressible Navier-Stokes equation. In these problems, the data are represented on a uniform equi-spaced grid, where discrete Fourier transformation (FNO, G.T./F.T.) and wavelet transformation (MWT) can be efficiently applied to.
The objective solution operators G aimed to learn are defined as (T' is the time horizon, u denotes solution function of interest, a denotes the coefficient function):
Burgers': G u(;,t)een + u(,, f)
(12)
Darcy flow: 9 + u,
(13)
Navier-Stokes:
u(:,t))eefo.1 ul., u(.,
(14)
On Navicr-Stokes equation, following prior work (Li et al. 2021a), we adopt datascts NS1, NS2, NS3 with different viscosities, and we generate more challenging and diverse dataset (denoted NS-mix) where is not constant, and it is uniformly sampled: ranging from 200 to 500. For the size of cach Navicr-Stokes dataset, NS2-full contains 9800/200 (train ttrain/tssi /tost) samples; NS-mix is larger dataset consisting of 10000/1000 samples; each of the rest datasets contains 1000/200 samples. For Burgers' equation and Darey flow, the dataset splitting is same as in Cao (0021
U[[0.4, 1]) 10-4, corresponding to Reynolds numbers roughly
Problems on non-uniform grid Other than uniform grid, the second kind of problems considered are based on unstructured non-uniform grid. We first investigate the 2D Poisson equation of magnetic/ eelectric field on different geometries, the objective to find the solution of field given boundary conditions and source-terms. We use the shape extrapolation dataset from Lötzsch al. (2022). The training contains grids of four shapes (square, circle with or without hole, L-shape) with 8000 samples, while testing set has 2000 samples consisting of U-shape grids that model has never seen during training. This experiment tests model's generalization capability to different solution domain. In addition, we study the 2D time-dependent compressible flow around the cross-section of airfoils using the dataset from Pfaff et al. (2020). In this problem, the underlying grid highly irregular, as the distance between two closest points ranges from 2e to 3.5. The dataset contains 1000/100 100 sequences different inflow speed (Mach number) and angles of attack.
The objective solution operators with respect to these problems are defined as:
Poisson: 9: f u, = 0 on an,
(15)
Airfoil: G u(, (keeja.o.5 u(: (E00.576,4.8000
(16)
8

---

Published in Transactions on Machine Learning Research (April/2023)
where Q is the solution spatial domain and f is the source term of the Poisson equation. For Poisson equation we predict the field vector in addition to the scalar potential For compressible flow we predict velocity, density and pressure jointly.
4.2 Results and discussion
For benchmark results of all baselines, we report the evaluation results from original paper when applicable. When not applicable, we train and evaluate the model following the suggested setting in the original paper, and mark the results with The best version of Galerkin /Fourier Transformer is reported for each problem. Details of baseline models' implementation and comparison of calculation time can be found in the Appendix
Calerkin/FFurier
Data settings
Relative L2 norm
Case
v, 7
FNO-21>
FNO-3D
MWT
G.T.*
OFormer
NS1
x 10-3, 50
0.0128
0.0086
0.0062
0.0098
0.0104 + 0.0005
NS2-part
x 10-4,30
0.1559
0.1918
0.1518
0.1399
0.1755 + 0.0050
NS2-full
1 x 10-4,30
0.0834
0.0820
0.0667
0.0792
0.0645 + 0.0011
NS3
1 x 10-5,20
0.1556
0.1893
0.1541
0.1340
0.1705
+ 0.0007
NS-mix*
[0.4, 1] x 10"", 30
0.1650
0.1654
0.1267
0.1462
0.1402 + 0.0018
# of parameters (M)
0.41/2.37
6.56
179.18
1.56
1.85
Table 1: Benchmark on 2D Navier-Stokes equation with fixed 64 x 64 grid. For large dataset (NS-mix) we use a larger version of FNO-2D by increasing modes and width. We adopt Galerkin Transformer (G.T.) with a larger hidden dimension (64 + 96) to match the size of other models.
Relative La norm (x10-3)
Relative L2 norm (x10-2)
Res.
Res.
FNO*
FNO-* * MWT
F.T.
OFormer
FNO
G.T.
OFormer
512
3.15
0.72
1.85
1.14
1.42 + 0.03
141
1.09
0.84
1.26 + 0.os
2048
3.07
0.72
1.86
1.12
1.30 + 0.07
211
1.09
0.84
1.28 + 0.02
8192
2.76
0.70
1.78
1.07
1.54 + 0.10
Table 3: Benchmark on 2D Darcy flow with different resolution. MWT's result is not re- ported since its implementation requires reso- lution to be power of 2 and thus it is trained on a different data.
Table 2: Benchmark 1D Burgers' equation with differ- ent resolution. FNO uses ReLU (Nair & Hinton 2010) whereas FNO+ uses GELU. Both FNO variants remove normalization compared to the original paper.
Performance equi-spaced grid The comparison results with several state-of-the-art baselines are shown in Table 0 We observe that for small data regime (NS2-part, NS3) and problems with relatively smooth target function (viscid Burgers', Darcy), OFormer generally does not show superiority over other operator learning-frameworkee framcworks. Architectures like spectral convolution (in FNO, Galerkin/Fouric Transformer) are baked in helpful inductive bias when target is smooth and periodic. As shown in Table 2 equipped with a smooth activation function, FNO+ outperforms all other methods signíficantly, as the target function u(; 1) relatively smooth due to the presence of diffusion term, and has periodic characteristic in the spatial domain. For more complex instances like Stokes equation, we find that OFormer has strong performance in data sufficient regime, where it is par with MWT on NS2-full with substantially less parameters. This highlights the learning capacity of attention-based architecture, which leverages flexible basis learned from data. Moreover, our model performs similarly under different resolutions, indicating its robustness to different resolution. (The temporal error trend can found in Appendix C Contour plot of the prediction and corresponding crror can be found in Appendix
?The electric field E is defined as the negative gradient of electric potential (ø): E Vo. For magnetic field B, it is defined as the curl magnetic potential A: A. this problem A, = A, = 0, and therefore magnetic potential reduces to: Ba 0Az/Oy, B, -0A./Or In the model we separate head to predict these fields.
9

---

Published in Transactions on Machine Learning Research (April/2023)
Training samples
Testing samples
Mean squared
(x10-3)
O
Li 1 L0
Mesh aug.
Model
Electric
Magnetic
Potential
Field
Potential
Field
OFormer GNN
0.174 1
1.690 1 0.201
0.138 +
1.470 +
0.327
2.723
0.161
2.640
OFormer GNN
0.019 + 0.004
0.328 + 0.02s
0.016 + 0.002
0.195 + 0.00m
Yes
0.006
1.821
0.006
1.338
Visualization train- ing/ /gg/testingaaa samples in 2D Poisson problem. Testing set contains new geometries that never appeared in the training set.
Figure 3:
Table 4:
Benchmark on electrostatics and magnetostatics Poisson equa- tion. The train rrain/evaluation protocol is adopted from Lötzsch et al. 2022 Mesh augmentation comprises set of random transformation on the training meshes, including varying node density and size of cull/c
Electric field Reference
(b) Airfoil
Prediction
Error
Prediction
Reference
Error
 LUD
Figure 4: Samples model's prediction, reference simulation result and absolute error on: (a) Electric potential and corresponding field; (b) Velocity field around airfoil, top top/bottom: x/y component.
Application to irregular grids The comparison between OFormer and GNN baseline on the 2D Poisson problem shown in Table I Both OFormer and GNN have the capability to learn and extrapolate to new geometries with reasonable accuracy under mesh augmentation. However, we can observe that OFormer outperforms GNN by a margin when no mesh augmentation is applied. This highlights an essential difference in the learning mechanism between OFormer and local graph convolution-based architecture. GNN that exploits local mesh information such edge distance and node connection is more prone to overfit the training geometries, while OFormer, leveraging global attention and not using any explicit graph information, is more robust under the change of local mesh topology. Other than grid with relatively uniform point distribution, OFormer also applicable to highly irregular meshes. We showcase that OFormer is able to predict the temporal evolution of compressible flow around the airfoil (exemplar visualization is shown Figure more visualization results can be found in Appendix The test root mean squared error (at region closed to airfoil) of the predicted momentum is 15.46911.007 (relative error: 8.17=0.06%)
Another important property of OFormer is that it can handle variable number of input/query points with the same set of weights. Here we showcase an application of such property. As shown in Figure 5 the model can still predict reasonably accurate results on a full-resolution output grid based on sparse input. The input is generated by randomly sampling 25% of the grid points in the spatial domain, which is 150% of the maximum dropping ratio during training (More quantitative results on varying sampling grids can be found in Appendix
During training, randomly drop some the input points with ratio ~~(([ 0.5)), still query full resolution and train full resolution.
10

---

Published in Transactions on Machine Learning Research (April/202))
(a)
04
T=0
Prediction: 7=20
Ground truth: 7-20
Error
(b)
06
05
04
Randomly placed sensors
Sensors' observation
Prediction: 7=20
Error
Figure 5: (a) Model's prediction on dense uniform input points; (b) Model's prediction sparse randomly sampled input points (25% of all grid points).
Learned spatio-temporal reprosentation To inspect OFormer's ability to learn meaningful represen- tations of the PDEs, we perform an analysis on our model trained on a dataset of Navier-Stoles equation where each sample has random coefficient log(v) 1010((33,550 -5.3], which resulted in falls the range of (5e 6, le -3.- 3). The model trained predict the future time steps, where it has never been informed of the coefficient values. After training, examine the latent space of the input encoder.
A single d-dimensional representation vector ((4) of k-th sample is obtainod by applying average pooling on the encodings + Avg( (f(*)))) F=)) over the input points. First, we project the representation vectors of all training samples into 2-dimensional space using Principal Component Analysis (PCA). We can observe the evident trend of colors w.r.t. # in Figure aa). Furthermore, we train linear regression model on the latent vectors from the training samples, and then we use it to predict the v on testing samples. Figure 6(b) shows
that a linear model can effectively map the latent representations to the viscosity coefficients which indicates the model's capability of finding compact representations such that a nonlinear mapping has become linear.
1e-3
Training samples
Relative Loss: 0.0569 Test R 0.997
Testing samples
Principal Component
Trained Encoder
Encoder
-10
0.0
02
0.4
0.6
0.8
1.0
1" Principal Component
Ground Truth
1c-3
...
Train Linear Regression
Predict with learned model
Figure 6: Learned representations of the model trained Navier-Stokes equations with different viscosity coefficients. (a) The 2-dimensional PCA the latent vectors. (b) A linear regression model trained on the latent vectors of training samples can accurately predict v of testing samples.
11

---

Published in Transactions on Machine Learning Research (April/202))
4.3
Ablation study
(a)
0.1107
Layerboom
Instancliomm
0.10
2.01
0.08
0.3774
0.0646
0.0646
3.5
Reletive
0.04
Relattve
0.02
0.3
0.00
Galerkk
(1.7%) Lange params)
RoFE
No RoPE
Pouneer
Golerih
Type attention
Type positional encoding
Type attention
Figure 7: Ablation study on (a) NS2-full dataset; (b) 1D Burgers' equation with resolution 2048.
Light color denotes option we adopted.
Vorticity
Error
Reference
Use RoPE
No RoPE
Use RoPE
No RoPE
15
000
ous
Figure 8: Influence of positional encoding, the visualized sequence is randomly chosen. Error measured by absolute error.
We study the influence of the following factors on the model's performance: (i) The type of attention; (ii) Model's scaling performance; (iii) Positional encoding; (iv) Different normalization methods, The quantitative ablation results are shown in Figure
First of all, we find that the performance of a specific type of attention problem-specific, which signifies the importance of choosing normalization targets (query/key key/vall key/value). In addition, observe that scale-sensitive problem (data not normalized for Burgers' equation to preserve the energy decaying property), InstanceNorm which normalizes cach column vector ||v = 1, performs better than LayerNorm, further facilitating the bases interpretation of two linear atlention mechanisms. Scaling up the model's size (increasing the hidden dimension) can benefit the model's performance datasot, the larger version of OFormer pushes the previous best result from 0.0667 (Gupta et al. 2021 0.0625. Lastly, we find that RoPE removed and replaced by concatenating coordinates into query and key vectors (after projection), the model's performance deteriorates significantly. As shown in Figure without.
data sufficient regime. On NSlfull
12

---

Published in Transactions on Machine Learning Research (April/2023)
Rotary Position Embedding (RoPE), the model's prediction becomes relatively blurrier and generally has larger error. This signifies the importance of relative position information in PDE operator learning.
5 Conclusion
In this work, we introduce OFormer, fully point- attention architecture for PDEs solution operator learning. The proposed model shows competitive performance on PDE operator learning and offers flcxibility with input/output discretizations. We also showcase that a trained data-driven model can learn to generalize to diverse instances of PDEs without imputting system parameters (NS-mix) and learn meaningful representation for system identification. The major limitation is that given few assumptions are made on the grid structure, the model needs sufficient data to reach its optimal performance, and linear attention mechanism is still relatively computationally expensive for higher resolution grids. In addition, just like most of the other data-driven models, the accumulated error of the model's prediction on long-term transient phenomena such as turbulent fluids is unnegloctable, unlike numerical solvers which are usually guaranteed to be stable.
6 Acknowledgments
This work is supported by the start-up fund from the Department of Mechanical Engineering, Carnegie Mellon University, United States.
References
Martin Alnæs, Jan Blechta, Johan Hake, August Johansson, Benjamin Kehlet, Anders Logg, Chris Richardson, Johannes Ring, Marie E Rognes, and Garth N Wells. The fenics project version 1.5. Archive of Numerical Software, 3(100), 2015.
Anima Anandkumar, Kamyar Avizzadenesheli, Kaushik Bhattacharya, Nikola Kovachki, Zongyi Li, Burigede Liu, and Andrew Stuart. Neural operator: Graph kernel network for partial differential equations. In ICLR 2020 Workshop on Inlegration of Deep Neural Models and Differential Equations, 2020. URL
https://openrevieet http://ppwnereee..eeffffooroooee
net/ net:foorumiddfgg22niia
Jimmy Lei Ba, Jamie Ryan Kiros, and Geolfrey E. Hinton. Layer normalization, 2016. URL https:
//axxiv. //axxiv.rg//bbsee
Damilry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate, 2014. URL hnttps://arxi: org/abs/ /abb//499... 0473
Saakaar Bhalnagar, Yaser Afshar, Shaowu Pan, Karthik Duraisamy, and Shailendra Kaushik. Prediction of aerodynamic flow fields using convolutional neural networks. Computational Mechanics, 64(2):525- 545, Aug 2019. ISSN 1432-0924. doi: 10.1007/500466-009-0074444 URL /ttp://///..oo/111000////// 600466-019-01700-0
Kaushik Bhattacharya, Bamdad Hosseini, Nikola B. Kovachki, and Andrew M. Stuart. Model Reduction And Neural Networks For Parametric PDEs. The SMAI journal of computational mathematics, 7:121-157, hwtp://www..ogrrreeeeeeeeeeeee/e
2021. doi: (0.5502//mmai-jmm.77) URL stttp:://mmai-. jem.
smai-jem 74/
Stella Biderman, Sid Black, Charles Foster, Leo Gao, Eric Hallahan, Horace He, Ben Wang, and Phil Wang. Rotary embeddings: A relative revolution, 2021. [Online; accessod ].
Johannes Brandstetter, Daniel E. Worrall, and Max Welling. Message passing neural PDE solvers. International Conference on Learning Representations, 2022. URL htttp:::/opaarrrewwweee///////
vSix3HPYKSU
Steven L. Brunton, Bernd R. Noack, and Petros Koumoutsakos. Machine learning for fluid mechanies. Anmual Review of Fluid Mechanics, 52(1))477-508, 2020. doi: fuud:/aanrre---i--000110001 URL
https://doi.n 10. 1146/ 1111/////Aamm----------000000000
13

---

Published in Transactions on Machine Learning Research (April/2023)
Shengze Cai, Zhicheng Wang. Lu Lu, Tamer A. Zaki, and George Em Karniadakis. Deepm&mmet: Inferring the electroconvection multiphysics fields based operator approximation by neural networks. Journal of Computational Physics, 436:110296, 2021. ISSN 0021-9991. doi: https:/ /to.://///ooo///////// 10tp
URL httpp://vvs www.ssscieeeeccccee http:///www..ooooooo....c./o///a
Shuhao Cao. Choose a transformer: Fourier or galerkin. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (ods.), Advances in Neural Information Processing Systems, 2021. URL
Tianping Chen and Hong Chen. Universal approximation to nonlinear operators by neural networks with arbitrary activation functions and its application to dynamical systems. IEEE Transactions on Newral Networks, ((4):911-917, 1995. doi: 10.1109/72.39222. 72.392253.
Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, David Belanger, Lucy Colwell, and Adrian Weller. Rethinking attention with performers, 2020. URL htttt:tt////rttttt httpp::/aaxx...r//////////// org httg/abb/22009
14794.
Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks, 2016. URL pestps://arxily org/ hase//a///// 1612.08083
Filipe de Avila Belbute-Peres, Thomas D. Economon, and J. Zico Kolter. Combining differentiable pde solvers and graph neural networks for fluid flow prodiction. In Proceedings of the 37th International Conference on Machine Learning, ICML'20. JMLR.org, 2020.
Jacob Devlin, Ming-Wol Wei Chang, Kenton Lec, and Kristina Toutanova. Bert: Pro-training of deep bidirectional transformers for language understanding, 2018, URL http:pss///rrxxx.oo////abb///
04805
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesuikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. URL
Weinan E and Bing Yu. The deep ritz method: A deep learning-based numerical algorithm for solving variational problems. Communications in Mathematics and Statistics, 6(1):1-12, Mar 2018. ISSN 2194-671X. doi: 10.1007 10.1007//00304401--002---
Nicholas Geneva and Nicholas Zabaras. Transformers for modeling pwiihlplpphpsllla systems. Neural Networks, 146:272 289, feb 2022. doi:
URL http:::://bi..org//101110///// neunet
2021.11.00
Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley, Oriol Vinyals, and George E. Dahl. Neural message passing for quantum chemistry, 2017. URL https://arxiv.orrt wwwpr///ss//0/..... 0111//11110.112
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Yee Whye Teh and Mike Titterington (eds.), Proceedings of the Thirteenth International Conference Artificial Intelligence and Statistics, volume of Proceedings of Machine Learning Research, pp. 249 256, Chia Laguna Resort, Sardinia, Italy, 13-15 May 2010. PMLR. URL https://proceeenngs mlr, prrrp/sss/ v9/g 3e/gglooot10...a html
Alex Graves, Greg Wayne, and Ivo Danihelka. Neural turing machines, 2014. URL https://arxiv.or abs/1410.54D
John Guibas, Morteza Mardani, Zongyi Li, Andrew Tao, Anima Anandkumar, and Bryan Catanzaro Efficient token mixing for transformers via adaptive fourier neural operators. In International Conference on Learning Representations, 2022. URL httpes///ppprrrreeweeee//trrrrrn
A3
Gaurav Gupta, Xiongye Xiao, and Paul Bogdan. Multiwavelet based operator learning for differential equations. In M. Ranzato, A. Beygelzmer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (ods.), Advances in Neural
Information Processing Systems, volume 34, pp. 24048 24062. Curran Associates, Inc., 2021. URL https
pdf
14

---

Published in Transactions on Machine Learning Research (April/2023)
Xu Han, Han Gao, Tobias Pffaf, Jian-Xun Wang. and Li-Ping Liu. Predicting physics in mesh-reduced space with temporal attention, 2022. URL httts:://arxiii al/////s/2222
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, 2015. URL
03385
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus), 2016. URL https://axxiv.rr//
abs/1606.085 08415
Jun-Ting Hsieh, Shengjia Zhao, Stephan Eismann, Lucia Mirabella, and Stefano Ermon. Learning neural PDE solvers with convergence guarantecs. In International Conference Learning Representations, 2019. URL https:://ppenreviee..eess httpp//aeeeiee..eee/oommmaaaaaaa
Valerii lakovlev, Markus Heinonen, and Harri Lähdesmäki. Learning continuous-lime {pde)s from sparse data with graph neural networks. In International Conference on Learning Representations, 2021. URL
Andrew Jaegle, Felix Gimeno, Andrew Brock, Andrew Zisserman, Oriol Vinyals, and Joao Carreira. Perceiver: General perception with iterative attention, 2021. URL https://axxix http://axxxvvv...aaaaa. htt//bbs21003 03206
George Karniadakis, Yannis Kevrekidis, Lu Lu, Paris Perdikaris, Sifan Wang, and Liu Yang. Physics-informed machine learning. pp. 1-19, 05 2021. doi: 101103//s425::400--001111-
Yuehaw Khoo, Jianfeng Lu, and Lexing Ying. Solving parametric pde problems with artificial neural networks. European Journal of Applied Mathematics, 32(3):421-435, 2021. doi: 101007/50966722200011
Georgios Kissas, Jacob H. Seidman, Leonardo Ferreira Guilhoto, Victor M. Preciado, George J. Pappas, and Paris Perdikaris. Learning operators with coupled attention. ArXiv, abs/2201.01032, 2022.
Dmitrii Kochkov, Jamie A. Smith, Ayya Alieva, Qing Wang, Michael P. Brenner, and Stephan Hoyer. Machine computational fluid dynamics. Proceedings of the National Academyy of Sciences, 118(21)::21111184111 2021. doi: 1..077/ppas.21000 10t.ppaasspppss....m...mm abs/10. 1073/ pnas.ppas... 2101784118
URL https: https://www..r pnas. org/ org/doi/
Nikola Kovachki, Zongyi Li, Burigede Liu, Kamyar Azizzadenesheli, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Neural operator: Learning maps between function spaces, 2021. URL https:
//axxiv /arraww.org/aas/2200..4.4
Aditi Krishnapriyan, Amir Gholami, Shandian Zhe, Robert Kirby, and Michael W Mahoney. Characterizing possible failure modes in
neural networks. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances Neural Information Processing Systems, volume 34, pp. 26548 26560. Curran Associates, Inc., 2021. URL htttps://proceedingss wwwpediiinnnnn.uuiiieee neurips cwturip..ccppppr///000 pc/p1/ pdf
Juho Lee, Yoonho Lee, Junglaek Kim, Adam R. Kosiorek, Seungjin Choi, and Yee Whye Teh. Set transformer: A framework for attention- iraention-bbsed permutation-invatti neural networks, 2018. URL https://arxiv.org
abs/1810.00m 00825
Kookjin Lee and Kevin T. Carlberg. Deep conservation: A latent-dynamics model for exact satisfaction of physical conservation laws. Proceedings of the AAAI Conference on Artificial Intelligence, 3511):277-285, May 2021. URL https://ojs. aaai. www..arg/aaaaa...a... wergg/indx..... phtp/AAA//articl//iiee////
Yunzhu Li, Hao He, Jiajun Wu, Dina Katabi, and Antonio Torralba. Learning compositional koopman operators for model- based control. In International Conference on Learning Representations, 2020a. URL https://openreviee hett/foruu11---11122Z20.
Zijic Li and Amir Barati Farimani. Graph neural networrk-cccllomtoo lagrangian fluid simulation. Computers & Graphics, 103:201-21 2211, 2022. ISSN 0097-8493. doi: http:pt/ts//dooo.////0..0../ w0w.:///.......2...2.0.00... URL
15

---

Published in Transactions on Machine Learning Research (April/202)
Zijie Li, Kazem Meidani, Prakarsh Yadav, and Amir Barati Farimani. Graph neural networks accelerated molecular dynamics. The Journal of Chemical Physics, 156(14):144103, 2022. doi: 10.1063/5.0083000
URL https://doi. http////...r//11.001000000.00000 00063/50008360
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Andrew Stuart, Kaushik Bhattacharya, and Anima Anandkumar. Multipole graph neural operator for parametric partial differential equations. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Newral Information Processing Systems, volume 33, pp. 7755-6766. Curran Associates, Inc., 2020b. |ttpss Paper pdf
Zongyi Li, Nikola Borislavov Kovachki, Kamyar Azizzadenesheli, Burigode liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. In International Conference on Learning Representations, 2021a. URL http:://oppnreviee.nnn//rrro
1d=c8P9NQVtmn0
Zongyi Li, Hongkai Zheng, Nikola Kovachki, David Jin, Haoxuan Chen, Burigede Liu, Kamyar Azizzadenesheli, and Anima Anandkumar. Plysics-informod neural operator for learning partial differential equations, 2021b. URL http:a///aari..oor//a///////111
Lu Lu, Pengzhan Jin, Guolei Pang, Zhongqiang Zhang, and George Em Karniadakis. Learning nonlinear operators via deeponet based on the universal approximation theorem of operators. Nature Machine Intelligence, 3(3):218 229, Mar 2021a. ISSN 2522-5839. doi: 10.1033/-42235-001--000000 7/ãoi. 1ttp://0000000/2222000000111-110
URL https:
Lu Lu, Xuhui Meng, Zhiping Mao, and George Em Karniadakis. Deepxde: A deep learning library for solving differential equations. SIAM Review, 63(1):208-22) 222,2,,00228,,... 2021b. doi: 10.1137/19M12740677 URL https:
Lu Lu, Xuhui Meng, Shengze Cai, Zhiping Mao, Somdatta Goswami, Zhongqiang Zhang, and George Em Karniadakis. A comprehensive and fair comparison of two neural operators (with practical extensions) based on fair dala. Compuler Methods in Applied Mechanics and Engineering, 393:1 14778, 2022. ISSN 0045-7825. doi: /t/p/////////00////2220000011111
http:///dd...rr11.1111/////// 202...0022.114775
URL sttp:::/www..iieecediieeee
com/
Minh-Thang Luong, Hicu Pham, and Christopher D. Manning. Effective approaches to attention-based neural machine translation, 2015. URL hattps://arxiv> arg/abs/1508 04025
Bethany Lusch, J. Nathan Kutz, and Steven L. Brunton. Dcep learning for universal linear embeddings of nonlinear dynamics. Nature Communicautions, 911):4950, Nov 2018. ISSN 2041-1723. doi: 10.1038/
s41467-018-072100- URL httpp:///ooo.ooo//00000001//1111
Winfried Lötzsch, Simon Ohler, and Johannes S. Otterbach. Learning the solution operator of boundary
value problems using graph neural networks. 2022. doi: 10.48550/ (0.48550/ARXIV.22000 2wW.IV..20..144990 URL https:
//axivv. wwwriv.org/abs//20 a20/aabs.../.2////////////eee 14092
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and
Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis, 2020. URL https: //axxiv.or//ab///00 08934
Jeremy Morton, Antony Jameson, Mykel J Kochenderfer, and Freddie Witherden. Deep dynamical modeling and control of unsteady fluid flows. In S. Bengio, H. Wallach, H. Larochelle, K. Grau- man, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL .pff
Vinod Nair and Geoffrey E. Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th International Conference on International Conference on Machine Learning, TCML/10, pp. 807-80 814, Madison, WI, USA, 2010. Omnipress. ISBN 675160555.77..
16

---

Published in Transactions on Machine Learning Research (April/2023)
Nicholas H. Nelsen and Andrew M. Stuart. The random feature model for input-output maps between banach spaces. STAM J. Sci. Compul.; 43:A3212-A332 A3243, 2021.
Francis Ogoke, Kazem Meidani, Amirreza Hashemi, and Amir Barati Farimani. Graph convolutional networks applied to unstructured flow field data. Machine Learning: Science and Tecknology, 2(4):045020, sep 2021. doi: 10.1088 12.1088/2632-20 201/5/263222222///1 2153/aclfc9. URL https: htw.:///orrr//0000///2//22222220
Francisco Palacios, Juan Alonso, Karthikeyan Duraisamy, Michael Colonno, Jason Hicken, Aniket Aranake, Alejandro Campos, Sean Copeland, Thomas Economon, Amrita Lonkar, Trent Lukaczyk, and Thomas Taylor. Stanford University Unstructured (SU"2): An open-source integrated computational environment. for multi-physics simulation and design. In 51st ATAA Aerospace Sciences Meeting including the New Horizons Forum and Aerospace Exposilion. American Instit ute of Aeronautics and Astronautics, 2013. doi: 20.2514/6.2013-22 287. URL httpp:://aoi.
2013--2001112 287
Shaowu Pan and Karthik Duraisamy. Physics-informett probabilistic learning of linear embeddings of nonlinear dynamics with guaranteed stability. SIAM Jourmal on Applied Dynamical Systems, 19(1):480 509, 2020.
doi: 10.1137 119111922277 URL httpss://ol. /tt///1.11111 1117719911127225
Pranshu Pant, Ruchit Doshi, Pranav Bahl, and Amir Barati Farimani.
Deep learning for reduced order modelling and efficient temporal evolution of fluid sinulations. Physics of Fluids, (3(10)1107101, 2021. doi:
10.1063/5.0062550 URL https:///oi. org/o0...... 10633...013/.............. 1664650566600
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjic Bai, and Soumith Chintala. Pytorch: An imperative style, high-performane deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Informa- tion Processing Systems 32, pp. 8024 8035. Curran Associates, Inc., 2019. URL http::/papers. http///apprr..uuippsspppp stgaaaiiieeetlll------ee--
neurips .pff
Jaideep Pathak, Shashank Subramanian, Peter Harrington, Sanjeev Raja, Ashesh Challopadhyay,, Morteza Mardani, Thorsten Kurth, David Hall, Zongyi Li, Kamyar Azizzadeneshelli, Pedram Hassanzadeh, Karthik Kashinath, and Animashrce Anandkumar. Fourcastnot: A global data-driven high-rosolution weather model using adaptive fourier neural operators, 2022. URL
11214
Tobias Pfaff, Meire Fortunato, Alvaro Ganzalz-Gonzalez, and Peter W. Battaglia. Learning mesh- based simulation with graph networks. 2020. doi:
URL https://arxiy org/ org/abs/
0010.03400
Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. In J. Platt, D. Koller, Y. Singer, and S. Roweis (eds.), Advances in Newral Information Processing Systems, vol- ume 20. Curran Associates, Inc., 2007. URL httpps:/prroceedings.oo www.:iiwnnnn.rrrr...ooo.... wttpp/pee/p/////ii/... pdf..
Md Ashiqur Rahman, Zachary F. Ross, and Kamyar Axiwadenesheli. U-no: U-shaped neural operators, 2022.
URL httpa///aaxxivvoo/aa///////
M. Raissi, P. Perdikaris, and G.E. Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378:686-7000 707, 2019. ISSN 0021-9991. doi:
Priyabrata Saha and Saibal Mukhopadhyay, A deep learning approach for predicting spatiotemporal dynamics from sparsely observed data. IEEE Access, 9:64200-64210, 2021. doi: 10.109/AACCES55.021.3111
Alvaro Sanchez-Gonzalee Jonathan Godwin, Tobias Pfaff, Rex Ying, Jure Loskovoe, and Peter W. Battaglia. Learning to simulate complex physies with graph networks, 2020. URL htttp:://axiv.orr//ee
09405
17

---

Published in Transactions on Machine Learning Research (April/2023)
Andrew M. Saxe, James L. McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks, 2013. URL https://arxiw.orr///aeo//
6120
William E. Schiesser. The numerical method of lines: Integration of partial differential equations. 1991.
Yidi Shao, Chen Change Loy, and Bo Dai. Sit: Simulation transformer for particle-based physics simulation, wtt.://oooormmnd-iiibbeellllll
2022. URT.
Justin Sirignano and Konstantinos Spiliopoulos. Dgm: A deep learning algorithm for solving partial differential equations. Journal of Compulational Physics, 375:1339-1364 2018. ISSN 0021-9991. doi: http://oo..rr//1100110///////// 201..00..... URL p11/50021999116305577
Kim Stachenfeld, Drummond Buschman Fielding, Dmitrii Kochkov, Miles Cranmer, Tobias Pfaff, Jonathan Godwin, Can Cui, Shirley Ho, Peter Battaglia, and Alvaro Gonzzlez. Learned simulators for turbulence. In International Conference on Learning Representations, 2022. URL httpps://penrrevlew
net/forrun'id<mmsBo]]jiz
Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfong Liu. Roformer: Enhanced transformer with rotary position embedding, 2021. URL attps://arxixs www./rgg//////e///e//..//.eeeeee abs//a/s///000//////////2////1aa .05///94444008444
Luning Sun, Han Gao, Shaowu Pan, and Jian-Xun Wang. Surrogate modeling for fluid flows based on plysics-constraaned deep learning without simulation data. Computer Methods in Applied Mechanics and Engineering, 361:112732 2020. ISSN 0045-7825. doi:
URL
Uya Sutskever, Oriol Vinyals, and Quoe Le. Sequence to sequence learning with neural networks. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger (eds.), Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc., 2014. URL htttps://proceedingg .dd.
Naoya Takeishi, Yoshinobu Kawahara, and Takehisa Yairi. Learning koopman invariant subspaces for dynamic mode dccomposition. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS'17, pp. 1 1130- 1140, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781510860964.
Matthew Tancik, Pratul P. Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T. Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains, 2020. URL http:///arra....rrra///////..../ 10739
Alasdair Tran, Alexander Mathews, Lexing Xie, and Cheng Soon Ong. Factorized fourier neural operators, 2021. URL https://arxiv. http://aaxxxv..g/abbaaa//sa/ 13802
Tapas Tripura and Souvik Chakraborty. Wavelet neural operator: neural operator for parametric partial differential equations, 2022. URL https://arxii: wttp://www.rr/bb///.....02
Kathryn Tunyasuvunakool, Jonas Adler, Zachary Wu, Tim Green, Michal Zielinski, Augustin Zidek, Alex Bridgland, Andrew Cowie, Clemens Meyer, Agata Laydon, Sameer Velankar, Gerard Kleywegt, Alex Bateman, Richard Evans, Alexander Pritzel, Michael Figurnov, Olaf Ronneberger, Russ Bates, Simon Kohl, and Demis Hassabis. Highly accurate protein structure prediction for the human proteome. Nature, 596: 1-9, 08 2021. doi: 10.1038//111586-01--033228
Dmitry Ulyanvv, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing ingrodient for fast stylization, 2016. URL
abs/1607 08022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https: htttps://proceednng. meereerrddngg.n.nnnip.se.... 2itee03255//de31110004444442a
neurips. cctppape//////////1...
.pff
18

---

Published in Transactions on Machine Learning Research (April/2023)
Petar Veličkovié, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pictro Lió, and Yoshua Bengio.
Graph attention networks, 2017. URL htttp::/arxiv.orrg/ wttp://wwww../aa//....00.... 10903.
Rui Wang, Karthik Kashinath, Mustafa Mustafa, Adrian Albert, and Rose Yu. Towards thyyscc--iformmn deep learning for turbulent flow prediction. In Proceedings of the 26th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, KDD '20, pp. 1457-1466, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 9781450379984. doi: 10. 10.1145/33944866440333 URL https://doi.org org/ 10. 11..11113334446...200
Sifan Wang, Hanwen Wang, and Paris Perdikaris. Learning the solution operator of parametric partial differential equations with physics-informed deeponets. Science Advances, 7(40))cabi8605, 2021a. doi:
10.1126/scaadv.abb88000 URL /ttt:/ttpp////orrreaaaaeeeeooo/0 http:////.......//.i........aaa
Sifan Wang, Hanwen Wang, and Paris Perdikaris. Improved architoctures and training algorithms for doep operator networks, 2021b. URL
Sifan Wang, Hanwen Wang, and Paris Perdikaris. On the eigenvector bias of fourier feature networks: From regression to solving multi-scale pdes with physics- informed neural networks. Computer Methods Applied Mechanics and Engineering, 384:113938, 2021c. ISSN 0045-7825. doi: httpss///doi.orr/1...o 10160. .021
113938. URL http:///ww..ccccccccc.cc.o/ooaaa
Gege Wen, Zongyi Li, Kamyar Azizzadenesheli, Anima Anandkumar, and Sally M. Benson. U-fno an enhanced fourier neural operator- based deep-learning model for multiphase flow. Advances Water Resources, 163:104180, 2022. ISSN 0309-1 1788. doi: http:///b...ogg//1000///////////
200.: 2022.104180. URL
Steffen Wiewel, Moritz Becher, and Nils Thürey. Latent space physics: Towards learning the temporal cvolution of fluid flow. Computer Graphics Forum, 38, 2019.
Ronald J. Williams and David Zipser. A learning algorithm for continually running fully recurrent neural networks. Newral Computation, 1(2):270-280, 1989. doi: 10.1162/neco. 1989.1.2.270.
10.1162//000.199111.220..0.
Yinhao Zhu and Nicholas Zabaras. Bayesian deep convolutional encoder-decoder networks for surrogate modeling and uncertainty quant ificatioon Journal of Compulational Physics, 366:415-444 2018. ISSN 0021- 9991. doi: httpss///oi..rg//// 10111/0000000111////////111 htt.://..p......0..... URL http:pes//a....ccccccceerrrrreoe
art.cle/ppL/50021999188444
19

---

Published in Transactions on Machine Learning Research (April/2023) 2023)
Appendix Summary
Section
A
Model implementation details;
Section
B
Baseline details;
Section
Further ablation study;
Section
Dataset details;
Section
Further results and visualization;
A
Model implementation details
Below provide the full implementation details of models used in different problems. All the models are implemented in PyTorch (Paszke et al. 2019
For notation, we use [d1, d2] to denote a linear /hidden layer with input dimension d1 and output dimension d2, which corresponds to mn.,inear(d1, d2) in Py Torch. For attention layer, d x indicates that there are h attention heads each with a dimension d.
A.1 Hyperparameter for equi-spaced grid problem
Input encoder top
Input encoder bottom
Problem
SA hidden dim
SA FFN
# SA Block
1D Burgers
[2, 96
96x1
[96, 192, 96
4
[96, 96)
2D Darey flow
[3, 96
96x4
[96, 192, 96
[96, 256]
2D Navier-Stokes
12, 128)
128x1
[128, 256, 128]
5
1188, 192]
Table 5: Architecture of input encoder. SA denotes self-attention, FFN denotes feed forward network.
Query encoder top
Query encoder bottom
Problem
CA hidden dim
CA FFN
1D Burgers
[1, 96, 96]
96x8
196, 192, 96]
2D Darcy flow
2, 256, 256
256x4
[256, 512, 256
2D Navier-Stokes
[2, 192, 192]
192x4
[192, 384, 196)
[192, 384]
Table 6: Architecture of query encoder. CA denotes cross-altention.
Problem
Propagator
Decoder
Total # params (M)
ID Burgers
[96, 96, 96, x6
196, 48, 1]
0.67
2D Darcy flow 2D Navier-Stokes
[256, 128, 64,
2.51
[384, 384, 384, 384]
[384, 192, 96, 1]
1.85
Table 7: Architecture of propagator and decoder. The number of parameters is the total number including encoders. For 1D Burgers' equation we use unshared MLPs for propagating the dynamics.
A.2 Hyperparameter for irregular grid problem
We GELU (Hnndrycks & Gimpel 2016 activation function for propagator and decoder, and opt for Gated-GELU DDuphhin et al. 2016) for FFN. On 1D Burgers, we train the model for 20k iterations using a batch size of 16. On Darey flow, we train the model for 32K iterations using a batch size of 8. For small dataset on Navier-Stoker (consisting of 1000 training samples), we train the model for 32k iterations using batch size of 16, and 128k itcrations for large datasct (consisting of roughly 10k training samples). On Electrostatics Magnetostatics we train the model for 32k iterations using batch size of 16. On Airfoil we train model for 50k iterations using batch size of 10.
20

---

Published in Transactions on Machine Learning Research (April/2023)
Input encoder top
Input encoder
Problem
SA hidden dim
SA FFN
# SA Block
bottom
2D Electrostatics
[11, 64, 64]
64x1
[64, 128, 64
2
[64, 64
2D> Magnetostatics
11, 96, 96]
96 1
[96, 128, 96
2
[96, 96]
2D Airfoil
(6, 128, 128]*
128x1
[128, 256, 128]
5
[128, 128]
Table 8: Architecture of input encoder. The input of 2D airfoil contains an additional time dimension of length lin, so the very top layer Conv2D layer with kernel size (lan. 1) stride (lin, 1) and no padding.
Query encoder top
Query encoder bottom
Problem
CA hidden dim CA FFN
SA hidden dim
2D Electrostatics
[3, 64, 64] [3, 96, 96]
64x4
664, 128, 64)
64x1
2D Magnetostatics
96x4
[96, 192, 96]
96x
2D Airfoil
[2, 128, 128]*
128x4
[128, 256, 128]
128x1
[128, 256]
Table 9: Archilecture of query encoder. On non-uniform grid, add self-attention layer after cross-attention layer. For 2D airfoil, we introduce a learnable embedding for each node type (boundary, open area, airfoil)
Problem
Propagator
Decoder
Total # of params (M)
2D Electrostatics
664, 32, 32, 3]
0.17
2D Magnetostatics
996, 48, 48, 3]
0.31
2D Airfoil
[384, 256, 256, 256
255,, 128, 128, 4]
1.35
Table 10: Architecture of propagator and decoder. The number of parameters is the total number including imput/ hupur/query encoders. For airfoil, before imputting into propagator block, node type embedding are concate- nated together with other features.
For training on the 2D Navier-Stokes and Airfoil problem, we adopt a curriculum stralegy grow the prediction time steps. During the initial stage, instead of predicting all the future states towards the end of time horizon, e.g. Wto ,....... WT, we truncate the time horizon with a ratio y (heuristically we choose y 0.5), train the network to predict ut, ......,, M.yr The motivation is that at the initial stage the training dynamics is less stable and gradient is changing violently, we truncate the time horizon to avoid stacking very deep recurrent neural network and thus alleviate the unstable gradient propagation. In practice, we found this makes the training more stable and converge slightly faster.
As suggested CCoo 2021 for scaling-sensitive problem the Burgers' equation, the initialization of the query /value projection layer has great influence on the model's performance. We use orthogonal initialization (Saxe al.] 2013) with gain 1/d and add an additional constant 1/d the diagonal elements (with d being the dimension of latent dimension at each head). The orthogonal weights have the benefit of preserving the norm of input and we found that with proper scaling it provides better performance than standard Xavier uniform initialization. Please refer to the Appendix C for the study on the influence of initialization.
For most of the problems we studied, we use instance normalization in the attention and observe slight improvement compared layer normalization. However, for 2D Electrostatics/ Magnctostaties' Poisson equation, we find that instance normalization is unstable as the grid points in this problem are much fewer (60 250) than other problems, we adopt layer normalization for this problem.
21

---

Published in Transactions on Machine Learning Research (April/2023)
B Baseline details
We use the open-sourced official implementation of Fourier Neural Operator (FNO ' Multiwavelet Operator D Gupta et al. 2021 and Mnberkin/Fourice Transformer (G.T./F.T. 2021h to carry out the benchmark. The experiments on these baselines are carried out following the settings in the official implementation with minimal changes.
Li ct al.
2021a CCoo
For FNO, when applying to 2D Navier-Stokes problem with relatively large number of data samples, we increase the number of modes in Fourier Transformation to 12 and increase the width (hidden dimension of network) to 32. This notably improves FNOO2DD's performance. For G.T./F.T., on 2D Navier-Stokes problem we increase the hidden dimension size from 64 to 96, and extend the training epochs from 100 to 200 (150 on large dataset with 10000 training samples).
Below we provide a computational benchmark of different models' training and inference efficiency on 2D Navier-Stokes equation. The benchmark was conducted on RTX-3090 GPU using Py PyTorch 1.8.1 (1.7 for MWT) and CUDA 11.0. The number of training samples is 10000, with time horizon T 30, and the batch size is set to 10.
Model
Iters / sec
Memory (GB)
# of params (M)
FNO-2D
9.71
0.33
2.37
FNO-3D
10.74
0.42
6.56
MWT
1.67
10.95
179.18
G.T.
1.79
16.65
1.56
OFormer
1.89
15.93
1.85
Table 11: Training computational cost of different models.
Pseudo-spectral
Model
OFormer
G.T.
FNO-2D
FNO-3D
MWT
Fine
Coarse
Time (sec)
2.32
3.22
0.58
0.50
2.21
5735.74
3510.97
Table 12: Inference computational cost of different models and ground truth numerical solver. The time indicates how long does it take for each model to finish simulating 200 sequences (data processing time is excluded). The pseudo-spectral method taken from (Li et al. 2021a), which implemented in PyTorch Fine grid has a resolution of 256 256 while coarse grid has a resolution of 64 x 64.
Note that as many traditional numerical solvers do not support GPU or not optimized for GPU computation, and they can trade-off accuracy for faster runtime (e.g. adopting a coarse grid and time step), so is hard conclude the exact acceleration of learned solvers on different problems, but in general we can observe that for time-dependent system learned solvers highly efficient they tolerate very large time step size (usually 3-4 orders of magnitudes compared to solver).
C Further ablation study
In this section we present further ablation study for the model.
Sparse reconstruction and prediction We study the model's performance under diffcrent discretization settings. At the training stage, if some of the input points (with probability 0.2 and dropping ratio ~ u([0, 0.5])) are randomly dropped, the model's performance are observed to deteriorate on most of the datasots. This indicates that while attention is flexible with respect to the input discrotization, its best performances are usually obtained by exploiting the very specific grid structures. But for dataset that contains more diverse system parameters, like NS-mix and NS-mix2, the random dropping has little influence
htttp://gfthubb.cm///uas/////// ooproto operator
http:://gthal.cm/ccommt////////p
22

---

Published in Transactions on Machine Learning Research (April/2023)
model's performance. While models trained with random dropping usually tend to perform worse than models trained with fixed grid points, they are much more robust on irregular discretization as shown in the Table 14 and can still generate reasonable prediction based on 25% of the input points.
Dataset
No random drop
Random drop
Performance diff.
NS1 NS2-part
0.0104
0.0133
0.0029 (27.9%)
0.1702
0.1773
0.0071 (4.2%)
NS2-full
0.0646
0.0672
0.0026 (4.0%)
NS3
0.1697
0.1747
0.0050 (2.9%)
NS-mix 0.1400
0.1413
0.0013 (0.9%)
ID Burgers 0.00126
0.00142
0.00016 (12.7%)
Table 13: Influence of randomly dropping some input grid points during the training process. No random drop indicates during training input and output grids are fixed at 64 x 64; random drop indicates that some are the input grid points are randomly dropped during training, while output grid is still fixed at 64 x 64. Error measured by relative C2 norm. The resolution of Burgers' equation 2048.
Training strategy
All
95%
75%
50%
25%
Random drop
0.0672
0.0687
0.0772
0.0939
0.1294
No random drop
0.0646
0.0795
0.1293
0.2065
0.3774
Table 14: Performance of models NS2-full's test set with randomly sampled input points. Percentage the top row indicates the ratio of sampled points' amount with respect to total number of all points. Error is measured by relative C2 norm on comparing full-resolution (64 x 64) prediction with ground truth.
Influence of initialization On Burgers' equation, to emulate the energy decaying scheme and preserve the magnitude of how much the energy has decayed, no data normalization or normalization layer (except for the instance normalization inside every attention layer) is used. While this improves model's performance, it also makes the model very sensitive to the initialization of projection matrix inside each attention layer, especially for the matrix where no instance normalization is applied (for Fourier type attention, it's the value matrix; for Galerkin type attention, it's the query matrix).
Orthogonal
Xavier uniform
271.45
165.13
Relative Es norm (x10 3)
1.25
1.40
1.36
1.35
Scaling-diagona bias
scaling Initialization method
No scaling scalingidiagonal
Figure 9: Influence of different initialization scaling scnsitive problem (1D Burgers' equation), using Fourier-type attention.
We modulate the initialization process of the weight of query (or value if using Fourier type attention) projection matrices (W dddd) in Galerkin type attention as:
W (g) cB + +
(17)
23

---

Published in Transactions on Machine Learning Research (April/2023)
where is scaling factor which set to 1/d, B is the original weight matrix (initialized either via orthogonal initialization (Saxe et al. 2013) or Xavier uniform (Glorot & Bengio, 2010)), and similar to Cao (2021) we add diagonal bias 51 to the matrix. The orthogonal initialization initializes a weight matrix that is norm proserving and projects the imput feature vectors onto orthogonal embedding vectors. As shown Figure due presence of skip connection: = z + Atin(z) and there is no normalization layer, both iniiialiition methods blow up and have much worse performance when no scaling is applied. Orthogonal initialization tend to perform the best when both scaling (set gain to 1/d) and diagonal bias applied.
Influence of data OFormer's performance is highly correlated with the amount of data available, especially for relatively complicated problem like Navicr-Stokes equation. As shown in Figure TO the performance of the trained OFormer improves almost linearly as data size grow exponentially. At low data regime (500-5000), OFormer benefits significantly from increasing the data amount. This indicates that data is crucial for OFormer to unlock its optimal approximation power and also demonstrates OFormer's strong capacity assimilate large data. However, this also hints potential bottleneck of OFormer that it might not have satisfactory performance when there is limited data.
0.20
Relative C2 norm
0::702
X
0.15
0.10
0,0852
D 0009
0.35
0.90
1000
2000
5000
10000
Training data amount (log scale)
Figure 10: Performance of OFormer on NS2 when trained with different amount of data.
lc
-2
Basis learned with MLP that does not contain any transformation on the top layer.
le 4
-2
(b) Basis learned with a MLP that has RFF on the top layer.
Figure 11: Visualization bases learned (the output of coordinate-based network in the decoder part) on 2D Darcy flow. The visualized bases are randomly chosen.
Influence of Random Fourier Feature We study the influence of Random Fourier Features (RFF) Recht 2007) on different problems 1D Burgers, 2D Darcy flow and Navier-Stokes. As shown in Figure 12 applying RFF to the input coordinates boosts the performance all
Taaniik et al.] 2020 Rahimi
24

---

Published in Transactions on Machine Learning Research (April/2023)
three problems. We hypothesize the main reason is the original MLP tends to learn set of bases with lower frequency while RFF broadens the spectrum of these earned bases (Tancik et al. 2020). As an example, Figure
reveals that without RFF, learned bases have relatively smoother pattern than those with RFF.
0.10
0.0329
0.020
Relative C. norm
Relative L2 norm
0.015
0.0152
Relative L2 norm
0.08
0.0712
0.0646
0.0126
10-2
0.06
0.010
0.04
0.02
0.0013
0.000
0.00
None
RFF
None
RFF
Type of mapping
Type of mapping
Type of mapping
(a) Performance on Burgers' equation with 2048 resolution.
(b) Performance on Darey flow with
(c)
Performance on NS2-full.
141 141 resolution.
Figure 12: Ablation on the influence of RFF.
0.30
0.018
Relative E, noo
0.014
Relative C2 norm
0.012
0.15
0.008
0.10
0.008
0.05
10
15
20
25
30
35
40
45
50
11
13
15
17
19
21
23
25
27
29
Time (seconds)
(seconds)
(a) Error trend NSL.
(b) Error trend NS2-parl.
0.12
0.250
Rel aevvvive Es horm
0.10
Relative E3 herm
0.200
0.08
0.175
0.06
0.04
0.300
0.075
11
1.3
15
17
19
21
73
75
27
79
11
13
17
Time (seconds)
Time (seconds)
(c) Error trend on NS2-full.
(d) Error trend on NS3.
Temporal trend of error We investigate the temporal (rend of OFormer's prediction error on time- dependent system (Figure 13] by computing the mean error model's prediction across all test sequences at particular time step. In general, we observe that the error will accumulate exponentially as time grew. The compressible flow's error (Figure 13f exhibits some levels of periodicity due the intrinsic periodic property of the flow. Increasing the number of training samples alleviate the error, but does not change the
25

---

Published in Transactions on Machine Learning Research (April/2023)
0.25
0.10
Relative Es norm
0.20
Relative L, norm
0.15
0.07
0.10
0.08
0.05
0.05
11
13
15
17
21
23
25
27
29
Time (seconds)
Time (seconds)
(e) Error trend on NS-mix.
(f) Error trend on Compressible flow around airfoil.
Figure 13: Error trend on different time-dependent systems.
exponential trend. How to alleviate the compounding prediction error for neural sperator/PDii solver can be interesting and important future venue.
Dataset details
D.1 Data generated equi-spaced mesh
Gupta et al. (2021) where the data is generated using equi-spaced simulation grid, and then dow nownmpeeeed to create dataset with different.
Following datasots are used as benchmark in Li ct al. (2021a) Cao 20221
resolution.
Burgers' Equation
The nonlinear Burgers' equation with the following form is considered:
Óeulz, 1) uudzu(x, vd t), x 10, E (0,1),,000 >0 u(x, (z)).
(18)
where и viscosity coefficient, the equation is under periodic boundary conditions, and initial condition un (z) sampled from Gaussian Random Field prior. The viscosity coefficient is set 0.1. The learning objective is an operator mapping from initial condition uo(z) the solution time 1, i.e., G up u(- 1). The data is generated via split step method on 8192 resolution grid. At each time step, the state of the system is first advanced with the diffusion term and then the convection term (both calculated via Fourier transform), using the forward Euler scheme.
Darcy Flow
The steady-state 2D Darcy flow equation considered here takes the form:
-V (a(z))v((()) = f(x), 1) un(x) 0, 000,1)²
(19)
where fx) is the forcing function (sct to constant 1) and we aim to learn the operator mapping diffusion coefficient a(-) to the solution u(-), i.e., G a + u. The coefficient function is sampled from a Gaussian Random Field prior with zero Neumann boundary conditions on the Laplacian kernel of the field. The data is generated via. second-order finite difference solver 421 421 resolution grid.
Dataset and numerical solver courtesy: htttps://gsthubb.cmm////////////
cectiieer_eurra__eee _pprrator
26

---

Published in Transactions on Machine Learning Research (April/2023)
Navier-Stokes Equation fluid reads as:
The vorticity form of 2D Navier Stokes equation for the viscous, incompressible
0wl(x,) b) + t(z Vw(a, t) = nAw(z, t) f(x),
xe (0,1),,t ,E (0.T),
V-u(z, 0, xE (0,1)),te te (0,7), w(r, = wo(a), x =
(20)
where w vorticity, u represents velocity field, and v is the viscosity, source term is set as: f(x) 0.1 (sin(2x(11 + æ2)) + (2s + x2))), and initial condition wz( sampled from a Gaussian random field. The equation is under periodic boundary conditions. The goal is to learn operator mapping the initial states of the vorticity field to the future time steps, i.e., G : w(,, (le[[,,10) + w(+, (le Generally, lower viscosity coefficients // result in more chaotic dynamics and makes the learning more challenging.
The data is generated by solving the stream-funetion (Φ u V x e)) formulation of above equation using pseudo-spectral method on a 256 256 grid. At each time step, the stream function is first calculated by solving a Poisson equation (V2p) -w). Then the non-linear term is calculated by differentiating voriticity in the Fourier space, dealiased and then multiplied with velocities in the physical space. Lastly, the Crank-Nicolson scheme is applied to update the state.
D.2 Data generated on non-equi-spaced mesh
Poisson equation
The 2D Poisson equation for electrostatics and magnetostatics defined as:
Electric: (²U(x) p(x))
x
c(x)
(21)
Magnetic: - (=))())=(())ddd x JEΩ
where p the charge density, the permittivity of the material, U the electric potential, μ is the permeability of the material, 1g is the component of current density vector and A₂ is the component of magnetic potential. The goal is to learn the solution operator of the Dirichlet boundary value problem: g (p.c) U (U(x) for 02) for electrostatics and G (I,, + Ag (A.(z) for 02) for magnetostaties. In the problem we studied, <(-). are held constant.
The data is generated by solving the above equation using finite element method implemented with FEniCS library AAnæs et al., 2015) 5 The element used is a quadratic triangle element. We use the pre-generates data from Lötzsch et al. (2022 for experiment.
Euler equation of compressible flow
The Euler equation of compressible flow takes the form:
0,p + V (pu) = f1,
@e(p)) + V (pu @ u + pl) = fg,
O((pF) + V (pFu + pu) = S3, p ,(x, u u(s, (), .p p(x, 1): x Ωt t [0,
(22)
where p is the density, u is the velocity field, p is the pressure, E is the total energy per unit mass (which can be closed via p (y 1)p [E - 0.5(u uu)), and fi, fz, fa are generic source terms. No-penetration condition is imposed at the airfoil. The solution operator of interest is: g {u, htl.P:lllensssoo.oo H {u, p. w...lee....000.7......
The dataset is generated by solving the above equation using finite volume method built in the SU2 library Palaacios et al. 2013). We use the pre-generated datase from Pfaff et al. 2020 to carry out the experiment. formulate this problem
Note that different from Pfaff et al. (2020), problem and use much large time slep size (Ai 0.192). In Pfaff et al. (002)), the learned solver is designed to learn the mapping: + U44i with At 0.008 and then rollout the simulation with a Markovian setting.
non-Markovian initial value
Dataset and numerical solver courtesy: htwp://wwwo//.../......o///////e Dataset courtesy:
27

---

Published in Transactions on Machine Learning Research (April/2023)
E Further results and visualization
In this section we present the visualization of model's prediction on different problems. All error plots are based on point-wise absolute error.
14
2.0
22
24
15
06
Figure 14: Model's prediction on 1D Burgers' equation, samples are randomly selected from test set. Green dotted lines denote the ground truth, and red lines denote model's prodiction.
28

---

Published in Transactions on Machine Learning Research (April/2023)
0.0150
Ref.
0.0100
0.0075
0.0050
Pred.
0.00075
0.00050
Err.
0.00025
0.010
Ref.
0.006
0.004
Pred.
0.002
0.000
0.0004
Err.
0.0002
Figure 15: Model's prediction on 2D Darcy flow, samples are randomly selected from test set.
T=14 18
22
26
30
38
46
Ref.
Pred.
0.02
Err.
Ref.
Pred.
Err.
0.02
Figure 16: Model's prediction NS1's test set (v le 3), with Reynolds number around 20.
29

---

Published in Transactions on Machine Learning Research (April/2023)
T-12
20
24
28
Ref.
Pred.
0.50
Err.
Ref.
Pred.
Err.
0.4
Figure 17: Model's prediction on NS2-full (v = le - 4), with a Reynolds number around 200.
T=11
12
17
19
Ref.
2
Pred.
Err.
Ref.
o
Pred.
Err.
Figure 18: Model's prediction on NS3 (v = le 5), which has a Reynolds number around 2000.
30

---

Published in Transactions on Machine Learning Research (Apri//222)
Velocity-x
T=1.92s
2.88s
3.84s
4.8s
150
Ref.
Pred.
-50
-100
20
Err.
10
Velocity-y
Ref.
-100
-10
Pred.
200
10
Err.
31

---

Published in Transactions on Machine Learning Research AApril/2233
Velocity-x
T=1.92s
2.88s
3.84s
4.8s
200
Ref.
Pred.
50
-100
40
Err.
30
20
10
Velocity-y
250
200
Ref.
150
100
50
-50
Pred.
40
Err.
30
32

---

Published in Transactions on Machine Learning Research (April/2023)
Velocity-x
T=1.92s
2.88s
3.84s
4.8s
200
Ref.
150
100
Pred.
-50
-100
50
40
Err.
20
Velocity-y
Ref.
-30
100
-150
Pred.
Err.
40
Figure 19: Model's prediction the compressible flow around airfoil. The test root mean squared error of predicted momentum is 15.477 at plotted region (multiplying velocities with density on each node).
33

---

Published in Transactions on Machine Learning Research (April/2233)
Electric
Magnetic
Ref.
(a)
Pred.
Err.
Ref.
Pred.
(b)
Err.
Pred.
(c)
Err.
Figure 20: Model's prediction (interpolated from non con-uniform grid points): (a) potential; (b) vector field's
component; (c)vector field's component.
34


Reviews:
[{"review_id": "9T2T25Zjwu", "reviewer": "Reviewer_duoN", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n- The connection of the cross attention with DeepONets is interesting.\n- Novel treatment for spatial-temporal problems using a shared propagator.\n- Comprehensive numerical experiments.\n\n### Weaknesses\n- The writings could enjoy some further polishing; see also request changes below for some suggestions.\n- No comparison with DeepONets in the non-equispaced case.\n- The usage of Rotary Position Embedding is not supported by a specifically designed experiment.\n- The modifications to the existing architectures lack theoretical motivations.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper further advances the methodology of PDE operator learning using Transformers proposed in (Cao, 2021). The main contribution is the adaptation of several techniques popularized by the Transformer research in NLP/CV, such as cross attention and relative positional encodings, to the context of PDE operator learning to make inference on non-equispaced grid possible. Overall it is a worthy addition to the literature if the writings and presentation can be improved (see below).", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- add a summary of the contribution after the introduction or literature review.\n- The number of parameters in DeepONets depends on the number of query points (grid size), since the final summation is with respect to the number of query points. While for Transformer there is no such dependence, the final sum is with respect to the embedding dimension $d$ that has nothing to do with the discretization size. It would be nice to add this clarification.\n- add an ablation study of the first layer (random Fourier projection vs vanilla MLP).\n- some clarification needs to be added for RoPE vs Euclidean coordinates. In (7) it is said that the random Fourier projection with Euclidean coordinates is used as the first layer, later on page 7 the Rotary Positional Embedding is used. As positional encodings are simply coordinates of discretizations in operator learning, if Euclidean coordinates are used in the first layer, where exactly is RoPE used? It would be better to add it on the diagram.\n- a table or a plot of numerical values of relative errors at different time steps should be given, instead of heat maps.\n\n### Things that need to be clarified\n- Page 2: \"an general architecture\".\n- Page 3: \"$d$ being the dimension of vectors\", here $d$ should be embedding/latent dimension or channels. \n- Page 3: $\\mathbf{q}_i\\cdot \\mathbf{k}_s$ should be approximating the kernel evaluated at $k(x_i, x_s)$.\n- Page 4: the writing on mathematics can be improved. For example, \"$q_l(y_{i'})$ as the trunk net\" is inaccurate as $q_l(y_{i'})$ is a scalar while a neural net is a mapping.\n- Page 4: is it LayerNorm or instance normalization used in the model?\n- Page 5, equation (8): in the first equation, $\\mathbf{z}'$ is used before being defined.\n- Page 8, magnetic and electric potentials are different, the current notation is confusing as the same $u$ is used. Moreover, in 2D, it is better to use the $\\text{rot}$ operator instead of $\\nabla \\times$. For example in Girault-Raviart, $\\nabla^\\perp$ is used in 2D.\n- Page 24: suggestion on expressing operator maps: it is better to use $\\mathcal{G}: a\\mapsto u$ directly instead of $\\mathcal{G}: a(x)\\mapsto u(x)$ as $a$ denotes a function while $a(x)$ means sampling $a$ at $x$.  \n- Page 25: The FEniCS's reference is not quite appropriate (currently given as (Scroggs et al., 2022)). \"The element used is a 2-degree triangle element\". I am not sure what \"2-degree\" is referring here.\n", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "B3dNA4y1XV", "reviewer": "Reviewer_nrTx", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- S1. The use of attention to solve PDEs is interesting, specially, taking advantage of the work by Cao to use attention without softmax to save computations.\n- S2. The model enables independent input-output meshes, as well as irregular grids.\n- S3. The proposed model has similar performance to existing methods.\n- S4. The experimental section is well done and honest, having an ablation study as well as a small qualitative study of the latent space.\n\n**Weaknessess**\n- W1. Writting could be improved. For example, the architecture description is not super clear. It took me several reads and looks at the code to understand how all the different parts of the model interconnect.\n- W2. Having a different input and output grid seems really odd/niche to me. Indeed, to my understanding this is not utilized at all in the experiments.\n- W3. While not seeking sota, the scenarios to prefer the proposed model over others are scarce (irregular grids where mesh-augmentation is not an option).\n- W4. To my understanding the main contribution is the usage of a Transformer (as latent marching has been used in the past), making the contributions rather weak.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work presents a deep learning model to solve partial differential equations (PDEs) using attention mechanisms. The model first encodes the input grid $x_1, \\dots, x_n$ into a key $K$ and value $V$ matrices using self-attention, and then encodes a different evaluation grid $y_1, \\dots, y_m$ into a query $Q$ matrix. The model then uses cross-attention to generate a latent representation for each instance $y_i$, which is then fed through an MLP $T$ times to simulate $T$ time steps of the equation. Finally, another MLP generate the prediction for the input $y_i$. \n\nThe main advantange of this work over others lays in the use of attention to encode the input, which allows the model to have different discretizations for the input and output grids, as well as using irregular grids. The latter could be achieved before with GNNs, yet the authors show that their architecture (OFormer) outperforms a proposed GNN architecture when no mesh-augmentation is applied.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "**Questions:**\n- Q1. Why not using relative encoding for the input grid ($x$) too?\n\n**Required changes**\n1. Clarity should be improved. The model should be well described (e.g., explain what is the output of each module, add notation to the figures, cross-attention should be described somewhere, etc.).\n2. If one of the advantages is the ability to use two grids, this should be showed/exploited in some way during experimentation.\n3. A clear motivation/advantage of using the proposed model over existing ones should still be found. While I know this is non-actionable, I can point out to some directions that come to mind. \n- Equation (4) shows that $K^\\top V$ could be pre-computed. Could you pre-compute it using the training data, and drop the first part of the encoder during testing?\n- Following on the previous point, what if the first part of the model contained information about the target, such that you could exploit training information?\n\n**Desirable changes**\n\n4. The connections with Transformer should be made clearer. To my understanding, _the entire encoder_ is a Transformer as in the original paper, with the only difference being that there is only positional encodings as inputs. This interpretation would ease a lot the encoder description.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "UyqdesiKQD", "reviewer": "Reviewer_dyW3", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Operator learning is an important problem\n- The paper is easy to follow\n- The model seems to perform well in the large-data, irregular-grid setting\n\nWeaknesses:\n- The model does not perform so well on the standard, regular-grid setting\n- It is unclear how relevant the irregular-grid setting is in practice\n\nMajor comments:\n- All the experimental results are lacking error bars, with makes it very hard to assess how significant the differences between the methods are. I would expect that the experiments would be run at least 3 times and for standard errors to be reported.\n- Since the proposed OFormer only performs well in settings where the dataset is large, the grid is irregular and no mesh augmentation is used, it would be important to motivate when this setting is realistic. Are there any scenarios in practice where mesh augmentation could not be used? Similarly, are there real-world datasets where the grids are irregular and the query points are in different locations than the measured points? Maybe in weather/climate data or so? It would be good to demonstrate the performance of the OFormer on at least one such dataset.\n\nMinor comments:\n- Eq. (5): maybe use a different index than $i$ in the sum in the numerator, since it is easy to confuse with $i'$\n- Eq. (8): should it say $z' = z^0 + \\dots$?\n- Sec. 4.1: imcompressible -> incompressible", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose a new model to learn operators for PDEs from observed data. Their model, named OFormer, encodes the observed points and query points independently and then uses a cross-attention architecture to learn latent encodings. It then uses a recurrent architecture in the latent space to evolve these encodings forward in time. The authors show that their proposed model does not outperform the baselines on standard PDE benchmarks, but that it can lead to advantages when used on irregularly sampled grids, where the datasets are large and no mesh augmentation is used.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "As mentioned above, I would like to see error bars in the experimental results and at least one real-world dataset where the OFormer performs well.", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors have successfully addressed all the review comments, and all the reviewers agree that the paper now meets the bar, and the audience of TMLR will be interested in reading it. 
Decision: Accept (poster)