Non-equispaced Fourier Neural Solvers for PDEs
Authors: Haitao Lin
Yongjie Xu
Lirong Wu
Yufei Huang
Siyuan Li
Guojiang Zhao
Stan Z. Li
Venue: Submitted to ICLR 2023 (2023)
Keywords: Neural Operators
PDE Solvers

Abstract:    Solving partial differential equations is difficult. Recently proposed neural resolution-invariant models,  despite their effectiveness and efficiency, usually require equispaced spatial points of data. However, sampling in spatial domain is sometimes inevitably non-equispaced in real-world systems, limiting their applicability. In this paper, we aim to propose a Non-equispaced Fourier PDE Solver (\textsc{NFS}) with adaptive interpolation on resampled equispaced points and a variant of Fourier Neural Operators as its components.  Experimental results on complex PDEs demonstrate its advantages in accuracy and efficiency. Compared with the spatially-equispaced benchmark methods, it achieves superior performance with $42.85\%$ improvements on MAE, and is able to handle non-equispaced data with a tiny loss of accuracy. Besides, to our best knowledge, \textsc{NFS} is the first ML-based method with mesh invariant inference ability to successfully model turbulent flows in non-equispaced scenarios, with a minor deviation of the error on unseen spatial points.     


Under review as a conference paper at ICLR 2023
NON-EQUISPACED FOURIER NEURAL SOLVERS FOR PDES
Anonymous authors Paper under double-bling review
ABSTRACT
Solving partial differential equations is difficult. Recently proposed neural resolution-invariant models, despite their effectiveness and efficiency, usually require equispaced spatial points of data. However, sampling in spatial domain is sometimes inevitably eew--eauuideedddeeeeeeeeee in real-world systems, limiting their appli- cability. In this paper, we propose a Non-equispaced Fourier PDE Solver (NFS) with adaptive interpolation on resampled equispaced points and variant of Fourier Neural Operators as its components. Experimental results on complex PDEs demonstrate its advantages accuracy and efficiency. Compared with the spatially- equispaced benchmark methods, it achieves superior performance with 42.85% improvements on MAE, and is able to handle non-equispaced data with a tiny loss of accuracy. Besides, to our best knowledge, NFS is the first ML-based method with mesh invariant inference ability to successfully model turbulent flows in non-equispaced scenarios, with a minor deviation of the error unseen spatial points.
INTRODUCTION
Solving the partial differential equations (PDEs) holds the key to revealing the underlying mechanisms and forecasting the future evolution of the systems. However, classical numerical PDE solvers require fine discretization in spatial domain to capture the patterns and assure convergence. Besides, they also suffer from computational inefficiency. Recently, data-driven neural PDE solvers revolutionize this field by providing fast and accurate solutions for PDEs. Unlike approaches designed to model one specific instance of PDE (E & Yu 2017 Bar & Sochen 2019; Smith al.) [2020] Pan & Duraisamy 2020; RRaisii et al., 2020). neural operators (Guo et al. 2016; [Sirignano & Spiliopoulos 2018
Bhatnagar al. 2019 KHOO et al. 2020 Li et al. 2020bb Bhattacharya et al. 2021 Brandstetter et al., 2022, 2022, Lin et al. 2022) directly learn the mapping between infiite-dimensionall spaces of
functions. They remedy the mesh-dependent nature of the finite-dimensional operators by producing single set of network parameters that may be used with different discretizations.
However, two problems still exist discretization-inneee computational inefficiency compared with convolutional neural networks in the
modeling for non-equispaced data and
setting. To alleviate the first problem, MPPDE (Brandstetter et al., 2022) lends basic modules in MPNN (Gilmer et al., 2017) model the dynamics for spatially non-equispaced data, but even intensifies the time complexity due to the pushforward trick and suffers from unsatisfactory accuracy in complex systems (See Fig. 2(a)). FNO (Li al., 2020c) has achieved success tackling the second problem of inefficiency and inaccuracy, while the spatial points must be equispaced due to its harnessing the fast Fourier transform (FFT).
To sum up, two properties should be available in neural PDE solvers: (7) discretizationiinaariii (2) padispace-unnceessityy Property (1) is shared by infiite-dimensionall neural operators, in which the learned pattern can be generalized to unseen meshes. By contrast, classical vision models and graph spatio-temporal models are not iitreeiiaationinvarriae handle pregularly-samplle spatial points. For example, graph spatio-temporal models do not require the data to be equispaced, but vision models are equispace-necesseee
and
Property (2) means that the model can
and limited to handling images as 2-d regular grids. And recently proposed methods can be classified into four types according to the two properties, as shown in Fig. [ As discussed, although the equispace- equispace-necesssary methods enjoy fast parallel computation and low prediction error, they lack the ability handle the spatially non-equispaced data. For these reasons, this paper aims to design a invvaraita model (defined in Fig. T called Non-equispaced Fourier neural Solver (NFS) with comparably low cost of computation

---

Under review as a conference paper at ICLR 2023
Equispace Unnecessity
Yes
Discretization Invariance
Vision Mixer
Graph spatio-temporal model
No
Finite-dimentional operator
Training
Inference
Training Graph spatio-temporal model (e.g. DCRNN, GraphPDE)
Inference
Vision/Video model (e.g. VIT, GFN)
Yes
Infiniteddimentioall operator
Training
Inforence
Training
Inference
Resolution-invariant model
Mesh-invariant model (e.g. MPPDE, MGKN)
(e.g.FNO) FNO)
Spatial point model
Spatial
by model
Figure 1: Four types of methods with without the two concluded limitations.
and high accuracy, by lending the powerful expressivity of FNO and vision models to efficiently solve the complex PDE systems. Our paper including leading contributions is organized as follows:
Sec. 2 we first give some preliminaries on neural operators as related work, with a brief introduction to Vision Mixers, to build a bridge between Fourier Neural Operator and Vision Mixers. Thus, we illustrate our motivation for the work: To establish mesh-invariant neural operator, by harnessing the network structure of Vision Mixers.
In Sec. we proposed Non-equispaced Fourier Solver (NFS), with adaptive interpolation operators and a variant of Fourier Neural Operators as the components. Approximation theorems that guarantee the expressiveness of the proposed interpolation operators developed. Further discussion gives insights into the relation between NFS, patchwise embedding and multipole graph models.
In Sec. 4 extensive experiments different types of PDEs are conducted to demonstrate the superiority of our methods. Detailed ablation studies show that both the proposed interpolation kernel and the architecture of Vision Mixers contribute to the improvements in performance.
BACKGROUND AND RELATED WORK
2.1 PROBLEM STATEMENT
Let D E Rd be the bounded and open spatial domain where n. nt,-point discretization of the domain D written as X (e; (a(") ....., (.! < S n. are sampled. The observation of input function E A(D; Rd,,) and output a E (D; Rdx) the n., points denoted by {a(æ:), ut u(=))) where A(D; Rd and LDD; Rd..)
separable Banach spaces of function taking values in Rdd and RR.. respectively. Suppose x ~ μ is i.i.d. sampled from the probability measure supported on D. An infinite-dimensional neural operator Go A(D; Rd.) (D; Rdd)) parameterized by θ, aims to build an approximation so that Go (a) R u. A cost functional C ((; R&d.,) x U(D; Rdd.. + R is defined to optimize the parameter of the operator by the objective
min E Eg (CGo(()) (() RJ lEe
C(Cola),
(1)
To establish a mesh-invariant operator, can be non-equispaced, and the learned Go should be transferred to an arbitary discretization X' D, where X' can be not necessarily contained in X. Because we focus on spatially non-equispaced points, when the PDE system is time-dependent, assume that timestamps (t,} are uniformly sampled, which means do not focus temporally irregular sampling or continuous time problem (Rubanova et al. 2019 Chen et al. 2019 Cagatay
Yildız al. 2019
lakovlev et al. 2020
2

---

Under review as a conference paper at ICLR 2023
2.2 DISCRETE FOURIER TRANSFORM
Let by (K(") ....., ....,hi ( the 1-th frequency corresponding to X, with ky Z The discrete Fourier transform of f: D + Rdd is denoted by (k)) E Cdr, with F as its inverse, then
F()) (a:)
())))))))) Σ
(2)
where j means the j-th dimension of f. General Fourier transforms have complexity Onn When the spatial points are distributed uniformly on equispaced grids, fast Fourier transform (FFT) and its inverse (IFFT) (Rader & Brenner 1976) can be implemented to reduce the complexity to O(n, n.).
2.3 FOURIER NEURAL OPERATOR
Neural Operators. To model one specific instance of PDEs, a line of neural solvers have been designed, with prior physical knowledge as constraints. Different from these methods, neural operators (Lu et al. 2021; Nelsen & Stuart (021) require knowledge of underlying PDEs, and only data. Finite-ding dimensional operator methods (Guo et al. [2016] Sirignano & Spiliopoulos/ 2018]] Bhatnagar et al. 2019: KHOO al. 2020) are discretizationnvarian
meaning that the model can only learn the patterns of the spatial points which have been fed to the model in the training process. By contrast, infinite-dimensional operator methods (Li et al., 2020b C (Bhattacharya et al. 202T
Brandstetter et al.) 2022) are proposed to be iissretizationninaaiieee enabling the learned models to generalize well to unseen meshes with zero-shot.
Kernel integral operator method (Li et al. 2020a) is a family of infinite-dimensional operators, in which (G)(a))<>) = Q oT o... o o (())(x) is formulated as an iterative architecture. A disber-dimensional representation function is first obtained by v = P(a) E U(D; Rd)), where P is a shallow fully- colne-con network. It updated by
r (x) == oWW' (a:) + (a))'()))))
Va D
where Kgp A L(L)) is kernel integral operator mapping, mapping a bounded linear operators, with parameters Φ. W is a linear transform and a is a non-linear activation function. After the final iteration, Q projects o (x) back U(D;
Fourier Neural Operator (FNO) Li et al., 2020c) as a member in kernel integral operator methods, updates the representation by applying the convolution theorem as:
(4) where R& as the Fourier transform of periodic kernel function Kg. is directly learned as the parameters in the updating process. To be resolution- inttiionniivrrantt,, FNO picks a finite-dimensional parameterization by truncating the Fourier series of both F(v) and R. as maximal number of modes (1(1) for 1 < S d. Because the sampled spatial points are equispaced in FNO, it can conduct FFT and IFFT to get the Fourier series, which can be very efficient.
(a(a))(=)== F (F((ss) (()) (R, F()))
2.4 VISION MIXER AND GRAPH TENTIO-EMMPORAL MODEL
Vision Mixers (Tolstikhin et al. 1021 Rao et al. with a stack of (token mixing)= (channel mixing) vision tasks. They are based on the assumption that the key component for the effectiveness of Vision Transformers (VIT) (Dosovitskiy et al.) 2020) is attributed to the proper mixing of tokens. The defined tokens are equivalent to equispaced spatial points the former definition, and the research on the mixing of them can be an analogy to modeling the proper interaction or pesssage-passig patterns among spatial points. In specific, VIT uses a non-Mercer kernel function (Wright & Gonzalez 2021 Kg to adaptively learn the pattern of gressage-passing through the iterative updating process
2021 Guibas al. 2021 are a line of models (token mixing) as their network structure for
et (x: (ChannclMMxx TokenMix(u (x))))
(okenMix(c(z))
Σ Ki (2 v(x), v(x,)) v(z;); (hannelMix(v(())) We(x), (5)
where W is linear transform called channel mixing layer because it transforms the input on the channel of an image whose dimension is equivalent to function dimension dy. Note that we omit the residual connection in Eq. (3) for simplicity.
3

---

Under review as a conference paper at ICLR 2023
Remark. The FNO can be regarded member of the family of Vision Mixers. The reason is that component in an iteration in Eq. (4) can be written as ( (Rg. (() = Rs E, 2nicm,a,> (((,,, lying
because in the equispaced scenarios, E; can be regarded scaling. The kernel Kg is parameterized by Ki (a 2i u( v(x;)) ) = e 22,,<,,., and the matrix multiplication of R& also performs mixing on channels. Besides, the inverse Fourier transform can also be regarded as token mixing layers, or so-called token demixing layers (Guibas et al. 2021
the same grids k after
However, the powerful fitting ability and efficiency of Vision Mixers are limited being applied to non-equispaced spatial points. Another option for non-equispaced data is graph spatio-temporal models, which interaction patterns among spatial points are modeled in a graph message- passsing
way (Gilmer et al. 2017 Atwood & Towsley [016) Defferrard et al. 2017). The mechanism is similar to the token mixing in Vision Mixers by means of the summation Eq. 5 conducted the predefined neighborhood of each point. Unfortunately, the graph spatio-temporal models (Seo et al., 2016) Li et al., 2018| Bai et al.
2020;] [in et al., (02T) suffer from high computational complexity
and unsatisfactory accuracy in solving complex dynamical systems (such turbulent flows).
2.5
MOTIVATION
Since FNO belongs Vision Mixers, this firstly raised question to us: Do models employing Vision Mixers's architecture have the potential to model complex PDE systems? Thus, experiments are conducted to give an intuitive explanation of our motivation as shown in Fig. [ The data are generated by Navier-Stokes equations. It is noted that graph spatio-temporal methods can also handle the equispaced data. Detailed setup is given in Sec.
HINI (C:1
CCN
LOBRY
AUTRA
NOYSE
130
200
300
500
(a) Equispaced comparison
(b) Non-equispaced comparison
(c) Loss Validation Set
Figure 2: Intuitive explanation of our motivation: In (a) and (b), 's' represents Vision Mixers, 'O' represents graph spatio-temporal models and 'O' is the proposed NFS. In (c), 'Eq' and 'Neq" mean the methods are trained in equispaced and non-equispaced scenarios respectively.
We find that (I) All of the evaluated Vision Mixers are able to model the dynamical systems effectively, in spite of FNO as the only pntrettizationinvariian model; (2) The complex dynamics of the systems are hardly captured by graph spatio-temporal models, whose performance on both accuracy and computational efficiency is very unsatisfactory in either equispaced or non-equispaced scenarios. Fig.
the problem of infeasibility of graph spatio-temporal models through the loss curves on the validation set, compared with Vision Mixers. However, Vision Mixers fail to handle the non-equispaced data. Therefore, we aim to (/) establish a mesh-invariant model, by harnessing the network structure of Vision Mixers to achieve competitive efficiency and effectiveness in equispaced scenarios, as shown in Fig. ((a) (2) Besides, it should allow applicability and comparable accuracy in non-equispaced scenarios for solving PDEs, as shown in Fig. 2(b)
PROPOSED METHOD
3.1 NON-EQUISPACED FOURIER TRANSFORM
Nonuniformly signals are unavoidable in certain real-world physics scenarios, such as signals obtained by meteorological stations the earth surface, which urge the fast Fourier transform (FFT) to be extended to non-equinnee eenisppude data with efficient implementation of FFT. Non-equisp FFTs usually rely mixture of interpolation and the judicious use of FFT, where the calculations of interpolation are no more than (n, log n.) operations (Kllmmaar et al.. 2012 Cheema et al. 2017). For example, I Lagrange interpolation is used approximate the signal values on m. resampled equispaced points
4

---

Under review as a conference paper at ICLR 2023
NEI Layer
NEI
FNO Layers
Equispaced signals
Equispaced signals
Precefined
Adaptiely leamaed interpoaaionn
interpolation
Nonequépaced signals
Fourier Tokwn Moking
Channel Kieng
Fourer Token No
Damicng
R4
Transfom
Prodefined interpcition kernll
Adrepivelly learned
interpotation
Nonequispaced signals
Figure 3: The architecture NFS: In non-equispaced inperpolation (NEI) layers, the interpolation kernels are adaptively learned rather than predefined, and the interpolated equispaced signals are processed through stack of FNO layers with the same structure of Vision Mixers.
{a, (ssssssm,, and then implement FFT on the interpolated points. A low rank approximation with complexity of O(n, log(1/o) is used to replace the interpolation with complexity of O(n)) with as the precision of computations (Dutt & Rokhlin 1995). Another example is commonly used Gaussian interpolation (Kestur al.) (010). Denote J as equispaced FFT in particular, and H as the interpolation operator, and the proposed non-equispaced FFT can be written as
my
(F (kf))(()) ~ T<ck>
Σ (( -x)).
(6)
-x;) interpolates values on resampled points via convolution with the periodic heat kernel 1 y) Elega (x-1))/// /1r with constant. Multiplication of the inperploation matrix (Hi)).s h, (x, (x:-2;) and the signal vector (f); f(x;) includes mn,. se O(n)) operations. For the kernel h,, it is summation of Gaussian kernel, and convolution with a single Gaussian in each points a;'s neighborhood (xx:) can yeid a tiny error depending on т, so the interpolation operator can be approximate via H())(a;)) Ea, Eaye) [(z;),, (æ: 2z)) Restricting the neighbor number [N(x))) < log nt, leads the complexity to reduce O(n, log n.).
3.2 NON-EQUISPACED FOURIER NEURAL PDE SOLVER
Non-equispaced interpolation. To harness the effectiveness of FNO, we use non-equispaced Fourier token mixing instead of the equispaced It generalizes the equispaced FFT in Eq. (4) as
F(v) = (F (a))
We denote H.y :A L(U) the interpolation operator mapping, which maps parametric function to bounded interpolation operator. (a, gets the inerploated values on m, resampled equispaced points via the convolution with kernel h1
(H,(()))(E,)) = - (x æ, x, a(x;)); The
(8)
where x, lies resampled equispaced grids. Another H' interpolates back on the n, non-equispaced ones in the same way via the convolution with kernel hç. To reduce the operations to no more than Onn, log nt),, the summation restricted in the neighborhood æ; and æ;, such that [W(x)) s [W(x)) < log nt, with points. We formulate the kernel with a shallow feed-forward neural networks. Thanks to the universal approximation of neural networks, the following theorem assures that the interpolation operator approximate the representation function v arbitrarily well. (For detailed proof, see Appendix. AA.3 Empirical observations the convergence of interpolation operators are given in Appendix C.
predefined constant determining the neighborhood size of spatial
5

---

Under review as a conference paper at ICLR 2023
Theorem 3.1 (Approximation Theorem of the Adaptive Interpolation). Assume the setting Theorem A2 in Appendix A.3 is satisfied. μ is the probability measure supported on D. For u E L. suppose u LD, Rd for any < p < 0C. Then, given 0, there exist neural network by Rd x Rd x Rd Rdd, such that || ollu S c, where o(x) Sn h, y, a. (())))(
Applicability of Layer-Norm. As shown in Fig. 3] besides the comparison of the proposed interpo- lation operator with the traditional ones, a notable difference between the original FNO and FNO layers in our Vision Mixer architecture is the applicability of normalization layers (Layer-Norm) which is usually used in Vision Mixers' architecture. FNO cannot adapt Layer-Norm layers, because the change of resolution will make the trained normalization parameters and spatial points disagree with each other. In comparison, the resampled equispaced points are fixed in our architecture, no matter how the discretization of the input changes. Therefore, the normalization layers can be added, in a similar way to Vision Mixers, bringing considerable improvements (See Sec. 4.3)
Mesh invariance. In the intermediate layers, which adopt equispaced FNO, the resampled points are fixed in both training and inference process, invariant to the input meshes. In the interpolation layers, the operator H, (a) is discretization-inaarii newly observed signals a(x), its coordinate and resampled spatial points' relative coordinates 2j - x. In the same way, Hi((a) is also mesh-invariant. This allows the NFS achieve zero-shot mesh-invariant inference ability, which is demonstrated in Sec. 4.2
because the kernel can be inductively obtained by the
Complexity analysis. The complexity of FNO is On, log ny + ny.kmax). In the interpolation layers, because the interpolated values of resampled points are determined by their neighbors, we set the size of each resampled point's neighborhood in G and observed non-equispaced points's neighborhood in G' as [W(=:) R [axx))] S. clog(n.), for Si n,, < j S m.. And in this way, the sparsity of the interpolation matrix reduces the complexity of the two interpolation layers to (c n, log n. + m, 12gn.). ). If we set the resampled points number n,, the overall complexity is O2cc n, log n. + n, log n, + kixx) ~ O(n, log n. + kmxx)
3.3 FURTHER DISCUSSION
Relation to Vision Mixer. The interpolation can be compared to patchwise embedding in Vision Mixers. For example, MLPMIXER learns the token mixing patterns adaptively with a feed-forward network, but the high resolution of input images does not permit the global mixing of tokens due to the complexity of O Therefore, the input images are firstly rearranged into patches, with each patch containing np pixels. In this way, the complexity is reduced to O(n (nn/nn). enabling feasible token mixing. The patchwise embedding very similar interpolating the values on resampled points, as the former one first chooses patches' centers as n²/ /nţ resampled points, and 'interpolates" the resampled points by lifting the embedding dimension and the rearranging of their neighbors' values as the interpolated values, rather than using kernel.
Relation to multipole graph model. The adaptively learned interpolation layer in NFS has similar formulation of multipole graph models (Li et al.) 2020b). In multipole graph mod- els, the high-level nodes aggregate messages from their low-level neighbors as Hligh (x,) = [aa (m,) vLow (x,)),, x,, a(xx), a(x:)). Compared to multipole graph models, the values of high-level resampled equispaced nodes are approximated with low-level non-equispaced nodes' values in NFS, but nodes' values of low levels are given in multipole graphs. This causes differences in multipole graph models' wesssaee-psssngg and NFS's ' interpolation: In the former one, messages flow circularly among different levels of nodes, while in NFS, messages only exchange twice between the nodes of two levels one is from low-level non-equispaced nodes to high-level resampled equispaced nodes, and the other is the opposite.
4 EXPERIMENTS
4.1 EXPERIEMNTAL SETUP
Benchmarks for comparison. For mnite-dimensional operators, we choose Vision Mixers includ- ing VIT (Dosovitskiy et al.) 2020), GFN (Rao et al. (021) and MLPMIXER (Tolstikhin al.) 202T) as equispaced problem solvers, with DEEPONET- V and DEEPONET-U as two variants for DeepONet(Lu et al. 2021) and graph spatio-temporal models including DCRNN (Li et al. 2018).

---

Under review as a conference paper at ICLR 2023
AGCRN (Bai et al. 2020) and GCGRU (Seo et al. 2016) non con-equispaced problem solvers. For -ineni--amenionnal operators, the state-of-the-ar FNO Li et al. 2020c) for equispaced problems and MPPDE (Brandstetter et al. 2022) for non-equispaced problems are chosen. A brief introduction to these models is shown in Appendix. B. In Vision Mixers, the different timestamps in temporal axis are also regarded 'tokens' in that timestamps uniformly sampled.
Protocol. The widely-used metrics Mean Absolute Error (MAE) and Root Mean Square Error (RMSE) are deployed measure the performance. The reported mean and standard deviation of metrics are obtained through 5 independent experiments. All the models for comparison are trained with target function of MSE, i.e. C(u, (²) = [|u(x>) - f(x))]" corresponding to Eq. ( and optimized by Adam optimizer in 500 epochs. The hyper-parameters are chosen through carefully tuning on the validation set. Every trial is implemented on a single Nvidia V V100
4.2 NUMERICAL EXPERIMENTS
Data. We choose four equations for numerical experiments, three of which are time-dependent (KdV, Burgers' and NS), while the other one is not (Darcy Flows). For 1-d problem, we consider Korteweg de Vries (KdV) and Burgers' equation (given in Appendix. B.2
For 2-d PDEs, we consider Darey Flow (given in Appendix. B.2) and Navier-Stokes (NS) equation for a viscous, incompressible fluid in vorticity form on the unit torus:
(w t) = vAw a. t) + f(x). 8 B,w(x, ( t) + u( t)
V (xx,t) = 0, w(2, 0) = Wi w)(
where E [0, [0. u is the velocity field, vorticity, v E IR is the viscosity coefficient, and f is the forcing function.
V x u is the vorticity, wo is the initial
The total number of instances is 1 1200, with percentages of 0.7, 0.1 and 0.2 for training, validating and testing, respectively. The original simulated resolutions of PDE signals are 128 x 128 in NS equation. For others, see Appendix. B.2 When evaluating their performance in equispaced scenarios of different resolutions, we can downsample the resolution for training to low-resolution data, e.g. 64 x 64 in NS equation. To evaluate their performance in non-equispaced scenarios of different meshes, we randomly choose nı spatial points for training.
Table 1: MAE(X 3) comparison with vision mixer benchmarks.
Rurgers' (n.
10)
harcy Flow
MS (n.
NS
10)
512
512
1024
64
128
256
64
10
40
20
1
1
10
40
20
10
40
20
VIT
0,5042
2.4269
1,5327
0.5073
0.9865
1.1078
9.3797
22.8565
15,7398
3,9609
12.3433
9,3010
MLPMIXER
0.1973
0.4210
0.3303
0.4970
0.8909
0.9125
7.5246
15.8652
14,9360
3.1530
7.9291
7.7410
GFN
0,2383
0.4187
0,3500
0.4739
0,8659
0.9618
3.5524
10,2250
6.3976
1.7396
5,4464
3.1261
FNO
0.0978
0.1815
0.1430
8.9857
2.4076
DEEPONET-U
0.4471
1.9624
0.6541
0.3753
0.9488
0.9692
7,4912
16.0440
14.3476
3.4436
10.2950
7.1394
DUIPONET-V
0.4782
2.1707
1,6131
0.5119
0.9614
1.3216
8.6986
18.5561
16.0587
3.9745
12.3314
9.3471
NFS
0.0958
0,1708
0,1474
0.1497
0.2254
4.7882
0.8636
3.1122
1.8406
Table 2: MAE(X 3) comparison with graph spatio-temporal benchmarks.
Burgers' (it, 10)
Daruy
(14,
NS (r, 10)
Na
612
512
256
-0096
16384
1024
40366
4096
10244
4006
4096
1024
10
20
40
10
20
DCRNN
2.6122
8.5880
4.6126
1.7629
00M
1.8146
30.6756
88. 13.3382
52.1290
8,7025
59.6602
27.1069
AGCRN
4.6667
15.6143
10.4900
1.7336
00M
1.6938
0OM
00M
59.9393
00M
00M
42,4197
GCGRU
5,7653
3.1000
1.7403
00M
1.7633
85.9303
6.3570
57.2493
21.3537
MPPDE
1.1271
4.1213
2-4554
0.5608
0.6384
0.6675
8.9810
54.2387
20.7453
5,4353
42,3057
17.5902
NES
0.1085
0.1983
0.1634
0.1430
0.2379
0.1727
2.1992
4.7865
3.9178
0.9335
3.2768
1.82399
Performance comparison. In this part, for time-dependent PDEs, our target is to map the observed physical quantities from initial condition u(X, T) E X*,, where T = {tt :t < Thissnn to quantities at some later time u( X, TT' E R". where T" (t: T <<< t; < T')isisn We set the input timestamp number n, as (initial state to future dynamics) and 10 (sequence to sequence), and prediction horizon nř, 10, 20 and 40 short-, mid- and long-term settings. For Darcy Flows, which are independent time, we directly build an operator to map a to u. In equispaced scenarios, the resolution is denoted by pdر ns. where d is the spatial dimension. In non-equispaced scenarios,
7

---

Under review as a conference paper at ICLR 2023
the spatial points number is denoted by n2. The comparison of benchmarks with without equispace- unnecessity are shown in Table. and rexpectvvyy and detailed results including KdV equations with RMSE and standard deviations equispaced scenarios, the proposed NFS obtains the lowest error in most 1-d PDE settings, and in solving 2-d PDEEs, its superiority over other Vision Mixers are significant, with 42.85% improvements on MAE according to the trials of NS (r = 64, the evaluated graph spatio-temporal models' performance is unsatisfactory, especially in NS equations. In comparison, NFS achieves comparable high accuracy to the equispaced scenarios, for instance, according to columns of NS (r 64, ne 10, ni 40) with (n, 4096, 74 10, ni 40). (3) In some trials such as Burgers' (n, 1) in Table. B4 in Appendix, B.3 Vison Mixers including FNO also suffer from non-convergence of loss; while NFS can still generate accurate predictions. The explanation of the phenomenon will be our future work.
given in Appendix. B.3 It can be concluded that (1) In
10, n 40). (2) In non-equispaced scenarios,
C-D
I-10 b-20
L-O
t-10
t-20
4:50
625
640
0.45
30
0.
6.5
1.0
0.0
0.5
1.0
0.0
0.5
140
0.0
05
1.0
0.0
0.5
1.0
64
0.5
1.0
Ground truth on seen meshgrid
Ground truth on unseen meshgrid
8.004
5.022
-.002
0.004
0.5
88.006
Pradiction meshgrid
Prediction meshgrid
Figure 4: Visualization on non-equispaced NS equation: The training mesh (n, 4096) differs from the mesh in inference process (n', 8192). Appendix, B.4 gives more visualization.
Table 3: Results NS equation: MAE(x10 3) of NFS and different variants of NFS seen and unseen meshes. 'Flex + LN' is the proposed NFS. 'Flex' represents the flexible interpolation layer
defined in Eq. 8 'LN' is the Layer-Norm and 'Gaus" is the predefined Gaussian interpolation.
Appendix. B.5 gives details and results on other equations.
(n, 1096, 10, n²,
10)
(R, 1024, n4 10, ni
20)
(n, 4096, R. 10.
40)
Flex Gaus + LN
Flex + LN
Flex + LN
Gaus
Flex +
Flex
Gaus LN
Flex + LN
10,
0.9335
1.6341
1.2138
1.8239
2.1976
2.5119
3.2768
3,6422
5.6761
2n.
0.9731
2,8589
1,4882
2.3530
3.7465
7,0203
3.5439
3.9092
5.8975
3n, 1.1011
3.4513
1.6384
2,5179
5.7712
7.9177
3.6584
4.2102
6.6622
4n,
3.4357
2.5919
5.5990
7.1962
4.2628
6.6951
Mesh-invariance evaluation. We use (u(X, T), u(X, T))) as the training set, and evaluate the model's performance of mesh-invariant inference ability mesh with X C X'. The visualization results of NS (n, = 4096, nú 40) are shown in Fig. For fixed n', randomly sampled different X' for 100 times, to get the mean errors and standard deviations (given in Appendix. B.5) of different spatial meshes. We can conclude from Table. that (I) The errors on unseen meshes effects. However, the errors on unseen meshes are acceptable, since they are even lower than other models' prediction error on seen meshes. (2) Larger n', leads to higher prediction error because large number of unseen points are likely to disturb the learned token mixing patterns. On the other hand, NS (n, 1024, ni = 10) implies that small spatial point numbers of training meshes (n.) hinder model's generalizing ability on unseen meshes, due to excessive loss of spatial information.
X', where | = n'.- X' is different
larger than the errors on seen meshes, showing the overfitting
4.3 ARCHITECTURE ANALYSIS
Two modules in NFS differ from FNO. The first is the interpolation layers at the beginning and the end of the architecture. The second is the extra Layer-Norm in the FNO layers, which can be applicable in NFS thanks to its fixed resampled equispaced points, but inapplicable to FNO for preserving its resolution-invariance. We aim figure out what makes NFS outperform FNO.
8

---

Under review as a conference paper at ICLR 2023
Effects of neighborhood sizes. It is widely believed that modeling the long-rangy dependency among tokens brings improvements (Naserr et al.] 2021; Tuli et al.] 2021; Mao et al. (022))
By contrast, some local kernel methods demonstrate their superionity (Yang et al., 2019; Liu
et al. 2021; Chu et at., 2021 Park & Kim 2022). For this reason, we first conjecture that
the large neighborhood sizes in the interpolation layer are conducive to predictive performance.
Besides, as demonstrated in Sec. 3.3 the patchwise
embedding in Vision Mixers can be an analogy to the resampling and interpolating, so we further es-
MAE PFNO
RMSE of PENO
tablish a patchwise FNO (PFNO), with patch size equaling to 4 and [4, 4] in 1-d and 2-d PDE problems, equivalent to each resampled points aggregating and 16 points in spatial domains in 1-d and 2-d sit- uations respectively. Layer-Norm is stacked in the FNO layers in PFNO, for fair comparison. Re- sults Fig. 1 show that the long-range dependency larger
1960
4.2
Out of Memory
4.1
3.0
3.1122
4.0
3.9
may even compromise the performance, mean neighborhood sizes often cause higher errors. However, matter how large is the neighbor size, the NFS outperforms PFNO. More details are given in Appendix. B.6 Therefore, we rule the possibility of performance gains brought form large neighborhood sizes and suppose that proposed kernel interpolation layers are the key, and is superior to the simple patchwise embedding methods.
3.1
4.3
4.8
6.0
7.0
loglMean Neighborhood Number)
Figure 5: Effects of neighborhood sizes on NS
(r = 64, n/ 10, n² 40).
Benefits from learned interpolation kernel. Since the kernel interpolation is likely to hold the key to improvements, we investigate the performance gains brought from adaptively learned interpolation kernels over the predefined one (See Fig. 3 We use an inflexible Gaussian kernel
h(z 27 exp((-(z; - x; (T) (xx; F; 1)) as predefined one as discussed in Sec. 3.1 where
...g (d)). and μ Rd 71)) ..... (()) R are learnable parameters. By setting all the other modules and the interpolation neighborhood sizes the same, we compare performance on different meshes of the two interpolation kernels in Table. (Gaus + LN), where the adaptively learned kernels achieve better accuracy.
Benefits from normalization layers. Previous works demonstrated the normalization is necessary for network architecture, for fast convergence and stable training (Dong et al.) 2021] Ba al.) 2016). A notable difference between NFS and FNO is that the Layer-Norm can be implemented NES's layers without disabling its discretization-ivvaraneee The improvements brought from the normalization layers are given in Table. 3 (Flex + LN), where the performance gap is obvious on unseen meshes.
4.4 NON-EQUISPACED VISION MIXERS
Since NFS can regarded as a combination of our interpolation layers with the revised FNO, our interpolation layers can also be implemented in the other Vision Mixers, so that these methods equipped with the ability to handle non-equispaced data. Details are given in Appendix. B.7 We find that (/) From Table. B10 and Table. T the degeneration of performance is obvious in other Vision Mixers. In comparison, FNO as intermediate equispaced layers, truncates the high frequency in its channel mixing and retains the low frequency shared by both resampled and original signals, so the loss of accuracy in non-equispaced scenarios is tiny in our NFS: (2) Although the performance on unseen meshes is more stable in these non-equispaced Vision Mixers, the performance gap still large, according to Table. B10 and Table. 3
4.5 COMPLEXITY COMPARISON
The discussed Fig. 2 shows time complexity of each method. Vision Mixers are the fastest, while graph ppatio-temporal models are far much slower. NFS falls in between because the interpolation layer can be analogy cquispaced token-channel mixing layers of Vision Mixers' structure. Memory usage shows storing each point's neighbors in the interpolation layer is very memory-consuumng..
gaaph-message-pasttiee
layer (See See. 3.3) and the intermediate are

---

Under review as a conference paper at ICLR 2023
REFERENCES
James Atwood and Don Towsley. Diffusiion-convllutiona neural networks, 2016.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization, 2016. URL
https://arxiie wwtpooggg///a/ooeeee 060.0064000
Lei Bai, Lina Yao, Can Li, Xianzhi Wang, and Can Wang. Adaptive graph convolutional recurrent network for traffic forecasting, 2020.
Leah Bar and Nir Sochen. Unsupervised deep learning algorithm for pde-hased forward and inverse problems, 2019.
Saakaar Bhatnagar, Yaser Afshar, Shaowu Pan, Karthik Duraisamy, and Shailendra Kaushik. Prediction of aerodynamic flow fields using convolutional neural networks. Computational
Mechanics, 64(2):525-545, jun 2019. doi: 10.1007/50446--01---774400
URL https
//dei.org/Ine org.10..00.... 10t.:/0000000720004444555530 -19- 019-0740-9----7---111 01740-0
Kaushik Bhattacharya, Bamdad Hosseini, Nikola B. Kovachki, and Andrew M. Stuart. Model reduction and neural networks for parametric pdes, 2021.
Johannes Brandstetter, Daniel Worrall, and Max Welling, Message passing neural pde solvers, 2022.
URL htttps///arxiv.ort abs /20o5/222222 2202 03376
Umer I. Cheema, Gregory Nash, Rashid Ansari, and Ashfaq Khokhar. Memory-optimized re-gridding architecture for non-uniform fast fourier transform. IEEE Transactions on Circuits and Systems I: Regular Papers, 64(7):1853-188 2017. doi: 10.1 10.T1
Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations, 2019.
Tianping Chen and Hong Chen. Universal approximation to nonlinear operators by neural networks with arbitrary activation functions and its application to dynamical systems. IEEE Transactions on Neural Networks, 6(4):91 1995. doi: 10.1 109/72.392253.
Xiangxiang Chu, Zhi Tian, Yuqing Wang, Bo Zhang, Haibing Ren, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Twins: Revisiting the design of spatial attention in vision transformers, 2021. URL
https://arxiy: org/ abs 2204. 13840
Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional neural networks on graphs with fast localized spectral filtering. 2017.
Yihe Dong, Sean-Baptiste Cordonnier, and Andreas Loukas. Attention is not all you need: Pure attention loses rank doubly exponentially with depth, 2021.
2103. 03404
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x 16 words: Transformers for image recognition at scale, 2020. URL /tttts://///........tt...e hattps:/aaxxvv.og//33...
A. Dutt and V. Rokhlin. Fast fourier transforms for nonequispaced data, ii Applied and Com- putational Harmonic Analysis, 201::85-100, 1995. ISSN 1063-5203. doi: https://doi.org/10 1006/acha. 1995 1007. URL http::tp///// /tt::://wwwww.. www.::/wwiinndddeeee... http://eedree.t..cccccccceeeeeee www.s/ccene///rticcc.......
011/310635203857000
Weinan E and Bing Yu. The deep ritz method: deep learning-basee numerical algorithm for solving variational problems, 2017.
Justin Gilmer, Samuel Schoenholz, Patrick F. Riley, Oriol Vinyals, and George E, Dahl. Neural message passing for quantum chemistry, 2017.
John Guibas, Morteza Mardani, Zongyi Li, Andrew Tao, Anima Anandkumar, and Bryan Catanzaro.
Adaptive fourier neural operators: Efficient token mixers for transformers, 2021. URL https
7/ rxiiv www./ba.//baaaaaaaoe.eeeeee /2111. 13587
10

---

Under review as a conference paper at ICLR 2023
Xiaoxiao Guo, Wei Li, and Francesco Iorio. Convolutional neural networks for steady flow ap- proximation. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '16, pp. 481-490, New York, NY. USA. 2016. Associa- tion for Computing Machinery. ISBN 9781450332322. doi: 10.1 00.1145/933967222999777 URL
Kurt Homik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are uni- versal approximators. Neural Networks, 2(5):359 366. 1989. ISSN 0893-6080. doi: https:
//db:./ng/...0010000000000000000 sci bccieencs/arttcceepa/17883333333
URL https: /tt:://wwwwwww. SC1 wnweeiicceiiiicccc rect. com/
Valerii lakovlev, Markus Heinonen, and Harri Lähdesmäki. Learning continuous-time pdes from sparse data with graph neural networks, 2020. URL
08956
Dhiraj D. Kalamkar, Joshua D. Trzaskoz, Srinivas Sridharan, Mikhail Smelyanskiy, Dachyun Kim, Armando Manduca, Yunhong Shu, Matt A. Bernstein, Bharat Kaul, and Pradeep Dubey. High performance non-uniform fft on modern x86-based multi-core systems. In 2012 IEEE 26th International Parallel and Distributed Processing Symposium, pp. 460, 2012. doi: 10.1109/
IPDPS
Srinidhi Kestur, Sungho Park, Kevin M. Irick, and Vijaykrishnan Narayanan. Accelerating the nonuniform fast fourier transform using fpgas. In 2010 18th IEEE Annual International Symposium on Beeld-Programmable Custom Computing Machines, pp. 19-26, 2010. doi: 10.1 10.1109/FCCM. 2010.13.
YUEHAW KHOO, JIANFENG LU, and LEXING YING. Solving parametric PDE prob- lems with artificial neural networks. European Journal of Applied Mathematics, 32(3):421- 435, jul 2020. doi: 1011077:0956992220000
URL cattps://dol. ort. 1017%
2FS055692520001222
Nikola Kovachki, Zongyi Li, Burigede Liu, Kamyar Azizzadenesheli, Kaushik Bhattacharya, An- drew Stuart, and Anima Anandkumar. Neural operator: Learning maps between function spaces, 2021. URL
cms www.eeec.ccc..ccccc... lelechhedd/useera. wttp/////e/////a///////a//////// anima/pubs
GraphPDE_ Journaal__ouoaa.a___m pdr]
Yaguang Li, Rose Yu, Cyrus Shahabi, and Yan Liu. Diffusion convolutional recurrent neural network:
Data-driven traffic forecasting, 2018.
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Neural operator: Graph kernel network for partial differential equations, 2020a. URL htttps::/arxxivorr////o/////
03485
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Multipole graph neural operator for parametric partial differential equations, 2020b.
Zongyi Li, Nikola Kovachki. Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations, 2020c. ftllitttss///rxiy
wwtp:bs/////ooooeoo /tab////111 2010 08895
Haitao Lin, Zhangyang Gao. Yongjie Xu, Lirong Wu, Ling Li, and Stan. Z. Li. Conditional local convolution for spatio- -patio-temporal meteorological forecasting, 2021.
Haitao Lin, Guojiang Zhao, Lirong Wu, and Stan Z. Li. Stonet: A neural-operator-drivn spatio- temporal network, 2022. http::tts://ara //aivvv org/ abs//////22220 08414
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows, 2021. URL.
htttss://arxisi
arg/abs. /et.//113......
Lu Lu, Pengzhan Jin, Guofei Pang, Zhongqiang Zhang, and George Em Karniadakis. Learning nonlinear operators via deeponet based on the universal approximation theorem of operators. Nature Machine Intelligence, 3:218-222 8-229, 2021.

---

Under review as a conference paper at ICLR 2023
Xiaofeng Mao, Gege Qi, Yuefeng Chen, Xiaodan Li, Ranjie Duan, Shaokai Ye, Yuan He, and Hui Xue. Towards robust vision transformer, 2021. URL htttp:://aaaaaata. oxg/ wwwpooa/////ooooooooot...ot..aet http//aab/22051....// 2005. 07926
Muzammal Naseer, Kanchana Ranasinghe, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and
Miuan Yang. Intriguing properties of vision transformers, 2021. URL httth::/p/aatt
org/ abs/ abs/2105. 2205. 10497
Nicholas H. Nelsen and Andrew M. Stuart. The random feature model for input-output maps between banach spaces, 2021.
Shaowu Pan and Karthik Duraisamy. Physics-informed probabilistic learning linear embeddings of nonlinear dynamics with guaranteed stability. SIAM Journal on Applied Dynamical Systems, 19
(1):480 -098 Jan 2020. ISSN 1536-0040. doi: 10.1137/19m12672466 URL http: /tatt://dx..m doi org 10. 1137 9M1267246
Namuk Park and Songkuk Kim. How do vision transformers work?, 2022. URL htttp:://axxii org/abs 2202. 06709
C. Rader and N. Brenner. A new principle for fast fourier transformation. IEEE Transactions on Acoustics, Speech, and Signal Processing. 24(3):264 266, 1976. doi: 10.1109/TASSP:199. 1162805.
Maziar Raissi, Alireza Yazdani, and George Em Karniadakis. Hidden fluid mechanics: Learning 2020.
velocity and pressure fields from flow visualizations. Science,
Yongming Rao, Wenliang Zhao, Zheng Zhu, Jiwen Lu, and Jie Zhou. Global filter networks for image classification, 2021.
2207. 00645
Yulia Rubanova, Ricky T. Q Chen, and David Duvenaud. Latent odes for ireeullarlyssampled time series, 2019.
Youngjoo Seo, Michaël Defferrard, Pierre Vandergheynst, and Xavier Bresson. Structured sequence modeling with graph convolutional recurrent networks, 2016.
Justin Sirignano and Konstantinos Spiliopoulos. DGM: A deep learning algorithm for solving partial differential equations. Journal of Computationall P 375: 1364, dee 2018. doi: 10.1016/ 1.jcp.2018.08 8.08.029. URL
10166222..11111211.. jep. 2018.028100 08. 029
Jonathan D. Smith, Kamyar Azizzadenesheli, and Zachary E. Ross. Eikonet: Solving the eikonal equation with deep neural networks, 2020.
Ilya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Un- terthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, Mario Lucic, and Alexey Dosovitskiy. Mlp-mixer: An all-mlp architecture for vision, 2021. URL https
7/arxiv. 7/arxiv.orrg/bb//2005.00
Shikhar Tuli, Ishita Dasgupta, Erin Grant, and Thomas L. Griffiths. Are convolutional neural networks or transformers more like human vision?, 2021. URL https:://axx...oo/////ooo
21C5.07199
Matthew A. Wright and Joseph E. Gonzalez. Transformers are deep ineeiiiiteedii non-mercer binary kernel machines, 2021. URL htR.ittttps///xx.oooo///sss//
01506
Baosong Yang, Longyue Wang, Derek Wong, Lidia S. Chao, and Zhaopeng Tu. Convolutional self-attention networks, 2019,
03107
Çağatay Yıldız, Markus Heinonen, and Harri Lähdesmäki. Ode?vae: Deep generative second order odes with bayesian neural networks, 2019.
12

---

Under review as a conference paper at ICLR 2023
A
METHODS
A.1 NOTATION
Symbol
Used for
X
discretization of the domain D, which is used to train the model.
X
A discretization of the domain D, which used to evaluate the model. Coordinate of spatial point in D.
ar
Number of spatial points of seen meshes for training, as X nu. Number of spatial points of unseen meshes for inference, as X n.
n4
Number of input timestamps as the number of input time- denedeennddnnt
PDE's initial states.
nl
Number of output timestamps as the number of output time-dependent PDE's future states.
a
Input function, where a A(D; Rd means for E D. a(x) R
time-dependent PDEs, u(-TT), where T [t,]
Target function for approximation, where (D; R means for a: E D, u(x) E IR"..
6
Representation function, where v E U(D;R means for a E D, v(x) Rd which is function obtained by lifter which project into a higher dimensional space.
Go
Approximation operator, where Go(a) The probability measure for sampling Spatial points æ, which is sup- ported on D.
/
C
Cost functional as the minimum optimization target.
Discrete Fourier transform for equispaced spatial points.
F
Discrete Inverse Fourier transform for equispaced spatial points.
Project operator, with (z))(x) E Rd
Q
Project operator, with Q(v)( ax) E Rd
t-th iterative representation function after kernel operators' update.
K.
Kernel integral operator mapping, which maps a to a bounded linear operator, with parameter Ø.
W
Linear transform on the d, dimension (channel) of v(x) d,. Fourier transform of periodic kernel function, which learnable parameters in a single iterative process in FNO.
Rs
ChannelMix
Channel-mixing operator as a linear transform in the dimensoion channel (d,)).
TokenMix
Token-mixing operator as a linear transform in the dimensoion spatial points (n,).
f
Proposed non-equispaced Fourier transform, where f (J H())).
m.
Spatial points' number on resampled equi-spaced points.
H.
Interpolation operator interpolate the non-equispaced spatial points on equispaced spatial grids.
T
Parameter in Gaussian interpolation kernels controlling smoothness of the kemnl..
μ
Parameter in Gaussian interpolation kernels, as the mean of Gaussian kernels.
Hn
Interpolation operator mapping, where (a,(a) is an interpolation op- erator used to map signals on the non-equispaced spatial points equispaced spatial grids.
H'r
Interpolation operator mapping, where (a) is interpolation op- erator used to map signals on the equispaced spatial points to non- equispaced spatial grids.
N(x)
Neighborhood of spatial point æ.
Table Al: Glossary of Notations used in this paper.
13

---

Under review as a conference paper at ICLR 2023
A.2 GRAPH CONSTRUCTION
Neighborhood construction. Instead of using K-nearest neighborhood method, the neighborhood system the interpolation layer is constructed by e-ball, because in equispace scenarios, there will be multiple points as K-th nearest neighbor at the same time. For point æ, its neighbor is defined according
d(æ, S
æ; E N(();;
(10)
d( >E
2 & (xx).
For given e defined in Sec. 3.2 we can restrict so that B [[(x)) clog(ns).
A.3 PROOF OF THEOREM 3.1.
Our proof is mostly based on Chen & Chen 1995) and Kovachki al. (2021). For notation simplicity, in the proof, we directly write H,,(a) as H. as the linear operator.
Lemma A1. Let X be Banach space, and U S a compact set, and K. a dense set. Then, for any > 0, there exists a number n E N, and a series of continuous, linear functionals G1, G2,,..,,.................. Gn C(U4; IR), and elements :f,....; PPn E K, such that
|10 wEl/
sup
Σ j=1
Gr(0)e,llx Sc
(11)
The proof is given in Lemma in Kovachki et al. (202T). and Theorem and 4. for reference.
Theorem A2. Let D C Rd be compact domain. Let be separable Banach space of real-valued functions on D, such that C(D, R) C U is dense. Suppose 4 LT; R) for any < p < co. is a probability measure supported on U and assume that, Eewollvlle is a probabilistie measure supported on D, which defines the inner product of Hilbert space U as <. f.g >4= Sn fa (x)g(()) du(()). Then, there exists neural network by IRd Rdd - R whose activation functiooon are of the Tauber- Wiener class, such that
for any U. l
|10 - H--)Hu S.
where H(v)(x)) = Sp (2, y) (.r.y)/())) ()ly)))
Proof. Since W a Polish space, we can find a compact set K., such that v(U) \ IC) S Therefore, Lemma A1 can be applied, to find a number N, series of continuous linear functionals Gy E C(U; R) and functions Py E C(D:R) such that
sup ||U vek
Σ S.
Denote (v) = Ey=1G(()))) and let < < xo be the Hölder conjugate of p. Since = LP(D,K)) by Reisz Representation Theorem, there exists functions 9j E [D(D, RR), such that Gz(c) S v(x)gg (19,(e) (x)du(=) for I,...,7 and v E LP(D;R), By density of C(D; R) in (D( R), find functions th ....... v/ C(D;R), such that
||Ve 9ill Se/n. sup JE(l...))
Then, define H.. IP(D; RR) C(D:R)
/ thy ()(())
HL. (v) = Σ
For the universal approximation (density) (Hornik a Multi-layer Feedforward network h, 1.th Tauber- Wiener class, such that
1989 of neural networks, we can find whose activation functions are of the
R2
sup a.yED
1,, v)
Σ (1)) Se.
14

---

Under review as a conference paper at ICLR 2023
Let Hnfx)) /o (z. y)o(y)du(y)) Then, there exists constant C1 > 0, such that [1Hn(v) - H(C)lle<(DR)) S ((((Hn)()) - H. (*)ll=(())))) + 1| (v) - H(D))LZP(DR)) For the first term, there is constant C2 0, such that
J " (x)) (olp000)--0/))0400000>0laa0aaaa
[[),(v) - Hn (V))llL S C2 Σ
C2 Σ
- -
(Ceelo(()ll
for some C3 > O. And for the second term,
||H - ((F)))))(DR))) = / f=1 (5)))))))) - - -
< D |
Therefore, there is constant C > 0, such that
[ [| (v) H., (v)lludu(() S eCEnsvllelle
Because of the assumption that Euow//u/lke oo, and is arbitrary, then
||v H(v)llu 1|v (v)lku [[H,.()) H(v) Iax.
the proof is complete.
Corollary A3. Define (v) = SD ( y. x, (g) the interpolation operator can also approximate any precision
Proof. We use a one-layer neural network hy D x D - IR as an example, which is defined as hyg (2, y, aly) or oEE wx,xx(1) Wy,y b). We rewrite it as
hy = 0 Σ We (x -y (0) + (wys + wa (
Wo (3)) + b),
where Wa.j = 0.
Corollary A4. The Theorem 12 and Corollary A3 can be extended for D + Rd where de > 1.
Proof. As v = (v(1), (2)) w(dd)), for each v()), a single neural network can be used for approxi-
mation. Moreover, in implementation, make hn fully-connected, to improve the expressivity. O
Remark. As Ex,ex (x x2, 2c, a(x;)) is the unbiased estimation of Ey (h, (x, y) ()))
use the Equation. (8) for the approximation.
B EXPERIMENTS
B.1 BENCHMARK METHOD DESCRIPTION
Vision Mixers. We provide a framework for vision mixers PDE solvers, including VIT, MLP- MIXER, FNET, GFN, FNO, PFNO and our NFS. The intermediate architecture of mixing layers shown in Fig. BT The code of our framework will be released soon. And the resampling and back-sampling methods are stacked before 'Equispaced Input' and 'Equispaced Output'. In this way, the description of the Vision Mixers included in our framework can described by different modules, as shown in Table. BI All the trials Vision Mixers set embedding size as 32, batch size as 4, layer number the intermediate equispaced mixing layers as 2. FNO and PFNO, the set as
truncated K Kmax is set 16. The patch size of Vision Mixers with patchwise embedding [4, 2] in 1-d PDEs and [4, 4, 2] in 2-d PDEs. The interpolation layers in NES are composed one layer of feed-forward network whose perceptron unit equal 4x embedding size of the model.
15

---

Under review as a conference paper at ICLR 2023
Tokan Mixing
Charnel Mixing
Token Demixing
Channel Mixing
Norm
Equisparced Input
Activation
Equispaccd Oulput
Residual Transform
Figure Bl: The architecture of Vision Mixers.
Table B1: Description of Vison Mixers in the unifying framework module by module.
Modules
VIT
MLPMINER
FNET
GFN
FNO
PFNO
NFS
Patchwise Embedding
Patchwise Embedding
Patchwise Embedding
Patchwise Embedding
Patchwise Embedding
Resampling
Identity
Interpolation
Token Mixing
Attention
MLP
Fourier
Fourier
Fourier
Fourier
Fourier
Channel Mixing
Elementwise Product
Low Frequency MatMultiply
Low Frequency MaiMultiply
Low Frequency
Lincar
Linear
Lincar
MatMultiply
Token Demixing
Identity
Identity
Identity
Inverse
Inverse
Fourier
Fourier
Fourier
Fourier
Channel Mixing
Identity
Identity
Identity
Linear
Identity
Identity
Identity
Complex
Normalization
LayerNorm
LayerNorm
LayerNorm
LayerNorm
Identity
LayerNorm
LayerNorm
Residual
Identity
Identity
Identity
Identity
1xl Conv
1xl Cony
1xl Conv
Complex
Activation
Gelu
Gelu
Gelu
Gelu
Gelu
Gelu
Gelu
Back Sampling
Lincar+ Rearrange
Lincar+ Rearrange
Linear+
Lincurt Rearrange
Identity
Linear+ Rearaagge
Interpolation
Rearrange
DeepONet Variants. Since vanilla DeepONet uses MLP as Branch Net, it cannot be implemented in
such high-resolution dataset, because for resolution like the trial (NS n, = 4096, = 10, n/ = 10 ). DeepONet assigns each data point a weight parameter in a single MLP, leading the MLP's parameter number reaches (n) = 409602 in single Branch Net, which is infeasible in practice. In the original paper, the spatial point's number in the experiments is set as 40, far less than the recent Neural Operator's evaluation protocol.
One feasible alternative to use other architecture to replace the original ML.,, thus allowing DeepONet to handle high-resolution data. For example, CNN and Vit. Therefore, we here conduct further experiments on the three equations in the context, to evaluate DeepONet-U (using UNet as the Branch Net) and DeepONet-V V (using Vit as the Branch Net) two variants of vanilla DeepONet for comparison. Note that the architecture of variants of DeepONet are all limited to equispaced data.
Graph Tentioaleempalll Models. The evaluated graph spatio-temporal neural networks are based on recurrent neural networks for dynamics modeling, where the spatial dependency is modeled by graph neural networks. The spatial and temporal modules for AGCRN, DCRNN and GCGRU are shown in Table. B2 MPPDE used different architecture, with the pushforward trick used for taining, with rolling equaling and time window equaling to 10. All the trials on these graph spatio-tempral models set embedding size as 64, except MPPDE 128. Batch size is set as 4. When the graph convolution needs multi-hop message-passing we set the hop as For MPPDE, the layer number of GNNs is 6. The embedding dimension in AGCRN is set as 2.
Table B2: Description of different graph spatio-tempont temporal models
Methods
Spatial module
Temporal module
GCGRU Seo et al. 2016
Cheb Conv Defferrard et al. 2017
GRU
DCRNN
LI et al. 2018
Diff' Cony Atwood & Towsley 2016
GRU GRU
AGCRN
Bai ot al
2020
Node Similarity Bai et al, 2020
16

---

Under review as a conference paper at ICLR 2023
B.2 DATA GENERATION
Burgers' Equation. The initial condition uo(x) is generated according to 40 N(00.25(-A 25/) =) with periodic boundary conditions. is set as 0.01. x E (0, 1] and [0. 1J. The spatial
resolution is 1024, and time resolution is 200. The dataset generation follows FNO's protocol, which can be downloaded from its source code on official Github.
KdV Equation. The equation is written as 0, t) + 30, (x, + (2, = 0,
(12)
where x E [0, 1). The initial condition (z) calculated as
u(z,0) 0) = Σ
0.5c; cos c; + b, a:)
i=1
where q (0, 0t), and a,, b; 0. The spatial resolution 1024. The dataset generated by scipy package, with fftpack. diff used as pesudo-differential method and cdeint used as forward Euler method.
Darcy Flow.
The equation is written as
V(((x))v(())) = (xx u(x) = 0
E (0,
(13)
0[0, 1]²
The original resolution is 256 x 256. a(x) is generated by Gaussian random field, and we directly establish the operator to learn the mapping of to u.
NS Equation. Our generation of NS Equation based on FNO's Appendix. A.3.3, with the forcing is kept fixed. The original spatial resolution is 128 x 128, and time resolution is 200.
B.3 COMPLETE RESULTS ON MODEL COMPARISON
Here we give complete results on the four Equations. Table. B3 give the performance comparison on Darcy flow of both equispaced and non-equispaced scenarios. Table. Bnd B5 gives performance comparison in equispaced scenarios on the other three time-dependent problems. Table. B6 and B7 gives performance comparison in non-equispaced scenarios on the other three time-dependent problems. In all the tasks except Darcy Flow, the depth of layer is set as 2, and kmax 16 in both NFS and FNO. However, we find in Darcy Flow, kmax should be set much larger, or the loss will not decrease. the reported results, kmax = 32, 64, 128 in Darcy Flow.
Table B3: Performance comparison Darcy Flow.
MAE (x10 9) RMSE(X10 3)
MAB(X10 RMSE(X10 MAE 3) RMSE(x10
Darcy Flow
Darcy Flow (r 128)
Darcy Flow
256)
VIT
0.5073-0041
0.8468400422
0.986520.000
1.619520.0m
1.1078aam21
1.844420021
MLPMIXER
0.4970
0.8228ah.0054
0.8909a0.099
1.42212001
0.9125aum24
1.6459aamaa
GEN
0.4739-00m1
0.8345an0001
0.8659aa.004
1.4237=0.0m1
0.
FNO
0.4289-00051
0.7740+n084
0.7086-0.045
0.1324-0.0m
0.9075-a0051
1 1.4940
NES
0.1497-0.008
0.1962-08007
0. 0.2254-00007
0.7245.n.omm
0.4216-a0ms
0.8578
Daycy
Darcy
Daray
1024)
(n,
4096)
(n, 16384)
DCRNN
1.8144-000000
2.6352+n.009
1.7629-a.00:
2.5760-n.00m/
0OM
00M
AGCRN
1. -0693
2.4444+nn00t
1.7336xx.00
2.4167-0.00m
0OM
00M
GCGRU
1.7633-oo000
2.5696400001
2.5363=0.00m
0OM
00M
MPPDE
0.6673
0.9290an.0012
0.5608
0.8424a0.0051
0.638420.005
0.8748
NFS
0.1727
0.231110000
0.143020.007
0.1014
0.2379a0007
0.3489-a.00
NFS fails model the non-equispaced Burgers' Equation when is set 1, in which the perfor- mance is far from it can achieve equispaced scenarios. Such problem will be our future work.
17

---

Under review as a conference paper at ICLR 2023
Table B4: Performance comparison with Vision Mixer benchmarks on different equations (n, 1).
Validation loss Burgers' (n4 = 1) of VIT, GFN, and FNO does not converge. The results show that the early-stopping occurs the begining of training.
MAE (x10 RMSE(
MAE(x10-9) R0
MAE( 1AEEx1000 10
RMSE(x10-3)
Vision Mixers
Burgers' (r 512, n/ 10)
Burgers' 512,
Burgers' (r 1024, ni 20)
40)
VIT
201.6539405584
231.9138assu:
183.6696-13011
210.6237
224.47122112772 195.5858=0.776
MLPMIXER
201.6547-00677
231.91634amm)
183.6535-005w
210.6160-novs
195.5960-60004
224.4791-0n
GFN
201.6557:uvess
231.9122suuss
183.6674-
210.6165:0ARs)
195.5918=00mm 8-00mm
224,4736-m091
FNO
201.6527-11445
231.9119
183.6696. 90105
210.6299:n-4m::
195. 995.5902-00944
224.4723-0022
NFS
0.1806-0000
0.2669:00m0
0.3570-00008
0.5340-000w
0.4344
0.6092-00017
Kdv
KdV
Kdv
G 512, m². = 10)
(r = 512, m4
40)
(r 1024, n/ = 20)
VIT
0,2808-0.00
0.3938-000m
0,3428 00012
0.6832-0001
0.3066
0,5461 -00003
MLPMIXER
0.2732=n
0.4259za008s
0.3336-0004
0.5923=00001
0.2872=0:006
0.5235_0.006
GFN
0.2587=00088
0.3490aa008s
0.3086-0023
0.5952=00838
0.201 1-0.0044
0.34644.0006
FNO
0,2619a0
0.3849a00107
0.5608=00053
0.8424
0.3925
0.4623-00007
NFS
0.251 4:000
0,4522-000s
0.6290
0.2254.0.00f
0.0745500:000
NS
n
10)
(r
64,
40)
(r 128, ni 20)
VIT
12.9291 400mm
22.8565-0009
29.1130-u
15.7398-0.075
20.6927-00064
MLPMIXER
7,5246:0000
10.4762sH0ows
15.8632-00575
20.1522:a0004
14.9360=0.00s
19.3268=000s
GFN
3.5524-n0087
4.707 +000ss
10.2250 00031
11.4451-00m0
6.3976-00084
8.2685
FNO
3.3425-00mm
5,256640000
8.9857-0000
1400171-00mm
4.4627-0.000
6.3047-00004
NFS
1.7425:0001
2.2847:00002
4.7882-n:006
6.1508:0002
2.6988-0.006
3.5121
Table B5: Performance comparison with Vision Mixer benchmarks on different equations (n, 10). MAE(x10-3)
MAE (x 10 RMSE(x10-3)
MAE(x10-3)
RMSE(x10-3)
RMSE(X10-3)
Burgers' (r 512, m/
Burgers' 512, m4
Burgers' (r= 1024, n 20)
Vision Mixers
10)
VIT 0.5042e00114
0.7667a0022
2.426920028s
3.7728
1.532720014
2.4093an0us
MLPMIXER
0.0973
0.2600zam097
0.4210an
0,5844200101
0.3303
0.4473200086
GFN
0.2383anmm4
0.30662n0114
0.4187
0.5407a00090
0.3500aam02
0.4489an008
FNO
0.0978
0.181 000000
00
0.1430
0.1871
NFS
0.0958200015
0.1347an000z
0.235 1000009
0.1474200026
0.1957400034
KdV
KdV
Xdv
(r
512, vo.
10)
512, n
40)
(r 1024, m/ 20)
VIT
0.2066xnm027
0.3525:00099
0.2376xamcc:
1
0.1897samts
0.3725xa0009
ML.MIXER
0.2152+0000 oons
0.3686-a0mr
0.2497.aom
0.5400
0.2062-am07
0.4429.a002
GEN
0. 000s
0.26074amm0
0.269 s0000r
0,5451 400004
0.1 1984400002
0.386944.0ms
FNO
0.3230
1.1105e0000
0.9605200004
2.7500aam0s
0.5999300000
1.64733000s
NFS
0.0678
0.121440003
0.270940000
0.5122a00013
0.1576
0.3114
NS
NS
NS
(r 64,n
= 10)
(r = 64,n
40)
(r = 128, n
20)
VIT
3.9609
6.0575aa0250
12.3433a00542
16.5238a00415
9.30 14nnn244
14:0027400390
MLPMIXER
3.1530em004p
4.4339amm07
7.929 12000s
10.41494nm0ss
7.7410zn0077
10.1934400082
GFN
1.7396208016
2.355 1a0mos
5.44644anocs
7.21 002130200002
3.126 1stu00s
4.169 1200047
FNO
2.4076:nm017
3.286 1:0004
7.6979kantes
10.6401 anooss
3,700
3.7001-amm44 1-00004
5.0047-aa003
NES
0.8636-00008
1.2264-0001
3.1122.0000
4.1950.000x7
1.840640m00
2.5620.0000s
18

---

Under review as a conference paper at ICLR 2023
Table B6: Performance comparison with graph spatio-temporee benchmarks (n, = 1).
Graph Spatio-
MAB (x10x))
RMSE(X1U-)) MAB(x10-))
RMSE(x10-) MAE(x10-3) RMSB(x10-3)
Temporal Models
Burgers' 512,
Buzgers' (n2 256,
Burgers' (n, 512, 40)
10)
DCRNN
146
29
368. 188
298.4097-001117
373.0938a00186
AGCRN
289.9 289
160.983440mm
272.6697 5404
340.1351
305.4976
GCGRU
288.4507
361.1175
294.9075
367, ao7
291.0365=n08s
365. 1668
MPPDE
24.4997
34.5123+0001
25.4357-000m
31.7015-00mm
25.3311
33.7808 0.005
NFS
16.1860sam01
28.15044coma
21.1634:0000
33.8976x000s
26.0818-0000
44.7962-01/00
KdV
KdV
KdV
(12 = 512, Ni
10)
(02 256, n/ 20)
(r =-= 512, mi 40)
DCRNN
1.6855a000
3.0875x000m
3.1267
4.8662-0000
5.7387-0000
8.3752--0000
AGCRN
4.0753a0.001
6.8043
5,4107a0000:
9,2333+0000)
8.4438-00000
13.8677-00000
GCGRU
1.6554x0.00
2.6839x0.00
3.0677:aomm
4.6557:0000)
5.8745-0.000
9.4528-n.000
MPPDE
1.5452a0.000
2.677440.000
2.9929a00007
5.4582a000010
3.0101=0:000
4,99446_n.001
NFS
0.08 0:0016
2.0012
0.1576:00007
4:00u
0.3210-0002
0.6873-0.0p
MS
(n1
4096, n1
10)
(n,
1024, n
20)
(n2
4096, nf 40)
DCRNN
30.6756xu100
41.7815xcux
52. 1290000
69. 69.7019su0mz
88.3382-00004
119.5021=0wo
AGCRN
OOM
0OM
59.9393+mnmm
79.0434gmmm)
OOM
OOM
GCGRU
28.8537400009
40.
49.9352-0oxox
67. 67.56233-0004
85.9303-00PM
117.9925-00172
MPPDE
8,98 04000014
20.7453+
32. 201
-4.2387=0000
90.01 90
NES
2.1992:0002 2.8280
3.9178 7.9178:00054
5.0182:0000
4.7865-00042
6.1384
11884-000
0.0009
Table B7: Performance comparison with graph spatio-temporal benchmarks (n+ = 10).
Graph Spatio-
MAE (x10-3) RMSE(x1003)
MAE(x10-3)
RMSE(x 10-3)
MAE(x10-> RMSE(x10-))
Temporal
Models
Burgers' (n2- = 512, nf 10)
Burgers'
Burgers'
(n, = 256, 20)
(ng = 512, m/. 40)
DCRNN AGCRN
2.6122400014
3.843520009
4.6126g00015
6.8853gau03
8.5880sn0020
12.739444M07
4.666740.001
6.2791200001
10.4900ea
13.9810aawee
15.61432nu00
21.0937a00001
GCGRU
1.664320.0002
2.5074
3.1400a00000
4.8008a00019
5.7653samo2
8.9335anmcs
MPPDE
1.1271
188838200077
2.4554g0000
4,4315220o0
4.1213:00006
6.1980g0000
NFS
0.1085
0.1504
0.1634:00015
0.1983:1u0m
0.2775
Kdv
KdV
kdy
(0,- 512, nf
256, n/
20)
512,
40)
DCRNN
2.3196x00001
4.1634s0mm
3.4503+0000
5.7450
4.9286--a010
8.39 2400008
AGCRN
3.9350
6. 16114aamm
5.663 1 400000
8.1191
8. 8.2893-amo::
11 11.568440000
GCGRU
1.664320000
2.5074
3 3.4205
5.6873+4000
2.5032+h000z
5.451 5500000
MPPDE
1. 40000
2.9708
5.38 Laquosso
2.4293shu00s
4.93 1400000
NFS
020816200012
0.0012200022
6.1576
0.311440001
0.3210
0.6873a0004
(n, 4096, v/ = 10)
(x_ 1024, ml 20)
(x1, 4096, n/ 40)
DCRNN
8.7025
12.5238-ammz
27.1069a00044
39.1259aa0001
59.6602-ami77
88.2946xan144
AGCRN
0OM
OOM
42.41972a000
60.5375aa
0OM
OOM
GCGRU
6.3570a0.001
9.7306xamm
21.3537:0002
32.967444008
57.24933su0s
84.1847
MPPDE
5.4353:0004
7.8838-0088
17.5902-40013
25.9372-amos
42.3057-na06s
76.3374
NES
0.9335:00011
1.3254.0092
1.8239
2.529 +0000s
3.2768-anmos
4.3988
19

---

Under review as a conference paper at ICLR 2023
B.4 MORE VISUALIZATION
Here we provide more visualization results on the three equations. See Fig.
B2
Fig.
B3 and Fig. B4
40
20
0.10
10
0.05
Ground Truth
-0.00
30
-0.05
-0.10
10
-0.15
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Prediction
Figure B2: Visualization on equispaced Burgers' equation.
40
20
0.1
10
0.0
Ground Truth
40
-0.1
30
10
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Prediction
Figure B3: Visualization on equispaced KdV equation.
20

---

Under review as a conference paper at ICLR 2023
t-10
1-20
Ied
t-10
E-20
mechgrid
meshgrid
0.00
0.25
0.5
00
6.5
00
05
3.0
Prediction on seen meshgrid
Prediction on unseen meshgrid
0.50
0.25
0.0
Ground tnth on unseen mechgrid
Ground truth on unseen meshgrid
0.00
30
05
0.0
10 Prediction meshirrd
1.0
meshgrid
Figure B4: Visualization on non-equispaced NS equation: The training mesh (n, 4096 in upper- left) is different from the meshes in inference process (n', = 8192 in upper-right, n', 12288 in lower-left and n's 16384 in lower-right).
21

---

Under review as a conference paper at ICLR 2023
B.5
MESH-INVARIANT EVALUATION
Table B8: Mesh-invariant performance NFS on Burgers' and KdV equations (ne = 10).
MAE (x10-9) RMSE(X10-))
MAE(X10-3)
RMSE(x10-3)
Burgers'
kdv
(n, 512,
10, 40)
(n, 512, 10, m4 40)
X
0.1983+0.000
0.2775+0.002
0.3210=0.0011
0.6873+0.004
Y 1.3ng
0.2371 100000
0.314310.001
0.376920.008
0.7805
1.7m,
0.2898+0.0133
0.3742+0.0002
0.4084=00007
0.8419=0.174
n 2.0mg
0.3052+t.008
0.4180+00000
0.411 +00022
0.8471 +0.074
Table B9: Performance of NFS with its variants of NS equations (n+ 10) on unseen meshes,
MAE (x 1x100 RMSE(x10-3) MAE(x10-3) RMSE(x10-))
MAE(x10-3) RMSE(x10->
Flex LN
NS
NS
NE
(n, = 4096, v = 10)
(n. 1024, n/.
20)
(R, 4096, n/ = 40)
X
0.9335xmm01
1.3254-amm2
1.8239.00w2
2.5291 +a00es
3.2768x00005
4.3988
ni
2n.
La0004
1.5042aamm77
2.3530a00081
3.3320aautr4
3.5439200085
4.7904smm1
n
3n,
1.107 s000011
1.5716
2.5179400089
3.5477=00125
3.658440018
8.8558a0246
n
4nl
1. 1.1015
1.5627-amm0
2.5919+00064
3.6526+a0m1
3.6608+0m00
4,952 +00000
MAE (x10
RMSE(x10
MAE(x10
RMSE(X10
MAE(X 10 3)
RMSE(X10
NS
NS
NS
Gaus LN
(n,
4096, of = 10)
(n, 1024, no.
20)
(n, 4096, n2
40)
X
40634
2.1992aa0m4
2.1976 20006s
3.02192am00
3.6422400026
5.0097
2ng
2.8589400002
4.05624a0126
3.746540041
5.1308200057
3.909240001
5,2402anmms
3ng
3.4513a00118
4.5199sa0377
5.7712400125
5.7137
4.2102480002
5.5057
4nl
3.4357200000
4.7382+00000
5.5990200066
5.5958s0m9w
4.2628+0000
5.7679500000
MAE (x10-
RMSE(x10-3)
MAE(x10-3)
RMSE(X10-3)
MAE(X 1000x110-
RMSE(X10-
NS
NS
NB
Flex LNN
(n. 4096, n = 10)
(n, 1024, n/. = 20)
(R, = 4096, n/. = 40)
X
1.2
1.7293aam47
2.5019
3.4923aumss
4.2083a0887
5.6761
2n
1.4882400146
2.1681 =00300
7.0203+00203
10.6096-a0345
5.8975+0m00
8.7704
3m
1.6384+0m0R
2.4130-am1e9
7.9177+00069
11.9825-a011
6.6622+0806
9.5874-0011
4ni
1.6975en0000
2.5008
7.1961000101
10.8860eanos
6.6951 400000
9.6334.00000
The mesh-invariant evaluation on Burgers' and KDV Equations of NFS are given in Table. B8
In Table. [B8] when the spatial resolution is just 512, inference performance unseen meshes deteriorates. This result also validate our conclusion (2) in the third paragraph in Sec. 4
Besides, we give full evaluation mesh-invairance of NFS in NS equation, with its variants detailed results corresponding to Table. B3
22

---

Under review as a conference paper at ICLR 2023
B.6 NEIGHBORHOOD SIZE'S EFFECTS
The effects of mean neighborhood size on the predictive performance on Burgers' (ns = 512, n4, = 10, ni = 40) and KDV (n, = 512, n4 110 ni 40) are shown in Fig. B5
MAE PfNO
RMBE of PENO
MAE of PTNO
RMSE PFNO of NFS
ofNES
OFS
MAE OFS
C.3100
0.8677
0.299
0.4855
0.21
0.35
MAE
RMSE
0.30
0.50
RMS
0.45
0.18
0.23
0.40
0.0
0.35
0.15
0.30
2,3
3.2
4.1
5.0
5.6
6.0
2.3
3.2
4.1
5.0
5.6
8.0
log(Moan Neighborhood Numbar)
leg[Mean Naighborhood Number)
(a) Burgers' (n, 512, ni 40).
(b) Kdv (n. 512, n 40).
Figure B5: The change of MAE and RMSE of NFS with the increase of neighborhood size on Burgers' (n, 512, 10, n/ 40) and Kdv (n, 512, 10, n² 40). PFNO is the baseline.
B.7 INTERPOLATION WITH OTHER VISION MIXERS
We conduct experiments on non-equispaced NS equations with the combination of our interpolation layers and other Vision Mixers to figure out they achieve camparable performance.
Table B10: Performance of different Vision Mixers combined with the interpolation layers in non- equispaced scenarios on NS equations (nz- = 10).
MAE (x10-3) RMSE( 3) MAE(x10 3) RMSE(x10- MAE(X10-3) 3) RMSE(x10
VIT
NS
(n, 4096, n² 10)
(n. 1024, n 20)
(n, 4096, 40)
X
OOM
OOM
0OM
OOM
0OM
0OM
MAE (x10-
RMSE(X10-5
MAB(x10-3) 9) RMSE(x10-
MAE 3)
RMSE(x10-
NS
NS
NS
MLPMIXER
(n, = 4096, n = 10)
(n, = 1024, ni = 20)
(0, 4096, nY. 40)
X
6.1854a00012
8.1556200001
9,4593ga.022
12.1316200022
10.1862aam45
13.15488a0051
n
2n
8. 8.1573=00126
11 11.2258
12.0706aam132
14.9460snaa199
10. 6000:
13.6872+0.23
n
3m
8. 1..20-000088
11.3840+00171
14.9910-anoa
5-0.000
10.5633-a6140
13.6394-am14
n,
4ri,
00000
1403313
14.9517-00125
17.7857-0.009
10.541440000
13.6106-4000
MAE (x10-3)
RMSE(X10-1
MAE(x10
RMSSE(100-)
MAE x 10
RMSE(x10-3)
NS
NS
GFN
4096, n//
1024, n/ 20)
(n, 4096, 40)
X
12.2373=00001
16.2902am0133
10.2768=a.x84
13.7852-0.078
14.7765aumes
19.4872+am16
n/
2n.
13.7752-na154
18.3108sn01st
1
24.1397:0.087
15.9083sa022s
n.
3r,
13.7054-0.112
98.2192+001s4
17.8238-anm1
24.2783-0.0096
15.8986-00156
20.99443-...s
n.
4nl
13.71 400000000
18.227 4000000
+0.8207
24.28333-0.044
15.8736-0000
20.9772=00000
23

---

Under review as a conference paper at ICLR 2023
B.8 COMPLEXITY COMPARISON
We here first give Table. BTT to show the complexity of time and memory of all the evaluated methods on NS (r = 64, n2 = 10, n = 40).
Table B11: comparison on complexity of the evaluated methods
Type
Methods
Time/Epoch
Peak Memory
Parameter Number
GCGRU
6'18"
8660MB
74945
Graph Spatio-
DCRNN
9'38"
11120MB
148673
Temporal Model
AGCRN
OOM
OOM
0OM
MPPDE
10'5-1"
23333MB
622161
VIT
3'14"
32166MB
773217
MLPMIXER
1'12"
4421MB
79749953
GFN
48"
3296MB
1361729
Vision Mixer
FNO
27"
3748MB
6299425
PFNO
43"
3380MB
9742145
NFS
202"
31938MB
37891937
Table B12: Detailed complexity of NFS
Interpolation Resampled Points
Neighbor Searching
Kernel Calculation Weighted Summation
3522MB
3 3102MB
6884MB
Interpolation back on Original Points
Neighbor Searching
Kernel Calculation
Weighted Summation
2506MB
2754MB
6884MB
It demonstrates that method has comparable efficiency to Vision Mixers. For the graph spatial- temporal models, they suffer from the recurrent network structures and thus are extremely time- consuming while the parameter number is small, limiting their flexibility.
Time. However, once compare the used time in PFNO and NFS, will find that the inter- polation layers are considerably consecon..min. Another module that cost time complexity the normalization layer, as the original FNO does include Layer-Norm in its architecture, but is stacked in PFNO. Theoretically, PFNO handles down-sampled grids in a low resolution, because of the patchwise embedding. However, takes more time than FNO. Therefore, we conclude that the time complexity brought from Layer-Norm is very significant, but it is affordable because of the performance improvements.
Memory, Besides, the operation of searching for each spatial point's neighborhood and calculating weighted summation in Eq. (9) and Eq. (10) are very memory consumin. We test it on the same experiment, and give the memory usage of different models in forward process, as shown Table, B12 The memory cost in backward process is 6902MB.
C EMPIRICAL OBSERVATION FOR THEOREM 3.
In Sec. 3.2, Theorem 3.1 is proved to assure the expressivity of NFS. However, no further evidence gives the assurance of the convergence of the kermel interpolation. Here we conduct empirical study to give some clues.
We conduct experiments on NS equation with n, 4096, n, = 10, nt = 40. In a single trial, NES is trained with fixed meshes. We repeated the trials 10 times with different meshes, and then give the
24

---

Under review as a conference paper at ICLR 2023
one-v.s.-all deviations of the representation states calculated by
H. Hzl Hil [1,
Diff 90
May nie
where H; is the representation states of the shape Vm, v my n4l, and | the element wise absolute value, and || || is the 1-norm of the matrix. If the Diff is small in the beginning and end, it can be inferred that the interpolation kernel function converges to a similar mapping since the final predictions are close to ground truth in these experiments, and the inputs are sampled from the same instance of PDEs. We give the Diff before the first FNO and after the final FNO layers in Table. CI The small values indicate that the trained model usually has similar representation states. Figure. and C2 give visualizations of representation states obtained by one instance of NS equation in two different trials. It indicates that the differences getting smaller during the training.
Table Cl: The defined Diff calculated by different epochs.
Epoch
Diff
Diff end
0.0676
0.0978
500
0.0102
0.0353
t-10
t-20
t-30
10
04
-02
100
0.8
0.6
-0.6
0.4
-0.8
0.2
0.0
-10
0.00
0.5
0.50
0.75
1.00
0.00
025
050
075
1.00
0.00
0.25
a5o
0.75
1.00
0.00
0.5
0.50
0.75
3.00
Dev= 0.0897
(a) Representation states at the beginning of FNO layers in two trials of Epoch 0
1.0
t-0
t-10
t-20
t-30
0.3
0.8
0.6
0.1
0.4
-0.1
-0.
0.2
-0.
0.0
0.0179
(b) Representation states at the beginning of FNO layers two trials of Epoch 500
Figure C1: Visualization on different representation states at the beginning of FNO layers.
25

---

Under review as a conference paper at ICLR 2023
1.0
0.8
1.0
0.6
0.4
0.2
0.0
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Dev 0.1182
(a) Representation states in the end of FNO layers in two trials of Epoch 0
t=0
t=10
t=20
t=30
1.0
0.8
0.6
0.4
0.2
0.0
1.
0.2
0.30
Dev 0.0394
(b) Representation states end of FNO layers in two trials Epoch 500
Figure C2: Visualization on different representation states the end of FNO layers.
As result, we present the one-v.s.-all differences of different epochs in the training process, to validate the convergence, as shown Figure. C
0.08
0.00
04
0.00
Epoch
Figure C3: Convergence of Diff.
26


Reviews:
[{"review_id": "YyE-M2uF35", "reviewer": "Reviewer_Gvhj", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper extends the Fourier neural operator to non-equispaced setting. It adds interpolation layers at the beginning and the end of FNO. Numerically, the proposed NFS model is more flexible than FNO, while more efficient than graph-based method.", "summary_of_the_review": "Overall, I think this paper brings a solid contribution. I think the paper is slightly above the threshold.", "strength_and_weaknesses": "Strength: the work is well-motivated and clearly written. It has a complete set of numerical experiments which show significant improvements. Also the paper has a careful ablation study.\n\nWeakness: using interpolation to address the mesh points dataset is not very new. Previous works have directly applied interpolation and used FNO as in (https://arxiv.org/abs/2111.05512). As shown in Table 3, the learned interpolation is not very significant from the plain Gaussian interpolation baseline.\n\nOther comments\n- In section 4.3, NFS outperforms FNO on the equispaced problem. This is somewhat counter-intuitive. I wonder if the improvement on equispaced problems attributes to differences in architecture such as channel mixing and layernorm, etc.\n- A recent work (Geo-FNO) proposes a similar architecture with a different theoretical formulation (https://arxiv.org/abs/2207.05209). Is it possible formulate the kernel interpolation as a deformation?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "NukZiPxwHf", "reviewer": "Reviewer_Y9dA", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors propose a new approach for solving PDEs using non-equispaced Fourier Neural Operators. The authors draw some analogs between vision mixer models and FNO and then build over it to obtain the non-equispaced FNO. For the non-equispaced FNO, the authors use a non-equispaced interpolation function that helps construct an equispaced signal and thus helps use FNO for the PDE solving. The idea is novel, and the results show improvement over FNOs, and further, they also overcome memory issues to some extent. ", "summary_of_the_review": "The work proposed by the authors is certainly novel; the paper proposed a non-equispaced Fourier Neural PDE solver, and the contributions include (I) some connection between vision mixers and FNOs (ii) the development of NEI layer to obtain non-equispaced FNO (iii) significant results compared to baseline. \n\nThe contributions are somewhat like incremental research with minimal improvement over the SoTA. Hence, I recommend a weak reject. What could convince me is (i) some more comparison with standard neural PDE solvers (ii) Some more theory on what kind of sampling strategies one should try or something (iii) and/or some more results which can be related to the theory proposed. ", "strength_and_weaknesses": "+ better memory efficiency over FNOs\n\n- It still has all several issues plagued with FNO as well.\n- The comparisons are done with standard vision mixers and other models. But methods such as DeepONets are cited but not compared against.\n- the theory (theorem 3.1) provided is not giving any insights but is only provided to prove something that the authors don't relate with the results obtained. It's as if the theory is not meant to prove anything but that approximations exist. I did a random search of where theorem 3.1 is cited in the whole paper and didn't find any references in the results or the rest of the main paper.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "LgcJ0Z2W6h", "reviewer": "Reviewer_gMaM", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper raises an important question as to how one can do operator learning when equispaced discretizations of the functions are not available. But the supposed solution proposed in this paper is quite unconvincing. ", "summary_of_the_review": "I believe there are very serious issues with the paper and hence my rating is on the lower side. ", "strength_and_weaknesses": "There is a total lack of transparency in this paper as to what is the new loss function - and this ambiguity starts from not having any clarity about how is a supposed interpolation being done from the sampled non-equispaced points to a new sample of equispaced points.  I believe that the $\\tilde{\\cal F}$ in equation 7 and 8 is what is replacing the ${\\cal F}$ in equation 4. But this is only my guess and this still doesn't really define what is the new replacement of the basic equation 3 of FNO.  \n\nEquations 6,7,8 and the periodic heat kernel are all seemingly defined in terms of their resampled equispaced points. Its entirely as to where and how the given non-equispaced points got used. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject