Guiding continuous operator learning through Physics-based boundary constraints
Authors: Nadim Saad
Gaurav Gupta
Shima Alizadeh
Danielle C. Maddix
Venue: ICLR 2023 poster (2023)
Keywords: partial differential equations
operator learning
physics-constraints
boundary conditions
kernel correction

Abstract: Boundary conditions (BCs) are important groups of physics-enforced constraints that are necessary for solutions of Partial Differential Equations (PDEs) to satisfy at specific spatial locations. These constraints carry important physical meaning, and guarantee the existence and the uniqueness of the PDE solution. Current neural-network based approaches that aim to solve PDEs rely only on training data to help the model learn BCs implicitly, however, there is no guarantee of BC satisfaction by these models during evaluation. In this work, we propose Boundary enforcing Operator Network (BOON) that enables the BC satisfaction of neural operators by making structural changes to the operator kernel. We provide our refinement procedure, and demonstrate the satisfaction of physics-based BCs such as Dirichlet, Neumann, and periodic by the solutions obtained by BOON. Numerical experiments based on multiple PDEs with a wide variety of applications indicate that the proposed approach ensures satisfaction of BCs, and leads to more accurate solutions over the whole domain. The proposed method exhibits a (2X-20X) improvement in accuracy (0.000084 relative $L^2$ error for Burgers' equation). Code available at: https://github.com/amazon-science/boon.

Published as a conference paper at ICLR 2023
GUIDING CONTINUOUS OPERATOR LEARNING THROUGH MAYSICS-BASER BOUNDARY CONSTRAINTS
Nadim Saad b Gaurav Gupta 2,1, Shima Alizadeh ?, Danielle C. Maddix 1 Stanford University (450 Serra Mall, Stanford, CA 94305) 2 AWS AT Labs (2795 Augustine Dr, Santa Clara, CA 95054) Equal contributions, order decided by coin toss.
nsaad31estanford edu, (gauravaz, alizshim, dmnaddix e amazzmmmazon zon. com
ABSTRACT
Boundary conditions (BCs) are important groups of physics-enforced constraints that are necessary for solutions of Partial Differential Equations (PDEs) to sat- isfy at specific spatial locations. These constraints carry important physical meaning, and guarantee the existence and the uniqueness of the PDE solution. Current neural-network based approaches that aim to solve PDEs rely only on training data to help the model learn BCs implicitly. There is no guarantee BC satisfaction by these models during evaluation. In this work, propose Boundary enforcing Operator Network (BOON) that enables the BC satisfaction of neural operators by making structural changes to the operator kernel. We provide our refinement procedure, and demonstrate the satisfaction of physics- based BCs, e.g. Dirichlet, Neumann, and periodic by the solutions obtained by BOON. Numerical experiments based on multiple PDEs with a wide variety of applications indicate that the proposed approach ensures satisfaction of BCs, and leads to more accurate solutions over the entire domain. The proposed correction method exhibits a (2X-20X) improvement over a given operator model in relative 1.2 error (0.000084 relative 1,2 error for Burgers' equation). Code available at:
science/boor https:: com/ amazon htt.:///.....oooaooooooonnee///n
INTRODUCTION
Partial differential equations (PDEs) are ubiquitous in many scientific and engincering applications. Often, these PDEs involve boundary value constraints, known as Boundary Conditions (BCs), in which certain values are imposed at the boundary of the domain where the solution is supposed to be obtained. Consider the heat equation that models heat transfer in a one dimensional domain as shown schematically in Figure [ The left and right boundaries are attached to insulator (zero heat flux) and heater (with known heat flux), respectively, which impose certain values for the derivatives of the temperature the boundary points. No-slip boundary condition for wall-bounded viscous flows, and periodic boundary condition for modeling isotropic homogeneous turbulent flows other examples of boundary constraints widely used in computational fluid dynamics. Violating these boundary constraints can lead to unstable models and non-physical solutions. Thus, it is critical for PDE solver satisfy these constraints in order to capture the underlying physics accurately, and provide reliable models for rigorous research and engineering design.
In the context of solving PDEs, there has been increasing effort in leveraging machine learning methods and specifically deep neural networks to overcome the challenges in conventional numerical
methods (Adler & Öktem 2017 Afshar et al. 2019 Guo al. 2016 Khoo al. 2020 Zhu & Zabaras, 2018). One main stream of neural- netrallnetwork approaches has focused on training models that predict the solution function directly. These methods typically are tied to a specific resolution and PDE parameters, and may not generalize well to different settings. Many of these approaches may learn physical constraints implicitly through training data, and thus do not guarantee their satisfaction at test time (Greenfeld et al. 2019 Raissi et al. 2019 Wang et al. (02I). Some previous works also attempt to formulate these physics based constraints in the form of a hard constraint
bhssics-based
Work completed during internship AWS AT Labs.

---

Published as a conference paper at ICLR 2023
Insulation
heater flux
du Ph 1=-1
4-
l
BOON (Ohm) FNO Esonee
teeadary one
16
11.00
10
0u
0r
0u
-2
BOON (Ours)
10
01
4-
0.000
0.006
0.010
du
0,900
0.996
1.000
og (0.
Oz
Figure 1: Heat equation with physical boundary constraints. Heat flow across a conductor with an insulator and heater at the left and right boundary, respectively. The insulator physically enforces zero heat flux (x 8z ), and the heater imposes a known heat flux oN (1, t) from the right boundary. Neumann denotes this derivative-based boundary constraint (see TableT]. Violation of the left boundary constraint by an existing neural operator (FNO), even when trained on data satisfying boundary constraint, suggests heat flowing across the insulator, which not aligned with the underlying physics. Disagreement at the right boundary violates energy conservation. The proposed boundary corrected model (BOON) produces physically relevant solution along with better overall accuracy (see Section(4.2.1).
0u
optimization problem, although they could be computationally expensive, and may not guarantee model convergence to more accurate results (Xu & Darve 2020 Krishnapriyan et al. 2021 Lu et al. Donti et al. 202T).
2021b
Neural Operators (NOs) are another stream of research which we pursue in our study here that
aim to learn the operator map without having knowledge of the underlying PDEs (Li et al. 2020a
Rackauckas et al. 2020 Tran et al., 202T Li et al.) 2020bb Guibas et al. 2021 Bhattacharya et al.
Gupa et al. 2021). NO-based models can be invariant to PDE discretization resolution, and
able transfer solutions between different resolutions. Despite the advantages NO-based models offer, they do not yet guarantee the satisfaction of boundary constraints. In Figure[] the variation of heat flux is shown at the boundary regions for a vanilla NO-based model (FNO) and our proposed model (BOON). FNO violates both boundary constraints, and results in a solution which deviates from the system underlying physics.
Physics-based boundary conditions are inherent elements of the problem formulation, and are readily available. These constraints are mathematically well-defined, and can be explicitly utilized in the structure of the neural operator. We show that careful leverage of this easily available information improves the overall performance of the neural operator. We propose Boundary enforcing Operator Network (BOON), which allows for the use of this physical information. Given integral kernel- based NO representing a PDE solution, a training dataset D, and a prescribed BC, BOON applies structural corrections to the neural operator to ensure the BC satisfaction by the predicted solution.
Our main contributions in this work can be summarized as follows: (i) A systematic change to the kernel architecture to guarantee BC satisfaction. (ii) Three numerically efficient algorithms to implement BC correction in lincar space complexity and no increase in the cost complexity of the given NO while maintaining its resolution-independent property. (iii) Proof of error estimates to show bounded changes in the solution. (iv) Experimental results demonstrating that our proposed BOON has state-of-the- performance on variety of physically-relevant canonical problems ranging from ID time-varying Burgers' equation to complex 2D time-varying nonlinear Navier-Stokes lid cavity problem with different BC, e.g. Dirichlet, Neumann, and periodic.
TECHNICAL BACKGROUND
Section 2.1 formally describes the boundary value problem. Then, we briefly describe the operators in Sections 2.2 and which are required for developing our solution in Section
2.1 BOUNDARY VALUE PROBLEMS
Boundary value problems (BVPs) are partial differential equations (PDEs), in which the solutions required to satisfy a set of constraints along given spatial locations. Primarily, the constraints assign physical meaning, and are required for uniqueness of the solution. Formally, BVP is written
2

---

Published as a conference paper at ICLR 2023
Fu(x, t) = 0, 2 x 52 C RR" u(z, 0) uo uo(z), x E
14 2 0
Gu(z,t) = 0, z E an,
where denotes a time-varying differential operator (non-linear, in general), the spatial domain with boundary 0Ω, un (2) the initial condition, & the boundary constraint operator, and u(x, t) the solution at time t. Formally, given the domain 22, we denote the interior as L which is the space of points x E S2 such that there exists an open ball centered at æ and completely contained in S2 The boundary of Ω2 is defined as an {x|x E 52 n & 52}. In given boundary domain ön, the boundary conditions (BCs) are specified through Gu = O, where G is taken be linear the current work. Three types of BCs which are widely used to model physical phenomena are: (1) Dirichlet, (2) Neumann, and (3) periodic. dasseiedescribes the mathematical definition of each boundary condition.
Bounday condition
Description
601
Dirichlet
Pixed-value boundary
(m((a. (om((u))) wlra.t) op(nact) (2
C
Neamann
Fixed-dericative the boundary
(u(r, G(Goa)) > (e4u))) onloigd) 0, ow CR, We 0. 0J
Perodlc
Equal values the boundary
Ollr.t)Geaatoo..oo O u(rt. () m0xy-11000 0. AB, Ve 0.W
Table 1: Boundary conditions. Examples of the most useful canonical boundary conditions, i.e. Dirichlet, Neumann, and periodic, which carry different physical meanings. We define Ou be the function space corresponding to given boundary condition. Illustration on a one-dimensional spatial domain Q [zn, xN- with boundary 0n (To, xN-1) shown for simplicity, and an extension to ligh-dimensional domain is straight-forward (see 4cction
2.2 OPERATORS
The solution of a PDE can be represented as an operator map, from either a given initial condition, forcing function, or even PDE coefficients (input a) the final solution w. Formally, an operator map I' is such that T' : A + U, where A and U denote two Sobolev spaces H with 8 > 0. Here, we choose p = 2. For simplicity, when T is linear, the solution can be expressed as an integral operator with kemee K Q x 1 - 1
u(x, 1) = Ta(x)= = K K(x, ((e,y)ay)dd
where a(x) A, u(x, E with specified 0. In this work, axx) is equivalent the initial condition, (I( in eq. (1)
Note: For a given PDE, the operator T is non-linear in general, and we concatenate multiple linear operators along with non-linearities (e.g., GeLU) model a non-linear operator (as done in Li et al. (2020a), Gupta et al. (202T). We refer to Section for more details.
2.3 BOUNDARY-CONSTRANEE
OPERATORS
T. T.Enay(T)
Consider BVP of the form in eq. I denote the true operator between initial condition to the solution at time
U
as P A Uhay with 2haa Lf nll where Uuay denotes the function space of solutions to the BVP, as illustrated in Figure 2 (See Appendix [A for summary of the notations). For an operator T A + Mody (learned from the data), the incurred boundary error is denoted as
.
T. Endy Enay(T))
Vody
au
Eady (T)
l[gT(a) - -P((a))laan, (GD
Σ
(3) Figure 2:
Operator refinement. A
D
given operator T and its refinement 7.
where [|-I|aze.. is taken as the L error at the boundary, or (San ddr D C (A, Mudy) ) is a given dataset (for example, evaluation data), and G is linear boundary operator (see eq. (D). Although the operator T' is trained using the data with samples drawn from Unay. it is not guaranteed to have small boundary errors the evaluation dataset. In this work, aim to develop correction
3

---

Published as a conference paper at ICLR 2023
mechanism such that the resulting operator T :AA A + Loay satisfies ETdy(T) over given data D, for an arbitrary small > 0.
Remark: Emphasizing small boundary error through model modification serves two purposes. first (i) satisfaction of boundary values which are necessary for physical importance of the problem, second and more importantly (ii) to act guiding mechanism for obtaining a correct solution profile (see Section and Figure
3 BOON: MAKING KERNEL CORRECTIONS FOR OPERATOR
We have seen in motivating example in Figure | that even though neural operator is provided with the training data which implicitly obeys the boundary constraints, the boundary conditions may be violated in the final solution. Explicit knowledge of the given boundary conditions can be exploited to guide the solution profiles, and ensure lower boundary errors in the evaluation. 3.1] motivation for an explicit BC satisfaction procedure, and Section discusses the computational algorithms for an efficient implementation.
3.1 MOTIVATION: KERNEL MANIPULATION
The integral kernel K (xx, y) in eq. (2) is the fundamental block in operator learning. We observe that the boundary conditions in Table Table can be exactly satisfied by appropriate kernel adjustment using the following result.
Proposition 1. For an integral operator T: HHSP + H with kernel K (x, y) mapping an initial condition uo(x) to w(x, t), the boundary conditions listed in Table Hare satisfied, if
1. Dirichlet: K(xo, 4) pp(zo.()
1 ro), Do E 02, Vy E Ω u(zo, t) ap (tx.)) t).
op(=n.0)
2. Neumann: K2 (xo. y) = ax( [ = mo), my E 02, Vy E Ω = u2 (xo. t) = a,( t).
3. Periodic:
K( y) K(EN-1, 8) Zo. 2N 82, Vy E Ω u(xo. u( t).
The corner cases, ( 0) = O for Dirichlet and (xoo ) = 0 for Neumann, are considered in the
proof of Proposition[]in in Appendix B Proposition motivates a key idea that kernel manipulation can propagate certain properties, e.g. boundary constraints from the input function wo(x)) (a(x)
Proposiiionmmottivtees
eq. (2) to the output ((xx,1),4 >> 0. For example, the input un(x) is periodic, or un (xn) = uo(xN-1) To, IN-1 02, then changing the kernel through Proposition I ensures that u (zo t)
u(ZN-1, t), > 0 as well, i.c. the output is periodic. Since boundary conditions need to be satisfied by all u(ar, t), ;), Vt (see formulation in eq. (D) can leverage kernel manipulation to communicate boundary constraints from the known input function at i = 0 to the unknown output at f > O.
3.2 KERNEL CORRECTION ALGORITHMS
The previous discussion motivates that should perform kernel manipulation for boundary constraint satisfaction. From a computational perspective, the input and the output functions are evaluated over discretized grid (see Section4]. We show that Gaussian elimination followed by normalization allows for Dirichlet and Neumann, and weighted averaging allows for periodic boundary corrections. We establish the boundary correction of given neural operator (or BOON) using the following result.
Theorem 1. Refinement: Given grid with resolution N, and an operator T A- + Ubdy charac- terized by the kernel (...)) as in eq. (2) For every 0u in Table there exists a kernel correction such that the resulting operator T A + Ldddy satisfies Ebdy (T) R 0, evaluated over the grid.
Figure (in Appendix(C) intuitively explains the existence of solution for Theorem where for given grid resolution N, the discretized version K = [K w.w.llooN........ E NNN shown in the blue grid, of the continuous kernel K ((xy)) y) is corrected by applying transformations for the boundary corrections. See Appendix C.1 for the choice and correctness of these transformations.
To efficiently implement our BOON boundary correction, we utilize given module X., such that y = K((x) := Kx. In Appendix C.2 show that matrix vector multiplications the only operations that our kernel correction requires. As common with other Neural Operator inplementations, we take K., for example to be Fourier-based i al. (2020a), or multivavelet--ased in Gupta al. (2021)
4

---

Published as a conference paper at ICLR 2023
to efficiently perform these matrix vector multiplications. The computational efficiency of BOON is established through the following result.
Theorem 2. Computation complexity. Given a grid resolution N, a kernel evaluation scheme K. for eq. (2) with cost NK, the kernel corrections in Theorem can be reduced to at most 3 kernel calls, requiring only O(Nx) operations and ((N) space.
The proof is provided in Appendix C.2 where we provide arithmetic expansions of the terms in for Dirichlet, Neumann, and periodic
Appendix C.I These derivations lead to Algorithms corrections, respectively. A straight-forward translation of these algorithms to neural operator architecture with module is provided in the Figure (see Appendix C.3
Algorithm 1: Dirichlet
Algorithm 2:
Neumann
Algorithm 3: Periodic
Input: K. Uo, aiD (2o, t)
Input: K. Uo. (zo,t),
Input: K. Uo 0, E ERT,
Output: Corrected Dir u(t) I: Ko,0 + XC)e))(O), with
c ER co #
a + -= =3=1
Output: Corrected Neu u(t)
Output: Corrected Per u(t)
eg 0,0.
Ko,0 f KC(c)) [0]. with
u f- K(uu))
2: z + K(uu))
eg [1, 0.O.
2: u[0] t ou(0) + Bu[-]
3: u - Ug
2: z t K(uu))
3: u u/-1] f u(0)
4: u(0] < 2up[0] z(0]/Ko.0 Ko.0
3: u < Up
5: u K((u)
4: n(0) f 2un(0] - 2[0]/Koo. Ko.0
6: u(0] f- tzo.
5: u f K(u)
6: u(0] + aN (zo, f)/co
If the given kernel module IC is grid resolution-inneeendeer i.e., train and evaluation at different resolution, then the boundary correction algorithms 12 and are resolution-independent as well by construction.
Note:
Lastly, show that the kernel correction in Theorem implemented to guide the operator T through boundary conditions, does not yield unrestricted changes to solution in the interior using the following result which is proved in Appendix C.4
Theorem 3. Boundedness: For any E A, and an operator T, with its refinement T (Theorem) such that = Ta and ū Ta, I| - úlll is bounded.
Interestingly, when the given operator T is FNO, then Theorem can be combined with the results of Kovachki et al. (2022) to bound the error between the refined operator π and the true solution.
With a straightforward modification, the refinement results and algorithms presented for a domain in this section can be extended to higher dimensions (and multi-time step predictions) we demonstrate empirically in Section
EMPIRICAL EVALUATION
In this section, we evaluate the performance of our BOON model on several physical BVPs with different types of boundary conditions.
For learning the operator, we take the input as a(x)) = uo(x) in eq. (T. and the output as T wo(x)) + [u(2,, 4),..., u(x, IM]] for the given ordered times < < IM E R., For single-step prediction, we take
1, and for multi-step prediction, we take M = 25.
Data generation: We generate dataset D {(a, u4.))x) where a; a(I;). Way := u(z;, tt) for x; E i0 O,...,NN .N- 1, and t, < t R4., L....,A M for given spatial grid-resolution N. and number of output time-steps M. Each data sample has size (N, NN x M) (See Appendix D) The train and test datasets, Duzin and Dusst are disjoint subsets of with sizes Perain and Alestt respectively (See Table 7).
We compare BOON model with the following baselines: (i) Fourier neural operator (FNO) (Li et al. 2020a), (ii) physics informed neural operator (PINO) (Li et al. 2021 in
Benchmark models:
5

---

Published as a conference paper at ICLR 2023
the multi-step prediction case or adapted PINO (APINO) in the one-step prediction case (iii) multi-pole graph neural operator (MGNO) (Li et al. 2020c), and (iv) DeepONet (Lu et al. Note that MGNO requires symmetric grid, and valid only for single-step predictions. We use FNO as the kernel-based operator in our work, and MGNO and DeepONet as the non-kernel based operators. See Appendix [F5 for study of applying BOON the multiwavelet-based neural operator (Gupta et al. 2021).
2021a).
The models are trained for total of 500 epochs using Adam optimizer with initial learning rate of 0.001. The learning rate decays every 50/ 100 epochs (ID/rest) with a factor of 0.5, We use the relative-L* error as the loss function (Li et (2020a)). We compute our BOON model by applying kernel corrections (see Section 3 on four stacked Fourier integral operator layers with GeI.U activation (Li et al. (2020a)) (See Table We use p3.8xlarge Amazon Sagemaker instance (Liberty et al. 2020) in the experiments. See Appendix F. for details.
Training:
true BOON
d(x. d(x, t):BOON
1.0
0.5
0.0
0.25
0.5
1.5
X
time
abs error
0.05
0(x t):FNO
0.02
10-6
0.5
0
0.25 0.5
X
Figure Multi-time step prediction with boundary correction: The predicted output 0(x t) for BOON (top) and FNO (bottom). Four uniformly-spaced time snapshots (from a total 25, see Dection[4.1.1] are shown from left right in the spatial domain = [0, 0.5). The color indicates absolute error at each spatial location. FNO has high boundary error which propagates to the interior, and impacts the overall solution profile. BOON (FNO with boundary correction) uses boundary values better learn the overall output across the entire time-horizon. Critical boundary mistake is visualized a 4x zoomed window, where BOON correctly learns the trend.
4.1 DIRICHLET BOUNDARY CONDITION
We here present our experimental results for BVPs with Dirichlet boundary conditions including one-dimensional Stokes' second problem (2D including time), one-dimensional Burgers' equation (2D including time), and two-dimensional lid-driven Cavity Flow (3D including time).
4.1.1 ID STOKES' SECOND PROBLEM
The one-dimensional Stokes" second problem is given by the simplified Navier-Stokes equation White (2006) in the y- cooordinate as follows.
44 Vilyy:
y E [0, 1),t 0. E 00. > 0.
un (y) = Ue by cos(ky);
" 0, t) U cos(wi),
PINO (Li et 2021 uses a PDF soft constraint from Raissi et al. (019)) which only supports the multi-step prediction For single step prediction,
adapted APINO, which only the houndary loss term.

---

Published as a conference paper at ICLR 2023
where U 2 0 denotes the maximal speed of the oscillatory plate located y 0, denotes the viscosity, denotes the oscillation frequency, and k (w/(2v) (See Appendix D.1 for the parameter values). We compute the multi-step prediction with final time lar = 2 for M = 25. We also apply exponential mollifier after the last layer in the model's architecture avoid numerical oscillations in the boundary regions.
Model
V = 0.1
e' = 0.02
V/ = 0.005
V/ == 0.002
= 0.001
BOON (Ours)
0.0089 (0)
0.0105 (0)
0,0075 (0)
0.0164 (0)
0,0183 (0)
FNO
0.0099 (0.0074)
0.0158 (0.0072)
0.0188 (0.0084)
0.0211 (0.0105)
0.0273 (0.0135)
PINO DeepONet
0.0241 (0.0841)
0.0256 (0.0578)
0.0221 (0.0342)
0.0188 (0.0223)
0.0191 (0.0175)
0.1471 (0.0556)
0.1047 (0.0382)
0.1408 (0.0515)
0.1335 (0.0488)
0.1538 (0.0549)
Table 2: Multi-step prediction for Stokes' with Dirichlet BC. Relative L error (L² boundary error) for Stokes' second problem with varying viscosities y at resolution N = 500 and M 25.
Table shows that our boundary constrained method results in the smallest relative L errors across various values of the viscosity by enforcing the exact boundary values. Figure B illustrates that our BOON model corrects the solution profile near the boundary area, and results in a more accurate solution in the interior of the domain. This observation demonstrates that the satisfaction of boundary constraints is necessary to predict an accurate solution that consistent with the underlying physics of the problem.
4.1.2 1D BURGERS' EQUATION
The -iedimmnsional viscous Burgers' equation with
0.025
BOON FNO
Dirichlet boundary conditions can be formulated as fol- lows:
0.020
Relative Urror
0.05
ur + (u² 12) VUzzr
x E [0, 11,t2 O,
(x(x) ULD if S 0.5, UR, i > 0.5,
x E [0, 1],
(5)
0.005
u(0 1) Uexact (0, 1), u(1, t) Mexact (1,
1 >> 0, t> 0,
0.000
0.0
0.2
0.4
0.6
0.8
1.0
where 2 O denotes the viscosity, Mexact (I. is the exact solution given in eq. (53) and ur UR E R. For the initial condition values given in Appendix D the
Figure 4: Relative error for Burgers' equation with v 0.02. The worst test error sample for BOON is shown.
exact solution involves the formation of sharp interface, known shock wave, which propagates towards the right boundary. We compute the single-step prediction at final time (s = 1.2 (M = 1) when the shock hits the right boundary. Figure indicates that the relative error of FNO is higher than BOON model throughout the domain and it grows significantly near the right boundary IN -1 = at /M = 1.2, while the relative error of BOON model stays relatively flat. Table 3 represents the L errors obtained by our model as well the benchmark models. BOON model exhibits a 4X- 30.X improvement in accuracy, and satisfies the prescribed boundary conditions exactly across various values of the viscosity v.
Model
0.1
0.05
0.02
0.005
0.002
BOON (Ours)
0.00012 (0)
0.00010 (0)
0.000184 (0)
0.00010 (0)
0.00127 (0)
FNO
0.0037 (0.0004)
0.0034 (0.0004)
0.0028 (0.0005)
0.0043 (0.0004)
0.0050 (0.0022)
APINO
0.0039 (0.0004)
0.0040 (0.0005)
0.0037 (0.0007)
0.0048 (0.0008)
0.006 (0.0016)
MGNO
0.0045 (0.0004)
0.0046 (0.0005)
0.0048 (0.0008)
0.0064 (0.0015)
0.0101 (0.0006)
DeepONet
0.00587 (0.00124)
0.0046 (0.001 30)
0.00580 (0.00128)
0.00623 (0.00140)
0.00773 (0.00141)
Table 3: Single-step prediction for Burgers' with Dirichlet BC. Relative L test error (L2 boundary
error) for Burgers' equation with varying viscosities at resolution N 500 and M 1.
7

---

Published as a conference paper at ICLR 2023
4.1.3 2D LAVIER-STOKER EQUATION: LID-DRIVEN CAVITY FLOW
The two-dimensional Navier-Stokes equation for viscous and incompressible fluid with no-slip boundary conditions can be written in terms of vorticity as follows:
Wc uT Vw (1/ (1/Re)V'w,
z,ye [0, 1| 0, r,y E [0, 112, 2 0, f,y E 113,
V 0
wo (r, y) V (00) (y 1)):
(6)
wrr, 0,t) = - (x, 0), w(2 1,t) - (z, w0O. v, 2 (0.y. t), w(..gtt) (1.
x E [0, 1], t 20
y E [0. 1,
(V x ú) 0,0, 1|" denotes the where = [w()) u22) denotes the 2D velocity vector, component of the vorticity. Re the Reynolds number and U > 0. See Appendix D.3 for the generation of the reference solution, and the choice of the different parameters. We compute the multi-step prediction for M = 25 with final time ts = 2. According to Table BOON results in the lowest errors compared to state-of-the-ar models for multiple values of the Reynolds numbers, and additionally reduces the boundary errors to absolute 0.
Model BOON (Ours) FNO
Re 10
Re 100
Re 1000
0.00213 (0)
0.00143 (0)
0.00176 (0)
0.00249 (0.7489)
0.00186 (0.6022)
0.00215 (0.9315) 0.00693 (0.7128)
PINO
0,00723 (0.6582)
0.00532 (0.4294)
Table 4: Multi-step prediction for Navier-Stokes with Dirichlet BC. Relative 7,² error (I.2 boundary error) for various benchmarks on the lid-driven cavity problem with varying Reynolds numbers Re at resolution N 100 and M 25.
4.2 NEUMANN BOUNDARY CONDITION
In this subsection, we present experimental results for BVPs with Neumann boundary conditions considering dieedimnsional heat equation (2D including time), and two-dimensional wave equation (3D including time). We use finite difference scheme estimate the given Neumann boundary conditions and enforce them in our operator learning procedure (See Lemma
4.2.1 ID HEAT EQUATION
The one-dimensional inhomogenous heat equation is given as:
ut kuzz = f(x, t), un (x) cos (OTT))))
x E [0, 00,1,,t2 0,
x E 00,
Ug (0, () = O, (1,(1)) = U sin
1 12 0,
where A 0 denotes the conductivity, f(x, Un cos denotes the heat source, and U, e R. The exact solution and more details about the choice of different parameters can be found in Appendix
D.4 We compute the multi-step solutions of the heat equation with final time f.r = 2 and M 25. Table 3 that BOON performs significantly better than the benchmark models in terms of both relative L and the L boundary errors. In particular, the large boundary errors of baseline models indicates their violation of the physical constraints of the problem, the zero flux on the left boundary (x = 0) imposed by the insulator, and the conservation system total energy that is controlled by the heat source on the right boundary (z 1). As illustrated in Figure by enforcing the exact boundary constraints, BOON corrects the solution profile near the boundary and leads to more accurate result in the interior domain well.
8

---

Published as a conference paper at ICLR 2023
Model
N 50
= 100
.V 250
500
BOON (Ours)
0.00675 (0)
0.00501 (0)
0.00893 (0)
0,01187 (0)
FNO
0,00915 (0.96924)
0.00845 (0,60455)
0.00925 (2.05607)
0,01239 (6.31774)
PINO DeepONet
0.01155 (1.23922)
0.00829 (0.25646)
0.06361 (0.01228)
0.07171 (0.03457)
0.10323 (7.1092)
0.11995 (10.1063)
0.18961 (16.6690)
0.27444 (24.3496)
Table 5: Multi-step prediction for the heat equation with Neumann BC. Relative T,2 error (I.² boundary error) for various benchmarks with varying resolutions N and 25.
4.2.2 2D WAVE EQUATION
The wave equation models the vibrations of an elastic membrane with Neumann BC at the boundary points (Cannon & Dostrovsky (981)). The mathematical formulation of this problem is as follows:
Ust = c² (Uzz + Uyn),
I E [0, 1]2 2 0,
uz(2, y) k cos((Tu) cos(TTy)
y,y E [0, 1]?,
(8)
ue (z, M/ 0) 0, Ug (0, y) = Uz (1, = (x, (z, u,,
J,y E [0, 1J²
t t2 0,
where the parameters c E 2 denotes the wavespeed and E TR. The details of our experiment
setup can be found in Appendix D.5 We compute the multi-step solution of the wave equation at final time (as = 2 with M = 25 Table shows that BOON obtains the lowest relative L error compared the benchmark models, and enforces the exact physical constraints the boundary points while capturing the displacement of the membrane with the highest accuracy.
Model
N 25
N 50
N 100
BOON (Ours)
0.00097 (0)
0.000893
0.00096 (0)
FNO
0.00140 (0.04296)
0.00093 (0.03402)
0.00119 (0.03350) 0.00635 (0.03572)
PINO
0.00113 (1.2873)
0.00526 (0.24901)
Table Multi-step prediction for the wave equation with Neumann BC. Relative L error (L2
boundary error) for various benchmarks with varying resolutions N and M = 25.
4.3 ADDITIONAL RESULTS
See Appendix F for the following additional experiments.
Single-step and multi-step predictions: We include the corresponding single-step predic- tion results for Stokes' (Dirichlet), Burgers' (periodic) and heat (Neumann) in Tables respectively. We also include the multi-step time-varying prediction results for the Riemann problem of viscous Burgers' with Dirichlet boundary conditions Table TO See Appendices F.T F3
Periodic boundary condition: We provide the periodic boundary condition results for Burgers'
from Li et al. (2020a) in Appendix F.2
F.2
Resolution independence for Neumann BC: In Appendix F.4 show that BOON trained lower
resolutions for the heat equation (eq. (7), e.g. N 256, can predict the output at finer resolutions,
e.g. N 512, 1024 and 2048 (See Table 14].
BOON applied to MWT: In Appendix [.5] we show that BOON can be successfully applied to other
Mfateoofthh-ar hernel-based operators, e.g. the multiwavelet-basee neural operator (MWT) (Gupta
et al. 2021
5 CONCLUSION
In this work, we propose BOON which explicitly utilizes prescribed boundary conditions in a given neural operators' structure by making systematic changes to the operator kernel to enforce their satisfaction. We propose a refinement procedure to enforce common BCs, e.g. Dirichlet, Neumann, and periodic. We also provide error estimates of the procedure. We show empirically, of physically-ifformatie informative PDE settings, that the proposed approach ensures the satisfaction of the boundary conditions, and leads to better overall accuracy of the solution in the interior of the domain.
range

---

Published as a conference paper at ICLR 2023
BIBLIOGRAPHY
Jonas Adler and Ozan Öktem. Solving ill-posed inverse problems using iterative deep neural networks. Inverse Problems, 33(12):124007, 2017.
Yaser Afshar, Saakaar Bhatnagar, Shaowu Pan, Karthik Duraisamy, and Shailendra Kaushik. Predic- tion of aerodynamic flow fields using convolutional neural networks. Computational Mechanics, 64:525- 545, 2019.
Bradley K. Alpert. A class of bases in 12 for the sparse representation of integral operators. SIAM Journal on Mathematical Analysis, 24(1):246-262, 1993.
Leah Bar and Nir Sochen. Unsupervised deep learning algorithm for pde-based forward and inverse problems, arXiv preprint arXiv:1904.0544 054417 2019.
Saakaar Bhatnagar, Yaser Afshar, Shaowu Pan, Karthik Duraisamy, and Shailendra Kaushik. Predic- tion of aerodynamic flow fields using convolutional neural networks. Computational Mechanics, 64(2):55--545, 2019.
Kaushik Bhattacharya, Bamdad Hosseini, Nikola B. Kovachki, and Andrew M. Stuart. Model Reduction And Neural Networks For Parametric PDEs. The SMAI journal of computational mathematics, 7:121-157, 2021.
J.T. Cannon and S. Dostrovsky. The Evolution of Dynamics: Vibration Theory from 1687 1742, volume 6. Veringer-Verlag, 1981.
Alexandre J. Chorin. Numerical solution of the navier-stokes equations. Mathematics of Computation, 22:745-762, 1968.
Tri Dao, Beidi Chen, Nimit Sohoni, Arjun Desai, Michael Poli, Jessica Grogan, Alexander Liu, Aniruddh Rao, Atri Rudra, and Christopher Ré. Monarch: Expressive structured matrices for efficient and accurate training. In Proceedings of the 39th International Conference Machine Learning, volume 162. PMLR, 2022.
Priya L. Donti, David Rolnick, and J Zico Kolter. DC3: A learning method for optimization with hard constraints. In International Conference on Learning Representations, 2021.
L.C. Evans. Partial Differential Equations. Graduate studies in mathematics. American Mathematical Society, 2010.
Yuwei Fan, Jordi Feliu-Faba, Lin Lin, Lexing Ying, and Leonardo Zepeda-Nunez. A multiscale neural network based on hierarchical nested bases. arXiv preprint 1rXhv:1808.02376 2018.
Yuwei Fan, Cindy Orozco Bohorquez, and Lexing Ying. Ber-net: A neural network based on the nonstandard wavelet form. Journal of Computational Physics, 184:1-15, 2019.
Jordi Feliu-Fabà, Yuwei Fan, and Lexing Ying. Meta-learning pseudo-differential operators with deep neural networks. Journal of Computational Physics, 408: 109309, 2020.
Daniel Greenfeld. Meirav Galun, Ronen Basri, Irad Yavnch, and Ron Kimmel. Learning to optimize multigrid PDE solvers. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 2415 24233. PMLR, 2019.
John Guibas, Morteza Mardani, Zongyi Li, Andrew Tao, Anima Anandkumar, and Bryan Catan- zaro. Adaptive fourier neural operators: Efficient token mixers for transformers. arXiv preprint arXiv:2111.13587 2021.
Xiaoxiao Guo, Wei Li, and Francesco Iorio. Convolutional neural networks for steady flow ap- proximation. In Proceedings of the 22nd ACM SIGKDD International Conference Knowledge Discovery and Data Mining, KDD '16, pp. 481-490, New York, NY, USA, 2016. Association for Computing Machinery.
10

---

Published as a conference paper at ICLR 2023
Gaurav Gupta, Xiongye Xiao, and Paul Bogdan. Multiwavelet- operator learning for differential equations. In Advances in Neural Information Processing Systems, volume 34, pp. 24048-24062, 2021.
Gaurav Gupta, Xiongye Xiao, Radu Balan, and Paul Bogdan. Non-linear operator approximations for initial value problems. In International Conference Learning Representations. 2022.
Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are universal approximators. Neural Networks, 2(5):359-3666 1989.
Chiyu "Max" Jiang, Karthik Kashinath, Prabhat, and Philip Marcus. Enforcing physical constraints in CNNs through differentiable PDE layer. In ICLR 2020 Workshop Integration of Deep Neural Models and Differential Equations, 2019.
Chiyu Max Jiang, Soheil Esmaeilzadeh, Kamyar Azizzadenesheli, Karthik Kashinath, Mustafa Mustafa, Hamdi A. Tchelepi, Philip Marcus, Prabhat, and Anima Anandkumar. Meshfreeflownet: A physice-constraineed deep continuous space- time super-resolution framework. arXiv preprins arXiv:2005.01663, 2020.
Xiaowei Jin, Shengze Cai, Hui Li, and George Em Karniadakis. NSFnets (navier-stokes flow nets): Physics-informed neural networks for the incompressible navier-stokes equations. Journal of Computational Physics, 426: 109951, 2021.
Yuchaw Khoo, Jianfeng Lu, and Lexing Ying. Solving parametric PDE problems with artificial neural networks. European Journal of Applied Mathematics, 32(3):421-435, 2020.
Dmitrii Kochkov, Jamic A. Smith, Ayya Alieva, Qing Wang. Michael P. Brenner, and Stephan Hoyer. Machine learning accelerated computational fluid dynamics. Proceedings of the National Academy of Sciences, 118(1)):21011841) 2021.
Nikola Kovachki, Samuel Lanthaler, and Siddhartha Mishra. On universal approximation and error bounds for fourier neural operators. J. Mach. Learn. Res., 22(1), 2022.
Aditi S. Krishnapriyan, Amir Gholami, Shandian Zhe, Robert Kirby, and Michael W Mahoney. Characterizing possible failure modes in physics-iffrrmdd neural networks. Advances in Neural Information Processing Systems, volume 34, pp. (6548-26560, 2021.
Randall Vequee Numerical methods for conservation laws (2. ed.). Lectures in mathematics. Birkhäuser, 1992.
Zongyi Li, Nikola Kovachki, Kamyar Arizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. arXiv preprint (XXi::00100088955 2020a.
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar, Neural operator: Graph kernel network for partial differential equations. arXiv preprint arXiv:2003.03485, 2020b.
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Multipole graph neural operator for parametric partial differential equations. arXiv preprint arXiv:2006.0953: 2020c.
Zongyi Li, Hongkai Zheng, Nikola Kovachki, David Jin, Haoxuan Chen, Burigede Liu, Kamyar Azizzadenesheli, and Anima Anandkumar. Physics-informed neural operator for learning partial differential equations. arXiv preprint arXiv:2111.0377 2021.
E. Liberty, Z. Karnin, B. Xiang, L. Rouesnel, B. Coskun, R. Nallapati, Delgado, A. Sadoughi, Y. Astashonok, P. Das, C. Balioglu, S. Chakravarty, M. Jha, P. Gautier, D. Arpin, T. Januschowski, V. Flunkert, Y. Wang, Gasthaus, L. Stella, S. Rangapuram, D. Salinas, S. Schelter, and A. Smola.
Elastic machine learning algorithms in amazon sagemaker. In 2020 ACM SIGMOD International Conference on Management of Data, SIGMOD '20, New York, NY, USA. Association for Computing Machinery, pp. 731-737, 2020.
11

---

Published as a conference paper at ICLR 2023
Lu Lu, Pengzhan Jin, Guofei Pang, Zhongqiang Zhang, and George Em Karniadakis. Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators. Nature Machine Intelligence, 3(3):218-22) 2021a.
Lu Lu, Raphael Pestourie, Wenjie Yao, Zhicheng Wang, Francese Verdugo, and Steven G. John- son. Physics-informed neural networks with hard constraints for inverse design. arXiv preprint 0t1::2102.006226, 2021b.
Ravi G. Patel, Nathaniel A. Trask, Mitchell A. Wood, and Eric Cyr. A physics-informed operator regression framework for extracting data-driven continuum models. Computer Methods in Applied Mechanics and Engineering, 373:1 13500, 2021.
Chris Rackauckas, Yingbo Ma, Julius Martensen, Collin Warner, Kirill Zubov, Rohit Supekar, Dominic Skinner, and Ali Ramadhan. Universal differential equations for scientific machine learning. arXiv preprint wrXiv:2001.04380 2020.
M. Raissi, P. Perdikaris, and G.E. Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378:686-707, 2019.
Jonathan D. Smith, Kamyar Azizzadenesheli, and Zachary E. Ross. EikoNet: Solving the eikonal equation with deep neural networks. IEEE Transactions on Geoscience and Remote Sensing, 59 (12):10685-1069)) 10696, 2021.
Alasdair Tran, Alexander Mathews, Lexing Xie, and Cheng Soon Ong. Factorized fourier neural operators. In International Conference on Learning Representations, 2021.
Rui Wang, Karthik Kashinath, Mustafa Mustafa, Adrian Albert, and Rose Yu. Towards physics- informed deep learning for turbulent flow prediction. arXiv preprint 1rXiv: 1.08655, 2019.
Sifan Wang, Hanwen Wang, and Paris Perdikaris. Learning the solution operator of parametric partial differential equations with physics- -nnfsrcs-nformee deeponets. Science Advances, 7(40):eabi8605, 2021.
E. Weinan and Bing Yu. The deep ritz method: A deep leamiig-basee numerical algorithm for solving variational problems. Communications in Mathematics and Statistics, 6(1):1-12, 2018.
F.M. White. Viscous Fluid Flow. McGraw-Hill international edition. McGraw-Hill. 2006.
Kailai Xu and Eric Darve. Physics constrained learning for data-driven inverse modeling from sparse observations. arXiv preprint urXiv:2002. 10521, 2020.
Yinhao Zhu and Nicholas Zabaras. Bayesian deep convolutional encoder-decodee networks for surrogate modeling and uncertainty quantification. Journal of Computational Physics, 366:415- 447, 2018
12

---

Published as a conference paper at ICLR 2023
The appendix is organized follows. Section A summarizes all the notations used in this work. Section[B] provides proof of Proposition listed in Section[.1] The proofs of main Theorems are provided in Section The details of data generation process is provided in Section Section further expands the experimental setup in addition to what deseribed in Section Finally, additional experiments supplement the main findings in Section of the paper are provided in Section
A NOTATIONS
PDEs, Boundary value problem (BVP) and operators
uo (x)
Initial condition of the given PDE.
A
Function space of initial conditions.
u(x,
Solution of the PDE at time 0.
Function space of solutions of given PDE.
Le
Function spaces such that the absolute value of the p-th power the element functions Lebesgue integrable.
HrP
Sobolev spaces that are subspaces of T such that the element functions and their weak derivatives up to order also belong to L..
Boundary constrained operators
auf
Function space such that each element function satisfies a given boundary condition.
Ubay
Function space such that each element function is a solution given BVP, and satisfies given boundary condition, i.c. Ubd = L au.
|1-1|azee
TP-norm computed at the boundary points.
Endy (T)
Boundary error incurred by a given operator T.
BOON kernel corrections
K
Kernel matrix K N for a given resolution > 0. Kerne matrix-vector multiplication computing module y K[xx) Kx.
IC
Kbdy
Boundary corrected Kernel matrix K E N for a given resolution N > 0. Kbty will differ for different boundary condi- tions.
B PROOF OF PROPOSITION
Proof.
Dirichlet: Given boundary location To 8Ω, boundary value function (ip t), input (n(x)) satisfying un (xo) = ap(xo, 0), we obtain u( t) using the integral kernel in eq. (2) with
a(x) = ux(x) as:
u(xo, t) uo(o)) = / K (xo, v))o(y)dy
I. op(() 1 zo)uo(y)dy (0) ap a !) (p 0) un un( an(
For the corner case (ap, 0) = 0. the division not defined. In such cases, we can still obtain the boundary corrected solution with arbitrary precision
0. First, note that, the operator is
13

---

Published as a conference paper at ICLR 2023
such that T : A - U, where A.U are H²? Therefore, the operator is compact, or it maps bounded
norm subspaces.
The operator kernel is K . Ω be the e' e'-perturbed version of a given imput uo( such that ||ube" - Up 122 S and up (To) = e. where e' > O. By taking the kernel K such that K K(ro,1) v) = an(a)s)) e-perturbed output ue (x))), > such that
L We first define sup, || (z, 1,- M. Next, let uo e (2)
(y = 20), we obtain an
ue (2o) =/o aD(ao.t) t) 1(y ro)uo, (v)dy = (20 t).
For any x + rv. we denote uz)) as the solution obtained by original input un (z). and
/ K Kxx, (zo,)) uo,e (2))dyy
lu(x) (x)(
|K (| u0. sup || (z, ||||_1//00000 uo,e'l1. 1.2
Me',
where, (a) uses Hölder's inequality. Given an oupptt-approximation we take e' e/M, Therefore. sup, |u(x) uz(2)) = e, or we obtain solution for any given precision such that u. (2o) (D t).
Note that, for the computational purpose, when we evaluate the functions and kernels on discretized grid (see Appendix
we simply replace the appropriate locations with the boundary values, and the modified kernel takes care of the continuity over a given input resolution.
Neumann: Given a boundary location zi E a9, boundary function ox( t). and an input uo(x), we obtain the derivative of the output uy ( f) by differentiating the integral kernel in eq. ( with
a(x) = ug (z) as:
T (Tuo(r))
Ue (zo, 1) =
dr
1==0
J. Ox
0
K K(x, (2,
1=00
K2 (5 y)uo (y)dd
/ aN(20. t) 1l (y zo)uo(y)ds uo(200)
ax(zo, 1) mo(2r)
un(z)) = aN(xo, t).
Note that, similar e' argument can be provided to deal with the special case of (o(o)) = as we did above for the Dirichlet BC.
Periodic: Given boundary locations Fp, 1 E an, and an input ux(x) satisfying periodicity, i.e.
(o())) (1(2NN1), we obtain u(xu, t) using the integral kernel in eq. (2) with a(x) un(x) as:
/ K( ())dyy)dy
u(zo. t) T (o(o))
K (xI (( 1,
T (o( = u(xN-11
14

---

Published as a conference paper at ICLR 2023
C PROOF OF THE THEOREMS
C.I REFINEMENT THEOREM
We first present the supporting Lemmas in Section C.1.I and then the proof in Section C.1.2
C.I.1 KERNEL CORRECTION LEMMAS
We have given resolution N, finite grid To T1...... ...................... -11 and discretized version of the continuous
kernel (x, y) as K [Ki,lo.N-1...N-] such that K E RNXN The operation in eq. (2) is
discretized as u(t) = Kug where ug E RV denotes the values the initial condition evaluated at the gridpoints, and utt) E RN denotes the approximated output the gridpoints for some t 0.
Lemma 1. Given a Dirichlet boundary condition u(zo, t) = aD( t), let 3 =
ao(zo.t) op(za.0) the
operator kernel can be modified for satisfying the boundary condition as follows:
Kidy
T
Dirichler
KT &
Dirichler
(9)
where,
9
0
K 3-N-1
T
Dirichler
Ko,o 0
Dirichler
1
Ko.0
(10)
IN-1
0
IN-
Proof.
The Kbay is computed as:
Kody =
Ko.0
Ko.0
KT
0.1:N-1
K I
Ro,a
IN-1
1
KENN-1,:NN1
IN-1
B
0
K
KINN-LIN-1
KINNL.K 011:N-1
Ko,0
With [0] ap(xo, 0), the output using modified kernel is u(t) Kbdy un such that u(t)[0] Gus [0] ap(ro.t) t).
Lemma 2. The continuous Neumann boundary condition U2 (z au((ag, f) after discretization
through general one-sided finite difference scheme, ut (z(o t) ~ En>o (t) + cou(0]
given coefficients Cg E R for k > O) and co E IR' [R){(),, and the following condition for discretized kernel
- k>0 Kkdj
i O,
Ko.j
(11)
E
d Kk.
ou(t) uo
j = 0,
0 are sufficient for a K to satisfy discretized Neumann boundary condition.
Proof. We show sufficiency through the discretized Neumann boundary condition, uz (z s Exso (I) (1)
Mar (2o, 1)
CRu (k)()))) + (L)
k>0
(Ku))
co (Kun)
k50
N-1
N-1
Ch Kk
4oli) + co
Σ
Ko. uolil
1=0
Ck Kr. + Kos
Co
uoli]
co
ON (2o, t)
[o aN(2o,!),
[o
where used eq. 11 to obtain the last equality.
15

---

Published as a conference paper at ICLR 2023
Given Neumann boundary condition (z t) (x. and its discretization through a general one-sided finite difference scheme. ug (z t) Exso (x + ct for given coefficients o E E. Co + O, and B ca ug(za)' the operator kernel can be modified through following operations to satisfy discretized Neumann boundary conditions.
Lemma 3.
au(
Kidy
T
Neumaun
KT K
Neumamn:
(12)
where,
GV
Neumaama
Ko.o 0
Neamann
Ko.0
IN-1
Proof.
The Kbay is evaluated as
Kydy
Kp,o
ON
co
Kg,0
KT
0.1N-1
0/
-Kopp 1
1
K
IN-1
K 1I:N-1.0
K 11:N-1,::-
0
IN-1
EnSo CKKEO
Σxso Ck CC(K (K'
(KE
K,I:N-1
Ko.o
1
Kn
Ki k(,NN-1)
11:N-1,0
KI
Ka,0
KI:N-1,0
K
U U,1:N-1
Now, (i Ki,0, Vi 0, therefore, (i) (Koayyo.o 3 Σ k20 ck (Kbay)a,, Next,
c0
I Ko, Vi,j > O, therefore, (ii) (Knaylo.j = Ekso (Kndy)k.j- Using (Kndylij = Kigj Ko.0
Lemma 2 Kody satisfies discretized Neumann boundary condition.
Lemma 4, Given periodic boundary condition (t) = u(N - 1](t) Vt, t, the operator kernel can be modified to satisfy the periodicity as follows.
K T periioticc K.
(13)
where,
periodic
a
0
IN-2
0
0
B
,
ORt ..Rt,.+0 E.B.Rttt+$ 8 ==
(14)
Proof. The Kıdy is computed
K = T periodic
0
3
KT
0
0
IN-2
K 12N:-
O
α
-11:NN-1
KT
- + K
0.O.N
KINN
(15)
KT 0.0:N-1
K
N-10.N111
With periodic input, i.e., un(0 Un N 1). the output through modified kernel in eq. u(t) Khay Uo is such that u (1) = uiN - (t).
15 or
1.2 PROOF OF THEOREM
Proof. Lemma and Lemma ensure sufficient condition for Dirichlet and periodic boundary conditions, respectively. An approximate satisfaction, through one-sided discretization derivative, shown in Lemma for Neumann boundary condition. Therefore, Theorem holds. n
16

---

Published as a conference paper at ICLR 2023
K
T Dirichlet
K
T
Dirichlet
T Periodic (c)
(a)
T Neumaon
K
T
Neumann
Figure 5: Operator refinement procedure. Kernel correction for Dirichlet, Neumann, and periodic in (a). (b), and (c), respectively. The purple color in (b) represents pre-computed weights for first-order derivative approximation (see Lemma
C.2 COMPLEXITY THEOREM
Given resolution N, the kernel matrix K E gg is, in-general, dense. Therefore, for an input un the operation u = K (un) using the proposed kernel corrections Lemma ! 3 and have a cost complexity O(N*), and space complexity of ( through a naive implementation. We show that if an efficient implementation exists for the matrix-vector operation y Kx, say K(x) which has cost complexity (NNx) (for example, Fourier bases in Li et al. (202aa), multiwavelets in Gupta et al. (202I)) then efficient implementation exists for our kernel correction (BOON) show in the following sections.
C.2.1 DIRICHLET
To implement the correction operation of Lemma we proceed with eq. (9) as follows.
u
T
KT2 Dirichle
Dirichler Ug.
(16)
First, let ý T Dirichlr Up therefore
[0]
KT - un
N 1]
Ko.0
Ug [1]
y
(17)
Ug N - 1]
[0]
Ko. KT 0,0:N -
Uo [1]
(18)
Up [N
2uo(0]
Ku,)
(Ke, 0
un [1]
(19)
un [N 1]
17

---

Published as a conference paper at ICLR 2023
where eo ........ 0J" is the first eigenvector, therefore, Ko,0 (Keu)[0). Now, if Kuo. then y in eq. (19) written as
O
[Kea) z
uo [1]
(20)
Mo N 1]
Next, let y KT
Dirichlet Up or y = Ky. Finally, we denote u T Dirichter y, or
(p(2).)) (ao.0)
[Kel] [[o]
[1]
(21)
y[N
Uo (Kep [(0]
an (mo.d)
y
(22)
: 8|N - 1]
Further, we observe that
y10]
KT
1
1
(o.o(uo!0
KT
Ko,IN-1u uo[1 N - 1J) - uo[1 N 1]
(23)
Ko,0
Ko.oua[O].
We finally express u as
aa t y
u
(24)
: y[N 1]
Note: For the above kernel correction implementation, never require an explicit knowledge the kernel matrix K but only a mearix-vectto multiplication module = K(x) = Kx. Therefore, the above operations for Dirichet correction can be implemented with kernel K. calls and O(N) space complexity. The steps are summarized in the Section 3.2 of main text Algorithm
Remark: For the special case when ( 0) 0, we note that Lemma is not defined but due to eq. 23 we can omit the division and still obtain boundary satisfied output through eq. (2))
C.2.2 NEUMANN
The correction using Lemma 3 is implemented via eq. 12 as follows.
T Neumanu KT Neumann Ug-
(25)
Since T Neumaan T Dirichler therefore, we follow the same initial steps outlined in Section C.2.1
first obtain y T Neumann Ug
2ua[0] [Kea] 2[0] Un 1]
(26)
Ug N
1]
where, Ko,0 = (0 and z = Kuo, as mentioned in Section C.2.1
Next, let = Ký and denote
T1 &. or
Kn
(27)
IN-1
18

---

Published as a conference paper at ICLR 2023
8 (Koo)l0)
5/0] - Ex>00
og
811]
(28)
8(N
ox (x(o
y
Σ k>0 5(k]
ý11]
(29)
y[N 1]
where the last equality uses from Lemma
Similar Section C.2. 1
observe that
8(0]
KT -1
1
Ko,o(uo(0]
K
- uol1 1ool11
N 1]) K K 1 N 1]
(30)
Ko,0
= [o)
Therefore, we finally express u as
(ou(2a.t)) 510] [0x(201))510
-E>o ot [[k]]
811]
(31)
SIN
Similar to Section C.2.. the Neumann kerel corrections also do not require an explicit knowledge of the kernel matrix and only a matrix-vector multiplication module y Kx K(x). Three calls to the kernel module I. are sufficient and can be implemented in O space with as an input. The steps are outlined in the Section 3.2 as
Note:
lifte-difference coefficients Co # 0, cg Algorithm
Remark: For the special case when on [0] = O, we note that Lemma is not defined. But due to eq. (30,, we can omit the division and still obtain boundary satisfied output through e1.
C.2.3 PERIODIC
The periodic kernel correction is implemented using Lemma and eq. eq. as follows.
T Periodic Kug-
(32)
Let Kuo, therefore, u is written as u T Periodic V,
u
T Periodic
(33)
O
8
0
I
N
y
(34)
0
3
Tay(0]
[9
IN
1]
{[1]
(35)
[ o [0] + 3 N
The steps are outlined as Algorithm in Section 3.2 Note that, periodic kernel correction only
requires one matrix-vector multiplication which can be obtained via y = Kx == K(x).
C.3 KERNEL CORRECTION ARCHITECTURE
The neural architectures BOON using Algorithm 2 and for Dirichlet, Neumann, and Periodic, respectively, are shown in Figure
19

---

Published as a conference paper at ICLR 2023
a
Kbdy
o
Kody
Kbdy
ax(20.0,ca
N//Noo
U1
11 Ug
11
IC
K:
Kox
K:
AC
Dirichlet Kody
Neumann Kody
ug
u
KC
Periodic Kody
Figure 6: BOON architectures. Kernel correction architectures for Dirichlet, Neumann, and periodic using Algorithm 1 2 and respectively. The line numbers of the algorithms shown at the corresponding locations. The complete BOON uses multiple corrected kernels K in concatenation along with non-linearity (.). Each correction stage uses kernel computing module A. from the base neural operator, for example, FNO (Li et al. 2020a), MWT (Gupla et al. (02T). The multiple appearances I. within each boundary correction architecture share the same parameters 0x. The MLP layers at the beginning and end are used to upscale and downscale, respectively, the channel dimensions for better learning. See Section for more details.
C.4 BOUNDEDNESS THEOREM
C.4.1 PERIODIC
Lemma 5. For given continuous input uolx) T.P. the output obtained by original operator T and kernel corrected T through Proposition for periodic are u(æ, t) and ū(x, t), respectively. The new kernel K (xx,y) such that
( K (wo, y) + 3 K(IN-1,1) == To S S TN-1,
k(z,y) v)
(36)
a K (xo. y) ((IN11.))
IN-1, To S y S EN-1.
where a, E. Bt, + 3 = 1. The solutions are bounded in L2-norm as follows.
||u(x, 1) 5(z f)222 a By²
|u(zo, t) u(IN-I,t)]
(37)
Additionally,
then the upper bound in 37 is minimized and we have,
|| t) - ū(zx t)),. S j((xx t) u(xN-1,t)l
(38)
20

---

Published as a conference paper at ICLR 2023
Proof. We take the continuous input domain s [No, the difference between the solutions is written as.
/ (u(z,t) ((c,tt))
||(xx,t) ()),,
/ I// K(ar, u(o,y))o(y)dy S. (xx, (a,ol)d)) 12 dx
1. 1// [K(x,g) - R(x, uol))dd)) dx.
Moreover, from Proposition
1 the kernels differ as
0, (1 a:) (xo, 2) 3K(
To < S IN-1,
K(r - k2
00, S 2N-1:
aKaa v) + (1 (xN-1)1)
I -N-1. S 2-
(39)
Therefore, using eq. (39). we have
[|u(,, t) ū(z t) 1) (u(ro. - ū(zo, t))² (u(IN-1,t) -,(
2 (1 a))(uo, 1) - Bul Bu(
+ - 1) + +(1 (B)_(uuxx--
(a) (1 a) (u((o, =1 t))² +(1 (1-3)((u(z00{)) -((=N1,)))
(1 a)² + (1 33²] (u(zo.t) u(xx-1,t))²
where in (a) we use + 8 1. Again, the bound can be further tightened
[|u(x, t) - ū(æ, 1) I V1 2a + 2a |u(zo, t) - u(zN-11
(40)
which attains minimum at 0.5.
Corollary 1. A discretized input un E IRN through discretized kernels K and K (from Lemma results in u(l) and ü(t) such that
4
- [((((t) (t)]
(41)
C.4.2 DIRICHLET
Lemma 6. For a given continuous input ug(x) ) E L?, the output obtained by original operator T and kernel corrected T through Proposition for Dirichlet are u(z, t) and ū(x, t), respectively. We take C ap(wa.)) the new kernel is written an(x0.0)
(C1y= 20)
Io. S IN-1
k(
(42)
((x,v)
o/w
The outputs are bounded
[|u( 1) - ū(x 1)1l
-v (uloo.t) -Vlulontt (x)) ))²² [((zo.t) K K(ro, 20))((2))
2r I K2 vo)de
1]
(43)
K² (2 20)
Proof. The difference between outputs is written
[u(x, 1) ū(., 0)1i. = / (u(x,
(](or,))-un - ūū(x, 2)
S 11/ [K(2, v) K k w) uo(y)dy) dr.
21

---

Published as a conference paper at ICLR 2023
The kernels differ follows
K (ro, x00 (zo. v)
x ro. y = 20
(xx y) K(x,y) v)
x = To. 2o < y S IN-1
(44)
0fW
Using eq. 44 we can bound the difference between ux () and oz t) as [1(a, 1) ū(x; t)1l? [/ʃ K (( (oooy))o(y)dd (ou(
(u(zo.t) (D)(t,t))²
Corollary 2. A discretized input uy E IRN through discretized kernels K and Kody (from Lemmal results in u(t) and ü(t) such that
K
[lu(t) ū(t)ll
(u(0](t) ap (2o. t))² (u[0](t) Ko,o
1)
(45)
K2.0
Proof. Let C = ap vo. ()/ap(zo,0) 0). The difference between the kernels, K and corrected version
Kidy from Lemma I
Ko,0 C
i= 0.j =
Kijj Kidy i.j
Kij Kiokos
i= i=0,iiisN-1.
(46)
ofw
Kao
Now, estimating the difference between utt) and ü(t), we have
u(t) ü(t)
Kun
Kidy Ug
[Ko,0 C)uo(0] + K
0, - 1 : 1]
KINN-1,0 KT K uo N - 1]
Kg.0
[(] (t)
KI:N-1KKK
0.1:N un [1: - 1]
Ko,0
Therefore, the norm of difference is written
||((t) l12 (t) (2o t))² +
K 1,0 K
D.I:N-11 .uo1
K2.0
N-1
N-1 K"D0
(u) ap t)) +
Ko.
1=1
1=1
K2.0
N-1 K
(u[0](() - ap 0)) + (t) (o)o
Σ
-1)
i=0
K
Note: When the kernel is diagonally dominant, i.e., ΣiN K2 so K2 then the overall difference
between the solutions is boundary error.
C.4.3 NEUMANN
Lemma 7. A discretized input un E N through discretized kernels and Kidy (from Lemma results in u(t) and ü(t) such that
[u(I) - ū(())a( < v V 13.
(47)
where,
2
f u(()(1) ax(
CkKio (Ko.ouo(0) u(((()))
+ cau(k](()
Co
Co
Co
Kn,0
k20
k>0
K2
2 = (u(()(() - Ko 2
Σ
1,0
0=0
K2
22

---

Published as a conference paper at ICLR 2023
Proof. Using 3 aN(ato co no [0)
the difference between the kernels, K and corrected version Kudy from
LemmaBis
Ko,0 - - Eksoo
i = 0, 0,
Ku,j + Ex Sh (KK Ko.0
1 KkoKKoj)
i = 0, > 0,
Kif Kb
(48)
0, 1 Ko,
i> 0, = 0, o/w.
No,0
Therefore, difference between u(t) and ü(t) is written u(t) ü(t) = (K (Kdy)up
(Ko,0 - 8 L Exso Ca
(x,x))000000
+ Σ>zo (Ko.s + Ekso Ckl Kk.
Ko.a
1 KLKKNN 1o KT
Ko,0
Therefore, the norm of difference is written as
|lu(t) ú(t)ll
(150.0 8
Σ c0oo0))o000+++
C0
k>0
Σ
(Ko.j +
[K.j
KR. Ko.oKooluolii)
Ko,0
j30
k>0
+
KT K 1
K2.0
2
[(] (N(x
CKK [Ko u[0](t))
cxu((t((t)
+
co
CO
coKo.0
k>0
k>0
+ (t) - oo
Σ
N-1
K2.0
i=0
K2
0000
C.4.4 PROOF OF THEERRM 3
Proof. Using Lemma 6
5 and
Theorem holds.
L
D DATA GENERATION
In this section, we discuss the data generation process to form the dataset D of size Taata consisting of input and output pairs. We take the input in the mapping a(x: == mo(r) to be the initial condition. We first discretize the initial condition a(x) on spatial grid of resolution N, i.c. a; := a(x;) for H; E 22 O,... N 1. We then discretize the exact reference solution u(x, on spatio-temporal grid of size N x N₁, iee. Us.j == u(x,,t,) for I; E Ω i = O...., ....,NN N 1, and t, ts+1 E R4, 1,..., N₂ with fixed spatial step sizes Ar (ÆN-1 2o)/N, and temporal step size At = NN, on time interval [0, tN.]. We construct the dataset using the solution at the last M time steps, that is, times IN, -++1 < < LN, RR.. We take different N, based on the dimensionality of the problem (See Table We take M = for single-step predictions, and M = 25 for multi-step predictions. the following subsections, we discuss the selection over various input and output pairs, by uniformly sampling different PDE parameters in the initial condition.
Dimension of Problem
T2jata
Araii
Thest
N₂
M
ID space
600
500
100
2D (1D space + time)
1200
1000
200
200
25
3D (2D space + time)
1200
1000
200
30
25
Table Data generation sizes and parameters for different dimensions (1D, 2D and 3D). Multi-step predictions are included in the 2D and 3D problems.
23

---

Published as a conference paper at ICLR 2023
D.1 ID STOKES' SECOND PROBLEM
The one-dimensional Stokes' second problem is given by the heat equation in the y y-coordinate as:
Ut yy uo y) Ue kw cos()));
[0, 1|. 0, y E [0, 1]. 12 0,
(49)
" - 0, 1) - U cos(wit,
with viscosity v 2 0, oscillation frequency w, k (////2// U 2 0, and analytical solution:
Mexact (z, t) Ue cos wt).
(50)
We construct the dataset D by fixing E {0.1, 0.02, 0.005, 0.002, 0.001). U 2 and sampling w uniformly in [3, 4] obtain the input and output pairs, consisting of the corresponding discretizations of the initial condition in eq. (49) and exact solution in eq. (50), respectively. The spatial resolution is N 500, and temporal resolution is N₂ for t [].
D.2 ID BURGERS' EQUATION
The one-dimensional viscous Burgers' equation is given as:
U4 + (u² 1/lzz
E [0, 11,
(51)
with lon-linear and convex flux function f(u) u² /2 and viscosity 2 O. The diffusive term Vu on the right hand side of eq. (51) is added to smooth out the sharp transitions in u(z,)) t). Depending on the initial condition, the solution can contain shocks and/or rarefaction waves that can be easily seen with classical Riemann problems Veque (1992)). In various applications, it critical that the For example, consider an aircraft
numerical solution has the correct shock speed dæ /tz/tt dt f'(w) wing flying at high enough speed for the flow to develop sonic shocks. The pressure distribution on the aircraft wing, which determines the lift and drag forces, is highly sensitive the shock location.
D.2.1 DIRICHLET BOUNDARY CONDITION
We consider various Riemann problems for which the initial condition uo(x) is given
un (x) =
uE
if æ S 0.5, uR-
(52)
x > 0.5,
where ut, uR e R. The exact solution of eq.
5T with initial condition in eq. (52) is given as:
u1 + un
Wr
un
tanh (x
st) un)
Mexucl (I, 1)
(53)
2
4u
where s (uy u2)/2 denotes the shock speed. We prescribe time- varying Dirichlet boundary conditions on both boundaries, i.e. u(0,t) = M0x f) and u(1,t) = Mexan (1, t), Vt 2 0. We choose us. >> Un leading to the formation of shock the solution u(x; 4) of eq. (5.). We construct the dataset ID by fixing un 0, and varying the value of uz, = 6/4, where
0.8, = 0.01
and μ ~ (0, 1) to generate different initial conditions in eq. (52) and exact solutions in eq. (51) for t e [0, 1.2).
D.2.2 PERIODIC BOUNDARY CONDITION
We construct the dataset D using a finite difference reference solution of Burgers' equation for every random sample un (x) and E [0, 1] done in Li et al. (2020a).
D.3 2D STORESSSOOKESSEEE EQUATION
We consider two-dimensional lid-driven cavity problem, which consists of a square cavity filled with fluid. The behavior of the fluid is described by the incompressible Navier-Stokes equations given as:
T + Z vū=-Vp + (1/Re)
2.y [0, 0,
(54)
V 0,
x,, V [0. 1|2 0,
24

---

Published as a conference paper at ICLR 2023
where [w(),, u(2)]² T denotes the 2D velocity vec- tor, the pressure and the viscosity. The initial condition is given uo(x, y) [U, 0]" (y 1) for y, E [0, 1]. We enforce Dirichlet conditions, Le. 1.) = IU, 0f for U O, ū((z,0, 27(0, =(,,)
a U,
w(x, 1,4) = - (x,10
(3'00))
DiDDee 2r = (9.01) 0= (ar0
0 ( cn
(1))
in '0ML
Ó. Figure Sosureeilluee iillustrates that at the top boundary, tangential velocity is applied drive the fluid flow in the cavity, and the remaining three walls are defined as no-slip boundary conditions with velocity. For computational efficiency,
6'4Dee
express the Navier-Stokes equation in terms of the scalar & component of the vor- ticity
w(x.0,1) u. (x. 0.t) u 0,
V x 2T) [0, 0, 1)" in eq. (6) done in Li iet al. (2020a). We construct the dataset D by varying the value of U' in the initial condition uniformly in [1, 1.5]
Figure 7: No-slip boundary conditions for the lid-driven cavity problem and streamlines showing the direction the fluid flow.
to generate the different samples of the initial condition and the corresponding reference solution of eq. (54p. We
use the projection method (Chorin (1688) on a staggered grid with N2 grid points (I:; Mj) with ž.j = I....
compute the reference solution. The pressure field is computed by solving a Poisson equation using second order central discretization, to enforce the incompressibility condition. For the time-stepping schemes, we use second-order a Crank-Nicolson method for the diffusive term (Le (1992)) for N, 30 timesteps over the interval [0, 2). Lastly, we compute (2 /ör Ou!!) /Oy using finite differences.
method for the non-linear term and
D.4 1D HEAT EQUATION
The one-dimensional heat equation is given as:
U kuzz f(x,t),
x x 10, x [0, 1].
(55)
uo (z) cos(wuzz).
where k > 0 denotes the conductivity, f(r, denotes a heat source, and w E R. We prescribe Neumann boundary conditions, i.e. uz u= 1) = 0 and (1, = U sin al with R, and choose
f(x, t) Un= cos शt. The exact solution of eq. (55) is given as:
Uk Mexaca (x,, t) = U x² sin(TT) (cos(TTt) - 1) + sin(wT) 2
an cos(nxare k(nn)² -(an)) (56)
π
WT
n=1
where
sinl(w + +n)m))ll n)m]/[(w n)r] sin((w n)>l/[(w n)m], if +
On
(57)
if n,
We construct the dataset D by sampling uniformly in (2.01,3.99) with fixed k: = 0.01 and U to obtain the input and output pairs, consisting of the corresponding discretizations the initial condition and exact solution in eq. (56). respectively grid with resolution N. The temporal resolution is given by N, on the time interval [0, 2].
D.5 2D WAVE EQUATION
The two-dimensional wave equation is given as:
( + Uyy),
3,y E [0, 10,1)2,4
(58)
where e E R2 denotes the wavespeed. We impose the following Neumann boundary conditions:
uz (0, y, t) u. (1, v) (1, uy (I, 0,t) (z, 1, = 0. for 0. The initial conditions given
as: un (ax, 3) = & cos(coa (TT) cos(TT)) with R and uy uz( y, 0) O. The exact solution of eq.
eq. (58) is
given as:
Mexaal (xr 1) l: c00((T))) cos(Tr)) (os(ev/2m))
(59)
25

---

Published as a conference paper at ICLR 2023
We construct the dataset D by fixing 1, and sampling uniformly in [3, 4] to obtain the imput and output pairs, consisting of the corresponding discretizations of the initial condition and exact solution two-dimensional uniform mesh with resolution N i... (I Vy)) with N. The temporal resolution given by N, on the time interval [0, 2).
in eq. [59). respectively
j
E EXPERIMENTAL SETUP
In this section, we list the detailed experiment setups and parameter searches for each experiment. We incorporate time into the problem dimension for multi-step predictions. Table|Bssows the parameters values selected for BOON and FNO. For PINO, we conduct multi-dimensional grid search, over the values {0, 0.0001, 0.001, 0.01, 0.1, 13. to find the optimal weights attributed the PDE, boundary and initial condition losses and report the minimum L errors in all our tables. We use the default hyper- preerpparametes for the other benchmarks in our experiments.
Dimension of Problem
Modes
Channels
ID space
16
64
2D (ID space time)
12
32
3D (2D space time)
8
20
Table 8: Training parameters in BOON for problems of different dimensions (ID, 2D and 3D).
F ADDITIONAL EXPERIMENTS
In this section, we include additional experimental results, which show that our proposed BOON has the lowest relative L error compared to the state-of-the-art baselines, and has the lowest absolute 2 boundary errors several single-step (M 1) and multi-step (M 25) predictions.
F.1 DIRICHLET BOUNDARY CONDITION
F.1.1 ID STOKES' SECOND PROBLEM
We are interested in learning the operator T un (v) + u(y.t). t 2, where uo(y) Uc-ky cky( and = /w/20 22 for the one-dimensional Stokes' second problem with Dirichlet boundary conditions (See Appendix D.I. Table |9 shows the results for single-step prediction (See Table 2 for the multi-step prediction results).
Model
0.1
0.02
= 0.005
0.002
0.001
ROON (Ours)
0.0189
0.0370
0.0339 (0)
0.0354
0.0370
FNO
0.0199 (0.0043)
0.0410(0.0093)
0.0578 (0.0135)
0.0370 (0.0077)
0.0557 (0.0116)
APINO MGNO
0.1343 (0.0914)
0.1398 (0.0636)
0.0972 (0.0271)
0.0898 (0.0181)
0.0986 (0.0172)
0.28457 (0.03530)
0.25150 (0.03778)
0.25007 (0.03574)
1659 (0.0208)
0.1609 (0.0238)
Table 9; Single-step prediction for Stokes' with Dirichlet BC. Relative 1,2 error (T.2 boundary error) for Stokes" second problem with varying viscosities y at resolution N 500 and 1.
F.1 ID BURGERS' EQUATION
We are interested in learning the operator T uo(z) u(x, f [0. 1.2], where un(x) is defined in eq. (52) for the -ie-dmmnnioonl viscous Burgers' equation with Dirichlet boundary conditions (See Appendix D.2.I). Table shoss the multi-step prediction results, where we train our model to predict solutions at the last M 25 time steps (See Table for the single-step prediction results).
26

---

Published as a conference paper at ICLR 2023
Model
0.1
0.05
0.02
0.005
0.002
BOON (Ours)
0.00040 (0)
0.000399 (0)
0,009366 (0)
0.00023 (0)
0,00045 (0)
FNO
0.000433 (0.00017)
0.00050 (0.00027)
0.00060 (0.00041)
0.00088 (0.00061)
0.00120 (0.00039)
PINO
0.01232 (0.05581)
0.01318 (0.06491)
0.01583 (0.08101)
0.02243 11506)
0.02358 (0. (2000)
DeepONet
0.00577 (0.00139)
0.00459 (0.00120)
0.00540 (0:00178)
0.00633 (0.00150)
0.00723 (0.00161)
Table 10: Multi-step prediction for Burgers' equation with Dirichlet BC. Relative L error (L² boundary error) for various benchmarks with varying viscosities v, and 25.
F.2 PERIODIC BOUNDARY CONDITION
F.2.1 ID BURGERS' EQUATION
are interested in learning the operator T un(x)) u(x, where up NO, 625
25 2)- and A and / denote the Laplacian and identity matrices, respectively, for the one- dimensional viscous Burger's equation with periodic boundary conditions, as done Li et al. (2020a) (See Appendix (.2.2). Table shows the results for the single-step prediction, where t I. Table 12 shows the multi-step predictions, where 2 O, M = 25 and final time tM = 1. We see that BOON has the highest performance compared to the other baselines across various resolutions N. As expected with Neural Operators, we see that the BOON error is approximately constant, as we increase the grid resolution N. In addition, BOON achieves small error in the low resolution case of N = 32, showing its effectiveness in learning the function mapping through low-resolution data.
Model
N 32
N = 64
N = 128
N 256
N 512
BOON (Ours)
0,006466 (0)
0,00243 (0)
0.00221 (0)
0.00176 (0)
0.00132 (0)
FNO
0.0066 (0.00046)
0.0028 (0. 00.00088)
0.0029 (0.00161)
0.0021 (0.00147)
0.0014 (0.00081)
APINO
0.0121 (0.00822)
0.0045 (0.00452)
0.0023 (0.00236)
0.0016 (0.00143)
0.0013 (0.00075)
MGNO
0,0390 (0.0137)
0.0473 (0.00751)
0.0606 (0.00786)
0.0706 (0.00702)
0.0807 (0.01109)
Table 11:
Single-step prediction for Burgers' equation with Periodic BC. Relative L error (L2
boundary error) for various benchmarks with varying resolutions N, viscosity 0.1, and M 1.
Model
N = 32
N = 64
N = 128
BOON (Ours)
0.00097 (0)
0.00093 (0)
0.00101 (0)
FNO
0.00133 (0.00160)
0.00104 (0.00175)
0.00201 (0.01152)
PINO DeepONet
0.00170 (0.00087)
0.00153 (0.00020)
0,00175 (0.00005)
0.08857 (0.00356)
0.08805 (0.00529)
0.08615 (0.00733)
Table 12: Multi-step prediction for Burgers' equation with Periodic BC. Relative I,2 error (I.2 boundary error) for various benchmarks with varying resolutions N, viscosity v 0.1 and M 25.
F.3 NEUMANN BOUNDARY CONDITION
F.3.1 ID HEAT EQUATION
We are interested in learning the operator T uo(x)) + u(z,t), t 2, where uo(x) cos(wTæ) for the one-dimensional heat equation with Neumann boundary conditions (See Appendix D.4). Table 13 shows the results for the single-step prediction (See 5 the multi-step prediction results).
27

---

Published as a conference paper at ICLR 2023
Model
N 50
100
250
500
BOON (Ours)
0.00901 (0)
0.01070 (0)
0.01170 (0)
0,00811 (0)
FNO
0.01636 (0.43876)
0.01711 (0,48925)
0.01694 (0.50789)
0.01730 (0.50619)
APINO
0.10457 (0, (0.24939)
0.09128 (0.33732)
0.07999 (0.21998)
0.08310 (0.21914)
MGNO
0.04685 15536)
0.05410 (0.5251)
0.07314 (1.7357)
0.05796 (1.3783)
Table 13: Single-step prediction for the heat equation with Neumann BC. Relative L error (L²
boundary error) for various benchmarks with varying resolutions N and M = =.
F.4 PREDICTION AT HIGH RESOLUTION FOR HEAT EQUATION
In this section, we evaluate the resolution extension property of BOON using the dataset generated
from heat equation with Neumann boundary conditions dataset (See Appendix D.4). We train BOON on dataset D irain with data of resolution N E (256, 512, 1024]. We test our model using a dataset Dest and finite difference coefficients ej generated using resolution N E {1024, 2048, 4096). Table 14 shows that on training with a lower resolution, for example, N 1024, the prediction error at 4X higher resolution N = 4096 is 0.01251 or 1.25%. Also, learning at an even coarser resolution of 512, the proposed model can predict the output of data with 8.X times the resolution, i.e.
N 4096, with relative le error of 2.16%.
Test
N 1024
N 2048
N 4096
Train
N 256
0,07197
0.08051
0,08447
N 512
0.01527
0.01947
0.02159
N 1024
0.00813
0.01086
0.01251
Table 14: Resolution invariance for Neumann BC with single-step prediction. BOON trained at lower resolutions can predict the output at higher resolutions.
F.5 KERNEL CORRECTION APPLIED TO MULTI-WAVELET BASED NEURAL OPERATOR
In this section, we show our proposed approach of kernel correction can also be applied other kernel-based neural operators, e.g. the Multiwavelet-basee neural operator (MWT) (Gupta et al. (202T). ). We evaluate the performance of MWT and MWT-BOON on Stokes' second problem and Burgers' equation with Dirichlet boundary conditions (See Appendices D.I D.2.1 respectively). Table 15 that BOON operations also improve the MWT-based neural operator.
Model
V = 0.1
== 0..05
= 0.02
L/ = 0.005
= 0.002
BOON-FNO
0.00012 (0)
0,000100 (0)
0,000084 (0)
0,00010 (0)
0.00127 (0)
BOON-MWT
0.00020 (0)
0.00025 (0)
0.00022 (0)
0.00020 (0)
0,00034 (0)
FNO
0.0037 (0.0004)
0.0034 (0.0004)
0.1028 (0.0005)
0.0043 (0.0004)
0.0050 (0.0022)
MWT
0.00020 (5.25e-6)
0.00020 (2.44e-05)
0.00026 (8.93e-05)
0.00027 (0.00030)
0.00054 (0.00013)
APINO
0.0039 (0.0004)
0.0040 (0.0005)
0.0037 (0.0007)
0.0048 (0.0008)
0.006 (0.0016)
MGNO
0.0045 (0.0004)
0.0046 (0.0005)
0.0048 (0.0008)
0.0064 (0.0015)
0.0101 (0.0006)
DeepONet
0.00587 (0.00124)
0.0046 (0.00130)
0.00580 (0.00128)
0.00623 (0.00140)
0.00773 (0.00141)
Table 15: Single-step prediction for Burgers' with Dirichlet BC. Relative L test error (L3
boundary error) for Burgers' equation with varying viscosities resolution N 500 and M 1.
28

---

Published as a conference paper at ICLR 2023
Model
== 0.1
0.02
0.005
0.002
0.001
BOON-FNO
0.0189 (0)
0.0370 (0)
0.0339 (0)
0.0354 (0)
0.0370 (0)
BOON-MWT
0,0701 (0)
0,0602 (0)
0.0631 (0)
0.0579 (0)
0,07992 (0)
FNO
0.0199 (0.0043)
0.0410 (0.0093)
0.0578 (0.0135)
0.0370 (0.0077)
0.0557 (0.0116)
MWT
0.0796 (0.0125)
0,1080 (0.0150)
0.0940 (0.0164)
0.0889 (0.0130)
0.1060 (0.0176)
APINO
0.1343 (0.0914)
0.1398 (0.0636)
0.0972 (0.0271)
0.0898 (0.0181)
0.0986 (0.0172)
MGNO
0.28457 (0.03530)
0.25150 (0.03778)
0,25007 (0.03574)
0,1659 (0.0208)
0.1609 (0.0238)
Table 16: Single-step prediction for Stokes' with Dirichlet BC. Relative L error (L² boundary error) for Stokes' second problem with varying viscosities y at resolution N 500 and M 1.
29


Reviews:
[{"review_id": "Y6kX-8Q9frp", "reviewer": "Reviewer_2TdD", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper adds support for boundary conditions to FNO’s, which are neural operators that learn the solution map of unknown PDE systems.\n", "summary_of_the_review": "This is a solid paper that succesfully includes proper boundary behavior to FNO models. \n", "strength_and_weaknesses": "\ns: Principled method.\n\ns: The results are convincing. The method achieves perfect behavior at boundaries (as expected), and also shows moderate interim accuracy improvement. \n\nw: This paper is incremental since it does an “FNO + BC” type model. The paper doesn’t motivate or demonstrate why adding BCs is important. I’m not sure if this is that important feature, and the author’s should more clearly argue why it’s important to include boundary conditions in FNOs.\n\nw: There are clarity issues. The algorithms 1-3 are the key contribution of this paper, and yet they are gibberish to me with no explanations. I can’t really follow the main idea of how the kernels are corrected.\n\n\nTechnical comments\n* Since PDEs are not linear in reality, how come it makes sense to still use the kernel form eq 1? I don’t think the “note” explains this\n* What does L^2 mean in eq 2? a(y) lives in \\A space, so how do we define a product between L^2 and \\A instances?\n* What does s and p denote?\n* In eq 1 G is a function, in eq 3 it's an operator. Is this intentional?\n* G only applies to values at boundary, by definition. In eq 3 we apply G to solutions of any points a \\in A. The G is thus mostly undefined in eq 3.\n* The U_bdy is an intersection of space and its boundary. Isn’t that then trivially just the boundary? Is this a typo?\n* What about the boundary of A, why is that not considered (around eq 3 and fig2)\n* How come in \\D we only have pairs from A and U_bdy. Why can’t we have points from boundary here? Is it intentional that this now restricts A implicitly (or does it)?\n* || is L2 error at boundary, yet most points inside the sum in eq 3 are not at the boundary. I don’t understand. Do you then skip some of the points inside the sum?\n* If we have boundary condition \\alpha=0, doesn’t that lead to division-by-zero in prop1?\n* I would expect a common boundary condition is that we have a time-invariant fixed value at boundary. This would then lead to a Dirichlet kernel that is simply an indicator function. Surely this is not a useful kernel and does nothing. What would we do then?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "D5i9RXzF4er", "reviewer": "Reviewer_Z7Rp", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes the additional use of boundary conditions for Neural Operator networks.\nThe boundary conditions are explicitly enforced in the parameterized operator kernel, such that they are in principle fulfilled \"by construction\".\nDirichlet, Neumann, and Periodic boundary conditions are considered.\n\nFor experiments, the proposed method is applied to Fourier Neural Operators (FNO) and compared to state-of-the-art operator learning architectures.\nIn multiple numerical experiments for PDEs in 1D and 2D (+ Time Dimension) with different boundary conditions,\na significant decrease in relative L2 error is demonstrated compared to vanilla FNO and other architectures.", "summary_of_the_review": "The paper provides a strong performance improvement to Neural Operator networks in settings where boundary conditions are available.\nThe novelty lies mainly in additionally considering boundary constraints in existing Neural Operator Architectures.\n\nHowever, due to the lack of clarity, I vote against accepting this submission.\nThis is mainly due to the lack of explanation for algorithms 1 to 3 and in general section 3.2.\n\nI'd like to clarify, that the method is a valuable contribution, but would greatly benefit from another iteration.\n\n\n***\n## Edit after Author's Response\nThe authors resolved with their response a few issues, namely:\n- Notation was clarified with additional info provided in the Appendix\n- Appendix C.2 was added, which explains the origin of the previously hard-to-understand Algorithms 1-3\n- The issue of repeated experiments was resolved, by providing information about very little variation in the experiment results.\n- A discussion of the special case for enforcing a boundary value to be zero was added to Appendix B., showing that in principle a solution with any given (but non-zero) precision is still possible.\n\nWith that, most of my major concerns have been resolved and I will raise my score.\nI still have my doubts regarding the structure of the paper and the usefulness of Algorithms 1-3 in the main part, while all of the actual explanations happen in the Appendix.", "strength_and_weaknesses": "Strengths:\n1. The proposed kernel correction results in a significant performance improvement in settings where the boundary conditions are known. This is exemplified with a wide range of extensive experiments.\n\n2. The kernel corrections are to my understanding generally applicable to Neural Operator architectures, not just FNOs. In the appendix, an experiment with application to the Multi-Wavelet based Neural Operator is showcased.\n\n3. Code is provided, and it seems to be well documented and maintained.\n\n\nWeaknesses:\n1. There is a lack of discussion on how the setting differs from other work on Neural Operators (e.g. FNO), i.e. what assumptions are made and what limitations the proposed kernel refinement has.\nFor example: Is this hard-coded approach for boundary conditions still applicable if we consider settings of the same PDE with differing boundary conditions?\n\n2. There are severe clarity issues. E.g. the algorithms for the kernel corrections are provided as pseudo-code, but there is no proper explanation of the underlying idea. One of the reasons is that a \"kernel module\" $\\mathcal{K}$ is suddenly used, which is hand-wavely mentioned in the paper but never properly defined.\n\nGeneral:\n\nI strongly recommend the authors to restructure the paper.\nInstead of the general and vague introduction of Neural Operators and the kernel correction, I'd suggest concretely introducing one architecture (e.g. FNO), and how the kernel correction can be applied to it.\nHighlighting the generality of the approach can still be done afterwards, but right now the lack of a concrete example is confusing.\nAlthough the extensive experiments are of course valuable, they could partially be moved to the appendix, offering some space for section 3.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "4ZwYvjIdYH8", "reviewer": "Reviewer_ESqy", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper uses the neural operator framework with kernel to learn solutions to PDE (see e.g. Neural Fourier Operator work, FNO) as a base method. The paper add modifications to the discretised kernel to achieve better predictions on the boundaries on Dirichlet, Neumann, Periodic boundary conditions.", "summary_of_the_review": "Generally a good paper, I have not checked through all equations. \n\nMinor points: \n1. In Eq.(1), would be more complete if the authors define the mapping of u.\n2. Is there a typo when the author define the mapping of kernel right before Eq. (2)?", "strength_and_weaknesses": "The kernel manipulation process shown in appendix or \"Algorithm\" table in page 5 is simple with low computational complexity. Yielding better numerical results.\n\nWhen the kernel changes (w.r.t. say FNO), the solutions at the internal regions also changes. Authors give a theorem to bound the new solution with respect to the solution without kernel manipulation.\n\nI may have missed it, it seems no bounds for the solutions with respect to the true solution is given.\n\nPaper is evaluated for 1D and 2D systems. Good if evaluation can also be done on 3D systems which is more useful in real world applications.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster