BENO: Boundary-embedded Neural Operators for Elliptic PDEs
Authors: Haixin Wang
Jiaxin LI
Anubhav Dwivedi
Kentaro Hara
Tailin Wu
Venue: ICLR 2024 poster (2024)
Keywords: AI for PDEs; physical simulation
neural operators
boundary-embedded
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
TL;DR: We introduce a boundary-embedded neural operator that incorporates complex boundary shape and inhomogeneous boundary values into the solving of Elliptic PDEs

Abstract: Elliptic partial differential equations (PDEs) are a major class of time-independent PDEs that play a key role in many scientific and engineering domains such as fluid dynamics, plasma physics, and solid mechanics. Recently, neural operators have emerged as a promising technique to solve elliptic PDEs more efficiently by directly mapping the input to solutions. However, existing networks typically neglect complex geometries and inhomogeneous boundary values  present in the real world. Here we introduce Boundary-Embedded Neural Operators (BENO), a novel neural operator architecture that embeds the complex geometries and inhomogeneous boundary values into the solving of elliptic PDEs. Inspired by classical Green's function, BENO consists of two Graph Neural Networks (GNNs) for interior source term and boundary values, respectively. Furthermore, a Transformer encoder maps the global boundary geometry into a latent vector which influences each message passing layer of the GNNs. We test our model and strong baselines extensively in elliptic PDEs with complex boundary conditions. We show that all existing baseline methods fail to learn the solution operator. In contrast, our model, endowed with boundary-embedded architecture, outperforms state-of-the-art neural operators and strong baselines by an average of 60.96%.

Published as a conference paper at ICLR 2024
BENO: BOUNDARY-EMBEDDER NEURAL OPERATORS FOR ELLIPTIC PDES
Haixin Wang' Jiaxin Li Anubhav Dwivedì", Kentaro Hara', Tailin Wu
1 National Engincering Research Center for Software Engineering, Peking University, 2 *Department of Engineering, Westlake University,
3 DDepartment of Astronautics and Aeronautics, Stanford University wang. hx@stu pku. edu. en, lijiaxii ansiiaiii@essslaeee edd.. (anubhavd, kenhara fenharaj@stanfore @stanford. edu,
cn,
ABSTRACT
Elliptic partial differential equations (PDEs) are major class of time-independent PDEs that play a key role in many scientific and engineering domains such as fluid dynamics, plasma physics, and solid mechanics. Recently, neural operators have emerged as promising technique to solve elliptic PDEs more efficiently by directly mapping the input to solutions. However, existing networks typically cannot handle complex geometries and inhomogeneous boundary values present in the real world. Here we introduce
Hoindary-Embedder Neural Operators (BENO), novel neural operator architecture that embeds the complex geometries and inhomogeneous boundary values into the solving of elliptic PDEs. Inspired by classical Green's 's function, BENO consists of two branches of Graph Neural Networks (GNNs) for interior source term and boundary values, respectively. Furthermore, a Transformer encoder maps the global boundary geometry into a latent vector which influences each message passing layer of the GNNs. We test our model extensively elliptic PDEs with various boundary conditions. We show that all existing baseline methods fail to learn the solution operator. contrast, our model, endowed with boundary-embedded architecture, outperforms state-of-the-arr neural operators and strong baselines by an average of 60.96%. Our source code can found https://github.com wtt.://.....ommAmAeeeeeaea West Westtakeel/beno.qr
INTRODUCTION
Partial differential equations (PDEs), which include elliptic, parabolic, and hyperbolic types, play a fundamental role in diverse fields across science and engineering. For all types of PDEs, but especially for clliptic PDEs. the treatment of boundary conditions plays an important role in the solutions. In particular, the Laplace and Poisson equations constitute prime examples of linear elliptic PDEs, which are used in a wide range of disciplines, including solid mechanies (Rivière 2008). plasma physics (Chen 2016), and fluid dynamics (Hirsch] (007].
Recently, neural operators have emerged as a promising tool for solving elliptic PDEs by directly mapping input to solutions (Li et al. 2020b a Lötzsch et al., 2022). Lowering the computation efforts makes neural operators more attractive compared with classical approaches like finite element methods (FEM) (Quarteroni & Valli 2008) and finite difference methods (FDM) (Dimov et al.) 2015). However, existing neural operators have not essentially considered the influence of boundary conditions on solving elliptic PDEs. A distinctive feature of elliptic PDEs is their sensitivity to boundary conditions, which can heavily influence the behavior of solutions.
In fact, boundary conditions pose two major challenges for neural operators in terms of inhomoge- neous boundary values and complex boundary geometry. First, inhomogeneous boundary conditions severe fluctuations in the solution, and have distinctive influence the solution com- pared to the interior source terms. For example, as shown in Fig. T the inhomogeneous boundary
"qqual contribution. Work done intern Westlake University. 'Corresponding author.

---

Published as a conference paper at ICLR 2024
(a) Forcing term
(b) Solution term
Figure 1: Examples of different geometries for the elliptic PDEs: (a) forcing terms and (b) solutions. The nodes in red-orange color-map represent the complex, inhomogeneous boundary values. The redder the area, the higher the boundary value it represents, whereas the orange the area, the lower the boundary value.
values cause high-frequency fluctuations in the solution especially near the boundary, which make extremely hard to learn. Second, since elliptic PDEs are boundary value problems whose solution describes the steady-state of the system, any variation the boundary geometry and values would influence the interior solution globally (Hirsch) 2007). The above challenges need to be properly addressed to develop a neural operator suitable for more general and realistic settings.
In this paper, we propose Coundary-Embedddee Neural Operators (BENO), novel neural operator architecture to address the above two key challenges. Inspired by classical Green's 's function, BENO consists of Graph Neural Networks (GNNs) that model the boundary influence and the interior source terms, respectively, addressing the first challenge. Moreover, to model the global influence of the boundary to the solution, we employ a Transformer (Vaswani al. (017) to encode the full boundary information to latent vector and feed it to each message passing layer of the GNNs. This captures how the global geometry and values of the boundary influence the pairwise interaction between interior points, addressing the second challenge. As whole, BENO provides a simple architecture for solving elliptic PDEs with complex boundary conditions, incorporating physics intuition into its boundary- boundary-embedded architecture. In Table BENO and prior deep learning methods for elliptic PDE solving.
[
we provide comparison between
Table Comparison of data-driven methods to time- iine-independent elliptic PDE solving.
1. PDE PDE agnostic prediction on ne- w initial condition
Evaluation at unobserved sp- atial locations
4. Free form spatial domain for boundary shape
Inhomageneous
Methods
2. Train/Test space grid indopendence
boundary condition
value
GKN
el. 2020h
FNO
ct
2020n
GNN-PDE (Laezsc et al 2022
MP-PDE Brandstetter et al
2022
BENO (ours)
To fully evaluate our model inhomogeneous boundary value problems, construct novel dataset encompassing various boundary shapes, different boundary values, different types of boundary conditions, and varying resolutions. The experimental results demonstrate that our approach not only outperforms the existing state-of-the-ar oftth--rrt methods by about an average of 60.96% in solving elliptic PDEs problems but also exhibits excellent generalization capabilities in other scenarios. In contrast, all existing baselines fail to learn solution operators for the above challenging elliptic PDEs.
2 PROBLEM SETUP
In this work, we consider the solution of elliptic PDEs in a compact domain subject to inhomogeneous boundary conditions along the domain boundary. Let E (²() be e-dimnesion-ddiffrentin function of interior grid nodes over an open domain Ω. Specifically, we consider the Poisson equation with Dirichlet (and Neumann in Appendix K) boundary conditions in a d-dimensional domain, and we consider = 2 in the following experiments:
V u ([x1, 22, = f ([21, Ø2, xd))
A ([x1, x.,l.,, E Ω ([zr, ma)) E 00,
(1)
([x1, .2,...,, ra)) g ([21, 22,...,, x.))),,
2

---

Published as a conference paper at ICLR 2024
where f and 9 are sufficiently smooth function defined on the domain Q2 { 22, ... Fd,i) Ji= and boundary 00, respectively. Eq. iss utilized in a range of applications in science and engineering to describe the equilibrium state, given by f in the presence of indeendneeenddnt boundary constraints specified by g- A distinctive feature of elliptic PDEs is their sensitivity to boundary values and shape o22 which heavily influence the behavior of their solutions. Appropriate boundary conditions must often be carefully prescribed to ensure well-posedness of elliptic boundary value problems.
3
METHOD
In this section, we detail our method BENO. We first motivate our method using Green's 's function, a classical approach to solving elliptic boundary value problems in Section 3.1 We then introduce our graph construction method in Section 3.2 Inspired by the Green's 's function, we introduce BENO's
architecture in Section 3.3
3.1 MOTIVATION
How to facilitate boundary-interior interaction? To design the boundary-embedded message passing neural network, we draw inspiration from the traditional Green's 's function (Stakgold & Holst 201T) method which is based on numerical solution. Take the Poisson equation with Dirichlet boundary conditions for example. Suppose the Green's function is G Q Ω R. which is the solution of the corresponding equation as follows:
{
V 8(x xo)6 - yo)
(2)
Glag = 0
Based the aforementioned equations and the detailed representation of the Green's function formula in the Appendix we can derive the solution in the following form:
uxx, v) = //, G(x, No, fo)f(zo Bol.ooodaa
@G y, zo, yo)
g( yo)
dla
(3)
Ono
Motivated by the two terms presented in Eq. our objective is to approach boundary embedding
by extending the Green's 's function. Following the mainstream work of utilizing GNNs as surrogate
models (Pfaff et al. 2020; Eliasof et al. 2021 Lötzsch et al.) 2022). we exploit the graph network simulator (Sanchez-Gonnzlez et al.| 2020) as backbone to mimic the Green's 's function, and add
the boundary embedding to the node update in the message passing. Besides, in order to decouple the learning of the boundary and interior, we adopt a dual-branch network structure, where one branch sets the boundary value g to to only learn the structural information of interior nodes, and the other branch sets the source term ff of interior nodes to 0 to only learn the structural information of the boundary. The Poisson equation solving then be disentangled into two parts:
(uu(z, y) = f(x, y)
V2u(E.y) =
V'ulx, y) = -(x, v) = g(x, v)
(w(x,))
uuxx,)) v) g(x,
Branch 1
Branch 2
Therefore, our BENO will use dual-branch design to build two different types of edges on the same graph separately. Branch considers the effects of interior nodes and Branch 2 focuses solely on how propagate the relationship between boundary values and interior nodes in the graph. Finally, aggregate them together to obtain a more accurate solution under complex boundary conditions.
How embed boundary? Since boundary conditions are crucially important for solving PDEs. how to better embed the boundary information into the neural network is key to our design. During a pilot study, we found that directly concatenating the interior node information with boundary information fails to solve for elliptic PDEs, and tends to cause severe over-fitting. Therefore, we propose to embed the boundary to represent its global information for further fusion. In recent years, Transformer (Vaswani et al.) 2017) has been widely adopted due to its global receptive field. By leveraging its attention mechanism, the Transformer can effectively capture long-range dependencies and interactions within the boundary nodes. This is particularly advantageous when dealing with complex boundary conditions (i.c., irregular shape and inhomogencous boundary values), as allows for the modeling of complex relationships between boundary points and the interior solution.
3

---

Published as a conference paper at ICLR 2024
3.2 GRAPH CONSTRUCTION
Figure 2: Visualization of the graph construction on our train/set samples from 5 different corner elliptic datasets. The interior nodes are black and the boundary one in purple.
Before designing our method, is an important step to construct graph 9 ((V,E) with the finite discrete interior nodes node set V on the PDE's solution domain Ω. In traditional solution methods such as FEM, the solution domain is initially constructed by triangulating the mesh graph (Bern & Eppstein 1995 Ho-Le 1988), followed by the subsequent solving process. Therefore, the first step is to implement Delaunay triangulation (Lee Schachter 1980) to construct mesh graph with edge set &mesks in which each cell consists of three edges. Then proceed to construct the edge set &kn by selecting the K -eneeesst nodes for each individual node. K the quantity of neighboring nodes that we deem as closely connected based the Buclidean distance Dij between node and j. The final edge set is 3 Emesh U Ekn- Examples of graph construction are shown in Fig. 2
3.3 OVERALL ARCHITECTURE
In this section, we will introduce the detailed architecture of our proposed BENO, as shown in Figure 3 Our overall neural operator is divided into two branches, with each branch receiving different graph information and boundary data. However, the operator architecture remains the same with the encoder,
message passing neural network and decoder. Therefore, will only focus on the common operator architecture.
3.3.1 ENCODER & DECODER
Encoder. The encoder computes node and edge embeddings. For each node i, the node encoder c" maps the node coordinates p; (Ia Vi). forcing term fi, and distances to boundary dæ; dy, to node embedding vector v e <"<;;, Vi: fi, dæ,, dy.)) RD in a high-dimensional space. The same mapping is implemented on edge attributes with edge encoder e for edge embedding vector eij. For both node and edge encoders e, we exploit two-layer Multi-Layer Perceptron (MLP) (Murtagh 1991) with Sigmoid Linear Unit (SiLU) activation (Elfwing et al.) 2018).
Decoder. We use a two-layer MLP to map the features to solutions. Considering our dual-branch architecture, we will add the outputs from each decoder to obtain the final predicted solution â.
3.3.2 BOUNDARY-EMBEDDEE MESSAGE PASSING NEURAL NETWORK (BE-MPNN)
To address the inherent differences in physical properties between boundary and interior nodes, we opt not to directly merge these distinct sources of information into a single network representation. Instead, we first employ the Transformer to specifically embed the boundary nodes. Then, the obtained boundary information is incorporated into the graph message passing processor. We will provide detailed explanations for these two components separately.
Embedding Boundary with Transformer. With the boundary node coordinates pd (x y the boundary value g. and the distance to the geometric center of solution domain de as input features, first utilize the position embedding to include relative position relationship for initial representation Transformer encoder with layers to embed the boundary information H The resulting boundary features, denoted as B, are obtained by applying global average pooling (Lin et al. 2013] to the encoder outputs H
H followed
Each self-attention layer applies multi-head self-attention and feed-forward neural networks to the input. The output of the i-th sell-attention layer denoted as 11,5 The self-attention mechanism calculates the attention weights A; as follows:
Q.H
A, = Softmax
(5)
Vdl
4

---

Published as a conference paper at ICLR 2024
Input 862
Enceder MLPs
RE-MPNN
Output
Decoder MLPS
jas
Boundary
Trunsformer
M/a
Embooking
Decodir ML.ss
P m
Enceder MLPs
BE-MPNN
Figure 3: Overall architecture of our proposed BENO. The pink branch corresponds to the first term in Eq. [ and the green branch corresponds to the second term. As the backbone of boundary embedding, Transformer provides boundary information as supplement for BE-MPNN, thereby enabling better prediction under complex boundary geometry and inhomogeneous boundary values.
where Qi K2, and Vi are linear projections of H with learnable weight matrices, and dą is the dimension of the key vectors. The attention output computed as:
H LayerNorm (A.V: (H1 + H
(6)
where LayerNorm denotes layer normalization, which helps to mitigate the problem of internal covariate shift. After passing through the self-attention layers, the output H is subject to global average pooling to obtain the boundary features: B AvgPool(119).
Message Passing Processor. The processor computes T steps of message passing, with intermediate graph representation , G and boundary representation B 8 The specific passing message ml in step / in our processor is formed by:
Boundary-Embedde
m MLPs of, vfn ey. -P))
where m represents the message sent from node j to i. p, Py is the relative position which can enhance the equivariance by justifying the symmetry of the PDEs.
Then we update the node feature of and edge feature ei as follows:
1 MLPs
Σ ml
(8)
JEN(()
(+1
MLPs (ely mly)
Here, boundary information is embedded into the message passing. (ii) represents the gathering of all the neighbors of node i.
Learning objective. Given the ground truth solution u and the predicted solution â, we minimize the mean squared error (MSE) of the predicted solution Ω.
EXPERIMENTS
We aim to answer the following questions: Compared with existing baselines, can BENO learn the solution operator for elliptic PDEs with complex geometry and inhomogeneous boundary values? (2) Can BENO generalize to aut-of-distribution boundary geometries and boundary values, and different grid resolutions? (3) Are all components of BENO essential for its performance? We first introduce experiment setup Sec. then answer the above three questions the following three sections.
4.1 EXPERIMENT SETUP
Datasets. For elliptic PDEs simulations, construct five different datasets with inhomogencous boundary values, including 4/3/2/1-comer squares and squares without corners. Each dataset consists of 1000 samples with randomly initialized boundary shapes and values, with 900 samples used for
5

---

Published as a conference paper at ICLR 2024
Table 2: Performances of proposed BENO and the compared baselines, which trained on 900 4-corners samples and tested on 5 datasets under relative L2 norm and MAE separately. The unit of the MAE metric is 1 10-3 Bold fonts indicate the best performance.
Tran on 4- Corners dalasel
Test sot
4-Comers
3-Comers
2-Corners
1-Comer
No-Comer
Metric
1.2
MAE
1.2
MAE
1.2
MAE
1.2
MAE
L22
MAE
1.1166:t
3.64974
10692:
3.70594
10673:
3.68224
1.103-4:
3,898:1
10728:
3,9551::
GKN
0.3936
1,1874
0,2034
0.9543
0.1393
0.9819
0.1905
0.9489
0,2074
0.9791
L0947:
2.27074
10742:
2.16574
10672:
2.26174
10921
2.39224
10762::
2.22814
FNO
0.3265
0.3361
0.3418
0.3976
0.3736
0.2449
0.2935
0.3526
0.4420
0.4192
1.0066:
3,1410
1.009:
3.28124
1.0015#
3.35574
1.0012+
3,1421#
1.001
3.7561
GNN-PDE
0,0093
0.8751
0.0101
0.8839
0.0099
0.8521
0,0153
0.8685
0.0152
1,0274
1.007
3. 3.1018
1.000
3.2464
0.9919
3.27654
0.9829
3.01634
0.9882-2
3 3.6522
MP-PDE
0.0677
0.8431
0.0841
0.8049
0.0699
0.8632
0,07199
0.8272
0.0683
0.8961
0.3523
0.9650 +
0.4308
1.2206:1
0.4910
1.388
0.5416:1:
1.4529
0.5542.1
1.7481
BENO (ours)
0.1245
0.3131
0.1994
0.4978
0.1888
0.5227
0.2133
0.4626
0.1952
0.5394
training and validation, and 100 samples for testing. Each sample covers a grid of x2x nodes and 128 boundary nodes. To further assess model performance, righetoresolution versions of each data sample, such as 64x64, are also provided. Details on data generation are provided Appendix C
Baselines. We adopt two of the most mainstream series of neural PDE solvers baselines, one is graph-based, including GKN (Li et al.) (020b). GNN-PDE (öözzcch et al.) (022), and MP- PDE (Brandstetter et al., )022); the other operator-based, including FNO (Li al. (2220a). For fair comparison and adaption to irregular boundary shapes in our datasets, all of the baselines are re-implemented with the same input as ours, including all the interior and boundary node features. Please refer to Appendix F for re-implementation details.
Implementation Details. All experiment are based on PyTorch (Paszke al, 2019) and PyTorch- Geometric (Fey & Lenssen 2019) on 2 NVIDIA A 100 GPUs (80G). Following (Brandstetter et al.) 2022), we also apply graph message passing neural network as our backbone for all the datasets. use Adam (Kingma & Ba 2014) optimizer with weight decay of 5 x 10 and a learning rate of x 10 obtained from grid search for all experiments. The relative L2 error measures the difference between the predicted and the ground truth values, normalized by the magnitude of the ground truth. MAE measures the average absolute difference between the predicted values and the ground truth values. Please refer Appendix D for more implementation details.
4.2
MAIN EXPERIMENTAL RESULTS
We first test whether our BENO has a strong capability to solve elliptic PDEs with varying shapes. Table 2 and summarize the results for the shape generalization task (more in Appendix H.
From the results, we see that recent neural PDE solving methods (i.c., MP-PDE) overall fail to solve elliptic PDEs with inhomogeneous boundary values, not to mention generalizing to datasets with different boundary shapes. This precisely indicates that existing neural solvers are insufficient for solving this type of boundary value problems.
In contrast, from Table 2 we see that our proposed BENO trained only on 4-Comers dataset consistently achieves significant improvement and strong generalization capability over the previous methods by large margin. More precisely, the improvements of BENO over the best baseline are 55. 17%, 52. 18%, 52.43%, 47.38%, and 52.94% terms of relative I.2 norm when testing on 10221//N-Cormee dataset respectively. We attribute the remarkable performance to two factors: (i) BENO comprehensively leverages boundary information, and fuses them with the interior graph (ii) BENO integrates dual-branch architecture to fully learn boundary and interior in a decoupled way and thus improves generalized solving performance.
message for solving.
Similarly, from Table 3 see that among mixed corner training results, BENO always achieves the best performance among various compared baselines when varying the test sets, which validates the consistent superiority of BENO with respect to different boundary shapes.

---

Published as a conference paper at ICLR 2024
Table 3: Performances of proposed BENO and the compared baselines, which trained 900 mixed samples (180 samples each from datasets) and tested on 5 datasets under relative L2 error and MAE separately. The unit of the MAE metric is 1 10
Train on mixed dalasel
Test set
4-Coreess
3-Comers
2-Corners
1-Comer
No-Comer
Metric
1.2
MAE
L2
MAE
1.2
MAE
L2
MAE
L.2
MAE
L0588:
3.5051
10651
3,70614
1.386:t
3.6033
1.03444
3.40841:
L0423:
3.901
GKN
0.1713
0.9401
0.1562
0,8563
0.1271
0.9392
0.1621
0.5199
02102
0.9287
L0834:
4.6401
109374
4.60924
10672:
4.5267=
10735
4.50274
10713:
FNO
0,0622
0.5327
0.0625
0.6713
0.0376
0.5581
0.0528
0.5371
0.0489
0.5565
L000:+
3.1311
1.0033
3.27814
1.0055::
3,3518=
0.9999
3,14224
1.0002:
3.7528
GNN-PDE
0.0036
0.8664
0.0039
0.8858
0,0038
0.8520
0.0042
0.8609
0.0041
1.0284
L00634
3.1238
1.00454
3.2537
0.99574:
3,2864
0,9822
3.0177
0.9912::
3.6658
MP-PDE
0.8502
0.0923
0,7867
0,0772
0.8607
0.0802
0.8363
0.0781
0.8949
0.4487.1
1.2150
0.478311
1.3509
0.4737
1.3516
0.516811
1.3728-1
0.4668
1.4213
BENO (ours)
0.1750
0,4213
0.1938
0.5432
0.1979
0.5374
0.1793
0.5148
0.2001
0.5262
Table 4: Performances of our BENO and the compared baselines, which are trained on 900 4-Corners samples and tested with zero boundary value samples. The unit of the MAE metric is 10-3
homogeneous houndary
Test set
Curners
3 3-Corees
Curners
1-Corner
No Corner
Metric
1.2
MAE
1.2
MAE
1.2
MAE
MAF
1.2
MAE
0.7092.1
0.1259
0.7390.1
0,23511
0.7491
0.3290
0.7593.:
0.4750.
0.7801
0.6808
GNN-PDE
0.0584
0.0755
0,0483
0.1013
0,085
0.1371
0.05269
0,1582
0.0371
0.1692
0.2598
0,0459=
0.3148
0.1066:t
0.3729
0.1778=
0.46344
0.3049+
0,5458#
0.4924
MP-PDE
0.1098
0.0359
0.0814
0.0618
0,0819
0.0969
0.0649
0.1182
0.0491
0.1310
0.0908=
0.0142=
0,1031
0.0288-
0.1652
0.0583= 0.0362
0.1783+
0.0862
0.2441
0.1622
BENO (ours)
0.07381
0.0131
0.0728
0.0189
0.1324
0.1508
0.0456
0.1665
0.0798
Additionally, we plot the visualization of the best baseline and our proposed BENO trained on 4- Corrners dataset in Figure 4 It can be clearly observed that the predicted solution of BENO is closed to the ground truth, while MP-PDE fails learn any features of the solution. We observe similar behaviors for all other baselines.
Output Predictions
Prediction Error
MP-PDE
Ours
Ground Truth
MP-DE
Ours
Figure 4: Visualization of two samples' prediction and prediction error from 4-Corners dataset. We render the solution u of the baseline MP-PDE, our BENO and the ground truth in Ω.
4.3
GENERALIZATION STUDY
4.3.1 RESULTS ON DIFFERENT BOUNDARY VALUES
To investigate the generalization ability on boundary value, we again train the models on 4-Corners dataset with inhomogeneous boundary value but utilize the test set with zero boundary value, which makes the boundary inhomogeneities totally different. Table 4 compares the best baseline and summarizes the results. From the results, reducing the L2 norm to around 0.1. In addition, our method outperforms the best baseline by approximately 60.96% in terms of performance improvement. This not only demonstrates BENO's
that BENO has significant advantage, successfully

---

Published as a conference paper at ICLR 2024
Table 5: Performances of our BENO and the compared baselines, which are trained on 900 4-Corness
32 x 32 samples and tested with 64 x 64 samples. The unit of the MAE metric is 1 x 10
Train 4-Comers datasct with 32 x 32 resolution
Test
4-Coreers(44xx6 64)
3.Comers(64x6))
2.Comeess(44x66
1-Comen(64x64
Nn-Comer(54 x 64)
Metric
MAE
MAE
1.2
MAE
MP-PDE
0.63534:
0.0596
0,74574
0.1138-
0,7926::
0.1565
0.8336
0,2445
0.8749::
0.3991 =
0.1009
0.0418
0.0738
0.0533
0.0505
0.0596
0.04467
0.0915
0.0298
0.1045
0.4596
0.0440
0,5483-1 0.0987
0.0860:1
0.6020 0.0842
0.1214 0.0537
0.66841
0.1995:1
0.7497:
0,3424
BENO (ours)
0.0349
0.0794
0.0851
0.0653
0.1000
Table 6: Ablation study of our BENO. The unit of the MAE metric is 10-3
Test set
4-Comens
3-Comers
2-Comers
I-Comer
No-Coree
Metric
L2
MAE
L2
MAE
L2
MAE
L2
MAE
L2
MAE
100304
3.1436::
10119::
3.3041
0.9999::
3.3007
100266
3,08424
0.9979:
3.68324
BENO M
0,0858
0.8667
0.0975
0.7906
0.0792
0.8504
0.0840
0,8202
0.0858
0.8970
0.4058
1.175:
0.4850::
1.3810
0.52731
1.5499
0.5795
1.56834
0.5835#
183824
BENO w/o. D
0,1374
0,3660
0,2230
0.6068
0.1750
0,4774
0.1981
0,4670
0.2232
0,5771
0.41134:
1.2020
0.462411
133569
0.5347
I. 1,5990
0.5891
1622244
0.584311
1.87901
BENO
0.1236
0.4048
0,2102
0,5453
0.1985
0.5004
0.2129
0,2016
0.2016
0,5952
0,903744
2.6795
0,8807
2.6992.1
0.8928 ::
2.8235
0.8849
2:561
0.8721 +
2.9851
BENO G
0.1104
0,5332
0.1298
0.6118
0.1208
0,5892
1462
0,5085
0.1569
0.5591
0.3823-1
0.9650.1
0.4308.1
1.2206
0.4910:1
1,438811 0.5227
0.5416 0,2133
1.4529:1 0.4626
0.5542.1 0.1952
1.74811
BENO (ours)
0.1245
0.3131
0.1994
0.4978
0.1888
0.5394
strong generalization ability regarding boundary values but also provides solid experimental evidence for the successful application of our elliptic PDE solver.
4.3.2 DIFFERENT GRID RESOLUTIOONS
Data-driven PDE solvers often face limitations in terms of the scale of the training data, making the ability to generalize to higher resolutions a crucial metric. Table provides a summary of our performance resolution generalization experiments. The model was trained on the 4-Corners homogeneous boundary value dataset with 32 x 32 resolution and tested with 64 x 64 samples not seen in training. The results demonstrate a significant advantage of our method over MP-PDE, with an improvement of approximately 22.46%. We attribute this advantage in generalization to two main factors. Firstly, it stems from the inherent capability of GNNs to process input graphs of various sizes. Secondly, it is due our incorporation of relative positions part of the network's 's edge features. Consequently, our approach can be deployed on different resolutions using the same setup.
4.4 ABLATION STUDY
To investigate the effectiveness of inner components in BENO, we study four variants of BENO. Table[]]showss the effectiveness of our BENO on ablation experiments, which is implemented based on 4-Corners dataset training. Firstly, BENO w. M replaces the BE-MPNN with vanilla message passing neural network (Gilmer et al. (017)) and merely keeps the interior node feature. Secondly, BENO w/o. D removes the dual-branch structure of BENO and merely utilizes single Encoder- W.MMPNN-Decodee procedure. Thirdly, BENO w. E adds the Transformer output for edge message passing. Finally, BENO w. G replaces the Transformer architecture with vanilla graph convolution network (Kipf & Welling 2016).
From the results we can draw conclusions as follows. Firstly, BENO w. M performs significantly worse than ours, which indicates the importance of fusing interior and boundary in BENO. Secondly, comparing the results of BENO w/o. D with ours we can conclude that decoupled learning of the interior and boundary proves to be effective. Thirdly, comparing the results of BENO w. E and ours, we can find that boundary information only helps in nove-level message passing. In other words, is not particularly suitable to directly inject the global information of the boundary into the edges. Finally, comparing results of BENO w. G with ours validates the design of Transformer for boundary embedding is crucial.
8

---

Published as a conference paper at ICLR 2024
5
RELATED WORK
5.1 CLASSIC ELLIPTIC PDE SOLVERS
The classical numerical solution of elliptic PDEs approximates the domain 52 and its boundary an in Eq. Ussng finite number of non-overlapping partitions. The solution to Eq. is then approximated over these partitions. A variety of strategies are available for computing this discrete solution. Popular approaches include finite volume method (FVM) (Hirsch) 2007). finite element method (FEM) (ughes 2012), and finite difference method (FDM) (LeVeque (007). In the present work we utilize the FVM to generate the dataset which can easily accommodate complex boundary shapes. This approach partitions the domains into cells, and the boundary is specified using cell interfaces. After numerically approximating the operator V2 over these cells, the numerical solution is obtained on the centers of the cells constituting our domain. Further details are provided in Appendix
5.2 GNN FOR PDE SOLVER
GNNs are initially applied in physics-based simulations on solids and fluids represented by parti-
cles Grabblz-Gonzzezzzccco et al. 2018). Recently, important advancement MeshGraphNets (Pfaff ct al. 2020) emerge to learn mesh-based simulations. Subsequently, several variations have been proposed, including techniques for accelerating finer-level simulations by utilizing GNNs (Belbute
Peres et al. 2020 Yang & Hong. 2022). combining GNNs with Physics-Infore Informed Neural Networks (PINNs) (Gao al.] 2022), solving inverse problems with GNNs and autodecoder-style priors (Zhao et al. 2022), and handling temporal distribution shift (Luo et al. (003). However, the research focus on addressing boundary issues is limited. T-FEN (Lienen& Günnemann 2022), FEONet (Lee et al.) 2002).. VQGraph Yang et al. (2024) and GNN-PDE (Lötzsch et al. (022) are pioncering efforts in this regard, encompassing complex domains and various boundary shapes. Nevertheless, the boundary values are still set to zero, which does not account for the presence of inhomogeneous boundary values. This discrepancy is precisely the problem that we aim address.
5.3 NEURAL OPERATOR AS PDE SOLVER
Neural operators map from initial/boundary conditions solutions through supervised learning in mesh-invariant manner. Prominent examples of neural operators include the Fourier neural operator (FNO) (Li et al. 2020a). graph neural operator (Li et al. 2020b), and DeepONet(Lu al. 2019).
Neural operators exhibit invariance to discretization, making them highly suitable for solving PDEs.
Moreover, neural operators enable the learning of operator mappings between
function spaces. Subsequently, further variations have been proposed, including techniques for solving arbitrary geometries PDEs with both the computation efficiency and the flexibility (Li et al. 2022). enabling deeper stacks of Fourier layers by independently applying transformations (Tran et al. 2021
utilizing Fourier layers as replacement for spatial self-attention (Guibas et al.) 2021). ), facilitating boundary condition satisfaction in neural operators by implementing structural modifications to the operator kernel (Saad et al. 2022) and incorporating symmetries in the physical domain using group
theory (Helwig et al. 2023) (Gupta et al.) 2021 2022 Xiao et al. 2023) continuously improve the design of the operator by introducing novel methods for numerical computation.
6 CONCLUSION
In this work, we have proposed Boundary-Enbeedded Neural Operators (BENO), a neural operator architecture to address the challenges posed by inhomogeneous boundary conditions with complex boundary geometry in solving elliptic PDEs. Our approach BENO incorporates physics intuition through boundary-embeddee architecture, consisting of GNNs and Transformer, to model the influence of boundary conditions on the solution. By constructing a diverse dataset with various boundary shapes, values, and resolutions, we have demonstrated the effectiveness of our approach in outperforming existing state-of-the-a methods by average of 60.96% in solving elliptic PDE problems. Furthermore, our method BENO exhibits strong generalization capabilities across different scenarios. The development of BENO opens up new possibilities for efficiently and accurately solving elliptic PDEs with complex boundary conditions, making them more useful to various scientific and engineering fields.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENT
We gratefully acknowledge the support of Westlake University Research Center for Industries of the Future, and Westlake University Center for High-performance Computing.
REFERENCES
Filipe De Avila Belbute-Peres, Thomas Economon, and Zico Kolter. Combining differentiable pde solvers and graph neural networks for fluid flow prediction. In International Conference Machine Learning, pp. 2402-2411, PMLR, 2020.
Marshall Bern and David Eppstein. Mesh generation and optimal triangulation. In Computing in Euclidean geometry, pp. 47-123. World Scientific, 1995.
Johannes Brandstetter, Daniel Worrall, and Max Welling, Message passing neural pde solvers. arXiv preprint arXiv:2202.03376, 2022.
Francis F Chen. Introduction to Plasma Physics and Controlled Fusion (3rd Ed.). Springer, 2016.
Ivan Dimov, István Faragó, and Lubin Vulkov. Finite difference methods, theory and applications. Springer, 2015,
Stefan Elfwing, Eiji Uchibe, and Kenji Doya. Wigmoid-weighted linear units for neural network function approximation in reinforcement learning. Neural networks, 107:3-11, 2018.
Moshe Eliasof, Eldad Haber, and Eran Treister. Pde-gen: Novel architectures for graph neural networks motivated by partial diffcrential equations. Advances in neural information processing systems, 34:3836-3849, 2021.
Matthias Fey and Jan Eric Lenssen. Fast Graph Representation Learning with PyTorch Geometric, 2019.
Han Gao, Matthew J Zahr, and Jian-Xun Wang. Physics-informed graph neural galerkin networks: unified framework for solving pde-governed forward and inverse problems. Computer Methods in Applied Mechanics and Engineering, 390:1 14502, 2022.
Justin Gilmer, Samuel S Schoenholz, Patrick Riley, Oriol Vinyals, and George E. Dahl. Neural message passing for quantum chemistry. In International conference on machine learning, pp. 1263-1272. PMLR, 2017.
John Guibas, Morteza Mardani, Zongyi Li, Andrew Tao, Anima Anandkumar, and Bryan Catan- zaro. Adaptive fourier neural operators: Efficient token mixers for transformers. arXiv preprint arXiv:2111.13587, 2021.
Gaurav Gupta, Xiongye Xiao, and Paul Bogdan. Multiwavelet-based operator learning for differential equations. Advances in neural information processing systems, 34:24048-2400 2021.
Gaurav Gupta, Xiongye Xiao, Radu Balan, and Paul Bogdan. Non-linear operator approximations for initial value problems. In International Conference Learning Representations, 2022.
Jacob Helwig, Xuan Zhang, Cong Fu, Jerry Kurtin, Stephan Wojtowytsch, and Shuiwang Group equivariant fourier neural operators for partial differential equations. arXiv preprint arXiv:2306.05699 056977 2023.
C. Hirsch. Numerical computation of internal and external flows: The fundamentals of computational fluid dynamics. Elsevier, 2007.
K Ho-Le. Finite element mesh generation methods: a review and classification. Computer-aided design, 20( 1):27-38, 1988.
T.J. R. Hughes. The finite element method: linear static and dynamic finite element analysis. Courier Corporation, 2012.
10

---

Published as a conference paper at ICLR 2024
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6990, 2014.
Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv: 00..6669999990 2016.
Der-Tsai Lee and Bruce Schachter. Two algorithms for constructing a delaunay triangulation.
International Journal Computer Information Sciences, 93):219-242, 1980.
Jae Yong Lee, Seungchan Ko, and Youngjoon Hong. Finite element operator network for solving parametric pdes. arXiv preprint S...v::2308.06990 2023.
R. J. LeVeque. Finite difference methods for ordinary and partial differential equations: steady-state and depedeenndent problems. SIAM, 2007.
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. arXiv preprint arXiv:2010.068: 00100089,, 2020a.
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Neural operator: Graph kernel network for partial differential equations. arXiv preprint arXiv:2003.03488 2020b.
Zongyi Li, Nikola Kovachki, Kamyar Avizzadenesheli, Burigede Liu, Andrew Stuart, Kaushik Bhattacharya, and Anima Anandkumar. Multipole graph neural operator for parametric partial differential equations. Advances in Neural Information Processing Systems, 33:6755-677 -766, 2020c.
Zongyi Li, Daniel Zhengyu Huang, Burigede Liu, and Anima Anandkumar. Fourier neural operator with learned deformations for pdes on general geometries. arXiv preprint arXiv:2207.05209 2022.
Marten Lienen and Stephan Günneeannn Learning the dynamics of physical systems from sparse observations with finite element networks. arXiv preprint arXiv:2203.08852 2022.
Min Lin, Qiang Chen, and Shuicheng Yan. Network in network. arXiv preprint arXiv: 1312.4400, 2013.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint htth::60080398833 2016.
Winfried Lötzsch, Simon Ohler, and Johannes S Otterbach. Learning the solution operator of boundary value problems using graph neural networks. arXiv preprint arXiv:2206.14000 2022.
Lu Lu, Pengzhan Jin, and George Em Karniadakis. Deeponet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators. arXiv preprint arXiv:1910.00 199,,3399, 2019.
Xiao Luo, Haixin Wang. Zijie Huang. Huiyu Jiang, Abhijeet Sadashiv Gangan, Song Jiang, and Yizhou Sun. Care: Modeling interacting dynamics under temporal environmental variation. In Thirth-seventh Conference on Neural Information Processing Systems, 2023.
Fionn Murtagh. Multilayer perceptrons for classification and regression. Neurocomputing, 2(5-6): 183-197, 1991.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vitoto Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Py PyTorch: An Imperative Style, High-Peeformance Deep Learning Library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' Alché Buc, F. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 8024-8035. Curran Associates, Inc., 2019.
SGOPAL Patro and Kishore Kumar Sahu. Normalization: preprocessing stage. arXiv preprint 103.042503.06462 2015.
11

---

Published as a conference paper at ICLR 2024
Tobias Pfaff, Meire Fortunato, Alvaro Manchez-Gonzaler and Peter W Battaglia. Learning mesh- based simulation with graph networks. arXiv preprint arXiv:2010.03409, 2020.
Alfio Quarteroni and Alberto Valli. Numerical approximation of partial differential equations, volume 23. Springer Science & Business Media, 2008.
Béatrice Rivière. Discontinuous Galerkin Methods for Solving Elliptic and Parabolic Equations. Society for Industrial and Applied Mathematics, 2008. doi:
Nadim Saad, Gaurav Gupta, Shima Alizadeh, and Danielle C Maddix. Guiding continuous operator learning through physics-basee boundary constraints. arXiv preprint a7Xiv:2212.07477,20000 2022.
Y. Saad. Iterative methods for sparse linear systems. SIAM, 2003.
Alvaro Sanchez-Gonzalez, Nicolas Heess, Jost Tobias Springenberg, Josh Merel, Martin Riedmiller, Raia Hadsell, and Peter Battaglia. Graph networks as learnable physics engines for inference and control. In International Conference on Machine Learning, pp. 4470 4479. PMLR, 2018.
Alvaro Sanchez-Gonzalez, Jonathan Godwin, Tobias Pfaff, Rex Ying, Jure Leskovee, and Peter Battaglia. Learning to simulate complex physics with graph networks. In International conference machine learning, pp. $459-8468. PMLR, 2020.
Ivar Stakgold and Michael J Holst. Green's 's functions and boundary value problems. John Wiley & Sons, 2011.
Alasdair Tran, Alexander Mathews, Lexing Xie, and Cheng Soon Ong. Factorized fourier neural operators. arXiv preprint arXiv:2111.1380) 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Xiongye Xiao, Defu Cao, Ruochen Yang, Gaurav Gupta, Gengshuo Liu, Chenzhong Yin, Radu Balan, and Paul Bogdan. Coupled multiwavelet neural operator learning for coupled partial differential equations. arXiv preprint arXiv:2303.02304 2023.
Ling Yang and Shenda Hong. Omni-granular ego-semantic propagation for self-supervised graph representation learning. In International Conference on Machine Learning, pp. 25022-25037. PMLR, 2022.
Ling Yang. Ye Tian, Minkai Xu, Zhongyi Liu, Shenda Hong, Wei Qu Wentao Zhang, Bin Cui, Muhan Zhang, and Jure Leskovec. Vggraph: Rethinking graph representation space for bridging gnns and mlps. In International Conference Learning Representations, 2024.
Qingqing Zhao, David B Lindell, and Gordon Wetzstein. Learning solve pde-constrained inverse problems with graph networks. arXiv preprint arXiv:2206.00711, 2022.
12

---

Published as a conference paper at ICLR 2024
APPENDIX
A DERIVATION OF THE GREEN'S S FUNCTION METHOD.
We first review the definition of the Green's function, which is G 02 x Ω - IR, which is the solution of the corresponding equation as follows:
{
V 8(x xo)6(y - yo)
(10)
Glan =
According to Green's identities,
SL. (uV = J
aG Ōn
/| (Vu. VG)da
dl
(11)
Since u and G are arbitrary, we can change the position to obtain that,
J1g (GV
= S
Gou
G ou
/1 (Vu. VG)da
dl
(12)
On
R
on
Subtract Eq. 12 from Eq.
11 we have,
( GV
8G On
du in
dl
(13)
Jon
Substitute Eq. 13
into Eq.
10
we can have that,
S
DG
G du
J/ (u 1..vGG-C G. v2 v²u)do
u
dl
In
On
(-uô(u- xn)ō(y vo) u)d
(14)
= -u(#o, 3/0) - 11 GV2udo
=u vo) + // Cf(z,y)do
Namely, we have that,
J/, G( v, 1o 20. y0) do)
u(x, v)
(15)
au mo)
aG(r, y, 2o, 10)
G(x, y. yo)
u( yo)
dlp
Ono
Ong
When considering the Dirichlet boundary conditions, we can simplify the solution in the following form:
u(x u) = //. G(x,, ((C,,,,0 To, fo) (20.
/ an
oG((,, y, 20, vo) dla
g( vo)
(16)
Ino
B NUMERICAL SOLUTION OF THE ELLIPTIC PDE
The strong solution to equation I can be expressed in terms of the Green's function (see Section 3.J and Appendix A for discussion). However, obtaining a closed form expression using the Green's function is typically
possible, except for some limited canonical domain shapes. In the present paper, we obtain the solution to equation T in arbitrary two dimensional domains Ω2 using the finite volume method (Hirsch) 2007). This numerical approach relies discretizing the domain Ω using cells. The surfaces of these cells at the boundary, which are called cell interfaces, are used to specify the given boundary condition. The solution of equation i then numerically approximated over N (c.g., for 32x32 cells, N = 1024) computational cells by solving.
Pû f,
(17)
13

---

Published as a conference paper at ICLR 2024
where P RNXN is an N N matrix which denotes second-order discretization of the V operator incorporating the boundary conditions, ú R xx is a vector of values at the cell centers, and f E Rx is vector with values f(-.-) at cell centers used to discretize the domain Q2. The matrix P resulting from this approach is positive definite and diagonally dominant, making it convenient to solve Equation 17 with matrix-free iterative approach such as the Gauss-Seidel method (Saad)
2003
C DETAILS OF DATASETS
In this paper, we have established a comprehensive dataset for solving elliptic PDEs to facilitate various research endeavors. The elliptic PDEs solver performed as follows. (1) square domain is set with Ne number of cells in both r and directions (note N NNN3). The number of corners set, however, the size of the corners chosen randomly. (2) The source term f(z. y) is assigned assuming a variety of basis functions, including sinusoidal, exponential, logarithmic, and polynomial distributions. (3) The values of the boundary conditions g(x, y) are set using continuous periodic functions with uniformly distributed wavelength E [1, 5]. (4) The Gauss-Seide method (Saad 2003) is used to iteratively obtain the solution u(x, 3). Each Poisson run generates two files: one for the interior cells with discrete values of y, f. and and the other for the boundary interfaces with discrete values æ, y. and g. The simulations are performed on the Sherlock cluster at Stanford University.
D MORE IMPLEMENTATION DETAILS
Our normalization process is performed using the z-score method (Patro & Sahu 2015), where the mean and standard deviation are calculated from the training set. This ensures that all features are normalized based on the mean and variance of the training data. We also apply the WarmRestarts scheduler (Loshchilov & Hutter 2016) during the training. Each experiment is trained for 1000 epochs, and validation performed after each epoch. For the final evaluation, we select the model parameters from the epoch with the lowest validation loss. Consistency is maintained across all experiments by utilizing the same random seed.
All our experiments are evaluated on relative L2 error, abbreviated as L2, and mean absolute error (MAF), which are two commonly used metries for evaluating the performance of models or algorithms. The relative 1.2 error, also known as the normalized 1.2 error, measures the difference between the predicted values and the ground truth values, normalized by the magnitude of the ground truth values. It is typically calculated the L2 norm of the difference between the predicted and ground truth values, divided by the L2 norm the ground truth values. On the other hand, MAE measures the average absolute difference between the predicted values and the ground truth values. It is calculated by taking the mean the absolute differences between each predicted value and its corresponding ground truth value.
E DETAILS OF BASELINES
Our proposed BENO compared with range of competing baselines follows:
GKN (Li et al. 2020b) develops approximation method for mapping in infiniteddimensione spaces by combining non-linear activation functions with set of integral operators. The integration of kernels is achieved through message passing graph networks. For fair comparison, re-implement by adding the boundary nodes to the graph. To better distinguish between nodes belonging the interior and those belonging the boundary, we have also added an additional column of one-hot encoding to the nodes for differentiation.
FNO LL et al. 2020a) introduces a novel approach that directly learns the mapping from functional parametric dependencies to the solution. The method implements a series of layers computing global convolution operators with the fast Fourier transform (FFT) followed by mixing weights in the frequency domain and inverse Fourier transform, enabling an architecture that is both expressive and computationally efficient. For fair comparison, we re-implement by fixing the value of out-domain nodes with large number, and then implement the global operation.
14

---

Published as a conference paper at ICLR 2024
GNN-PDE (ötzzch et al. 2022) represents the pioneering effort in training neural networks on simulated data generated by a finite element solver, encompassing various boundary shapes. It evaluates the generalization capability of the trained operator across previously unobserved scenarios by designing a versatile solution operator using spectral graph convolutions. For fair comparison, we re-implement by adding the boundary nodes to the graph. To better distinguish between nodes belonging to the interior and those belonging to the boundary, we have also added an additional column of one-hot encoding to the nodes for differentiation.
MP-PDE (Brandstetter et al., 2022) presents a groundbreaking solver that utilizes neural message passing for all its components. This approach replaces traditionally bouristic-dssinne elements in the computation graph with neural function approximators that are optimized through backpropaga- tion. For fair comparison, we re-implement it by adding the boundary nodes to the graph. To better distinguish between nodes belonging to the interior and those belonging to the boundary, we have also added an additional column of one-hot encoding to the nodes for differentiation.
F DIFFERENCES WITH OTHER NEURAL OPERATORS
In this section, we compare our method, BENO, with existing approaches in terms of several key aspects according to Table [
PDE-agnostic prediction on new initial conditions: GKN, FNO, GNN-PDE, MP-PDE, and BENO are capable of predicting new initial conditions.
Train/Test space grid independence: GKN, GNN-PDE, and BENO exhibit independence between the training and testing spaces, while FNO and MP-PDE lack this independence.
Evaluation at unobserved spatial locations: GKN, FNO, and BENO capable of evaluating the PDE at locations that are not observed during training, while GNN-PDE and MP-PDE do not possess this capability.
Free-form spatial domain for boundary shape: Only GNN-PDE and BENO are capable of dealing with arbitrary boundary shapes, while GKN and MP-PDE are limited in this aspect.
Inhomogeneous boundary condition value: Only our method, BENO, has the ability to handle inhomogeneous boundary conditions, while GKN, FNO, GNN-PDE, and MP-PDE are unable to handle them.
In summary, compared to the existing methods, our method, BENO, possesses several distinct advantages. can predict new initial conditions regardless of the specific PDE, maintains grid independence between training and testing spaces, allows evaluation at unobserved spatial locations, handles free-form spatial domains for boundary shapes, and accommodates inhomogeneous boundary condition values. These capabilities make BENO versatile and powerful approach for solving iee-independeet elliptic PDEs.
G ALGORITHM
The whole learning algorithm of BENO is summarized in Algorithm |
H MORE EXPERIMENTAL RESULTS
H.1 SENSITIVITY ANALYSIS
In this section, discuss the process of determining the optimal values for the number of MLP layers (M) and the number of Transformer layers (L) using grid search, a systematic approach for hyper-parameteer tuning.
Grid search involves defining a parameter grid consisting of different combinations of M and values. We specified M in the range of [2, 3, 4] and L in the range of [1, 2. 3] to explore diverse set configurations. We build multiple models, each with a different combination of M and values, and train them 4-Corness training dataset. The models then evaluated using appropriate
15

---

Published as a conference paper at ICLR 2024
Algorithm Learning Algorithm of the proposed BENO
Input: The foreing term f. the inhomogeneous boundary condition g on 022
Output: The solution prediction à of the elliptic PDEs.
1: Construct the graph G {(V,E) E)} following Section 3.2
2: Initialize the parameters in our model;
# Training procedure
3: while not convergence do
4: for each training input do
5: Set the boundary value of one branch to zero following Eq.
16
6: Set the source term of interior in the other branch to zero;
7:
Feed the node/edge attributes to encoder following Section 3.3.
8:
Feed the boundary the Transformer for boundary features 8;;
9:
Add B to the message passing processor following Eq. [
Feed output features into a decoder get the predictions i;;
11: Calculate the loss using MSE;
Update the parameters BENO using back propagation; 13: end for
14:
end while
evaluation metrics on separate validation set. The evaluation results allowed us to compare the performance of models across different parameter combinations.
After evaluating the models, we select the combination M and that yield the best performance according our chosen evaluation metric. This combination became our final choice for M and L, representing the optimal configuration for our model. To ensure the reliability our chosen parameters, validate them independent validation set. This step confirmed that the model's performance remained consistent and reliable.
The grid search process provided systematic and effective approach to determine the optimal values for M = and I. = 1, allowing us to fine-tune our model and achieve improved performance.
H.2 MORE EXPERIMENTAL RESULTS
We have successfully validated our method's performance on the 4-Corners and mixed corners datasets during training and testing on other shape datasets, yielding favorable results. In this section, we will further supplement the evaluation by training on the No Corner dataset and testing on other shape datasets. Since the No Corner dataset does not include any corner scenarios, the remaining datasets present completely unseen scenarios for it, thereby providing a stronger test of the model's generalization performance.
Table summarizes the results of training on 900 No-Corner samples and tested on all datasets. We can infer similar conclusions those in the experimental section above. Our BENO performs well in learning on No-Corner cases, yielding more accurate solutions. Additionally, our method demonstrates stronger generalization ability, it can obtain good solutions even in cases where corners of any shape have not been encountered.
H.3 CONVERGENCE ANALYSIS
We draw the training curve the train L2 norm and test L2 norm of three models trained on the 4-Corners dataset with inhomogeneous boundary value in Figure 5 It is obviously that although two baselines also contains the boundary information, they fail to learn the elliptic PDEs with non- decreasing convergence curves. However, our proposed BENO is capable of successfully learning complex boundary conditions with the use the CosineAneealing Warttestarts scheduler, converges satisfactory result.
16

---

Published as a conference paper at ICLR 2024
Table 7: Performances of proposed BENO and the compared baselines, which trained on 900
No-Corner samples and tested on 5 datasets under relative L2 Norm and MAE separately. The unit of the MAE metric is 1 10-3
Train No- Corner dalaset
Test
4-Comers
3-Comers
2-Corners
1-Comer
No-Comer
Metric
1.2
MAE
1.1474
3.3790
1.1199
3.54191
10047.
3.53011
1.072.1
3.2295
1.028.1
3,6899
GKN
0.1128
0.8922
0.1212
0.8643
0.1166
0.9514
0.1098
0.8520
0.1060
0.8987
0.9714=
3.3210=
0,9745
3.3187
0.9733+
3.3319
0
3.3511
0
3,3427
FNO
0.0128
0.6546
0.0175
0.6639
0.0137
0.6298
0,0210
0.6109
0.0121
0.6981
0.9988
3.11824
0.9997
3.2748
0.9994
3,34754
1.002
3,14474
0.9998
3.7518:
GNN-PDE
0.0051
0,8543
0.0054
0.8902
0.0054
0.8533
0.0056
0,8559
0.0056
1,0314
1.0029
3.10054
1.0099
3.2488
0.9986+
3,2902:4
0.9855+
3.0356
0.9977
3.6648
MP-PDE
0.0808
0.8158
0.0891
0.7941
0.0822
0.8651
0,0769
0.8133
0,07670
0.8949
0.6870
18830
0.6036-1
1.7293
0,5760
166034
0.6192:1
L6749:
0,4093
1.2505
BENO (ours)
0.2038
0.6083
0.1940
0,5844
0.1998
0.6605
0.2259
0.5773
0.1873
0.5752
Train Convergence
Test Convergence
1.2
1.0
Relative L2 Norm
Relative L.2 Norm
1.0
0.8
GNN-PDE MP-PDE Ours
0.8
GNN-PDE MP-PDE Ours
0.6
0.6
0.4
0.4
0.2
200
400
600
800
200
400
600
800
Epoch
Epoch
Figure 5: Visualization of the convergence curve of our BENO and two baselines.
LIMITATIONS & BROADER IMPACTS
Limitations, Although this paper primarily focuses on Dirichlet boundary conditions, is essential acknowledge that there other types of boundary treatments, including Neumann and Robin boundary conditions. While the framework presented in this study may not directly address these alternative boundary conditions, it still retains its usefulness. Future research should explore the extension of the developed framework to incorporate these different boundary treatments, allowing for more comprehensive and versatile solution for broader range of practical problems.
Broader Impact. The development of a fast, efficient, and accurate neural network for solving PDEs holds significant potential for numerous physics and engineering disciplines. The impact of such advancements cannot be understated. By providing more streamlined and computationally efficient approach, this research can revolutionize fields such as computational fluid dynamics, solid mechanics, electromagnetics, and many others. The ability to solve PDEs more efficiently opens up new possibilities for modeling and simulating complex physical systems, leading to improved designs, optimizations, and decision-making processes. The resulting advancements can have far-reaching implications, including the development more efficient and sustainable technologies, enhanced understanding of natural phenomena, and improved safety and reliability in engineering applications. is crucial to continue exploring and refining these neural network-baid based approaches to maximize their potential impact across wide range of scientific and engineering disciplines.
17

---

Published as a conference paper at ICLR 2024
MP-PDE
Ours
Ground truth
MP-PDE
Ours
Figure 6: Visualization of prediction and prediction error 64 x 64 grid resolution.
MP-PDE
Ours
Ground truth
MP-PDE
Ours
4008
400
Figure 7: Visualization of prediction and prediction error on data with zero boundary value.
MORE VISUALIZATION ANALYSIS
In this section. we visualize the experimental results on a broader range of experiments. Figure presents the comparison of solution prediction on 64 64 grid resolution. Figure / prresents the comparison of solution prediction data with zero boundary value. T the qualitative results of training on the 4-Corners dataset and testing on data with various other shapes.
18

---

Published as a conference paper at ICLR 2024
Output Predictions
Prediction Error
MP-PDE
Ours
Ground Truth
MP-PDE
Ours
Figure 8: Visualization of prediction and prediction error from 3/2/1/No-Corner dataset, and each has two samples. We render the solution the baseline MP-PDE, our BENO and the ground truth in Ω.
19

---

Published as a conference paper at ICLR 2024
Table 8: Performances of our proposed BENO and the compared baselines under Neumann boundary condition, which are trained on 900 4-corners samples and tested on 5 datasets under relative L2 norm and MAE separately. The unit of the MAE metric is 10 Bold fonts indicate the best.
Tran on 4- Corners dalaset
Test sot
4-Comers
3-Comers
2-Corners
1-Comer
No-Comer
Metric
1.2
MAE
1.2
MAE
1.2
MAE
1.2
MAE
1.2
MAE
L0118:
3.71054
10046:
3.60134
103014:
3.3880
1.025:::
3.3675:
0.92774:
3,5691
GKN
0.1031
1.0699
0.1284
1,2937
0.1417
1.0127
0.1326
0.112
0.1270
1,2194
L0547:
4.03161
L0587
4.02194
105194
4.0308
L0533:t
4.0276:
1.549:
4.04174
FNO
0.1643
0.8953
0.1761
0.8210
0.1822
0.8369
0.1782
0.8554
0.1842
0.8063
10005
2.3685
0.9977
2.5474
1.0124
2.7348
0,9821
2.9824
0
3,4930
GNN-PDE
0.0898
0.6933
0,1054
0.8863
0.1208
0.8461
0.1225
0.8106
0.1071
1,1110
1.000
2. 2.3595
0.9895
2.540
1.1144
2.7345:
0.9782-
2.96794
0.9670
3.4807
MP-PDE
0.0813
0.6941
0,0973
0.8955
0.1120
0.8393
0.1240
0,7958
0.1164
1.1143
0.3568
0.8311
0.4201
1081441
0.5020
1.3918:1
0.5074:1
1.5676:1
0.5221
186491
BENO (ours)
0.0988
0.2864
0.1170
0.3938
0.1648
0.5454
0.1422
0,4815
0.1474
0.5472
Table 9: Performances of our proposed BENO and the compared baselines under Neumann boundary condition, which are trained 900 mixed samples (180 samples each from 5 datasets) and tested on 5 datasets under relative 1.2 norm and MAE separately. The unit of the MAE metric is x 10-3 Bold fonts indicate the best.
Train on mixed datasets
Test
4-Comers
3-Comers
2-Comers
1-Comer
No-Comer
Metric
L2
MAB
L2
MAE
L22
MAE
L2
MAE
L2
MAE
L0578:
3. 3,89924
L0399
3.6554
10995
3.5980+
L.649
3.6030+
10373:
3.7000+
GKN
0.1859
1.2048
0,2116
1,2122
0.1912
0.9631
0.3096
0.9310
0,2210
1.0831
1.046611
3.581711
1.008
3.618711
1.048
3.6338
1.0592.1
3.653111
1.575
3.6498.1 1
FNO
0.0917
0.8212
0.0925
0.8338
0.1239
0.8042
0.1065
0.8448
0.1019
0.8239
2.3648.1
1.00001
2.64041
0.999911
2.7425.1
1.00001
1.1488
1.00011
3.7167.1
GNN-PDE
0.0008
0.7703
0.0010
0.9250
0.0010
0.9808
0.0010
0.9672
0.0010
1.3370
MP-PDE
L0245.1
2.397311
0.9989.1
2.5510.1.
1.027711
2.7722
0.9940.1
2.9998
0.9731
3,4930.1
0.1448
0.7015
0.1277
0.8717
0.1399
0.8091
0.1543
0.7781
0.1414
1.0867
0.4237 1 0.1237
10114
0.3970
L0378 1
0.3931 11
1O8811 11
0.3387 1
L0550 1:
0.3344
1.2261 1:
BENO (ours)
0.1279
K
EXPERIMENTS ON NEUMANN BOUNDARY
In this section, we consider to solve the Poisson equation with Neumann boundary conditions using our proposed BENO. In the context of Neumann boundary conditions, the equation takes the form:
VullI, v) = f(x,y), v(r, v) E E
@u(x,v) g(x,v), V(r. v) E 0n,
(18)
On
where ff represents the source term, typically represents the unit normal vector perpendicular to the boundary surface, and g specifies the prescribed rate of change normal to the boundary an. The challenge in solving Poisson's equation with Neumann boundary conditions lies in the proper treatment of the boundary derivative term, which requires sophisticated numerical schemes to approximate accurately.
Specifically, the model is trained exclusively on dataset consisting of 900 4-corners samples. The robustness and gemeralizability of approach were then evaluated on 5 different test datasets, which represent various boundary configurations encountered in practical applications. Each dataset is constructed to challenge the model with different boundary complexities.
The results are shown in Table g and Table Our proposed BENO still demonstrates superior performance across all test datasets in comparison to the baselines, including GNN-PDE, and MP- PDE models. Particularly, BENO achieves the lowest MAE and relative L2 norm scores in the majority of the scenarios. In Table g when tested on the 4-Corners dataset, BENO exhibites
20

---

Published as a conference paper at ICLR 2024
Branch1 Output
Branch2
Output
Figure 9: Visualization of the output from GNN branches.
Table 10: Performances of our proposed BENO and the compared baselines on Darcy flow, which are trained on 900 4-corners samples and tested on datasets under relative 1.2 norm and MAE separately. The unit of the MAE metric is 10-3 Bold fonts indicate the best.
Train 4-Comers dataset
3-Comers
2-Comers
1-Camer
No-Comer
Metric
L2
MAB
L2
MAB
L2
MAB
L2
MAE
L2
MAB
0.5802
0.3269
0.5332-
0.46224
0.6197
0.63074
0.6906+t
0.8469
0,7406
1.0906
MP-PDE
0.1840
0.2085
0.1742
0.2999
0.1709
0.3282
0.1432
0.4087
0,1271
0.3949
0.2431
0.1664
0,2542:1
0.2150
0.2672::
0.2585
0,2466
0.3091
0.2366
0.3591
BENO (ours)
0.0895
0.0773
0.1252
0.1270
0.1497
0,1313
0.1405
0.2350
0.1104
0.2116
L2 norm of 0.3568 and an MAE of 0.8311, outperforming all other methods and showcasing the effectiveness of our approach under strict 4-corners conditions.
When trained on mixed boundary conditions in Table 9 BENO still maintains the highest accuracy, yielding an relative 1.2 norm of 0.4237 and an MAE of 1.01 14 on the 4-Corners test set, confirming its robustness to varied training conditions. Notably, the improvement is significant in the more challenging No-Corner test set, where BENO's L2 is 0.3344, remarkable enhancement over the baseline methods. The bolded figures in the tables highlight the instances where BENO outperforms techniques.
all other models, underscoring the impact of our
The consistency of BENO's performance under different boundary conditions underscores its po- tential for applications in computational physics where such scenarios prevalent. Besides, the experimental outcomes affirm the efficacy BENO handling complex boundary problems the context of PDEs. It is also worth noting that the BENO model not only improves the prediction accuracy but also exhibits a significant reduction in error across different test cases, which critical for high-stakes applications such as numerical simulation in engineering and physical sciences.
L EXPERIMENTS ON DARCY FLOW
In this section, we consider the solution of the Darcy flow using our proposed BENO approach. The 2-d Darcy flow is second-order linear elliptic equation of the form
V (K( V Vu(r, v)) f(x,y, V(z,y) Ω
(19)
u(x, y) = g(x,v), v(a,y) E an,
where the coefficients x is generated by taking a linear combination of smooth basis function in the solution domain. The coefficients of the linear combination of these basis functions is taken from
21

---

Published as a conference paper at ICLR 2024
uniform distribution of random numbers. Dirichlet boundary condition is imposed along the boundary 0s2 using the function g which is sufficiently smooth. The objective of BENO is to map from the coefficient to solution u of the PDE in Equation 19
The model was exclusively trained on a dataset comprised of 900 samples, each featuring 4-corner configurations. To assess the robustness and adaptability of our method, we conduct evaluations five distinct test datasets. These datasets are deliberately chosen to represent a variety of boundary conditions commonly encountered in real-world applications, with each one designed present the model with different levels of boundary complexity. The outcomes of these evaluations are detailed in Table IO Our proposed BENO model consistently outperforms the best baseline across all test datasets. Notably, BENO achieves the lowest Mean MAE and relative L2 norm in the majority of these scenarios. This performance underscores the effectiveness of our approach, particularly under the stringent conditions of 4-corner boundaries.
M VISUALIZATION OF TWo BRANCHES
In this section, the visualized outputs of two distinct branches offer deeper insight into our model's functionality. Branch1, with the boundary input set zero, posited to approximate the impact emanating from the interior, while Branch2, nullifying the interior inputs, is conjectured to capture the boundary's influence on the interior. The observations from Figure lend credence to our hypothesis, indicating discernible delineation of roles between the two branches.
Extending this analysis, we further postulate that the interplay between Branchl and Branch2 is critical for accurately modeling the PDE solution landscape. The synergy of these branches, evidenced in our results, showcases a composite model that effectively balances the intricate houndary-interrs dynamics. This balance is crucial in situations where boundary conditions significantly dictate the behavior of the system, further emphasizing the robustness and adaptability of our model. The innovative dual-branch strategy presents promising avenue for enhancing the interpretability and precision of PDE solutions in complex domains.
N HYPER-PARAMETER LIST
Table 11: Hyper-parameters Configuration
Hyper-parameter Name
Hlper-parameter Value
Boundary Dimension
128
Node Dimension
128
Edge Dimension
128
Epochs
1000
Learning Rate
5e-05
MLP Layers Eq. 7
Nearest Node Number K
8
Message Passing Steps T
5
Transformer Layers
Number of Attention Head
2
Number of iterations for the first restart
Scheduler
CosineAnneellingaamee Reratss
Activation Function
Sigmoid Linear Unit (SiLU)
GPU Device
Nvidia A100 GPU
22


Reviews:
[{"review_id": "Ebg1BJt5MF", "reviewer": "Reviewer_QC4z", "summary": "This work introduces a novel neural operator architecture called BENO for solving elliptic PDEs with complex geometries and inhomogeneous boundary values. The BENO model outperforms state-of-the-art neural operators. The paper also discusses the potential impact of this research on various scientific and engineering domains, including computational fluid dynamics, solid mechanics, and electromagnetics.", "questions": "1. How can the dataset used to evaluate the BENO model be expanded to include more diverse scenarios and improve the generalization performance of the model? \n2. Would the authors be able to provide the number of parameters to quantify each model's complexity?\n3. How can the BENO model be adapted to handle time-dependent PDEs or other types of dynamic systems?\n4. Can the BENO model be used to generate physically meaningful insights and interpretability?\n5. There are some latest related operator works that should be referred to for a boarder audience:[Xiao, Xiongye, et al. \"Coupled Multiwavelet Neural Operator Learning for Coupled Partial Differential Equations.\"] [Gupta, Gaurav, et al. \"Multiwavelet-based operator learning for differential equations.\"]; [Gupta, Gaurav, et al. \"Non-linear operator approximations for initial value problems.\"].", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The manuscript is well-written and easy to follow.\n2. The paper introduces a novel neural operator architecture that embeds complex geometries and inhomogeneous boundary values into the solving of elliptic PDEs.\n3. The BENO model outperforms state-of-the-art neural operators and strong baselines.", "weaknesses": "1. Lack of comparison with more SOTA works like multiwavelet neural operator.\n2. Focusing on solving elliptic PDEs with complex geometries and inhomogeneous boundary values may limit the applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NTEtf5z0HL", "reviewer": "Reviewer_L19w", "summary": "The paper presents a modified neural operator for solving elliptic PDEs where in the boundary conditions are independently encoded as part of the architecture. The basic idea relies on constructing two GNN based operators for the interior source and the boundary values respectively and further encoding the boundary geometry using a Transformer. The approach is tested on a dataset of geometries with homogenous and inhomogenous boundary conditions.", "questions": "See weaknesses above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Overall the paper is well written and the details are clear and concise.\n\n2. The method appears to generalize to a variety of boundary conditions that competing methods have trouble solving.\n\n3, The experimental and ablation results are exhaustive and show the effect of each design choice.", "weaknesses": "1. The grid sizes for the domain are quite small (32x32, 64x64) when compared with FNO. I would like to see if the approach generalizes to larger and more practical grid sizes as well. This is especially important given the sequence length constraints that come with using transformers.\n\n2. It would also be useful to get some additional information about training and inference times for BENO and baselines.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CU3eU75hRw", "reviewer": "Reviewer_MqG8", "summary": "The existing models for operator learning have struggled to handle complex domains and boundary conditions effectively. To address this challenge, this paper introduces the Boundary-Embedded Neural Operators (BENO) model. BENO focuses on elliptic partial differential equations (PDEs) and employs separate graph neural networks for learning the source term inside the domain and the boundary values at the boundaries. It leverages message passing and transformers to learn the solution operator. Experimental results demonstrate that BENO offers more accurate predictions for areas with diverse boundaries compared to existing models.\n\n### Post-rebuttal\nI apologize for the delay in my response. Thank you for addressing most of my inquiries and concerns despite the limited time. I appreciate the swift inclusion of various experiments, which has alleviated my concerns. I hope that my questions and suggestions contribute to making the paper even better.\n\nHowever, I still have reservations about the generalization of this method. As observed in the current experiments, the domain handled is limited to rectangles with cut corners. I remain skeptical about its applicability to smoother circles or more complex domains, as I originally inquired. Instead of conducting experiments on similar domains, it would be beneficial to showcase its performance on entirely different shapes and more intricate domains. Although the authors mentioned this as future work in their response, it seems that further exploration in this direction is warranted. \n\nTherefore, based on my personal opinion, I will revise my score to a maximum of 5 points. I greatly appreciate your kind and diligent responses to my review.", "questions": "* In the paper, it is mentioned that other methods overfit in challenging boundary conditions, but a more explicit explanation of what this means would be helpful. Are there graphs or figures that can demonstrate this concept?\n\n* It would be beneficial to provide a clearer explanation of what the red and orange parts in Figure 1 represent. What does the red part signify, and what about the orange part? How do they differ?\n\n* Regarding Table 1, as mentioned in the Strengths section, is it impossible for FNO to handle 5.? Is it also impossible for GKN to handle 5.? Is there no prior research in the field of operator learning models that can handle 5.?\n\n* In the Problem Setup (2. Problem Setup), it is not clear whether BENO can be applied to 3D or higher-dimensional domains. While the use of graph neural networks suggests that it might be possible, further explanation would be helpful.\n\n* In the context of 3.1 Motivation, where branch1 is interpreted as the interior and branch2 as the exterior from a Green function perspective, it's important to know whether the output of Branch1 precisely reaches 0 at the boundary and whether the output of Branch2 reaches exactly \"g\" at the boundary. Is this a process of approximating \"g,\" or is the model capable of obtaining the exact boundary values of \"g\"?\n\n* In Section 3.2, a more detailed explanation of the definition of $\\mathcal{E}_{kn}$ would be appreciated. Furthermore, the part around this section involving attributes is somewhat challenging to comprehend in written form. It might be a good idea to exclude this part if it ultimately relates to the explanation in Section 3.3.2.\n\n* Are dx, dy, and dc in Section 3.3.1 uniquely determined for each node? Also, why is dc necessary?\n\n* On page 5, is \"B\" ultimately equal to \"B^1\"? It would be beneficial to provide a more detailed explanation of where \"t\" changes and what its range is.\n\n* The definition of MAE for Table 2 is present in the appendix but not in the main text.\n\n* In Section 4.3.2, the explanation of training at 32x32 and testing at 64x64, is this capability attributed to BENO's use of graph neural networks, or are there other features enabling this super-resolution? A more detailed explanation would be appreciated.\n\n* In Appendix K, when looking at Figure 6, it appears that the results for \"Ours\" are different from other solution profiles, forming square blocks with discontinuities. What is the reason or cause behind this phenomenon?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 1, "strengths": "The issue raised in the paper, where existing neural operator models struggle to handle complex domains and challenging boundary conditions, is indeed a significant problem that needs to be addressed. Many existing neural operator models often disregard PDEs and neglect boundary conditions when working with data. Furthermore, the problem posed in the paper is crucial in this field since operator learning is typically performed in straightforward domains. Moreover, the idea of using triangulation to locally partition complex domains is a promising approach to tackling these challenges.", "weaknesses": "However, the paper's focus on elliptic PDEs seems somewhat restrictive, especially considering that it primarily deals with the simplest case of the Poisson equation with Dirichlet boundary conditions. This might give the impression that the novelty of the paper is somewhat limited.\n\nFor example, can the proposed BENO method be applied to Poisson equations with Neumann boundary conditions or mixed Robin boundary conditions? The current model design appears to be tailored to the Dirichlet boundary conditions, which might be considered quite restrictive. To truly highlight the advantages of this approach, it would be beneficial to demonstrate its applicability to a wider range of PDEs, including parametric PDEs such as convection-diffusion equations or nonlinear Burgers' equations. Moreover, exploring its applicability to time-dependent parabolic PDEs could lead to a more generalized operator learning model.\n\nIn this context, the paper experimentally investigates a single type of operator, the one between (f, g) and the solution u. However, it's intriguing to consider whether other types of solution operators could also be learned (for instance, as seen in the FNO paper with Darcy flow, where the coefficient could be an input to the operator, or the initial condition of PDE can be an input).\n\nFurthermore, the experiments in complex domains appear to be limited to similar, complex domains, such as the \"5 different corner elliptic dataset.\" It would be interesting to explore whether the BENO method can be extended and applied to different scenarios, such as domains with circular holes in the center of a square or corners at non-right angles (or more simple, circle domain as in (Lotzsch et al., 2022)). By conducting experiments in more diverse domains and presenting results in a variety of scenarios, the paper could better showcase the novelty and versatility of the BENO method.\n\nTherefore, it would be beneficial to generalize the limitations of BENO, as described in this paper. For instance, in terms of boundary conditions, there are relevant discussions in [1], and graph-based methods have addressed complex domains and a variety of equations and boundary conditions, as seen in [2] and [3], which leveraged Finite Element Method (FEM). Referencing these works might provide valuable insights.\n\nFurthermore, there is some curiosity regarding whether the results of the baseline methods were adequately compared. It appears from Appendix F that the internal and boundary grids for the baseline methods are distinguished using one-hot encoding. It's worth considering if this method provides the fairest basis for comparison. As mentioned in the paper, FNO suggests the use of Fourier continuation for different boundary conditions and domains ('Non-uniform and Non-periodic Geometry' part in [4]). Therefore, it raises the question of whether the paper's approach is indeed the best for addressing complex scenarios. Additionally, there is a model for operator learning called DeepONet [5], which can be applied directly to challenging boundary conditions and complex domains. It would be interesting to know whether the paper conducted a comparison with this model.\n\n*[1] Horie, M., & Mitsume, N. (2022). Physics-Embedded Neural Networks: Graph Neural PDE Solvers with Mixed Boundary Conditions. Advances in Neural Information Processing Systems, 35, 23218-23229.*\n\n*[2] Lienen, M., & Günnemann, S. (2022). Learning the dynamics of physical systems from sparse observations with finite element networks. arXiv preprint arXiv:2203.08852.*\n\n*[3] Lee, J. Y., Ko, S., & Hong, Y. (2023). Finite Element Operator Network for Solving Parametric PDEs. arXiv preprint arXiv:2308.04690.*\n\n*[4] Kovachki, N. B., Li, Z., Liu, B., Azizzadenesheli, K., Bhattacharya, K., Stuart, A. M., & Anandkumar, A. (2023). Neural Operator: Learning Maps Between Function Spaces With Applications to PDEs. J. Mach. Learn. Res., 24(89), 1-97.*\n\n*[5] Lu, L., Meng, X., Cai, S., Mao, Z., Goswami, S., Zhang, Z., & Karniadakis, G. E. (2022). A comprehensive and fair comparison of two neural operators (with practical extensions) based on fair data. Computer Methods in Applied Mechanics and Engineering, 393, 114778.*", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uYu5Nlxyr6", "reviewer": "Reviewer_tbt5", "summary": "The paper proposed a novel neural operator for elliptic equations, particularly Poisson equations with Dirichlet boundary conditions in 2D domain, that has better flexibility to the geometry transformation of the problem domain.\n\nThe neural operator named BENO has several advantages over previous methods. No previous method can have train/test space grid independence, evaluation at on observed spatial locations, free-form spatial domain for boundary shape, inhomogeneous boundary condition values, at the same time.", "questions": "None.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Originality:** The paper has deliberately designed a neural operator for PDE with Dirichlet boundary conditions on free-form geometry domains. \n\n**Quality:** The paper has extensively benchmarked BENO with many existing NOs, and BENO shows great advantages.\n\n**Clarity:** The paper is well organized and clear presented.\n\n**Significance:** To have NO be able to deal with Dirichlet boundary conditions on domains of complex geometry is very important to the community, which can greatly improve the flexibility of NO and the range of scenario where NO can be useful.", "weaknesses": "The paper only deal with Poisson equations with Dirichlet boundary conditions, which limits the effect on convincing readers that BENO is superior than baselines.\n\nHow about more general elliptic equations?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YgaL1uWxhv", "reviewer": "Reviewer_113v", "summary": "The paper presents a method that involves conditioning a Message Passing Neural Network (MPNN) with boundary conditions to predict solutions for the Poisson equation. The approach involves training a model to learn the representation of input boundary conditions using a Transformer encoder. Subsequently, this representation is integrated into two message-passing neural networks, which work together to predict the solution. The model can condition the networks with boundaries of different shapes and values.", "questions": "- What motivates the use of different implementations for encoding the boundary condition (with Transformer) and the forcing term (with MPNNs)? Even if the goal is to learn a global representation for the boundary condition, couldn't MPNN achieve the same objective?\n- How are the 'distances to boundary $(dx_i, dy_i)$' defined? It appears to be more like an offset w.r.t the closest point on the boundary. If there are multiple closest points, how is the choice made?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper tackles the interesting problem of conditioning neural networks with varying boundary conditions and offers a reasonable solution for encoding both the boundary condition and its shape. \n- The paper is relatively well-written, providing comprehensive details on the model and the experiments. I appreciate the attention to aligning prior information given to different baselines.", "weaknesses": "- Several modeling choices require further clarification through ablation studies:\n  - It remains unclear to what extent the separation of the model into two terms is necessary without further empirical proof. Notably, both Branch 1 and Branch 2 share an identical architecture, differing only in their input, which comprises the forcing term and the boundary condition.\n  - As illustrated in Figure 1, it seems that the encoding of boundary conditions is achieved using a shared transformer, while the forcing term employs two distinct MPNNs. Is this particular choice of architecture a necessary one?\n- Despite the acknowledged disadvantages of concatenating the boundary conditions alongside the other inputs, it is indeed surprising to see the catastrophic failure of recent advances, such as MP-PDE, as illustrated, especially in Figure 5. The authors have mentioned that the learning rate and the weight decay factor are fixed for all experiments through grid search, but it remains unclear with which model these hyperparameters are searched for. Furthermore, aligning optimization hyperparameters across all architectures may benefit certain models but harm the performance of others, and this aspect should be considered. Therefore, more justification should be issued to prove that other baselines do not work at all to support more concretely the claim of a performance gain.\n- From the visualization, it appears that there are visible artifacts aligned with boundaries. These artifacts may be introduced by including $(dx_i, dy_i)$ in the node embedding. This raises questions about the necessity of incorporating $(dx_i, dy_i)$, as the Transform already appears to serve in learning a representation of the boundary shape.\n\n----------------------------------------------\nPost-rebuttal update: Having reviewed your explanations regarding the clarification of your architectural choices and engaging in discussions with other reviewers, most of my concerns have been addressed. Consequently, I have adjusted my rating to 6.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)