Nonparametric Boundary Geometry in Physics Informed Deep Learning
Authors: Scott Alexander Cameron
Arnu Pretorius
Stephen J. Roberts
Venue: NeurIPS 2023 poster (2023)
Keywords: PINNs
physics informed neural networks
geometric deep learning
neural operator
PDEs

Abstract: Engineering design problems frequently require solving systems of
partial differential equations with boundary conditions specified on
object geometries in the form of a triangular mesh. These boundary
geometries are provided by a designer and are problem dependent.
The efficiency of the design process greatly benefits from fast turnaround
times when repeatedly solving PDEs on various geometries. However,
most current work that uses machine learning to speed up the solution
process relies heavily on a fixed parameterization of the geometry, which
cannot be changed after training. This severely limits the possibility of
reusing a trained model across a variety of design problems.
In this work, we propose a novel neural operator architecture which accepts
boundary geometry, in the form of triangular meshes, as input and produces an
approximate solution to a given PDE as output. Once trained, the model can be
used to rapidly estimate the PDE solution over a new geometry, without the need for
retraining or representation of the geometry to a pre-specified parameterization.

Nonparametric Boundary Geometry in Physics Informed Deep Learning
Scott Cameron Oxford University, Instadeep Ltd. United Kingdom
Arnu Pretorius Instadeep Ltd. South Africa
Stephen Roberts Oxford University United Kingdom
scameron@@intaadeep. com
Abstract
Engineering design problems frequently require solving systems of partial differ- ential equations with boundary conditions specified on object geometries the form of triangular mesh. These boundary geometries are provided by designer and are problem dependent. The efficiency of the design process greatly benefits from fast turnaround times when repeatedly solving PDEs various geometries. However, most current work that uses machine learning to speed up the solution process relies heavily
fixed parameterization of the geometry, which cannot be changed after training. This severely limits the possibility of reusing a trained model across variety of design problems. In this work, propose novel neural operator architecture which accepts boundary geometry, in the form of triangular meshes, input and produces Once trained, the model can be used to rapidly estimate the PDE solution over a new geometry, without the need for retraining representation of the geometry with a pre-specified parameterization.
approximate solution given PDE as output.
Introduction
The prospect of applying machine learning complex computational problems in science and gineering is appealing for a variety of reasons. In particular, surrogate models allow for rapid pre- dictions be made without performing lengthy and expensive computations. Surrogate models are especially beneficial in iterative design of technical systems that traditionally rely on sophisticated simulations. One challenge in such applications is the principled inclusion of inductive biases to preserve properties of the underlying physics. Various approaches have been proposed, such as di- rectly parameterizing models obey conservation laws [1; 10], or by the inclusion of constraints [22; 18). Of key relevance to this paper is the approach popularly known as Physics Informed Neural Networks (PINNs) [12].
1.1 Physies informed neural networks
In their simplest form, PINNs are multilayer perceptrons trained to minimize a loss function based on the residual of some partial differential equation (PDE). The PDE in question is system of equations, with solution é. defined some open region Ω with boundary 5Ω:
F(Ø) = F(x, o Vo....) f(x) x Ω G(Ø) G(z, o.
(1)
80,
(2)
for smooth F.G.J. and g. For the vast majority of realistie physical problems, the domain Ω a subset three dimensional Euclidean space R Here we shall furthermore assume that Ω is bounded and the boundary 8Ω2 is compact. Many algorithms and techniques solve such systems
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

have been developed and extensively studied over the previous decades; perhaps the most popular approach being that of the finite element method (FEM), which approximates the solution piece- wise on (volumetric) mesh discretization of Q?. See Reddy [13] for an in depth introduction to FEM. The PINN approach approximates the solution & using a smooth neural network with parameters O. The PINN loss function measures the discrepancy of the network approximation from the PDF:2
[(0) = - [11(000) - (fx))))
dr + / ||G(o)) - -=(x)] |2 { dr.
(3)
Here, y and A are densities which may be chosen to increase or decrease importance of different points in the domain. In practice, these integrals are approximated using Monte Carlo sampling and the PDE residuals calculated using automatic differentiation.
Hypothetically speaking, with universal function approximator, the global optimizer of Equa- tion (3) should be able to provide an arbitrarily accurate solution to the the PDE. However, this is rarely realized in practice and one often needs to carefully tune the model and problem setup to give acceptable results. Various proposals have been made to improve the stability and robustness of PINNS such as enforcing exact boundary conditions using signed distance fields [16], highway networks [19; 15], alternative loss functions based on the variational formulation of PDEs [23], and causal training for time-dependent problems [20]. Furthermore, in industry applications, one would likely have access to (possibly sparse) simulation data, which would be used in a supervised fashion, with the PINN loss acting as a regularizer. For simplicity, in this paper, as proof of concept, we will only consider a basic scheme for training PINNs without simulation data; however it should be noted that in practice more advanced methods would typically be required.
One considerable downside to PINNS is that the time taken to train the network may be comparable to even significantly longer than applying classical techniques like FEM. Hence PINNs do not generally provide any advantage for one-off problems and traditional techniques provide better ac- curacy guarantees. However, it frequently the case that one wants solve a PDE for a family of conditions; for example by varying the parameters of the PDE or with a variety of different source terms. In these situations, PINNs can be trained conditionally to predict the solution of the PDE as function of the source term or PDE parameters. These models learn a map from one function space to another and are thus called neural operators [6]. The advantage provided by neural operators lies in their capacity to make extremely fast predictions once they have been trained, whilst an FEM solver would generally have to start from scratch for each new variant of the problem.
One important aspect of modeling in science and engineering applications is the definition of bound- ary conditions. It is frequently the case in engineering design that one defines some geometry in computer-aided design (CAD) software, and subsequently runs an FEM simulation to calculate physical properties such as stress, heat transfer, or aerodynamics. After observing the results of the simulation, the designer may need to modify the geometry and repeat the simulation several more times. In such cases, the actual boundary conditions are often defined by the physics in question (perhaps based on material properties), but the geometry is modified between simulations. Hennigh al. [3] provide an interesting example of a multi-physies design optimization problem involving a heatsink, where the geometry to be varied is described by fixed set of parameters. They train a conditional PINN and optimize over the geometry using the trained PINN as a surrogate model for the heat transfer. This approach, however, requires fixed parameterization of the geometry decided upon at training time and therefore the trained model limited this particular design problem.
In this paper, we propose a new neural operator architecture to learn the PDE solution operator, taking as input the boundary geometry in the form of triangular mesh This has many benefits:
* Once trained, the model may be used to rapidly predict the solution to the given PDE on new geometry without requiring expensive simulation.
Since boundary geometry is provided as triangular mesh, this allows the model to be reused across a wide variety of problems without retraining, and without requiring the geometry to conform to some fixed parameterization.
Liee of the same differentiabilitty class as the PDE. For example, ReLU is not everywhere differentiable. Although it is weakly differentiable, its second derivative almost everywhere zero, which makes inadequate for general second order PDEs.
Other norms may be used such L or Huber loss instead of 2

---

0. This would A fast predictive model enables the use of numerical optimization in the design process, po- tentially allowing for improved productivity and reduced iteration time. In particular, since the neural operator is differentiable, gradient-based optimization can be used to optimize geometry.
The increased applicability and utility of this type of model provides better justification for the time and resource costs involved in large-scale training in comparison to other PINNs which have limited reusability.
2 Geometry encoding with mesh-based neural operators
Solid geometry in CAD software is frequently represented by its boundary in the form of a triangular mesh. For physically realistic geometry, such boundary mesh is always manifold. This means the mesh is closed (has no holes in the surface, nor boundary), and each edge separates precisely two faces. Meshes which are non-triangular can always be triangulated, and we therefore restrict ourselves to this case. As consequence of these properties, we may employ the architecture of MeshCNN [2]. MeshCNN may be thought of as a direct generalization of convolutional neural networks to non- Euclidian geometries. This approach exploits a key feature of triangular manifold meshes- that every edge has precisely four immediate neighbours: those of the two triangles connected by that edge. Assuming we consider only positively oriented triangles (triangles whose normal vector points to the exterior the enclosed volume), there are two possible orderings for the edges making up these two triangles. To enforce invariance to this ordering, Hanocka et al. [2] propose using feature vectors which are symmetric under change of order. This is done by mapping the edge features of the four neighbours as follows
(c1, E c32 64) + (er + ez, lei egl, 62 + c4, lc2 -4l)-
Let ēt, denote the above transformed feature vector for the hth edge for j = 1,.... 4, and ē be the feature vector of edge i. The mesh convolution operation then defined
conv(K,ē), Σ Kyēs.
(5)
where each K, is a matrix of kernel weights. The fixed number of neighbours involved in the convolution greatly simplifies the implementation as compared to other neural network on graph- like domains. Hanocka al. [2] also design pooling operation the mesh which collapses certain edges, in analogy with pooling operations typically found in convolutional neural networks; however, this mechanism is more complicated to implement and instead standard nonlinear activation functions as is also commonly seen in normal convolutional neural networks.
2.1
PDE solution operator
The solution to the PDE is function of the boundary geometry. The dependency on the boundary geometry may be complex and highly nonlinear. The best method we have found to incorporate such dependence is in the use of the attention mechanism [17]. For this, we calculate encoded boundary geometry features using the MeshCNN. This takes into account local geometric information of the mesh. For the approximate PDE solution, use Transformer decoder architecture which uses cross-attention to aggregate these geometric features. Attention layers allow the model to take into account global geometric information, and allows the model to attend to different parts of the surface in different proportions depending on the point at which the function Φo is being evaluated.
Importantly, as a deviation from the standard Transformer decoder, we do not include a self-attention layer over the points of the domain being evaluated. Naturally, the value of de at a point should be independent of whether or not one is simultaneously calculating its value at another point . In other words, the forward pass of the neural network should give identical results, regardless of whether it is performed over the domain one point at time, or performed over those points in parallel as batch. This can be stated mathematically with the chain rule: de(z) de(z) öe
0r"
tr or
no longer be the case if self-attention layers were included in the decoder.
'This is analogous to the von Neumann neighbourhood in cellular automata as opposed to the Moore neigh- bourhood
3

---

((3)
F
Add & Norm
Fully Connected
Alzention
ReLU
MeshCorn
Fully Connected
Figure 1: Neural operator architecture.
Barring the lack of self-attention, our decoder largely the same as that given by Vaswani et al. [17]: sequence of blocks, each of which consists of multi-head dot-product attention, with residual connection and layer normalization, followed by a single hidden-layer feed-forward neural network, also with a residual connection and layer normalization. In the attention layers, the query vectors are linear projections of the outputs from the previous layer, while the key and value vectors are linear projections of the encoded edge features which are the output of the MeshCNN. The final layer output is just a linear layer which projects down to the dimension of ø. Figure gives a visual representation of this architecture.
We attempted many variations in building up to this architecture. One specific approach we tried was to use an operator of a similar form to DeepONet [9]. DeepONet is neural operator defined by two neural networks; the branch which takes as input point evaluations locations, and as output produces the final layer weights for the second network, the frunk, which itself is the result of applying the operator. Lu et al. [9] show that under mild assumptions (such as non-polynomial activation functions), any continuous nonlinear operator can be approximated arbitrarily accurately on a compact set by a neural operator this form. Our first approach was to DeepONet, using MeshCNN as the branch network and simply taking the mean over edges to reduce its output to a fixed dimensionality. Despite its theoretical properties, soon became clear that, in practice, this model did not provide enough flexibility for this application.
function at some predefined
As an improvement, we thereafter tried a single attention layer to perform a weighted average over MeshCNN output, which seemed to help somewhat, but still failed to give any results of satisfactory accuracy. This led us to consider using multiple layers of attention, which showed a significant per- formance increase and proved to be flexible enough model the PDE solution operator accurately."
2.2 Invariance geometry preserving transformations
In similar manner to standard convolutional neural networks' invariance to translations, MeshCNN invariant to transformations which preserve the local topology of the mesh. When the input fea- tures of the mesh depend on local geometry (such as curvature, torsion, or a given vector field) then MeshCNN is invariant only to transformations which preserve this local geometry. This invariance property is manifest in the definition of the convolution in Equation (5). Ideally, for application to boundary geometry, our model would also invariant transformations of the continuum manifold which preserve local geometry; however, the discretization of the continuum manifold into trian-
'Wile the architecture is flexible enough give accurate PDE solutions, this does not guarantee that simply training Equation (3) will achieve accurate fit in practice without more advanced schemes.

---

gular mesh breaks this invariance, and so the MeshCNN is not guaranteed to be truly invariant under different mesh discretizations. Despite this, Hanocka et al. [2] report that, in experiments, this in- variance is approximately preserved, and trained MeshCNNs may give quantitatively similar results on two different triangulations of the same mesh. Importantly, this discretization should be done in way which best preserves the overall geometry and maintains a fine enough resolution in order to be valid. Geometric features are calculated in a manner which approximates their continuous counterparts in the limit of an infinite resolution mesh. Naturally, poor quality discretization (with either a very low resolution or one which poorly approximates the geometry) should be expected to adversely affect the quality of the results. Approximate invariance discretization depends on the assumption that the geometric features themselves are approximately invariant to the discretization process, which is only true for meshes of sufficiently fine resolution. The extent which the bound- ary mesh resolution can be reduced without significant loss of quality in model predictions poses an interesting research question, albeit one which we leave for future work.
To maintain the appropriate invariance properties of the PDE solution map, edge features on the mesh must be chosen accordingly. For mesh segmentation and classification purposes, Hanocka et al. [2] employ features which are invariant to similarity transformations, such as translation, rotation and scaling. This is appropriate because the predictions of the network transform scalars. In our case, the mesh is assumed to be embedded in R3 and the output of the network is a function on the interior domain which is subset of the same R Thus applying a transformation T, which is symmetry of the PDE, to the boundary geometry, should result in the same solution with the inverse transformation applied to the interior domain 6(x) o(T (z)). The solution operator is thus contravariant as opposed to invariant under symmetry transformations. Although it is not obvious how this contravariance could be explicitly encoded in the operator network architecture, we can encourage the network to learn contravariance by using covariant or contravariant edge features, and data augmentation by random transformations of the boundary geometry during training, As examples of such edge features, we use the midpoint and length of each edge, local surface normal vectors, and local curvature.
3 Experiments
For boundary geometry training data, we use a collection of meshes created in Fusion 360 [21]. After preprocessing and filtering, our data set contains about 12 thousand triangular meshes. For validation data, we use an FEM solver on a small collection of meshes, which we treat as our ground truth. While FEM solvers are only approximate methods, in the cases we consider they are extremely accurate and so their errors are negligible. We measure the mean squared error (MSE) of our neural operator approximation compared the FEM solution and report these errors as percentages of the mean square value (the average intensity) of the FEM solution. For a comparison of our model to vanilla PINNs trained on individual meshes, see Appendix A. During training, we save the model parameters with the lowest average MSE over the validation set.
We generate gcometry-encodiing edge features described in Section 2.2, consisting of the edge midpoint, length, normal vector, and sealar curvature. Interior and boundary points are generated by sampling uniformly inside and over the surface respectively. These points used to estimate the loss function in Equation (3), where we replace the L norm with a Huber loss for stability. We use data augmentation scheme, consisting of random translations, rotations, and nonuniform scaling of the meshes.
For our network architecture, we use a MeshCNN with ReLU nonlinearities, and SiLU nonlinear- ities for all activation functions in the Transformer decoder. As input to the Transformer decoder we found that (learnable) linear combination of Fourier features and SiLU embeddings worked better than either alone. We also attempted to use only Fourier features as input to the decoder but found that it gave considerably worse results. Our MeshCNN uses four convolutional layers, each having 512 output channels. For the Transformer decoder, we use blocks, using attention layers
'Here we are assuming the value of ((z) transforms as a scalar on the domain. Similar notions of covari- ance/contravariance apply when o(x) transforms as a tensor.
'Available https: /tap:://gitae //gtthtt.b , com/ AutodeskA Law./Laaaaaaaaeeee aatruiiooooalleeaaattaaaaa
"The smoothness requirement for PINNs only applies to the part of the network being differentiated, ie. the decoder which takes x as input should be smooth but the MeshCNN, which does not depend on æ, can have arbitrary nonlinearities.
5

---

with 8 heads and feature size of 512. The hidden-layer size of 2048.
network sublayers in each block have
We use stochastic gradient descent with momentum as we found other optimizers (Adam [5], RAdam [8] and RMSProp) to be very unstable. We cosine annealing learning rate sched- ule with linear warmup to a maximum learning rate of 0.001. To improve stability we use gradient clipping and gradient accumulation to increase the effective batch size.
For the experiments below, unless otherwise stated, we use a Gaussian source term centered at the origin as smoothed-out approximation point-like source. During data augmentation, ensure that the origin point is always on the interior of the PDE domain (0 E 92).
All experiments use the same network architecture and hyper-parameters. They were trained on an RTX 3070 with 8Gb of VRAM.
3.1 Poisson equation
The Poisson equation with Dirichlet boundary conditions is defined as
-2 = f(x) x E S <(z)
(6)
x E 652.
This equation arises a number of applications including steady state diffusion processes, Newto- nian gravity and the study of electromagnetism.
For this equation, as sanity check, we first use the so-called "method of manufactured solu- tions" to compare against simple analytic solution. By substituting in the analytic solution:
(1) sin(y), we arrive the Poisson equation with the appropriate source term and boundary conditions as follows:
V2g 2 cos(2) sin(y) 0E Ω <(x) = cos(xr sin x E 6Ω.
(8)
(9)
This method is commonly used to test numerical solvers against known analytic solutions. Figure shows a comparison between the manufactured solution the neural operator prediction on an example test geometry. In all figures below, the color intensities of the FEM or true solution and those the neural operator share the same normalization. In this example, the MSE 4% of the average intensity, and the model prediction had average error of 5% across the validation set.
0.86
4,00813
0.800
0.000687
(a) True solution
(b) Neural operator
(c) Absolute difference
Figure 2: Poisson equation: manufactured solution. a shows the manufactured analytical solution, shows the output of our trained model the same geometry, and show the absolute error of the model compared to the true solution.
For the Gaussian point-like source term, Figure 3 shows the comparison of the neural operator prediction to the FEM solution on two test meshes, having an MSE of 3% and 8% of the average intensity respectively.

---

1.51
14
0.307
1.13
1.00
0.20
0.788
0.584
0.154
0.78
0.19
0.0700
-2400-3
0.307
8.72w008
(a) FEM
(b) Neural operator
(c) Absolute difference
1.37
0,190
1.03
.07
0.149
0.636
0747
0.108
0348
0,420
0.0883
4.10 34
0.0941
0.0250
(d) FEM
(c) Neural operator
(f) Absolute difference
Figure 3: Poisson equation. a, d show the FEM solution, b, show the output of our trained model on the same geometry, and c, f show the absolute enror of the model compared to the FEM solution.
3.2 Helmholtz equation
The Helmholtz equation is the eigenvalue equation for the Laplace operator. The Helmholtz problem with Neumann boundary conditions is given as:
= f(x) x x Q V + -
(10)
Vo(z) =0 82,
(11)
where x² is the eigenvalue and is the unit normal vector the surface 8Ω The solutions to this equation form a basis which diagonalizes the Laplace operator, and hence this equation arises in many applications. In particular, this equation is used in studying wave propagation, and therefore specifically in acoustics and optics. Here we assume the eigenvalue of interest to be x² 1
Figure 4 shows the FEM solution and neural operator prediction for the Helmholtz equation. On both these test meshes the MSE of the neural operator is approximately 2% of the average intensity.
3.3 Steady state reaction-difflsion system
Reaction-diffuse diffusiidfffoonnno equations frequently used in modeling chemical reactions, biological sys- tems such as in predator-prey models, combustion, population dynamics, and spread of disease in epidemiology. We consider the case of quadratic reaction term, known Fisher's equation, with Neumann boundary conditions in the stationary (0,0 = 0) limit:
v + 6(1 f(x) x S
(12)
n. Vo(x) = 0 x E 52.
(13)
Figure 5 shows our neural operator predictions on three test meshes. The MSEs for these predictions are I 1%, 1,, and 5% of the mean intensities respectively. There noticeable qualitative difference between the neural operator solution and the FEM solution in the third row (Figures 5g to 5i). The small relative volume covered by this error suppresses contribution to the MSE, and to the loss function in Equation (3).
7

---

0.0744
0.0008
0.105
-0120
0,0020
0.0798
0.164
0.188
0,0929
4889
0.174
0,0885
0225
00215
3.62-055
(a) FEM
(b) Neural operator
(c) Absolute difference
000873
0.187
-0210
0.0458
4024
0,238
0.0246
0.577
0.282
0.000219
(d) FEM
(e) Neural operator
(I) Absolute difference
Figure 4: Helmholtz equation. a, d show the FEM solution, b, e show the output of our trained model on the same geometry, and show the absolute error of the model compared the FEM solution.
0.0900
9.0751
0.0644
0.124
0.118
0,0408
0.157
0.150
0.0272
0.180
0,001
0.0188
0.221
0.244
2.88 05
(a) FEM
(b) Neural operator
(c) Absolute difference
0.0832
0.0925
0.130
-.169
0.174
0,0882
0.207
0,216
0.0131
0.259
4.0ee005
(d) FEM
(e) Neural operator
(f) Absolute difference
0.221
0.254
0.26
0.8211
0.291
0.323
250e00
FEM
(h) Neural operator
(i) Absolutc difference
Figure 5: Reaction-diffusion system. a, d, g show the FEM solution, b. e, show the output of our trained model on the same geometry, and c, show the absolute enror of the model compared to the FEM solution.

---

4
Related work
There have been many advances in physics informed learning in recent years. However, relatively few these advances specifically focus on boundary geometry.
On fixed rectangular geometries. convolutional neural networks and Fourier neural operators [7] can be used predict solutions to PDEs as a function of the souree term, or solve inverse problems such as inference of initial conditions of dynamical field from its current configuration. They may be trained with or without physies informed approach. However, the these models cannot be used on irregular domains.
Using graph neural networks, can extend these approaches irregular domains using volu- metric mesh. Pfaff et al. [1i] build a model to predict the result of an expensive simulation on a mesh, by training on data generated by the simulation. However, this approach does not encode physical inductive biases as PINNs do, using a standard supervised approach. Hamiltonian graph neural networks [14] can be used on meshes similarly, with the difference that they explicitly encode energy conservation by simulating a Hamiltonian system.
Kashefi and Mukerji [4] have proposed model to approximate the solution to given PDE on multiple irregular geometries. Their model is trained in a physics informed manner and is based on PointNet, which is neural operator designed for processing point clouds. These point clouds form an implicit representation of the geometry and their model can be used to make predictions on new geometries just by generating interior samples. However, compared point clouds, meshes contain far more geometric information [2] such as local connectivity and curvature.
5 Conclusion
In this paper, we propose novel neural operator architecture for the purposes of learning PDE solutions as a function of boundary geometry without restricting the geometry to a fixed, finite- dimensional parameterization. This provides variety of benefits and significantly extends the possible applications and reusability of trained models over other PINNs. We believe this is an important step toward feasible PINN technology at industry scale, and reducing time and resource costs associated with traditional simulations.
We showed that our neural operator is capable of approximating the PDE solution operator, even when trained without simulation data or more advanced PINN methodologies. While we chose not to incorporate more sophisticated training techniques for the sake of simplicity and proof of concept, we highly recommend that they be used in any practical application, along with example data from simulations to improve the reliability and robustness of the model. We believe this is especially important as the geometry and domain of interest increase in complexity.
5.1 Limitations
There are a variety of limitations with using neural networks to approximate solutions to PDEs. The subject itself is still relatively immature when compared to traditional methods like FEM or finite volume methods, and lacks the theoretical guarantees and wealth of insight that have been developed for these methods. Problems of exploding or vanishing gradients are exacerbated by the triple, or even higher order derivatives appearing from the optimization of Equation (3). When training on the PDE itself, loss curves may vary by multiple orders of magnitude from one epoch to the next, sometimes increasing even when the error of the solution is decreasing. This makes the learning curves very difficult to interpret. Furthermore, small loss values do not necessarily mean that the solution itself is accurate, and this complicates error estimation and uncertainty quantification.
For our particular experiments, we found that the lowest errors for each validation mesh occurred different epochs and so while we were able to get very accurate results (occasionally even below 1 % MSE) for each individual validation mesh if we monitored its eтror and used early stopping, we were not able to achieve such accurate results on each mesh simultaneously. We believe this is mostly due to the difficulties of fitting to the PDE directly rather than a limitation of the model itself, and we expect to achieve better results with supervised learning on data from simulations in addition to the PINN loss function.

---

Acknowledgments and Disclosure of Funding
This work was sponsored by InstaDeep Ltd.
References
[1] S. Greydanus, M. Dzamba, and J. Yosinski. Hamiltonian Neural Networks. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alhhé Buc, E. Fox, and R. Gar- nett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https:/ https://proceedings Becdeaaaccckkerccccaaassdddd
[2] R. Hanocka, A. Hertz, N. Fish, R. Giryes, S. Fleishman, and D. Cohen-Or. Meshenn: A network with an edge. ACM Transactions on Graphics (TOG), ((44):90:1990111 2019.
[3] O. Hennigh, S. Narasimhan, M. A. Nabian, A. Subramaniam, K. Tangsali, M. Rietmann, del Aguila Ferrandis, W. Byeon, Z. Fang, and S. Choudhry, Nvidia simnet: an ai-accelerated multi-physics simulation framework, 2020.
[4] A. Kashefi and T. Mukerji. Physics-informed pointnet: A deep learning solver for steady- state incompressible flows and thermal fields on multiple sets of irregular geometries. Jour- nal of Computational Physics, 468:111510, 2022. ISSN 0021-9991. doi: https://doi.org/ 212..111000. URL https: //ww.. sciencedirect. com/sccience//rticll... pi1/$0021999112005722
[5] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. CoRR, abs/ 1 1412.6980, 2014.
[6] N. Kovachki, Z. Li, B. Liu, K. Arizzadenesheli, K. Bhattacharya, A. Stuart, and A. Anandku- mar. Neural operator: Learning maps between function spaces, 2023.
[7] Z. Li, N. Kovachki, K. Azizzadenesheli, B. Liu, K. Bhattacharya, A. Stuart, and A. Anandku- mar. Fourier neural operator for parametric partial differential equations, 2021.
[8] L. Liu, H. Jiang, P. He, W. Chen, X. Liu, J. Gao, and J. Han. On the variance of the adaptive learning rate and beyond, 2021.
[9] Lu, P. Jin, G. Pang, Z. Zhang, and G. E. Karniadakis. Learning nonlinear operators via Deep- ONet based on the universal approximation theorem of operators. Nature Machine Intelligence, 333)2218229, 2021. doi: 10. 211038/42226-022-0000000
[10) M. Lutter, C. Ritter, J. Peters, C. Science, T. Universit, D. Hochschulstr, and L. Mechanics. Deep Lagrangian Networks: Using Physics as Model Prior for Deep Learning. Ielr.
[11] T. Pfaff, M. Fortunato, A. Sanchez-Goozalezz, and P. Battaglia. Learning mesh-based simu- lation with graph networks. In International Conference on Learning Representations, 2021. URL https:// wtt.:eeeeeettttommdiddiddo/
XP.
[12] M. Raissi, P. Perdikaris, and G. E. Karniadakis. Physics informed deep learning (part i): Data- driven solutions of nonlinear partial differential equations.
[13] N. Reddy. An introduction the finite element method. McGraw-Hie Hill, fourth edition, 2019.
[14] A. Sanchez-Gonzale V. Bapst, K. Cranmer, and P. Battaglia. Hamiltonian Graph Networks with ODE Integrators, 2019.
[15] Sirignano and K. Spiliopoulos. DGM: deep learning algorithm for solving partial differ- ential equations. Journal of Computational Physics, 375:1339-1364 2018. ISSN 0021-9991. doi: iitaac///riicee///////0199991111
029. URL https: / /wwww//www//////////////.... www.seedireett.c....mm.mm com/
10

---

[16] N. Sukumar and A. Srivastava. Exact imposition of boundary conditions with distance func- tions in inysics-informed deep neural networks. Computer Methods Applied Mechan- ics and Engineering, 389:114333 2022. ISSN 0045-7825. doi: https://doi.org/100 10161/
J,cma.2021.114333 URL https://wwtt http://www..ciineeereeet.t. comf//cceence//rtiiclepp... $0045782521006186.
[17] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need.
Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan. and R. Gamett, editors, Ad- vances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https: httpse://proceeedngg. www.dedniniinnnnnnneeeeeeeee wwtpccppppcceeeeeee httpr//lellaappp////////111/ #fee225477ee1110033cccaaaee Paparaaapppppppppp pdf.
[18] H. Wang and X. Zhou. Explicit estimation of derivatives from data and differential equations by gaussian process regression.
[19] Wang. Teng, and Perdikaris. Understanding and mitigating gradient flow pathologies in physics-informed neural networks. SIAM Journal on Scientific Computing, 43(5):A A305- A3081, 2021. doi: 10. 10.113700MM318044 URL https://ddot org/ 10. 1137/20M1318043.
[20] S. Wang, S. Sankaran, and P. Perdikaris. physics-informet neural networks, 2022.
Respecting causality is all you need for training
[21] K. D. Willis, P. K. Jayaraman, H. Chu, Y. Tian, Y. Li, D. Grandi, A. Sanghi, L. Tran, J. G. Lambourne, A. Solar-Lezama, and W. Matusik. Joinable: Learning bottom-up assembly of parametric cad joints. arXiv preprini urXiv:2111.12772 2021.
[22] Yang. S. W. Wong, and S. Kou. Inference of dynamic systems from noisy and sparse data via manifold-constrained gaussian processes.
[23] Y. Zang, G. Bao, X. Ye, and H. Zhou. Weak adversarial networks for high-dimensional partial differential equations. Journal of Computational Physics, 411:109409, 2020. ISSN 0021- 9991. doi: http::/hooo.rrr//l0.ll1........p
URL http://www.iieeeeieee.ttt

---

A Vanilla PINN results
For few selected test meshes, we attempted training standard PINN, without conditioning on the geometry. Here we show comparison of these PINNs to our trained neural operator. The training code identical for both the operator and standard PINNs, with the only differences being that the PINN model does not include any mesh encoding or attention layers, and the dataset is restricted to only a single mesh.
The PINN model architecture consists of 5 blocks, each of which is a fully connected feed-forward network with skip connection and SiLU nonlinearities. The inputs and outputs of each block are of size 512, and each feed-forward network has a hidden-layer size of 2048. We use the linear combination of Fourier features and SiLU embeddings in the first layer. These parameters are chosen to match our neural operator for fair comparison. This model has 12.6 million parameters.
In all cases, the standard PINN fails miserably, producing far less accurate results than the neural op- erator. It appears that in this case, the process of learning on all geometries simultaneously improves the ability of the model find an accurate solution to the PDE. Further investigation would need to be done to assess the extent to which this behaviour occurs, but we leave that for future work.
0.473
0.314
0348
0.420
155
4.16-34 14:34
0.0041
0,00857
(a) FEM
(b) Neural operator
(c)
PINN
Figure 6: Comparison to vanilla PINNS on the Poisson equation
0.0784
0.0888
0.3883
-0.100
4.0000
-00000
0.184
0.155
0.0758
0.899
0.174
0,0845
02253
-.215
0.0032
(a) FEM
(b) Neural operator
(c) PINN
Figure 7: Comparison vanilla PINNs on Helmholtz equation
0.132
0,216
0.0480
0.245
0.299
0.0515
(a) FEM
(b) Neural operator
(c) PINN
Figure 8: Comparison to vanilla PINNs on reaction diffusion equation
12


Reviews:
[{"review_id": "I4Jcba3ovF", "reviewer": "Reviewer_L8FG", "summary": "The article describes a method to obtain the solution of a PDE given just the boundary mesh as an input. A variety of edge features are first transformed using MeshCNN, which are then used with a Transformer decoder to obtain the solution. The method is demonstrated to work for a few different PDEs on relatively simple domains.", "questions": "It is not clear to me how one goes from the input (mesh manifold) to the solution at any physical point in the domain.", "limitations": "It is not clear whether the method can be effective for large scale problems (think 10k+ elements) where the benefits of a fast solution estimator would actually be helpful.", "rating": 3, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The proposed framework is potentially powerful considering it is supposed to work off of the mesh manifold, which is typically the starting point of the computational analyses. ", "weaknesses": "The method is demonstrated to work qualitatively, and only on toy problems.\nThe framework presentation should be improved to demonstrate the framework architecture and how one goes from input (boundary mesh) to output (solution at a physical point in the domain).. I have gone through the text several times and it is still not clear to me. While the authors refer to other papers, I think it is important to make the article self-sufficient to an extent for the reader.\nWithout sufficient evidence, it is hard to see how this framework would work for complex domains which require large number of elements to accurately solve the PDE of interest.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Please see our other response and the attached diagram for an explanation of our architecture. We hope this provides a clearer picture of our model.\n\nRegarding the complexity of the geometry, it is important to state that the quality of predictions will be highly dependent on the size of the model and the dataset being trained on. This is no different from image processing problems using CNNs. We expect that, with a high quality dataset and sufficient compute, it should in theory be possible to train a MeshCNN on complex geometry with high resolution detail. However, due to limited compute budget, we are not able to scale up our model at this current time.\n\nAnother important point is that we are only considering the boundary geometry, i.e. a closed 2 dimensional surface. This is in contrast to FEM methods and the like which operate on a 3 dimensional volumetric (eg. tetrahedral) mesh. Volumetric meshes require orders of magnitude mode elements for a similar resolution than their corresponding boundary surfaces. A boundary mesh may only need one or two thousand edges to represent very complex geometry, while the interior would require tens or hundreds of thousands of elements for an accurate FEM simulation.\n"}, {"review_id": "fI05xpX43w", "reviewer": "Reviewer_ECF7", "summary": "Physics Informed Neural Nets (PINN) have gained considerable attention lately. However, they are significantly expensive as compared to FEM or other classical PDE solvers. Moreover, any trained PINN is specific to the object geometry it has been trained upon. This paper proposes a solution to reuse the trained model to various object geometries by learning a PDE solution without restricting to a certain object parametrisation. It takes into account the bounding conditions on the object in terms of meshes and deploys MeshCNN instead of regular CNN to encode local geometric properties under an attention mechanish [17] to learn the solutions to PDE.", "questions": "see weakness section", "limitations": "The limitations have been discussed", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The proposed method addresses an important problem of making the physics-based learning more generic.\n\nThe paper discusses an important aspect of learning on meshes: contravariance/invariance of geometric properties under mesh formulations. It enforces the preservation of geometric quantities in order to solve the PDE.\n\nThe experiments show various well-known PDE systems are solved quite accurately using the proposed approach.", "weaknesses": "The writing is a bit difficult to follow. The use of footnotes is weird and interrupts the reader's flow.\n\nSome design aspects of the paper are not clear. Lines 118-120 suggest that the self-attention block of the Transformer is removed.  The authors argue that the value of learned parameter at a point x should be independent of whether or not one is simultaneously calculating its value at another point x′. This argument needs to be explained further as geometrically close points manifest similar properties and learning them together (or simultaneously) can be beneficial to obtain the solution to the PDE. It is understandable that the geometrically distant points need not to considered simultaneously. Choosing to remove the self-attention layer seems to be a compromise and should be explained in more details.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Please see our other response and the attached diagram for an explanation of our architecture. We hope this provides a clearer picture of our model.\n\nWe have attempted to explain above why it is undesirable for information to be shared between different points. For a bit of further clarification, it is certainly possible to build a model which does explicitly share information using eg. self-attention, but this would be a different species of operator. If $\\mathcal{M}$ is the space of 2 dimensional closed boundary manifolds in $\\mathbb{R}^{3}$ and $C^{2}(\\mathbb{R}^{3})$ is the space of twice differentiable functions, our operator has type $\\mathcal{M} \\to C^{2}(\\mathbb{R}^{3})$. By partial application of the MeshCNN part of the model, one arrives at a function $\\phi \\in C^{2}(\\mathbb{R}^{3})$, which can be evaluated independently at any point within the domain. On the other hand, if we were to allow mixing of information between points with self-attention, we would have an operator of type $\\mathcal{M} \\to (\\mathbb{R}^{3n} \\to \\mathbb{R}^{n})$ for all values of $n$. Partially applying the MeshCNN in this case would result in an object which is not an element of $C^{2}(\\mathbb{R}^{3})$, but is itself an operator on a more complex space: something like $\\hat\\phi(x_1,x_2,\\ldots) = (\\phi(x_1), \\phi(x_2), \\ldots)$. In this case, the Jacobian of $\\hat\\phi$ would not be diagonal and the loss calculation would need to take this into account. \n\nThis type of approach is better suited for cases when the source term in the PDE is an input to the model. For example, something like $\\nabla^2\\phi(x) = f(x)$, in which case the solution does in fact have that $\\frac{\\delta\\phi(x)}{\\delta f(y)} = G(x - y) = \\frac{1}{4\\pi \\|x - y\\|} \\ne 0$, $G$ being the Green’s function. We leave this case to future work.\n\nRegarding similarity of nearby points, this is captured implicitly by the smoothness of the network.\n"}, {"review_id": "DwZ4GhVqo7", "reviewer": "Reviewer_4pp2", "summary": "This paper proposes a neural operator methods that can take different boundary geometry, in the representation of a mesh, as input to solve different PDEs. To the best of my understanding, the proposed method can take geometry represented in different triangular meshes as input and predict PDE solutions (i.e. a function of the geometry). The proposed method used MeshCNN to extract geometry features, and use cross attention transformer as the decoder to produce the solution. The proposed model needs to be trained only once for a specific PDE, then it can be used to produce solution for different conditions.\n\nThis work is trying to address a very important problem, but due to the presentation issue and some concern with the results, I am leaning toward rejection. I think the paper can be greatly strengthen with an iteration with better presentation and more evaluation.", "questions": "See the weakness section.", "limitations": "Yes, the author has a good discussion of the limitation.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. I think the problem this paper set-up to solve is of very important. When the engineers are using a model to speed up their simulation pipelines, they usually require solving the same problem with different geometries. If the trained model can only work on one specific geometry, then it might not be the best fit of such pipeline. With this said, if this paper is successful, it can enable neural operator to be used in more practical scenario.\n2. The proposed method (if I understand it correctly) can indeed end handle different geometry while maintaining respected invariances required for the problem set-up.", "weaknesses": "1. Presentation issues. I found the paper (especially the method section) difficult to understand as it lacks of rigorous definition of the problem set-up (e.g. input/output of the model; how do we represent it; what’s the network architecture;). As the paper is proposing a specific way to achieve this goal, I think including a more rigorous definition of the model can improve reproducibility and help understand why specific model design choice is necessary.\n2. The results does not necessarily support the claims. A important claim of the paper is that their proposed model can be trained over different geometries of the same PDE, thus provide advantage by amortizing the training compute to benefit multiple simulation rounds. To show this, I think it’s necessary to compare to the baseline where for each geometry, we train a neural operator (which we might not be able to do without data) or we solve for a PINN for each geometry (this is valid as PINNs can be solved for scratch). I expect this comparison turns out to be that the proposed method is comparable with the baseline, while being faster since PINNs requires optimization while the proposed method requires only forward pass. Right now the results only show comparison with the ground truth, which makes it difficult to see 1) whether the proposed method is accurate enough, and 2) whether the hypothesized advantage in amortizing the training across different geometry exists. \n3. Agnostic to discrete action. While the paper cite MeshCNN to argue that in practice the feature extraction pipeline is not adversely affected by the discretization quality, I think this argument is not sufficient as simulation using FEM usually requires a careful discretization and it’s simulation is different from the task MeshCNN were designed to do (classification and segmentation). The fact that the method section mentioned the manifold assumption in L88-100 strengthen my concern. While this can be better fit into the future-works scope, I would recommend the authors to stress-test how different discretization affects the performance, and probably discuss how this method can be applied to non-curated meshes such as mesh obtained from real-world scans.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Please see our other response and the attached diagram for an explanation of our architecture. We hope this provides a clearer picture of our model.\n\nWe have added in the attached pdf two examples of vanilla PINNs trained on test geometries for comparison with our neural operator. For the architecture, we use the same feed forward layers that appear in our operator with silu non-linearities and skip connections. In other words, we train a PINN consisting of 5 blocks, where each block is of the form:\n$h_{t+1} = \\operatorname{LayerNorm}(h_t + W_t \\operatorname{silu}(U_t h_t + a_t) + b_t)$,\nwhere each $h_t$ is 512 dimensional, and each block has 2048 hidden neurons.\nThese values are chosen in an attempt to make a fair comparison.\nEven though we observe the losses converging, the fitted model is often quite different from the ground truth solution. Without more sophisticated training techniques or simulation data (see the introduction of our paper) we have thus far been unable to attain better performance with vanilla PINNs on these geometries than what is shown in the attached pdf. \n\nThe point regarding the discretization quality of the boundary is a very important one. The MeshCNN paper claims that in practice they noticed their model was able to maintain accurate predictions on different discretizations of the same object. However, it is important to note that these different discretizations were done in a way which preserves the overall geometry (this  requirement is much more restrictive on surface meshes than on volumetric meshes), while also maintaining similar resolution. Geometric features are calculated in a manner which approximates their continuous counterparts in the limit of an infinite resolution mesh. It does make sense then that a poor quality discretization, with either a very different resolution or one which poorly approximates the geometry should indeed adversely affect the quality of the results. Approximate invariance to discretization depends on the assumption that the geometric features themselves are approximately invariant to the discretization process, which is only true for meshes of fine enough resolution.\nAs you mention, we will leave a rigorous assessment of mesh discretization to future work, but we will ensure that this point is more clearly discussed in the paper.\n"}, {"review_id": "F3Vi2sYvIi", "reviewer": "Reviewer_mGk1", "summary": "The Authors introduce a cross-attention (without self-attention)  based decoder architecture for neural operators for solving PDEs. The architecture contains an encoder component that has the boundary defined by a triangular mesh as an input. The mesh encoder uses graph convolutions on the edges with uniform 4-nearest neighbour architecture across  the triangular mesh to define the features of the boundaries, the decoder attention layers propagate the information to create the PDE solution in the interior domain. The decoder uses ReLU, the transformer SiLU non-linearities. The latter to make sure the automatic differentiation for the PDE solution is smooth enough everywhere.\n\nThe system is trained with 12000 example meshes (geometries), validated by FEM solutions. The lowest validation error parameters from each epoch is used.  The methods is demonstrated for a set of simple geometries and equations with mostly encouraging results.\n\nThe capability of solving PDEs with varying boundaries in an efficient was is valuable for many industrial applications, and is an impactful addition to the  physical system simulation toolbox.", "questions": "Poisson  equation solutions have corner singularities in certain geometries. Now, analytically the PDE solution of triangulation based geometric behave badly on all vertices of the boundary mesh. It is possible to solve the problem by smoothing the surface, but this would require a different parametrisation of the surface, for example splines - as often used in industry. How would the current neural operator architecture address this problem?", "limitations": "NA", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The introduction is clear and easy to follow, also  the manuscript brings clearly up the benefits of using PINNS in the industrial domain.\n\nThe neural network operator that can take a triangularisation define boundary as an input for the trained system, and hence allows for solution of gPDEs with geometries using only an inference of a neural network, with convolution and a layers of cross-attention has the potential to provide solution much faster than traditional  FEM solvers.", "weaknesses": "The architecture of the solution, especially its hyper parameters are not described.  Figures showing the components of the architecture are missing, and verbal explanation with references to similar architectures MeshNet and decoder from the original attention is all you need makes it hard to read. As introducing the novel architecture is the main result of the paper, one should concentrate on describing it in simple visual manner that allows a wider audience to grasp the essential.\n\nAlso, regularisation of PDE solutions when geometries are given with triangularisation have singularities. The manuscript should address this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Please see our other response and the attached diagram for an explanation of our architecture. We hope this provides a clearer picture of our model.\n\nIt is true that non-smooth boundaries (eg. sharp points or corners) can lead to PDE solutions which are non-differentiable, discontinuous or even have singularities. Non-smooth behavior is an important consideration for PINNs and they usually perform very poorly in these cases.\n\nThere are at least 2 separate scenarios to consider here. (1) The geometry of interest is smooth, and the triangular mesh is a non-smooth approximation to the geometry, and (2) the geometry of interest does in fact have important non-smooth points such as corners.\n\nIn the first case, we assume the true solution to the PDE is smooth, and so it may be that the PINN doesn’t have any trouble, since it is smooth by design. Essentially, the PINNs inherent smoothness naturally regularizes it. Even so, it may be valuable to reparameterize the boundary geometry with spline patches instead of flat triangles as is done in many FEM applications. The structure of this is still a mesh, but with additional information about the spline curvature. There is nothing to prevent this information from being used as feature information in the input to the MeshCNN. The MeshCNN doesn’t care whether the edges it is given are flat or curved, only the connectivity matters. Of course, during training, boundary and interior points still need to be sampled accordingly.\n\nIn the second case PINNs are generally incapable of exhibiting this kind of non-smooth behavior without specific modifications. This is an important and rich line of research, and there are many ideas that address it; however, it is somewhat orthogonal to the points in our paper. We do mention some of this work in the introduction, but we do not ourselves directly address this point.\n"}]

Author rebuttal:
# Response to all Reviewers
We kindly thank all reviewers for their time and helpful feedback on our paper. All reviewers agree that our description of our model architecture was confusing and unclear. We aim to give a more concrete and precise description below, which we hope will improve the clarity of our paper. We have furthermore included a figure to give a visual representation. We will of course include this in the final version of our paper.

## Network architecture
The model has two inputs: (1) the boundary geometry in the form of a triangular mesh, and (2) a point $x$ at which the function is being evaluated. These inputs are passed into two subnetworks. See the attached pdf for a diagram of this architecture.

The first subnetwork is the MeshCNN, consisting of alternating MeshConv layers and nonlinear activation functions. The triangular mesh is passed into this network as a tensor of shape `(n_edges, n_features)`, along with an integer index array of shape `(n_edges, 4)` representing the adjacency lists of each edge; each edge having 4 neighbors. The output of the MeshCNN is a tensor of shape `(n_edges, d_embedding)` considered as a latent representation of the boundary geometry. MeshConv layers are able to extract local geometric information about each edge and its neighborhood.

The second subnetwork is based on a Transformer decoder architecture. It is this network which is the PINN, i.e. the one which represents the function $\phi$ in the PDE. This is trained as a PINN conditioned on the geometric information encoded by the MeshCNN. The first layer of this subnetwork is a fully-connected feed-forward network with 1 hidden layer. The input to this layer is the point $x$ which is a 3 dimensional vector. The outputs from this first layer are then fed through a sequence of Transformer blocks. Each block consists of a cross attention layer, followed by another fully-connected feed-forward layer, each with residual connections and layer normalization. The cross attention layers are calculated as follows: $h_\mathrm{out} = \operatorname{attn}(K,V,Q)$, where $K$ and $V$ are linear projections of the edge embeddings (the output of the MeshCNN), and $Q$ is a linear projection of the output from the previous hidden layer (a latent embedding of the point $x$). The softmax and weighted average in the attention operation are computed by summing over the edges. The output of the final Transformer block is then linearly projected to obtain $\phi(x)$.
The purpose of the attention layers is to allow the model to condition on relevant geometric information over the entire mesh.

Here we have described the model assuming that a single mesh and a single point are given as inputs. Of course these can be given in batches. During each training step, we pass to the model a single mesh, and a batch of points sampled within the domain of the PDE. In analogy to typical sequence models, one might say that these points are a batch of length 1 sequences, whilst a single mesh would be analogous to a single sequence (a batch of size 1). In this way, attention weights are computed between each point-edge pair. These weights are only averaged over the edges (the sequence dimension) and not over points (the batch dimension). For each point, the network conditions on information from each edge, but information is not shared between points. If we were to allow information to be shared across the batch dimension, it would break underlying assumptions of the system being solved. For example, consider two points $x$, $y$ in the domain. We have that $\frac{\partial x}{\partial y} = 0$, since these are independent points. However, if we allow information to be shared across the batch dimension, we would have $0 \ne \frac{\partial \phi(x)}{\partial y} = \frac{\partial\phi(x)}{\partial x} \frac{\partial x}{\partial y} = 0$, which is nonsensical. It is of course possible to build a model which attends over many points in the domain; however, this type of model would need to be trained differently to correct for this mixing.


Meta-review: The reviews are somewhat conflicting for this paper, but the AC gave it a read and decided to suggest accepting this work.  The idea of learning a PDE solution as a function of boundary mesh geometry is a creative and interesting one, so the AC is willing to overlook some weaknesses (most importantly, limited stress testing).

In the final revision, please address the reviewer comments, which were thoughtful and reasonable.  In particular, the exposition needs to be revised to include clear descriptions of the input/output, architecture, and hyperparameters.  Optionally, "stress-testing" by trying more complicated boundary geometries or PDEs would place the paper on more rigorous footing.
Decision: Accept (poster)