Geometry-Informed Neural Operator for Large-Scale 3D PDEs
Authors: Zongyi Li
Nikola Borislavov Kovachki
Chris Choy
Boyi Li
Jean Kossaifi
Shourya Prakash Otta
Mohammad Amin Nabian
Maximilian Stadler
Christian Hundt
Kamyar Azizzadenesheli
Anima Anandkumar
Venue: NeurIPS 2023 poster (2023)
Keywords: partial differential equation
computational fluid dynamics
neural operator
TL;DR: We propose geometry-informed neural operator, an efficient neural operator-based method for learning the solution operator of 3d large-scale partial differential equations with varying geometries.

Abstract: We propose the geometry-informed neural operator (GINO), a highly efficient approach for learning the solution operator of large-scale partial differential equations with varying geometries. GINO uses a signed distance function (SDF) representation of the input shape and neural operators based on graph and Fourier architectures to learn the solution operator. The graph neural operator handles irregular grids and transforms them into and from regular latent grids on which Fourier neural operator can be efficiently applied. We provide an efficient implementation of GINO using an optimized hashing approach, which allows efficient learning  in a shared, compressed latent space with reduced computation and memory costs. GINO is discretization-invariant, meaning the trained model can be applied to arbitrary discretizations of the continuous domain and applies to any shape or resolution.  To empirically validate the performance of our method on large-scale simulation, we generate the industry-standard aerodynamics dataset of 3D vehicle geometries with Reynolds numbers as high as five million. For this large-scale 3D fluid simulation, numerical methods are expensive to compute surface pressure. We successfully trained GINO to predict the pressure on car surfaces using only five hundred data points. The cost-accuracy experiments show a 26,000x speed-up compared to optimized GPU-based computational fluid dynamics (CFD) simulators on computing the drag coefficient. When tested on new combinations of geometries and boundary conditions (inlet velocities), GINO obtains a one-fourth reduction in error rate compared to deep neural network approaches.

Geometry-Informed Neural Operator for Large-Scale 3D PDEs
Zongyi Li
Nikola Borislavov Kovachki
Chris Choy
Boyi Li
Jean Kossaifi
Shourya Prakash Otta
Mohammad Amin Nabian
Maximilian Stadler
Christian Hundt
Kamyar Azizzadenesheli
Anima Anandkumar
NVIDIA
Abstract
We propose the geometry-informed neural operator (GINO), highly efficient approach for learning the solution operator of large-scale partial differential equa- tions with varying geometries. GINO uses a signed distance function (SDF) and point-cloud representations of the input shape and neural operators based on graph and Fourier architectures to learn the solution operator. The graph neural operator handles irregular grids and transforms them into and from regular latent grids on which Fourier neural operator can be efficiently applied. GINO discretization- convergent, meaning the trained model can be applied to arbitrary discretizations the continuous domain and it converges to the continuum operator as the dis- cretization is refined. To empirically validate the performance of our method on large-scale simulation, we generate the industry-standard aerodynamics dataset of 3D vehicle geometries with Reynolds numbers as high five million. For this large-scale 3D fluid simulation, numerical methods are expensive to com- pute surface pressure. We successfully trained GINO predict the pressure car surfaces using only five hundred data points. The cost-accuracy experiments show 26, 000x speed-up compared optimized GPU-based computational fluid dynamics (CFD) simulators on computing the drag coefficient. When tested on new combinations of geometries and boundary conditions (inlet velocities). GINO obtains one-fourth reduction in error rate compared to deep neural network approaches.
1 Introduction
Computational sciences aim understand natural phenomena and develop computational models to study the physical world around us. Many natural phenomena follow the first principles physics and are often described as evolution function spaces, governed by partial differential equations (PDE). Various numerical methods, including finite difference and finite element methods, have been developed computational approaches for solving PDEs. However, these methods need to be run at very high resolutions to capture detailed physics, which are time-consuming and expensive, and often beyond the available computation capacity. For instance, in computational fluid dynamics (CFD), given a shape design, the goal is to solve the Navier- Stokes equation and estimate physical properties such as pressure and velocity. Finding the optimal shape design often requires solving thousands of trial shapes, each of which can take more than ten hours even with GPUs [
To overcome these computational challenges, recent works propose deep learning- methods, particularly neural operators [2], to speed up the simulation and inverse design. Neural operators
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Input Samples
discretize
GNO
FNO
GNO
query
Prediction
Operates on input geometry
Operates (latent) regular geometry
Projects output geometry
MWW
FNO loyers
The architecture of GINO. The input geometries are irregular and change for each sample. These are discretized into point clouds and passed geometry to a latent regular grid. The output of this GNO layer is concatenated with the SDF features and passed into FNO model. The output from the FNO model is projected back onto the domain of the input geometry for each query point using another GNO layer. This is used to predict the target function (e.g., pressure), which is used to compute the loss that is optimized end-to-end for training.
Figure 1:
GNO layer, which maps from the given
generalize neural networks and learn operators, which are mappings between infiniteddimensione function spaces 22. Neural operators are discretization convergent and can approximate general operators 3 The input function to neural operators can be presented at any discretization, grid, resolution, or mesh, and the output function can be evaluated at any arbitrary point. Neural operators have shown promise in learning solution operators in partial differential equations (PDE) [3] with numerous applications in scientific computing, including weather forecasting 44, carbon dioxide storage and reservoir engineering [5], with tremendous speedup over traditional methods. Prior works on neural operators developed a series of principled neural operator architectures to tackle variety of scientific computing applications. Among the neural operators, graph neural operators (GNO) 2, and Fourier neural operators (FNO) 6 have been popular in various applications.
GNO implements kernel integration with graph structures and is applicable to complex geometries and irregular grids. The kernel integration in GNO shares similarities with the message-passing implementation of graph neural networks (GNN) 77. which is also used in scientific computing [8- 10 However, the main difference is that GNO defines the graph connection in a ball defined on the physical space, while GNN typically assumes fixed set of neighbors, c.g., k-nearest neighbors, see Figure 5 Such neirest connectivity in GNN violates discretization convergence, and it degenerates into pointwise operator at high resolutions, leading to poor approximation of the ground-truth operator using GNN. In contrast, GNO adapts the graph based on points within a physical space, allowing for universal approximation of operators. However, one limitation of graph- based methods is the computational complexity when applied to problems with long-range global interactions. To overcome this, prior works propose using multi-pole methods or multi-level graphs to help with global connectivity. However, they do not fully alleviate the problem since they require many such levels capture global dependence, which still makes them expensive.
While GNO performs kernel integration in the physical space using graph operations, FNO leverages Fourier transform to represent the kernel integration in the spectral domain using Fourier modes. This architecture is applicable to general geometries and domains since the (continuous) Fourier transform can be defined on any domain. However, becomes computationally efficient when applied regular input grids since the continuous Fourier transform can then be efficiently approximated using discrete Fast Fourier transform (FFT) [13, giving FNO significant quasi- computational complexity. However, FFT limits FNO to regular grids and cannot directly deal with complex geometries and irregular grids. A recent model, termed GcoFNO, learns deformation from a given geometry to a latent regular grid [14] so that the FFT can be applied in the latent space. In order to transform the latent regular grid back to the irregular physical domain, discrete Fourier transform (DFT) on irregular grids is employed. However, DFT on irregular grids is more expensive than FFT, quadratic vs. quasi- linear, and does not approximate the Fourier transform in discretization convergent manner. This is because, unlike in the regular setting, the points are not sampled at regular intervals, and therefore the integral does not take into account the underlying measure. Other attempts share a similar computational barrier as shown in Table [ which we discussed in Section
In this paper, we consider learning the solution operator for large-scale PDEs, in particular, 3D CFD simulations. We propose the coometry-infommdd neural operator (GINO), neural operator

---

Table 1: Computational complexity of standard deep learning models. N is the number of mesh points; is the dimension of the domain and degree is the maximum degree of the graph. Even though GNO and transformer both work on irregular grids and are discretization convergent, they become too expensive on large-scale problems.
Model
Range
Complexity
Irregular grid Discretization convergent
GNN CNN
local
((Ndggre))
local
(N)
X
UNet
global
(N)
Transformer
global
O(N2)
GNO (kernel)
radius r
( ((Ndege
FNO (FFT)
global
( log N)
GINO [Ours]
global
O log N + Ndegree)
architecture for arbitrary geometries and mesh discretizations. It uses signed distance function (SDF) to represent the geometry and composes GNO and FNO architectures together in a principled manner to exploit the strengths of both frameworks.
The GNO by itself can handle irregular grids through graphs but is able to operate only locally under limited computational budget, while the FNO can capture global interactions, but requires a regular grid. By using GNO to transform the irregular grid into a regular one for the FNO block, we can get the best of both worlds, i.e., computational efficiency and accuracy of the approximation. Thus, this architecture tackles the issue of expensive global integration operations that were unaddressed in prior works, while maintaining discretization convergence.
Specifically, GINO has three main components, (i) Geometry encoder: multiple local kernel integration layers through GNO with graph operations, (ii) Global model: a sequence of FNO layers for global kernel integration, and (éii) Geometry decoder: the final kernel integral layers, as shown in Figure The input to the GINO is the input surface (as a point cloud) along with the SDF, representing the distance of each 3D point to the surface. GINO is trained end-to-end to predict output (e.g., car surface pressure in our experiments), function defined on the geometry surfaces.
Geometry encoder: the first component in the GINO architecture uses the surface (i.e., point cloud) and SDF features as inputs. The irregular grid representation of the surface is encoded through local kernel integration layers implemented with GNOs, consisting of local graphs that can handle different geometries and irregular grids. The encoded function is evaluated on regular grid, which concatenated with the SDF input evaluated on the same grid. Global model: the output of the first component is encoded on a regular grid, enabling efficient learning with an FNO using FFT. Our second component consists of multiple FNO layers for efficient global integration. practice, we find that this step can be performed at lower resolution without significantly impacting accuracy, giving further computational advantage. Geometry decoder: the final component is composed of local GNO-based layers with graph operations, that decode the output of the FNO and project back onto the desired geometry, making it possible to efficiently query the output function on irregular meshes. The GNO layers in our framework are accelerated using our GPU-based hash-table implementation of neighborhood search for graph connectivity of meshes.
We validate our findings two large-scale 3D CFD datasets. We generate our own large-scale industry-standard Ahmed's body geometries using GPU-based OpenFOAM [15,, composed of 500+ car geometries with O(10²) mesh points on the surface and O(10²) mesh points in space. Each simulation takes 7- hours on 16 CPU cores and 2 Nvidia V 100 GPUs. Further, we also study lower resolution dataset with more realistic car shapes, viz., Shape-Net car geometries generated by [6]. GINO takes the point clouds and SDF features as the input and predicts the pressure fields on the surfaces of the vehicles. We perform full costraccuracy trade-off analysis. The result shows GINO is 26, 000x faster at computing the drag coefficients over the GPU- OpenFOAM solver, while achieving 8.31% % (Ahmed-body) and 7.29% (Shape-Net car) error rates on the full pressure field. Further, GINO is capable of zero-shot super-resolution, training with only one-eighth of the mesh points, and having a good accuracy when evaluated on the full mesh that is not seen during training.
3

---

2
Problem setting
We are interested in learning the map from the geometry of a PDE to its solution. We will first give a general framework and then discuss the Navier-Stokes equation in CFD as example. Let D c R be a Lipschitz domain and A Banach space of real-valued functions on D. We consider the set of distance functions T A so that, for each function T T, its zero set Sy {ar E D T(x) = 0} defines a (d - 1) dimeensiona sub-manifold. We assume Sr is simply connected, closed, smooth, and that there exists
0 such that B, n 0D 2 for every by QT C D, the open volume enclosed by the sub-manifold ST and assume that Qr is a Lipschitz domain with 8Qr ST. We define the Lipschitz domain Ω2 == D \ QT so that, 02r 0D U ST. Let L denote a partial differential operator and consider the problem
ST and T T. We denote
((u) = f, in SIT: == 9, in UNT..
(I)
for some E F. B where B, F denote Banach spaces of functions on R& with the assumption that the evaluation functional continuous in B. We assume that L is such that, for any triplet (T', f) the PDE ( has unique solution u E Ur where Up denotes Banach space of functions on Q2. Let U denote a Banach space of functions on D and let {Er : UT - u T E T be a family of extension operators which are linear and bounded. We define the mapping from the distance function to the solution function
V T x F B 4
by f.9) + ET( which is our operator of interest.
Navier-Stokes Equation. We illustrate the above abstract formulation with the following example. Let D = (0, 1)dh be the unit cube and let A = C(D). We take T C A to be some subset such that the zero level set of every element defines a (d 1)-dimensional closed surface which can be realized as the graph of Lipschitz function and that there exists > 0 such that each surface is at least distance e away from the boundary of D. We now consider the steady Naiver-Stoker equations.
~v + (c. V)U + Vp-
in Ωe, in Dr: in OD,
V U = 0,
q.
U 0, in ST,
where ;: 22 + Rd is the velocity, p 12 + IR is the pressure, v/ the viscosity, and f, Rd - Rd are the forcing and boundary functions. The condition that v 0 in Sr is commonly known "no slip" boundary and is prevalent in many engineering applications. The function g, on the other hand, defines the inlet and outlet boundary conditions for the flow. We assume that f E H- H Rd and q E C(Rd; Rd). We can then define our boundary function y E C(Rd; Rd) such that g(x) = 0 for any x E D with dist(xr AD) 2 and g(x) = q(x) for any x E D with, dist(r, @D) > e/2 as well as any x & D. Continuity of g can be ensured by an appropriate extension for any x E D such that dist(z, @D) < and dist x, aD) 2 e/2 [17]. We define u 22r + IR+11 by = (v, as the unique weak solution of (3) with Ur = H (2r;; x (22 /R [18]. We define U = H (D: x (D)RR and the family of extension operators (Er 24 + U1 by Ey (u) = (E) (E))).
where E.. H (Ωy; [T: IRd) + H IRd)) and ET. T,2 (I2r)) /TR - (D) are defined as the restriction onto D of the extension operators defined [19 Chapter VI, Theorem 5]. This establishes
the existence of the operator v : T x HI (IR;; Rd xd) C(R:; Rd) + ID;; x2)xx (DD) /D mapping
x the geometry, forcing, and boundary condition the (extended) solution of the steady Navier-Stokes equation 3 Homomorphic extensions of deformation based operators have been shown in (20]. We
leave for future work studying the regularity properties of the presently defined operator.
3 Geometric-Informe Neural Operator
We propose a geometry-infowe informed neural operator (GINO), neural operator architecture for varying geometries and mesh regularities. GINO is deep neural operator model consisting of three main components, (i) multiple local kernel integration layers, (M) sequence FNO layers for global kernel integration which precedes (iii) the final kernel integral layers. Each layer of GINO follows the form of generic kernel integral of the form (5). Local integration computed using graphs, while global integration is done in Fourier space.

---

3.1 Neural operator
neural operator 33] maps the input functioon a a= (T, () to the solution function u. The neural operator v/r is composed of multiple layers of point-wise and integral operators,
V Q o Kr .... K o P.
(4)
The first layer is pointwise operator parameterized by a neural network. transforms the input function a into a higher-dimensional latent space P a + Vi. Similarly, the last layer acts as projection layer, which is pointwise operator Q it H u, parameterized by neural network Q. The model consists of L layers of integral operators K2 2 + in in between.
v)(x)) = J, (2 v))
(5)
where is learnable kernel function. Non-linear activation functions are incorporated between each layer.
3.2 Graph operator block
To efficiently compute the integral in equation ()) we truncate the integral local ball at with radius r >> 0, as done in [2].
J
vr (x)
<(x,, ()) dy-
(6)
(x)
We discretize the space and use Riemann sum to compute the integral. This process involves uniformly sampling the input mesh points and connecting them with a graph for efficient parallel computation. Specifically, for each point D, randomly sample points (y1,-.., (us) (x)
and approximate equation (6 as
M
(1 ~
Σ kkl, (
where y denotes the Riemannian sum weights corresponding to the ambient space of (x).. For a fixed input mesh of N points, the computational cost of equation (7) scales with the number of edges, denoted as (F.) = O(M (MN). Here, the number of sampling points M is the degree of the graph. can be either fixed to a constant sampling size, scale with the area of the ball.
Encoder. Given an input point cloud {.f...., xW C ST, we employ a GNO-encoder to transform
it to a function on uniform latent grid {-1 grid
...., x.S gndd C. D. The encoder is computed as discretization of an integral operator (n Ei= k(22rrid .u(" y("")((("") over ball Br,, (2:rd))) To weights y(v!"). Further, we use Fourier
inform the grid density, GINO computes Riemannian features in the kernel [21]. For simple geometries, this encoder can be omitted, see Section 4
Decoder. Similarly, given function defined on the uniform latent grid { ....., L C D. we use GNO-dccoder to query arbitrary output points {-TTh ..... N C Ωr. The output evaluated grid
as u( xe ΣM ,6 .vt oy (u. uridl (u over ball Br.. (u. Here, the Riemannian weight, (( 1/S since we choose the latent space be regular grid. Since the queries are independent, we divide the output points into small batches and run them in parallel, which enables us to use much larger models by saving memory.
Efficient graph construction. The graph construction requires finding neighbors to each node that are within a certain radius. The simplest solution is to compute all possible distances between neighbors, which requires (NN)) computation and memory. However, as the N gets larger, e.g., 10 100 million, computation and memory become prohibitive even on modern GPUs. Instead, we use hash grid-based implementation to efficiently prune candidates that are outside of a ball first and then compute the 8² distance between only the candidates that survive. This reduces the computational complexity to O(Ndr3) where d denotes unit density and r is the radius. This can be efficiently done using first creating hash table of voxels with size r. Then, for each node, we go over all immediate neighbors to the current voxel that the current node falls into and compute the distance between all points these neighboring voxels. Specifically, we use the CUDA implementation from Open3D 22 Then, using the neighbors, we compute the kernel integration using gather-scatter operations from torch-scatter
(33). Further, the degree of the graph gets larger, we can add Nyström approximation by sampling nodes (2).
5

---

Ground-truth pressure
pressure
error
-187.
5
Figure 2: Visualization of ground-truth pressure and corresponding prediction by GINO from the Shape-Net Car (top) and Ahmed-bout -bdd (bottom) datasets, as well as the absolute error.
3.3 Fourier operator block
The geometry encoding Vo and the geometry specifying map T, both evaluated on regular grid discretizing D are passed to a FNO block. We describe the basic FNO block as first outlined in [6]. We will first define global convolution in the Fourier space and it to build the full FNO operator block. To that end, we will work on the d-dimensional unit torus T We define an integral operator with kernel T.2 (TT, R=xm) the mapping C T.2 T.2 Rm) (TM, R") given by
C(v)=
F(w)),
VUE L'(
Here F,F-1 are the Fourier transform and its inverse respectively, defined for 1,2 by the appropriate limiting procedure. The Fourier transform of the function will be parameterized directly by some fixed number of Fourier modes, denoted a E particular, we assume
<(x))
Σ Cettta NET
for some index set I C Zd with [ = a and coefficients by E C"xm Then we may view F: (TT, R= + ( so that F(n)(n) = c. if E I and F(x)(1) = if & We directly learn the coefficients C, without ever having to evaluate in physical space. We then L (TTTT) by
define the full operator block K. (Td,
[())(x)= - (W o(x) C(u)), VIETd
where is pointwise non-linearity and W Rxm is learnable matrix. We further modify the layer by learning the kernel coefficients in tensorized form, adding skip connections, normalization layers, and learnable activations outlined in (24] We refer the reader to this work for further details.
Adaptive instance normalization. For many engineering problems of interest, the boundary in- formation a fixed, scalar, inlet velocity specified on some portion of OD. In order to efficiently incorporate this scalar information into our architecture, we use learnable adaptive instance nor- malization (25 combined with Fourier feature embedding 21.. In particular, the scalar velocity embedded into vector with Fourier features. This vector then goes through a learnable MLP, which outputs scale and shift parameters of an instance normalization layer (26). In problems where the velocity information is not fixed, we replace the normalization layers of the FNO blocks with this adaptive normalization. We find this technique improves performance, since the magnitude of the output fields usually strongly depends on the magnitude of the inlet velocity.

---

Table 2: Shape-Nett Car dataset (3.7k mesh points).
Model
training error
test error
GNO
18%
18.77%
Geo-FNO (sphere)
10.79%
15.85%
UNet (interp)
12.48%
12.83%
FNO (interp)
9.65%
9.42%
GINO (encoder-decoder)
7.95%
9.47%
GINO (decoder)
6.37%
7.12%
We do benchmark study with several standard machine-learning methods on the Shape-Net and Ahmed body datasets. The training error is normalized L2 error; the test error is de-normalized L2.
Experiments
We explore a range of models on two CFD datasets. The large-scale Ahmed-Body dataset, which we generated, and also the Shape-Net Car dataset from [16. Both datasets contain simulations of the -yynold-Aeerged Navier-Stokes (RANS) equations for chosen turbulence model. The goal estimate the full pressure field given the shape of the vehicle as input. We consider GNO [2]. MeshGraphNet [9]. GeoFNO [14] 3D UNet [27] with linear interpolation, FNO [6], and GINO. We train each model for 100 epochs with Adam optimizer and step learning rate scheduler. The implementation details can be found in the Appendix. All models run on single Nvidia V100 GPU.
4.1 Ahmed-Body dataset
We generate the industry-level vehicle aerodynamics simulation based on the Ahmed-body shapes 28). The shapes are parameterized with six design parameters: length, width, height, ground clearance, slant angle, and fillet radius. We also vary the inlet velocity from 10m/s to 70m/s, leading to Reynolds numbers ranging from 4.35 x 10 to 6.82 10t. We use the GPU-accelerated OpenFOAM solver for steady state simulation using the SST - w turbulence model [29] with 7.2 million mesh points in total with 100k mesh points the surface. Each simulation takes 7-19 hours 2 Nvidia v100 GPUs with 16 CPU cores. We generate 551 shapes total and divide them into 500 for training and 51 for validation.
4.2
Shape-Net Car dataset
We also consider the Car dataset generated by (16]. The input shapes are from the ShapeNet Car category [30]. In [16), the shapes are manually modified to remove the side mirrors, spoilers, and tires. The RANS equations with the k turbulence model and SUPG stabilization are simulated to obtain the time-averaged velocity and pressure fields using a finite element solver [31]. The inlet velocity fixed 20m/s (72km/h) and the estimated Reynolds number is 10 Each simulation takes approximately 50 minutes. The car surfaces are stored with 3.7k mesh points. We take the 61 wighr-tight shapes out of the 889 instances, and divide the 611 instances into 500 for training and 111 for validation.
As shown in Table 2 and Figure 2 GINO achieves the best error rate with large margin compared with previous methods. On the Ahmed-bood dataset, GINO achieves 8.31 % while the previous best method achieve 11 11.16%. On the Shape-Net Car, GINO achieves 7.12% error rate compared 9.42% on FNO. It takes 0.1 seconds to evaluate, which is 100,000x faster than the GPU-parallel OpenFOAM solver that take 10 hours to generates the data. We further performance a full cost-accuracy analysis in the following section.
For ablations, we consider channel dimensions [32, 48, 64, 80]. latent space [32, 48, 64, 80), and radius from 0,025 to 0.055 (with the domain size normalized to [-1, IJ). As depicted in Figure ((a) and Table larger latent spaces and radii yield superior performance.
4.3 Cost-accuracy analysis on drag coefficient
To compare the performance of our model against the industry-standard OpenFOAM solver, we perform a full cost-accuracy trade-off analysis on computing the drag coefficient, which is the standard design objective vehicles and airfoils. The result shows GINO 26,000x faster at computing the drag coefficients. Figure 3(b) below shows the cost-accuracy curve, measured in terms of inference
7

---

Table 3: Ahmed-body dataset (100k mesh points).
Model
training error
test error
MeshGraphNet
9.08%
13.88%
UNet (interp)
9.93%
11.16%
ENO (interp)
12.97%
12.59%
GINO (encoder-decoder)
9.36%
9.01%%
GINO (decoder)
9.34%
8.31% %
Previous works such as GNO and Geo-FNO cannot scale to large meshes with 100k points. We instead add the MeshGraphNet for graph comparison. Again, for UNet, FNO, and GINO, we fix the latent grid to 64 x x 64. The training error normalized L2; the test error is de-normalized L2.
0.310
The Srag
Ofe-fOAM (drag loss)
Filte rd drag
Anaaage Rulative Eirur in Divig
7.0060
0.905
Bulse diig
relative
0.055
comcere
0.050
0.300
0.045
0.000
0.035
0.200
0.000
0.025
10a
-07
10
10f
1ime (eecondu)
Time Cseconosi
(a) Example drag coefficient computed after every iteration of the OpenFOAM solver. The last point
(b) Cost-accuracy trade-off curve for OpenFOAM vs.
GINO. Increasing cost of GINO is achieved by increas- ing the size of the latent space. Loss function models on the red line includes only error in pressure and wall shear stress, while, for models on orange line, the loss also includes drag.
reference drag corresponds the filtered signal. Triangle indicates the the time when the solver reaches 3% relative error with respect to the reference drag.
Figure 3: Cost-accuracy trade-off analysis for the drag coefficient.
time needed for a relative error in the drag coefficient for GINO and OpenFOAM. The detailed setup discussed in the appendix.
4.4 Discretization-comverennee and ablation studies
We investigate discretizationncovvereneee by varying different parts of GINO. Specifically, we vary the latent grid resolution and the sampling rates for input-output meshes. In these experiments, fixed the training and test samples to be the same, i.e., same latent grid resolution or sampling rate, but varied the shape and input conditions.
cisereena wrt the latent grid. Here, each model trained and tested on (the same) latent resolutions, specifically 32, 48, 64, 80, and 88, and the architecture is the same. As depicted Figure ((a) GINO demonstrates a comparable error rate across all resolutions. A minor improvement in errors is observed when employing a larger latent space. Conversely, the errors associated with the UNet model grow as resolution is decreased due to the decreasing receptive field of its local convolution kernels.
Discretization-convers
in the input-output mesh. Here, GINO is trained and tested with sub-sampled input-output meshes at various sampling rates (2x, 4x, 6x, 8x). As illustrated in Figure 4(b) GINO exhibits consistent error rate across all sampling rates. slight increase in errors is observed on coarser meshes.
Zero-shot super-resolution. GINO possesses the ability to perform zero-shot super-resolution.
The model is trained on a coarse dataset, sub-sampled by 2x, 4x, 6x, and 8x, and subsequently tested on the full mesh, that is not seen during training. The emoor remains consistent across all sampling rates [c)) This characteristic enables the model to be trained at coarse resolution when the mesh dense, consequently reducing the computational requirements.

---

-GINO (enc-dec) fixed degres
0.2
0.2
relative 12 error
0.4
GINO (dec) fixed degree
GINO amncoder-decoder
GINO (encoder-decocer)
GINO (dec) fixed radius
0.15
0.15
0.3
UNet
0.1
0.1
0.2
0.1
0.05
0.05
48
64
80
4
2
8
latent grid resolution
mesh sampling rate
mesh sampling rate
(a) Varying resolutions of the latent grid (same resolution for training and testing).
(b) Varying the sampling rates of the input-output mesh (same rate for training and testing).
(c) Train with a low sampling rate and test on full mesh (zero-shot super- resolution).
Figure 4: Discretization-convrgeen studies and zero-shot super-resolution.
5 Related Work
The study of neural operators and their extended applications in learning solution operators in PDE has been gaining momentum B 34. A method that stands out is FNO, which uses Fourier transform [6] The FNO and its variations have proven to highly accelerate the simulations for large-scale flow problems. including weather forecasting [4), seismology [35 36) and multi-phase flow (5). However, a challenge with the FNO is that its computation superiority is gained when applied a regular grid, where the Fourier transform is approximated using FFT. Therefore, its reliance on FFT limits its use with irregular grids or complex geometries. There have been attempts to modify the FNO work with these irregular structures, but scalability to large-scale 3D PDEs remains an issue. One such attempt is GeoFNO, which learns a coordinate transformation to map irregular inputs regular latent space 14. This method, while innovative, requires a geometric discrete Fourier transform, which is computationally demanding and lacks discretization insurance. To circumvent this, GINO limits the Fourier transform to a local GNO to improve efficiency. The locality is defined assuming the metrics of the physical space.
Additionally, the Non- -ou-squisp Fourier neural solvers (NFS) merge the FNO with non-equispaced interpolation layers, a method similar to global GNO 33 However, at the architecture level, their method replaces the integration of GNO with the summation of the nearest neighbor points on the graph. This step transitions this method to neural network, failing to deliver a discretization convergent approach. The Domain-Agnostic Fourier Neural Operators (DAFNO) represents another attempt improvement, applying an FNO to inputs where the geometry is represented as in- dicator function [38]. However, this method lacks a strategy for handling irregular point clouds. Simultaneously, researchers exploring the combination of FNO with the attention mechanisms [ for irregular meshes. This includes the Operator Transformer (OFormer) [39], Mesh- -ndepennddn Neural Operator (MINO) [40]. and the General Neural Operator Transformer (GNOT) [41). Besides, the Clifford neural layers use the Clifford algebra to compute multivectors, which provides Clifford-FNe implementations
an extension of FNO. The work 43 uses physics-informed loss with U-Net for 3D channel flow simulation with several shapes. The work 44 innovatively proposes the use of multigrid training for neural networks that improves the convergence. Although these methods incorporate attention layers, which are special types of kernel integration
3 with quadratic
complexity, they face challenges when scaling up for large-scale problems.
GNNs are incorporated in the prior attempts in physical simulations involving complex geometry, primarily due to the inherent flexibility of graph structures. Early research 7
45 laid the foundation for GNNs, demonstrating that physical entities, when represented as graph nodes, and their interactions, as edges, could predict the dynamics of various systems. The introduction of graph element networks [48) marked significant development, being the first to apply GNNs to PDEs by discretizing the domain into elements. The similar idea has been explored in term of continuous 3D point cloud convolution 49 [1]. Another line of work, mesh graph networks 8 further explored PDEs in the context of fluid and solid mechanics. [2 533 train a Graph convolutional neural works on the ShapeNet car dataset for inverse design. However, GNN architectures' limitations hinder their use in operator learning for PDEs. GNNs connect each node to its nearest neighbors according to the graph's 's metries, not the metrics of the physical domain. As the input function's 's discretization becomes finer, each node's 's nearest neighbors eventually converge to the same node, contradicting the expectation of improved model performance with finer discretization. Furthermore, GNNs'

---

Input geometries
Discretized geometries at various fine to coarse levels of discretizations
Coarser
discretization
discrretzztionn
(a) An imput geometry (continuous function) is first discretized into series of points by subsampling it. Note that in practice, the discretization applications that we want a method that can work on arbitrary geometries, but also that is discretization convergent, meaning that the method converges to desired solution operator as we make the discretization finer.
be highly irregular. key challenge with several scientific computing
Finer discetization
discetization
Target
Geometry
Geometry
(b) GNN connects each point in the latent subspace (red) to its nearest neighbors the original space (top). This is very discretization dependent, and as we increase the resolution (sample points more densely), the method becomes increasingly local and fails capture context. addition, operator at dis- cretization limit is non-unique and depends on how the discretization is done.
GNO instead connects each point in the latent subspace (red) to all its neighbors within an epsilon ball the original space (top). This induces con-
vergence to continuum solution operator we in- the resolution (sample points means GNO converges to unique operator as the discretization becomes finer and scales to large problems without becoming overly local.
densely).
Figure 5: Comparison of GNN and GNO as the discretization becomes finer. GNN is discretization dependent, while GNO is discretization convergent.
model behavior at the continuous function limit lacks unique definition, failing the discretization convergence criterion. Consequently, as pointwise operators in function spaces at the continuous limit, GNNs struggle to approximate general operators between function spaces, Figure 5
6 Conclusion
In this work, we propose the GINO model for 3D PDEs with complex geometries. The GINO model consists of the graph-kems blocks for the encoder and decoder that go latent uniform space, where the Fourier blocks run on the latent space to capture the global interaction. We experiment on two CFD datasets: Shape-Net car geometries and large-scale Ahmed's body geometries, latter encompassing over 600 car geometries featuring hundreds of thousands of mesh points. The evidence from these case studies illustrates that our method offers substantial speed improvement, with factor of 100,000 times acceleration in comparison the GPU-based OpenFOAM solver. Concurrently, our approach has achieved one-fourth to one-half the error rates compared to prevailing neural networks such as 3D U-Net. This underscores the potential of our method to significantly enhance computational efficiency while maintaining a competitive level accuracy within the realm of CFD applications. Limitation: The trained surrogate model is limited specific category of shapes. The quality of the model depends on the quality the training dataset. For CFD with more complex shapes, it is not easy obtain large training dataset. We will explore physics-informed approaches (54) and generate time-dependent high-fidelity simulations in the future.
10

---

Acknowledgments and Disclosure of Funding
ZL. is supported by the Nvidia fellowship. NBK is grateful to the NVIDIA Corporation for support through full-time employment.
References
[1] Ashley M Korzun, Gabriel Nastac, Aaron Walden, Eric J Nielsen, William T Jones, and Patrick Moran. Application of detached eddy simulation approach with finite-rate chemistry to mars-relevant retropropulsion operating environments. In AIAA SciTech 2022 Forum, page 2298, 2022.
[2] Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar, Neural operator: Graph kernel network for partial differential equations. arXiv preprint arXiv:2003.034:0 2020.
[3] Nikola Kovachki, Zongyi Li, Burigede Liu, Kamyar Azizzadenesheli, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Neural operator: Learning maps between function spaces, arXiv preprint arXiv:2108.0848) 2021.
[4] Jaideep Pathak, Shashank Subramanian, Peter Harrington, Sanjeev Raja, Ashesh Chattopadhyay, Morteza Mardani, Thorsten Kurth, David Hall, Zongyi Li, Kamyar Azizzadenesheli, al. Fourcastnet: A global data-driven high-resolution weather model using adaptive fourier neural operators. arXiv preprint arXiv:2202. 1214, 2022.
[5] Gege Wen, Zongyi Li, Qirui Long, Kamyar Arizzadenesheli, Anima Anandkumar, and Sally M Benson. Real-time high-resolution co 2 geological storage prediction using nested fourier neural operators. Energy Environmental Science, 173--1173,- 2023.
[6] Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differen- tial equations, arXiv preprint arXiv:2010.088/0 2020.
[7] Peter Battaglia, Razvan Pascanu, Matthew Lai, Danilo Jimenez Rezende, et al. Interaction networks for learning about objects, relations and physics. Advances in neural information processing systems, 29, 2016.
[8] Alvaro Sanzhez-Conzzlez,, Jonathan Godwin, Tobias Pfaff, Rex Ying, Jure Leskovec, and Peter Battaglia. Learning to simulate complex physics with graph networks. In International conference on machine learning, pages 8459-8468. PMLR, 2020.
[9] Tobias Pfaff, Meire Fortunato, Alvaro Sanchez- Gonzalez, and Peter W Battaglia. Learning mesh-based simulation with graph networks. arXiv preprint arXiv:2010.03409, 2020.
[10] Kelsey Allen, Tatiana
Guevara, Kimberly Stachenfeld, Alvaro Sanchez Peter Battaglia, Jessica Hamrick, and Tobias Pfaff. Physical design using differentiable learned simulators. arviv preprint arXiv:2202.0072. 007288 2022.
Sanchez-Gonzale Gonzalez,
[11] Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Andrew Stuart, Kaushik Bhattacharya, and Anima Anandkumar. Multipole graph neural operator for parametric partial differential equations. Advances in Neural Information Processing Systems, 33, 2020.
[12] Remi Lam, Alvaro Sanchez-Gonzabee Matthew Willson, Peter Wirnsberger, Meire Fortunato, Alexander Pritzel, Suman Ravuri, Timo Ewalds, Ferran Alet, Zach Eaton-Rosen et al. Graphcast: Learning skillful medium-range maangee global weather forecasting. arXiv preprint arXiv:2212.12794. 2022.
[13] James W Cooley and John W Tukey. An algorithm for the machine calculation of complex fourier series. Mathematics of computation, 19(90):297-30, 1965.
[14] Zongyi Li, Daniel Zhengyu Huang, Burigede Liu, and Anima Anandkumar. Fourier neural oper- ator with learned deformations for pdes on general geometries. arXiv preprint 2rXiv:2207.05209 2022.
11

---

[15] Hrvoje Jasak, Aleksandar Jemcov, Zeljko Tukovic, et al. Openfoam: A c++ library for complex physics simulations. In International workshop on coupled methods in numerical dynamics. volume 1000, pages 1-20, 2007.
[16] Nobuyuki Umetani and Bernd Bickel. Learning three-dimensional flow for interactive aerody- namic design. ACM Transactions on Graphics (TOG), 37(4):1-10, 2018.
[17] Hassler Whitney, Analytic extensions of differentiable functions defined closed sets. Trans- actions of the American Mathematical Society, 36(1):63-89, 1934.
[18] R. Temam. Navier-Stokes Equations: Theory and Numerical Analysis. Studies in mathematics and its applications. North-Hollande 1984.
[19] Elias M. Stein. Singular Integrals and Differentiability Properties of Functions. Princeton University Press, 1970.
[20] Albert Cohen, Christoph Schwab, and Jakob Zech. Shape holomorphy of the stationary navier- stokes equations. SIAM Journal on Mathematical Analysis, 50(2):1200-1752 2018.
[21] Vincent Sitzmann, Julien Martel, Alexander Bergman, David I Lindell, and Gordon Wetzstein, Im- plicit neural representations with periodic activation functions. Advances in Neural Information Processing Systems, 33:7462-7473, 2020.
[22] Qian-Y, Yi Zhou, Jaesik Park, and Vladlen Koltun. Open3D: modern library for 3D data processing,
2018.
[23] Matthias Fey et al. Pytorch extension library of optimized scatter operations, 2023. URL https:///ithhut wtt.t//wwww///ttt///////ooorosaa
[24] Jean Kossaifi, Nikola Kovachki, Kamyar Avizzadenesheli, and Anima Anandkumar. Multi-grid tensorized fourier neural operator for high-resolution pdes, 2023.
[25] Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normalization. CoRR,
2017.
[26] Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky, Instance normalization: The missing ingredient for fast stylization. arXiv preprint arXiv:1607.08000 2016.
[27] Adrian Wolny, Lorenzo Cerrone, Athul Vijayan, Rachele Tofanelli, Amaya Vilches Barro, Marion Louveaux, Christian Wenzl, Sören Strauss, David Wilson-Sánchhr Rena Lymbouridou, Susanne Steigleder, Constantin Pape, Alberto Bailoni, Salva Duran-Nebreda, George W Bassel, Jan Lohmann, Miltos Tsiantis, Fred A Hamprecht, Kay Schneitz, Alexis Maizel, and Anna Kreshuk. Accurate and versatile 3d segmentation of plant tissues at cellular resolution. eLife, 9:e57613, jul 2020. ISSN 2050-084X. doi: 10. 10.7554/eLifc.57613 URL https:////oi org/10. 7554/ eLife. 57613
[28] Syed Ahmed, G Ramm, and Gunter Faltin. Some salient features of the time-averaged ground vehicle wake. SAE Iransactions, pages 473-503, 1984.
[29] Florianr Menter. Zonal equation kw turbulence models for aerodynamic flows. In 23rd fluid dynamics, plasmadynamics, and lasers conference, page 2906, 1993.
[30] Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li. Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information -ich 3d model repository. arXiv preprint arXiv: 151.::55120011.. 2015.
[31] Olek C Zienkiewicz, Robert Leroy Taylor, and Perumal Nithiarasu. The finite element method for fluid dynamics. Enneewoort--Heieemane 2013.
[32] Kaushik Bhattacharya, Bamdad Hosseini, Nikola B Kovachki, and Andrew M Stuart. Model reduction and neural networks for parametric pdes. The SMAI journal of computational mathematics. 7:121-157,202
12

---

[33] Georgios Kissas, Jacob H Seidman, Leonardo Ferreira Guilhoto, Victor M Preciado, George J Pappas, and Paris Perdikaris, Learning operators with coupled attention. Journal of Machine Learning Research, 23(215):1 2022.
[34] Md Ashiqur Rahman, Zachary E Ross, and Kamyar Azizzadenesheli. U-no: U-shaped neural operators. arXiv preprint arXiv:2204.11127, 2022.
[35] Yan Yang, Angela Gao, Jorge C Castellanos, Zachary Ross, Kamyar Azizzadenesheli, and Robert W Clayton. Seismic wave propagation and inversion with neural operators. The Seismic Record, 1(3): 126-134, 2021.
[36] Hongyu Sun, Yan Yang, Kamyar Azizzadenesheli, Robert W Clayton, and Zachary E Ross. Accelerating time-reversal imaging with neural operators for real-time earthquake locations. arXiv preprint arXiv:2210.0663 2022.
[37] Haitao Lin, Lirong Wu, Yongjie Xu, Yufei Huang, Siyuan Li, Guojiang Zhao, Li Cari, et al. Non-equispaced fourier neural solvers for pdes, arXiv preprinl arXiv:2212.0468) 2022.
[38] Ning Liu, Siavash Jafarzadch, and Yue Yu. Domain agnostic fourier neural operators. arXiv preprint 2305::2305.00478, 2023.
[39] Zijie Li, Kazem Meidani, and Amir Barati Farimani. Transformer for partial differential equations' operator learning. arXiv preprint arXiv:2205.13677 2022.
[40] Seungjun Lee. Mesh-independent operator learning for partial differential equations. ICML 2022 2nd for Science Workshop.
[41] Zhongkai Hao, Chengyang Ying. Zhengyi Wang, Hang Su, Yinpeng Dong, Songming Liu, Ze Cheng, Jun Zhu, and Jian Song. Gnot: A general neural operator transformer for operator learning. arXiv preprint 2rXiv:2302.143377 2023.
[42] Johannes Brandstetter. Rianne van den Berg, Max Welling, and Jayesh K Gupta. Clifford neural layers for pde modeling. arXiv preprint arXiv:2209.0493 2022.
[43] Nils Wandel, Michael Weinmann, and Reinhard Klein. Teaching the incompressible navier- stokes equations to fast neural surrogate models in three dimensions. Physics of Fluids, 33(4), 2021.
[44] Aditya Balu, Sergio Botelho, Biswajit Khara, Vinay Rao, Soumik Sarkar, Chinmay Hegde, Adarsh Krishnamurthy, Santi Adavani, and Baskar Ganapathysuuramaanan Distributed multi- grid neural solvers on megavoxel domains. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1-14, 2021.
[45] Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional 2016.
networks. arXiv preprint
[46] Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning large graphs. Advances in neural information processing systems, 30, 2017.
[47] Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et al. Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv:1806.01266 2018.
[48] Ferran Alet, Adarsh Keshav Jeewajee, Maria Bauza Villalonga, Alberto Rodriguez, Tomas Lozano-Perez, and Leslie Kaelbling. Graph element networks: adaptive, structured computation and memory. In International Conference on Machine Learning, pages 212-222. PMLR, 2019.
[49] Yangyan Li, Rui Bu, Mingchao Sun, Wei Wu, Xinhan Di, and Baoquan Chen. Pointenn: Convolution on x-transformed points. Advances in neural information processing systems, 31, 2018.
[50] Pedro Hermosilla, Tobias Ritschel, Pere-Pau Vázquez, Älvar Vinacua, and Timo Ropinski. Monte earlo convolution for learning on non-uniformly sampled point clouds. ACM Transactions on Graphics (TOG), 37(6):1112, 2018
13

---

[51] Wenxuan Wu, Zhongang Qi, and Li Fuxin. Pointconv: Deep convolutional networks on 3d point clouds. Proceedings of the IEEE/CVE Conference on computer vision and pattern recognition, pages 9621-9630 2019.
[52] Edoardo Remelli, Artem Lukoianov, Stephan Richter, Benoit Guillard, Timur Bagautdinov, Pierre Baque, and Pascal Fua. Meshsdf: Differentiable iso-surface extraction. Advances in Neural Information Processing Systems, 53:22468-22443 2020.
[53] Nikita Durasov, Artem Lukoyanov, Jonathan Donier, and Pascal Fua. Debosh: Deep bayesian shape optimization. arXiv preprint arXiv:2109. 021220099.337 2021.
[54] Zongyi Li, Hongkai Zheng, Nikola Kovachki, David Jin, Haoxuan Chen, Burigede Liu, Kamyar Azizzadenesheli, and Anima Anandkumar. Physics-informed neural operator for learning partial differential equations. arXiv preprint arXiv:21 2021.
[55] Matthias Fey and Jan Eric Lenssen. Fast graph representation learning with pytorch geometric. arXiv preprint arXiv: 024482228, 2019.
14

---

7 Appendix
7.1 Experiments and Ablations
Table 4: Ablation the Ahmed-body with different sizes of the latent space
Model
latent resolution
radius
fraining error
fest error
GINO
32
0.055
14. 1%
13.59%
GINO
48
0.055
8,99%
10.20%
GINO
64
0.055
6.00%
8.47%
GINO
80
0.055
5.77%
7.87 %
GINO
0.110
8.66%
10.10%
GINO
48
0,073
7.25%
9.17%
GINO
0.055
6.00%
8.47%
GINO
80
0.044
6.22%
7.89 %
When fixing the radius, larger latent resolutions lead better performance. The gaps become smaller when fixing the number of edges and scaling the radius correspondingly.
Benchmarks. This study analyzes several existing models, including GNO, GeoFNO, 3D UNet, and MeshGraphNet. All models are trained using the Adam optimizer for 100 epochs, with the learning rate halved at the 50th epoch. We consider starting learning rates such as [0.002, 0.001, 0,0005, 0.00025, 0,0001], with the most favorable results attained at the rates 0,00025 and 0.0001. For GeoFNO, 2D spherical latent space is employed due to instability with 3D deformation, leading to faster runtime than other 3D-based models. In the 3D UNet model, we evaluate channel dimensions ranging from [64, 128, 256] and depths from [4, 5, 6]. Utilizing larger model can reduce UNet's error rate to 11.1%.
We added an experiment comparing GINO's performance with GNN. We used the MeshGraphNet 19], which is a common GNN method for physical simulations. The MeshGraphNet has three main parts: an encoder, a decoder, and a processor. Both the encoder and decoder use the same setup, with channel size of 256. The processor has 15 layers of information passing, using edge and node blocks. also has size of 256. In total, the GNN model has about 10M parameters. The total number of edges is around 280k, which saturates single NVIDIA V100 GPU with 32GB of memory. We set the learning rate to be le-4 with an exponential decay with a rate of 0.99985, similar to the original setup in
For the GINO model, consider channel dimensions [32, 48, 64, 80], latent space [32, 48, 64, 80), and radius from 0.025 to 0.055 (with the domain size normalized to [-1, IJ). As depicted in Figure 4(a) and Table larger latent spaces and radii yield superior results.
Table 5: Ablation on the Ahmed-body with different choices of the radius.
Model
radius 0.025
radius 0.035
training error
test error
training error
test error
GINO (encoder-decoder)
12.91% %
13.07%
8.65%
10.32%
GINO (encoder-decoder, weighted)
12.94%
12.76%
9.26%
9.90%
GINO (decoder)
12.62%
12.74%
8.82%
%
The choice of radius is significant. A larger radius leads to better performance for all models.
Encoder and Decoder. For the GINO model, we contemplate two configurations: an encoder- decoder design utilizing GNO layers for both input and output, and a decoder-only design which takes the fixed SDF input directly from the latent space and employs the GNO layer solely for output. When the input mesh significantly exceeds information extraction. However, when the size of the latent grid matches or surpasses the input mesh, an encoder becomes redundant. As depicted Table both the encoder-decoder and decoder-only designs exhibit comparable performance.
latent grid in size, the encoder proves beneficial in
15

---

Parallelism. Data parallelism is incorporated in the GNO decoder. In each batch, the model sub-samples 5,000 mesh points calculate the pressure field. As query points are independent, they can be effortlessly batched. This parallel strategy allows for a larger radius in the decoder GNO. Without a parallel implementation, a radius of 0.025 leads to 300,000 edges, rapidly depleting GPU memory. Yet, with parallelism, the algorithm can handle radius of 0.055. Implementing parallelism in the encoder is left for future exploration.
Weights in the Riemann Sum. As mentioned in the GNO section, the integral approximated as a Riemann sum. In the decoder, the weight su()) is constant, reflecting the uniformity of the latent space. Conversely, in the encoder, weights are determined as the area of the triangle. For increased expressiveness, the weight is also integrated into the kernel, resulting in a kernel of the form a(x, y, μ())). However, it's worth noting that the encoder's significance diminishes when a large latent space is in use.
Sub-sampling and Super-resolution. The computational cost of the models increases rapidly with the number of mesh points. Training models with sub-sampled meshes saves significant models can achieve such super-resolution; they
computational resources. can be trained on coarse mesh points and generalized to fine evaluation mesh. We investigate the super-resolution capabiliiiies of UNet (interp). FNO (interp), and GINO. As demonstrated in Table o GINO maintains consistency across all resolutions. UNet (interp) and FNO (interp) can also adapt to denser test meshes, albeit with marginally higher error rate, based on linear interpolation. The results corroborate GINO's enecretzaationccoverggee nature.
Hash-table- based Graph Construction. For graph construction and kernel integration compu- lation in this work, we utilize the CUDA implementation from Open3D [2] and torch-scatter 233. respectively. This approach is 40% faster compared to previous GNO implementation that constructed the graphs with pairwise distance and used the PyTorch Geometric library These implementations are Incorporated into the GNO encoder and decoder in GINO.
In addition, the CUDA hash based implementation requires less memory footprint O dr compare to the standard pairwise distance which requires O(N2) memory and computation complexity. For 10k points, hash-based implementation requires 6GB of GPU memory while the pairwise method requires 24GB of GPU memory; making the hash-based method more scalable for larger graphs.
Table 6: Super-resolution sub-sampled meshes
Model
Sampling rate
1/2
1/4
1/6
1/8
Unet (interp)
16.5%
13.8%
13.9%
15.6%
FNO (interp)
14.2%
14.1% 1
13.3%
11.5%
GINO (encoder-decoder)
8.8%
9.4%
9.4%
9.7%
7.2 Drag Coefficient Comparison
For many engineering tasks, the goal is often to determine a single quantity of interest from a simulation which can then be used within an overall design process. In the design of automobiles, a sought after quantity is the drag coefficient of the vehicle. Intuitively, it is a number inversely proportional to the efficiency with which vehicle passes through fluid. Engineers are therefore often interested in designing geometries with minimal drag coefficients. For fluid with unit density, the drag coefficient is defined as
/ Tu (x) i(x) dx AZd p(a)(((a)) i(x)) dx + Jun
2
(8)
where an C IR3 is the surface of the car, p RR3 - TR is the pressure, ñ an - @ is the outward unit normal vector of the car surface, i: R + S is the unit direction of the inlet flow, T,, : 8n2 + R33 is the wall shear stress on the surface of the car, v E is the speed of the inlet flow, and area of the smallest rectangle enclosing the front of the car.
the
For our Ahmed-body dataset, we train several GINO models (decoder) to predict the pressure on the surface of the car well as the wall shear stress. Since the inlet flow is always parallel to the z-axis
16

---

Table 7: Design of the Ahmed-body shapes
Parameters
steps
lower bound
upper bound
Length
20
644
1444
Width
10
239
539
Height
5
208
368
Ground Clearance
2.5
30
90
Slant Angle
2.5
0
40
Fillet Radius
2.5
80
120
Velocity
4
10
70
L.e. i(x) = (-1,0,0, we predict only the first component of the wall shear stress. Since our results from Table indicate that varying the size of the latent space has a significant effect on predictive performance, we make this the only hyper-parameter of the model and fix all others. We choose latent resolutions varying from 24 to 86. Furthermore, we consider two different loss functions. The first is simply the average of the relative L errors for pressure and wall shear stress. The second includes in this average the relative error in the drag coefficient computed using equation (8). The drag is always computed from our full-field predictions and is never given as part of the output from the model.
To compare the performance of our model against the perform a full cost-accuracy trade-off analysis. During data generation, we keep track of the drag coefficient predicted by OpenFOAM after every iteration. While the coefficient converges with more iterations, this convergence is not monotone and can often appear quite noisy. This makes computing the error from the raw data not possible. We therefore apply box filter the raw signal compute a filtered version of the drag which acts as smoother. We take as the reference drag, the drag at the last iteration of the filtered signal. To compute the number of iterations takes for the solver to predict a drag coefficient at given relative error, we trace back the predictions from the filtered signal and return the first time at which this prediction incurs the given error with respect to the reference drag. An example of this methodology is shown in Figure 3(a) The errors for our GINO
OpenFOAM solver,
model are computed with respect the true drag coefficient from the last iteration of the solver. This
is because we take as ground truth the pressure and wall shear stress from this last iteration and train our model to predict them.
Figure ]b))shows the cost-accuracy curve, measured in terms of inference time needed for relative error in the drag coefficient for GINO and OpenFOAM. The cost of GINO is computed as the time, averaged over the test set, needed to predict the drag coefficient by running the model. This time includes both data pre-processing (computing the SDF) well the model run-time and the drag calculation given the predicted fields. All models are ran on a single NVIDIA V100 GPU, The cost for OpenFOAM is computed as described in the previous paragraph and is averaged over the test set. The solver is ran on two NVIDIA V100 GPUs in parallel. We observe a four to five order of magnitude speed-up when using GINO. At 3% relative error, we find the speed-up from model which includes drag in the loss to be 26, x00 As we increase the size of the latent space, the cost of GINO grows, however, we observe a plateau in the drag error. This is common in machine learning models as the error from using finite data starts to dominate the approximation error. Furthermore, we use only the size of the latent space as pypae-parameter, keeping the number of learnable parameters fixed. It is interesting to explore further how parametrically scaling the model impacts predictive power.
7.3 Data Generation
industry-standart vehicle aerodynamics simulations are generated in this study, utilizing the Ahmed- body shapes as foundation (28). Examples are illustrated in Figure These shapes are characterized by six design parameters: length, width, height, ground clearance, slant angle, and fillet radius, as outlined in Table 7 In addition to these design parameters, we include the inlet velocity to address a wide variation in Reynolds number. The inlet velocity varies from 10m/s to 70m/s, consequently resulting in Reynolds numbers ranging from 4.35 10 to 6.82 x 105. This varying input adds complexity to the problem. We identify the design points using the Latin hypercube sampling scheme for space filling design of experiments.
17

---

The simulations employ the GPU-accelerated OpenFOAM solver for steady-state analyses, applying the SST & w turbulence model. Consisting of 7.2 million mesh points in total, including 100k surface mesh points, each simulation is run on 2 NVIDIA V100 GPUs and 16 CPU cores, taking between to hours to complete.
For this study, the focus is solely on the prediction of the pressure field. It is our hope that this dataset can be utilized in future research, potentially aiding in full-field simulation of the velocity field, as well as the inverse design.
18

---

034-00
Mayyhaa
01400 400
300
200
Figure 6: Illustrations of the Ahmed-body dataset
19


Reviews:
[{"review_id": "rapygDutj8", "reviewer": "Reviewer_zqvK", "summary": "This paper proposes a geometry-informed neural operator for arbitrary geometry, to facilitate the learning on the solution operator for large-scale 3D CFD simulation.", "questions": "As the authors claim the significant speedup provided by the proposed method. Could the authors report the scale of the parameters of the proposed method, and the comparison to the other related methods? ", "limitations": "As pointed out by the authors, this work is constrained to a specific category and limited to CFD with more complex shapes. It'll be with practical significance if the proposed work could tackle these limitations.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The proposed GINO model applies graph-kernel blocks for the encoder and decoder, for processing features in the latent uniform space, with the Fourier blocks running on the latent space to capture the global interaction. The proposed method provides significant runtime speedup.", "weaknesses": "For the experiment, the authors only perform the evaluation on the car category of ShapeNet dataset. It’ll be more persuasive to have the proposed model evaluate the other categories rather than only a single category.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer zqvK\n\n> **Q1:** For the experiment, the authors only perform the evaluation on the car category of the ShapeNet dataset. It’ll be more persuasive to have the proposed model evaluate the other categories rather than only a single category.\n\n**A1:** Thanks to the reviewer for the comment. We want to first point out that we considered two datasets: one is the existing ShapeNet Car data; the other is the Ahmed body dataset we generated. 3D Aerodynamic simulations are extremely expensive. For the example of the Ahmed body dataset, each simulation takes 7-19 hours on 2 Nvidia v100 GPUs with 16 CPU cores.  It is extremely costly to generate a 3D dataset with multiple shapes. We have been searching through the literature for a very long but could not find another dataset resembling industry standard settings. If the reviewer would point out an existing dataset, we will be happy to try it.\n\n> **Q2:** As the authors claim the significant speedup provided by the proposed method. Could the authors report the scale of the parameters of the proposed method, and the comparison to the other related methods?\n\n**A2:** Thanks for the questions. For the larger Ahmed-body dataset, GINO uses O(100M) parameters which take around 5mins to train per each epoch. We match the FNO and UNet baselines to have a similar model size and training time. \nFor smaller the Car-CFD dataset, we compared it against two other baselines: GeoFNO and GNO. The GeoFNO baseline is a 2D surface model. It's smaller and faster (10M parameters and 1 min training time), while its error is higher than the above 3D models. The GNO also has a smaller number of parameters (10M), but it has more edges that saturate the memory on an NVIDIA V100 GPU.  \n\nTo further support the results, we add two more experiments that compare GINO with the solver and GNNs (MeshGraphNet). When comparing against the solver, we observe a four to five order of magnitude speed-up when using GINO. At a $3\\%$ relative error, we find the speed-up from our model which includes drag in the loss to be $26,000 \\times$. When comparing against GNN, we implemented the model following MeshGraphNet, which consists of an encoder, decoder, and processer. The encoder and decoder have MLPs of channel size 256. The processer is 15 layers of message passing of edge blocks and node blocks. The model has a size of around 10M parameters. The details can be found in the general response. Note that the memory footprint of this model is the same as our 100M parameter model as they both saturate the memory of a single NVIDIA V100 GPU.\n\nAs a conclusion, the proposed model is faster than the solver while more accurate than other ML methods given a similar level of cost."}, {"review_id": "JAe7g02b57", "reviewer": "Reviewer_rW9d", "summary": "This paper proposes a framework to learn a neural operator for large-scale 3D PDEs. The framework uses a well-implemented Graph Neural Operator (GNO) to transform the irregular grid into a regular grid, so that it enables the powerful Fourier Neural Operator (FNO) to work on irregular input data, such as point clouds, in a discretization invariant way. \n\nThe main contribution of the paper is to use GNO to transform large-scale irregular grids (point clouds and SDF) into regular grids, to make the FNO suitable. Based on this, GINO realized 100,000x speed-up compared to GPU-based simulators on large-scale stable CFD problems. The model is proven to be working on datasets with a high level of complexity and realism. The paper also generates two large-scale CDF datasets, which require a large amount of time to simulate and generate, which is valuable.  \n", "questions": "1. What is the difference between your proposed GNO and Continuous Convolution (CtsConv) from “Lagrangian Fluid Simulation with Continuous Convolutions”? It seems there is no obvious difference based on equations. And one of the claimed contributions is a well-implemented GNO, which makes it much more efficient, but CtsConv is also well-implemented based on Open3D and hashing table. \n\n2. What is the performance for much finer latent resolutions such as 128^3 and 256^3? Can we keep increasing the test error by increasing the latent resolution? ", "limitations": "1. As the paper stated, the trained model is limited to a specific observed category of shapes. The training for the operator requires a training dataset of high quality. \n\n2. The proposed framework should be general for various kinds of PDEs, but only stable NS equations are tested. \n\n3. The framework is for 3D PDEs, which could be modified for time-dependent ones. ", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. Clear written about the problem setting and equations and symbols. \n\n2. The main idea is useful but not too complicated to understand. An efficient combination of existing methods. \n\n3. The method has a general potential for various kinds of PDEs. \n\n4. The method has shown great performance in engineering-level experiments, fulfilling its great potential for applications. ", "weaknesses": "Although the experiments have demonstrated the main ability of the model, the experiments are not complete enough to support all claims and novelties. \n\n1. Since the model is a new combination of two existing methods GNO and FNO, the main contribution is the new usage of GNO. Then the main thing required to be demonstrated should be “GNO is more proper and has great encoding and decoding ability between irregular and regular grids”. Regarding this, some different encoding methods should be compared, such as GNN, kNN.\n\n2. Discretization invariance is said to be important, but no direct experiments could support this. Although Table 5 is relevant to this, but the variables are not kept so it’s not a direct support. In other words, a GNN+FNO baseline should be added. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer rW9d\n\n> **Q1:** Although the experiments have demonstrated the main ability of the model, the experiments are not complete enough to support all claims and novelties.\n\n**A1:** To further support the results, we add two more experiments that compare GINO with the solver and GNNs (MeshGraphNet). When comparing against the solver, we observe a four to five order of magnitude speed-up when using GINO. At a $3\\%$ relative error, we find the speed-up from our model which includes drag in the loss to be $26,000 \\times$. When comparing against GNN, we implemented the model following MeshGraphNet, which consists of an encoder, decoder, and processer. The model has a size of around 10M parameters, at a similar cost as other models. As shown in the updated experiments, the proposed model is faster than the solver while more accurate than other ML methods given a similar level of cost.\n\n> **Q2:** Since the model is a new combination of two existing methods GNO and FNO, the main contribution is the new usage of GNO. Then the main thing required to be demonstrated should be “GNO is more proper and has great encoding and decoding ability between irregular and regular grids”. Regarding this, some different encoding methods should be compared, such as GNN, kNN. \n\n**A2:** We agree that our main contribution is the combination of two existing methods GNO and FNO. Where GNO handles interaction on local irregular meshes and FNO learns global physics on the uniform latent grid. However, our goal is not to compare GNO against GNN or kNN in this paper. The main spirit of GNO is to design a graph structure with ball connectivity instead of nearest-neighbor connectivity, so the message passing is well-defined as the kernel integral on a ball. Indeed, modern graph neural networks as such GraphCast [1] also design the encoder and decoder with ball connectivity, following the definition of GNO. For a more comprehensive study, we do add a new experiment on GNN (MeshGraphNet) as discussed above.\n\n> **Q3:** Discretization invariance is said to be important, but no direct experiments could support this. Although Table 5 is relevant to this, the variables are not kept so it’s not a direct support. In other words, a GNN+FNO baseline should be added.\n\n**A3:** Discretization invariance means the model can be trained on one mesh discretization and used on another. For example, we have the super-resolution experiment where we train with sub-sampled (lower resolution meshes) and evaluate the model with the full mesh. If we design the encoder or decoder with the nearest-neighbor connectivity, then the neighbors (receptive field) will change along with the sampling rate, which makes it hard for the GNNs to generalize across different resolutions. Regarding the question, could the reviewer clarify which variables are not kept? We kept the same hyperparameters for all sampling rates. \n\n> **Q4:** What is the difference between your proposed GNO and Continuous Convolution (CtsConv) from “Lagrangian Fluid Simulation with Continuous Convolutions”? It seems there is no obvious difference based on equations. And one of the claimed contributions is a well-implemented GNO, which makes it much more efficient, but CtsConv is also well-implemented based on Open3D and hashing table.\n\n**A4:** Thanks for pointing out the paper. We will be happy to add a reference to it. CtsConv learns a continuous convolution for learning fluid mechanics which is based on linear interpolation.  CtsConv is an efficient local convolution layer that competes with GNNs and GNOs. Indeed our GNO implementation is similar to CtsConv. However, we consider 3D industry-level aerodynamic simulations. In the Ahmed-body dataset, the airflow has 7M particles and the surface mesh has 100k particles. As a comparison, the previous work only considers smaller-scale simulations with particles. For the large-scale problem, it's crucial to have an efficient FNO model to capture the global physical interaction, which is our main contribution.\n\n> **Q5:** What is the performance for much finer latent resolutions such as 128^3 and 256^3? Can we keep increasing the test error by increasing the latent resolution?\n\n**A5:** It's a very exciting direction to use much finer latent resolutions, which will likely further improve the performance as projected based on Table 3. However, 80^3 resolution already saturates 32GB Nvidia V100 GPUs. 128^3 and 256^3 resolution will take 4x and 32x more memory respectively. It's possible to push for higher resolution with the model parallel FNO [2], which we leave as future work.\n\n> **Q6:** As the paper stated, the trained model is limited to a specific observed category of shapes. The training for the operator requires a training dataset of high quality.\nThe proposed framework should be general for various kinds of PDEs, but only stable NS equations are tested.\nThe framework is for 3D PDEs, which could be modified for time-dependent ones.\n\n**A6:** Thanks for the comments, we would want to emphasize that 3D aerodynamics is a very one of the biggest industry problems. The time-averaged NS (RANS) is still the industry standard in car design.  We look forward to exploring time-dependent problems in future work.   \n\n[1] Lam, Remi, et al. \"GraphCast: Learning skillful medium-range global weather forecasting.\" arXiv preprint arXiv:2212.12794 (2022).\n[2] Grady, Thomas J., et al. \"Model-parallel Fourier neural operators as learned surrogates for large-scale parametric PDEs.\" Computers & Geosciences (2023): 105402.\n"}, {"review_id": "sPt8tiRMbX", "reviewer": "Reviewer_RPer", "summary": "The authors address the task of learning to solve large-scale PDEs based on a geometry-informed neural operator. The combination of graph neural operators (GNO) and Fourier neural operators (FNO) allows the exploration of the benefits of being able to handle irregular grids and locality of operations to allow efficiency (due to GNO) as well as capturing global interactions (due to FNO), thereby overcoming the limitations of the individual approaches. In more detail, the surface (e.g. point cloud) is input to a geometry encoder that encodes the irregular grid information on a regular grid structure based on local kernel integration layers through GNO with graph operations. Then the result is concatenated with signed distance features. Then, a sequence of FNO layers is used (i.e. on latent space) for global kernel integration. The respective intermediate result is projected back to the domain of the input geometry.\n\nTo show the potential of their approach, the authors carry out experiments on a novel own dataset as well as a ShapeNet car dataset, where they report quantitative results regarding training and test errors.\n", "questions": "Please discuss the comments mentioned under 'Weaknesses'.", "limitations": "Limitations are shortly discussed, but seem to be severely limiting (category-specific approach, lacking generalization capabilities, unclear relation to speed-ups achieved based on physics-informed approaches).", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Technical soundness and novelty:\nThe method seems novel and performs beneficial over some potential alternatives in terms of speed and accuracy, since it allows a significant speed-up in comparison to the OpenFOAM solver and offers more accuracy than GNO, GeoFNO and U-Net.\n\nEvaluation:\nThe authors provide quantitative and qualitative results including comparisons to baselines regarding training/test errors.\n\nExposition:\nThe paper is well-structured and mostly readable. Figure/tables and their captions are informative. \n", "weaknesses": "Technical soundness and novelty\n- The discussion of limitations provided by the authors is rather short. Blending out the benefits of physics-informed approaches, that led to extreme speed-ups over OpenFOAM particularly for fluid simulation as well as offer generalization to novel scenes without being limited to object categories, limits the conclusions drawn from the presented work. The relation to these should be better emphasized to clarify what the presented method adds and whether it can be combined with these.  \n- The datasets are described in a very short manner. Especially for the new dataset, it would be relevant to see more details such as a systematic overview on the key aspects. In addition, clarifications on whether the trainings of different approaches converged within the used 100 iterations would be relevant. In Figure 2, the rightmost part is also difficult to interpret given the used color scale.\n\nEvaluation:\n- Reporting training and test errors only gives limited insights on where the errors are better/worse in comparison to previous approaches. E.g., it is not demonstrated that physical phenomena (such as the Magnus effect or Karman vortex streets) are accurately represented.\n- What are limitations/failure cases that cannot be handled that well with the presented operator?\n\nReferences:\nDiscussing other developments such as\n\nBrandstetter et al., CLIFFORD NEURAL LAYERS FOR PDE MODELING\n-> usage of multivector representations together with Clifford convolutions. The authors show the benefit of Clifford neural layers by replacing convolution and Fourier operations in common neural PDE surrogates by their Clifford counterparts on 2D Navier-Stokes tasks\n\nor physics-informed upgrades of U-Net such as e.g.\n\nWandel et al., Teaching the incompressible Navier–Stokes equations to fast neural surrogate models in three dimensions\n-> an example of physics-informed fluid simulation based on physics-informed U-Net\n\nor the splitting of the solver into region-wise optimization such as\n\nBalu et al., Distributed Multigrid Neural Solvers on Megavoxel Domain\n\nwould improve the discussion of the presented approach and its potential in the context of related work, especially since the presented approach involves quite strong assumptions (dependence on training data/shape category) and not leveraging physics-informed models therefore should be discussed. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer RPer\n\n> **Q1:** The discussion of limitations provided by the authors is rather short. Blending out the benefits of physics-informed approaches, that led to extreme speed-ups over OpenFOAM particularly for fluid simulation as well as offer generalization to novel scenes without being limited to object categories, limits the conclusions drawn from the presented work. The relation to these should be better emphasized to clarify what the presented method adds and whether it can be combined with these.\n\n**A1:** Thanks for pointing this out. The physics-informed approach is the future direction that may overcome the limitation. We will separate the future work from the limitation. We will add a more detail limitation as the space allows.\n\n> **Q2:** The datasets are described in a very short manner. Especially for the new dataset, it would be relevant to see more details such as a systematic overview on the key aspects.  \n\n**A2:** Thanks for the suggestion. We will add a more systematic description of the dataset. \n\n> **Q3:** In addition, clarifications on whether the trainings of different approaches converged within the used 100 iterations would be relevant.\n\n**A3:** Most of the models such as FNO and UNet converge around 60 epochs. We are happy to include the training curves.\n\n> **Q4:** In Figure 2, the rightmost part is also difficult to interpret given the used color scale.\n\n**A4:** In Figure 2, the rightmost part represents the error. We plot the error with the same color bar as the truth and prediction. It shows that the error is near zero almost everywhere except the bumper. We are happy to plot the error with the relative scale in the revision.\n\n> **Q5:** Reporting training and test errors only gives limited insights on where the errors are better/worse in comparison to previous approaches. E.g., it is not demonstrated that physical phenomena (such as the Magnus effect or Karman vortex streets) are accurately represented.What are limitations/failure cases that cannot be handled that well with the presented operator?\n\n**A5:** It is a good question. Usually, a small L2 error implies the two fields are identical, including the same physical behaviors. Since we are not predicting the velocity field but only the pressure field, we cannot study the Magnus effect or Karman vortex. We instead add a drag coefficient study since the drag can be computed from the pressure and the wall shear stress (we predict both). The drag coefficient is one of the major objectives in aerodynamics design. We will add the worst-case example and analysis.\n\n> **Q6:** \n> References: Discussing other developments such as\n> - Brandstetter et al., CLIFFORD NEURAL LAYERS FOR PDE MODELING -> usage of multivector representations together with Clifford convolutions. The authors show the benefit of Clifford neural layers by replacing convolution and Fourier operations in common neural PDE surrogates by their Clifford counterparts on 2D Navier-Stokes tasks\n> \n> or physics-informed upgrades of U-Net such as e.g.\n> \n> - Wandel et al., Teaching the incompressible Navier–Stokes equations to fast neural surrogate models in three dimensions -> an example of physics-informed fluid simulation based on physics-informed U-Net\n> \n> or the splitting of the solver into region-wise optimization such as\n> \n> - Balu et al., Distributed Multigrid Neural Solvers on Megavoxel Domain\n> \n> would improve the discussion of the presented approach and its potential in the context of related work, especially since the presented approach involves quite strong assumptions (dependence on training data/shape category) and not leveraging physics-informed models therefore should be discussed.\n\n**A6:** Thank you for giving these references. We will add the discussion in the related work section in the revision."}, {"review_id": "M8VBTnk7Gd", "reviewer": "Reviewer_5VBN", "summary": "This paper introduces a novel approach for applying Fourier or other Neural operators to complex geometries by prepending a “learnable projection step” via Graph Neural Operator (GNO). Unlike previous methods that morphe complex geometries into regular domains, this approach projects (learnable) sampled nodes onto nearby regular grids, providing advantages such as discretization invariance and reduced computational overhead by sub-sampling.\n\nHowever, the method is limited to simple geometries and fails to account for variations in geometry and subtle geometry features. **The paper also lacks sufficient datasets and comparisons with relevant methods, such as the GNN family.**\n\nWhile the ideas are somewhat novel, though not groundbreaking, **the paper requires further evaluation to demonstrate its strengths and limitations.**\n\n**A borderline rejection is assigned, with reconsideration if the issues are addressed during the revision period.**\n\n## After rebuttal\n- The authors improved their references, presentation, and empirical evaluation; Hence I increased my score to 5", "questions": "See the last point in **Weaknesses**", "limitations": "The method is limited to very simple geometries due to 2 facts:\n1. Only relies on the point cloud. A counter-example is porous material where the point cloud can be the same but the tunnel is very different.\n   - In other words, although this method is discretization invariant, it is also geometry-ignorant, which is not desired.\n2. Sub-sampling, if the key flow feature is determined by some subtle geometries, which is very common, this method fails again. The limitation also is reflected in the dataset, as the pressure seems to look really smooth for all cases, and all geometries are very simple.\n\nI recommend the authors objectively illustrate these facts, even at the beginning (you do not have to emphasize them; mentioning them is enough). More experiments objectively showing the limitation (maybe in the appendix) are also appreciated.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Discretization invariant\n- Improved efficiency and scalability by sub-sampling\n- Improved empirical performance", "weaknesses": "- Insufficient number of datasets and comparisons:\n  - More datasets should have been included, such as cylinders and airfoils from [1].\n  - Comparisons with representative methods from the GNN family, such as MeshGraphNet[1], MSGNN-Grid[2], and BSMSGNN[3], should have been discussed or ideally conducted.\n  - Among the mentioned papers [1] to [3], a crucial comparison would be with [2], which also utilizes a background grid as a helper but differs in the backbone as it did not use FNO. This would have provided a valuable benchmark for evaluation.\n\n- The overly ambitious illustrations: such as the claim in the abstract that \"...(the method) can be applied to any geometry\", which is not the case. A more objective approach in illustrating both the strengths and limitations would be appreciated by readers.\n\n- Lack of clarity in dataset presentation and results:\n  - Fig.2 suggests that the shapes are overly simplistic and the pressure distributions appear uniformly smooth, indicating ease of learning. It is essential for the author to provide a more comprehensive and clear explanation, including typical examples of datasets and variations in geometry among examples.\n\n[1] Pfaff, Tobias, Meire Fortunato, Alvaro Sanchez-Gonzalez, and Peter W. Battaglia. \"Learning Mesh-Based Simulation with Graph Networks.\" Link: https://openreview.net/forum?id=roNqYL0_XP\n\n[2] Lino, Mario, Chris Cantwell, Anil A Bharath, and Stathi Fotiadis. \"Simulating continuum mechanics with multi-scale graph neural networks.\". Link: https://arxiv.org/abs/2106.04900\n\n[3] Cao, Yadi, Menglei Chai, Minchen Li, and Chenfanfu Jiang. \"Efficient learning of mesh-based physical simulation with bi-stride multi-scale graph neural network.\". Link: https://openreview.net/forum?id=2Mbo7IEtZW", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer 5VBN\n\n> **Q1:** Insufficient number of datasets and comparisons:\nMore datasets should have been included, such as cylinders and airfoils from [1].\nComparisons with representative methods from the GNN family, such as MeshGraphNet[1], MSGNN-Grid[2], and BSMSGNN[3], should have been discussed or ideally conducted.\nAmong the mentioned papers [1] to [3], a crucial comparison would be with [2], which also utilizes a background grid as a helper but differs in the backbone as it did not use FNO. This would have provided a valuable benchmark for evaluation.\nThe overly ambitious illustrations: such as the claim in the abstract that \"...(the method) can be applied to any geometry\", which is not the case. A more objective approach in illustrating both the strengths and limitations would be appreciated by readers.\n\n**A1:** We thank the reviewer for the suggestions. As suggested by the reviewer, we added the comparison with MeshGraphNet [1]. The GNN model learns the local interaction with flexible meshes. However, it has the challenge of learning the long-range global interaction. On the other hand, the GINO model has the advantages of both the graphs structure and Fourier methods. It has the flexibility of graphs as well as the efficiency of Fourier methods. The details can be found in the general response. We are further experimenting with GraphCast [4] which has a latent graph similar to MSGNN-Grid [2], so far it underperforms the MeshGraphNet model [1]. We will continue to experiment to present a faithful comparison.\nWe are happy to add the reference to MSGNN-Grid and BSMSGNN too.\n\nIn this work, we aim to study industry-level 3D aerodynamics simulation. We generate the Ahmed-body dataset, which has 7M nodes in the space and 100k nodes on the surface. The 2D datasets such as the cylinders flow and airfoils are interesting and widely studied, but they are much smaller with only a few thousand nodes. The 3D fluid problems are several order more expensive to generate. We have been searching for complex 3D simulations for long, but such public data is still lacking in the community. Therefore we decided to generate a new dataset. If the reviewer is aware of existing 3D simulations with multiple complex shapes, we will be very happy to experiment with them.\n\n\n\n> **Q2:** Lack of clarity in dataset presentation and results:\nFig.2 suggests that the shapes are overly simplistic and the pressure distributions appear uniformly smooth, indicating ease of learning. It is essential for the author to provide a more comprehensive and clear explanation, including typical examples of datasets and variations in geometry among examples.\n\n\n**A2:** Thanks for the comments. We added a few illustrations in Figure 6 in the appendix. We also provide a figure ([3] in the general response) of the collection of shapes below. They are significantly more complex compared to the existing 2D problems such as the channel flow and airfoils, where the airfoils are smooth curves determined by a few parameters. It is unfair to require graphics-level meshes. Even with these relatively simple meshes, the physics is highly complicated as shown in Figure 6. High-fidelity 3D simulations on graphics-level 3d shapes would take up to billion of mesh points to solve with NASA FUN3D solver [1], which is sincerely beyond the scope of this work.\n\n\n[1] Carlson, Jan-Renee, et al. \"High-Fidelity Simulations of Human-Scale Mars Lander Descent Trajectories.\" AIAA AVIATION 2023 Forum. 2023."}, {"review_id": "BS3oo6R2Nh", "reviewer": "Reviewer_zgWV", "summary": "This paper presents geometry-informed neural operator (GINO), solving computational fluid dynamics (CFD)  problems. It combines graph neural operations (GNO) and Fourier neural operators (FNO) to adapt to irregular discretized grids. The authors have tested the model on two large-scale datasets.", "questions": "See weakness.", "limitations": "See weakness.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "\n1. The authors propose combining GNO and FNO to leverage the advantages of both methods, such as analyzing local and global information and efficiently processing irregular grids. They also conduct experiments demonstrating that GINO exhibits discretization invariance over the latent grid and the input-output mesh.\n\n2. The authors have generated two CFD datasets using various vehicle datasets. It would greatly benefit the learning physical simulation community if these datasets were made publicly available upon the paper's acceptance.", "weaknesses": "1. The definition of the $\\kappa$ operator in the graph operator block is unclear. It is not specified whether it measures the distance between two points or the similarity of their features. Additionally, it would be helpful to know if the $\\kappa$ operator has any learnable parameters. Providing more details in the paper would make it self-contained and enable readers to understand the methodology.\n\n2. The paper mentions the input of SDF features and surface points. It would be beneficial to clarify if these two types of data are fed in a uniform manner. For instance, are the point locations associated with their corresponding SDF values, while the surface points are assigned a value of 0?\n\n3. The authors claim efficient graph construction in the paper. However, it appears that once the graph operator block finishes processing the input, it results in a regular grid. It would be important to address whether the implementation takes this into consideration and ensures the efficiency of the graph construction process.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer zgWV\n\n> **Q1:** The definition of the κ  operator in the graph operator block is unclear. It is not specified whether it measures the distance between two points or the similarity of their features. Additionally, it would be helpful to know if the κ operator has any learnable parameters. Providing more details in the paper would make it self-contained and enable readers to understand the methodology.\n\n**A1:** Thank you for the comments. The κ operator in GNO learns the transform of meshes. It can be viewed as a learned interpolation, where the kernel measures the weights (similarity) between two points. The Radial basis function (RBF) interpolation is one simple example of the kernel. In GNO blocks, the kernel function k is parameterized as a neural network. Its weights are learnable parameters. As shown in the table, the learnable GNO encoder-decoder outperforms fixed interpolation. We will try to add more backgrounds and make it self-contained.\n\n> **Q2:** The paper mentions the input of SDF features and surface points. It would be beneficial to clarify if these two types of data are fed in a uniform manner. For instance, are the point locations associated with their corresponding SDF values, while the surface points are assigned a value of 0?\n\n**A2:** Thanks for the question. The SDF features are measured on the uniform latent space (64x64x64), and the surface points are a point cloud of the geometry. The SDF is directly fed to the latent FNO model, while the surface is fed to the encoder GNO. They don't have to have the same format. We will add these clarifications to the paper.\n\n> **Q3:** The authors claim efficient graph construction in the paper. However, it appears that once the graph operator block finishes processing the input, it results in a regular grid. It would be important to address whether the implementation takes this into consideration and ensures the efficiency of the graph construction process.\n\n**A3:** Thanks for the suggestion. Since the ending mesh is a uniform grid, it's possible to round up the node in the starting mesh and find out its neighbors. In our implementation, we used a hash-table-based graph construction, which is similar to the rounding process."}]

Author rebuttal:
# General response

We are grateful to the reviewers for their insightful feedback and constructive comments.
It is encouraging to note that part of the reviewers agree with the
- scalability of the proposed model to realistic 3D aerodynamic simulations,
- acknowledge the significant speed-up offered by our approach,
- and recognize our new dataset has been a valuable contribution.

However, the primary concerns lie in the comprehensiveness of the experiments. In response to these concerns and support our results, we add **two new experiments** that further compare our model with GNN methods and numerical solver.
- we added an experiment comparing GINO's performance with GNN. It shows GINO has a much smaller error rate compared to GNN (8.31% vs 13.88%).
- we perform a full cost-accuracy trade-off analysis. The result shows GINO is 26,000x faster at computing the drag coefficients.
- and we further expand the Ahmed body dataset from 500 instances to 800, making it the first and largest 3D RANS design dataset available for the community.

We sincerely hope these experiments make our study more comprehensive and clarify the concerns of the reviewers.

## 1. Comparison against GNNs

We added an experiment comparing GINO's performance with GNN. It shows GINO has a much smaller error rate compared to GNN (8.31% vs 13.88%). 
We used the MeshGraphNet [1], as suggested by Reviewer 4. This model is a common GNN method for physical simulations. The MeshGraphNet has three main parts: an encoder, a decoder, and a processor. Both the encoder and decoder use the same setup, with a channel size of 256. The processor has 15 layers of information passing, using edge and node blocks. It also has a size of 256. In total, the model has about 10 million parameters.  The total number of edges is around 280k, which saturates a single NVIDIA V100 GPU with 32GB of memory. We set the learning rate to be 1e-4 with an exponential decay with a rate of 0.99985 similar to the original setup in [1].

The validation error of GNN for the pressure field is **13.88%**. The training curve is shown in the report. The error of GNN model is much higher than the GINO model (64x64x64 resolution, 100M params) of **8.31%**.  The smaller GINO (32x32x32 resolution, 10M params) can achieve 10.10% error rate. The GNN model is good at learning the local interaction with flexible meshes. However, it cannot easily learn the long-range global interactions. On the other hand, the GINO model has the advantages of both the graph and Fourier method. It has the flexibility of graphs as well as the efficiency of Fourier methods for capturing long-range dependencies.

[1] Pfaff, Tobias, Meire Fortunato, Alvaro Sanchez-Gonzalez, and Peter W. Battaglia. "Learning Mesh-Based Simulation with Graph Networks." Link: https://openreview.net/forum?id=roNqYL0_XP

## 2. Comparison against solver on Drag coefficients

To compare the performance of our model against the industry-standard OpenFOAM solver, we perform a full cost-accuracy trade-off analysis. The result shows GINO is **26,000x** faster at computing the drag coefficients. Figure (2, left) shows the cost-accuracy curve, measured in terms of inference time needed for a relative error in the drag coefficient for GINO and OpenFOAM. 

The cost of GINO is computed as the time, averaged over the test set, needed to predict the drag coefficient by running the model. This time includes both data pre-processing (computing the SDF) as well as the model run-time and the drag calculation given the predicted fields. All models are run on a single NVIDIA V100 GPU. The cost for OpenFOAM is computed as described in the proceeding paragraph and is averaged over the test set. The solver is run on two NVIDIA V100 GPUs in parallel. We observe a four to five order of magnitude speed-up when using GINO. At a **3%** relative error, we find the speed-up from our model which includes drag in the loss to be 26,000 times. As we increase the size of the latent space, the cost of GINO grows, however, we observe a plateau in the drag error. This is common in machine learning models as the error from using finite data starts to dominate the approximation error. Furthermore, we use only the size of the latent space as a hyper-parameter, keeping the number of learnable parameters fixed. It is interesting to explore further how parametrically scaling the model impacts predictive power.

During data generation,
we keep track of the drag coefficient predicted by OpenFOAM after every iteration. 
While the coefficient converges with more iterations, this convergence is not monotone and can often appear quite noisy. This makes computing the error from the raw data not possible. We therefore apply a box filter to the raw signal to compute a filtered version of the drag which acts as a smoother. We take as the reference drag, the drag at the last iteration of the filtered signal. 
To compute the number of iterations it takes for the solver to predict a drag coefficient at a given relative error, we trace back the predictions from the filtered signal and return the first time at which this prediction incurs the given error with respect to the reference drag. An example of this methodology is shown in Figure (2, right). The errors for our GINO model are computed with respect to the true drag coefficient from the last iteration of the solver. This is because we take as ground truth the pressure and wall shear stress from this last iteration and train our model to predict them.


Meta-review: Reviewers for this paper agree that the proposed method seems solid and that the results are extremely impressive.  The main issue here seems to be that the method is undertested.  I am happy to suggest accepting this work with the understanding that the authors incorporate the new experiments/descriptions shared during the rebuttal phase.  Please also pay close attention to reviewer RPer's suggestions for ways to improve the exposition and avoid missing details.
Decision: Accept (poster)