SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathcal {K} (v) (x) = \sigma \big (W v (x) + \mathcal {C} (v) \big), \quad \forall x \in \mathbb {T} ^ {d}
$$
where  $\sigma$  is a pointwise non-linearity and  $W\in \mathbb{R}^{n\times m}$  is a learnable matrix. We further modify the layer by learning the kernel coefficients in tensorized form, adding skip connections, normalization layers, and learnable activations as outlined in [24]. We refer the reader to this work for further details.
Adaptive instance normalization. For many engineering problems of interest, the boundary information is a fixed, scalar, inlet velocity specified on some portion of  $\partial D$ . In order to efficiently incorporate this scalar information into our architecture, we use a learnable adaptive instance normalization [25] combined with a Fourier feature embedding [21]. In particular, the scalar velocity is embedded into a vector with Fourier features. This vector then goes through a learnable MLP, which outputs the scale and shift parameters of an instance normalization layer [26]. In problems where the velocity information is not fixed, we replace the normalization layers of the FNO blocks with this adaptive normalization. We find this technique improves performance, since the magnitude of the output fields usually strongly depends on the magnitude of the inlet velocity.
Table 2: Shape-Net Car dataset (3.7k mesh points).
Model	training error	test error
GNO	18.16%	18.77%
Geo-FNO (sphere)	10.79%	15.85%
UNet (interp)	12.48%	12.83%
FNO (interp)	9.65%	9.42%
GINO (encoder-decoder)	7.95%	9.47%
GINO (decoder)	6.37%	7.12%
We do a benchmark study with several standard machine-learning methods on the Shape-Net and Ahmed body datasets. The training error is normalized L2 error; the test error is de-normalized L2.
We explore a range of models on two CFD datasets. The large-scale Ahmed-Body dataset, which we generated, and also the Shape-Net Car dataset from [16]. Both datasets contain simulations of the Reynold-Averaged Navier-Stokes (RANS) equations for a chosen turbulence model. The goal is to estimate the full pressure field given the shape of the vehicle as input. We consider GNO [2], MeshGraphNet [9], GeoFNO [14], 3D UNet [27] with linear interpolation, FNO [6], and GINO. We train each model for 100 epochs with Adam optimizer and step learning rate scheduler. The implementation details can be found in the Appendix. All models run on a single Nvidia V100 GPU.
4.1 Ahmed-Body dataset
We generate the industry-level vehicle aerodynamics simulation based on the Ahmed-body shapes [28]. The shapes are parameterized with six design parameters: length, width, height, ground clearance, slant angle, and fillet radius. We also vary the inlet velocity from  $10\mathrm{m / s}$  to  $70\mathrm{m / s}$ , leading to Reynolds numbers ranging from  $4.35\times 10^{5}$  to  $6.82\times 10^{6}$ . We use the GPU-accelerated OpenFOAM solver for steady state simulation using the SST  $k - \omega$  turbulence model [29] with 7.2 million mesh points in total with 100k mesh points on the surface. Each simulation takes 7-19 hours on 2 Nvidia v100 GPUs with 16 CPU cores. We generate 551 shapes in total and divide them into 500 for training and 51 for validation.
4.2 Shape-Net Car dataset
We also consider the Car dataset generated by [16]. The input shapes are from the ShapeNet Car category [30]. In [16], the shapes are manually modified to remove the side mirrors, spoilers, and tires. The RANS equations with the  $k - \epsilon$  turbulence model and SUPG stabilization are simulated to obtain the time-averaged velocity and pressure fields using a finite element solver [31]. The inlet velocity is fixed at  $20\mathrm{m / s}$  (72km/h) and the estimated Reynolds number is  $5\times 10^{6}$ . Each simulation takes approximately 50 minutes. The car surfaces are stored with 3.7k mesh points. We take the 611 water-tight shapes out of the 889 instances, and divide the 611 instances into 500 for training and 111 for validation.
As shown in Table 2 3 and Figure 2, GINO achieves the best error rate with a large margin compared with previous methods. On the Ahmed-body dataset, GINO achieves  $8.31\%$  while the previous best method achieve  $11.16\%$ . On the Shape-Net Car, GINO achieves  $7.12\%$  error rate compared to  $9.42\%$  on FNO. It takes 0.1 seconds to evaluate, which is  $100,000x$  faster than the GPU-parallel OpenFOAM solver that take 10 hours to generate the data. We further performance a full cost-accuracy analysis in the following section.
For ablations, we consider channel dimensions [32, 48, 64, 80], latent space [32, 48, 64, 80], and radius from 0.025 to 0.055 (with the domain size normalized to [-1, 1]). As depicted in Figure 4(a) and Table 4, larger latent spaces and radii yield superior performance.
4.3 Cost-accuracy analysis on drag coefficient
To compare the performance of our model against the industry-standard OpenFOAM solver, we perform a full cost-accuracy trade-off analysis on computing the drag coefficient, which is the standard design objective of vehicles and airfoils. The result shows GINO is 26,000x faster at computing the drag coefficients. Figure 3(b) below shows the cost-accuracy curve, measured in terms of inference
Table 3: Ahmed-body dataset (100k mesh points).
Model	training error	test error
MeshGraphNet	9.08%	13.88%
UNet (interp)	9.93%	11.16%
FNO (interp)	12.97%	12.59%
GINO (encoder-decoder)	9.36%	9.01%%
GINO (decoder)	9.34%	8.31%
Previous works such as GNO and Geo-FNO cannot scale to large meshes with 100k points. We instead add the MeshGraphNet for graph comparison. Again, for UNet, FNO, and GINO, we fix the latent grid to  $64 \times 64 \times 64$ . The training error is normalized L2; the test error is de-normalized L2.
(a) Example drag coefficient computed after every iteration of the OpenFOAM solver. The reference drag corresponds to the last point of the filtered signal. Triangle indicates the time when the solver reaches a  $3\%$  relative error with respect to the reference drag.
(b) Cost-accuracy trade-off curve for OpenFOAM vs. GINO. Increasing cost of GINO is achieved by increasing the size of the latent space. Loss function for models on the red line includes only error in pressure and wall shear stress, while, for models on orange line, the loss also includes drag.
Figure 3: Cost-accuracy trade-off analysis for the drag coefficient.
time needed for a relative error in the drag coefficient for GINO and OpenFOAM. The detailed setup is discussed in the appendix.
4.4 Discretization-convergence and ablation studies
We investigate discretization-convergence by varying different parts of GINO. Specifically, we vary the latent grid resolution and the sampling rates for input-output meshes. In these experiments, we fixed the training and test samples to be the same, i.e., same latent grid resolution or sampling rate, but varied the shape and input conditions.
Discretization-convergence wrt the latent grid. Here, each model is trained and tested on (the same) latent resolutions, specifically 32, 48, 64, 80, and 88, and the architecture is the same. As depicted in Figure 4(a), GINO demonstrates a comparable error rate across all resolutions. A minor improvement in errors is observed when employing a larger latent space. Conversely, the errors associated with the UNet model grow as the resolution is decreased due to the decreasing receptive field of its local convolution kernels.
Discretization-convergence in the input-output mesh. Here, GINO is trained and tested with sub-sampled input-output meshes at various sampling rates (2x, 4x, 6x, 8x). As illustrated in Figure 4(b), GINO exhibits a consistent error rate across all sampling rates. A slight increase in errors is observed on coarser meshes.
Zero-shot super-resolution. GINO possesses the ability to perform zero-shot super-resolution. The model is trained on a coarse dataset, sub-sampled by 2x, 4x, 6x, and 8x, and subsequently tested on the full mesh, that is not seen during training. The error remains consistent across all sampling rates 4(c). This characteristic enables the model to be trained at a coarse resolution when the mesh is dense, consequently reducing the computational requirements.
(a) Varying resolutions of the latent grid (same resolution for training and testing).
(b) Varying the sampling rates of the input-output mesh (same rate for training and testing).
(c) Train with a low sampling rate and test on full mesh (zero-shot superresolution).
Figure 4: Discretization-convergence studies and zero-shot super-resolution.
The study of neural operators and their extended applications in learning solution operators in PDE has been gaining momentum [3, 32-34]. A method that stands out is FNO, which uses Fourier transform [6]. The FNO and its variations have proven to highly accelerate the simulations for large-scale flow problems, including weather forecasting [4], seismology [35, 36] and multi-phase flow [5]. However, a challenge with the FNO is that its computation superiority is gained when applied on a regular grid, where the Fourier transform is approximated using FFT. Therefore, its reliance on FFT limits its use with irregular grids or complex geometries. There have been attempts to modify the FNO to work with these irregular structures, but scalability to large-scale 3D PDEs remains an issue. One such attempt is GeoFNO, which learns a coordinate transformation to map irregular inputs to a regular latent space [14]. This method, while innovative, requires a geometric discrete Fourier transform, which is computationally demanding and lacks discretization insurance. To circumvent this, GINO limits the Fourier transform to a local GNO to improve efficiency. The locality is defined assuming the metrics of the physical space.
Additionally, the Non-Equispaced Fourier neural solvers (NFS) merge the FNO with non-equispaced interpolation layers, a method similar to global GNO [37]. However, at the architecture level, their method replaces the integration of GNO with the summation of the nearest neighbor points on the graph. This step transitions this method to a neural network, failing to deliver a discretization convergent approach. The Domain-Agnostic Fourier Neural Operators (DAFNO) represents another attempt at improvement, applying an FNO to inputs where the geometry is represented as an indicator function [38]. However, this method lacks a strategy for handling irregular point clouds. Simultaneously, researchers are exploring the combination of FNO with the attention mechanisms [3] for irregular meshes. This includes the Operator Transformer (OFformer) [39], Mesh-Independent Neural Operator (MINO) [40], and the General Neural Operator Transformer (GNOT) [41]. Besides, the Clifford neural layers [42] use the Clifford algebra to compute multivectors, which provides Clifford-FNO implementations as an extension of FNO. The work [43] uses a physics-informed loss with U-Net for 3D channel flow simulation with several shapes. The work [44] innovatively proposes the use of multigrid training for neural networks that improves the convergence. Although these methods incorporate attention layers, which are special types of kernel integration [3] with quadratic complexity, they face challenges when scaling up for large-scale problems.
GNNs are incorporated in the prior attempts in physical simulations involving complex geometry, primarily due to the inherent flexibility of graph structures. Early research [7, 45-47] laid the foundation for GNNs, demonstrating that physical entities, when represented as graph nodes, and their interactions, as edges, could predict the dynamics of various systems. The introduction of graph element networks [48] marked a significant development, being the first to apply GNNs to PDEs by discretizing the domain into elements. The similar idea has been explored in term of continuous 3D point cloud convolution [49-51]. Another line of work, mesh graph networks [8-10], further explored PDEs in the context of fluid and solid mechanics. [52, 53] train a Graph convolutional neural works on the ShapeNet car dataset for inverse design. However, GNN architectures' limitations hinder their use in operator learning for PDEs. GNNs connect each node to its nearest neighbors according to the graph's metrics, not the metrics of the physical domain. As the input function's discretization becomes finer, each node's nearest neighbors eventually converge to the same node, contradicting the expectation of improved model performance with finer discretization. Furthermore, GNNs'
(a) An input geometry (continuous function) is first discretized into a series of points by subsampling it. Note that in practice, the discretization can be highly irregular. A key challenge with several scientific computing applications is that we want a method that can work on arbitrary geometries, but also that is discretization convergent, meaning that the method converges to a desired solution operator as we make the discretization finer.
(b) GNN connects each point in the latent subspace (red) to its nearest neighbors in the original space (top). This is very discretization dependent, and as we increase the resolution (sample points more densely), the method becomes increasingly local and fails to capture context. In addition, the operator at the discretization limit is non-unique and depends on how the discretization is done.
(c) GNO instead connects each point in the latent subspace (red) to all its neighbors within an epsilon ball in the original space (top). This induces convergence to a continuum solution operator as we increase the resolution (sample points more densely). This means GNO converges to a unique operator as the discretization becomes finer and scales to large problems without becoming overly local.
Figure 5: Comparison of GNN and GNO as the discretization becomes finer. GNN is discretization dependent, while GNO is discretization convergent.
model behavior at the continuous function limit lacks a unique definition, failing the discretization convergence criterion. Consequently, as pointwise operators in function spaces at the continuous limit, GNNs struggle to approximate general operators between function spaces, Figure 5.
In this work, we propose the GINO model for 3D PDEs with complex geometries. The GINO model consists of the graph-kernel blocks for the encoder and decoder that go to a latent uniform space, where the Fourier blocks run on the latent space to capture the global interaction. We experiment on two CFD datasets: Shape-Net car geometries and large-scale Ahmed's body geometries, the latter encompassing over 600 car geometries featuring hundreds of thousands of mesh points. The evidence from these case studies illustrates that our method offers a substantial speed improvement, with a factor of 100,000 times acceleration in comparison to the GPU-based OpenFOAM solver. Concurrently, our approach has achieved one-fourth to one-half the error rates compared to prevailing neural networks such as 3D U-Net. This underscores the potential of our method to significantly enhance computational efficiency while maintaining a competitive level of accuracy within the realm of CFD applications. Limitation: The trained surrogate model is limited to a specific category of shapes. The quality of the model depends on the quality of the training dataset. For CFD with more complex shapes, it is not easy to obtain a large training dataset. We will explore physics-informed approaches [54] and generate time-dependent high-fidelity simulations in the future.
ZL is supported by the Nvidia fellowship. NBK is grateful to the NVIDIA Corporation for support through full-time employment.
[51] Wenxuan Wu, Zhongang Qi, and Li Fuxin. Pointconv: Deep convolutional networks on 3d point clouds. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pages 9621-9630, 2019.
[52] Edoardo Remelli, Artem Lukoianov, Stephan Richter, Benoit Guillard, Timur Bagautdinov, Pierre Baque, and Pascal Fua. Meshsdf: Differentiable iso-surface extraction. Advances in Neural Information Processing Systems, 33:22468-22478, 2020.
[53] Nikita Durasov, Artem Lukoyanov, Jonathan Donier, and Pascal Fua. Debosh: Deep bayesian shape optimization. arXiv preprint arXiv:2109.13337, 2021.
[54] Zongyi Li, Hongkai Zheng, Nikola Kovachki, David Jin, Haoxuan Chen, Burigede Liu, Kamyar Azizzadenesheli, and Anima Anandkumar. Physics-informed neural operator for learning partial differential equations. arXiv preprint arXiv:2111.03794, 2021.
[55] Matthias Fey and Jan Eric Lenssen. Fast graph representation learning with pytorch geometric. arXiv preprint arXiv:1903.02428, 2019.
7.1 Experiments and Ablations
Table 4: Ablation on the Ahmed-body with different sizes of the latent space
Model	latent resolution	radius	training error	test error
GINO	32	0.055	14.11%	13.59%
GINO	48	0.055	8.99%	10.20%
GINO	64	0.055	6.00%	8.47%
GINO	80	0.055	5.77%	7.87%
GINO	32	0.110	8.66%	10.10%
GINO	48	0.073	7.25%	9.17%
GINO	64	0.055	6.00%	8.47%
GINO	80	0.044	6.22%	7.89%
When fixing the radius, larger latent resolutions lead to better performance. The gaps become smaller when fixing the number of edges and scaling the radius correspondingly.
**Benchmarks.** This study analyzes several existing models, including GNO, GeoFNO, 3D UNet, and MeshGraphNet. All models are trained using the Adam optimizer for 100 epochs, with the learning rate halved at the 50th epoch. We consider starting learning rates such as [0.002, 0.001, 0.0005, 0.00025, 0.0001], with the most favorable results attained at the rates 0.00025 and 0.0001. For GeoFNO, a 2D spherical latent space is employed due to instability with 3D deformation, leading to a faster runtime than other 3D-based models. In the 3D UNet model, we evaluate channel dimensions ranging from [64, 128, 256] and depths from [4, 5, 6]. Utilizing a larger model can reduce UNet's error rate to  $11.1\%$ .
We added an experiment comparing GINO's performance with GNN. We used the MeshGraphNet [9], which is a common GNN method for physical simulations. The MeshGraphNet has three main parts: an encoder, a decoder, and a processor. Both the encoder and decoder use the same setup, with a channel size of 256. The processor has 15 layers of information passing, using edge and node blocks. It also has a size of 256. In total, the GNN model has about 10M parameters. The total number of edges is around 280k, which saturates a single NVIDIA V100 GPU with 32GB of memory. We set the learning rate to be 1e-4 with an exponential decay with a rate of 0.99985, similar to the original setup in [9].
For the GINO model, we consider channel dimensions [32, 48, 64, 80], latent space [32, 48, 64, 80], and radius from 0.025 to 0.055 (with the domain size normalized to [-1, 1]). As depicted in Figure 4(a) and Table 4, larger latent spaces and radii yield superior results.
Table 5: Ablation on the Ahmed-body with different choices of the radius.
Model	radius 0.025	radius 0.035
training error	test error	training error	test error
GINO (encoder-decoder)	12.91%	13.07%	8.65%	10.32%
GINO (encoder-decoder, weighted)	12.94%	12.76%	9.26%	9.90%
GINO (decoder)	12.62%	12.74%	8.82%	9.39%
The choice of radius is significant. A larger radius leads to better performance for all models.
Encoder and Decoder. For the GINO model, we contemplate two configurations: an encoder-decoder design utilizing GNO layers for both input and output, and a decoder-only design which takes the fixed SDF input directly from the latent space and employs the GNO layer solely for output. When the input mesh significantly exceeds the latent grid in size, the encoder proves beneficial in information extraction. However, when the size of the latent grid matches or surpasses the input mesh, an encoder becomes redundant. As depicted in Table 5, both the encoder-decoder and decoder-only designs exhibit comparable performance.
Parallelism. Data parallelism is incorporated in the GNO decoder. In each batch, the model sub-samples 5,000 mesh points to calculate the pressure field. As query points are independent, they can be effortlessly batched. This parallel strategy allows for a larger radius in the decoder GNO. Without a parallel implementation, a radius of 0.025 leads to 300,000 edges, rapidly depleting GPU memory. Yet, with parallelism, the algorithm can handle a radius of 0.055. Implementing parallelism in the encoder is left for future exploration.
Weights in the Riemann Sum. As mentioned in the GNO section, the integral is approximated as a Riemann sum. In the decoder, the weight  $\mu(y)$  is constant, reflecting the uniformity of the latent space. Conversely, in the encoder, weights are determined as the area of the triangle. For increased expressiveness, the weight is also integrated into the kernel, resulting in a kernel of the form  $\kappa(x,y,\mu(y))$ . However, it's worth noting that the encoder's significance diminishes when a large latent space is in use.
Sub-sampling and Super-resolution. The computational cost of the models increases rapidly with the number of mesh points. Training models with sub-sampled meshes saves significant computational resources. Discretization-convergent models can achieve such super-resolution; they can be trained on coarse mesh points and generalized to a fine evaluation mesh. We investigate the super-resolution capabilities of UNet (interp), FNO (interp), and GINO. As demonstrated in Table 6, GINO maintains consistency across all resolutions. UNet (interp) and FNO (interp) can also adapt to denser test meshes, albeit with a marginally higher error rate, based on linear interpolation. The results corroborate GINO's discretization-convergent nature.
Hash-table-based Graph Construction. For graph construction and kernel integration computation in this work, we utilize the CUDA implementation from Open3D [22] and torch-scatter [23], respectively. This approach is  $40\%$  faster compared to a previous GNO implementation that constructed the graphs with pairwise distance and used the PyTorch Geometric library[55]. These implementations are Incorporated into the GNO encoder and decoder in GINO.
In addition, the CUDA hash based implementation requires less memory footprint  $O(N dr^3)$  compared to the standard pairwise distance which requires  $O(N^2)$  memory and computation complexity. For 10k points, hash-based implementation requires 6GB of GPU memory while the pairwise method requires 24GB of GPU memory; making the hash-based method more scalable for larger graphs.
Table 6: Super-resolution on sub-sampled meshes
Model	Sampling rate	1/2	1/4	1/6	1/8
Unet (interp)	16.5%	13.8%	13.9%	15.6%
FNO (interp)	14.2%	14.1%	13.3%	11.5%
GINO (encoder-decoder)	8.8%	9.4%	9.4%	9.7%
7.2 Drag Coefficient Comparison
For many engineering tasks, the goal is often to determine a single quantity of interest from a simulation which can then be used within an overall design process. In the design of automobiles, a sought after quantity is the drag coefficient of the vehicle. Intuitively, it is a number inversely proportional to the efficiency with which a vehicle passes through a fluid. Engineers are therefore often interested in designing geometries with minimal drag coefficients. For a fluid with unit density, the drag coefficient is defined as
$$
c _ {d} = \frac {2}{v ^ {2} A} \left(\int_ {\partial \Omega} p (x) (\hat {n} (x) \cdot \hat {i} (x)) d x + \int_ {\partial \Omega} T _ {w} (x) \cdot \hat {i} (x) d x\right) \tag {8}
$$
where  $\partial \Omega \subset \mathbb{R}^3$  is the surface of the car,  $p: \mathbb{R}^3 \to \mathbb{R}$  is the pressure,  $\hat{n}: \partial \Omega \to \mathbb{S}^2$  is the outward unit normal vector of the car surface,  $\hat{i}: \mathbb{R}^3 \to \mathbb{S}^2$  is the unit direction of the inlet flow,  $T_w: \partial \Omega \to \mathbb{R}^3$  is the wall shear stress on the surface of the car,  $v \in \mathbb{R}$  is the speed of the inlet flow, and  $A \in \mathbb{R}$  is the area of the smallest rectangle enclosing the front of the car.
For our Ahmed-body dataset, we train several GINO models (decoder) to predict the pressure on the surface of the car as well as the wall shear stress. Since the inlet flow is always parallel to the  $x$ -axis