SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $\kappa^2$  is the eigenvalue and  $n$  is the unit normal vector to the surface  $\delta \Omega$  at  $x$ . The solutions to this equation form a basis which diagonalizes the Laplace operator, and hence this equation arises in many applications. In particular, this equation is used in studying wave propagation, and therefore specifically in acoustics and optics. Here we assume the eigenvalue of interest to be  $\kappa^2 = 1$ .
Figure 4 shows the FEM solution and neural operator prediction for the Helmholtz equation. On both these test meshes the MSE of the neural operator is approximately  $2\%$  of the average intensity.
3.3 Steady state reaction-diffusion system
Reaction-diffusion equations are frequently used in modeling chemical reactions, biological systems such as in predator-prey models, combustion, population dynamics, and spread of disease in epidemiology. We consider the case of a quadratic reaction term, known as Fisher's equation, with Neumann boundary conditions in the stationary  $(\partial_t\phi = 0)$  limit:
$$
\nabla^ {2} \phi + \phi (1 - \phi) = f (x) \quad x \in \Omega , \tag {12}
$$
$$
n \cdot \nabla \phi (x) = 0 \quad x \in \delta \Omega . \tag {13}
$$
Figure 5 shows our neural operator predictions on three test meshes. The MSEs for these predictions are  $1\%$ ,  $1\%$ , and  $5\%$  of the mean intensities respectively. There is a noticeable qualitative difference between the neural operator solution and the FEM solution in the third row (Figures 5g to 5i). The small relative volume covered by this error suppresses its contribution to the MSE, and to the loss function in Equation (3).
(a) FEM
(d) FEM
Figure 4: Helmholtz equation. a, d show the FEM solution, b, e show the output of our trained model on the same geometry, and c, f show the absolute error of the model compared to the FEM solution.
Figure 5: Reaction-diffusion system. a, d, g show the FEM solution, b, e, h show the output of our trained model on the same geometry, and c, f, i show the absolute error of the model compared to the FEM solution.
(g) FEM
(e) Neural operator
(h) Neural operator
(f) Absolute difference
(i) Absolute difference
There have been many advances in physics informed learning in recent years. However, relatively few of these advances specifically focus on boundary geometry.
On fixed rectangular geometries, convolutional neural networks and Fourier neural operators [7] can be used to predict solutions to PDEs as a function of the source term, or to solve inverse problems such as inference of initial conditions of dynamical field from its current configuration. They may be trained with or without a physics informed approach. However, the these models cannot be used on irregular domains.
Using graph neural networks, one can extend these approaches to irregular domains using a volumetric mesh. Pfaff et al. [11] build a model to predict the result of an expensive simulation on a mesh, by training on data generated by the simulation. However, this approach does not encode physical inductive biases as PINNs do, using a standard supervised approach. Hamiltonian graph neural networks [14] can be used on meshes similarly, with the difference that they explicitly encode energy conservation by simulating a Hamiltonian system.
Kashefi and Mukerji [4] have proposed a model to approximate the solution to a given PDE on multiple irregular geometries. Their model is trained in a physics informed manner and is based on PointNet, which is a neural operator designed for processing point clouds. These point clouds form an implicit representation of the geometry and their model can be used to make predictions on new geometries just by generating interior samples. However, compared to point clouds, meshes contain far more geometric information [2] such as local connectivity and curvature.
In this paper, we propose a novel neural operator architecture for the purposes of learning PDE solutions as a function of boundary geometry without restricting the geometry to a fixed, finite-dimensional parameterization. This provides a variety of benefits and significantly extends the possible applications and reusability of trained models over other PINNs. We believe this is an important step toward feasible PINN technology at industry scale, and reducing time and resource costs associated with traditional simulations.
We showed that our neural operator is capable of approximating the PDE solution operator, even when trained without simulation data or more advanced PINN methodologies. While we chose not to incorporate more sophisticated training techniques for the sake of simplicity and proof of concept, we highly recommend that they be used in any practical application, along with example data from simulations to improve the reliability and robustness of the model. We believe this is especially important as the geometry and domain of interest increase in complexity.
There are a variety of limitations with using neural networks to approximate solutions to PDEs. The subject itself is still relatively immature when compared to traditional methods like FEM or finite volume methods, and lacks the theoretical guarantees and wealth of insight that have been developed for these methods. Problems of exploding or vanishing gradients are exacerbated by the triple, or even higher order derivatives appearing from the optimization of Equation (3). When training on the PDE itself, loss curves may vary by multiple orders of magnitude from one epoch to the next, sometimes increasing even when the error of the solution is decreasing. This makes the learning curves very difficult to interpret. Furthermore, small loss values do not necessarily mean that the solution itself is accurate, and this complicates error estimation and uncertainty quantification.
For our particular experiments, we found that the lowest errors for each validation mesh occurred at different epochs and so while we were able to get very accurate results (occasionally even below  $1\%$  MSE) for each individual validation mesh, if we monitored its error and used early stopping, we were not able to achieve such accurate results on each mesh simultaneously. We believe this is mostly due to the difficulties of fitting to the PDE directly rather than a limitation of the model itself, and we expect to achieve better results with supervised learning on data from simulations in addition to the PINN loss function.
This work was sponsored by InstaDeep Ltd.
[1] S. Greydanus, M. Dzamba, and J. Yosinski. Hamiltonian Neural Networks. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https://proceedings.neurips.cc/paper/2019/file/26cd8ecadce0d4efd6cc8a8725cbd1f8-Paper.pdf.
[2] R. Hanocka, A. Hertz, N. Fish, R. Giryes, S. Fleishman, and D. Cohen-Or. MeshCNN: A network with an edge. ACM Transactions on Graphics (TOG), 38(4):90:1-90:12, 2019.
[3] O. Hennigh, S. Narasimhan, M. A. Nabian, A. Subramaniam, K. Tangsali, M. Rietmann, J. del Aguila Ferrandis, W. Byeon, Z. Fang, and S. Choudhry. Nvidia simnet: an ai-accelerated multi-physics simulation framework, 2020.
[4] A. Kashefi and T. Mukerji. Physics-informed pointnet: A deep learning solver for steady-state incompressible flows and thermal fields on multiple sets of irregular geometries. Journal of Computational Physics, 468:111510, 2022. ISSN 0021-9991. doi: https://doi.org/10.1016/j.jcp.2022.111510. URL https://www.sciencedirect.com/science/article/pii/S0021999122005721.
[5] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. CoRR, abs/1412.6980, 2014.
[6] N. Kovachki, Z. Li, B. Liu, K. Azizzadenesheli, K. Bhattacharya, A. Stuart, and A. Anandkumar. Neural operator: Learning maps between function spaces, 2023.
[7] Z. Li, N. Kovachki, K. Azizzadenesheli, B. Liu, K. Bhattacharya, A. Stuart, and A. Anandkumar. Fourier neural operator for parametric partial differential equations, 2021.
[8] L. Liu, H. Jiang, P. He, W. Chen, X. Liu, J. Gao, and J. Han. On the variance of the adaptive learning rate and beyond, 2021.
[9] L. Lu, P. Jin, G. Pang, Z. Zhang, and G. E. Karniadakis. Learning nonlinear operators via Deep-ONet based on the universal approximation theorem of operators. Nature Machine Intelligence, 3(3):218229, 2021. doi: 10.1038/s42256-021-00302-5.
[10] M. Lutter, C. Ritter, J. Peters, C. Science, T. Universit, D. Hochschulstr, and L. Mechanics. Deep Lagrangian Networks: Using Physics as Model Prior for Deep Learning. Iclr.
[11] T. Pfaff, M. Fortunato, A. Sanchez-Gonzalez, and P. Battaglia. Learning mesh-based simulation with graph networks. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=roNqYLO_XP.
[12] M. Raissi, P. Perdikaris, and G. E. Karniadakis. Physics informed deep learning (part i): Data-driven solutions of nonlinear partial differential equations.
[13] J. N. Reddy. An introduction to the finite element method. McGraw-Hill, fourth edition, 2019.
[14] A. Sanchez-Gonzalez, V. Bapst, K. Cranmer, and P. Battaglia. Hamiltonian Graph Networks with ODE Integrators, 2019.
[15] J. Sirignano and K. Spiliopoulos. DGM: A deep learning algorithm for solving partial differential equations. Journal of Computational Physics, 375:1339-1364, 2018. ISSN 0021-9991. doi: https://doi.org/10.1016/j.jcp.2018.08.029. URL https://www.sciencedirect.com/science/article/pii/S0021999118305527.
[16] N. Sukumar and A. Srivastava. Exact imposition of boundary conditions with distance functions in physics-informed deep neural networks. Computer Methods in Applied Mechanics and Engineering, 389:114333, 2022. ISSN 0045-7825. doi: https://doi.org/10.1016/j.cma.2021.114333. URL https://www.sciencedirect.com/science/article/pii/S0045782521006186.
[17] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf.
[18] H. Wang and X. Zhou. Explicit estimation of derivatives from data and differential equations by gaussian process regression.
[19] S. Wang, Y. Teng, and P. Perdikaris. Understanding and mitigating gradient flow pathologies in physics-informed neural networks. SIAM Journal on Scientific Computing, 43(5):A3055-A3081, 2021. doi: 10.1137/20M1318043. URL https://doi.org/10.1137/20M1318043.
[20] S. Wang, S. Sankaran, and P. Perdikaris. Respecting causality is all you need for training physics-informed neural networks, 2022.
[21] K. D. Willis, P. K. Jayaraman, H. Chu, Y. Tian, Y. Li, D. Grandi, A. Sanghi, L. Tran, J. G. Lambourne, A. Solar-Lezama, and W. Matusik. Joinable: Learning bottom-up assembly of parametric cad joints. arXiv preprint arXiv:2111.12772, 2021.
[22] S. Yang, S. W. K. Wong, and S. C. Kou. Inference of dynamic systems from noisy and sparse data via manifold-constrained gaussian processes.
[23] Y. Zang, G. Bao, X. Ye, and H. Zhou. Weak adversarial networks for high-dimensional partial differential equations. Journal of Computational Physics, 411:109409, 2020. ISSN 0021-9991. doi: https://doi.org/10.1016/j.jcp.2020.109409. URL https://www.sciencedirect.com/science/article/pii/S0021999120301832.
A Vanilla PINN results
For a few selected test meshes, we attempted training a standard PINN, without conditioning on the geometry. Here we show a comparison of these PINNs to our trained neural operator. The training code is identical for both the operator and standard PINNs, with the only differences being that the PINN model does not include any mesh encoding or attention layers, and the dataset is restricted to only a single mesh.
The PINN model architecture consists of 5 blocks, each of which is a fully connected feed-forward network with a skip connection and SiLU nonlinearities. The inputs and outputs of each block are of size 512, and each feed-forward network has a hidden-layer size of 2048. We use the linear combination of Fourier features and SiLU embeddings in the first layer. These parameters are chosen to match our neural operator for fair comparison. This model has 12.6 million parameters.
In all cases, the standard PINN fails miserably, producing far less accurate results than the neural operator. It appears that in this case, the process of learning on all geometries simultaneously improves the ability of the model to find an accurate solution to the PDE. Further investigation would need to be done to assess the extent to which this behaviour occurs, but we leave that for future work.
Figure 6: Comparison to vanilla PINNs on the Poisson equation
(c) PINN
Figure 7: Comparison to vanilla PINNs on Helmholtz equation
(c) PINN
Figure 8: Comparison to vanilla PINNs on reaction diffusion equation