SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $\rho$  denotes the mass density,  $\pmb{u}$  and  $\sigma$  represent the corresponding displacement and stress fields, respectively. The computational domain is enclosed by a unit cell  $[0,1]^2$  of which the center exists a randomly shaped void, as described by its radius  $r = 0.2 + \frac{0.2}{1 + \exp(\tilde{r})}$  and  $\tilde{r} \sim \mathbb{N}(0,4^2 (-\nabla +3^2)^{-1})$ . The bottom edge is fixed and the top edge is subjected to a tensile traction of  $\pmb{t} = [0,100]$ . The underlying hyperelastic material is of the incompressible Rivlin-Saunders type. For training, we directly adopt the dataset in Li et al. (2022a), where a total of 1000, 200, 200 samples are selected for training, validation and testing, respectively. For this problem, the input is represented as point clouds and the target is the resulting stress field.
Table 1: The total number of parameters (in millions) of selected models for hyperelasticity dataset.
model	eDAFNO	iDAFNO	FNO	IFNO	Geo-FNO	GNO	DeepONet	UNet	F-FNO
nparam	2.37	0.60	2.37	0.60	3.02	2.64	3.10	3.03	3.21
Table 2: Test errors for the hyperelasticity problem, where bold numbers highlight the best method.
Model	# of training samples
10	100	1000
Proposed model	eDAFNO	16.446%±0.472%	4.247%±0.066%	1.094%±0.012%
iDAFNO	16.669%±0.523%	4.214%±0.058%	1.207%±0.006%
Baseline model	FNO w/ mask	19.487%±0.633%	7.852%±0.130%	4.550%±0.062%
IFNO w/ mask	19.262%±0.376%	7.700%±0.062%	4.481%±0.022%
Geo-FNO	28.725%±2.600%	10.343%±4.446%	2.316%±0.283%
GNO	29.305%±0.321%	18.574%±0.584%	13.007%±0.729%
DeepONet	35.334%±0.179%	25.455%±0.245%	11.998%±0.786%
F-FNO	35.672%±3.852%	12.135%±5.813%	3.193%±1.622%
UNet	98.167%±0.236%	34.467%±2.858%	5.462%±0.048%
Ablation study	FNO w/ smooth χ	17.431%±0.536%	5.479%±0.186%	1.415%±0.025%
IFNO w/ smooth χ	17.145%±0.432%	5.088%±0.146%	1.509%±0.018%
Ablation study We first carry out an ablation study by comparing (a) the proposed two DAFNO models with (b) the baseline FNO/IFNO with the sharp characteristic function as input (denoted as FNO/IFNO w/ mask), and (c) the original FNO/IFNO with our proposed smoothened boundary
characteristic function as input (denoted as FNO/IFNO w/ smooth  $\chi$ ). In this study, scenarios (b) and (c) aim to investigate the effects of our proposed boundary smoothing technique, and by comparing scenarios (a) and (c) we verify the effectiveness of encoding the boundary information in eDAFNO architecture. In addition, both FNO- and IFNO-based models are tested, with the purpose to evaluate the model expressivity when using layer-independent parameters in iterative layers. Three training dataset sizes (i.e., 10, 100, and 1000) are employed to explore the effect of the proposed algorithms on small, medium, and large datasets, respectively. The number of trainable parameters are reported in Table 1. Following the common practice as in Li et al. (2022a), the hyperparameter choice of each model is selected by tuning the number of layers and the width (channel dimension) keeping the total number of parameters of the same magnitude.
The results of the ablation study are listed in Table 2. Firstly, by directly comparing the results of FNO (and IFNO) with mask and with smooth boundary encoding, one can tell that the boundary smoothing technique helps to reduce the error. This is supported by the observation that FNO and IFNO with smooth  $\chi$  consistently outperform their counterparts with mask in all data regimes, especially when a sufficient amount of data becomes available where a huge boost in accuracy can be achieved (by over  $300\%$ ). On the other hand, by encoding the geometry information into the iterative layer, the prediction accuracy is further improved, where eDAFNO and iDAFNO outperform FNO and IFNO with smoothed  $\chi$  by  $22.7\%$  and  $20.0\%$  in large-data regime, respectively. Another interesting finding is that eDAFNO is  $9.4\%$  more accurate compared to iDAFNO in the large-data regime, although only a quarter of the total number of parameters is needed in iDAFNO due to its layer-independent parameter setting. This effect is less pronounced as we reduce the amount of available data for training, where the performance of iDAFNO is similar to that of eDAFNO in the small- and medium-data regimes. This is because iDAFNO has a much smaller number of trainable parameters and therefore is less likely to overfit with small datasets. Given that the performance of eDAFNO and iDAFNO is comparable, it is our opinion that both architectures are useful in different applications. In Figure 3, an example of the computational domain, the smoothened boundary encoding, the ground truth solution, and the eDAFNO prediction are demonstrated. To demonstrate the capability of prediction across resolutions, we train eDAFNO using data with  $41\times 41$  grids then apply the model to provide prediction on  $161\times 161$  grids—one can see that eDAFNO can generalize across different resolutions.
Comparison against additional baselines We further compare the performance of DAFNO against additional relevant baselines, including GNO, Geo-FNO, F-FNO, DeepONet, and UNet. Note that the results of GNO, DeepONet, and UNet are obtained using the same settings as in Li et al. (2022a). Overall, the two DAFNO variants are significantly superior to other baselines in accuracy, with eDAFNO outperforming GNO, DeepONet, UNet, Geo-FNO, and F-FNO by  $1088.9\%$ ,  $975.0\%$ ,  $399.3\%$ ,  $111.7\%$ , and  $191.9\%$ , respectively, in large-data regime. DAFNO is also more memory efficient compared to Geo-FNO (the most accurate baseline), as it foregoes the need for additional coordinate deformation network. As shown in Table 1, when the layer-independent parameter setting in iDAFNO is taken into account, DAFNO surpasses Geo-FNO by  $407.4\%$  in memory saving.
4.2 Airfoil design
Figure 4: An illustration on a test sample from the airfoil design problem. From left to right: an illustration of the discretized computational domain, the smoothed boundary encoding (i.e., smoothed  $\chi$ ), the ground truth, and the eDAFNO prediction.
In this example, we investigate DAFNO's performance in learning transonic flow around various airfoil designs. Neglecting the effect of viscosity, the underlying physics can be described by the Euler equation:
$$
\frac {\partial \rho}{\partial t} + \nabla \cdot (\rho \boldsymbol {v}) = 0, \quad \frac {\partial \rho \boldsymbol {v}}{\partial t} + \nabla \cdot (\rho \boldsymbol {v} \otimes \boldsymbol {v} + p \mathbb {I}) = 0, \quad \frac {\partial E}{\partial t} + \nabla \cdot ((E + p) \boldsymbol {v}) = 0, \tag {9}
$$
Table 3: Results for the airfoil design problem, where bold numbers highlight the best method.
Model	Train error	Test error
Proposed model	eDAFNO	0.329%±0.020%	0.596%±0.005%
iDAFNO	0.448%±0.012%	0.642%±0.020%
eDAFNO on irregular grids	0.331%±0.003%	0.659%±0.007%
Baseline model	Geo-FNO	1.565%±0.180%	1.650%±0.175%
F-FNO	0.566%±0.066%	0.794%±0.025%
FNO w/ mask	2.676%±0.054%	3.725%±0.108%
UNet w/ mask	2.781%±1.084%	4.957%±0.059%
with  $\rho, p$  and  $E$  being the fluid density, pressure and the total energy, respectively, and  $\mathbf{v}$  denoting the corresponding velocity field. The applied boundary conditions are:  $\rho_{\infty} = 1$ , Mach number  $M_{\infty} = 0.8$ , and  $p_{\infty} = 1$  on the far field, with no penetration enforced on the airfoil. The dataset used for training is directly taken from Li et al. (2022a), which consists of variations of the NACA-0012 airfoil and is divided into 1000, 100, 100 samples for training, validation and testing, respectively. For this problem, we aim to learn the resulting Mach number field based on a given mesh as input. An example of the computational domain, the smoothened boundary encoding, the ground truth, and the eDAFNO prediction is illustrated in Figure 4.
We report in Table 3 our experimental observations using eDAFNO and iDAFNO, along with the comparison against FNO, Geo-FNO, F-FNO and UNet, whose models are directly attained from Li et al. (2022a); Tran et al. (2022). We can see that the proposed eDAFNO achieves the lowest error on the test dataset, beating the best result of non-DAFNO baselines by  $24.9\%$  and Geo-FNO by  $63.9\%$ , respectively. Additionally, iDAFNO reaches a similar level of accuracy with only a quarter of the total number of parameters employed, which is consistent with the findings in the previous example. With the properly trained DAFNO models, efficient optimization and inverse design are made possible.
Figure 5: eDAFNO applied to the airfoil dataset on irregular grids. (a): the highly irregular and adaptive grids in the physical space is firstly deformed (via either an analytical mapping  $f$  or a trainable neural network for grid deformation) to uniform grids in the computational space, on which eDAFNO is applied to learn the underlying physics. The learned prediction is then converted back to the physical space via the inverse mapping  $f^{-1}$  or with another trainable neural network. (b): an illustration of the absolute error distribution of predictions in Geo-FNO, eDAFNO trained on uniform grids, and eDAFNO trained on irregular grids.
Aside from the enhanced predictability, DAFNO can also be easily combined with the grid mapping technique in Geo-FNO to handle non-uniform grids. In particular, no modification on the model architecture is required, where one just needs to include an analytical or trainable mapping from
non-uniform/irregular mesh grids to uniform mesh grids. As a demonstration, we consider irregular grids of the airfoil dataset and use a pre-computed function to map irregular grids to regular grids. In the irregular grid set, we place more grid points near the airfoil to provide a better resolution near the important parts. The test error is provided in Table 3, where the test loss using the eDAFNO learned model is  $0.659\% \pm 0.007\%$ , which is similar to the DAFNO results on uniform grids. In Figure 5, we demonstrate the irregular mesh and the absolute error comparison across Geo-FNO, DAFNO on regular grids, and DAFNO on irregular grids. One can see that, while both DAFNOs substantially outperform Geo-FNO, the error contours from eDAFNO with an irregular mesh show a smaller miss-match region near the airfoil, illustrating the flexibility of DAFNO in meshing and its capability in resolving fine-grained features.
4.3 Crack propagation with topology change in domain
In this example, we aim to showcase DAFNO's capability in handling evolving domains by modeling crack propagation in brittle fracture. We emphasize that DAFNO represents the first neural operator that allows for learning with topology change. In the field of brittle fracture, a growing crack can be viewed as a change in topology, which corresponds to an evolving  $\chi(t)$  in the DAFNO architecture. In particular, we define the following time-dependent characteristic function:
$$
\chi (\boldsymbol {x}, t) = \left\{ \begin{array}{l l} 1 & \boldsymbol {x} \in \Omega (t) \\ 0 & \boldsymbol {x} \in \mathbb {T} \backslash \Omega (t) \end{array} , \right. \tag {10}
$$
where  $\Omega(t)$  denotes the time-dependent domain/evolving topology. Employing time-dependent  $\chi$  in (6) keeps the neural operator informed about the evolving topology. In general, the topology evolution rule that determines  $\Omega(t)$  can be obtained from a separate neural network or from physics as is the case in the current example. The high-fidelity synthetic data in this example is generated using 2D PeriFast software (Jafarzadeh et al., 2022a), which employs a peridynamics (PD) theory for modeling fracture (Bobaru et al., 2016). A demonstration of the evolving topology, as well as further details regarding the governing equations and data generation strategies, is provided in Appendix B.3.
In this context, we select eDAFNO as the surrogate model to learn the the internal force density operator. Specifically, let  $u_{1}(\boldsymbol{x},t)$ ,  $u_{2}(\boldsymbol{x},t)$  denote the two components of the displacement field  $\boldsymbol{u}$  at time  $t$ , and  $L_{1}$ ,  $L_{2}$  be the two components of the internal force density  $\mathcal{L}[\boldsymbol{u}]$ . Given  $u_{1}$ ,  $u_{2}$  and  $\chi$  as input data, we train two separate eDAFNO models to predict  $L_{1}$  and  $L_{2}$ , respectively. Then, we substitute the trained surrogate models in the dynamic governing equation and adopt Velocity-Verlet time integration to update  $\boldsymbol{u}$  for the next time step, whereafter  $\Omega$  and  $\chi$  are updated accordingly.
The problem of interest is defined on a  $40\mathrm{mm}\times 40\mathrm{mm}$  thin plate with a pre-crack of length  $10\mathrm{mm}$  at one edge, which is subjected to sudden, uniform, and constant tractions on the top and bottom edges. Depending on the traction magnitude, crack grows at different speeds, may or may not bifurcate, and the final crack patterns can be of various shapes. Our training data is comprised of two parts, one consisting of 450 snapshots from a crack propagation simulation with a fixed traction magnitude of  $\sigma = 4\mathrm{MPa}$ , and the other consisting of randomized sinusoidal displacement fields and the corresponding  $L_{1}, L_{2}$  fields computed by the PD operator. The sinusoidal subset contains 4,096 instances without fracture and is used to diversify the training data and mitigate overfitting on the crack data. For testing, we use the trained models in two scenarios with traction magnitudes different from what is used in training, which allows us to evaluate the generalizability of eDAFNO. Note that these tests are highly challenging for the trained models, as the predictions of the previous time step are used as the models' input in the current step, which leads to error accumulation as the simulation marches forward.
Figure 6 displays the test results on the crack patterns under different traction magnitudes, where the low traction magnitude (left column) shows a slow straight crack, the mid-level traction results in a crack with a moderate speed and a bifurcation event (middle column), and the highest traction leads to a rapid crack growth and initiates a second crack from the other side of the plate (right column). The trained eDAFNO model is able to accurately predict the left and right examples while only seeing the middle one, indicating that it has correctly learned the true constitutive behavior of the material and generalized well to previously unseen loading scenarios and the correspondingly changed domain topology. To further verify eDAFNO's generalizability to unseen geometries, in Figure 7 we compare eDAFNO with the baseline FNO w/ smoothed mask model, and plot their relative errors in  $\chi$  and  $u$ . As observed, the FNO predictions become unstable at a fairly early time in the two test scenarios,
Figure 6: Comparison of the fracture patterns with different loading scenarios between the high-fidelity solution and eDAFNO prediction.
Figure 7: Comparison of relative errors in the characteristic function  $\chi$  (left) and displacement fields (right) of the evolving topology predicted using the trained eDAFNO and FNO with mask at each time step for the training (loading  $= 4\mathrm{MPa}$ ) and testing (loading  $= 2\mathrm{MPa}$  and  $6\mathrm{MPa}$ ) scenarios.
since it creates a mask which is out of training distribution, while DAFNO can handle different crack patterns by hard coding it in the architecture, so the results remain stable for a much longer time.
We introduce two DAFNO variants to enable FNOs on irregular domains for PDE solution operator learning. By incorporating the geometric information from a smoothed characteristic function in the iterative Fourier layer while retaining the convolutional form, DAFNO possesses the computational efficiency from FFT together with the flexibility to operate on different computational domains. As a result, DAFNO is not only highly efficient and flexible in solving problems involving arbitrary shapes, but it also manifests its generalizability on evolving domains with topology change. In two benchmark datasets and a real-world crack propagation dataset, we demonstrate the state-of-the-art performance of DAFNO. We find both architectures helpful in practice: eDAFNO is slightly more accurate while iDAFNO is more computationally efficient and less overfitting with limited data.
**Limitation:** Due to the requirement of the standard FFT package, in the current DAFNO we focus on changing domains with uniformly meshed grids. However, we point out that this limitation can be lifted by using nonuniform FFT (Greengard & Lee, 2004) or an additional mapping for grid deformation, as shown in the airfoil experiment. Additionally, in our applications, we have focused on applying the same types of boundary conditions to the changing domains (e.g., all Dirichlet or all Neumann). In this context, another limitation and possible future direction would be on the transferability to PDEs with different types of boundary conditions.
S. Jafarzadeh would like to acknowledge support by the AFOSR grant FA9550-22-1-0197, and Y. Yu would like to acknowledge support by the National Science Foundation under award DMS-1753031 and the AFOSR grant FA9550-22-1-0197. Portions of this research were conducted on Lehigh University's Research Computing infrastructure partially supported by NSF Award 2019035.
Agwai, A., Guven, I., and Madenci, E. Predicting crack propagation with peridynamics: a comparative study. International journal of fracture, 171(1):65, 2011.
Benson, D. J. and Okazawa, S. Contact in a multi-material Eulerian finite element formulation. Computer Methods in Applied Mechanics and Engineering, 193(39):4277-4298, 2004. ISSN 0045-7825.
Besnard, G., Hild, F., and Roux, S. "finite-element" displacement fields analysis from digital images: application to portevin-le châtelier bands. Experimental mechanics, 46(6):789-803, 2006.
Bobaru, F., Foster, J. T., Geubelle, P. H., and Silling, S. A. Handbook of peridynamic modeling. CRC press, 2016.
Brigham, E. O. The fast Fourier transform and its applications. Prentice-Hall, Inc., 1988.
Cai, S., Mao, Z., Wang, Z., Yin, M., and Karniadakis, G. E. Physics-informed neural networks (PINNs) for fluid mechanics: A review. Acta Mechanica Sinica, pp. 1-12, 2022.
Cao, S. Choose a transformer: Fourier or galerkin. Advances in neural information processing systems, 34:24924-24940, 2021.
Carleo, G., Cirac, I., Cranmer, K., Daudet, L., Schuld, M., Tishby, N., Vogt-Maranto, L., and Zdeborova, L. Machine learning and the physical sciences. Reviews of Modern Physics, 91(4): 045002, 2019.
Day, S. J., Mullineux, N., and Reed, J. Developments in obtaining transient response using fourier transforms: Part i: Gibbs phenomena and fourier integrals. International Journal of Electrical Engineering Education, 3(4):501-506, 1965.
De Luycker, E., Benson, D. J., Belytschko, T., Bazilevs, Y., and Hsu, M.-C. X-FEM in isogeometric analysis for linear fracture mechanics. International Journal for Numerical Methods in Engineering, 87:541-565, 2011.
Ghaboussi, J., Garrett Jr, J., and Wu, X. Knowledge-based modeling of material behavior with neural networks. Journal of engineering mechanics, 117(1):132-153, 1991.
Ghaboussi, J., Pecknold, D. A., Zhang, M., and Haj-Ali, R. M. Autoprogressive training of neural network constitutive models. International Journal for Numerical Methods in Engineering, 42(1): 105-126, 1998.
Goswami, S., Bora, A., Yu, Y., and Karniadakis, G. E. Physics-informed neural operators. 2022 arXiv preprint arXiv:2207.05748, 2022.
Gottlieb, D. and Shu, C.-W. On the gibbs phenomenon and its resolution. SIAM review, 39(4): 644-668, 1997.
Greengard, L. and Lee, J.-Y. Accelerating the nonuniform fast fourier transform. SIAM review, 46(3): 443-454, 2004.
Gupta, G., Xiao, X., and Bogdan, P. Multiwavelet-based operator learning for differential equations. Advances in Neural Information Processing Systems, 34:24048-24062, 2021.
Ha, Y. D. and Bobaru, F. Studies of dynamic crack propagation and crack branching with peridynamics. International Journal of Fracture, 162(1):229-244, 2010.
Ong, Y. Z., Shen, Z., and Yang, H. IAE-NET: Integral autoencoders for discretization-invariant learning. 03 2022. doi: 10.13140/RG.2.2.25120.87047/2.
Pfau, D., Spencer, J. S., Matthews, A. G., and Foulkes, W. M. C. Ab initio solution of the many-electron schrödinger equation with deep neural networks. Physical Review Research, 2(3):033429, 2020.
Ronneberger, O., Fischer, P., and Brox, T. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention, pp. 234-241. Springer, 2015.
Shadden, S. C., Astorino, M., and Gerbeau, J.-F. Computational analysis of an aortic valve jet with Lagrangian coherent structures. *Chaos*, 20:017512, 2010.
Silling, S. A. Dynamic fracture modeling with a meshfree peridynamic code. Computational Fluid and Solid Mechanics, 2003:641-644, 2003.
Simo, J. C. and Laursen, T. A. An augmented Lagrangian treatment of contact problems involving friction. Computers & Structures, 42(1):97-116, 1992. ISSN 0045-7949.
Tran, A., Mathews, A., Xie, L., and Ong, C. S. Factorized fourier neural operators. In The Eleventh International Conference on Learning Representations, 2022.
Tripura, T. and Chakraborty, S. Wavelet neural operator: a neural operator for parametric partial differential equations. arXiv preprint arXiv:2205.02191, 2022.
Yin, M., Ban, E., Rego, B. V., Zhang, E., Cavinato, C., Humphrey, J. D., and Em Karniadakis, G. Simulating progressive intramural damage leading to aortic dissection using DeepONet: an operator-regression neural network. Journal of the Royal Society Interface, 19(187):20210670, 2022a.
Yin, M., Zhang, E., Yu, Y., and Karniadakis, G. E. Interfacing finite elements with deep neural operators for fast multiscale modeling of mechanics problems. Computer Methods in Applied Mechanics and Engineering, in press, pp. 115027, 2022b.
Yin, Y., Kirchmeyer, M., Franceschi, J.-Y., Rakotomamonjy, A., et al. Continuous pde dynamics forecasting with implicit neural representations. In The Eleventh International Conference on Learning Representations, 2022c.
You, H., Yu, Y., D'Elia, M., Gao, T., and Silling, S. Nonlocal kernel network (NKN): A stable and resolution-independent deep neural network. Journal of Computational Physics, pp. arXiv preprint arXiv:2201.02217, 2022a.
You, H., Zhang, Q., Ross, C. J., Lee, C.-H., Hsu, M.-C., and Yu, Y. A physics-guided neural operator learning approach to model biological tissues from digital image correlation measurements. Journal of Biomechanical Engineering, accepted for publication, pp. arXiv preprint arXiv:2204.00205, 2022b.
You, H., Zhang, Q., Ross, C. J., Lee, C.-H., and Yu, Y. Learning deep implicit fourier neural operators (IFNOs) with applications to heterogeneous material modeling. Computer Methods in Applied Mechanics and Engineering, 398:115296, 2022c. doi: https://doi.org/10.1016/j.cma.2022.115296.
Zhang, L., Han, J., Wang, H., Car, R., and Weinan, E. Deep potential molecular dynamics: a scalable model with the accuracy of quantum mechanics. Physical Review Letters, 120(14):143001, 2018.