SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Ning Liu*
Global Engineering and
Materials, Inc.
Princeton, NJ 08540
ningliu@umich.edu
Siavash Jafarzadeh*
Bethlehem, PA 18015
sij222@lehigh.edu
Yue  $\mathbf{Y}\mathbf{u}^{\dagger}$
yuy214@lehigh.edu
Fourier neural operators (FNOs) can learn highly nonlinear mappings between function spaces, and have recently become a popular tool for learning responses of complex physical systems. However, to achieve good accuracy and efficiency, FNOs rely on the Fast Fourier transform (FFT), which is restricted to modeling problems on rectangular domains. To lift such a restriction and permit FFT on irregular geometries as well as topology changes, we introduce domain agnostic Fourier neural operator (DAFNO), a novel neural operator architecture for learning surrogates with irregular geometries and evolving domains. The key idea is to incorporate a smoothed characteristic function in the integral layer architecture of FNOs, and leverage FFT to achieve rapid computations, in such a way that the geometric information is explicitly encoded in the architecture. In our empirical evaluation, DAFNO has achieved state-of-the-art accuracy as compared to baseline neural operator models on two benchmark datasets of material modeling and airfoil simulation. To further demonstrate the capability and generalizability of DAFNO in handling complex domains with topology changes, we consider a brittle material fracture evolution problem. With only one training crack simulation sample, DAFNO has achieved generalizability to unseen loading scenarios and substantially different crack patterns from the trained scenario. Our code and data accompanying this paper are available at https://github.com/ningliu-iga/DAFNO.
Deep learning surrogate models provide a useful data-driven paradigm to accelerate the PDE-solving and calibration process of scientific modeling and computing problems. Among others, a wide range of scientific computing applications entail the learning of solution operators, i.e., the learning of infinite dimensional function mappings between any parametric dependence to the solution field. A prototypical instance is the case of solving Navier-Stokes equations in fluid mechanics, where the initial input needs to be mapped to a temporal sequence of nonlinear parametric solutions. The demand for operator learning has sparked the development of neural operator based methods (Li et al., 2020a,b,c; You et al., 2022a,b,c; Goswami et al., 2022; Liu et al., 2022; Gupta et al., 2021; Lu et al., 2019; Cao, 2021; Hao et al., 2023; Li et al., 2022b; Yin et al., 2022c), with one of the most popular architectures being Fourier Neural Operators (FNOs) (Li et al., 2020c; You et al., 2022c).
The success of FNOs can be mostly attributed to its convolution-based integral kernel that learns in a resolution-invariant manner and the computationally efficient evaluation achieved via Fast Fourier Transform (FFT) (Brigham, 1988). While learning in the spectral domain is fast, the latter comes at a cost: the computational domain of the underlying problem needs to be rectangular with uniformly
meshed grids. This is often intractable as the domain of interest is, more often than not, irregular. An often taken trick for applying FNO to irregular domains is to embed the original domain into a larger rectangular domain and zero-pad or extrapolate on the redundant space (Lu et al., 2022). This poses two potential problems, one being the possible numerical errors and even instabilities due to the discontinuity at the original domain boundary (e.g., the Gibbs phenomenon (Gottlieb & Shu, 1997)) and the other, perhaps more importantly, being the fact that the padding/extrapolating techniques cannot handle domains with shallow gaps, as is the case in object contact and crack propagation problems. Meanwhile, another line of work emphasizes the learning of a diffeomorphic mapping between the original domain and a latent domain with uniform grids on which FNO can be applied (Li et al., 2022a). However, in this approach the changes on the boundary and the domain topology can only be informed via the learned diffeomorphism, which results in approximation errors when tested on a new domain geometry and possible failure when a change in topology is involved on the domain geometry.
In this work, we aim to design FNO architectures that explicitly embed the boundary information of irregular domains, which we coin Domain Agnostic Fourier Neural Operator (DAFNO). This is inspired by the recent work in convolution-based peridynamics (Jafarzadeh et al., 2022b) in which bounded domains of arbitrary shapes are explicitly encoded in the nonlocal integral formulation. We argue that, by explicitly embedding the domain boundary information into the model architecture, DAFNO is able to learn the underlying physics more accurately, and the learnt model is generalizable to changes on the domain geometry and topology. Concretely, we construct two practical DAFNO variants, namely, eDAFNO that inherits the explicit FNO architecture (Li et al., 2022a) and iDAFNO that is built upon the implicit FNO (IFNO) architecture characterizing layer-independent kernels (You et al., 2022c). Moreover, a boundary smoothening technique is also proposed to resolve the Gibbs phenomenon and retain the fidelity of the domain boundary. In summary, the primary contributions of the current work are as follows:
- We propose DAFNO, a novel Fourier neural operator architecture that explicitly encodes the boundary information of irregular domains into the model architecture, so that the learned operator is aware of the domain boundary, and generalizable to different domains of complicated geometries and topologies.
- By incorporating a (smoothened) domain characteristic function into the integral layer, our formulation resembles a nonlocal model, such that the layer update acts as collecting interactions between material points inside the domain and cuts the non-physical influence outside the domain. As such, the model preserves the fidelity of the domain boundary as well as the convolution form of the kernel that retains the computational efficiency of FFT.
- We demonstrate the expressivity and generalizability of DAFNO across a wide range of scientific problems including constitutive modeling of hyperelastic materials, airfoil design, and crack propagation in brittle fracture, and show that our learned operator can handle not only irregular domains but also topology changes over the evolution of the solution.
The goal of this work is to construct a neural network architecture to learn common physical models on various domains. Formally, given  $\mathcal{D} \coloneqq \{(g_i|_{\Omega_i}, u_i|_{\Omega_i})\}_{i=1}^N$ , a labelled set of function pair observations both defined on the domain  $x \in \Omega_i \subset \mathbb{R}^s$ . We assume that the input  $\{g_i(x)\}$  is a set of independent and identically distributed (i.i.d.) random fields from a known probability distribution  $\mu$  on  $\mathcal{A}(\mathbb{R}^{d_g})$ , a Banach space of functions taking values in  $\mathbb{R}^{d_g}$ .  $u_i(x) \in \mathcal{U}(\mathbb{R}^{d_u})$ , possibly noisy, is the observed corresponding response taking values in  $\mathbb{R}^{d_u}$ . Taking mechanical response modeling problem for example,  $\Omega_i$  is the shape of the object of interest,  $g_i(x)$  may represent the boundary, initial, or loading conditions, and  $u_i(x)$  can be the resulting velocity, pressure, or displacement field of the object. We assume that all observations can be modeled by a common and possibly unknown governing law, e.g., balance laws, and our goal is to construct a surrogate operator mapping,  $\hat{\mathcal{G}}$ , from  $\mathcal{A}$  to  $\mathcal{U}$  such that
$$
\tilde {\mathcal {G}} [ \boldsymbol {g} _ {i}; \theta ] (\boldsymbol {x}) \approx \boldsymbol {u} _ {i} (\boldsymbol {x}), \forall \boldsymbol {x} \in \Omega_ {i}. \tag {1}
$$
Here,  $\theta$  represents the (trainable) network parameter set.
In real-world applications, the domain  $\Omega_{i}$  can possess different topologies, e.g., in contact problems (Benson & Okazawa, 2004; Simo & Laursen, 1992) and material fragmentation problems (De Luycker et al., 2011; Agwai et al., 2011; Silling, 2003), and/or evolve with time, as is the case
in large-deformation problems (Shadden et al., 2010) and fluid-structure interaction applications (Kuhl et al., 2003; Kamensky et al., 2017). Hence, it is desired to develop an architecture with generalizability across various domains of complex shapes, so that the knowledge obtained from one geometry can be transferable to other geometries.
2.1 Learning solution operators of hidden physics
In real-world physical problems, predicting and monitoring complex system responses are ubiquitous in many applications. For these purposes, physics-based PDEs and traditional numerical methods have been commonly employed. However, traditional numerical methods are solved for specific boundary, initial, and loading conditions  $\pmb{g}$  on a specific domain  $\Omega$ . Hence, the solutions are not generalizable to other domains and operating conditions.
To provide a more efficient and flexible surrogate model for physical response prediction, there has been significant progress in the development of deep neural networks (NNs) and scientific machine learning models (Ghaboussi et al., 1998, 1991; Carleo et al., 2019; Karniadakis et al., 2021; Zhang et al., 2018; Cai et al., 2022; Pfau et al., 2020; He et al., 2021; Besnard et al., 2006). Among others, neural operators (Li et al., 2020a,b,c; You et al., 2022a; Ong et al., 2022; Gupta et al., 2021; Lu et al., 2019, 2021b; Goswami et al., 2022; Tripura & Chakraborty, 2022) show particular promise in learning physics of complex systems: compared with classical NNs, neural operators are resolution independent and generalizable to different input instances. Therefore, once the neural operator is trained, solving for a new instance of the boundary/initial loading condition with a different discretization only requires a forward pass. These advantages make neural operators a useful tool to many physics modeling problems (Yin et al., 2022a; Goswami et al., 2022; Yin et al., 2022b; Li et al., 2020a,b,c; Lu et al., 2022, 2021a).
2.2 Neural operator learning
Here, we first introduce the basic architecture of the general integral neural operators (Li et al., 2020a,b,c; You et al., 2022a,c), which are comprised of three building blocks. First, the input function,  $g(\pmb{x}) \in \mathcal{A}$ , is lifted to a higher-dimensional representation via  $h^0(\pmb{x}) = \mathcal{P}[g](\pmb{x}) \coloneqq P[\pmb{x}, g(\pmb{x})]^T + \pmb{p}$  where  $P \in \mathbb{R}^{(s + d_g) \times d_h}$  and  $\pmb{p} \in \mathbb{R}^{d_h}$  define an affine pointwise mapping. Then, the feature vector function  $h^0(\pmb{x})$  goes through an iterative layer block, where the layer update is defined via the sum of a local linear operator, a nonlocal integral kernel operator, and a bias function:  $h^{l+1}(\pmb{x}) = \mathcal{J}^{l+1}[h^l](\pmb{x})$ . Here,  $h^l(\pmb{x}) \in \mathbb{R}^{d_h}$ ,  $l = 0, \dots, L$ , is a sequence of functions representing the values of the network at each hidden layer.  $\mathcal{J}^1, \dots, \mathcal{J}^L$  are the nonlinear operator layers defined by the particular choice of networks. Finally, the output  $u(\pmb{x}) \in \mathcal{U}$  is obtained via a projection layer by mapping the last hidden layer representation  $h^L(\pmb{x})$  onto  $\mathcal{U}$  as:  $u(\pmb{x}) = \mathcal{Q}[h^L](\pmb{x}) \coloneqq Q_2\sigma(Q_1h^L(\pmb{x}) + q_1) + q_2$ .  $Q_1, Q_2, q_1$  and  $q_2$  are the appropriately sized matrices and vectors that are part of the learnable parameter set, and  $\sigma$  is an activation function (e.g., ReLU (He et al., 2018) or GeLU).
Then, the system response can be learnt by constructing a surrogate operator of (1):  $\tilde{\mathcal{G}}[\pmb{g};\theta](\pmb{x}) := \mathcal{Q} \circ \mathcal{J}^1 \circ \dots \circ \mathcal{J}^L \circ \mathcal{P}[\pmb{g}](\pmb{x}) \approx \pmb{u}(\pmb{x})$ , by solving the network parameter set  $\theta$  via an optimization problem:
$$
\min  _ {\theta \in \Theta} \mathcal {L} _ {\mathcal {D}} (\theta) := \min  _ {\theta \in \Theta} \sum_ {i = 1} ^ {N} \left[ C \left(\tilde {\mathcal {G}} [ \boldsymbol {g} _ {i}; \theta ], \boldsymbol {u} _ {i}\right) \right]. \tag {2}
$$
Here,  $C$  denotes a properly defined cost functional (e.g., the relative mean square error) on  $\Omega_{i}$ .
2.3 Fourier neural operators
The Fourier neural operator (FNO) is first proposed in Li et al. (2020c) with its iterative layer architecture given by a convolution operator:
$$
\mathcal {J} ^ {l} [ \boldsymbol {h} ] (\boldsymbol {x}) := \sigma \left(W ^ {l} \boldsymbol {h} (\boldsymbol {x}) + \boldsymbol {c} ^ {l} + \int_ {\Omega} \kappa (\boldsymbol {x} - \boldsymbol {y}; \boldsymbol {v} ^ {l}) \boldsymbol {h} (\boldsymbol {y}) d \boldsymbol {y}\right), \tag {3}
$$
where  $W^{l} \in \mathbb{R}^{d_{h} \times d_{h}}$  and  $c^{l} \in \mathbb{R}^{d_{h}}$  are learnable tensors at the  $l$ -th layer, and  $\kappa \in \mathbb{R}^{d_{h} \times d_{h}}$  is a tensor kernel function with parameters  $v^{l}$ . When a rectangular domain  $\Omega$  with uniform meshes is considered, the above convolution operation can be converted to a multiplication operation through discrete Fourier transform:
$$
\mathcal {J} ^ {l} [ \boldsymbol {h} ] (\boldsymbol {x}) = \sigma \left(W ^ {l} \boldsymbol {h} (\boldsymbol {x}) + \boldsymbol {c} ^ {l} + \mathcal {F} ^ {- 1} \left[ \mathcal {F} \left[ \kappa (\cdot ; \boldsymbol {v} ^ {l}) \right] \cdot \mathcal {F} [ \boldsymbol {h} (\cdot) ] \right] (\boldsymbol {x})\right),
$$
Figure 1: A schematic of extending an arbitrarily shaped domain  $\Omega$  to a periodic box  $\mathbb{T}$  and its discretized form in 2D.
where  $\mathcal{F}$  and  $\mathcal{F}^{-1}$  denote the Fourier transform and its inverse, respectively, which are computed using the FFT algorithm to each component of  $h$  separately. The FFT calculations greatly improve the computational efficiency due to their quasilinear time complexity, but they also restrict the vanilla FNO architecture to rectangular domains  $\Omega$  (Lu et al., 2022).
To enhance the flexibility of FNO in modeling complex geometries, in Lu et al. (2022) the authors proposed to pad and/or extrapolate the input and output functions into a larger rectangular domain. However, such padding/extrapolating techniques are prone to numerical instabilities (Gottlieb & Shu, 1997), especially when the domain is concave and/or with complicated boundaries. As shown in Lu et al. (2022), the performance of dgFNO+ substantially deteriorates when handling complicated domains with notches and gaps. In Geo-FNO (Li et al., 2022a), an additional neural network is employed and trained from data, to continuously map irregular domains onto a latent space of rectangular domains. As a result, the vanilla FNO can be employed on the rectangular latent domain. However, this strategy relies on the continuous mapping from the physical domain to a rectangular domain, hence it is restricted to relatively simple geometries with no topology change.
3 Domain Agnostic Fourier Neural Operators
In this section, we introduce Domain Agnostic Fourier Neural Operator (DAFNO), which features the generalizability to new and unseen domains of arbitrary shapes and different topologies. The key idea is to explicitly encode the domain information in the design while retaining the convolutional architecture in the iterative layer of FNOs. In what follows, we present the eDAFNO architecture based on the standard/explicit FNO model (Li et al., 2020c), while the iDAFNO architecture based on the implicit FNO model (You et al., 2022c) is provided in Appendix A.
Concretely, we enclose the physical domain of interest,  $\Omega$ , by a (slightly larger) periodic box  $\mathbb{T}$ , as shown in Figure 1. Next, we define the following domain characteristic function:
$$
\chi (\boldsymbol {x}) = \left\{ \begin{array}{l l} 1 & \boldsymbol {x} \in \Omega \\ 0 & \boldsymbol {x} \in \mathbb {T} \backslash \Omega \end{array} , \right. \tag {4}
$$
which encodes the domain information of different geometries. Inspired by Jafarzadeh et al. (2022b), we incorporate the above-encoded domain information into the FNO architecture of (3), by multiplying the integrand in its convolution integral with  $\chi (\pmb {x})\chi (\pmb {y})$  ..
$$
\mathcal {J} ^ {l} [ \boldsymbol {h} ] = \sigma \left(\int_ {\mathbb {T}} \chi (\boldsymbol {x}) \chi (\boldsymbol {y}) \kappa (\boldsymbol {x} - \boldsymbol {y}; \boldsymbol {v} ^ {l}) (\boldsymbol {h} (\boldsymbol {y}) - \boldsymbol {h} (\boldsymbol {x})) d \boldsymbol {y} + W ^ {l} \boldsymbol {h} (\boldsymbol {x}) + \boldsymbol {c} ^ {l}\right). \tag {5}
$$
Herein, we have followed the practice in You et al. (2022a) and reformulated (3) to a nonlocal Laplacian operator, which is found to improve training efficacy. By introducing the term  $\chi(\pmb{x})\chi(\pmb{y})$  the integrand vanishes when either point  $\pmb{x}$  or  $\pmb{y}$  is positioned inside  $\Omega$  and the other is positioned outside. This modification eliminates any undesired interaction between the regions inside and outside of  $\Omega$ . As a result, it tailors the integral operator to act on  $\Omega$  independently and is able to handle different domains and topologies. With this modification, the FFT remains applicable, since the convolutional structure of the integral is preserved and the domain of operation yet spans to the whole rectangular box  $\mathbb{T}$ . In this context, (5) can be re-organized as:
$$
\mathcal {J} ^ {l} [ \boldsymbol {h} ] = \sigma \left(\chi (\boldsymbol {x}) \left(\int_ {\mathbb {T}} \kappa (\boldsymbol {x} - \boldsymbol {y}; \boldsymbol {v} ^ {l}) \chi (\boldsymbol {y}) \boldsymbol {h} (\boldsymbol {y}) d \boldsymbol {y} - \boldsymbol {h} (\boldsymbol {x}) \int_ {\mathbb {T}} \kappa (\boldsymbol {x} - \boldsymbol {y}; \boldsymbol {v} ^ {l}) \chi (\boldsymbol {y}) d \boldsymbol {y} + W ^ {l} \boldsymbol {h} (\boldsymbol {x}) + \boldsymbol {c} ^ {l}\right)\right).
$$
Note that multiplying  $W^l \pmb{h}(\pmb{x}) + \pmb{c}^l$  with  $\chi(\pmb{x})$  does not alter the computational domain. Now that the integration region is a rectangular box  $\mathbb{T}$ , the FFT algorithm and its inverse can be readily applied,
Figure 2: An illustration of the proposed DAFNO architecture. We start from the input function  $\pmb{g}(\pmb{x})$ . After lifting, the iterative Fourier layers are built that explicitly embed the encoded domain information,  $\chi$ . Lastly, we project the last hidden layer representation to the target function space.
and hence we can further express the eDAFNO architecture as:
$$
\mathcal {J} ^ {l} [ \boldsymbol {h} ] := \sigma \left(\chi (\boldsymbol {x}) \left(\mathcal {I} \left(\chi (\cdot) \boldsymbol {h} (\cdot); \boldsymbol {v} ^ {l}\right) - \boldsymbol {h} (\boldsymbol {x}) \mathcal {I} \left(\chi (\cdot); \boldsymbol {v} ^ {l}\right) + W ^ {l} \boldsymbol {h} (\boldsymbol {x}) + \boldsymbol {c} ^ {l}\right)\right), \tag {6}
$$
where  $\mathcal{I}(\circ ;\pmb{v}^l)\coloneqq \mathcal{F}^{-1}\left[\mathcal{F}[\kappa (\cdot ;\pmb{v}^l)]\cdot \mathcal{F}[\circ ]\right]$
An illustration of the DAFNO architecture is provided in Figure 2. Note that this architectural modification is performed at the continuum level and therefore is independent of the discretization. Then, the box  $\mathbb{T}$  can be discretized with structured grids (cf. Figure 1), as is the case in standard FNO.
Although the proposed DAFNO architecture in (6) can handle complex generalization in domains, it has a potential pitfall: since the characteristic function is not continuous on the domain boundary, its Fourier series cannot converge uniformly and the FFT result would present fictitious wiggling near the discontinuities (i.e., the Gibbs phenomenon (Day et al., 1965)). As a consequence, the introduction of  $\chi$  can potentially jeopardize the computational accuracy. To improve the efficacy of DAFNO, we propose to replace the sharp characteristic function,  $\chi$ , with a smoothed formulation:
$$
\tilde {\chi} (\boldsymbol {x}) := \tanh  (\beta \operatorname {d i s t} (\boldsymbol {x}, \partial \Omega)) (\chi (\boldsymbol {x}) - 0. 5) + 0. 5. \tag {7}
$$
Here, the hyperbolic tangent function  $\tanh(z) := \frac{\exp(z) - \exp(-z)}{\exp(z) + \exp(-z)}$ ,  $\mathrm{dist}(\pmb{x}, \partial \Omega)$  denotes the (approximated) distance between  $\pmb{x}$  and the boundary of domain  $\Omega$ , and  $\beta$  controls the level of smoothness, which is treated as a tunable hyperparameter. An illustration of the effect of the smoothed  $\tilde{\chi}$  is displayed in Figure 3, with additional plots and prediction results with respect to different levels of smoothness,  $\beta$ , provided in Appendix B.1. In what follows, the  $\tilde{\cdot}$  sign is neglected for brevity.
Remark: We point out that the proposed smoothed geometry encoding technique, although simple, is substantially different from existing function padding/extrapolation techniques proposed in Lu et al. (2022) who cannot handle singular boundaries (as in the airfoil tail of our example 2) nor notches/shallow gaps in the domain (as in the crack propagation of our example 3). Our proposed architecture in (5) is also more sophisticated than a trivial zero padding at each layer in that the characteristic function  $\chi(\boldsymbol{x})$  is multiplied with the integrand, whereas the latter breaks the convolutional structure and hinders the application of FFTs.
4 Numerical examples
In this section, we demonstrate the accuracy and expressivity of DAFNO across a wide variety of scientific problems. We compare the performance of DAFNO against other relevant scientific machine learning models, including FNO (Li et al., 2020c), Geo-FNO (Li et al., 2022a), IFNO (You et al., 2022c), F-FNO (Tran et al., 2022), GNO (Li et al., 2020a), DeepONet (Lu et al., 2019), and UNet (Ronneberger et al., 2015). In particular, we carry out three experiments on irregular domains, namely, constitutive modeling of hyperelasticity in material science, airfoil design in fluid mechanics, and crack propagation with topology change in fracture mechanics. For fair comparison, the hyperparameters of each model are tuned to minimize the error on validation datasets, including initial learning rate, decay rate for every 100 epochs, smoothing parameter, and regularization parameter, while the total number of epochs is restricted to 500 for computational efficiency. The
Computational Domain 41x41
Figure 3: An illustration on a hyperelasticity sample: (a) sharp characteristic function  $\chi$ , (b) smoothed characteristic function  $\chi$ , (c) ground truth, (d) eDAFNO (trained using  $41\times 41$  discretization) prediction from the same resolution, and (e) zero-shot super-resolution prediction from eDAFNO (trained using  $41\times 41$  discretization and evaluated directly on  $161\times 161$  discretization).
Boundary Encoding  $\beta = 10$
Ground Truth 41x41
eDAFNO Prediction 41x41
eDAFNO Prediction 161x161
relative L2 error is reported as comparison metrics for both the training and test datasets. The experiments of each method are repeated on 5 trials with 5 different random seeds, and the mean and standard deviation of the errors are reported. Further details and additional results are provided in Appendix B.
4.1 Constitutive modeling of hyperelasticity
We start with a hyperelasticity problem in material science that models the fundamental principle of constitutive relations. The high-fidelity synthetic data in this benchmark is governed by
$$
\rho \frac {\partial^ {2} \boldsymbol {u}}{\partial t ^ {2}} + \nabla \cdot \boldsymbol {\sigma} = 0, \tag {8}
$$