SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Fabien Casenave Brian Staber Xavier Roynard
Safran Tech, Digital Sciences & Technologies
78114 Magny-Les-Hameaux, France
{fabien.casave, brian.staber, xavier.roynard}@safrangroup.com
When learning simulations for modeling physical phenomena in industrial designs, geometrical variabilities are of prime interest. While classical regression techniques prove effective for parameterized geometries, practical scenarios often involve the absence of shape parametrization during the inference stage, leaving us with only mesh discretizations as available data. Learning simulations from such mesh-based representations poses significant challenges, with recent advances relying heavily on deep graph neural networks to overcome the limitations of conventional machine learning approaches. Despite their promising results, graph neural networks exhibit certain drawbacks, including their dependency on extensive datasets and limitations in providing built-in predictive uncertainties or handling large meshes. In this work, we propose a machine learning method that do not rely on graph neural networks. Complex geometrical shapes and variations with fixed topology are dealt with using well-known mesh morphing onto a common support, combined with classical dimensionality reduction techniques and Gaussian processes. The proposed methodology can easily deal with large meshes without the need for explicit shape parameterization and provides crucial predictive uncertainties, which are essential for informed decision-making. In the considered numerical experiments, the proposed method is competitive with respect to existing graph neural networks, regarding training efficiency and accuracy of the predictions.
Many problems in science and engineering require solving complex boundary value problems. Most of the time, we are interested in solving a partial differential equation (PDE) for multiple values of input parameters such as material properties, boundary conditions, initial conditions, or geometrical parameters. Traditional numerical methods such as the finite element method, finite volume method, and finite differences require fine discretization of time and space in order to be accurate. As a result, these methods are often computationally expensive, especially when the boundary value problem needs to be repeatedly solved for extensive exploration of the input parameters space. To overcome this issue, machine and deep learning have been leveraged for various tasks in computational physics, namely, solving and learning solutions to PDEs [37, 56, 65, 81], accelerating linear solvers [5, 34], reduced-order modeling [49], domain decomposition [41], closure modeling [52], and topology optimization [74], to name a few. As reported in the review papers [13, 17, 75], most of the recent advances have been relying on deep neural networks for their flexibility and expressiveness. In this work, we focus on learning simulations of physical phenomena, that are discretized on a non
parameterized unstructured mesh. In this situation, traditional machine learning approaches cannot easily be leveraged as the inputs of the problem are given by graphs with different numbers of nodes and edges. In contrast, deep learning models such as graph neural networks (GNNs) [67] can easily overcome this limitation thanks to their ability to operate on meshes with different resolutions and topologies. While GNNs show promising results and their flexibility is highly appealing, they still suffer from a few shortcomings that prevent their deployment in engineering fields where decisions involve high stakes. Training GNNs usually requires large datasets and computational resources, and predicting their uncertainties is still an open and challenging problem of its own [32].
We propose a novel methodology, called Mesh Morphing Gaussian Process (MMGP), that relies on standard and well-known morphing strategies, dimensionality reduction techniques and finite element interpolation for learning solutions to PDEs with non-parameterized geometric variations. In contrast to deep learning methods, such as GNNs, the model can easily and efficiently be trained on CPU hardware and predictive uncertainties are readily available. Our method shares some limitations with any machine learning regressor for PDE systems: (i) within the predictive uncertainties, our method produces predictions with an accuracy lower than the reference simulations, (ii) unlike many methods used in reference simulators, like the finite element method, our method provides no guaranteed error bounds and (iii) our method requires a well sampled training dataset, which has a certain computational cost, so that the workflow becomes profitable only for many-query contexts where the inference is called a large number of times. Regarding (i), rough estimates may be sufficient in preproject phases, and accuracy can be recovered by using the prediction as an initialization in the reference simulator, or by allowing the designer to run the reference simulator on the identified configuration if the regressor is used in an optimization task.
We start by providing the background and assumptions of our method while mentioning some related works in Section 2. Then, the proposed methodology is detailed in Section 3. Three numerical experiments are presented in Section 4. Finally, a conclusion is given in Section 5.
2 Preliminaries and related works
Notations. Vectors and matrices are denoted with bold symbols. The entries  $i$  of a vector  $\mathbf{v}$  and  $i, j$  of a matrix  $\mathbf{M}$  are respectively denoted  $v_{i}$  and  $M_{i,j}$ . The i-th row of a matrix  $\mathbf{M}$  is denoted by  $\mathbf{M}_i$ .
Background. Let  $\mathcal{U}_{\mathrm{true}}: \Omega \to \mathbb{R}^d$  be a solution to a boundary value problem, where  $\Omega \subset \mathbb{R}^{d_{\Omega}}$  denotes the physical domain of the geometry under consideration, and  $d_{\Omega} = 2$  or 3. The domain  $\Omega$  is discretized into a conformal mesh  $\mathcal{M}$  as  $\mathcal{M} = \cup_{e=1}^{N_e} \Omega_e$ . In traditional numerical approaches such as the finite element method [82], an approximation  $\mathcal{U}$  of the solution  $\mathcal{U}_{\mathrm{true}}$  is sought in the finite-dimensional space spanned by a family of trial functions,  $\{\varphi_I(\mathbf{x})\}_{I=1}^N$ , supported on the mesh  $\mathcal{M}$ :
$$
\mathcal {U} _ {k} (\mathbf {x}) = \sum_ {I = 1} ^ {N} U _ {k, I} \varphi_ {I} (\mathbf {x}), \quad k = 1, \dots , d, \tag {1}
$$
where  $N$  is the total number of nodes in the mesh  $\mathcal{M}$ ,  $\mathbf{U} \in \mathbb{R}^{d \times N}$  is the discretized solution (featuring  $d$  fields), and  $\mathbf{x} \in \mathbb{R}^{d_{\Omega}}$  denotes the spatial coordinates. For simplicity of the presentation and without loss of generality, we consider the particular case of a Lagrange  $\mathbb{P}_1$  finite element basis, so that the solution is uniquely determined by its value at the nodes of  $\mathcal{M}$ . In this setting, the basis  $\{\varphi_I\}_{I=1}^N$  spans the space  $\{v \in \mathcal{C}^0(\mathcal{M}) : v|_{\Omega_e} \in \mathbb{P}_1, \forall \Omega_e \in \mathcal{M}\}$ , and the discretized solution  $\mathbf{U}$  is determined by solving the discretized weak formulation of the underlying boundary value problem. This problem also depends on some parameters  $\pmb{\mu} \in \mathbb{R}^p$ , such as material properties and boundary conditions. It is assumed that there are scalar output quantities of interest  $\mathbf{w} \in \mathbb{R}^q$  that depend on the discretized solution  $\mathbf{U}$ , and possibly on  $\mathcal{M}$  and  $\pmb{\mu}$ . We restrict ourselves to stationary, time-independent, scalars and fields of interest, which still falls in the scope of many industrial problems of interest. The learning task that we consider herein consists in learning the mapping
$$
\mathcal {F}: (\boldsymbol {\mu}, \mathcal {M}) \mapsto (\mathbf {U}, \mathbf {w}). \tag {2}
$$
For this purpose, it is assumed that we are given a training set of size  $n$  made of input pairs  $(\pmb{\mu}^i,\mathcal{M}^i)$  of parameters and meshes, and output pairs  $(\mathbf{U}^i,\mathbf{w}^i)$  of discretized fields and scalars. Each input mesh  $\mathcal{M}^i$  has a number of nodes denoted by  $N^i$ , and corresponds to a finite element discretization of an input geometry  $\Omega^i$ . The associated discretized solution  $\mathbf{U}^i$  is a matrix of size  $(d\times N^i)$ . For
any  $i = 1, \dots, n$ , the mesh  $\mathcal{M}^i$  can be represented as an undirected graph  $G^i = (V^i, E^i)$ , where  $V^i$  denotes the set of nodes and  $E^i$  is the set of edges.
Assumptions and limitations. We assume that the observed input geometries,  $\Omega^1,\ldots ,\Omega^n$  , share a common topology. The parameterization that generates the input geometries is unknown, and we are left with the associated finite element meshes  $\mathcal{M}^1,\dots ,\mathcal{M}^n$  . Being the discretization of physical domains involved in a boundary value problem, the input meshes inherit important features such as boundary conditions applied to subsets of nodes and elements. In finite element methods, error estimates strongly depend on the quality of the mesh [66]. Hence, in our context, it is assumed that the input meshes exhibit good quality in terms of elements aspect ratios and node densities, adapted to the regularity of the fields of interest. Our focus centers on the design optimization of industrial components with respect to specific physical phenomena. Consequently, we assume precise control over the geometry, free from any noise. Additionally, the employed geometrical transformations are constrained to avoid extreme distortions, as they are selected from sets of admissible designs that adhere to limitations on mass, volume, and mechanical resistance.
Related works. In recent years, there has been a substantial focus on advancing neural networks to emulate solutions to physical systems, either through the integration of domain-specific knowledge [44] or by devising efficient architectures for GNNs [63]. GNNs learn the mapping  $\mathcal{F}$  by relying on the message passing framework introduced by Gilmer et al. [33] and extended by Battaglia et al. [11]. In the context of physical systems, only a few contributions address non-parameterized geometric variabilities. The early work of Baque et al. [10] explores the use of GNNs to emulate physics-based simulations in the presence of geometric variabilities by relying on geodesic convolutions [58, 61]. More recently, Pfaff et al. [63] develop the MeshGraphNets (MGNs) model, a GNN that updates nodes and edges features in order to learn time-dependent simulations. Most notably, the model can handle various physics, three-dimensional problems, and non-parameterized geometric variabilities. Fortunato et al. [28] introduce MultiScale MGNs that relies on two different mesh resolutions in order to overcome the computational cost of the message passing algorithm on dense meshes, and to increase the accuracy of MGNs. The efficiency of MGNs has been illustrated by Allen et al. [7] for inverse problems, and by Harsch et al. [39] for time-independent systems. There exist several variants of such GNNs for learning mesh-based solutions. A multi-scale GNN that learns from multiple mesh resolutions has been proposed in Lino et al. [53] and is illustrated on two dimensional PDEs with geometric variabilities. Lino et al. [54] also devise a multi-scale and rotation-equivariant GNN that extrapolates the time evolution of the fluid flow, and Cao et al. [19] propose a novel pooling strategy that prevents loss of connectivity and wrong connections in multi-level GNNs. Regarding morphing strategies, Gao et al. [30] and Li et al. [51] deform irregular meshes into a reference one in order to learn solution of PDEs, but rely on complex coordinate transformation to compute a physical residual-based loss in the reference domain, and on input meshes with equal number of nodes. It is worth emphasizing that while the aforementioned works show promising results, they do not provide predictive uncertainties. There exist several methods for quantifying the uncertainties of deep neural networks [32], but it remains an open problem to provide well calibrated uncertainty estimates at a reasonable computational cost.
3 MMGP methodology
The proposed methodology is based on two main ingredients that allow us to leverage classical machine learning methods for regression tasks in the context of non-parameterized geometrical variability: (i) the data is pretreated by morphing each input mesh into a reference shape, and resorting to finite element interpolation to express all fields of interest on a common mesh of this reference shape, and (ii) a low-dimensional embedding of the geometries is built by considering the coordinates of the nodes as a continuous input field over the meshes. Formally, the proposed methodology consists in constructing a graph kernel by relying on three well chosen transformations such that the transformed inputs can be compared with any classical kernel functions defined over Euclidean spaces. Figure 1 illustrates the proposed strategy for a two-dimensional problem where we aim at predicting output fields of interest. The first transformation morphs the input mesh onto a chosen common shape. The second transformation performs a finite element interpolation on the chosen reference mesh of the common shape. Finally, a dimensionality reduction technique is applied to obtain low-dimensional embeddings of the inputs and outputs. These three steps are
all deterministic and described in the following subsections. The proposed kernel function can be plugged into any kernel method. Herein, we rely on Gaussian process regression in order to learn steady-state mesh-based simulations in computational fluid and solid mechanics.
Figure 1: Illustration of the MMGP inference workflow for the prediction of an output field of interest. The lower rectangle in the illustration of the input of the GP represents the scalar inputs  $\mu^i$ .
3.1 Deterministic preprocessings of the input meshes and fields of interest
In this section, we describe the methodology for building low-dimensional representations of the input meshes and output fields.
Mesh morphing into a reference shape  $\overline{\Omega}$ . Each input mesh  $\mathcal{M}^i$ ,  $i = 1, \dots, n$ , is morphed onto a mesh  $\overline{\mathcal{M}}^i$  associated to a fixed reference shape  $\overline{\Omega}$ . The morphed mesh has the same number of nodes and same set of edges as the initial mesh  $\mathcal{M}^i$ , but their spatial coordinates differ. In this work, we consider two morphing algorithms, namely, Tutte's barycentric mapping [71] onto the unit disk, and the Radial Basis Function (RBF) morphing [9, 21] onto a chosen reference shape. Regardless of the morphing algorithm, physical features inherited from the boundary value problem are carefully taken into account. More precisely, points, lines and surfaces of importance in the definition of the physical problem are mapped onto their representant on the reference shape. Doing so, rigid body transformations that may occur in the database are corrected in the mesh morphing stage, and boundary conditions of same nature are matched together.
Common mesh  $\overline{\mathcal{M}}_c$  of the reference shape  $\overline{\Omega}$ . At this stage, it should be noted that although the morphed meshes  $\overline{\mathcal{M}}^i$  are associated to a common reference shape  $\overline{\Omega}$ , they do not share the same nodes and edges. This prevents us from measuring similarities between the input meshes and output fields with classical techniques. A strong advantage of the finite element method is that it provides accurate solution fields with a continuous description over the mesh, and a natural way to transfer fields from one mesh to another. This motivates us to introduce a common mesh  $\mathcal{M}_c$  of the reference shape  $\overline{\Omega}$ , as a common support for all the sample fields data. A possibility is to choose an input mesh in the training set, e.g.  $\mathcal{M}^1$ , and define  $\overline{\mathcal{M}}_c$  as its morphing onto the chosen reference shape. The aim is twofold. First, it allows us to express the output fields on the common morphed mesh  $\overline{\mathcal{M}}_c$ , leading to vector representations of same sizes. Second, the coordinates fields of the meshes  $\overline{\mathcal{M}}^i$  are also transferred onto this common mesh in order to build a shape embedding. These procedures rely on classical finite element interpolation that is described in the rest of this section.
Transporting the fields of interest on the common mesh  $\overline{\mathcal{M}}_c$ . The discretized solution  $\mathbf{U}^i$ ,  $i = 1, \dots, n$ , is first transferred on the morphed mesh  $\overline{\mathcal{M}}^i$  as follows:
$$
\bar {\mathcal {U}} _ {k} ^ {i} (\bar {\mathbf {x}}) = \sum_ {I = 1} ^ {N ^ {i}} U _ {k, I} ^ {i} \bar {\varphi} _ {I} ^ {i} (\bar {\mathbf {x}}),, \quad k = 1, \dots , d, \tag {3}
$$
where  $\{\overline{\varphi}_I^i\}_{I = 1}^{N^i}$  is the finite element basis associated to the morphed mesh  $\overline{\mathcal{M}}^i$ . The transported fields  $\overline{\mathcal{U}}_k^1,\ldots ,\overline{\mathcal{U}}_k^n$  share the same geometric support (the reference shape). This implies that they can be
interpolated onto the common mesh  $\overline{\mathcal{M}}_c$  using the finite element interpolation operator  $P$  defined as:
$$
P \left(\bar {\mathcal {U}} _ {k} ^ {i}\right) (\bar {\mathbf {x}}) = \sum_ {J = 1} ^ {N _ {c}} \bar {\mathcal {U}} _ {k} ^ {i} \left(\bar {\mathbf {x}} _ {J} ^ {c}\right) \bar {\varphi} _ {J} ^ {c} (\bar {\mathbf {x}}) = \sum_ {I = 1} ^ {N ^ {i}} \sum_ {J = 1} ^ {N _ {c}} U _ {k, I} ^ {i} \bar {\varphi} _ {I} ^ {i} \left(\bar {\mathbf {x}} _ {J} ^ {c}\right) \bar {\varphi} _ {J} ^ {c} (\bar {\mathbf {x}}), \tag {4}
$$
where  $\{\overline{\varphi}_I^c\}_{I = 1}^{N^i}$  is the finite element basis associated to  $\overline{\mathcal{M}}_c$ ,  $\overline{\mathbf{x}}_J^c$  is the coordinates of the  $J$ -th node of  $\overline{\mathcal{M}}_c$  and  $\overline{\mathcal{U}}_k^i (\overline{\mathbf{x}}_J^c)$  is evaluated using Equation (3). We are now in the much more favorable situation where all the fields of interest are expressed on a common mesh  $\overline{\mathcal{M}}_c$ . More specifically, for each field of interest  $k$  and input mesh  $\mathcal{M}^i$ , we let  $\widetilde{\mathbf{U}}_k^i\in \mathbb{R}^{N_c}$  be the transported output fields onto the common mesh, such that  $\widetilde{\mathbf{U}}_{k,I}^{i} = \overline{\mathcal{U}}_{k}^{i}(\overline{\mathbf{x}}_{I}^{c})$ . In this setting, the vector representations of the output fields now have the same sizes  $N_{c}$ . Notice that the derivation of the finite element interpolation is identical with higher-order Lagrange finite elements.
Transporting the coordinates fields on the common mesh  $\overline{\mathcal{M}}_c$ . The same procedure can be applied to the coordinate fields of the input meshes in order to build a shape embedding of the input meshes. Let  $\mathcal{Z}_{\ell}^{i}$  be the  $\ell$ -th component of the coordinate field over the mesh  $\mathcal{M}^i$ ,  $\ell = 1, \ldots, d_{\Omega}$ . Using the finite element basis associated to the mesh  $\mathcal{M}^i$ , the coordinates fields can be written as
$$
\mathcal {Z} _ {\ell} ^ {i} (\mathbf {x}) = \sum_ {I = 1} ^ {N ^ {i}} Z _ {\ell , I} ^ {i} \varphi_ {I} ^ {i} (\mathbf {x}),
$$
where  $Z_{\ell,I}^{i}$  denotes the  $\ell$ -th of the coordinates of the node  $I$  in the mesh  $\mathcal{M}^i$ . Notice that the notation  $Z_{\ell,I}^{i}$  is preferred to  $x_{\ell,I}^{i}$ , since it denotes here the degrees of freedom of the coordinate fields defined over  $\mathcal{M}^i$ , whose continuity property is essential for the finite element interpolation stage. Then, in the same fashion as for the fields of interest, the coordinates fields are transferred on the morphed mesh  $\overline{\mathcal{M}}_i$  and interpolated on the common mesh  $\overline{\mathcal{M}}_c$  using the operator  $P$  given by Equation (4). For each coordinate  $\ell = 1,\ldots,d_{\Omega}$  and input mesh  $\mathcal{M}^i$ , we have the common representations  $\widetilde{\mathbf{Z}}_{\ell}^{i}\in \mathbb{R}^{N_c}$  of the coordinate fields on the common mesh  $\overline{\mathcal{M}}_c$ .
Dimensionality reduction. At this stage, the input coordinates fields of the meshes and the output fields are expressed on the same common mesh  $\overline{\mathcal{M}}_c$ , and can be compared using standard machine learning techniques. We propose to build low-dimensional embeddings of these quantities using Principal Component Analysis (PCA). For each output field, PCA is applied to the set of observations  $\{\widetilde{\mathbf{U}}_k^i\}_{i=1}^n$ , leading to the fields low-dimensional embeddings that we denote by  $\{\widehat{\mathbf{U}}_k^i\}_{i=1}^n$ . Similarly, PCA is applied to concatenated transported coordinate fields,  $\{(\widetilde{\mathbf{Z}}_1^i, \ldots, \widetilde{\mathbf{Z}}_{d_\Omega}^i)\}_{i=1}^n$ , leading to low-dimensional embeddings of the input geometries  $\{\widehat{\mathbf{Z}}^i\}_{i=1}^n$ , that we refer to as the shape embeddings.
3.2 MMGP training
Once the operations of mesh morphing, finite element interpolation on a common mesh and dimensional reduction described in the previous subsection have been carried out, we are left with reduced-size objects of same dimension. Let  $\{\mathbf{X}^i\}_{i=1}^n \in \mathbb{R}^{l_{\mathbf{Z}} + p}$ , where  $p$  the number of nongeometrical parameters and  $l_{\mathbf{Z}}$  is the size of shape embedding, be such that  $\mathbf{X}^i = (\widehat{\mathbf{Z}}^i, \boldsymbol{\mu}^i)$ . Denoting  $l_{\mathbf{U}_k}$  the size of the embedding of field  $\mathbf{U}_k$ , the machine learning task given by Equation (2) can be approximated by the following set of scalar and vector regression problems:
$$
\hat {\mathcal {F}} _ {\text {s c a l a r}, m}: \quad \mathbf {X} ^ {i} \mapsto w _ {m} ^ {i} \in \mathbb {R}, \quad m = 1, \dots , q, \tag {5a}
$$
$$
\widehat {\mathcal {F}} _ {\text {v e c t o r}, k}: \quad \mathbf {X} ^ {i} \mapsto \widehat {\mathbf {U}} _ {k} ^ {i} \in \mathbb {R} ^ {l} \mathbf {u} _ {k}, \quad k = 1, \dots , d. \tag {5b}
$$
Gaussian processes can be trained in a classical fashion to address the regression problems (5a)-(5b).
MMGP for a scalar output. Let  $\mathcal{D} = \{(\mathbf{X}^i, w_{m_0}^i)\}_{i=1}^n$  be a training dataset for one of the problems given by Equation (5a), i.e. for the  $m_0$ -th output scalar. It can be shown by standard conditioning [76] that the posterior mean and variance of the prediction on some given test input  $\mathbf{X}^\star$  are given by
$$
\mathbb {E} [ w ^ {\star} ] = \mathbf {k} _ {\star} ^ {T} (\mathbf {K} + \sigma^ {2} \mathbf {I}) ^ {- 1} \mathbf {w} _ {m _ {0}},
$$
$$
\mathbb {V} [ w ^ {\star} ] = K _ {\star \star} - \mathbf {k} _ {\star} ^ {T} (\mathbf {K} + \sigma^ {2} \mathbf {I}) ^ {- 1} \mathbf {k} _ {\star},
$$
where  $\mathbf{w}_{m_0} = \{w_{m_0}^i\}_{i=1}^n$ , and  $\mathbf{K}$  is the Gram matrix such that  $K_{i,j} = c(\mathbf{X}^i, \mathbf{X}^j)$  for  $1 \leq i, j \leq n$ , the vector  $\mathbf{k}_\star$  such that  $k_{\star j} = c(\mathbf{X}^\star, \mathbf{X}^j)$ , and the scalar  $K_{\star \star} = c(\mathbf{X}^\star, \mathbf{X}^\star)$ , with  $c$  denoting the chosen kernel function which lengthscales are optimized, and  $\sigma$  denotes the optimized nugget parameter. This training procedure is repeated for the  $q$  scalar outputs.