Multi-Fidelity Transfer Learning for accurate
database PDE approximation

Wenzhuo Liu
IRT SystemX & INRIA, U. Paris-Saclay
Wenzhuo.liu@irt-systemx.fr

Mouadh Yagoubi
Irt-SystemX
mouadh.yagoubi@irt-systemx.fr

David Danan
Irt-SystemX
david.danan@irt-systemx.fr

Marc Schoenauer
INRIA, U. Paris-Saclay
marc.schoenauer@inria.fr

Abstract
Data-driven approaches to accelerate computation time on PDE-based physicalproblems have recently received growing interest. Deep Learning algorithms areapplied to learn from samples of accurate approximations of the PDEs solutions
computed by numerical solvers. However, generating a large-scale dataset withaccurate solutions using these classical solvers remains challenging due to theirhigh computational cost. In this work, we propose a multi-fidelity transfer learning
approach that combines a large amount of low-cost data from poor approximations
with a small but accurately computed dataset. Experiments on two physical problems (airfoil flow and wheel contact) show that by transferring prior-knowledge
learned from the inaccurate dataset, our approach can predict well PDEs solutions,even when only a few samples of highly accurate solutions are available.

1

Introduction

In recent years, there has been a rapid growth in the use of machine learning algorithms to numericallysolve problems from physical domains for which accurate numerical solvers are too computationallyexpensive. In such context, data-driven methods have been proposed [1, 2, 3, 4, 5, 6] to learn Deep
Learning models (DL) that approximate the solutions of complex partial differential equations (PDEs)
from large amounts of data created by classical numerical solvers. A widely used family of suchclassical numerical methods is the Finite Element Method (FEM), which uses a discretization of
the physical domain into small polygonal elements, aka a mesh. The accuracy of the FEM solution
depends on the size of the elements and hence on the size of the mesh, i.e., the number of elements.
In particular, fine meshes (i.e., with a large number of elements) are needed for acceptable accuracy.
However, beyond issues related to generalization, that will not be addressed here, a most critical issue
in the data-based approach is the computational cost of the generation of training datasets: Complex
phenomena can only be captured accurately using large enough deep networks that require large
training datasets. Furthermore, the total error of the trained model is the sum of the training error of
the network and the numerical error of the samples in the training set, that hence should be madeof accurate enough solutions, i.e., in the case of FEM, solutions computed on very fine meshes. To
tackle this challenge, we introduce MFT, a Multi-Fidelity Transfer learning approach that uses twomeshes of different granularity: On the coarse mesh, FEM approximate solutions of the PDE at hand
can be computed at low cost – but only poorly approximate the exact solution of the PDE. On thefine mesh, on the other hand, the FEM solutions are good approximations of the exact solution of the
PDE, but are very costly to compute, making it unrealistic to generate enough samples to train an
accurate-enough deep model. The goal of this work is to train such an accurate-enough deep model
Machine Learning and the Physical Sciences workshop, NeurIPS 2022.

Figure 1: Diagram of the multi-fidelity transfer learning approach on Wheel Contact problem
using only a small dataset of highly accurate solutions, but leveraging the result of the training on the
many samples computed on the coarse mesh, using principles from Transfer Learning: the modeltrained on the large dataset generated on the coarse mesh is used as a starting point for the learning
process on the small dataset of accurate solutions. We demonstrate the effectiveness of MFT on two
PDEs describing respectively the air flow around an airfoil and the contact of a tire on the road.

2

The multi-fidelity transfer learning approach

In the following, we aim at learning a data-based model (see below) to solve a given PDE for different
values of some input quantities x that govern the system on a given physical domain. We will use
a given FEM numerical solver on two different meshes of that domain, a coarse mesh Mc and afine mesh Mf . The solver can compute so-called low-fidelity solutions at low cost on the coarse
mesh Mc , and high-fidelity solutions on the fine mesh Mf , but at a high cost. The goal is to train a
deep model that can compute high-fidelity solutions of the PDE for any given input, using as few as
possible high-fidelity samples but taking advantage of as many low-fidelity samples as needed.
2.1

Prediction on the Coarse Mesh

The first step consists in generating a dataset Dc of low-fidelity solutions by applying the FEM solver
on the coarse mesh Mc using a representative set of inputs x. A deep model fc is then learned fromDc (details follow). As many samples as needed can be computed efficiently at low cost, and the loss
function is the MSE between the network output and the numerical solution in Dc , considered as
ground truth at this step. Even in the ideal case where the error of the learned network is zero, thanks
to very large dataset Dc data, the accuracy of the learned model w.r.t. the exact solution of the PDE
is not satisfactory since the solutions in Dc are poor approximations of these exact solutions.
2.2

Integration of low-fidelity prior knowledge on the Fine Mesh

The second step starts from this low-fidelity model fc , which was fully trained on the low-fidelity
dataset Dc and from a dataset Df of high-fidelity solutions assumed to be too small to allow direct
training of an accurate enough deep model. Inspired by the ideas from Transfer Learning algorithms[7], where the knowledge gained while solving a task is re-used for a different but related task, we
propose to leverage some knowledge that has been encapsulated in fc to improve the learning of adeep model from the small dataset Df . Transfer Learning is widely used nowadays to solve textand image-related tasks [8, 9, 10], especially in similar situations when only a few data samples areavailable or training.
Latent Feature Extraction The basic assumption here is that the pre-trained model fc has learned ameaningful representation of the problem and can be viewed as a feature extractor for the high-fidelity
task. For each input x, the vector h, outputs of all neurons before the last layer of fc is considered asa vector of latent features of the problem. It is first upsampled to the fine mesh Mf using k-nearest
neighbors interpolation and used as additional inputs for the high-fidelity learning, as illustrated inFig. 4.
2

Transfering Using the high-fidelity dataset Df and additional input h, we train a transfer model ft
using again the MSE loss. The final output is:
y = ft (x, h) = ft (x, UpSample[fc (x)])
2.3Model Architecture

Mesh to Graph A mesh can be viewed as a graph whose nodes are the vertices of the polygonal
elements, and whose edges are the edges of the elements. Formally, mesh data can then be expressedas M = (N, E, V, E), where N is the number of nodes of the mesh, E the set of edges, V ∈ RN ×F
are the F -dimensional node features attached to each of the N nodes, and E ∈ RE×D the attachedattributes on each edge.
Graph Neural Networks GNNs [11, 12, 13, 14, 15] are designed to treat data from non-Euclidean
space such as graphs, meshes or manifolds, and are now routinely used to handle mesh data [4, 5, 6].
Needed here are a convolutional operator, invariant by permutation of the nodes, and a pooling
operator, allowing to exchange data between meshes of different granularities. Among the different
types of GNN, we have chosen the MoNet GNN [16].
Convolution operator In MoNet, it is defined as follows. Consider a weighted graph G =(N, E, V, E). Let xi ∈ RF be the feature vector of node i, and xeij ∈ RD be the feature of the edgei, j defining the set of neighbours N (i) of node i. The basic idea of MoNet is to define a trainable
function w that computes an edge weight wij from the edge feature xeij . MoNet then defines the
convolutional operator on node i as
x′i =

K
X 1 X
1
wk (xeij ) ⊙ Θk xj|N (i)|K
k=1

j∈N (i),where K is the user-defined kernel size, Θk ∈ RM ×N stands for the trainable matrix applying
a linear transformation on the input data, ⊙ is the element-wise product, and wk , k = 1, . . . , K
are trainable edge weights. Following [16], we use Gaussian kernels: wk (xeij ) = exp(− 12 (xeij −eµk )T Σ−1k (xij − µk )). Both µk and Σk are trainable variables.
Pooling Operator We use simply the k-nearest interpolation proposed in PointNet++ [17] when
designing pooling operators. Let y be a node from M1 , and assume its k nearest neighbors on M2
are (x1 , . . . , xk ). The interpolated feature for y is defined from those of the xi ’s as:
Pkw(xi )f (xi )1, where w(xi ) =f (y) = i=1
Pk||y−xi ||2w(x)
ii=1
MFT Architecture Both learning phases of MFT are performed using the Graph U-Net architecturethat was proposed in [18]. As argued above, Graph Neural Networks (GNNs) can directly handle
mesh data. On the other hand, the U-Net architecture allows to extract hierarchical spatial features.
Modern mesh generators today easily allow us to build a set of hierarchical meshes for the same
domain without sacrificing geometric information. We benefit from these algorithms and use such ahierarchy of meshes to construct the U-Net architecture. A set of graph layers is applied on the samemesh level while pooling operators transfer data between different mesh levels.
The edge attribute xeij ∈ E of edge (i, j) is xeij = pj − pi , where pi and pj are the coordinatesof nodes i and j respectively. The node attribute at node i is the vector of values of the unknown
variables of the PDE at node i, the output of the network.

3

Experimental Results

The MFT approach is validated on two different PDEs:
Airfoil Flow : Incompressible fluid flow around an airfoil, modeled by the Navier Stokes PDEs.
Flow fields on 80 distinct 2D NACA airfoils are simulated by the CFD solver OpenFOAM. Inputs are
the Mach number, in [0.03, 0.3], and the Angle-of-Attack (AoA), in [-22.5, 22.5] (in degrees). The
target variable are the velocity v = (vx , vy ) and the pressure p. The low-fidelity dataset is generated
with meshes of approx. 500 nodes and the high-fidelity dataset uses meshes with around 3800 nodes.
3Wheel Contact : The target is the deformation field u = (ux , uy ) of a tire on a fixed 2D wheel
under some external force f . The FEM solver is GetFEM++. The input quantities are the mechanicalcarateristics of the rubber: Young modulus E ∈ [5, 7] × 106 and Poisson’s ratio ν ∈ [0.38, 0.495];
the strength A ∈ [0.1, 0.5], and the angle α ∈ [−0.78, −2.35] of the external forces; and the friction
coefficient of the road µ ∈ [0.5, 0.8]. The low-fidelity dataset is generated on a fixed coarse meshwith 504 nodes, while the high-fidelity dataset uses a fixed mesh with 3398 nodes. Note that the
network is asked to predict the correction w.r.t. the projection of the prediction by the low-fidelity
model fc rather than directly the deformation y.
Datasets For each problem, we generate a coarse mesh Mc and a fine mesh Mf . The low-fidelity
datasets Dc contains 2 000 samples, while the high-fidelity datasets Df is made of only 400 samples.
Baselines We compare MFT with three baselines. An interpolation model LF-Int predicts theoutputs by simply up-sampling the output of model fc . Model HF-400 is directly trained on the
400 high-fidelity samples without any knowledge transfer. Finally, in order to be fair with the highfidelity-only approach, we add some high-fidelity samples to Df to compensate for the computational
cost of creating Dc . Model HF-ST is trained on this extended high-fidelity dataset.
Training Both the coarse model fc and the transfer model ft use the Graph U-Net architecture
described above. Hyper-parameters setting and training process are detailed in Appendix B. Standard
10-fold cross-validation is applied when training ft to assess the robustness of the approach, and wereport the averages and standard deviations of test errors over the different folds. Models are trainedon a single Nvidia A100 GPU, and each training takes from 3 to 6 hours.
Evaluation We create 800 new samples on Mf for each task to form the ultimate test set andevaluate the results of all approaches using the rooted mean squared error (RMSE) metric.
Results Table 3 reminds the characteristics of the datasets (right side) and the error on the test set
described above (left). The latter clearly demonstrates that the MFT model significantly outperforms
the three baselines on both physical problems. The performance of the interpolation baseline LF-Int
shows that the transfer model MFT largely improved the predictions based on the pre-trained modelfc . In the meanwhile, the prior knowledge extracted from Dc does help the prediction on finemeshes compared with HF-400, which has only access to the high-fidelity dataset Df . Moreover, thecomparison between HF-ST and MFT indicates that the multi-fidelity datasets containing many more
samples tend to be more informative than a pure high-fidelity dataset with the same computational
budget. The MFT model combines the benefit of the low-fidelity dataset for the fast generation withthe high-fidelity simulations to create accurate ground truth.
Table 1: Rooted mean squared error (RMSE) for both Airfoil Flow and Wheel Contact problem (left);dataset composition for each discussed model (right); and generation time for each sample in the last
two rows.
Tasks

Df

ModelsAirfoil Flow (e-2)

Wheel Contact (e-4)

D

LF-IntHF-400
HF-ST
MFT

9.81
2.98 ± 0.30
2.43 ± 0.39
1.95 ± 0.05

27.82
6.62 ± 0.25
4.80 ± 0.19
4.57 ± 0.10

2 000
\
\
2 000

\
400
400 + 127/240 1
400∼ 2.33s∼ 0.20s∼ 36.7s∼ 1.64s

Generation
Time (each)

4

Datasetsc

Airfoil Flow
Wheel Contact

Conclusions

This work introduced MFT, a multi-fidelity transfer learning model, to use Machine Learning
approaches (namely GNNs) to numerically solve PDEs on fine meshes efficiently. MFT benefits froma large amount of low-fidelity data to extract some prior-knowledge, and then transfers it to predict
1

We generated 127 additional high-fidelity examples on fine meshes for the Airfoil Flow problem, and 240
for Wheel Contact problem.

4

high-fidelity solutions. The training process on fine meshes can then be achieved by using only asmall training set. The experimental results on two complex physical problems are a first proof ofthe concept that MFT can solve PDEs accurately when a small number of high-fidelity samples isavailable. On-going and future work is to evaluate the proposed approach on more complex physicalsystems such as three-dimensional problems.

5

Broader Impact

This paper investigates the ability to learn PDEs solutions when large training datasets are required
to solve the problem accurately. Our proposed method can be widely used in data-driven methods
for physical systems as the lack of a large-scale and high-fidelity dataset is a common issue in thephysical domain.

References[1] Saakaar Bhatnagar, Yaser Afshar, et al. Prediction of aerodynamic flow fields using CNNs.
Computational Mechanics, 64(2):525–545, 2019.
[2] Jonathan Tompson, Kristofer Schlachter, Pablo Sprechmann, and Ken Perlin. Accelerating
eulerian fluid simulation with convolutional networks, 2017.
[3] Nils Thuerey, Konstantin Weißenow, Lukas Prantl, and Xiangyu Hu. Deep learning methods forreynolds-averaged navier–stokes simulations of airfoil flows. AIAA Journal, 58(1):25–36, 2020.
[4] Ali Kashefi, Davis Rempe, and Leonidas J. Guibas. A point-cloud deep learning framework forprediction of fluid flow fields on irregular geometries, 2020.
[5] Filipe de Avila Belbute-Peres, Thomas D. Economon, and J. Zico Kolter. Combining differentiable PDE solvers and GNNs for fluid flow prediction. In 37th ICML, 2020.
[6] Tobias Pfaff, Meire Fortunato, Alvaro Sanchez-Gonzalez, and Peter W Battaglia. Learning
mesh-based simulation with graph networks. arXiv preprint arXiv:2010.03409, 2020.
[7] Fuzhen Zhuang, Zhiyuan Qi, Keyu Duan, Dongbo Xi, Yongchun Zhu, Hengshu Zhu, Hui
Xiong, and Qing He. A comprehensive survey on transfer learning. Proceedings of the IEEE,
109(1):43–76, 2020.
[8] Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification, 2018.
[9] Jui-Ting Huang, Jinyu Li, Dong Yu, Li Deng, and Yifan Gong. Cross-language knowledgetransfer using multilingual deep neural network with shared hidden layers. In 2013 IEEE
International Conference on Acoustics, Speech and Signal Processing, pages 7304–7308. IEEE,
2013.
[10] Maxime Oquab, Leon Bottou, Ivan Laptev, and Josef Sivic. Learning and transferring midlevel image representations using convolutional neural networks. In Proceedings of the IEEE
conference on computer vision and pattern recognition, pages 1717–1724, 2014.
[11] Joan Bruna, Wojciech Zaremba, Arthur Szlam, and Yann LeCun. Spectral networks and locallyconnected networks on graphs, 2014.
[12] Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. CNNs on Graphs with Fast
Localized Spectral Filtering. In NeurIPS, 2017.
[13] Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional
networks. In 5th ICLR, 2017.
[14] James Atwood and Don Towsley. Diffusion-convolutional neural networks, 2016.
[15] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua
Bengio. Graph attention networks. In ICLR, 2018.
5

[16] Federico Monti, Davide Boscaini, et al. Geometric deep learning on graphs and manifolds using
mixture model CNNs. In CVPR, 2016.
[17] Charles R. Qi, Li Yi, Hao Su, and Leonidas J. Guibas. Pointnet++: Deep hierarchical featurelearning on point sets in a metric space, 2017.
[18] Wenzhuo Liu, Mouadh Yagoubi, and Marc Schoenauer. Multi-resolution graph neural networks
for pde approximation. In International Conference on Artificial Neural Networks, pages
151–163. Springer, 2021.
[19] Automatic airfoil c-grid generation.

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope? [Yes](b) Did you describe the limitations of your work? [Yes] See Section 4(c) Did you discuss any potential negative societal impacts of your work? [No](d) Have you read the ethics review guidelines and ensured that your paper conforms tothem? [Yes]2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A](b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)? [No] The code andthe data are proprietary.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they
were chosen)? [Yes] partly in Section 3 and partly in Appendix B(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [Yes] In section 3(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)? [Yes] See Section 3
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes](b) Did you mention the license of the assets? [N/A](c) Did you include any new assets either in the supplemental material or as a URL? [N/A](d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating? [N/A](e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable? [N/A](b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable? [N/A](c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation? [N/A]6

A

Additional dataset details

A.1

Airfoil Flow Dataset

We generate a database of cases with various angles of attack (AoA), Mach numbers, and airfoil
shapes. The CFD solver OpenFOAM is applied to generate ground truth.
Airfoil Generation We create 80 NACA 4-digit airfoil shapes characterized by their camber C, theposition of their maximum camber P , and their thickness T . The NACA airfoil section is drawn by acamber line and a thickness distribution plotted vertically to the camber line. The camber line andthe thickness distribution are controlled by three NACA parameters: the camber C, the position of
maximum camber P , and the thickness T . While the thickness distribution is decided by a polynomial
function w.r.t the coordinate of axis x.
Tyt =(a0 x0.5 + a1 x + a2 x2 + a3 x3 + a4 x4 )
0.2
,where {ai |i = 0, .., 4} are constants. We sample the 3 parameters by uniform distributions C ∼U[0, 0.09], P ∼ U[0.4, 0.6] and T ∼ U[0.1, 0.3] 80 times to generate these distinct NACA airfoils.
Automatic Mesh Generation For each NACA airfoil, we automatically mesh it in a C-grid
quadrilateral format using the algorithm provided by [19]. The algorithm will create a C-grid mesh bysimply giving some inputs concerning mesh size or physical domain and the coordinates to describe
the airfoil shape. C-grid meshes are widely used for CFD analysis of an airfoil. In most cases, thesemeshes give a better convergence of flow over the airfoil.
Underlying PDEs The Spalart-Allmarasa equation is employed to model eddy viscosity combinedwith RANs equations forms a system of four PDEs in two-dimensional space. To solve such a system,
we use the semi-implicit method for pressure-linked equations (SIMPLE) algorithm assuming the
CFD problem to be incompressible.
Data Pre-processing We apply the data pre-processing steps described by [3]. Data Generation by
CFD solvers requires a large computational domain where the approximation of CFD simulation is
calculated. An appropriate distance between the airfoil object and the borders of the computational
domain should be large enough so that the boundary conditions assigned to the outer domain don’t
affect the quality of the flow simulation around the airfoil. When training the neural networks, it is
not necessary to compute the prediction for areas far away from the airfoil that is less interesting. We
focus on a small domain close to the airfoil. Additionally, the output quantities velocity v and pressure
p are normalized relative to the magnitude of the freestream velocity to make them dimensionless,thus:v̄ = v/||v0 ||, p̄ = p/||v0 ||2
A.2

Wheel Contact Dataset

In wheel contact problem, we are interested in finding the displacement of a wheel in contact with arigid foundation by applying an external force.
We use the 2D linear plane strain elasticity equation system to model the wheel contact problem. To
simplify the mathematical model, we suppose that the displacements of the material particles aremuch smaller than any relevant dimension of the body, and the inner rim of the wheel is rigid. With
these preliminaries, the formulation of the wheel contact problem is as follows:Π = ∂F W (F) in Ω,
Div Π + f 0 = 0 in Ω,u = ud on Γ1 ,
Πν = f 2 on Γ2 ,uν ≤ g, Πν ≤ 0, (uν − g) Πν = 0 on Γ3 ,∥Πτ ∥ ≤ µ |Πν | ,−Π = µ |Π | uτ if u ̸= 0. on Γ3τν ∥uτ ∥

τ

(1)
(2)
(3)
(4)
(5)
(6)

Equation (1) represents the hyperelastic constitutive law of the material. Equation (2) is the equilibrium equation. The two conditions (3) and (4) represent the displacement and boundary conditions
respectively. Finally, (5) and (6) describe the rubbing contact condition.
7

Figure 2: Physical Domain for Wheel Contact problem

B

Experimental Details

Models are trained to by minimizing the mean square error (MSE) between the output and groundtruth. We utilize the Adam optimizer to minimize the loss function. Moreover, a step decay strategy
is applied during training, where the learning rate is decreased to its half every 500 epochs. The
model architecture we applied is the graph U-Net described in Section 2.3. The graph U-Net model
consists of a series of graph blocks and sampling operator. A graph block C contains multiple graphlayers, each of which is followed by an activation function "elu". Each block C is parametrized by the
number of layers l, a channel factor c and a kernel size k. For example, C = (l, c, k) = (4, 128, 5)
implies a GNN block with 4 graph layers, and each graph layer has 128 hidden channels with a kernel
size of 5. After each GNN block, a sampling operator is applied to convert data between two meshlevels. The sampling operator has one hyper-parameter, the number of nearest neighbours n. We set
n = 6 for all our experiments. Finally, a graph layer is applied to map high-dimensional features tosolution space.
Airfoil Flow For airfoil flow problem, to train the model fc which predicts the solution on a coarse
mesh Mc , we down-sample once Mc and re-sample to Mc . A total of three blocks C = (4, 128, 5)
are applied. The transfer model ft on high resolution meshes down-samples progressively threetimes in the encoding part, and recover the mesh resolution with three up-sampling operators during
decoding. ft contains seven GNN blocks C = (2, 48, 5).
Wheel Contact For wheel contact problem, the model fc contains three mesh levels and 5 GNN
blocks C = (4, 48, 5). For the transfer model ft , same as that on airfoil flow, three are seven GNN
blocks C = (2, 64, 5).

C

Additional Figures8

Figure 3: An example of wheel contact prediction

Figure 4: An example of airfoil flow prediction

9