EAGLE: Large-scale Learning of Turbulent Fluid Dynamics with Mesh Transformers
Authors: Steeven JANNY
Aurélien Bénéteau
Madiha Nadri
Julie Digne
Nicolas THOME
Christian Wolf
Venue: ICLR 2023 poster (2023)
Keywords: Learning Fluid Mechanics
Simulation
Graph networks

Abstract: Estimating fluid dynamics is classically done through the simulation and integration of numerical models solving the Navier-Stokes equations, which is computationally complex and time-consuming even on high-end hardware. This is a notoriously hard problem to solve, which has recently been addressed with machine learning, in particular graph neural networks (GNN) and variants trained and evaluated on datasets of static objects in static scenes with fixed geometry. We attempt to go beyond existing work in complexity and introduce a new model, method and benchmark. We propose EAGLE: a large-scale dataset of ∼1.1 million 2D meshes resulting from simulations of unsteady fluid dynamics caused by a moving flow source interacting with nonlinear scene structure of varying geometries, with 600 different scenes of three different types in total. To perform future forecasting of pressure and velocity on the challenging EAGLE dataset, we introduce a new mesh transformer. It leverages node clustering, graph pooling and global attention to learn long-range dependencies between spatially distant data points without needing a large number of iterations, as existing GNN methods do. We show that our transformer outperforms state-of-the-art performance on, both, existing synthetic and real datasets and on EAGLE. Finally, we highlight that our approach learns to attend to airflow, integrating complex information in a single iteration.

Published as a conference paper at ICLR 2023
EAGLE: LARGE-SCALE LEARNING OF TURBULENT FLUID DYNAMICS WITH MESH TRANSFORMERS
Steeven Janny
Aurélien Béneteau
Madiha Nadri
LIRIS, INSA Lyon, France steeven. jannyd insa- fy
SupAero, France
LAGEPP, Univ. Lyon 1, France
surelien.benezee estuuent. saae-suparoo.rt
madiha nadri Maadri-woEfPan www.wwoiw-
Julie Digne
Nicolas Thome Sorbonne University, CNRS, ISIR, Paris, France
Christian Wolf
LIRIS, CNRS, France julie. diilie.dggneRe dignneenrs.fe
Naver Labs Europe, France
christian wol sol.fneverlsssscom
ni colas, theme fiiii..ue www...upme.....mm
ABSTRACT
Estimating fluid dynamies is classically done through the simulation and integra- tion of numerical models solving the Navier-Stokes equations, which is computa- tionally complex and time-consuming even on high-end hardware. This is a no- toriously hard problem to solve, which has recently been addressed with machine learning. in particular graph neural networks (GNN) and variants trained and eval- uated on datasets of static objects in static scenes with fixed geometry. We attempt to go beyond existing work in complexity and introduce a new model, method and benchmark. We propose EAGLE, large-scale dataset of ~1.1 million 2D meshes resulting from simulations of unsteady fluid dynamics caused by a moving flow source interacting with nonlinear scene structure, comprised of 600 different scenes of three different types. To perform future forecasting of pressure and ve- locity on the challenging EAGLE dataset, we introduce new mesh transformer. It leverages node clustering, graph pooling and global attention to learn long- ong-range dependencies between spatially distant data points without needing large number of iterations, existing GNN methods do. We show that our transformer outper- forms state-of-the-art performance on, both, existing synthetic and real datasets and on EAGLE. Finally, highlight that our approach learns to attend to airflow, integrating complex information in a single iteration.
INTRODUCTION
Despite consistently being at the center of attention of mathematics and computational physies, solv- ing the Navier-Stokes equations governing fluid mechanics remains an open problem. the absence of analytical solution, fluid simulations are obtained by spatially and temporally discretizing dif- ferential equations, for instance with the finite volume or finite elements method. These simulations computationally intensive, take up to several weeks for complex problems and require expert configurations of numerical solvers.
Neural network- bessed physics simulators may represent a convenient substitute in many ways. Be- yond the expected speed gain, their differentiability would allow for direct optimization of fluid mechanies problems (airplane profiles, turbulence resistance, etc.), opening the way to replace tradi- tional trial-and-error approaches. They would also be an alternative for solving complex PDEs where numerical resolution is intractable. Yet, the development such models slowed down by the dif- ficulty of collecting data in sufficient quantities reach generalization. Velocity and pressure field measurements on real world systems require large and expensive devices, and simulation faces the problems described above. For all these reasons, few datasets freely available for training high- capacity neural networks, and the existing ones either address relatively simple problems which can be simulated in reasonable time and exhibiting very similar behaviors (2D flow cylinder, airfoil

---

Published as a conference paper at ICLR 2023
Timestep
Timosoap
Mettsllns Afterso
We introduce EAGLE, a large-scale dataset for learning complex fluid mechanies, accu- rately simulating the air flow created by 2D drone in motion and interacting with scenes of varying 2D geometries. We address the problem through an autoregressive model and self-attention over kens in a coarser resolution, allowing to integrate long-range dependencies in single hop sap--showw shown in the given example by the attention distributions for which follows the airflow.
Figure I:
Pfaff et 2021 Han et 2022)) or simulations of very high precision, but limited to a few different samples only GGaham et al. 2016 Wu et al., 2017).
In this paper, we introduce EAGLE, a large-scale dataset for learning unsteady fluid mechanics. We accurately simulate the airflow produced by two-dimensional unmanned aerial vehicle (UAV) mov- ing in 2D environments with different boundary geometries. This choice has several benefits. It mod- els the complex ground effect turbulence generated by the airflow of an UAV following a control law, and, up to our knowledge, thus significantly more challenging than existing datasets. leads to highly turbulent and non-periodic eddies, and high flow variety, the different scene geometries generate completely different outcomes. At the same time, the restriction to a 2D scene (similar to existing datasets) makes the problem manageable and allows for large-scale amounts of simulations (~1.1m meshes). The dataset will be made publically available upon publication.
As second contribution, propose new multi-scale attention- based model, which circumvents the quadratic complexity of multi-head attention by projecting the mesh onto a learned coarser rep- resentation yielding fewer but more expressive nodes. Conversely standard approaches based on graph neural networks, we show that our model dynamically adapts to the airflow in the scene by focusing attention not only locally, but also over larger distances. More importantly, attention for specific heads seems to align with the predicted airflow, providing evidence of the capacity of the model integrate long range dependencies in a single hop see Figure T We evaluate the method on several datasets and achieve dfate-of-the-ar art performance on two public fluid mechanics datasets (Cylinder- Flow, (Pffff et 202T) and Scalar- Flow (Eckert et al. 2019)), and on EAGLE.
2 RELATED WORK
Fluids datasets for deep learning are challenging to produce in many ways. Real world mea- surement is complicated, requiring complex velocimetry devices (Wang et al. 2020; Discetti Co- letti 2018 Erichson et al. 2020). Remarkably, (Eckert al. 2019 De Bézenac et al. 2019) lever- age alignment with numerical simulation to extrapolate precise GT flows on real world phenomena
(smoke clouds and sea surface temperature). Fortunately, accurate simulation data can by acquired
through several solvers, ranging from computer graphics-oriented simulators (Takahashi et al. 2021
Pfaff & Thuerey 2016) accurate computational fluid dynamics solver (OpenFOAM' Ansys Flu-
ent, ...). A large body of work (Chen et al.) [202 Plaff et al. 2021 Han et al. 2022, Stachenteld al. 2021 Pfaff al. 202I) introduces synthetic datasets limited simple tasks, such as 2D flow past cylinder. EAGLE falls into this synthetic category, but differs in two main points: (a) simu- lations rely on hundreds of procedurally generated scene configure gortguuttiinns, requiring several weeks of regh-performanc computer, and (b) used engineer-grate fluid solver with demanding turbulence model and a fine domain discretization. For a comparison, see table
calculations
Learning of fluid dynamics - is mainly addressed with message passing networks. Recent work focuses in particular on amoothed-partice hydrodynamics (SPH) (Shao et al. 2022 Ummenhofer
et al. 2020 Shlomi et al.] 2021 Li et al. [219 Allen et al. 2022), somehow related Lagrangian representation of fluids. Lanchez-Gozzlle et al. (2020) proposes to chain graph neural networks
in an Encode-Proed Decooe pipeline to learn interactions between particles. SPH simulations
2

---

Published as a conference paper at ICLR 2023
Dyn. Scene
Dyn.
nodes
#
Dataset
Size
Public
Mesh
(avg)
meas.
CylinderFlow
15Gh
.885
0.72M
Pfaff et al. (2021)
AirFoil
56Gb
5,233
0.72M
KS Equation
64
1,200
Incomp, Dec.
N.A
X
X(Grid)
2.304
210
Stachenfeld et al. 2021
Comp. Dec.
32,768
35
Rad. Cooling
32,768
30
Han ct al. 2022
Vascular Flow
N.A. 351Gb
X
7,561
5,250
Eckert ct aL. 201
X
X(Grid)
1.7M
0.015M
Bezenac EAGLE(OUrS)
(2019)
SST
N.A
X(Grid)
4,096
0.1M
270Gb
3,388
1.18M
Table 1: Fluid mechanics datasets in the literature. To the best of our knowledge, EAGLE is the first dataset of such scale, complexity and variety. Smaller-scale datasets such as [Li et al. (2008); Wu et al. (2017) have been excluded, they favor simulation accuracy over size. Thee datasets in Stachenfeld et al. (202T) are not public, but can be reproduced from the information in the paper.
very suitable for applications with reasonable number of particles but larger-scale simulations (ve- hicle aerodynamic profile, sea flows, etc.) remain of scope. In fluid mechanics, Eulerian rep- resentations are more regularly used, where the flow of quantities are studied on fixed spatial cells. The proximity to images makes uniform grids appealing. which lead to the usage of convolutional networks for simulation and learning (De Bézenac et al., 2019 Ravuri et al., 2021; Ren et al.) [022]
Liu et al. 2022] Le Guen & Thome 2020). For instance, Stachenfeld et al. (2021) takes the prin-
ciples introduced in SSanceez-Gozzak et al. (2020) applied to uniform grids for the prediction of turbulent phenomena. However, uniform grids suffer from limitations that hinder their generalized use: they adapt poorly complex geometries, especially strongly curved spatial domains and their spatial resolution is fixed, requiring a large number of cells for a given precision.
convenient way of solving the issues raised by uniform grids. Nodes can then be sparser in some areas and denser in areas of interest. Graph networks (Battaglia al. (016)) are well suited for this type of structure. The task was notably introduced in Praff et al. 2021) with MeshGraphNer, an Prooode-rocess-Deccode pipeline solving mesh-based physics problems. (Lienen & Günnemann 2022) introduced a graph network structure algorithmically aligned with the fimite element method and show good performances on several public datasets. Close to our work, Haan al. (2022) leverages temporal attention mechanism on coarser mesh to enhance forecasting accuracy over longer horizon. In contrast, our model is based on spatial transformer, allowing node to communicate not only with its neighbors but also over greater distances by dynamically adapting attention to the airflow.
Deep Learning non-uniform meshes
EAGLE comprised of fine-grained fluid simulations defined on irregular triangle meshes, which we argue is more suited to a broader range of applications than regular grids and thus more represen- tative industrial standards. Compared to grid-based datasets De Bézenac et al, (2019); Stachenfeld al. (2021), irregular meshes provide better control over the spatial resolution, allowing for finer dis- cretization near sensitive areas. This property is clearly established for most fluid mechanics solvers (Versteeg & Malalasekera 2007) and seems transfer well to simulators based on machine learning (Praff et al. (2021). However, using triangular meshes with neural networks is not as straightfor regular grids. Geometric deep learning (Bronstein et al.) 2021> and graph networks (Battaglia eet al. 2018) have established known baselines but this remains an active domain of research. Ex- isting datasets focus on well-studied tasks such the flow past an object (Chen al. 202 Pfaff eet al. 2021) or turbulent flow on an airfoil (Thuerey et al. 2020 Sckar et al. 2019) These are well studied problems, for some of which analytical solutions exist, and they rely on a large body of work from the physics community. However, the generated flows, while being turbulent, are merely
steady or periodic despite variations in the geometry. With EAGLE, we propose a complex task, with convoluted, unsteady and turbulent air flow with minimal resemblance across each simulation.
3 THE EAGLE DATASET AND BENCHMARK
we built EAGLE in order to meet a growing need for a fluid mechanics dataset in ac- cordance with the methods used in engineering, i.e. reasoning irregular meshes. To significantly increase the complexity the simulations compared to existing datasets, we propose a proxy task consisting in studying the airflow produced by dynamically moving UAV in many scenes with
Purpose
3

---

Published as a conference paper at ICLR 2023
Step
Spline Triangulor
Figure 2: Velocity field norm over time for three episodes, one for each geometry type. Turbulence is significantly different from one simulation to another and strongly depends on the ground surface.
variable geometry. This is motivated by the highly non-steady turbulent outcomes that this task gen- crates, yielding challenging airflow be forecasted. Particular attention has also been paid to the practical usability of EAGLE with respect to the state-of-the-arr in future forecasting of fluid dy- namics by controlling the number of mesh points, and limiting confounders variables to a moderate amount (i.e. scene geometry and drone trajectory).
Simulation and task definition - we simulate the complex airflow generated by 2D unmanned aerial vehicle maneuvering in 2D scenes with varying floor profile. While the scene geometry varies, the UAV trajectory is constant: the UAV starts in the center of the scene and navigates, hovering near the floor surface. During the flight, the two propellers generate high-paced air flows interacting with each other and with the structure of the scene, causing convoluted turbulence. To produce wide variety of different outcomes, we procedurally generate a large number of floor profiles by interpolating set of randomly sampled points within a certain range. The choice of interpolation order induces drastically different floor profiles, and therefore distinct outcomes from one simulation to another. EAGLE contains three main types of geometry depending on the type of interpolation (see Figure 2): (i) Step: surface points are connected using step functions (zero-order interpolation), which produces very stiff angles with drastic changes of the air flow when the UAV hovers over step. (ii) Triangular: surface points are connected using lincar functions (first-order interpolation). causing the appearance of many small vortices different location in the scene. (iii) Spline: surface points are connected using spline functions with smooth boundary. causing long and fast trails of air, occasionally generating complex vortices.
EAGLE contains about 600 different geometries (200 geometries of each type) corresponding to roughly 1,200 flight simulations (one geometry gives flight simulations depending whether the drone is going to the right or to the left of the scene), performed at 30 fps over 33 seconds, resulting in 990 time steps per simulation. Physically plausible UAV trajectories are obtained through MPC control of (flow agnostic) dynamical system we design for a 2D drone. More details and statistics are available in appendix A
We simulated the temporal evolution of the velocity field as well as the pressure field (both static and dynamic) defined over the entire domain. Due to source motion, the triangle mesh on which these fields are defined need to be dynamically adapted to the evolving scene geometry. More formally, the mesh is a valued dynamical graph M' = (A'', V., PP) where N is the set of nodes, the edges, 1 is field of velocity vectors and P is field of scalar pressure values. Both physical quantities are expressed at node level. Note that the dynamical mesh is completely flow-agnostic, thus no information about the flow be extrapolated directly from the future node positions. Time dependency will be omitted when possible for sake of readability.
Numerical simulations Reynolds Averaged Navier-Stokes equations of the Reynolds stress model. It uses five equations to model turbulence, more accurate than standard k-e k-w models (two equations). This resulted in
carried out using the software Ansys R Fluent, which solves the
4

---

Published as a conference paper at ICLR 2023
Graph Foolang
Desader Up Sampling
Clustering
Figure 3:
The mesh transformer encodes the input mesh node values (positions, pressure and ve- locity), reduces the spatial resolution through clustering graph pooling, and performs multi-head self-attention on the coarser level of cluster centers. A decoder upsamples the token embeddings to the original resolution and predicts pressure and velocity time step / 1.
3.9TB of raw data with ~ 162,760 control points per mesh. We down- sampled this to 3,388 points in average, and compressed it to 270GB. Details and illustrations are given in appendix A
Task for what follows, define x; the 2D position of node i, D; its velocity, P: pressure and n, the node type, which indicates if the node belongs to a wall, an input or an output boundary. We interested in the following task: given the complete simulation state time namely M', as well as future mesh geometry ,/rth, gr+h forecast the future velocity and pressure fields Br+h, pr+t, i.e. for all positions we predict 6/ f/th over horizon h. Importantly, consider the dynamical re- re-meshing step .\' _/*hh to be known during inference and thus not required to be forecasted.
LEARNING UNSTEADY AIRFLOW
Accurate flow estimations require data on a certain minimum spatial and temporal scale. Deviations from optimal resolutions. i.c. data sampled with lower spatial resolutions or lower frame rates, are typically very hard to compensate through models of higher complexity, in particular when the es- timation is carried out through numerical simulations with an analytical model. The premise of our work is that machine learning can compensate loss in resolution by picking up longer rate regular- ities in the data, trading data resolution for complexity in the modeled interactions. Predicting the outcome for given mesh position may therefore require information from larger neighborhood, whose size can depend on factors like resolution, compressibility. Reynolds number etc.
Regularities and interactions meshes and graphs have classically been modeled with probabilistic graphical models (MRFs (Geman & Geman 1984). CRFs (Lafferty et al. |200I] RBMs (Smolen- sky 1986) etc.), and in the DL era through geometric DL (Bronstein et al. 2021) and graph net- works (Battaglia et al. 2018), or through deep energy-based models. These models can capture long-range dependencies between distant nodes, but need to exploit them through multiple itera- tions. In this work we argue for the benefits of transformers and self-attention Vaswani et al. (2017), which in principle capable of integrating long-range interactions in single step.
However, the quadratic complexity of transformers in terms of number of tokens makes its direct application large meshes expensive. While low-complexity variants do exist, e.g. (Katharopoulos ctt al. 2020), we propose a different Ansatz, shown in Figure 3 we propose to combine graph cluster- ing and learned graph pooling perform full attention on coarser scale with higher-dimensional node embedding. This allows the dot dot-product similarity of the transformer model - which is at the heart of the crucial attention operations input signals, similar to the settings in other applications. In NL., attention typically operates on word embeddings Vaswani et al. (2017), and in vision either on patch embeddings Dosovitskiy et al.) (2021) or on convolutional feature map cells Waan et al. (2018). In the sequel, we present the main modules our model; further details are given in appendix B
to operate on a semantic representation instead of on raw
down-scale mesh resolution through geometric clustering, which is indepen- dent of the forecasting operations and therefore pre-computed offline. A modified k-means clustering applied to the vertices '/ of each time step and creates clusters with a constant number of nodes, details are given in appendix B.I The advantages are twofold: (a) the irregularity and adaptive res- olution of the original mesh preserved, as high density region will require more clusters, and (b)
Offfine Clustering
5

---

Published as a conference paper at ICLR 2023
constant cluster sizes facilitate parallelization and allow to speed up computations. In what follows, let Cg be the kdah cluster computed on mesh MM'
Encoder- the initial mesh MM' is converted into graph 9 using the encoder in Pfaff et al. 2021 More precisely, node and edge features are computed using MLPs & node and &edger giving
n mnode n;). e) = Pedge (x; x,, -x,11x;-x,II1 x,II).
The encoder also computes an appropriate positional encoding based upon spectral projection F(x). We also leverage the local position of each node in its cluster. Let X be the barycenter of cluster Ct. then the local encoding of node i belonging cluster is the concatenation S; LF(x,) F($g-x,)] Finally, series of Graph Neural Networks (GNN) extracts local features through message passing:
+ 4 odge (Inf Sil. 1y S
=n, + W Wnode (Ir S1l
Σ
The superscript e indicates the layer, and w mode and wh respectively. The exact architecture ability, in whats follows, will note 11; = n,, and eij ej
"dgee MLPs which encode nodes and edges, are given in appendix B For the sake of read-
Graph Pooling summarizes the state of the nodes of the same cluster C₁ in single high- dimensional embedding Wig on which the main neural processor will reason. This is performed with a Gated Recurrent Unit (GRU)[Cho et (2014) where the indi nodes integrated sequentially in random order. This allows to learn more complex integration of features than a sum. Given an inital GRU state So node embeddings are integrated iteratively, indicated by superscript n.
n++1 = GRU Mes f.). ND), i E C2. W² = &eluster (h
where N |C&l and Ceuuster is MLP. GRU(.) denotes the update equations of a GRU, where omitted gating functions from the notation. The resulting set of cluster embeddings W Wglk=I.K significantly reduces the spatial complexity of the mesh.
Attention Module - consists a transformer with M layers of multi-head attention (MHA) Vaswani al. 2017) working on the embeddings W of the coarse graph. Setting w =0g, we get for layer
m:
w MHA @= ww F(X)). K=W,=W
(4)
where Q, K and V are, respectively, the query, key and value mappings of a transformer. We to Vaswani et al. (017) for the details of multi-head attention, denoted as MHA(-).
Decoder the output of the attention module is calculated on the coarse scale, one embedding per cluster. The decoder upsamples the representation and outputs the future pressure and velocity field on the original mesh. This upsampling is done by taking the original node embedding 7Ι and catenating with the cluster embedding w.M followed by the application of GNN, whose role is to take the information produced on a coarser level and correctly distribute it over the nodes i. To this end, the GNN has access the positional encoding of the node, which is also concatenated:
{ 0 = g+1 ppl=p'' 6
(6 6,) GNN (Dn: w f.l).
(5)
where i E Cg and GNN(-) is the graph network variant described in equation parameters are not shared. Our model trained end-to-end, minimizing the forecasting error over horizon where balances the importance of pressure field over velocity field:
L= Σ MSE vtt (),+++),++)))) + Σ MSE ()
(6)
5
EXPERIMENTS
We compare method against three competing methods for physical reasoning: MeshGraphNet (Praff et al. 202T) (MGN) is Graph Neural Network based model that relies on multiple chained message passing layers. GAT is based upon MGN where the GNNs interactions replaced by

---

Published as a conference paper at ICLR 2023
+100
+200
+250
MashCr Grand
Figure 4: Qualitative comparisons with the state of the art on EAGLE. We color code the norm of the velocity field.
Dataset
Cylinder Flow
Scalar Flow
EAGLE
Horizon
+1
+50
+250
+1
+50
+100
+1
+50
+250
MeshGraphNet
0.0058
0.0405
0.0792
0.0374
0.3193 0.5944
0.0916
0.5698
0.9896
GAT
0.01
0.1774
1.3336
0.0434
0.3991
0.6414
7.5587
19.260
26.867
DilResNet
0.0429
0.0626
0.1295
0.0372
0,2212
0.3975
0.2987
0.5650
0.8944
Ours
0.0030 0.0221 0.0735
0.0128
0.0869 0.1467
0.0811
0.3495
0.6357
Table 2: Norm. RMSE (velocity and pressure) for our model (cluster size = 10) and the baselines.
graph attention transformers (Veličković et al., 2017). Compared to our mesh transformer, here attention is computed over the one-ring of each node only. DilResNet (DRN) Stachenfeld et al. (202T) differs from the other models as it does not reason over non uniform meshes, but instead uses dilated convolution layers to perform predictions on regular grids. To evaluate this model on EAGLE, we interpolate grid-based simulation over the original mesh, see appendix A.2 During validation and testing, we project airflow back from the grid to the original mesh in order to compute comparable metrics. All baselines have been adapted to the dataset using hyperparameter sweeps, which mostly lead to increases in capacity, explained by EAGLE's complexity.
We also compare to two other datasets: Cylinder-Flow (Pfaff et al. 2021 simulates the airflow behind cylinder with different radius and positions. This setup produces turbulent yet periodic airflow corresponding to Karman vortex. Scalar-Flow (Eckert et al. 2019) contains real world mea- surements of smoke cloud. This dataset built using velocimetry measurements combined with numerical simulation aligned with the observations. Following Lienen & Güinneman]](203) Kohl al. 2020), reduce the data 2D grid-based simulation by averaging along the x-direction.
We evaluate all models reporting the sum of the root mean squared error (N-RMSE) on both pressure and velocity fields, which have been normalized wrt to the training set (centered and reduced), and we provide finer-grained metrics in appendix C.T
Existing datasets show little success discriminate the performances of fluid mechanics mod- els (see table 2). On Cylinder-Flow, both ours and MeshGraphNet reach near perfect forecasting accuracy. Qualitatively, flow fields hardly distinguishable from the ground truth at least for the considered horizon (see appendix C.2 As stated in the previous sections, this dataset is a great task to validate fluid simulators, challenging benchmark, as these real world measurements are limited in resolution and quantity. Our model obtains good quantitative results, especially on a longer horizon, showing robustness to er- accumulation during negr-reegessive forecasting. Yet, model achieved visually satisfactory results, the predictions remain blurry and leave room for improvements (cf figure in appendix).
may considered as saturated. Scalar-Flow is a much more
more clearly assessed EAGLE. Our model gives excellent results and outperforms competing baselines. succeeds in forecasting turbulent eddies even after long prediction horizon. Our model outperforms MeshGraphNet, which provides evi-
Comparisons with the state-of-the-art-

---

Published as a conference paper at ICLR 2023
(a)
(b)
II May + 1)1l: MeshGraphNer
Receptive field per layer of GNN
13
IV Mit + 1)]l Ours
Figure 5: Locality of reasoning. (a) velocity of example flow and selected point D: (b): The receptive field for this point for the MeshGraphNee model (Pfaff et al. 2021) 1) restricted to local neighborhood, also illustrated through the overlaid gradients IIV Mo+ 1l1, (c): the receptive field of our method covers the whole field and the gradients indicate that this liberty is exploited; (d) the attention distributions for point D. certain maps correlate with airflow. Attention maps can be explored interactively using the online tool at https:: leggle-dataset liiiaaset.ittub.t.... github.ic
dence for the interest in modeling long-range interactions with self-attention. GAT seems to struggle on our challenging dataset. The required increase in capacity difficult do for this resource hungry model, we failed even on the 40GB A100 GPUs of high-end Nvidia DGX.
DilResNet shows competitive performances on EAGLE, consistent with the claims of the original paper. However, it fails to predict details of the vortices (cf. Figure 4 This model leverages grid- based data, hence was trained on voxelled simulation, finally projected back on the triangular mesh during testing. This requires precaution in assessment. We try to limit projection error by setting images contains ten times more pixels than nodes in the actual mesh. Yet. even that scale, we measure that the reconstruction error represents roughly a third of the final N-RMSE This points out that grid-based are not suited complex fluid problem such as EAGLE, which require finer spatial resolution near sensitive areas. We expose failure cases in appendix C.3
Self-attention - is a key feature our models, as shown in FigureBp and c, which plots the gradient intensity of selected predicted point D situated on the trail wrt. to all input points, for fixed trained model weights. MeshGraphNet is inherently limited to a neighborhood determined by the number chained GNNs, the receptive field, which is represented concentric black circles overlaid over the gradients. In contrast, our model not spatially limited and can exchange information across the entire scene, even possible in a single step. The gradients show that this liberty is exploited.
N-RMSE
Ablation
Clustering
(+250)
GNN
20
1.3484
1.0258
One-ring
20
0.7976
1
0.7876
Average
20
0.7797
Ours
20
0.6572
Figure 6: Ablations: GNN
places global attention by a set of L GN the coarser mesh. One- ring constrains attention to the one- ring. Average forces uniform atten- tion.
In the same figure we also show the attention maps, per head and layer, for the selected point D near the main trail Fig- ure 5t. Interestingly, our model discovers to attend not only to the neighborhood (as GNN would), but also much farther areas. More importantly, observe that certain heads explicitly (and dynamically) focus the airflow, which provides evidence that attention is guided by the regularities the input data. We re- leased an online tool allowing interactive visualization and exploration of attention and predictions, available at /tttts:////agg/// http::///eelleeedaattteeteeee hitpaab.tttttt.t.....e
Ablation studies indicate how global attention impacts performance: (a) closer to MeshGraphNet, we replace the attention layers by GNNs operating on the coarser mesh, allowing message passing between nearest cluster only; (b) we limit the receptive field of MHA the one-ring of the cluster;
8

---

Published as a conference paper at ICLR 2023
Figure 7: Impact of cluster size. Left: We color code RMSE in logarithmic scale on the velocity
field near a relatively turbulent at a horizon of h=400 steps of forecasting. Right: Error (+50), inference time and FLOPs for different cluster sizes and the baselines.
Model
Ours
MGN
Ablated Geometry
Stp
Spl
Tri
Stp
Spl
Tri
Stp
0.927
0.865
1.132
0.828
2,062
1.236
1.347
1.116
N-RMSE (+250)
Spl
0.595
0,584
0.857
0.488
1.257
0.941
1.037
0.807
Tri
0.730
0.732
1.049
0.647
1.685
1.100
1.131
1.037
Table 3: Generalization to unseen geometries: we evaluate our model and MeshGraphNet in dif- ferent setups, evaluating on all geometry types but removing one from training. Our model shows satisfactory generalization, also highlighting the complementarity of each simulation type.
(c) we enforce uniform attention by replacing it with average operation. As shown in table 6 attention is a key design choice. Disabling attention to distant points has a negative impact on RMSE, indicating that the model leverages efficient long-range dependencies. Agnostic attention to the entire scene is not pertinent either: to be effective, attention needs to dynamically adapt to the predicted airflow. We also conduct a study on generalization down-sampled meshes in appendix C.4
The role of clustering is to summarize a set of nodes into a unique feature vector. Arguably, with bigger clusters, more node-wist wise information must be aggregated in finite dimensional vector. We indeed observe slight increase in N-RMSE when the cluster size increases (Figure 77)). Nonethe- less, our model appears be robust even aggressive graph clustering as the drop remains limited and still outperforms the baselines. A qualitative illustration is shown figure (left, where we sim- ulate the flow up to 400 time- tme-steps forward and observe the егrоr on a relatively turbulent region. Clustering also acts on the complexity of our model by reducing the number of tokens on which at- tention is computed. We measure a significant decrease in inference time and number of operations (FL.Ps) even when we limit clusters to a small size (Figure 7 and c).
Generalization experiments - highlight the complementarity of the geometry types in EAGLE, since the removal of one geometry in the training set impacts the performances the others. Mesh- GraphNet suffers the most, resulting in a drop ranging from 10% in average (ablation of Spline) to 67% (ablation of Step). On model, the performance losses are limited for the ablation of Step and Spline. The most challenging geometry is arguably Triangular, as the ground profile tends to generate more turbulent and convoluted flows.
CONCLUSION
We presented a new large-scale dataset for deep learning in fluid mechanics. EAGLE contains accu- rate simulations of the turbulent airflow generated by a flying drone in different scenes. Simulations are unsteady, highly turbulent and defined on dynamic meshes, which represents a real challenge for existing models. To the best of our knowledge, we released the first publicly available dataset of this scale, complexity, precision and variety. We proposed a new model leveraging mesh transform- ers to efficiently capture long distance dependencies on a coarser scale. Through graph pooling, we show that our model reduces the complexity of multi-head attention and outperforms the competing state-of-the-art -rrt on, both, existing datasets and EAGLE, We showed across various ablations and il- lustration that global attention is a key design choice and observed that the model naturally attends to airflow. Future work will investigate the impact of implicit representations on fluid mechanics, and we discuss the possibility of an extension to 3D data in appendix D
Acknowledgements- recognize support through French grants "Delicio" (ANR- CAATR-19CCO 19-CE23-0000 of call CE23 " "ntelligencee Artificielle" and " "emember" (ANR-20-CHIA0018)) 8), of call "Chaires IA hors centres".

---

Published as a conference paper at ICLR 2023
REFERENCES
Kelsey R. Allen, Tatiana Lopez-Guevara, Kimberly Stachenfeld, Alvaro Sanchez-Gonvalez, Peter Battaglia, Jessica Hamrick, and Tobias Pfaff. Physical design using differentiable learned simula- tors. In arXiv, 2022.
Peter Battaglia, Razvan Pascanu, Matthew Lai, Danilo Jimenez Rezende, et al. Interaction net- works for learning about objects, relations and physics. Neural Information Processing Systems (NewrIPS), 2016.
P.W. Battaglia, J.B. Hamrick, V. Bapst, A. Sanchez-Gonzalez V.F. Zambaldi, M. Malinowski, A. Tacchetti, D. Raposo, A. Santoro, R. Faulkner, Ç. Güllehree F. Song, AJ.. Ballard, J. Gilmer, G.E. Dahl, A. Vaswani, K. Allen, C. Nash, V/ Langston, C. Dyer, N. Heess, D. Wierstra, P. Kohli, M. Botvinick, O. Vinyals, Y. Li, and R. Pascanu. Relational inductive biases, deep learning, and 2018.
graph networks.
Michael M. Bronstein, Joan Bruna, Taco Cohen, and Petar Veličković. Geometric deep learning: Grids, groups, graphs, geodesies, and gauges, 2021.
J. Chen, E. Hachem, and J. Viquerat. Graph neural networks for laminar flow prediction around random two-dimensional shapes. Physics of Fluids, 2021a.
Shengyu Chen, Shervin Sammak, Peyman Givi, Joseph P Yurko, and Xiaowei Jia. Reconstructing high-resolution turbulent flows using physics--gidde neural networks. In International Conference on Big Data (Big Data), 2021b.
Kyunghyun Cho, Bart van Merrienboer, Çaglar Gülçehre, Dzmitry Bahdanau, Fethi Bougares, Hol- ger Schwenk, and Yoshua Bengio. Learning phrase representations using encoder-decoder for statistical machine translation. In EMNLP, 2014.
Mengyu Chu and Nils Thuerey. Data-driven synthesis of smoke flows with cnn-based feature de- scriptors, ACM Transactions on Graphics (TOG), 36(4):1-14, 2017.
Robert L. Cook. Stochastic sampligg in computer graphics. ACM Trans. Graph., 1986. URL https:
7 //doi. (166...org11..115//555555333 8927
Jean-Yves Dantan, Zhicheng Huang, Edoh Goka, Lazhar Homri, Alain Etienne, Nicolas Bonnet, and
Mickael Rivette. Geometrical variations management for additive manufactured product. CIRP Annals Manufacturing Technology, 2017.
Emmanuel De Bézenac, Arthur Pajot, and Patrick Gallinari. Deep learning for physical processes: Incorporating prior scientific knowledge. Statistical Mechanics: Theory and Experiment, 2019.
Stefano Discetti and Filippo Coletti. Volumetric velocimetry for fluid flows. Measurement Science and Technology. 2018.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image worth 16x16 words: Transformers for image recognition scale. International Conference on Learning Representation (ICLR), 2021.
Marie-Lena Eckert, Kiwon Um, and Nils Thuerey. Scalarflow: a large-scale volumetric data set of real-world scalar transport flows for computer animation and machine learning. Transactions on Graphics (TOG), 2019.
Benjamin Erichson, Lionel Mathelin, Zhewei Yao, Steven Brunton, Michael W Mahoney, and J Nathan Kutz. Shallow neural networks for fluid flow reconstruction with limited sensors. Royal Society A, 2020.
Enrico Fonda, Ambrish Pandey, Jörg Schumacher, and Katepalli Sreenivasan. Deep learning in turbulent convection networks. National Academy of Sciences, 2019.
S. Geman and D. Geman. Stochastic Relaxation, Gibbs Distributions, and the Bayesian Restoration of Images. IEEE Transactions on Pattern Analysis and Machine Intelligence, 6(6):721-741, 11 1984.
10

---

Published as a conference paper at ICLR 2023
J. Graham, K. Kanov, X. I. A. Yang, M. Lee, N. Malaya, C. C. Lalescu, R. Burns, G. Eyink, A. Szalay, R. D. Moser, and C. Meneveau. A web services accessible database of turbulent channel flow and its use for testing new integral wall model for les. Journal of Turbulence, 2016.
Xu Han, Han Gao, Tobias Pfaff, Jian-Xun Wang, and Liping Liu. Predicting physics in mesh-reduced space with temporal attention. In International Conference on Learning Reprexentations (ICLR), 2022.
Ali Kashefi, Davis Rempe, and Leonidas J Guibas. A point-cloud deep learning framework for prediction of fluid flow fields on irregular geometries. Physics of Fluids, 2021.
A. Katharopoulos, A. Vyas, N. Pappas, and F. Fleuret. Transformers are RNNs: Fast autoregressive transformers with linear attention. In ICML, 2020.
Corentin Kervadec, Theo Jaunet, Grigory Antipov, Moez Baccouche, Romain Vuillemot, and Chris- tian Wolf. How transferable are reasoning patterns in vqa? In Conference on Computer Vision and Puttern Recognition (CVPR), 2021.
Hyun-Goo Kim. A method of accelerating the convergence of computational fluid dynamics for micro-siting wind mapping. Computation, 2019.
Georg Kohl, Kiwon Um, and Nils Thuerey. Learning similarity metrics for numerical simulations. In International Conference on Machine Learning (ICML). PMLR, 2020.
J. Lafferty, A. McCallum, and F. Pereira. Conditional random fields: Probabilistic models for seg- menting and labeling sequence data. In ICML, 2001.
Vincent Le Guen and Nicolas Thome. Disentangling physical dynamics from unknown factors for unsupervised video prediction. In Computer Vision and Pattern Recognition (CVPR), 2020.
Yi Li, Eric Perlman, Minping Wan, Yunke Yang, Charles Meneveau, Randal Burns, Shiyi Chen, Alexander Szalay, and Gregory Eyink. A public turbulence database cluster and applications to study lagrangian evolution of velocity increments in turbulence. Journal of Turbulence, 2008
Yunzhu Li, Jiajun Wu, Russ Tedrake, Joshua B. Tenenbaum, and Antonio Torralba. Learning par- ticle dynamies for manipulating rigid bodies, deformable objects, and fluids. In International Conference on Learning Representations (ICLR), 2019.
Marten Lienen and Stephan Günnemann. Learning the dynamics of physical systems from sparse ob- servations with finite element networks. In International Conference on Learning Representation (ICLR), 2022.
Yang Liu, Rui Hu, Adam Kraus, Prasanna Balaprakash, and Aleksandr Obabko. Data-driven mod- eling of coarse mesh turbulence for reactor transient analysis using convolutional recurrent neural networks. Nuclear Engineering and Design, 2022.
Arvind T Mohan, Nicholas Lubbers, Daniel Livescu, and Michael Chertkov. Embedding hard phys- ical constraints in neural network coarse-graining of 3d turbulence. arXiv preprini, 2020a.
Arvind Mohan, Dima Tretiak, Misha Chertkov, and Daniel Livescu. Spatio-temporal deep learning models of 3d turbulence with physics informed diagnostics. Journal of Turbulence,
2020b. 2020b.
Tobias Pfaff and Nils Thuerey. Mantaflow, 2016. URL http:://mantaallorccom//ii html
Tobias Pfaff, Meire Fortunato, Alvaro Sanzhez-Gonzalez, and Peter W. Battaglia. Learning mesh- based simulation with graph networks. In International Conference on Learning Representations (ICLR), 2021.
Suman Ravuri, Karel Lene, Matthew Willson, Dmitry Kangin, Remi Lam, Piotr Mirowski, Megan Fitzsimons, Maria Athanassiadou, Sheleem Kashem, Sam Madge, et al. Skilful precipitation now- casting using deep generative models of radar. Nature, 2021.
11

---

Published as a conference paper at ICLR 2023
Pu Ren, Chengping Rao, Yang Liu, Jian-Xun Wang, and Hao Sun. Phyernet: Physics-informed convolutional-recuree network for solving spatiotemporal pdes. Computer Methods in Applied Mechanics and Engineering, 2022.
Angel Romero, Sihao Sun, Philipp Foehn, and Davide Scaramuzza. Model predictive contouring control for time-optimal quadrotor flight. Transactions on Robotics (TAC), 2022.
Alvaro Sanchez-Gonzalez, Jonathan Godwin, Tobias Pfaff, Rex Ying, Jure Leskovee, and Peter Battaglia. Learning to simulate complex physics with graph networks. In International Con- ference on Machine Learning (ICML), 2020.
Vinothkumar Sekar, Qinghua Jiang, Chang Shu, and Boo Cheong Khoo. Fast flow field prediction over airfoils using deep learning approach. Physics of Fluids, 2019.
Yidi Shao, Chen Change Loy, and Bo Dai. Sit: Simulation transformer for particle-based physics simulation, 2022.
Jonathan Shlomi, Peter Battaglia, Jean Vlimant. Graph neural networks in particle physies. Machine Learning: Science and Technology, 2021.
Smolensky. Information processing in dynamical systems: Foundations of harmony theory. In Parallel Distributed Processing: Volume 1: Foundations. MIT Press, 1986.
Kimberly Stachenfeld, Drummond B Fielding, Dmitrii Kochkov, Miles Cranmer, Tobias Pfatf, Jonathan Godwin, Can Cui, Shirley Ho, Peter Battaglia, and Alvaro Sanchez-Gonzale Learned coarse models for efficient turbulence simulation. In International Conference Learning Rep- resentation (ICLR), 2021.
Tetsuya Takahashi, Junbang Liang. Yi-Ling Qiao, and Ming C Lin. Diffcrentiable fluids with solid coupling for learning and control. In Proceedings of the AAAI Conference Artificial Intelligence, 2021.
Nils Thuerey, Konstantin Weißenow, Lukas Prantl, and Xiangyu Hu. Deep learning methods for reynolds-averaged navier-stokes simulations of airfoil flows. American Institute of Aeronautics and Astronautics (AIAA), 2020.
Benjamin Ummenhofer, Lukas Prantl, Nils Thuerey, and Vladlen Koltun. Lagrangian fluid simu- lation with continuous convolutions. In International Conference Learning Representations
(ICLR), 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Neural information processing systems (NeurIPS), 2017.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. In International Conference on Learning Representation (ICLR), 2017.
Henk Kaarle Versteeg and Weeratunge Malalasekera. An introduction to computational fluid dynam- ics the finite volume method.
2007. ISBN 978-0-582-21884
Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In ICLR, 2018
Yuanhao Wang. Ramzi Idoughi, and Wolfgang Heidrich. Stereo event-based particle tracking ve- locimetry for 3d fluid flow reconstruction. In European Conference on Computer Vision (ECCV), 2020.
Steffen Wiewel, Byungsoo Kim, Vinicius C Azevedo, Barbara Solenthaler, and Nils Thuerey. Latent space subdivision: stable and controllable time predictions for fluid flow. In Computer Graphics Forum, 2020.
Xiaohua Wu, Parviz Moin, James M. Wallace, Jinhie Skarda, Adrián Lozano-Durán, and Jean-Pierre Hickey. Transitional turbulent spots and turbulent turbulent spots boundary layers. National Academy of Sciences, 2017.
12

---

Published as a conference paper at ICLR 2023
Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng. Shuxin Zheng, Chen Xing. Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tieyan Liu. On layer normalization in the transformer architecture. In International Conference on Machine Learning (ICML), 2020.
13

---

Published as a conference paper at ICLR 2023
Appendix
A DATASET DETAILS
A.1 STRUCTURE AND
The EAGLE dataset is composed of exactly 1,184 simulations of 990 time-steps (33 seconds at 30 fps). Scene geometries are arranged in three categories based on the order of the interpolation used to generate the ground structure: 197 Step scenes, 199 Triangular and 196 Spline. A geometry gives two simulations depending on whether the drone is crossing the left or the right part of the scene. A proper train/valid/test splitting provided ensuring that each geometry type equally represented. The train split contains 948 simulations, while test and valid splits each contain 1 118 simulations.
Simulation details 'The scene described as a mx2.5 m 2D surface. Wall boundary conditions (zero velocity) are applied to the frontiers, except for the top edge, which is an outlet (zero diffusion of flow variables). The propellers is modeled as two squares starting the middle of the scene, with wall boundary conditions on the left, right and top edges, and inlet condition for the bottom edge (normal velocity of intensity proportional to the rotation speed of the propeller). We mesh the scene with triangular cells of an average size of 15 mm, and add inflation near wall boundaries. We let the simulator updates the mesh during time with default parameters.
Drone trajectory control (MPC) of a dynamical model of a 2D drone allowing realistic trajectory tracking. The model is obtained by constraining the dynamics of 3D drone model (Romero et al.) (2022) to motion in a 2D plane and reducing the number of rotors to two. The drone can therefore move along the axis x and y, and pivot around the z-axis perpendicular the simulation plane follows:
has recei ved special care, and is obtained using model predictive control
---K (2 + 22) K;(2 + 22)x
ý (2 + (2) cos(O) g K2(2, + +2,
Ø Kg(22 - 2 2
where x, yss the 2D position of the drone and 0 its orientation, Q and 52 the left/right propeller rotation speed, g 9.88 m/s is acceleration (gravity), and K2 10-4 K₂ 5x 5x1005, K3 5.5x10 are physical constants depending on drone geometry. The resulting trajectories represent physically plausible outcomes, taking into account inertia and gravity.
Mesh down-sampling - consists in simplifying the raw simulation data, as they are not suitable for direct deep learning applications, and require post-processing (see Figure Saj. The simulation software leverages a very fine- tree-graied mesh dynamically updated in order to accurately solve the Navier-Stokes equations. The main step thus consists simplifying the mesh to a reasonable number of nodes. Formally, our goal is to construct a new coarser mesh (N) E(r') based upon the raw mesh proposed by the simulation software (N()), E()). To cope with the dynamic nature of the simulation mesh, our approach consists in dividing the target node set into static and dynamic part (1) S D()).
The static mesh is obtained by subsampling the simulation point cloud using Poisson Disk Sampling (CCook) 1986)). However. the spatial density of (z()) evolves over time (certain areas of space are more densely populated at the end of the simulation than at the start). To preserve finer resolution near relevant regions, we thus concatenated regularly spaced point clouds N((r,) into single set. We then sub-sample the resulting set by randomly selecting a point, and deleting all neighbors repeated until no more point is at distance less than R from another. We used an adaptive radius R correlated to the density map: when the original point cloud is dense, the radius is smaller. Conversely, the radius increases in sparse areas. An example of the density map is provided in Figure 8b
sphere of radius R around the chosen point. This operation is
The dynamic mesh is mandatory to track drone motion accurately. We therefore complete the static mesh with dynamical part that follows the boundaries of the UAV. To do so, used
14

---

Published as a conference paper at ICLR 2023
(a)
(c)
(
Figure 8: (a) sample of raw simulation measurements obtained on a high resolution mesh. This single snapshot contains 158.961 nodes. (b) example of node density map controlling the sampling disk radius. The raw mesh is more dense near the boundaries and on the left side, as this sample is taken from simulation where the drone explores the left region of the scene. (c) Mesh simulation at final resolution. We drastically simplified the mesh while maintaining a satisfactory level of details.
the ground truth trajectory to track drone position and orientation across time and extrapolate bounding boxes, which are then transformed into point clouds by sub-dividing the box into several points.
Finally, the edge set E'()) is computed using constrained Delaunay triangulation to prevent triangles to spawn outside the domain. Once (1),,(()) has been computed, we evaluate the pressure and velocity field V(), P()) the nodes by averaging the three nearest points in raw simulation data. We illustrate the final result in figure &c Better mesh simplification algorithm exists, notably minimizing the interpolation error, yet such algorithms rely on the simulated flow to compute the mesh, which may embed unwanted biases or shortcuts in the mesh geometry.
A.2 GRID BASED DATASET
One of the baselines, DilResNet (Stachenteld et al. 2021), relies on convolutional layers for future forecasting of turbulent flows, and therefore requires projecting EAGLE and Cylinder-Flow on uniform rectangular grid. However, such discretization scheme can not adapt its spatial resolution as function of the geometry of the scene, which therefore constitutes disadvantage with respect to an irregular triangular mesh. To limit this effect, the resolution of the grid is chosen such that the number pixels is at least ten times larger than the number points the triangular mesh.
We project Cylinder-Floo Flow onto a uniform 256 x 64 grid and EAGLE onto 256 x 1 128 grid (the dimensions were chosen to respect the height-width ratio the original data). The value of the pressure and velocity fields at each point in the grid is extrapolated from the nearest point in the raw simulated data. We illustrate this projection figure 9 While the grid-based simulation (figure 9b) seems visually more accurate than the mesh-based simulation (figure dd) we observed that the re- projection error (ie. the error obtained after projecting the grid based data onto the triangular mesh) greater near sensible regions, as for example near the scene boundaries.
MODEL DETAILS
B.1 CLUSTERING
We use our own implementation of the same size Kmeans algorithm described here" Using equally sized clusters has two main advantages:
means
15

---

Published as a conference paper at ICLR 2023
(a) Cylinder Flow on uniform rectangular grid
(c) Cylinder Flow on irregular triangle mesh
(b) EAGLE on uniform rectangular grid
(d) EAGLE on irregular triangle mesh
Figure 9: Illustration of the pixellisation process. The left column (a and shows snapshots of simulations from the grid-based datasets, used to train DilResNet. For comparison, we show the same snapshots the mesh-based datasets (c and d). While resolution seems better on grid-based simulation, it lacks precision near sensible region, which are primordial for accurate forecasts.
Areas of high density will be covered by a greater number of clusters, allowing the adaptive resolution of irregular meshes to be preserved on the coarser mesh.
The model can be implemented efficiently, maximizing parallelization, since clusters can be easily stored as batched tensors.
Since the clustering depends solely on the geometric properties of the mesh (and not on the prediction of the neural network), it is possible apply the clustering algorithm a pre-processing step to reduce the computational burden during training. Note that since the mesh is dynamic, so are the clusters: the k cluster time will necessarily contain the same points time 1.
B.2 ARCHITECTURE AND TRAINING DETAILS
We kept the same training setup for all datasets and trained our model for 10,000 steps with the Adam optimizer and a learning rate of 10 4 to minimize equation 6 with a = 10-1 and H = 8, Velocity and pressure are normalized with statistics computed on the train set, except for Scalar-Flow where better results are obtained without normalization.
&node and &edge are one-layer MLPs with ReLU activations. hidden size and output size of 128 (1), eiy) IR 128). We used L=4 chained graph neural network layers composed of two identical MLP Wedge and Wnode with hidden layer of dimension 128, ReLU activated, followed by layer normalization. The positional encoding function F is defined as follows:
Encoder
F(x) = (xx)(2 sin(2' xx)]
(8)
where x is a 2D vector modeling the position of node i.
Graph Pooling used single layer gated recurrent unit with hidden size of dimension W followed by single layer MLP with hidden and output size of W. This step produces a cluster feature representation 10g R W For Cylinder-Flon Flow and EAGLE, W=512, For Scalar-Flow, W-128.
16

---

Published as a conference paper at ICLR 2023
Attentional module Following (Xiong et al. 2020) attention block is defined follows for an imput w E R"
20 F(()))
U02 MHA 11 111
W03 = w + (in
1C4 LN((w3)
405 = MLP((4)
W6 3/3 + 4/5
where LN layer norm functions, Linearis linear function (with bias), MHA is multi-head attention and MLP is a multi-layer perceptron with one hidden layer of size W. We denote the barycenter of cluster & as xg. We used M =4 chained attention block, with four attention head each. The last attention layer is followed by final layer norm.
Decoder The decoder takes as input the node embeddings '/ the cluster features updated by the attentional module w and the node-wise positional encoding f.. We applied a graph neural network composed of two identical MLP (two hidden layers with hidden size of 128, ReLU activated and layer norm). The resulting node embeddings are fed to a final MLP with two hidden layers and hidden size of 128, with TanH as activation function.
B.3 BASELINES TRAINING DETAILS
After performing grid search to select the best options, we found that training each baseline to minimize equation 6 with Adam optimizer and learning rate of 10-4 produces best results. We vary the weighting factor maintain balance between pressure and velocity. For Cylinder-Flow and EAGLE, we trained the basclines over = 5 time-steps. For Scalar-Flow, we set = 20.
MeshGraphNet- we performed grid search over the number of GNN layers to fit to each dataset, but best results were obtained with the recommended depth L = 15 for each dataset. Conversely to what is suggested in [Pafff et al. (202I) we found that training MeshGraphNet over a longer horizon improves the general performances. We used our own implementation the baseline and make sure to reproduce the results presented in the main paper (for Cylinder-Flow only). We get the best trade-off between velocity and pressure with a = 10.
we performed grid-search over the number of heads per layer and the number of layers. Best results were obtained for 10 layers of graph attention transformer and two attention heads per layer (except for Cylinder-Flow, where four heads slightly improves the performances).
GAT
DilResNet- - we found that increasing the number of blocks improves overall performance, setting the number of convolutional blocks from 20.
The baselines are structurally built to predict pressure field ('+hh) and velocity field p'(th) de- scribed on the mesh geometry current time (t)). Auto-regressive forecasting on a longer horizon thus requires interpolation of the predicted flow to the (provided) future mesh W/r + h). We do not want interpolation to disturb our problem of interest, which is turbulent flow prediction. Therefore, made the interpolation from ( to time // + 1) straightforward. As the vast majority of the mesh remains static (see previous section), only the nodes linked to the UAV need lated. Since they readily be associated in one-to-one relation, nearest point interpolation can be performed automatically by assigning ('(t+)) at these points to v(1+h).
interpo-
MORE RESULTS
C.I DETAILED METRICS
Formally, used the following metrics report our results the test set D:
"
[|x(t) - 000ll2 + ll p()
N-RMSE = D
Σ
Σ
(9)
p
17

---

Published as a conference paper at ICLR 2023
100
100
100
100
100
100
100
100
100
100
100
100
100
100
The maps show the k-number computed for each cluster, that is, the number nodes required to reach 90% of the attention. A low k-number indicates very specialized head (attending to few nodes), while a high k-number indicates uniform attention.
Figure 10:
where e and f are standard deviation of velocity and pressure field computed on the train set.
Detailed metric raw root mean squared error (RMSE) on each field is reported in figure as well as temporal evolution of N-RMSE across prediction horizon. On Cylinder-Flow (Figure velocity error is very similar between MeshGraphNet and ours. Our model slightly outperforms the baseline on the pressure field, yielding overall better performances. However, temporal evolution of the N-RMSE indicated that both models converge to the same accuracy for very long roll-out prediction. On EAGLEE, our model shows excellent stability over long horizon, and produces accurate velocity and pressure estimates.
property which can be calculated for attention maps, and which consists in the number of tokens required to reach 90% of attention (Kervadec al. 22221) This property can be used to characterize the shape of attention maps, varying from peaky attention (requiring few tokens to reach 90%) to more uniform attention heads. We show k-numbers in Figure 10 Interestingly, the k-number maps can be compared with attention maps figure 5it: peaky heads (in blue) are correlated with relatively local attention maps, and conversely, more uniform heads (in red) correspond to tention maps focusing on larger distances, often following the airflow. Some heads have different behavior depending on the selected cluster, and are peaky in some areas (mainly around the bound- aries of scene), but more uniform elsewhere. These cues support the importance of global attention in our model.
K-number
18

---

Published as a conference paper at ICLR 2023
Nomalaed PRGC
Horizon
+50
+250
Field
MGN
0.0004
0.0016
0.0047
0.0095
0.0144
0.0145
GAT
0.0015
0.0025
0.0278
0.0360
0.2595
0.2314
DRN
0.0098
0.0063
0.0152
0.0085
0.0344
0.0152
Ours
0.0003
0.0007
0.0044
0.0035
0.0179
0.0079
(a) CylinderFlow: (Right) RMSE on velocity V and pressure P fields. (Left) Normalized RMSE over forecast- ing borizon. Our mesh transformer overcomes the baselines by a small margin. qualitative results tends to indicates that Cylinder Flow is already well mastered task.
Horizon
+1
+50
+ +00
Field
MGN
0.0009
0.0059
0.0105
0.0568
0.0231
0.1130
GAT
0.0009
0.0066
0.0097
0.0578
0.0200
0.1091
DRN
0.0014
0.0101
0.01 30
0.0750
0.0237
0.1217
Ours
0.0005
0.0024
0.0035
0.0145
0.0059
0.0239
(b) Scalar Flow: (Right) RMSE on velocity V and density fields. (Left) Normalized RMSE over forecasting horizon. Our model shows improvements over the baselines on both fields.
Horizon
+1
+50
+250
Field
MGN
0,0810
0.4256
0.5926
2.2492
1.0702
3.7220
1
GAT 0.1698
64.546
0.8551
162.56
1.0959
227.20
DRN
0.2517
1.4453
0.5374
2.4568
0.9188
3.5824
Ours
0.0537
0.4590
0.3494
1.4432
0.6826
2.4130
(c) EAGLE: (Right) RMSE on velocity and pressure P fields. (Leftt) Normalized RMSE over forecasting horizon. Our largely and consistently and reliably outperforms the competing baselines. While MeshGraphNet and DilResNet shows comparable performances during first time-steps, our model succeed to control enror accumulation for reasonable horizons and eventually presented better sinulations.
Figure 11: Detailed metrics on Cylinder-Flon Flow, Scalar-Flow and EAGLE, evaluated for each base- lines and our model.
19

---

Published as a conference paper at ICLR 2023
C.2
QUALITATIVE RESULTS
1100
1350 stop
1200
1250 90p
stpp
+30 stop
$150 STAP
-200 stap
+210 stop
Figure 12: Examples of prediction forward in time on Cylinder-Flow
20

---

Published as a conference paper at ICLR 2023
+1 step
+25 Step
+50 step
+75 Step
+100 Step
Ground Truth
NechCrannte
CAr
Dilte Mex
Cors
+1 step
+25 Step
+50
+75 slep
+100 step
Ground Truhh
NechCrappatee
GAT
Diibe Neet
Onns
Figure 13: Examples of prediction forward in time on Scalar-Flos
21

---

Published as a conference paper at ICLR 2023
step
+100 step
+150 step
+200 step
+250 step
Ground Trubh
CAT
Dilteette
Clrs
+1 stet
+100 step
+150 step
+200 step
+250 step
Ground Thuch
GAr
Dille Bett
OuT
Figure 14: Examples of prediction forward in time on EAGLE
22

---

Published as a conference paper at ICLR 2023
C.3 FAILURE CASE
Despite the excellent performance of our model against competitive baselines, there is still room for improvement. Some difficult configurations give rise very turbulent flows, widely extended in the scene. The evolution of these flows is more difficult to predict and the models we evaluated failed remains accurate. In these cases, the precision with which the small vortices are simulated is essential, because some of them will grow to become the majority.
Moreover, our model suffers from an error accumulation problem, like any auto-regressive model. Experimentally, we observe that the airflow tends to be smoothed by deep learning models when the prediction horizon increases.
+1 steps
+100 steps
+200 +300 steps
+400 steps
Ground truth
Curs
N-AUMSE
+1 steps
+100 steps
+200 steps
+300 steps
+400 steps
Ground truth
Ours
NC-RMISE
Figure 15: We expose failure cases of our mesh transformer Eagle. The error increases when the flow tends to intensify throughout the scene, and when turbulence dominates. Over a longer prediction horizon, the airflow tends to be smoother and less uubulent.
23

---

Published as a conference paper at ICLR 2023
Training
N-RMSE
90%
80%
70%
60%
90%
0.454
0.497
0.513
0,502
Testing
80%
0.427
0.446
0.467
0.440
70%
0,406
0.405
0.416
0.394
60%
0.401
0.370
0.368
0.348
Table 4: Mesh down-sampling: We train our mesh transformer under different regimes of down- sampling by keeping a fix percentage of points from the initial mesh and removing the others. We evaluate the resulting models on regimes different from training, and observe very little variations in N-RMSS among them.
C,4 GENERALIZATION TO DIFFERENT MESH RESOLUTION
In EAGLE. the number of points varies from one simulation to another, forcing the model to general- ize on meshes of different sizes. We explicitly demonstrate the performance of our mesh transformer on this task in table 4 Four instances of the model are trained on particular regime in which the simulation meshes are randomly down-sampled, respectively at 90%, 80%, 70% and 60% the initial mesh resolution. These models are then evaluated in different regime from the one used for train- ing, either higher (more points on average during test than during training), or lower (fewer points in test than in training). We show that our model generalizes well to these different regimes by giving relatively close N RMSE measurements for a given down- sampligg regime.
D EXTENSION TO 3D FLUID SIMULATION
While we think that 3D simulations are indeed the long-term future this subject, argue that the complexity in factors of variation we need for large-scale machine learning is currently not possible in 3D simulations, and this has motivated our choice for challenging 2D dataset. In this section, discuss the possible extension of our dataset and the mesh transformer method to problems in three dimensions. We address two aspects: data generation itself, and the extension of the method.
D.1 DATA GENERATION
Fluids datasets in 3D are very limited, due the computation time required for simulation. Mesh- based simulation in three dimensions greatly increases the number of points in the mesh, and thus exponentially increases computing time (see numerical evidences Kim (2019); Dantan et al.) (2077). Classical workarounds rely on relaxing physical accuracy or versatility of the solver, e.g. with SPH simulations. Accurate 3D simulations are mostly conducted on bridd meshes, and for rather simple, theoretic problems (Mohan al. 2020a, Chen et al., 2021b Stachenfeld al. (02I). The John Hopkins Turbulent Database (Li et al. (008)) contains nine direct numerical simulation datasets (i.e. direct resolution of Navier-Stokes equations) but with only a single scene per dataset simulated on very fine grid and low time resolution.
Therefore, extending EAGLE to 3D simulations is very difficult without sacrificing one of the funda- mental principles on which our dataset relies: (i) accuracy, guaranteed by the resolution RANS equations with demanding turbulence model on a very fine mesh (ii) irregular meshes, which are much more versatile and widespread in engineering and (iii) large scale, with nearly 1200 different scene configurations.
D.2 MODELS AND METHODS
Forecasting models in the literature mostly focus on 2D simulations (Li et al. 2019 Kashefi et al., 202 Han et al. 2022 Thuerey et al. 2020), To the best of our knowledge, there published work on large-scale scale machine learning models performing flow prediction on irregular 3D meshes. Therefore, publishing 3D version of EAGLE seems premature, not mention the difficulty of distributing such dataset and training models on reasonable setups. For grid-based simulation on
the other hand, few works leverage CNN-like structures for flow prediction (Stachenfeld al. 2021
24

---

Published as a conference paper at ICLR 2023
Mohan al. 2020b Fonda et al. 2019) computer graphics Wiewel al. (2020) Chu & Thuerey 2017). yielding the limitations discussed in the main paper.
However, we emphasize that GNN-based model are theorctically not restricted to 2D and readily manage 3D simulations. The main challenge is the memory requirements to train graph neural net- works larger point-clouds. We argue that mesh transformer is step towards handling larger meshes by reducing the number features using clustering
25


Reviews:
[{"review_id": "ldIAAU6HRJ", "reviewer": "Reviewer_DQoT", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This manuscript proposed a method to train neural networks to predict turbulent fluid dynamics. For the neural architecture, this manuscript introduced the concept of a transformer into the fluid dynamics field, which outperforms previous methods due to long-range receptive fields. To support the training of the mesh transformer, this manuscript generates a large-scale 2D aerodynamics dataset using a controlled drone. The experiments showed the superior performance of their method over several different baselines.", "summary_of_the_review": "The dataset is valuable to the community. However, I would like to see the authors justify my questions about the dataset. I am open to changing the score based on the response.", "strength_and_weaknesses": "Strength:\n- The dataset is large-scale. It covers a big fraction of the interest of the physical datasets. Previous datasets are mostly steady-state simulations.\n- The manuscript compared their method with several well-known baselines.\n\nWeaknesses:\n- In my opinion, the weaknesses are obvious. First, the dataset only contains 2D data, whose usefulness is suspicious due to the diffusion of fluid. I wonder if the authors can provide more insights on how they are useful in real engineering.\n- Several previous datasets are working in CFD, e.g., http://turbulence.pha.jhu.edu/. How is the proposed dataset compared to theirs?\n- The only variation in the dataset is the geometry. However, in reality, there are more descriptions of fluid (it is true even for air itself), e.g., the Reynolds number, the temperature, the boundary conditions, and the wind speed. I wonder how generalizable a network trained on this dataset can be.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "_jSH_iD0vQ", "reviewer": "Reviewer_gcRV", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a dataset for predicting fluid turbulences and a method using a Transformer to learn the dynamics. It adds clustering and graph pooling before the attention block so that the receptive field is not constrained in a local neighborhood. Results show that the new method outperforms the previous works.", "summary_of_the_review": "I would recommend weak reject because of the weaknesses described above. I am happy to change my score if the issues are addressed in the rebuttal.", "strength_and_weaknesses": "Strength:\n1. Although different shapes of the boundary affect how the mesh is constructed and how they are clustered, it seems that the model still generalizes well.\n2. The results are promising and supportive of the technical contribution.\n\nWeakness:\n1. Missing reference: Takahashi, Tetsuya, et al. \"Differentiable fluids with solid coupling for learning and control.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. No. 7. 2021.\n2. I wonder how this can be applied to 3D cases. Since the 2D task only there is already 270Gb data, it is probably not practical to prepare a good amount of data for the model to converge and generalize. Also, dealing with tetrahedra instead of triangles may introduce unknown challenges. Please add more insights regarding the extension to 3D in the conclusion section.\n3. How does the mesh resolution affect the training/test accuracy? If the training and test images are of different resolutions, will the model perform better or worse? It would be great to see such comparisons in the experiments.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "_3QOyaSAEZ", "reviewer": "Reviewer_jvKK", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper makes two contributions: First, it presents a large-scale 2D fluid simulation dataset called EAGLE. The dataset consists of 2D scenes with (dynamically changing) rigid boundaries and Reynold-Averaged Navier-Stokes (RANS) fluids. Second, it proposes a neural network model (mesh transformer) for solving fluid problems and compares its performance with popular neural network simulators (MeshGraphNet, etc).", "summary_of_the_review": "Currently, I am leaning toward rejection, although my score is not final yet. It is definitely a respectable submission, and I welcome different opinions from the authors and other reviewers.", "strength_and_weaknesses": "The proposal of a large-scale fluid simulation dataset is very timely. I believe having a high-quality dataset will foster relevant research in the learning community.\n\nOn the other hand, part of me feels this dataset is still premature. It is probably better than many existing fluid simulation datasets in previous papers, but I think it could have been more polished.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "SdX_70K8gJ", "reviewer": "Reviewer_912F", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a novel GNN architecture and benchmark dataset to analyze the 2D turbulent fluid dynamics caused by a single moving flow source. By utilizing the autoregressive model and self-attention modules, the proposed method achieved outperforming performance on the testing dataset.", "summary_of_the_review": "In summary, I think this paper is well written and provide some solid results with respect to the 2D dynamic fluid analysis. I think it will be better if the authors can include a short paragraph discussing the real scene application and failure case discussion.", "strength_and_weaknesses": "Strength:\n+ The paper is well written. Algorithms are well explained and easy to understand. \n+ The results are good, from both qualitative and quantitative perspective.\n+ The experiments are comprehensive and well demonstrate the robustness of the method\n\nWeakness:\n- The dataset is simulated by some existing dynamic fluid simulator, which means that the predicted results are highly impact the the quality of the simulator. \n- Since there's no real experiments has been carried out and the analysis are based on 2D, I am not sure how and how well this model can work in the 3D real scene scenarios. It will be great if the authors can briefly discuss its applications or include some real results.\n- It will be great if there's a failure case analysis or discussion. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster