Year: 2021

Abstract:
Physics Informed Neural Networks has been quite successful in modelling the complex nature of fluid flow. Computational Fluid Dynamics using parallel processing algorithms on GPUs have considerably reduced the time to solve the Navier Stokes Equations. CFD based approaches uses approximates to make the modelling easy but it comes at the cost of decrease in accuracy. In this paper, we propose an attention based network architecture named AA-PINN to model PDEs behind fluid flow. We use a combination of channel and spatial attention module. We propose a novel loss function which is more robust in handling the initial as well as boundary conditions imposed. Using evaluation metrics like RMSE, divergence and thermal kinetic energy, our network outperforms previous PINNs for modelling Navier Stokes and Burgers Equation.

heading: 1 INTRODUCTION
text: Computational Fluid Dynamics (CFD) has become the core technology behind almost every fluid simulation. Fluid mechanics has been traditionally concerned with big data, thus making deep learning an obvious choice in modelling the inherent complexity of the problem. Neural Networks of late has been quite successful in understanding, predicting, optimizing, and controlling fluid flows. Neural Network has proven to improve optimization performance and reduce convergence time drastically. Neural network is also used for turbulence modelling and identifying low and high dimensional flow regimes. Deep learning algorithms are able to take into account inherent complexity of the problem thus optimizing for the performance, robustness or convergence for complex tasks. Understanding the physics behind fluid flows is a complex problem which can be solved using neural networks by feeding lots of training data. It helps in providing a general purpose framework for interpretability, generalizability and explainability of the results achieved.

heading: 2 RELATED WORK
text: Neural network to solve Reynolds Averaged Navier Stokes Equation was proposed by (Ling et al., 2016). The Reynolds stress term was modelled using DNS equation by (Wang et al., 2017). Neural network was used to model turbulent flows using Large Eddy simulation (Zhou et al., 2019). A convolutional neural network was used to model the velocity field over a cylinder (Jin et al., 2018). (Wu et al., 2020) proposed a similar CNN based method to model the unsteady flow in arbitrary fluid regimes. A thorough study of data driven methods using machine learning approaches for modelling the turbulence was studied by (Duraisamy et al., 2019). (Brunton et al., 2020) also did a comprehensive study of machine learning approaches for modelling different kind of problems in fluid mechanics. (Raissi et al., 2017) proposed physics informed neural networks for solving nonlinear partial differential equations using neural network.
This work was further improved in (Raissi et al., 2019). The theoretical exact solution of the 3d Navier Stokes equation was shown by (Ethier and Steinman, 1994). Deep feedforward neural networks was used (Lui and Wolf, 2019) for modelling complex flow regimes. CNN were used for making faster fluid simulation (Tompson et al., 2017). A novel neural network was proposed for solving the function approximation and inverse PDE problems (Meng and Karniadakis, 2020). (Khoo et al., 2021) used neural network for solving parametric PDEs. (Meng et al., 2020) proposed a neural network for solving unsteady PDEs. Bayesian neural network was used to quantify the uncertainty
while solving PDEs (Yang et al., 2021). Data driven approaches for solving PDEs was proposed by (Long et al., 2018) and (Long et al., 2019). c(Sirignano et al., 2020) used neural network for solving PDEs in the context of large-eddy simulation.
(Bar and Sochen, 2019) was one of the first works to use unsupervised learning for solving PDEs. (Thuerey et al., 2020) proposed a deep learning approach for solving Reynolds-averaged NavierStokes equation around airfoils. Another approach for solving PDEs using deep learning was used (Miyanawala and Jaiman, 2017) in the context of unsteady wake flow dynamics. A comprehensive study of deep learning approaches for modelling and solving fluid mechanics problem was done by (Kutz, 2017). (Ranade et al., 2021) proposed a deep learning based solver for Navier–Stokes equations using finite volume discretization. Neural networks was used for solving incompressible Navier-Stokes equations (Jin et al., 2021). A method for predicting turbulent flows using deep learning was proposed by (Wang et al., 2020).
Our main contributions can be summarized as:
• A novel network architecture combining channel and spatial attention mechanism is used for modelling the inherent complexity in fluid flow problems.
• We train and test our network using a more robust loss function for solve PDEs behind incompressible Navier Stokes and Burgers Equation.
• Our network achieves better results than previous PINNs using commonly used evaluation metrics while still running at good enough speed.

heading: 3 BACKGROUND

heading: 3.1 NAVIER STOKES EQUATION
text: The incompressible transient two dimensional Navier-Stokes equations for mass and momentum conservation are written as defined in the below set of equations:
∇ · u = 0 (1)
ux ∂ux ∂x + uy ∂ux ∂y = −1 ρ ∂p ∂x + ν∇2ux + gx (2)
ux ∂uy ∂x + uy ∂uy ∂y = −1 ρ ∂p ∂y + ν∇2uy + gy (3)
in which u is the velocity field (with x and y components for 2 dimensional flows). Here g represents the gravitational acceleration and µ the dynamic viscosity of the fluid.

heading: 3.2 MOMENTUM EQUATIONS
text: When the difference operators are expanded using uniform grid spacing h and time step k results in:
ui,j − kν
h2 (ui−1,j + ui,j−1 − 4ui,j +ui,j+1 + ui+1,j) +
k h
( ūni+1,j ūi+1,j − ūni,j ūi,j ) + k
h
( v̄ni,j ũi,j − v̄ni,j−1ũi,j−1 ) = uni,j − k
h (pi+1,j − pi,j)
(4)
Where variables without superscripts denote advanced time level results to be computed. Using the formulas of the averages and collecting the terms results in equation below:
−A1ui−1,j −A2ui,j−1 +A3ui,j −A4ui,j+1 −A5ui+1,j = bi,j − k
h (pi+1,j − pi,j) (5)
The various coefficients in the above equation are given using the set of equations as follows:
A1 = k
h
( ν
h +
1 2 ūni,j
) (6)
A2 = k
h
( ν
h +
1 2 v̄ni,j−1
) (7)
A3 = 1 + 4 kν
h2 +
k
2h
( ūni+1,j − ūni,j + v̄ni,j − v̄ni,j−1 ) (8)
A4 = k
h
( ν
h − 1 2 v̄ni,j
) (9)
A5 = k
h
( ν
h − 1 2 ūni+1,j
) (10)
It is to be noted that in the continuous equations, we ignored effects of correction quantities in advective and diffusive terms. The u-component velocity correction can be written in the form as defined in Equation below:
u′i,j = k
A3h
( p′i,j − p′i+1,j ) (11)
We now present analogous results for the y-momentum equation. The v-component velocity correction can be written in the form as defined in Equation below:
v′i,j = k
B3h
( p′i,j − p′i,j+1 ) (12)

heading: 3.3 PRESSURE POISSON EQUATION
text: By substituting the velocity corrections into the discrete continuity equation for grid cell (i, j) results in:
ui,j − ui−1,j hx + vi,j − vi,j−1 hy = 0 (13)
After substituting the decomposed velocity components, the above equation can be written as:
(u∗ + u′)i,j − (u∗ + u′)i−1,j hx + (v∗ + v′)i,j − (v∗ + v′)i,j−1 hy = 0 (14)
For simplicitiy, we set hx = hy = h, and rewrite this as:
− 1 A3,i−1,j p′i−1,j − 1 B3,i,j−1 p′i,j−1 +
( 1
A3,i,j +
1
A3,i−1,j +
1
B3,i,j +
1
B3,i,j−1
) p′i,j
− 1 B3,i,j p′i,j+1 − 1 A3,i,j p′i+1,j = −
h2
k D∗i,j
(15)
It can alternatively written in a more compact form similar to that used for the momentum equations:
C1p ′ i−1,j + C2p ′ i,j−1 + C3p ′ i,j + C4p ′ i,j+1 + C5p ′ i+1,j = d ∗ i,j (16)
The various coefficients in the above equation is defined as follows:
C1 ≡ 1A3,i−1,j , C2 ≡ 1 B3,i,j−1 , C3 ≡ −
( 1
A3,i,j + 1A3,i−1,j + 1 B3,i,j + 1B3,i,j−1 ) C4 ≡ 1B3,i,j , C5 ≡ 1 A3,i,j , d∗i,j ≡ h 2 k D ∗ i,j
(17)

heading: 3.4 BURGER’S EQUATION
text: In one space dimension, the Burger’s equation along with Dirichlet boundary conditions is defined using the below set of equations:
ut + uux − (0.01/π)uxx = 0, x ∈ [−1, 1], t ∈ [0, 1] u(0, x) = − sin(πx) u(t,−1) = u(t, 1) = 0
(18)
Here, tiu, x i u, ui Nu i=1 denotes the initial and boundary training data on u(t, x) and t i f , x i f Nf i=1 denotes the collocations points for f(t, x). The loss MSEu corresponds to the initial and boundary data while MSEf enforces the structure used by equation at a finite set of collocation points.

heading: 4 METHOD

heading: 4.1 SPATIAL ATTENTION MODULE
text: The spatial attention module is used for capturing the spatial dependencies of the feature maps. The spatial attention (SA) module used in our network is defined below:
fSA(x) = fsigmoid (W2 (fReLU (W1(x)))) (19)
where W1 and W2 denotes the first and second 1× 1 convolution layer respectively, x denotes the input data, fSigmoid denotes the sigmoid function, fReLU denotes the ReLu activation function. The spatial attention module used in this work is shown in Figure 1:

heading: 4.2 CHANNEL ATTENTION MODULE
text: The channel attention module is used for extracting high level multi-scale semantic information. The channel attention (CA) module used in our network is defined below:
fCA(x) = fsigmoid(W2(fReLU (W1f 1 AvgPool(x)))) (20)
where W1 and W2 denotes the first and second 1 × 1 convolution layer, x denotes the input data. f1AvgPool denotes the global average pooling function, fSigmoid denotes the Sigmoid function, fReLU denotes ReLU activation function.
The channel attention module used in this work is shown in Figure 2:

heading: 4.3 NETWORK ARCHITECTURE
text: We use deep convolutional neural network in this work. The input is the spatial and temporal coordinates of the points in the fluid flow domain. This information is propagated to three Residual blocks sequentially. In between the blocks, channel attention module is used to weight the usefulness of important features and spatial attention module is used for modelling the inter-spatial relationship of features. Fusion operator is used to merge the individual features. The output is the spatio-temporal pressure and velocity fields predicted. The complete network architecture used in this work is shown in Figure 3:

heading: 4.4 LOSS FUNCTIONS
text: The MSE loss function is used for both the X and Y components of momentum equation which is defined as:
MSEu = 1
Nu Nu∑ i=1 ∣∣u (tiu, xiu)− ui∣∣2 (21) MSEf = 1
Nf Nf∑ i=1 ∣∣f (tif , xif)∣∣2 (22) The shared parameters between the neural networks u(t, x) and f(t, x) can be learned by minimizing the mean squared error loss as defined using in the equation below:
MSEt = αMSEu + βMSEf (23)
The weighting coefficients α and β are used to balance different terms of the loss function and accelerate convergence in the training process. The individual loss function terms Le, Lb and Li represent loss function components corresponding to the residual of the Navier-Stokes equations, the boundary conditions, and the initial conditions, respectively. The loss function is defined using the set of equations below:
Le = 1
Ne 4∑ i=1 Ne∑ n=1 |enV Pi| 2 (24)
Lb = 1
Nb Nb∑ n=1 |un − unb | 2 (25)
Li = 1
Ni Ni∑ n=1 |un − uni | 2 (26)
Where Nb, Ni and Ne denote the number of training data for different terms. The above 3 terms can be combined to give:
Lt = γLe + δLb + ρLi (27)
The weighting coefficients γ, δ and ρ are used to balance different terms of the loss function and accelerate convergence in the training process. The complete loss function for training the parameters of our network is defined as follows:
Lfinal = MSEt + Lt (28)

heading: 4.5 OPTIMIZATION DETAILS
text: For a general gradient descent algorithm, the iterative formulation of the parameters of our network can be expressed as:
θ(k+1) = θ(k) − ηγ∇θLe − ηδ∇θLb − ηρ∇θLi − ηα∇θMSEu − ηβ∇θMSEf (29)
where θ denotes the parameters of the neural network, namely the weights of all the layers, k is the iteration step, and η is the learning rate.

heading: 4.6 EVALUATION METRICS
text: Root Mean Square Error (RMSE) is the most popularly used metric for quantifying the prediction performance. The downside of using it is that it only measures indivdual pixel differences. There is a need to check whether the predictions are physically meaningful and preserve desired physical quantities, such as Turbulence Kinetic Energy and Divergence. In this work, we use following metrics for evaluation.
1. Root Mean Square Error: We calculate the RMSE of all predicted values from the ground truth for each pixel.
2. Divergence: We use the average of absolute divergence over all pixels at each prediction step as an evaluation metric.
3. Turbulence Kinetic Energy: The turbulence kinetic energy is characterised by measured root mean square velocity fluctuations as defined by:
( (u′) 2 + (v′) 2 ) /2, (u′) 2 = 1
T T∑ t=0 (u(t)− ū)2 (30)
where t is the time step. We calculate the turbulence kinetic energy for each predicted sample of 100 velocity fields.

heading: 4.7 IMPLEMENTATION DETAILS
text: An adaptive optimization algorithm, Adam (Kingma and Ba, 2014), is used to optimize the loss function. The parameters of the neural networks are randomly initialized using the Xavier intitalization scheme. We simulate turbulent channel flow at Reτ = 9.99× 102 using our network. The time step value of 0.005 is used for evaluating the residuals our network. We feed the training data using mini-batches to train our network in this study. There are three parts in the input data corresponding to the initial conditions, the boundary conditions and the residuals of equations respectively. We place 100,000 points inside the domain and 25,000 points on the boundary sampled at each time step, and 150,000 points at the initial time step to determine the loss function. The total number of iterations in one training epoch used is 100. The hyper-parameter values are α = 100, β = 100.

heading: 5 RESULTS
text: The performance comparison of our network with previous state of the art is shown in Table 1:
The exact and learned dynamics solution for the Burgers equation using our network is shown in Figure 4:
The exact and learned dynamics solution for the Navier Stokes equation using our network is shown in Figure 5:
The actual and and the predicted dynamics of the velocity components u and v using our network at different timeframes is shown in Figure 6:

heading: 5.1 ABLATION STUDIES
text: A study of with and without using channel and spatial attention module on the performance is shown in Table 2:

heading: 6 CONCLUSIONS
text: In this paper, we present a attention based physics informed neural network named AA-PINN to simulate incompressible Navier Stokes and Burgers Equations. We formulate our network using Pressure-Velocity coupling. The spatial and temporal co-ordinates of the domain are input while instantaneous pressure and velocity fields are output. We use the initial and boundary conditions as supervised data-driven parts, while residual of the Navier-Stokes and Burgers equations as the unsupervised part in the loss function while training our network. We propose a more robust loss function to handle both the boundary conditions as well as initial conditions. We test the performance our network using RMSE, divergence and TKE as the evaluation metrics. We demonstrate our designed network is more robust while modelling the complex flow physics. In the future, we would like to study the effect of attention mechanism for solving compressible and steady Navier Stokes Equations.

heading: ACKNOWLEDGMENTS
text: We would like to thank Nvidia for providing the GPUs for this work.

heading: 7 ADDITIONAL MATERIALS

heading: 7.1 GALERKIN PROCEDURE
text: Navier-Stokes equations can also be written as using the set of equations below:
ut + ( u2 ) x + (uv)y = −px + 1
Re ∆u (31)
vt + (uv)x + ( v2 ) y = −py + 1
Re ∆v (32)
ux + vy = 0 (33)
The dependent variables expressed in terms of Fourier Series is defined using the set of equations below:
u(x, y, t) = ∞∑ k ak(t)ϕk(x, y) (34)
v(x, y, t) = ∞∑ k bk(t)ϕk(x, y) (35)
p(x, y, t) = ∞∑ k ck(t)ϕk(x, y) (36)
The exponential are used as basis functions as shown below:
ϕk(x, y) = e ik·x = ei(k1x+k2y) = eik1xeik2y (37)
Substituting the above equation in momentum Navier Stokes Equation results in:
i ∞∑ k (k1ak + k2bk)ϕk = 0 (38)
Since, this is true for all the points in the domain considered, hence it can be written as:
k1ak + k2bk = 0 ∀k (39)
On substituting these expansions into the x-momentum equation, we obtain:
∂ ∂t ∑ ` a`ϕ` + ∂ ∂x ∑ `,m a`amϕ`ϕm + ∂ ∂y ∑ `,m a`bmϕ`ϕm =
− ∂ ∂x ∑ ` c`ϕ` + 1 Re
[ ∂2
∂x2 ∑ ` a`ϕ` + ∂2 ∂y2 ∑ ` a`ϕ` ] (40) The process of commuting, summation and differentiation gives: ∑ ` ȧ`ϕ`+i ∑ `,m (`1 +m1) a`amϕ`ϕm+i ∑ `,m (`2 +m2) a`bmϕ`ϕm = −i ∑ ` `1c`ϕ`− 1 Re ∑ ` ( `21 + ` 2 2 ) a`ϕ`
(41)
Theorem 1.1: Let u0, FB and tf > 0 be given such that
u0(x) ∈ H(Ω), and FB(x, t) ∈ L2 (0, tf ;H) (42)
Then ∃a unique solution u ∈ (u1, u2)T
ui, ∂ui ∂xj ∈ L2 (Ω× (0, tf )) , i, j = 1, 2 (43)
and u is continuous from [θ, tf ] intoH . Moreover, the following energy equation holds for t ∈ [θ, tf ]:
1 2 d dt ‖u(t)‖2L2 + ν‖u(t)‖2H1 = 〈FB(t),u(t)〉 (44)
Theorem 1.2: Suppose u0, FB are given and are such that:
u0(x) ∈ V (Ω), and FB(x, t) ∈ L2 (0, tf ;H) (45)
for tf > 0. Then ∃ a unique (strong) solution u = (u1, u2)T with
ui, ∂ui ∂t , ∂ui ∂xj , ∂2ui ∂xj∂xk ∈ L2 (Ω× (0, tf )) , i, j, k = 1, 2 (46)
and u is continuous from [0, tf ] into V .

heading: 7.2 SIMPLE ALGORITHM
text: SIMPLE algorithm is a finite-volume based scheme, which uses a staggered grid which is derived using terms of discrete equations rather than from discretization of a continuous function.
The flow variables can be expressed at any given time step as defined below:
u = u∗ + u′, v = v∗ + v′ and p = p∗ + p′ (47)
where ∗ denotes an initial estimate, and ′ represents a correction term. This can be substituted in the Navier Stokes Equation to produce the equation below:
(u∗ + u′)t + ( (u∗ + u′) 2 ) x + ((v∗ + v′) (u∗ + u′))y = − (p ∗ + p′)x + ν∆ (u ∗ + u′) (48)
We ignore effects of the corrections on advective and diffusive terms thus giving the set of equations below in terms of both x and y momentum:
u∗t + (u nu∗)x + (v nu∗)y = −p n x + ν∆u ∗ (49)
u′t = −p′x (50)
v∗t + (u nv∗)x + (v nv∗)y = −p n y + ν∆v ∗ (51)
v′t = −p′y (52)
The differentiation with respect to time of Pressure Poisson Equation is defined in the equation below:
D′ = u′x + v ′ y (53)
∂D′
∂t =
∂ ∂t
( u′x + v ′ y ) = u′tx + v ′ ty (54)
Using a simple forward-difference approximation, we have:
∂D′ ∂t = D′n+1 −D′n ∆t (55)
Using the set of Navier-Stokes equations, this can be written as:
∂D′
∂t = −p′xx − p′yy (56)
Next, a PPE for the correction pressure can be obtained as:
∆p′ = D∗
∆t (57)
The boundary conditions to be employed with this PPE are defined using equations below:
p′ = 0 (58)
∂p′ ∂n = 0 (59)
Once p ′
has been calculated, we can then update the pressure and velocity components using the set of equations below:
pn+1 = pn + p′ (60)
un+1 = u∗ − ∂p ′
∂x ∆t ≡ u∗ + u′ (61)
vn+1 = v∗ − ∂p ′
∂y ∆t ≡ v∗ + v′ (62)

Review summaries:
This manuscript introduces the attention module into the framework of physics-informed neural networks. The contributions are: (1) proposing a network architecture that marries the popular attention mechanism to physics-informed neural networks. (2) separating the large attention module by operating on the channel and spatial dimensions individually. The authors empirically studied the efficacy of this architecture on PDEs including Navier-Stokes fluid simulation and Burgers equation.

The authors propose a new architecture and loss function for training physics-informed neural networks (PINNs) on fluid flow problems. The idea is centered around augmenting a basic residual network with two additional attention blocks that are placed before and after residual blocks. These two new blocks aim to introduce channel and spatial attention into the model. In addition, they propose a new loss function that is tailored to solving PDEs such as Navier Stokes and Burgers Equation.