AA-PINN: ATTENTION AUGMENTED PHYSICS INFORMED NEURAL NETWORKS

1 INTRODUCTION . Computational Fluid Dynamics ( CFD ) has become the core technology behind almost every fluid simulation . Fluid mechanics has been traditionally concerned with big data , thus making deep learning an obvious choice in modelling the inherent complexity of the problem . Neural Networks of late has been quite successful in understanding , predicting , optimizing , and controlling fluid flows . Neural Network has proven to improve optimization performance and reduce convergence time drastically . Neural network is also used for turbulence modelling and identifying low and high dimensional flow regimes . Deep learning algorithms are able to take into account inherent complexity of the problem thus optimizing for the performance , robustness or convergence for complex tasks . Understanding the physics behind fluid flows is a complex problem which can be solved using neural networks by feeding lots of training data . It helps in providing a general purpose framework for interpretability , generalizability and explainability of the results achieved . 2 RELATED WORK . Neural network to solve Reynolds Averaged Navier Stokes Equation was proposed by ( Ling et al. , 2016 ) . The Reynolds stress term was modelled using DNS equation by ( Wang et al. , 2017 ) . Neural network was used to model turbulent flows using Large Eddy simulation ( Zhou et al. , 2019 ) . A convolutional neural network was used to model the velocity field over a cylinder ( Jin et al. , 2018 ) . ( Wu et al. , 2020 ) proposed a similar CNN based method to model the unsteady flow in arbitrary fluid regimes . A thorough study of data driven methods using machine learning approaches for modelling the turbulence was studied by ( Duraisamy et al. , 2019 ) . ( Brunton et al. , 2020 ) also did a comprehensive study of machine learning approaches for modelling different kind of problems in fluid mechanics . ( Raissi et al. , 2017 ) proposed physics informed neural networks for solving nonlinear partial differential equations using neural network . This work was further improved in ( Raissi et al. , 2019 ) . The theoretical exact solution of the 3d Navier Stokes equation was shown by ( Ethier and Steinman , 1994 ) . Deep feedforward neural networks was used ( Lui and Wolf , 2019 ) for modelling complex flow regimes . CNN were used for making faster fluid simulation ( Tompson et al. , 2017 ) . A novel neural network was proposed for solving the function approximation and inverse PDE problems ( Meng and Karniadakis , 2020 ) . ( Khoo et al. , 2021 ) used neural network for solving parametric PDEs . ( Meng et al. , 2020 ) proposed a neural network for solving unsteady PDEs . Bayesian neural network was used to quantify the uncertainty while solving PDEs ( Yang et al. , 2021 ) . Data driven approaches for solving PDEs was proposed by ( Long et al. , 2018 ) and ( Long et al. , 2019 ) . c ( Sirignano et al. , 2020 ) used neural network for solving PDEs in the context of large-eddy simulation . ( Bar and Sochen , 2019 ) was one of the first works to use unsupervised learning for solving PDEs . ( Thuerey et al. , 2020 ) proposed a deep learning approach for solving Reynolds-averaged NavierStokes equation around airfoils . Another approach for solving PDEs using deep learning was used ( Miyanawala and Jaiman , 2017 ) in the context of unsteady wake flow dynamics . A comprehensive study of deep learning approaches for modelling and solving fluid mechanics problem was done by ( Kutz , 2017 ) . ( Ranade et al. , 2021 ) proposed a deep learning based solver for Navier–Stokes equations using finite volume discretization . Neural networks was used for solving incompressible Navier-Stokes equations ( Jin et al. , 2021 ) . A method for predicting turbulent flows using deep learning was proposed by ( Wang et al. , 2020 ) . Our main contributions can be summarized as : • A novel network architecture combining channel and spatial attention mechanism is used for modelling the inherent complexity in fluid flow problems . • We train and test our network using a more robust loss function for solve PDEs behind incompressible Navier Stokes and Burgers Equation . • Our network achieves better results than previous PINNs using commonly used evaluation metrics while still running at good enough speed . 3 BACKGROUND . 3.1 NAVIER STOKES EQUATION . The incompressible transient two dimensional Navier-Stokes equations for mass and momentum conservation are written as defined in the below set of equations : ∇ · u = 0 ( 1 ) ux ∂ux ∂x + uy ∂ux ∂y = −1 ρ ∂p ∂x + ν∇2ux + gx ( 2 ) ux ∂uy ∂x + uy ∂uy ∂y = −1 ρ ∂p ∂y + ν∇2uy + gy ( 3 ) in which u is the velocity field ( with x and y components for 2 dimensional flows ) . Here g represents the gravitational acceleration and µ the dynamic viscosity of the fluid . 3.2 MOMENTUM EQUATIONS . When the difference operators are expanded using uniform grid spacing h and time step k results in : ui , j − kν h2 ( ui−1 , j + ui , j−1 − 4ui , j +ui , j+1 + ui+1 , j ) + k h ( ūni+1 , j ūi+1 , j − ūni , j ūi , j ) + k h ( v̄ni , j ũi , j − v̄ni , j−1ũi , j−1 ) = uni , j − k h ( pi+1 , j − pi , j ) ( 4 ) Where variables without superscripts denote advanced time level results to be computed . Using the formulas of the averages and collecting the terms results in equation below : −A1ui−1 , j −A2ui , j−1 +A3ui , j −A4ui , j+1 −A5ui+1 , j = bi , j − k h ( pi+1 , j − pi , j ) ( 5 ) The various coefficients in the above equation are given using the set of equations as follows : A1 = k h ( ν h + 1 2 ūni , j ) ( 6 ) A2 = k h ( ν h + 1 2 v̄ni , j−1 ) ( 7 ) A3 = 1 + 4 kν h2 + k 2h ( ūni+1 , j − ūni , j + v̄ni , j − v̄ni , j−1 ) ( 8 ) A4 = k h ( ν h − 1 2 v̄ni , j ) ( 9 ) A5 = k h ( ν h − 1 2 ūni+1 , j ) ( 10 ) It is to be noted that in the continuous equations , we ignored effects of correction quantities in advective and diffusive terms . The u-component velocity correction can be written in the form as defined in Equation below : u′i , j = k A3h ( p′i , j − p′i+1 , j ) ( 11 ) We now present analogous results for the y-momentum equation . The v-component velocity correction can be written in the form as defined in Equation below : v′i , j = k B3h ( p′i , j − p′i , j+1 ) ( 12 ) 3.3 PRESSURE POISSON EQUATION . By substituting the velocity corrections into the discrete continuity equation for grid cell ( i , j ) results in : ui , j − ui−1 , j hx + vi , j − vi , j−1 hy = 0 ( 13 ) After substituting the decomposed velocity components , the above equation can be written as : ( u∗ + u′ ) i , j − ( u∗ + u′ ) i−1 , j hx + ( v∗ + v′ ) i , j − ( v∗ + v′ ) i , j−1 hy = 0 ( 14 ) For simplicitiy , we set hx = hy = h , and rewrite this as : − 1 A3 , i−1 , j p′i−1 , j − 1 B3 , i , j−1 p′i , j−1 + ( 1 A3 , i , j + 1 A3 , i−1 , j + 1 B3 , i , j + 1 B3 , i , j−1 ) p′i , j − 1 B3 , i , j p′i , j+1 − 1 A3 , i , j p′i+1 , j = − h2 k D∗i , j ( 15 ) It can alternatively written in a more compact form similar to that used for the momentum equations : C1p ′ i−1 , j + C2p ′ i , j−1 + C3p ′ i , j + C4p ′ i , j+1 + C5p ′ i+1 , j = d ∗ i , j ( 16 ) The various coefficients in the above equation is defined as follows : C1 ≡ 1A3 , i−1 , j , C2 ≡ 1 B3 , i , j−1 , C3 ≡ − ( 1 A3 , i , j + 1A3 , i−1 , j + 1 B3 , i , j + 1B3 , i , j−1 ) C4 ≡ 1B3 , i , j , C5 ≡ 1 A3 , i , j , d∗i , j ≡ h 2 k D ∗ i , j ( 17 ) 3.4 BURGER ’ S EQUATION . In one space dimension , the Burger ’ s equation along with Dirichlet boundary conditions is defined using the below set of equations : ut + uux − ( 0.01/π ) uxx = 0 , x ∈ [ −1 , 1 ] , t ∈ [ 0 , 1 ] u ( 0 , x ) = − sin ( πx ) u ( t , −1 ) = u ( t , 1 ) = 0 ( 18 ) Here , tiu , x i u , ui Nu i=1 denotes the initial and boundary training data on u ( t , x ) and t i f , x i f Nf i=1 denotes the collocations points for f ( t , x ) . The loss MSEu corresponds to the initial and boundary data while MSEf enforces the structure used by equation at a finite set of collocation points . 4 METHOD . 4.1 SPATIAL ATTENTION MODULE . The spatial attention module is used for capturing the spatial dependencies of the feature maps . The spatial attention ( SA ) module used in our network is defined below : fSA ( x ) = fsigmoid ( W2 ( fReLU ( W1 ( x ) ) ) ) ( 19 ) where W1 and W2 denotes the first and second 1× 1 convolution layer respectively , x denotes the input data , fSigmoid denotes the sigmoid function , fReLU denotes the ReLu activation function . The spatial attention module used in this work is shown in Figure 1 : 4.2 CHANNEL ATTENTION MODULE . The channel attention module is used for extracting high level multi-scale semantic information . The channel attention ( CA ) module used in our network is defined below : fCA ( x ) = fsigmoid ( W2 ( fReLU ( W1f 1 AvgPool ( x ) ) ) ) ( 20 ) where W1 and W2 denotes the first and second 1 × 1 convolution layer , x denotes the input data . f1AvgPool denotes the global average pooling function , fSigmoid denotes the Sigmoid function , fReLU denotes ReLU activation function . The channel attention module used in this work is shown in Figure 2 : 4.3 NETWORK ARCHITECTURE . We use deep convolutional neural network in this work . The input is the spatial and temporal coordinates of the points in the fluid flow domain . This information is propagated to three Residual blocks sequentially . In between the blocks , channel attention module is used to weight the usefulness of important features and spatial attention module is used for modelling the inter-spatial relationship of features . Fusion operator is used to merge the individual features . The output is the spatio-temporal pressure and velocity fields predicted . The complete network architecture used in this work is shown in Figure 3 :

Summary: This manuscript introduces the attention module into the framework of physics-informed neural networks. The contributions are: (1) proposing a network architecture that marries the popular attention mechanism to physics-informed neural networks. (2) separating the large attention module by operating on the channel and spatial dimensions individually. The authors empirically studied the efficacy of this architecture on PDEs including Navier-Stokes fluid simulation and Burgers equation.