AA-PINN: ATTENTION AUGMENTED PHYSICS INFORMED NEURAL NETWORKS
Authors: Abhinav Sagar
Venue: ICLR 2022 Submitted (2022)

Abstract: Physics Informed Neural Networks has been quite successful in modelling the complex nature of fluid flow. Computational Fluid Dynamics using parallel processing
algorithms on GPUs have considerably reduced the time to solve the Navier Stokes
Equations. CFD based approaches uses approximates to make the modelling easy
but it comes at the cost of decrease in accuracy. In this paper, we propose an
attention based network architecture named AA-PINN to model PDEs behind fluid
flow. We use a combination of channel and spatial attention module. We propose a
novel loss function which is more robust in handling the initial as well as boundary
conditions imposed. Using evaluation metrics like RMSE, divergence and thermal
kinetic energy, our network outperforms previous PINNs for modelling Navier
Stokes and Burgers Equation.

Under review as a conference paper at ICLR 2022
AA-PINN: ATTENTION AUGMENTED PHysICS IN- FORMED NEURAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
Physics Informed Neural Networks has been quite successful in modelling the com- plex nature of fluid flow. Computational Fluid Dynamics using parallel processing algorithms on GPUs have considerably reduced the time to solve the Navier Stokes Equations. CFD based approaches uses approximates to make the modelling easy but it comes at the cost of decrease in accuracy. In this paper, we propose an attention based network architecture named AA-PINN to model PDEs behind fluid flow. We use a combination of channel and spatial attention module. We propose a novel loss function which is more robust in handling the initial as well as boundary conditions imposed. Using evaluation metrics like RMSE, divergence and thermal kinetic energy, our network outperforms previous PINNs for modelling Navier Stokes and Burgers Equation.
INTRODUCTION
Computational Fluid Dynamics (CFD) has become the core technology behind almost every fluid simulation. Fluid mechanics has been traditionally concerned with big data, thus making deep learning an obvious choice modelling the inherent complexity of the problem. Neural Networks of late has been quite successful in understanding, predicting, optimizing, and controlling fluid flows. Neural Network has proven to improve optimization performance and reduce convergence time drastically. Neural network is also used for turbulence modelling and identifying low and high dimensional flow regimes. Deep learning algorithms are able to take into account inherent complexity of the problem thus optimizing for the performance, robustness convergence for complex tasks. Understanding the physics behind fluid flows is a complex problem which can be solved using neural networks by feeding lots of training data. It helps in providing general purpose framework for interpretability, generalizability and explainability of the results achieved.
2 RELATED WORK
Neural network to solve Reynolds Averaged Navier Stokes Equation was proposed by (Ling et al., 2016). The Reynolds stress term was modelled using DNS equation by (Wang al., 2017). Neural network was used to model turbulent flows using Large Eddy simulation (Zhou et al., 2019). A convolutional neural network was used to model the velocity field over a cylinder (Jin et al., 2018). (Wu et al., 2020) proposed a similar CNN based method to model the unsteady flow in arbitrary fluid regimes. A thorough study of data driven methods using machine learning approaches for modelling the turbulence was studied by (Duraisamy et al., 2019). (Brunton et al., 2020) also did a comprehensive study of machine learning approaches for modelling different kind of problems in fluid mechanics. (Raissi et al., 2017) proposed physics informed neural networks for solving nonlinea partial differential equations using neural network.
This work was further improved in (Raissi et al., 2019). The theoretical exact solution of the 3d Navier Stokes equation was shown by (Ethier and Steinman, 1994). Deep feedforward neural networks was used (Lui and Wolf, 2019) for modelling complex flow regimes. CNN were used for making faster fluid simulation (Tompson et al., 2017). A novel neural network was proposed for solving the function approximation and inverse PDE problems (Meng and Karniadakis, 2020). (Khoo et al., 2021) used neural network for solving parametric PDEs. (Meng et al., 2020) proposed a neural network for solving unsteady PDEs. Bayesian neural network was used quantify the uncertainty

---

Under review as a conference paper at ICLR 2022
while solving PDEs (Yang al., 2021). Data driven approaches for solving PDEs proposed by (Long et al., 2018) and (Long et al., 2019). c(Sirignano et al., 2020) used neural network for solving PDEs in the context of large-eddy simulation.
(Bar and Sochen, 2019) was one of the first works to use unsupervised learning for solving PDEs. (Thuerey al., 2020) proposed deep learning approach for solving Reynolds-averager Navier- Stokes equation around airfoils. Another approach for solving PDEs using deep learning was used (Miyanawala and Jaiman, 2017) in the context of unsteady wake flow dynamics. A comprehensive study of deep learning approaches for modelling and solving fluid mechanics problem was done by (Kutz, 2017). (Ranade et al., 2021) proposed deep learning based solver for Navier-Stokes equations using finite volume discretization. Neural networks was used for solving incompressible Davier-Stokes equations (Jin et al., 2021). A method for predicting turbulent flows using deep learning was proposed by (Wang et al., 2020).
Our main contributions can be summarized as:
A novel network architecture combining channel and spatial attention mechanism is used for modelling the inherent complexity in fluid flow problems.
We train and test our network using more robust loss function for solve PDFs behind incompressible Navier Stokes and Burgers Equation.
Our network achieves better results than previous PINNs using commonly used evaluation metrics while still running at good enough speed.
3
BACKGROUND
3.1
NAVIER STOKES EQUATION
The incompressible transient two dimensional Navier-Stokes equations for mass and momentum conservation written defined in the below set of equations:
u 0
Our
Our
1 8pp pxx
Us
+ My
2V22z + Jr
Ix
Oy
Juy
aug
1 10p p By
Ug dx
by
vV2u,
9g
oy
in which u is the velocity field (with æ and y components for 2 dimensional flows). Here g represents the gravitational acceleration and μ the dynamic viscosity of the fluid.
3.2 MOMENTUM EQUATIONS
When the difference operators are expanded using uniform grid spacing h and time step & results in:
kv
4uy +ugg-1 + u4+1,) + I h (uht1) U1tis - u.s)
Uij (u--1 + U1 h
(4)
k (piti. - pig)
h (oifūis - ( ufis
Where variables without superscripts denote advanced time level results to be computed. Using the formulas of the averages and collecting the terms results in equation below:
t.
- -A14-115 - A2 + Agus, Aqui,ft1 Asu4t15 = bij (pitis - Pi.)) (5)
2

---

Under review as a conference paper at ICLR 2022
The various coefficients in the above equation are given using the set of equations follows:
k 1 h
A1
(6)
2
A
+
h
kv
h
A3 1 +
(1 (1411
+
h
2h
k
1
A
Di.j
(9)
h
2
k
1/
Ag
h
h
2
W
+ 15
(10)
It is to be noted that in the continuous equations, we ignored effects of correction quantities in advective and diffusive terms. The u-component velocity correction can be written in the form as defined in Equation below:
h: Agh
d/
(P',s Pl+4.5)
(11)
We now present analogous results for the y-momentum equation. The v-component velocity correction be written in the form defined in Equation below:
k
1/ = Bah (PLs - phasta)
(12)
3.3 PRESSURE POISSON EQUATION
By substituting the velocity corrections into the discrete continuity equation for grid cell (i,j) results in:
Wi.j
04 1.j
Ui.j
Ui.j O
(13)
h2
hy
After substituting the decomposed velocity components, the above equation can written as:
(u" + u')as - (u" + u (0" + +2),, (v" + t')
(14)
ho
by
For simplicitiy, we set hn = by h. and rewrite this as:
1
1
1
1
Ag,i-13 P
P
+
+
+
P/.s
Ba
Aaii
A3
Baiii
Ba
(15)
1
112
Phot1
Di,
Baa
Ag.i
It can alternatively written in compact form similar to that used for the momentum equations:
Cirl-19 + Carku-1 Caph. + Capha41 Csph414 di,s
(16)
3

---

Under review as a conference paper at ICLR 2022
The various coefficients in the above equation is defined as follows:
1
A1,--13'
C2
B3
C3
As,1s + A + Ba,ss +
Ba
(17)
C4 Bana"
C5 As.
dig
h²a k
Dij
3.4 BURGER'S EQUATION
In space dimension, the Burger's s equation along with Dirichlet boundary conditions is defined using the below set of equations:
Ut + ut, (0.01/m) 1uzz =0, xE1-1,1], Le [0.
u 00, x) sin TT)
(18)
utt -1) u(t,
Here, tu, 2/, 14 N i=1 denotes the initial and boundary training data on u(t, x) and fyy 2² Ny i=1 denotes the collocations points for 2() The loss MSE,, corresponds to the initial and boundary data while F, enforces the structure used by equation a finite set of collocation points.
4
METHOD
4.1
SPATIAL ATTENTION MODULE
The spatial attention module is used for capturing the spatial dependencies of the feature maps. The spatial attention (SA) module used in network is defined below:
fSA(F) Ssigmoid (W2 (SEeL (x))))
(19)
where Wi and W2 denotes the first and second 1 x convolution layer respectively, æ denotes the input data, fsigmoid denotes the sigmoid function, fREL denotes the ReLu activation function.
The spatial attention module used in this work is shown in Figure
Spatial Attention Module
(HWW)X(WW))
Output leature: map
fealure map
Reshape
Reshape
Figure 1: Details of our spatial attention module
4.2 CHANNEL ATTENTION MODULE
The channel attention module is used for extracting high level multi-scale semantic information. The channel attention (CA) module used in network is defined below:
fCA(z) = fiigmoria (W (Wi. xx)) (z))))
(20)
where W, and W. denotes the first and second 1 x 1 convolution layer, x denotes the input data. f Pool denotes the global average pooling function, f sigmoid denotes the Sigmoid function, FREU denotes ReLU activation function.
The channel attention module used in this work is shown in Figure 2:
4

---

Under review as a conference paper at ICLR 2022
Reshapp
Reshaape
icature mnp
fsature
map
CXC
Channel Attention Module
Figure 2: Details of our channel attention module
4.3 NETWORK ARCHITECTURE
We use deep convolutional neural network in this work. The input is the spatial and temporal co- ordinates of the points in the fluid flow domain. This information is propagated to three Residual blocks sequentially. between the blocks, channel attention module used to weight the usefulness of important features and spatial attention module is used for modelling the inter-spatial relationship of features. Fusion operator is used to merge the individual features. The output is the spatio-temporal pressure and velocity fields predicted. The complete network architecture used in this work is shown in Figure 3:
CA
SA
Output pl upx.). v(x.)
Input domain(x.0
Residual Block
Residual Block
Figure 3: Illustration of our network architecture. A residual block denotes convolution, max pooling, relu activation function and batch normalization layer sequentially; CA and SA denotes channel and spatial attention module respectively; denotes fusion operator.
4.4 Loss FUNCTIONS
The MSE loss function used for both the and Y components of momentum equation which is defined as:
No Σ
MSE. = Nu
u (t)
r0¹
(21)
i=1
N E1s MSE, N
(22)
The shared parameters between the neural networks u(t, x) and f(t, x) can be learned by minimizing the mean squared error loss as defined using in the equation below:
MSE, aMSE, + SMSEJ
(23)
The weighting coefficients and B are used to balance different terms of the loss function and accelerate convergence in the training process. The individual loss function terms Le. L, and L; represent loss function components corresponding the residual of the Navier-Stokes equations, the boundary conditions, and the initial conditions, respectively. The loss function is defined using the set of equations below:
5

---

Under review as a conference paper at ICLR 2022
1
N₂
Le
evpl
(24)
Ne
im
n=1
No
Lb
u - 73
(25)
Nb nm
N:
L
Σ
1
u -
(26)
N
Where No. N and Ne denote the number of training data for different terms. The above terms be combined to give:
L = T + 8Lg + pL:
(27)
The weighting coefficients 7, 5 and p are used to balance different terms of the loss function and accelerate convergence in the training process. The complete loss function for training the parameters of our network is defined as follows:
Linal MSEE + L1
(28)
4.5 OPTIMIZATION DETAILS
For a general gradient descent algorithm, the iterative formulation the parameters of our network be expressed as:
V, - - ((++1) = (k) 17 - - Vo -
(29)
where 0 denotes the parameters of the neural network, namely the weights of all the layers, k is the iteration step, and v is the learning rate.
4.6 EVALUATION METRICS
Root Mean Square Error (RMSE) is the most popularly used metric for quantifying the prediction performance. The downside of using that it only measures indivdual pixel differences. There is need to check whether the predictions are physically meaningful and preserve desired physical quantities, such as Turbulence Kinetic Energy and Divergence. this work, we use following metrics for evaluation.
Root Mean Square Error: We calculate the RMSE of all predicted values from the ground truth for each pixel.
We use the average of absolute divergence over all pixels at each prediction step as an evaluation metric.
Divergence:
Turbulence Kinetic Energy: The turbulence kinetic energy is characterised by measured root mean square velocity fluctuations defined by:
(w/²² +()=)((0))))) /2,
(w/2
T Σ a)²
(30)
where is the time step. We calculate the turbulence kinetic energy for each predicted sample 100 velocity fields.

---

Under review as a conference paper at ICLR 2022
4.7 IMPLEMENTATION DETAILS
An adaptive optimization algorithm, Adam (Kingma and Ba, 2014), is used to optimize the loss function. The parameters of the neural networks are randomly initialized using the Xavier intitalization scheme. We simulate turbulent channel flow at Re, = 9.99 x 10 using our network. The time step value of 0.005 used for evaluating the residuals our network. We feed the training data using mini-batches to train our network in this study. There are three parts in the input data corresponding to the initial conditions, the boundary conditions and the residuals of equations respectively. We place 100,000 points inside the domain and 25,00 25,000 points on the boundary sampled at each time step, and 150,000 points at the initial time step to determine the loss function. The total number of iterations in one training epoch used is 100. The hyper-parameter values are 100, B 100.
5 RESULTS
The performance comparison of our network with previous state of the art is shown in Table 1:
Table I: Comparion of SOTA networks using the number of parameters, the best number of input
frames, the best number of accumulated errors for back-propogation and training time for one epoch.
Models
TF-net
U-net
GAN
ResNet
ConVLSTM
SST
DHPM
Ours
number of purameters(10*)
15.9
25.0
26.1
21.2
11.8
49.9
2.12
0.53
input length
25
25
24
26
27
23
23
20
accumulated errors
4
6
5
5
4
5
5
2
time for one epoch(min)
0.39
0.57
0.73
1.68
45.6
0.95
4.591
0.72
The exact and Iearned dynamics solution for the Burgers equation using our network is shown in Figure 4:
Ficec Dymamics
Teareed Dysamics
0:75
0.00
0.00
0:25
0:55
0
0.00
0-
0.0
-0.21
-0.35
-5
-0.7
-0.0
-0.50
-0.75
-100
0.0
2.
5.3
7.5
10.0
0.0
2.5
3.01
7.5
20.0
Figure 4: A solution of the Burger's equation (left panel) is compared to the corresponding solution of the learned partial differential equation (right panel).
The exact and learned dynamics solution for the Navier Stokes equation using our network is shown Figure 5:
Dynommis
Dynamiis
Figure 5: A solution of the NavierStokes equation (left panel) is compared to the corresponding solution of the learned partial differential equation (right panel).
The actual and and the predicted dynamics of the velocity components u and using our network at different timeframes is shown in Figure 6:
7

---

Under review as a conference paper at ICLR 2022
1-99
T-L0
T-20
1220
721
130
T-50
Dynamics of
Dynamics of
Figure 6: The first row shows the images of the dynamics. The last two show the images of the predicted dynamics using our network.
5.1 ABLATION STUDIES
study with and without using channel and spatial attention module on the performance shown in Table 2:
Table 2: Ablation study using variations of spatial and channel attention modules.
Metrics
Only SA
Only CA
Both
number parameters(10*)
1.23
0.71
0.53
accumulated enrors
5
3
2
time for one epoch(min)
1.05
1.16
0.72
CONCLUSIONS
In this paper, present attention based physics informed neural network named AA-PINN to simulate incompressible Navier Stokes and Burgers Equations. We formulate our network using Pressure-Veloo Velocity coupling. The spatial and temporal co-ordinates of the domain input while instantaneous pressure and velocity fields are output. We use the initial and boundary conditions as supervised data-driven parts, while residual of the Anvier-Stoker and Burgers equations as the unsupervised part in the loss function while training our network. We propose a more robust loss function to handle both the boundary conditions as well initial conditions. We test the performance our network using RMSE, divergence and TKE as the evaluation metrics. We demonstrate our designed network is more robust while modelling the complex flow physics. In the future, we would like to study the effect of attention mechanism for solving compressible and steady Navier Stokes Equations.
ACKNOWLEDGMEN EDGMENTS
We would like to thank Nvidia for providing the GPUs for this work.
REFERENCES
L. Bar and N. Sochen. Unsupervised deep learning algorithm for pde-based forward and inverse problems. arXiv preprint arXiv: 1..900055111 2019.
A. G. Baydin, B. A. Pearlmutter, A. A. Radul, and J. M. Siskind. Automatic differentiation in machine learning: survey. Journal of machine learning research, 18, 2018.
S. L.. Brunton, B. R. Noack, and P. Koumoutsakos. Machine learning for fluid mechanics. Annual Review of Fluid Mechanics, 52:477-508, 2020.
S. Cai, S. Zhou, C. Xu, and Q. Gao. Dense motion estimation of particle images via convolutional neural network. Experiments in Fluids, 60(4):1-16, 2019.
8

---

Under review as a conference paper at ICLR 2022
K. Duraisamy, G. laccarino, and H. Xiao. Turbulence modeling in the age of data. Annual Review of Fluid Mechanics, 51:357-377,2011
C. R. Ethier and D. Steinman. Exact fully 3d navier-stokes solutions for benchmarking. International Journal for Numerical Methods in Fluids, 1965):39-3375, 1994.
X. Jin, P. Cheng, W.-L. Chen, and H. Li. Prediction model of velocity field around circular cylinder over various reynolds numbers by fusion convolutional neural networks based on pressure on the cylinder. Physics of Fluids, 30(4):047105, 2018.
X. Jin, S. Cai, Li, and Karniadakis. Nsfnets (navier-stokes flow nets): Physics-informed neural networks for the incompressible navier-stokes equations. Journal of Computational Physics, 426: 129510 2021.
Y. Khoo, J. Lu, and L. Ying. Solving parametric pde problems with artificial neural networks. European Journal of Applied Mathematics, 32(3):421-435, 2021.
D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint 2014.
J. N. Kutz. Deep learning in fluid dynamics. Journal of Fluid Mechanics, 814:1-4, 2017.
J. Ling, A. Kurzawski, and J. Templeton. Reynolds averaged turbulence modelling using deep neural networks with embedded invariance. Journal of Fluid Mechanics, 807:155-166, 2016.
Z. Long, Y. Lu, X. Ma, and B. Dong. Pde-net: on Machine Learning, pages 3208-3216 PMLR, 2018.
Learning pdes from data. In International Conference
Z. Long, Y. Lu, and B. Dong. Pde-net 2.0: Learning pdes from data with numeric-symbolic hybrid deep network. Journal of Computational Physics, 1999:00925. 2019.
H. F. Lui and W. R. Wolf. Construction reduced order models for fluid flows using deep feedforward neural networks. arXiv preprint arXiv:1903.05200 2019.
K. O. Lye, S. Mishra, and D. Ray. Deep learning observables in computational fluid dynamics. Journal of Computational Physics, 410:1093330 2020.
X. Meng and G. E. Karniadakis. A composite neural network that learns from multi-fidelity data: Application function approximation and inverse pde problems. Journal of Computational Physics, 401:109020,, 2020.
X. Meng, Z. Li, D. Zhang, and G. E. Karniadakis. Ppinn: Parareal physics-inforee neural network for time-dependent pdes. Computer Methods in Applied Mechanics and Engineering, 370:1 13250, 2020.
T. P. Miyanawala and R. K. Jaiman. An efficient deep learning technique for the navier-stokes equations: Application to unsteady wake flow dynamics. arXiv preprint arXiv: 2017.
M. Raissi. Deep hidden physics models: Deep learning of nonlinear partial differential equations. The Journal of Machine Learning Research, 19(1):932-955, 2018.
M. Raissi, P. Perdikaris, and G. E. Karniadakis. Physics informed deep learning (part i): Data-driven solutions of nonlinear partial differential equations. arXiv preprint arXiv:1711.10566 2017.
M. Raissi, P. Perdikaris, and G. E. Karniadakis. framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378:686-707, 2019.
informed neural networks: A deep learning
R. Ranade, C. Hill, and J. Pathak. Discretizationnet: A machine-learning based solver for navier- stokes equations using finite volume discretization. Computer Methods in Applied Mechanics and Engineering, 378: 13722, 2021.
A. Sagar. Aa3dnet: Attention augmented real time 3d object detection, 2021a.
A. Sagar. Dmsanet: Dual multi scale attention network. arXiv preprint 10X.::2/06.08382,

---

Under review as a conference paper at ICLR 2022
V. Sekar, Q. Jiang, C. Shu, and B. C. Khoo. Fast flow field prediction over airfoils using deep learning approach. Physics of Fluids, (1(5)057703,, 2019.
J. Sirignano, J. F. MacArt, and J. B. Freund. Dpm: A deep learning pde augmentation method with application to large-eddy simulation. Journal of Computational Physics, 423:109811, 2020.
L. Sun, H. Gao, Pan, and J.-X. Wang- Surrogate modeling for fluid flows based on physics- constrained deep learning without simulation data. Computer Methods in Applied Mechanics and Engineering, 361:1 12732, 2020.
N. Thuerey, K. Weilienow, L. Prantl, and X. Hu. Deep learning methods for reynolds-avcraged navier-stokes simulations of airfoil flows. AIAA Journal, 58(1):25-36, 2020.
J. Tompson, K. Schlachter, P. Sprechmann, and K. Perlin. Accelerating eulerian fluid simulation with convolutional networks. In International Conference on Machine Learning, pages 3424-3433. PMLR, 2017.
J.-X. Wang, J.-L. Wu, H. Xiao. Physics-informed machine learning approach for reconstructing reynolds stress modeling discrepancies based on dns data. Physical Review Fluids, 2(3):034603, 2017.
R. Wang, K. Kashinath, M. Mustafa, A. Albert, and R. Yu. Towards physics-informed deep learning for turbulent flow prediction. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1457-1466, 2020.
P. Wu, J. Sun, X. Chang, W. Zhang, R. Arcucci, Y. Guo, and C. C. Pain. Data-driven reduced order model with temporal convolutional neural network. Computer Methods Applied Mechanics and Engineering, 360:1 12766, 2020.
L. Yang. X. Meng, and G. E. Kamiadakis. B-pinns: Bayesian physics-infomme neural networks for forward and inverse pde problems with noisy data. Journal of Computational Physics, 425:109913, 2021.
Z. Zhou, G. He, S. Wang, and G. Jin. Subgrid-scale model for large-eddy simulation of isotropic turbulent flows using an artificial neural network. Computers & Fluids, 195:104319, 2019.
ADDITIONAL MATERIALS
GALERKIN PROCEDURE
Navier-Stokes equations can also be written as using the set of equations below:
1
U4 (u²)z (uvv) P Re
Au
(31)
1 Re
Ut + (uv) + (u²) = - -
Av
(32)
02 + 0y = 0
(33)
The dependent variables expressed in terms of Fourier Series defined using the set of equations below:
u(x, v.
Σ au (1)))(z))))
(34)
((x,).t) =
Σ
by (t)
(35)
b
10

---

Under review as a conference paper at ICLR 2022
p(a, v. Σ (x(0)p(()))))
(36)
The exponential are used as basis functions as shown below:
( y) = e
i = (kk _ eaky
(37)
Substituting the above equation in momentum Navier Stokes Equation results in:
(ki + kzbb) pa = 0
(38)
Since, this is true for all the points in the domain considered, hence it can be written as:
ki + k2bb 0 vk
(39)
On substituting these expansions into the x-momentum equation, we obtain:
a Σ
σt Σ appt +
Σ
aeampPiPm +
aebmpprim
dx
dy 8.
2m
(40)
82 d.
82 On² 1
Σ
Σ
Σ
cePe
+
appe +
appe
dx
Re
The process of commuting, summation and differentiation gives:
Σ Eespeté Σ (f1 m1) am0m E,m
Σ
(l2 m2) aebm -i- Eiccpp
Σ (G
E(G+
L aepe
Re
t.m
(41)
Theorem 1.1: Let un. Fn and l > 0 be given such that
uo(x)= E (2), and E (0,t
(42)
Then 3a unique solution (u1, T
Du: Ui dr,
L (Q (0,ty)), i,j 1,2
(43)
and u is continuous from [0, ty] into H. Moreover, the following energy equation holds for e [0, tyl:
2 di [u(t)] + (ll (FF(t),
(44)
Theorem 1.2: Suppose FB are given and are such that:
uo(a) V(2), and Fa(æ, 1) E 1. (0,4;; H)
(45)
forty ty 0. Then 3 unique (strong) solution (u1, uz)) with
Ou Ou: Sus Ui, 0t Ozy dr,0nr
E T. (Ω x (0,tr)), =
(46)
and is continuous from [0, 1y] into V.
11

---

Under review as a conference paper at ICLR 2022
7.2 SIMPLE ALGORITHM
SIMPLE algorithm a finite-ollume based scheme, which uses a staggered grid which is derived using terms of discrete equations rather than from discretization of continuous function.
The flow variables can be expressed at any given time step as defined below:
t == + +u', a= + and p - p +p'
(47)
where denotes initial estimate, and represents correction term. This be substituted in the Navier Stokes Equation to produce the equation below:
(u" + u.), + + ((v" + + (u² + u')), = - (p" p) + vA (u² + (48)
We ignore effects of the corrections on advective and diffusive terms thus giving the set of equations below in terms of both x and y momentum:
u + (u² + (v"u"), = =-p" + vAu"
(49)
14 = -p's
(50)
ot + (u²v"), + (2" = =Pp" + -vA="
(51)
1/ P'y
(52)
The differentiation with respect to time of Pressure Poisson Equation is defined in the equation below:
D' wly + v'y
(53)
aD'
Öt Øt (u', + v/y) uhe + r/yy
(54)
Using simple forward-difference approximation, have:
aD' D Dm
(55)
0L Al
Using the set of Navier-Stokes equations, this can be written as:
OD'
-P P'yv
(56)
Øt
Next, a PPE for the correction pressure can be obtained as:
D*
Ap's
(57)
At
The boundary conditions be employed with this PPE are defined using equations below:
p' 0
(58)
Op' On
0
(59)
12

---

Under review as a conference paper at ICLR 2022
Once p' has been calculated, we can then update the pressure and velocity components using the set of equations below:
p n+ = p" +pp
(60)
0p'
u²
At u²" u
(61)
0p'
a+1 0-
At = n² +
(62)
By
13


Reviews:
[{"review_id": "n6AXwNqIBXN", "reviewer": "Reviewer_kPHW", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I appreciate the idea of combining attention mechanisms with physics-informed neural networks, but I expect more than merely an addition. Also, there are a few weaknesses I have to point out:\n\n1. The author did not provide any explanation on the intuition behind refactorizing the channel and spatial dimensions. Is it for memory efficiency? How does it compare to non-local neural networks?\n2. Physical process usually comes with tons of hyperparameters, which is understandable because of the complexity. However, a lot of the hyperparameters are left unexplained or unexperimented. For example, how were $\\alpha$ and $\\beta$ selected? What does the overall network architecture look like? Does the number of attention modules stacked matter to the result?\n3. One key question to physical simulation with neural networks is generalizability. It will be very helpful to demonstrate how this method generalizes to unseen boundary conditions and initial fields.\n4. Attention mechanisms intrigue me by not only a superior performance on multiple tasks but also the explainability. It will be interesting to see the visualization of them so that it might shed some light on what and how the networks learn physics.\n\nIn addition to these technical unclearnesses, my main concern is the novelty of mixing multiple popular topics. I expect more solid theories or more exciting demonstrations.", "summary_of_the_paper": "This manuscript introduces the attention module into the framework of physics-informed neural networks. The contributions are: (1) proposing a network architecture that marries the popular attention mechanism to physics-informed neural networks. (2) separating the large attention module by operating on the channel and spatial dimensions individually. The authors empirically studied the efficacy of this architecture on PDEs including Navier-Stokes fluid simulation and Burgers equation.", "summary_of_the_review": "This manuscript talks about two popular topics: attention mechanisms and physics-informed neural networks. I believe something significant will happen in this particular overlap but not in a \"1+1=2\" form. I expect more explanatory theories or experiments and domain-specific modifications.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "mMT1mM164g8", "reviewer": "Reviewer_Ck1m", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is not well organized. In some sections e.g. BACKGROUND, there is a whole page describing the MOMENTUM EQUATIONS and PRESSURE POISSON EQUATION, however later in the AA-PINN experiment, these equations are never used. Also in the Results section, the authors just display the outcomes from the AA-PINN without any discussion about why AA-PINN works. The ablation study of the channel and spatial attention module on the performance looks interesting, but again there is no comment from the authors at all.\n\nI would not recommend this paper to be published in the current form.\n\nMy addition comments are as follows:\n\n1.\tOn page 4, the BURGER’S EQUATION Eq. (18) doesn’t show f(t, x) at all. Later in the loss function Eq. (22) on page 5, f(t, x) is used. This will cause confusion. \n2.\tThe definition of t_u_i is given right after Eq. (18), but it is not used in that part. Instead, it is used later in Eq. (22). This will confuse reader as well.\n3.\tThe terms of t_u_i, x_u_i, u_i, N_u, etc. should be explained respectively instead of as a whole.\n4.\tOn page 4, Figure 1 contains two operations 1) reshape with x, 2) reshape with +. What are they about, respectively?\n5.\tOn page 8, Figure 6 has three rows of plots, what are the last two rows about, respectively?\n6.\tOn page 7, Table 1 shows the performance comparison of the proposed network with previous state of the art. What equation are those networks solving? Where is the data for those neural networks, e.g., TF-net, from?\n", "summary_of_the_paper": "The manuscript entitled “AA-PINN: ATTENTION AUGMENTED PHYSICS INFORMED\nNEURAL NETWORKS” use a combination of channel and spatial attention module in addition to the normal Physics Informed Neural Networks. Using evaluation metrics like RMSE, divergence and thermal kinetic energy, AA-PINN network outperforms previous PINNs for modelling Navier Stokes and Burgers Equation.\n", "summary_of_the_review": "This paper proposed attention augmented physics informed neural networks (AA-PINN). It uses a combination of channel and spatial attention module in addition to the normal Physics Informed Neural Networks. This paper would be an interesting paper however it is poorly organized in the current version and also has a lot of flaws as I pointed out above earlier.\n\nIn summary, I would not recommend its publication in ICLR until all my comments are addressed.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "bHOEfHBfWgs", "reviewer": "Reviewer_qXxf", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n\nThe presentation is very poor, experimental details are missing, and it is not clear \nagainst which models the authors are benchmarking. They report increased performance\naccording to a metric they don't even define. \n\n- Missing references in the introduction\n\n- Open problems in the field are not discussed and \nmotivation for the proposed model is missing. \n  \n- Poor English throughout the paper and generally a poor presentation. \nThe first sentence\n\"Computational Fluid Dynamics (CFD) has become the core technology behind almost every fluid\nsimulation.\" is almost tautologic. Many ill-composed sentences such as\n\"A thorough study of data-driven methods using machine learning approaches for\nmodeling the turbulence was studied by ...\".\n\n- tilde and bar variables not defined in Eq. 4\n\n- No outline of the paper in the introduction. Not clear what is the purpose of some sections like 3.2 and 3.3.\n\n- The author should care to explain what is the output of the network. \n\n- Not clear what models the authors are benchmarking against, as\nthey don't provide details and references.\n Possibly they should apply their attention augmentation to  State of the art\n architectures such as the ones analyzed in the NSFNets paper.\n\n- The authors say they evaluate on a few different metrics, but all I can see reported is\n\"accumulated errors\", which I have no idea what it is since it has not been defined. \n\n- A figure is not a detailed explanation of a NN block but only a visual help, please provide definitions in formulas.", "summary_of_the_paper": "The authors consider Navier-Stokes equation in 2D and the 1D Burger's equation.\nUsing a PINN approach to model the solution with neural networks, they propose an architecture that alternates spatial and channel attention blocks to convolutional blocks. \n", "summary_of_the_review": "The proposed model may have some selling points to it, but the presentation\nis so poor and so many details are missing that it is impossible to tell.\nThis manuscript is far from being up to ICLR standards. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "JbBhmRV-j-S", "reviewer": "Reviewer_oeTZ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "PINNs are a vibrant research area that combine ideas from deep learning and numerical methods and scientific computing. Advancing this area is of interest for the scientific machine learning community and work that advances the state-of-the-art can have significant impact. Designing new architectures that are better suited for certain scientific problems is a plausible research direction since (a) standard PINNs rely on very simple fully-connected architectures and (b) architectures that are designed for CV and NLP problems might not be optimal for scientific problems.  Hence, the research question that the authors address is of interest. \n\nHowever, the paper has several weaknesses and there are several open questions after reading the paper.\n\n1. Method:\n\n* The proposed spatial and channel attention models are insufficiently discussed. It is not clear to follow the logic and what the motivation for the design of these modules is, in the specific context. The concept of attention is not explained. \n\n* It would be helpful to have more details about the 1x1 conv layers to understand whether the dimensions are increased or reduced. I assume that the authors use zero-padding and stride of 1 (details are not provided). Feeding a given tensor x of shape (B, K, H, W) into the proposed spatial attention model that applies twice a 1x1 convolution layer with F_1 and F_2 filters should return a new tensor of shape (B, F_2, H, W), i.e., the spatial attention model is changing the filter dimension from K to F_1 to F_2. Since this output is added to the input tensor x, F_2=K. But, details about F_1 are not provided and it is not unclear whether the number of channels are increased or decreased here, i.e., if this module is introducing some bottleneck. \n\n* It is not clear what is meant by `reshape' and what the addition operation is doing in Figure 1 and 2. \n\n* The ablation study does indicate that there is some effect of using both modules together, but the discussion of the ablation study is insufficient. On which dataset has the study been performed? Why does the model with both modules has less parameters? Does this mean the modules essential reduce the dimensions? Also a baseline model that does not use any additional module is missing.\n\n* Why is the new loss function more robust? How do you tune the many new parameters that are introduced? A detailed ablation study is missing. \n\n\n2. Experiments\n\n* Following the reasoning behind the experiments and understanding the content in the tables is difficult. First, it is not clear where all the other models are coming from. For example, has a GAN been trained on the specific dataset by someone else, or by the authors? In the former case, some references are needed; in the latter case, more information about the setup is needed.\n\n* Next, I don't know what the authors mean by `accumulated errors'. The author introduce other metrics for evaluation, but I cannot find any table that shows, for instance, the RMSE. \n\n* In Figure 6, what is the difference between the second and third row?\n\n\n3. General comments\n\n* It would be helpful to clearly state and discuss the model / PINN framework that the authors consider.\n\n* The related work jumps between general deep neural networks for scientific applications and PINNs. It is confusing to follow the related works.\n\n* What do the authors mean by `running at good enough speed'?\n\n* The background section can be moved to the appendix. It is not relevant to understand other parts of the paper.\n\n* The additional provided materials seem not relevant for the general discussion. \n\n* I strongly suggest that the authors expand the discussion and provide additional details to clarify the working mechanisms of both the spatial and channel module. ", "summary_of_the_paper": "The authors propose a new architecture and loss function for training physics-informed neural networks (PINNs) on fluid flow problems. The idea is centered around augmenting a basic residual network with two additional attention blocks that are placed before and after residual blocks. These two new blocks aim to introduce channel and spatial attention into the model. In addition, they propose a new loss function that is tailored to solving PDEs such as Navier Stokes and Burgers Equation.", "summary_of_the_review": "The authors introduce a new architecture for PINNs, which can be seen to be novel up to a certain extend. However, modules of similar flavor have been proposed in previous works and simply proposing a new architecture that augments an existing architecture with additional modules or layers is not sufficient. The experiments need to clearly study and demonstrate the advantages on a range of problems, in particular if no theoretical justifications are provided. The provided experiments do not live up to the standards that I would expect from an empirical paper. Further, the presentation and discussion of the ideas is insufficient and many details are missing. Also, no research code is provided to reproduce or better understand the proposed architecture. The overall quality of this paper is below the acceptance threshold for ICLR. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: All four reviewers agree that the paper should be rejected in its current form, but make numerous suggestions for improving it. The main points of concern were the motivation of the proposed method, novelty and the quality of the presentation of the work. The authors did not provide a response. The AC agrees with the reviewers and recommends rejecting the paper.
Decision: Reject