We evaluate the performance of the physics informed deep learning paradigm for solving the Biot system modeling coupled ﬂow and poromechanics using Mandel’s problem analytical solution. The solution presents a unique set of challenges to the deep learning paradigm, such as the disparity in expected orders of magnitude of the output variables as well as the non-monotonicity and steep gradient in one of the output variables in a certain spatio- temporal domain. We tackle those challenges in this work and comment on the effect of activation function and minimization algorithm on the deep learning framework. Keywords Physics informed deep learning Biot systemMandel’s problemTensorFlow SciPyActivation function Batch gradient descent 1 Introduction input layer hidden layers output layer targets bias (a) Basic neural net construction (b) Activity at the neuron level Figure 1: Aspects of neural net architecture The world of business has been leveraging machine learning in several diverse application areas like analyzing sales data towards streamlining the data, real-time mobile personalization to promote user experience, fraud detection with the help of pattern detection, product recommendations based on customer feedback, dynamic pricing based on user demand and natural language processing to better the ability of machines to interact with humans. The advent of open-source packages like TensorFlow [1], PyTorch [2], Theano [3] along with the availability of Python packages like SciPy [4], numpy [5], scikit-learn [6] has spawned interest in exploring the deep learning paradigm in engineering sciences as well (see [7 –27]. As shown in Figure 1a, a neural network is composed of several connected layers of artiﬁcial neurons and biases where the data is Physics informed deep learning for coupled ﬂow and poromechanics. Mandel’s problem fed into the input layer and ﬂows through some hidden layers. As shown in Figure 1b, the output of each neuron is computed by multiplying the outputs from the previous layer with the corresponding weights followed by an alteration by an activation function [28, 29]. In the training phase, the weights of the neural network are initialized [30, 31], followed by continual updates on the weights using an algorithm such that the objective function is minimized. The objective function is the cumulative error due to training of the data set on the spatio-temporal boundary of the domain and the physics informed error corresponding to the residue of the partial differential equations. To formalize the framework mathematically, let N(X;W;b) be a L-layer neural network with input vector X, output vector yand network parameters Wandbcalled weights and biases respectively. Each layers creates data for the next layer through the nested transformations: ol='l(Wlol1+bl); l= 1;;Lwhere o0xandoLyand'lare activation functions and the mapping is represented as y7!N (X;W;b). In this work, we test the neural net construction framework’s metrics in the coupled ﬂow and poromechanics space for the prevalent Mandel’s problem [32, 33] which presents its unique set of challenges to the deep learning paradigm. This paper is structured as follows: the Biot system is presented in this section, the physics informed machine learning paradigm for Mandel’s problem is presented in section 2, results are presented in section 3, and conclusions and outlook are presented in section 4. 1.1 Coupled ﬂow and poromechanics Let be the domain under consideration. The Biot system consists of the ﬂow model and the poromechanics model. Let the boundary @ = f D[f Np D[p Nwhere f D,f Nandf D,f Nare the Dirichlet and Neumann boundaries for the ﬂow and poromechanics models respectively. The system of equations is Mass conservation :@ @t(1 Mp+ ru) +rz=q Darcy law : z=K (rp0g) =(rp0g) Linear momentum balance : r+f=0 Constitutive law : =(ru)I+ 2G pI Small strain kinematics : (u) =1 2(ru+ (ru)T) BCs :p=gon f D;zn=qDon f N;un1=uDon p D;Tn2=ton p NICs :p(x;0) =p0(x); (x;0) =0(x); (x;0) = 0(x);u(x;0) =u0(x)(1) wherep: (0;T]!Ris the ﬂuid pressure, z: (0;T]!R3is the ﬂuid ﬂux, Mis the Biot modulus, qis the source or sink term, Kis the uniformly symmetric positive deﬁnite absolute permeability tensor, is the ﬂuid viscosity, 0is a reference density, is the porosity, =K is a measure of the hydraulic conductivity of the pore ﬂuid, cis the ﬂuid compressibility, T >0is the time interval, u: [0;T]!R3 is the solid displacement, fis the body force per unit volume, is the strain tensor, is the Cauchy stress tensor,Gis shear modulus and is lame parameter. With regards to the conventional forward modeling paradigm, the ﬁxed stress split staggered solution algorithm [34 –43] is popularly employed to solve the system of equations arising out of the physics of coupled ﬂow and poromechanics. 1.2 Mandel’s problem and its importance in deep learning As shown in Figure 2, a rectangular isotropic specimen is sandwiched between rigid, frictionless plates and the lateral sides are traction-free. At t= 0+, a force intensity of 2FN=m is applied to the rigid plates. Plane strain condition is applicable i.e. zz= 0. The boundary conditions are aZ ayyjy=bdx=2F;pjx=a= 0;vjy=b= 0;xxjx=a=xyjx=a= 08t>0 Given the biaxial symmetry of the problem, only a quarter of the domain needs to be modeled. With U representing the analytical solution for the ydisplacement at y=b, the boundary conditions are recast as ujx=0=vjy=0=pjx=a=xxjx=a=xyjx=a= 0;vjy=b=U8t>0 We use the following material parameters: a= 1m,b= 1m,T= 1s,= 8:48107Pa, = 1, k= 100D,= 10cp,F= 1N=m to generate the analytical solution which is plotted in Figure 3. The 2 Physics informed deep learning for coupled ﬂow and poromechanics. Mandel’s problem 2a2bxy2F 2Fp= 0 σxx= 0 σxy= 0F u·n= 0 u·n= 0 Figure 2: Circles indicate rollers and solid black boxes indicate rigid frictionless plates t0.0 0.2 0.4 0.6 0.8 1.0y 0.00.20.40.60.81.0p 0.00.10.20.30.40.50.6Pressure at x/a=0 (a) Non-monotonic pressure response at center of specimen t0.0 0.2 0.4 0.6 0.8 1.0y 0.00.20.40.60.81.0u1e9 012345x displacement at x/a=1(b) The x-displacement at the lateral free edge is roughly 8 orders of magnitude less than the pressure Figure 3: Mandel’s problem important aspects of the solution that present a challenge to the machine learning paradigm are the disparity in expected order of magnitude in the target variables and the non-monotonicity in one of the target variables in a certain spatio-temporal subdomain. 2 Physics informed deep learning The deep learning model used for training is a fully-connected neural network where the number of hidden layersNhand the number of neurons Nnper hidden layer is adjusted depending on the problem under consideration [44], [45]. With X  (x;y;t )representing the spatio-temporal input vector, we deploy a separate neural net for each primary variable as follows ^ 7!N (X;W;b); =u;v;p;xx;xy;yy We rewrite the governing differential equations for the Mandel’s problem as @ @t(1 Mp+ (@u @x+@v @y))r2p= 0 xx=(@u @x+@v @y) + 2G@u @x p xy=G@u @y+@v @x yy=(@u @x+@v @y) + 2G@v @y p @xx @x+@xy @y= 0 @yx @y+@yy @y= 0 3 Physics informed deep learning for coupled ﬂow and poromechanics. Mandel’s problem The corresponding residuals are p(X) :=@ @t(1 M^p+ (@^u @x+@^v @y))(@2^p @x2+@2^p @y2) xx(X) := ^xx((@^u @x+@^v @y) + 2G@^u @x ^p) xy(X) := ^xyG@^u @y+@^v @x yy(X) := ^yy((@^u @x+@^v @y) + 2G@^v @y ^p) u(X) :=@2^u @x2+@2^v @x@y + 2G@2^u @x2 @^p @x+G@2^u @y2+@2^v @x@y v(X) :=@2^u @x@y+@2^v @y2 + 2G@2^v @y2 @^p @y+G@2^v @x2+@2^u @x@y 2.1 Loss function for Mandel’s problem LetN0be number of initial points sampled, Nbbe number of boundary points sampled and Nfbe the number of collocation points sampled in the interior of the domain using the Latin hypercube method [46]. With the notationjj()jj:=j()j2, the loss functionLto be minimised is L=Lb+P 2fu;v;p;xx;xy;yyg(L0 +L ) L0 =1 N0N0P i=1jj^ (Xji t=0) (Xji t=0)jj Lb=1 NbNbP i=1 jj^u(Xji x=0)jj+jj^v(Xji y=0)jj+jj^p(Xji x=a)jj+jj^v(Xji y=b)U(Xji y=b)jj +jj^xx(Xji x=a)jj+jj^xy(Xji x=a)jj L =1 NfNfP i=1(jj (Xi)jj+jj^ (Xi) (Xi)jj) whereL0andLbcorrespond to loss due to training on initial and boundary conditions respectively, whereas Lp;Lu;Lv;Lxx;LxyandLyycorrespond to loss due to training on the variables p;u;v;xx;xyandyy respectively. 2.2 Training process We test three different activation functions i.e. sigmoid, ReLU [47], and hyperbolic tangent. Weights are initialized in accordance with Xavier scheme [30]. The residuals involve computation of derivatives which are obtained using automatic differentiation [48]. The loss function is optimized with two competing schemes: Limited memory Broyden-Fletcher-Goldfarb-Shanno, L-BFGS [49] and Batch gradient descent, BGD, [50]. There are many BGD algorithmic variants and we choose the Adam [51] variant due to its popularity in the machine learning community [52]. 3 Numerical results We use TensorFlow [1] and run simulations on a dual core AMD Ryzen 3 3200U processor with Radeon Vega 3 Graphics card. We generate the exact solution points on a rectangular mesh ( [0;1][0;1]) with 20 equidistant intervals in both xandydirection. Using 49equidistant intervals in the temporal domain [0;1], in total, we have Ntotal= 202050 = 20000 examples. We use N0= 200 for initial conditions, Nb= 500 data points for boundary conditions on each of the four boundaries, Nf= 15000 data points in the interior of the domain, Nh= 4andNn= 10 . The stopping criterion for both schemes iskLk+1Lkk kLk+1k<1106 3.1 Capturing the non-monotonic pressure response As shown in Figure 4, we attempt to capture the non-monotonic pressure response using the different activation functions for the Adam BGD with a batch size of 1000 and learning rate of 0.0005. While none 4 Physics informed deep learning for coupled ﬂow and poromechanics. Mandel’s problem 0.0 0.2 0.4 0.6 0.8 1.0 Time0.00.10.20.30.40.50.6Pressure at x/a=0Learning rate=0.0005, batch size=1000 Analytical Adam-tanh Adam-relu Adam-sigmoid 0.0 0.2 0.4 0.6 0.8 1.0 Time0.00.10.20.30.40.50.6Pressure at x/a=0Learning rate=0.01, batch size=1000 Analytical Adam-tanh Adam-relu Adam-sigmoid Figure 4: Comparison of pressure response using different learning rates for Adam Batch Gradient Method of the combinations does a perfect job, we observe that a learning rate of 0.0005 is more optimal from the standpoint of capturing this critical physical phenomena 3.2 Relative error in pressure 50 100 150 200 Training epochs0.30.40.50.60.70.80.91.0Relative error in pressureAdam, learning rate=0.0005 relu sigmoid tanh 50 100 150 200 Training epochs0.30.40.50.60.70.80.91.0Relative error in pressureAdam, learning rate=0.01 relu sigmoid tanh Figure 5: Relative error in pressure iskp^pk kpk As shown in Figure 5, we track the relative error in pressure as the training epochs proceed. We observe that the ReLU activation function with a batch size of 1000 and a learning rate of 0.0005 is optimal from the standpoint of the error metrics. This is consistent with our observation from the standpoint of capturing the non-monotonicity in the previous module. 3.3 Decay in loss function We plot the decay in loss function versus training epochs in Figure 6. We again observe that the batch size of 1000 with a learning rate of 0.0005 is optimal from the standpoint of avoiding the ﬂuctuations in the loss function as the training epochs proceed. 5 Physics informed deep learning for coupled ﬂow and poromechanics. Mandel’s problem 50 100 150 200 Training epochs1.4 1.2 1.0 0.8 0.6 0.4 0.2 0.0logLAdam, learning rate=0.0005 relu sigmoid tanh 50 100 150 200 Training epochs1.6 1.5 1.4 1.3 1.2 1.1 1.0 0.9 0.8 logLAdam, learning rate=0.01 relu sigmoid tanh Figure 6: Loss function decay 2 1 0 1 2 31.0 0.5 0.00.51.01.52.02.53.0 sigmoid tanh ReLU truncated sine truncated cosine Figure 7: Activation functions employed 3.4 Introducing the truncated sine and cosine activation functions We now introduce the truncated sine and cosine activation functions, and proceed to observe the effects of this choice on the overall behavior of the deep learning framework. The argument for choosing these functions is that the non-monotonicity would be better captured is the activation function itself is non-monotonic in behavior. The truncated sine function is such that f(x) = 0 forx<0 sin(x) forx>0 and the truncated cosine function is such that f(x) = 0 forx<0 cos(x) forx>0 The functions are plotted in Figure 7. We compare the output of the deep learning with the truncated sine and cosine activation functions as shown in Figure 8. It is evident that the truncated sine activation function with BFGS optimizer holds the most promise in terms of being able to capture the non-monotonic behavior. 6 Physics informed deep learning for coupled ﬂow and poromechanics. Mandel’s problem 0.0 0.2 0.4 0.6 0.8 1.0 Time0.00.10.20.30.40.50.6Pressure at x/a=0Learning rate=0.0005, batch size=1000 Analytical Adam-sine Adam-cosine 0.0 0.2 0.4 0.6 0.8 1.0 Time0.00.10.20.30.40.50.6Pressure at x/a=0BFGS Analytical Adam-sine Adam-cosine Figure 8: Comparing of the deep learning framework with truncated sine and cosine activation functions 4 Conclusions It is evident that a truncated sine activation function holds a lot of promise for the problem under consideration. It is also clear that the loss function deﬁnition needs to be looked at in more detail to carefully capture the non-monotonic pressure response. Aspects like regularization to avoid overﬁtting would be critical in this respect, since we would ideally like the deep learning framework to not be too dependent on the hyperparameters like number of layers and number of neurons per hidden layer as well as the number of data points being factored into consideration for training the deep learning model.