Legendre Deep Neural Network (LDNN) and its application for approximation of nonlinear Volterra–Fredholm–Hammerstein integral equations

1 INTRODUCTION . Deep neural networks are a main and beneficial part of machine learning family which are applied in various areas including speech processing , computer vision , natural language processing and image processing ( LeCun et al. , 2015 ; Krizhevsky et al. , 2012 ) . Also , the approximation of the functions is a significant branch in scientific computational and achieving success in this area is considered by some research ( Tang et al. , 2019 ; Hanin , 2019 ) . Solving differential equations is the other main branch of scientific computational which neural networks and deep learning have been shown success in this area . ( Lample & Charton , 2019 ; Berg & Nyström , 2018 ; Raissi et al. , 2019 ) . Various phenomena in biology , physics , finance , neuroscience and engineering are modeled by differential equations ( Courant & Hilbert , 2008 ; Davis , 1961 ) . In recent years , several researchers studied the solving differential equations via deep learning or neural networks . differential equations consists of ordinary differential equations , partial differential equations and integral equations . ( Sirignano & Spiliopoulos , 2018 ; Lu et al. , 2019 ; Meng et al. , 2020 ) . It is notable that the various numerical methods are applied for solving differential equations . Homotopy analysis method ( HAM ) ( Liao , 2012 ) and variational iteration method ( VIM ) ( He & Wu , 2007 ) are known as analytical/semi-analytical methods . Usually , spectral methods ( Canuto et al. , 2012 ) , Runge-Kutta methods ( Hairer et al. , 2006 ) , the finite difference methods ( FDM ) ( Smith , 1985 ) and the finite element methods ( FEM ) ( Johnson , 2012 ) are considered as the popular numerical methods . When the complexity of the model does not allow us to obtain the solution explicitly , numerical methods are a proper selection for finding the approximate solution for the models . Recently , some of the machine learning methods are applied for solving differential equations . Chakraverty & Mall ( 2017 ) introduced orthogonal neural networks which used orthogonal polynomials in the structure of the network . Raja et al . ( 2019 ) applied meta-heuristic optimization algorithm to neural network for obtaining the solution of differential equations . Moreover , other methods of machine learning such as support vector machine ( Vapnik , 2013 ) are used to approximate the solution of the models . Least squares support vector machines are considered in these researches ( Hajimohammadi et al. , 2020 ; Mehrkanoon & Suykens , 2015 ) . Baker et al . ( 2019 ) selected deep neural networks for solving the differential equations . Pang et al . ( 2019 ) introduced a new network to find the solution of the different equations . Han et al . ( 2018 ) solved high-dimensional problems via deep networks . Also , Long et al . ( 2018 ) and Raissi et al . ( 2019 ) introduced a group of the equations which solved by deep learning . Furthermore , He et al . ( 2018 ) and Molina et al . ( 2019 ) investigated the effect of the activation function on networks . In this paper , we concern nonlinear Volterra–Fredholm–Hammerstein integral equations ( V-F-HIEs ) and try to obtain the solution of them via deep neural network . We present a new numerical approach of machine learning which is a combination of deep neural network and Legendre collocation method . This approach is useful for solving the differential equations and we applied it for solving nonlinear V-F-H-IEs . We used Legendre collocation method to our network for perfect the numerical computations and enhancement the performance the network . 2 LEGENDRE DEEP NEURAL NETWORK ( LDNN ) . The main purpose of introducing LDNN is to apply it for solving differential models . Indeed , this purpose is to expand the utilization of deep learning networks in the field of scientific computing , especially the solution of differential equations . Moreover , this network has the advantages of solving equations by deep learning as well as numerical methods such as collocation method used to achieve better solution to the equations . LDNN presents a combination of a deep neural network and Legendre collocation method . In fact , our network consists of two networks which have connected consecutive to each other . The first network is a feed forward neural network which has an orthogonal Legendre layer . The second network includes operation nodes to create the desired computational model . In recent decades , numerical methods especially collocation method are popular methods for solving differential equations . In the collocation method , first an approximation of the solution is expanded by using the sum of the basic functions . The basic functions consists of the orthogonal polynomials such as Legendre polynomials.Then this approximation is placed in the differential equation . By considering the appropriate set of candidate points , an attempt is made to obtain the unknown coefficients of the basic functions so that the solution satisfies the equation in a set of candidate points . The first network is applied to creat the approximation of the solution . This approximation can be known as the scattered data interpolation problem . The second network is used to obtain the desired equation so that the solution satisfies it . The structure of LDNN is described in detail at the following rest . Consider that the first network has aM-layer which defined as follows : H0 = x , x ∈ Rd , H1 = L ( W ( 1 ) H0 + b ( 1 ) ) , Hi = f ( W ( i ) Hi−1 + b ( i ) ) , 2 ≤ i ≤M− 1 , HM = W ( M ) HM−1 + b ( M ) . where H0 is the input layer with d dimension . Hi , 1 ≤ i ≤ M − 1 are hidden layers , L = [ L0 , L1 , ... Ln ] T which Li are i-th degrees of Legendre orthogonal polynomials , H1 is an orthogonal layer , f is the hyperbolic tangent activation function or other commonly used activation functions . W ( i ) , i = 1 , · · · , M are the weight parameters and b ( i ) , 1 ≤ i ≤ M are the bias parameters . HM is the output layer . It is notable that the second network is applied to obtain the desired differential model . This aim is possible by using operation nodes including integrals , derivatives , and etc . These nodes are applied to the output of the first network . Moreover , automatic differentiation ( AD ) ( Baydin et al. , 2017 ) and Legendre Gaussian integration ( Shen et al. , 2011 ) have been used in network computing to obtain more accurate and fast calculations . How to train the network and set the parameters are also important points . Supervised learning method is used to train network . The cost function for setting parameters is defined as follows : CostFun = min ( yt − yp ) + min ( Rm ) . ( 1 ) where yt is an exact value of the model and yp is a predicted value of the LDNN . The definition of Rm is explained in section 3.The minimization of CostFun is obtained by performing Adam algorithm ( Kingma & Ba , 2015 ) and the L-BFGS method ( Liu & Nocedal , 1989 ) on mean squared errors of training data set . 2.1 LEGENDRE POLYNOMIALS . Legendre polynomials ( Shen et al. , 2011 ) are a main series of orthogonal polynomials which denoted by Ln ( η ) , are defined as : Ln ( η ) = 1 2n [ n2 ] ∑ ` =0 ( −1 ) ` ( 2n− 2 ` ) ! 2n ` ! ( n− ` ) ! ( n− 2 ` ) ! ηn−2 ` ( 2 ) Legendre polynomials are defined in [ −1 , 1 ] domain and have the recurrence formula in the following form : ( n+ 1 ) Ln+1 ( η ) = ( 2n+ 1 ) ηLn ( η ) − nLn−1 ( η ) , n ≥ 1 , L0 ( η ) = 1 , L1 ( η ) = η . ( 3 ) Orthogonality relation for these polynomials is as follows : ∫ 1 −1 Ln ( η ) Lm ( η ) dη = γδn , m , ( 4 ) where δn , m is a delta Kronecker function and γ = 22n+1 . The weight function of them isW ( η ) = 1 . Some following useful properties of Legendre polynomials are defined : Ln ( −η ) = ( −1 ) nLn ( η ) , ( 5 ) |Ln ( η ) | ≤ 1 , ∀η ∈ [ −1 , 1 ] , n ≥ 0 , ( 6 ) Ln ( ±1 ) = ( ±1 ) n , ( 7 ) ( 2n+ 1 ) Ln ( η ) = L ′ n+1 ( η ) − L′n−1 ( η ) , n ≥ 1 . ( 8 ) 3 NONLINEAR VOLTERRA–FREDHOLM–HAMMERSTEIN INTEGRAL EQUATIONS AND LDNN . The general form of nonlinear Volterra–Fredholm–Hammerstein integral equations ( V-F-H-IEs ) is as follows : y ( x ) = g ( x ) + ξ1 ∫ x 0 K1 ( x , s ) ϕ1 ( s , y ( s ) ) ds+ ξ2 ∫ 1 0 K2 ( x , s ) ϕ2 ( s , y ( s ) ) ds , x ∈ [ 0 , 1 ] . ( 9 ) where ξ1 , ξ2 are fixed , g ( x ) , K1 ( x , s ) and K2 ( x , s ) are given functions and ϕ1 ( s , y ( s ) ) , ϕ2 ( s , y ( s ) ) are nonlinear functions . The aim is to find the proper y ( x ) . In order to use the LDNN , reformulated Eq . ( 9 ) in the following form : Rm = −y ( x ) +g ( x ) + ξ1 ∫ x 0 K1 ( x , s ) ϕ1 ( s , y ( s ) ) ds+ ξ2 ∫ 1 0 K2 ( x , s ) ϕ2 ( s , y ( s ) ) ds , x ∈ [ 0 , 1 ] . ( 10 ) y ( x ) is approximated by the first network of the LDNN . y ( x ) ≈ HM . ( 11 ) Furthermore , we applied Legendre–Gauss integration formula ( Shen et al. , 2011 ) : ∫ 1 −1 h ( X ) dX = N∑ j=0 ωjh ( Xj ) ( 12 ) where { Xj } Nj=0 are the roots of Ln+1 and { ωj } Nj=0 = 2 ( 1−X2j ) ( L′n+1 ( Xj ) ) 2 . Here , we should transfer the [ 0 , x ] and [ 0 , 1 ] domains into the [ −1 , 1 ] domain . It is possible by using the following transformation : t1 = 2 x s− 1 , t2 = 2s− 1 . Consider Z1 ( x , s ) = K1 ( x , s ) ϕ1 ( s , y ( s ) ) , Z2 ( x , s ) = K2 ( x , s ) ϕ2 ( s , y ( s ) ) . we have Rm = −y ( x ) + g ( x ) + ξ1 x 2 ∫ 1 −1 Z1 ( x , x 2 ( t1 + 1 ) ) dt1 + ξ2 2 ∫ 1 −1 Z2 ( x , x 2 ( t2 + 1 ) ) dt2 . ( 13 ) by using Legendre–Gauss integration formula , the below form is concluded : Rm = −y ( x ) + g ( x ) + ξ1 x 2 N1∑ j=0 ω1jZ1 ( x , x 2 ( t1j + 1 ) ) + ξ2 2 N2∑ j=0 ω2jZ2 ( x , x 2 ( t2j + 1 ) ) . ( 14 ) The second network of LDNN and its nodes makes Rm . The architecture of LDNN for solving nonlinear V-F-H-IEs is represented in Figure 1 .

Summary: The authors present a neural network based method to solve a special class of integral equations. Their approach involves training a neural network with Legendre polynomial based activation functions to approximate the solution $y(x)$ for a given $x$. The network is trained in a supervised fashion to minimize a loss function with two term- (1) the $\ell_2$ error between the true solution and $y(x)$ and (2) the residual of the given integral  equation when analysed at $x$. They show impressive numerical results for several instances of VFH-IEs with very low errors. The primary contributions as claimed by the authors are the use of Legendre polynomial based activation functions and creating a differentiable approximation for the integral equation by using Legendre polynomials and Quadrature methods to analyse the integral.