LEGENDRE DEEP NEURAL NETWORK (LDNN)
Year: 2020

Abstract:
Various phenomena in biology, physics, and engineering are modeled by differential equations. These differential equations including partial differential equations and ordinary differential equations can be converted and represented as integral equations. In particular, Volterra–Fredholm–Hammerstein integral equations are the main type of these integral equations and researchers are interested in investigating and solving these equations. In this paper, we propose Legendre Deep Neural Network (LDNN) for solving nonlinear Volterra–Fredholm–Hammerstein integral equations (V-F-H-IEs). LDNN utilizes Legendre orthogonal polynomials as activation functions of the Deep structure. We present how LDNN can be used to solve nonlinear V-F-H-IEs. We show using the Gaussian quadrature collocation method in combination with LDNN results in a novel numerical solution for nonlinear V-F-H-IEs. Several examples are given to verify the performance and accuracy of LDNN.

heading: 1 INTRODUCTION
text: Deep neural networks are a main and beneficial part of machine learning family which are applied in various areas including speech processing, computer vision, natural language processing and image processing (LeCun et al., 2015; Krizhevsky et al., 2012). Also, the approximation of the functions is a significant branch in scientific computational and achieving success in this area is considered by some research (Tang et al., 2019; Hanin, 2019). Solving differential equations is the other main branch of scientific computational which neural networks and deep learning have been shown success in this area. (Lample & Charton, 2019; Berg & Nyström, 2018; Raissi et al., 2019). Various phenomena in biology, physics, finance, neuroscience and engineering are modeled by differential equations (Courant & Hilbert, 2008; Davis, 1961). In recent years, several researchers studied the solving differential equations via deep learning or neural networks. differential equations consists of ordinary differential equations, partial differential equations and integral equations. (Sirignano & Spiliopoulos, 2018; Lu et al., 2019; Meng et al., 2020). It is notable that the various numerical methods are applied for solving differential equations. Homotopy analysis method (HAM) (Liao, 2012) and variational iteration method (VIM) (He & Wu, 2007) are known as analytical/semi-analytical methods. Usually, spectral methods (Canuto et al., 2012), Runge-Kutta methods (Hairer et al., 2006), the finite difference methods (FDM) (Smith, 1985) and the finite element methods (FEM) (Johnson, 2012) are considered as the popular numerical methods. When the complexity of the model does not allow us to obtain the solution explicitly, numerical methods are a proper selection for finding the approximate solution for the models. Recently, some of the machine learning methods are applied for solving differential equations. Chakraverty & Mall (2017) introduced orthogonal neural networks which used orthogonal polynomials in the structure of the network. Raja et al. (2019) applied meta-heuristic optimization algorithm to neural network for obtaining the solution of differential equations. Moreover, other methods of machine learning such as support vector machine (Vapnik, 2013) are used to approximate the solution of the models. Least squares support vector machines are considered in these researches (Hajimohammadi et al., 2020; Mehrkanoon & Suykens, 2015). Baker et al. (2019) selected deep neural networks for solving the differential equations. Pang et al.
(2019) introduced a new network to find the solution of the different equations. Han et al. (2018) solved high-dimensional problems via deep networks. Also, Long et al. (2018) and Raissi et al. (2019) introduced a group of the equations which solved by deep learning. Furthermore, He et al. (2018) and Molina et al. (2019) investigated the effect of the activation function on networks. In this paper, we concern nonlinear Volterra–Fredholm–Hammerstein integral equations (V-F-HIEs) and try to obtain the solution of them via deep neural network. We present a new numerical approach of machine learning which is a combination of deep neural network and Legendre collocation method. This approach is useful for solving the differential equations and we applied it for solving nonlinear V-F-H-IEs. We used Legendre collocation method to our network for perfect the numerical computations and enhancement the performance the network.

heading: 2 LEGENDRE DEEP NEURAL NETWORK (LDNN)
text: The main purpose of introducing LDNN is to apply it for solving differential models. Indeed, this purpose is to expand the utilization of deep learning networks in the field of scientific computing, especially the solution of differential equations. Moreover, this network has the advantages of solving equations by deep learning as well as numerical methods such as collocation method used to achieve better solution to the equations. LDNN presents a combination of a deep neural network and Legendre collocation method. In fact, our network consists of two networks which have connected consecutive to each other. The first network is a feed forward neural network which has an orthogonal Legendre layer. The second network includes operation nodes to create the desired computational model. In recent decades, numerical methods especially collocation method are popular methods for solving differential equations. In the collocation method, first an approximation of the solution is expanded by using the sum of the basic functions. The basic functions consists of the orthogonal polynomials such as Legendre polynomials.Then this approximation is placed in the differential equation. By considering the appropriate set of candidate points, an attempt is made to obtain the unknown coefficients of the basic functions so that the solution satisfies the equation in a set of candidate points. The first network is applied to creat the approximation of the solution. This approximation can be known as the scattered data interpolation problem. The second network is used to obtain the desired equation so that the solution satisfies it. The structure of LDNN is described in detail at the following rest. Consider that the first network has aM-layer which defined as follows:
H0 = x, x ∈ Rd, H1 = L(W (1)H0 + b(1)), Hi = f(W (i)Hi−1 + b(i)), 2 ≤ i ≤M− 1, HM = W (M)HM−1 + b(M).
where H0 is the input layer with d dimension. Hi, 1 ≤ i ≤ M − 1 are hidden layers, L = [L0, L1, ...Ln]
T which Li are i-th degrees of Legendre orthogonal polynomials, H1 is an orthogonal layer, f is the hyperbolic tangent activation function or other commonly used activation functions. W (i), i = 1, · · · ,M are the weight parameters and b(i), 1 ≤ i ≤ M are the bias parameters. HM is the output layer. It is notable that the second network is applied to obtain the desired differential model. This aim is possible by using operation nodes including integrals, derivatives, and etc. These nodes are applied to the output of the first network. Moreover, automatic differentiation (AD) (Baydin et al., 2017) and Legendre Gaussian integration (Shen et al., 2011) have been used in network computing to obtain more accurate and fast calculations. How to train the network and set the parameters are also important points. Supervised learning method is used to train network. The cost function for setting parameters is defined as follows:
CostFun = min(yt − yp) + min(Rm). (1)
where yt is an exact value of the model and yp is a predicted value of the LDNN. The definition of Rm is explained in section 3.The minimization of CostFun is obtained by performing Adam algorithm (Kingma & Ba, 2015) and the L-BFGS method (Liu & Nocedal, 1989) on mean squared errors of training data set.

heading: 2.1 LEGENDRE POLYNOMIALS
text: Legendre polynomials (Shen et al., 2011) are a main series of orthogonal polynomials which denoted by Ln(η), are defined as:
Ln(η) = 1
2n [n2 ]∑ `=0 (−1)` (2n− 2`)! 2n`!(n− `)!(n− 2`)! ηn−2` (2)
Legendre polynomials are defined in [−1, 1] domain and have the recurrence formula in the following form:
(n+ 1)Ln+1(η) = (2n+ 1)ηLn(η)− nLn−1(η), n ≥ 1, L0(η) = 1, L1(η) = η. (3)
Orthogonality relation for these polynomials is as follows:∫ 1 −1 Ln(η)Lm(η)dη = γδn,m, (4)
where δn,m is a delta Kronecker function and γ = 22n+1 . The weight function of them isW(η) = 1. Some following useful properties of Legendre polynomials are defined:
Ln(−η) = (−1)nLn(η), (5) |Ln(η)| ≤ 1, ∀η ∈ [−1, 1], n ≥ 0, (6) Ln(±1) = (±1)n, (7) (2n+ 1)Ln(η) = L ′ n+1(η)− L′n−1(η), n ≥ 1. (8)

heading: 3 NONLINEAR VOLTERRA–FREDHOLM–HAMMERSTEIN INTEGRAL EQUATIONS AND LDNN
text: The general form of nonlinear Volterra–Fredholm–Hammerstein integral equations (V-F-H-IEs) is as follows:
y(x) = g(x) + ξ1 ∫ x 0 K1(x, s)ϕ1(s, y(s))ds+ ξ2 ∫ 1 0 K2(x, s)ϕ2(s, y(s))ds, x ∈ [0, 1]. (9)
where ξ1, ξ2 are fixed, g(x), K1(x, s) and K2(x, s) are given functions and ϕ1(s, y(s)), ϕ2(s, y(s)) are nonlinear functions. The aim is to find the proper y(x). In order to use the LDNN, reformulated Eq. (9) in the following form:
Rm = −y(x)+g(x)+ ξ1 ∫ x 0 K1(x, s)ϕ1(s, y(s))ds+ ξ2 ∫ 1 0
K2(x, s)ϕ2(s, y(s))ds, x ∈ [0, 1]. (10)
y(x) is approximated by the first network of the LDNN.
y(x) ≈ HM. (11)
Furthermore, we applied Legendre–Gauss integration formula (Shen et al., 2011):∫ 1 −1 h(X)dX = N∑ j=0 ωjh(Xj) (12)
where {Xj}Nj=0 are the roots of Ln+1 and {ωj}Nj=0 = 2(1−X2j )(L′n+1(Xj))2 . Here, we should transfer the [0, x] and [0, 1] domains into the [−1, 1] domain. It is possible by using the following transformation:
t1 = 2
x s− 1, t2 = 2s− 1.
Consider Z1(x, s) = K1(x, s)ϕ1(s, y(s)),
Z2(x, s) = K2(x, s)ϕ2(s, y(s)).
we have
Rm = −y(x) + g(x) + ξ1 x
2 ∫ 1 −1 Z1(x, x 2 (t1 + 1))dt1 + ξ2 2 ∫ 1 −1 Z2(x, x 2 (t2 + 1))dt2. (13)
by using Legendre–Gauss integration formula, the below form is concluded:
Rm = −y(x) + g(x) + ξ1 x
2 N1∑ j=0 ω1jZ1(x, x 2 (t1j + 1)) + ξ2 2 N2∑ j=0 ω2jZ2(x, x 2 (t2j + 1)). (14)
The second network of LDNN and its nodes makes Rm. The architecture of LDNN for solving nonlinear V-F-H-IEs is represented in Figure 1.

heading: 4 NUMERICAL RESULTS
text: In order to present the accuracy and performance of the LDNN for solving nonlinear V-F-H-IEs and justify the efficiency of the proposed method, several examples are given. The convergence behavior of the LDNN is reported by using the following parameters: The exact value yt, the predicted value yp and the absolute error (Error) in some points of test data are reported in various tables. The number of the train data m1, the number of Legendre quadrature points (N1, N2), the number of the test data m2, the structure of network M-layers, Ltrain2 and Ltest2 are shown in Table 1. L train 2 and L test 2 are calculated as follows:
Ltrain2 = ||yt − yp||2 = [ mtr∑ j=1 (yt(xj)− yp(xj))2] 1 2 ,
Ltest2 = ||yt − yp||2 = [ mte∑ j=1 (yt(xj)− yp(xj))2] 1 2 , (15)
The Tensorflow package of Python version 3.7.0. is applied for writing the code of all experiments. Adam algorithm is stoped when the number of iteration is up to 5000 and L-BFGS method is stoped when it converges. The figures are obtained on the test data set.

heading: 4.1 EXPERIMENT 1
text: Suppose that we have the following model (Yousefi & Razzaghi, 2005):
y(x) = ex − 1 3 e3x + 1 3 + ∫ x 0 y3(s)ds, x ∈ [0, 1]. (16)
It has the exact solution y(x) = ex. Table 2 represents the exact value, the predicted value and the absolute error (Error) in several test points on [0, 1] domain. 50 points of shifted Legendre quadrature points are applied for training LDNN. The number of train data set is 500 and the number of test data set is 100. Figure 2 shows the illustrated comparison between yt and yp.

heading: 4.2 EXPERIMENT 2
text: Suppose that we have the following model (Razzaghi & Ordokhani, 2002):
y(x) = 1 + sin2(x) + ∫ 1 0 K(x, s)y2(s)ds, x ∈ [0, 1]. (17)
where
K(x, s) = {−3 sin(x− s), 0 ≤ s ≤ x;
0 x ≤ s ≤ 1. (18)
It has the exact solution y(x) = cos(x). The exact value, the predicted value and the absolute error (Error) in several test points on [0, 1] domain are reported in Table 3. 50 points of shifted Legendre quadrature points are applied for training LDNN. The number of train data set is 500 and the number of test data set is 100. Figure 3 shows the illustrated comparison between yt and yp.

heading: 4.3 EXPERIMENT 3
text: Suppose that we have the following model (Babolian et al., 2007):
y(x) = g(x) + ∫ x 0 (x− s)y2(s)ds+ ∫ 1 0 (x+ s)y(s)ds, x ∈ [0, 1]. (19)
where
g(x) = − 1 30 x6 + 1 3 x4 − x2 + 5 3 x− 5 4 (20)
It has the exact solution y(x) = x2 − 2. Table 4 illustrates the exact value, the predicted value and the absolute error (Error) in several test points on [0, 1] domain. 50 points of shifted Legendre quadrature points are applied for training LDNN. The number of train data set is 500 and the number of test data set is 100. Figure 4 represented the comparison between yt and yp.

heading: 4.4 EXPERIMENT 4
text: Suppose that we have the following model (Hadizadeh & Mohamadsohi, 2005):
y(x) = − 1 10 x4 + 5 6 x2 + 3 8 + ∫ x 0 1 2x y2(s)ds, x ∈ [0, 1]. (21)
It has the exact solution y(x) = x2 + 12 . The exact value, the predicted value and the absolute error (Error) in several test points on [0, 1] domain are reported in Table 5. 50 points of shifted Legendre quadrature points are applied for training LDNN. The number of train data set is 500 and the number of test data set is 100. Figure 5 shows the illustrated comparison between yt and yp.

heading: 5 CONCLUSION
text: Legendre deep neural network (LDNN) is introduced in this paper. LDNN and its application for solving nonlinear Volterra–Fredholm–Hammerstein integral equations (V-F-H-IEs) are proposed.
LDNN includes two networks. The first network approximates the solution of a nonlinear V-F-H-IE y(x) which hasM-layers feed forward neural network structure. The first hidden layer of this has a orthogonal layer consists of Legendre polynomials as activation functions. The last network adjusts the output of the sooner network to fit to a desired equation form. The better performance of the network has been obtained by using Legendre Gaussian integration and automatic differentiation. Some experiments of nonlinear V-F-H-IEs are given to investigate the reliability and validity of LDNN. The results show that this network is an efficient and has high accuracy.

Review summaries:
The paper proposed the Legendre Deep Neural Network (LDNN) to solve Volterra–Fredholm–Hammerstein integral equations. Specifically, the network uses Legendre polynomials as the activation in the first layer and uses Gaussian quadrature to discretize the integral operator as a summation. The numerical examples are performed to verify the performance of LDNN. However, the method is not novel and the numerical examples are too simple.

The authors present a neural network based method to solve a special class of integral equations. Their approach involves training a neural network with Legendre polynomial based activation functions to approximate the solution $y(x)$ for a given $x$. The network is trained in a supervised fashion to minimize a loss function with two term- (1) the $\ell_2$ error between the true solution and $y(x)$ and (2) the residual of the given integral  equation when analysed at $x$. They show impressive numerical results for several instances of VFH-IEs with very low errors. The primary contributions as claimed by the authors are the use of Legendre polynomial based activation functions and creating a differentiable approximation for the integral equation by using Legendre polynomials and Quadrature methods to analyse the integral.