Legendre Deep Neural Network (LDNN) and its application for approximation of nonlinear Volterra–Fredholm–Hammerstein integral equations
Authors: Kourosh Parand
Zeinab Hajimohammadi
Ali Ghodsi
Venue:  (2021)
Keywords: Deep neural network
Volterra–Fredholm–Hammerstein integral equations
Legendre orthogonal polynomials
Gaussian quadrature method
Collocation method

Abstract: Various phenomena in biology, physics, and engineering are modeled by differential equations. These differential equations including partial differential equations and ordinary differential equations can be converted and represented as integral equations. In particular, Volterra–Fredholm–Hammerstein integral equations are the main type of these integral equations and researchers are interested in investigating and solving these equations.  In this paper, we propose Legendre Deep Neural Network (LDNN) for solving nonlinear Volterra–Fredholm–Hammerstein integral equations (V-F-H-IEs). LDNN utilizes Legendre orthogonal polynomials as activation functions of the Deep structure. We present how LDNN can be used to solve nonlinear V-F-H-IEs. We show using the Gaussian quadrature collocation method in combination with LDNN results in a novel numerical solution for nonlinear V-F-H-IEs. Several examples are given to verify the performance and accuracy of LDNN.

Under review as a conference paper at ICLR 2021
LEGENDRE DEEP NEURAL NETWORK (LDNN) AND ITS APPLICATION FOR APPROXIMATION OF NON- LINEAR COLTERRA-FEEDOLLN-HHAWEESSS TEGRAL EQUATIONS
IN-
Anonymous authors Paper under double-blind review
ABSTRACT
Various phenomena in biology, physics, and engineering are modeled by differen- tial equations. These differential equations including partial differential equations and ordinary differential equations can be converted and represented as integral equations. In particular, the main type of these integral equations and researchers are interested in inves- tigating and solving these equations. In this paper, we propose Legendre Deep Fredholm-Hammnnnn
integral equations are
Neural Network (LDNN) for solving nonlinear integral equations (V-F-H-IEs). LDNN utilizes Legendre orthogonal polynomials as activation functions of the Deep structure. We present how LDNN can be used to solve nonlinear V-F-H-IEs. We show using the Gaussian quadrature colloca- tion method in combination with LDNN results in novel numerical solution for nonlinear V-F-H-IEs. Several examples are given to verify the performance and accuracy of LDNN.
INTRODUCTION
Deep neural networks are main and beneficial part of machine learning family which are applied in various areas including speech processing, computer vision, natural language processing and image processing (LeCun et al. 2015 Krizhevsky et al. 2012). Also, the approximation of the functions significant branch in scientific computational and achieving success in this area is considered by some research (Tang et al.
2019 Hanin 2019). D. Solving differential equations is the other main
branch of scientific computational which neural networks and deep learning have been shown suc- cess in this area. (Lample & Charton 2019 Berg & Nyström 2018 Raissi et al. 2019). Various phenomena biology, physics, finance, neuroscience and engineering are modeled by differential equations (Courant & Hilbert [008] Davis 1961). In recent years, several researchers studied the solving diffcrential equations via deep learning or neural networks. differential equations consists of ordinary differential equations, partial differential equations and integral equations. (Sirignano & Spiliopoulos 2018 Lu et al. 2019 Meng et al.) 2020). is notable that the various numerical meth- ods are applied for solving differential equations. Homotopy analysis method (HAM) (Liao 2012)
and variational iteration method (VIM) (He & Wu (007)) are known as analytical/semi-ane satea//semm-analyticaa
methods. Usually, spectral methods (Canuto al. 2012),, Runge-Kutta methods (Hairer et al., 2006).
the finite difference methods (FDM) (Smith) 1985) and the finite element methods (FEM) (Johnson 2012) are considered as the popular numerical methods. When the complexity of the model does not allow us to obtain the solution explicitly. numerical methods are a proper selection for finding the approximate solution for the models. Recently, some of the machine learning methods are applied for solving differential equations. [Chakraverty & Mall] (2017) introduced orthogonal neural net- works which used orthogonal polynomials in the structure of the network. Raja et al. (2019) applied meta-heuristic optimization algorithm to neural network for obtaining the solution of differential equations. Moreover, other methods of machine learning such as support vector machine (Vapnik) 2013) are used to approximate the solution of the models. Least squares support vector machines are considered in these researches (Hajinohammadi et al. 2020; Mehrkanoon & Suykens 2015).. Baker et al. (2019) selected deep neural networks for solving the differential equations. Pang al.

---

Under review as a conference paper at ICLR 2021
2019 introduced new network to find the solution of the different equations. Han al. (2018)
solved high- nigh-dimensional problems via deep networks. Also, Long et al. (2018) and Raissi et al.
2019 introduced group of the equations which solved by deep learning. Furthermore, Hee et al.
2018) and Molina et al. 2019) investigated the effect of the activation function on networks. -araa--Feedooo--aaanaae
In this paper, we concern nonlinear IEs) and try to obtain the solution of them via deep neural network. We present new numerical approach of machine learning which is a combination of deep neural network and Legendre collo- cation method. This approach is useful for solving the differential equations and we applied it for solving nonlinear V-F-H-IEs. We used Legendre collocation method to our network for perfect the numerical computations and enhancement the performance the network.
integral equations (V-F-H-
2 LEGENDRE DEEP NEURAL NETWORK (LDNN)
The main purpose of introducing LDNN is to apply it for solving differential models. Indeed, this purpose is to expand the utilization of deep learning networks in the field of scientific computing, especially the solution differential equations. Moreover, this network has the advantages of solv- ing equations by deep learning well as numerical methods such collocation method used to achieve better solution to the equations. LDNN presents a combination of a deep neural network and Legendre collocation method. In fact, our network consists of two networks which have nected consecutive to each other. The first network is a feed forward neural network which has an orthogonal Legendre layer. The second network includes operation nodes to create the desired computational model. In recent decades, numerical methods especially collocation method are pop- ular methods for solving differential equations. In the collocation method, first an approximation of the solution is expanded by using the sum of the basic functions. The basic functions consists of the orthogonal polynomials such as Legendre polynomials. Then this approximation placed in the differential equation. By considering the appropriate set of candidate points, an attempt is made to obtain the unknown coefficients of the basic functions so that the solution satisfies the equation in set of candidate points. The first network is applied to creat the approximation of the solution. This approximation can be known as the scattered data interpolation problem. The second network is used to obtain the desired equation so that the solution satisfies it. The structure of LDNN is described in detail at the following rest.
Consider that the first network has a MM-layer which defined as follows:
I E Rd, H1 = T(W(1) Ho + b());
Ho = x,
H. f(W()) +6 b(t). 25iSM-1.
HM WWMHHMM-1+b + (((M)
where Ho is the input layer with d dimension. H., 1 1 M --1 are hidden layers, == Lo, Ln," which L, are i-th degrees of Legendre orthogonal polynomials, Hi is an orthogonal layer. f is the hyperbolic tangent activation function or other commonly used activa- tion functions. (()) i= I,... .M are the weight parameters and 6(0), bias parameters. HM is the output layer. It is notable that the second network is applied to obtain the desired differential model. This aim possible by using operation nodes including integrals, derivatives, and etc. These nodes are applied to the output of the first network. Moreover, automatic differentiation (AD) (Baydin et al. 2017) and Legendre Gaussian integration (Shen et al. (01T) have been used in network computing obtain more accurate and fast calculations. How to train the network and set the parameters are also important points. Supervised learning method is used to train network. The cost function for setting parameters is defined as follows:
.M are the
CostFun min(y, Up) min(R....
where yi is an exact value of the model and yp is a predicted value of the LDNN. The definition of R., is explained in section The minimization of CostFun is obtained by performing Adam algorithm (Kingma & Ba 2015) and the L-BFGS method (Liu & Nocedal 1989) on mean squared emrrs of training data set.
2

---

Under review as a conference paper at ICLR 2021
2.1 LEGENDRE POLYNOMIALS
Legendre polynomials (Shen el al. 2011 are a main series of orthogonal polynomials which denoted by (1), are defined as:
(2n 26)!
L (n)
Σ
1-
22
2nL - E)!(n 20)!
Legendre polynomials are defined in [-1.1] domain and have the recurrence formula in the follow- ing form:
(n + 10++mI++100 = (2n + - 2 1,
To(n) 1, (n) =
(3)
Orthogonality relation for these polynomials is as follows:
[ Ln( Lim Lm((dddn y
(4)
where mm is delta Kronecker function and y 2n+I The weight function of them W(n) 1. Some following useful properties of Legendre polyno- mials are defined:
(.n(-n) Ln
(5)
[n)] 1, Vn E [-1, n n2 0,
(6)
Ln(+1) = (+1)",
(7)
(2n + 1)),,()) (1)) I' (n),, n n2
3 NONLINEAR DOUTERRA-RREEHOLL---AMMEEEES EQUATIONS AND LDNN
INTEGRAL
The general form of nonlinear Volterra-Fredholm- Hammmerseennnnnmeeettee integral equations (V-F-H-IEs) as follows:
y(अ) = g(x)+ + & S/ Ki(z, s) + / K2(2, s)2sss,(s)( x [0, (9)
where E1. & are fixed, (()), Ki (1 8) and (2) are given functions and yp (s, v()), p2(ss v()) are nonlinear functions. The aim find the proper y(r). order use the LDNN, reformulated Eq. (9) in the following form:
Rn, --(())+(((())+)) ' (((( uss)) + & S' (g(2,)))2(9. v(s))d, x E [0, 1].
(10)
y(x) is approximated by the first network of the LDNN.
g(x) ~ H.M
(11)
Furthermore, we applied Legendre-Gaun Gauss integration formula (Shen et al. 2011:
S M(X)dX Σ w))((,))
(12)
where {o the roots of Ln+1 and fwz 1 (1- (1-x3) x_X3xE. Here, should transfer
the [0, and [o, domains into the 1-1, 1] domain. It is possible by using the following transfor- mation:
2 f
1, to = 2s 1.

---

Under review as a conference paper at ICLR 2021
Consider
(1(2,8) (1(, (2, (2((,,)
we have
Rm = --)()) g(z) + Ei Zi (z, 2 (t) 1))dtt + &2
/ Zz(a, (tz 1))dtz.
(13)
2
by using Legendre- Gauss integration formula, the below form concluded:
Rm - -y(x) g(x) 2
Σ 11, (1, + 1)) j=0
Σ w2y 22 (1/5 1)). (14)
The second network of LDNN and its nodes makes Rm, The architecture of LDNN for solving
nonlinear V-F-H-IES is represented in Figure
g(x)
Lo (n)
1
L (n)
Fully Connected
Fully Connected
Fully Connected
y
Connected
Fully
(n)
y
Rm
X Z
2
X
52
LP (T)
1
1
Zz
H
H 2
H M
2
first network
second network
The architecture of LDNN for solving nonlinear V-F-H-IES. The first network approxi- mates the solution of IE y(r). This network has MM-layer and feed forward neural network is the structure of it. Hi is introduced (Legendre polynomials) as activation functions. Other layers have f. hyperbolic tangent activa- tion functions. The second network with the nodes makes the desired model and the output of it, is Rm (consider Eq. (14)). The outputs of LDNN are y(x) and Rm.
Figure 1:
orthogonal layer which consists of p neurons with {L;) ==o
NUMERICAL RESULTS
In order present the accuracy and performance of the LDNN for solving nonlinear V-F-H-IES and justify the efficiency of the proposed method, several examples are given. The convergence behavior of the LDNN reported by using the following parameters:
The exact value yt, the predicted value y/P and the absolute error (Error) in some points of test data are reported in various tables. The number of the train data m1, the number of Legendre quadrature points (N1, N2), the number of the test data m2, the structure of network M-layers, L.trein and T Irarnn and gest are calculated as follows:
test are shown in Table 1
may
L Lirain
111e 2/1122 = Σ (ur(2,) (x;))))))
men
Ilve ypl12 2 Σ L
(M)(2,)) - -b,(,,))
(15)
4

---

Under review as a conference paper at ICLR 2021
Table I: The LDNN parameters for all the experiments. The structure of M-Layers indicates by [d, NN N ..... (L 1]. This network has d dimension input layer, M 1 hidden layers with NT,(0), 2 S & M 1, neurons in each layer and one output which approximates the v(r). All the experiments have 4 hidden layers.
Experiment
M-Layers
M1
(N1, N2)
L2
Irran
m2
L
fest
Experiment 1 Experiment 2
10, 30, 20, 1
500
(50. -)
3.937867e- 09
100
4.015095c- 09
10,10,00 20, 20,
500
150,50)
7.156029e
100
7.537263e-09
Experiment 3
11 10,00,3,
20,
500
(50,50)
1.347132e 09
100
1.659349ee
Experiment 4
1,,10, 20, 10,
500
(50,50)
9.182442c
09
100
1.107755e 09
Table The exact value, the predicted value and the absolute error (Error) in several test points on [0, 1] domain for Experiment 1.
exact value (y/-
predicted value (yp)
Error
0.0
1.0
1.000000049
4.90000001e-00 08
0.2
1.22140276
402715
4.99999997e
0.4
1.4918247
1.49182494
2.40000000e 07
0.6
1.8221188
1.82211831
4.90000000c
0.8
2.22554093
2.2255-10981
5.09999998c 08
1.0
2.71828183
2.7182817
4.00000002e 08
The Tensorflow package of Python version 3.7.0. is applied for writing the code of all experiments. Adam algorithm is stoped when the number of iteration is up to 5000 and L-BFGS method is stoped when it converges. The figures are obtained on the data set.
4.1 EXPERIMENT
Suppose that we have the following model Yousefi & Razzaghi 2005):
1 e
/ y (s)ds, x E [0,
u(x) = ea
(16)
3
It has the exact solution y(x) _
Table represents the exact value, the predicted value and the absolute error (Error) several test points on [0. 1] domain. 50 points shifted Legendre quadrature points applied for training LDNN. The number of train data set is 500 and the number of test data set is 100. Figure shows the illustrated comparison between y₁ and yp.
250
1755
150
125
109
00
0.2
04
06
08
10
Figure 2: Results of Experiment 1. Exact solution ye(xx) = =e",, ppredicted solution Up(xr) by LDNN.

---

Under review as a conference paper at ICLR 2021
Table The exact value, the predicted value and the absolute error (Error) in several test points on [0, 1] domain for Experiment 2.
æ
exact value (z) cos(x))
predicted value (yp)
Error
0.0
1.0
1 1.000000559
5.90000000e 08
0.2
0.98006658
0.98006683
2.50000000e 07
0.4
0.92106099
0.92106083
1.50000000e 07
0.6
0.82533561
0.82533555
6.00000000e - 08
0.8
0.69670671
0.69670670
9.99999994e 09
1.0
0.54030231
0.54030237
6.00000001c-00 08
4.2
EXPERIMENT 2
Suppose that we have the following model (Razzaghi & Ordokhani 2002):
π(x) = 1 sin sin)(x) + /. K(z,s)y x [0,1]
(x)+
(17)
where
K(x, 8) 3sin(x 5), 0 0S5 5 S z 8S855 1.1.
(18)
0
has the exact solution y(²) = cos(x). The exact value, the predicted value and the absolute
error (Error) in several test points on [0, 1] domain are reported in Table 3 50 points of shifted
Legendre quadrature points are applied for training LDNN. The number of train data set is 500 and the number of test data is 100. Figure shows the illustrated comparison between yr and yp-
10
YpP
06
10
X
X
Figure 3: Results of Experiment 2. Exact solution (e(x) cos(Ir, predicted solution (p(x) by LDNN.
4.3 EXPERIMENT 3
Suppose that we have the following model (Babolian et al. 2007
π(अ) g(x) / (x (s- (-)s)
/ (ar + s)x(s)ds, 00, 1].
(19)
where
g(x) 30
x
2
(20)
3
has the exact solution y(z) = x² 2. Table illustrates the exact value, the predicted value and the absolute error (Error) in several test points [0, 1] domain. 50 points of shifted Legendre quadrature points are applied for training LDNN. The number train data set is 500 and the number of test data set is 100. Figure represented the comparison between yt and yp.

---

Under review as a conference paper at ICLR 2021
Table The exact value, the predicted value and the absolute error (Error) in several points on [0, 1] domain for Experiment 3.
exact value (f/- = x²² 2)
predicted value (3p)
Error
0.0
-2.0
-200000001
9.99999994e- 09
0.2
-1.96
-1.96000049
1.90000000e-000 07
0.4
-1.84
-1.840000009
8.99999986e 09
0.6
-1.64
-1.64000036
3.60000000c 07
0.8
1.36
-1.35999998
2.00000001c
1.0
-1.0
-0.99999999
1.0000000ee-00 08
-.0
42
-12
-14
-18
-18
-2.
X
X
Figure 4: Results of Experiment 3. Exact solution (x) = x²2 2, predicted solution (r(x) by
LDNN.
4.4
EXPERIMENT 4
Suppose that have the following model (Hadizadeh & Mohamadsohi 2005):
5
3
1
((x)
2x (s)dss x E [0.
(21)
has the exact solution y(x) = x² 2. The exact value, the predicted value and the absolute
error (Error) in several test points [0, 1] domain are reported in Table 5 50 points of shifted
Legendre quadrature points are applied for training LDNN. The number of train data set is 500 and
the number of test data is 100. Figure shows the illustrated comparison between y/ and yp-
12
1.10
0.
06
04
08
X
X
Figure 5: Results of Experiment 4. Exact solution (x) = x² + 1.
2. predicted solution (,(x) by
LDNN.
5 CONCLUSION
Legendre deep neural network (LDNN) is introduced this paper. LDNN and its application for solving nonlinear Volerra-Frcdholn-HHammeeses integral equations (V-F-H-IEs) proposed.

---

Under review as a conference paper at ICLR 2021
Table The exact value, the predicted value and the absolute error (Error) in several test points on [0, 1] domain for Experiment 4.
exact value (t/e = x +
predicted value (yp)
Error
0.0
0.5
5.0000004
4.00000000e:
0.2
0.54
0.54000001
9.999999944
09
0.4
0.66
0.66000002
2.00000000e
0.6
0.86
0.85999998
2.00000000e 08
0.8
1.14
1.13999999
9.99999994cc
1.0
1.5
1.49999999
9.99999994c- 09
LDNN includes two networks. The first network approximates the solution of nonlinear V-F-H-IE (xx) which has M-layers feed forward neural network structure. The first hidden layer of this has orthogonal layer consists of Legendre polynomials as activation functions. The last network adjusts the output of the sooner network to to a desired equation form. The better performance of network has been obtained by using Legendre Gaussian integration and automatic differentiation. Some experiments of nonlinear V-F-H-IEs are given to investigate the reliability and validity of LDNN. The results show that this network is efficient and has high accuracy.
REFERENCES
Esmail Babolian, F Fattahzadeh, and E Golpar Raboky. A chebyshev approximation for solving nonlinear integral equations of hammerstein type. Applied Mathematics and Computation, 189 (1):641-646, 2007.
Nathan Baker, Frank Alexander, Timo Bremer, Aric Hagberg, Yannis Kevrekidis, Habib Najm, Man- ish Parashar, Abani Patra, James Sethian, Stefan Wild, al. Workshop report basic research needs for scientific machine learning: Core technologies for artificial intelligence. Technical report, USDOE Office of Science (SC), Washington, DC (United States), 2019.
Atilim Günes Baydin, Barak Pearlmutter, Alexey Andreyevich Radul, and Jeffrey Mark Siskind. Automatic differentiation in machine learning: a survey. The Journal of Machine Learning Re- search,
2017.
Jens Berg and Kaj Nyström. A unified deep artificial neural network approach to partial differential equations in complex geometries. Neurocomputing, 317:28 41, 2018.
Claudio Canuto, M Yousuff Hussaini, Alfio Quarteroni, A Thomas Jr, et al. Spectral methods in fluid dynamics. Springer Science & Business Media, 2012.
Snehashish Chakraverty and Susmita Mall. Artificial neural networks for engineers and scientists: solving ordinary differential equations. CRC Press, 2017.
Richard Courant and David Hilbert. Methods of Mathematical Physics: Partial Differential Equa- fions. John Wiley Sons, 2008
Harold Thayer Davis. Introduction to nonlinear differential and integral equations. US Government Printing Office, 1961.
M Hadizadeh and M Mohamadsohi. Numerical solvability of a class of gral equations with noncompact kernels. Journal of Applied Mathematics, 2005, 2005.
inte-
Ersst Hairer, Christian Lubich, and Michel Roche. The numerical solution of differential-aggerrii systems by Runge-Kutte Kutta methods, volume 1409. Springer, 2006.
Hajimohammadi, F Baharifard, and K Parand. new numerical learning approach to solve general fikkeer-kan model. Engineering with Computers, pp. 1-17, 2020.
8

---

Under review as a conference paper at ICLR 2021
Jiequn Han, Arnulf Jentzen, and E Weinan. Solving high-dimennional partial differential equations using deep learning. Proceedings of the National Academy of Sciences, 115((4):8505-855)) -5100, 2018.
Boris Hanin. Universal function approximation by deep neural nets with bounded width and relu activations. Mathematics, 7(10):992, 2019.
Ji-Huan He and Xu-Hong Wu. Variational iteration method: new development and applications. Computers & Mathematics with Applications, (4(7-8):881-899) 2007.
Juncai He, Lin Li, Jinchao Xu, and Chunyue Zheng. Relu deep neural networks and linear finite 0007003973, 2018.
elements. arXiv preprint
Claes Johnson. Numerical solution of partial differential equations by the finite element method.
Courier Corporation, 2012.
Diederik P Kingma and Jimmy Ba. Adam (2014), method for stochastic optimization. In Pro- ceedings of the 3rd International Conference on Learning Representations (ICLR), arXiv preprint arXiv,, volume 1412, 2015.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
Guillaume Lample and François Charton. Deep learning for symbolic mathematics. arXiv preprint 192.011912. 2019.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521(7553):436-444 2015.
Shijun Liao. Homotopy analysis method nonlinear differential equations. Springer, 2012.
Dong Liu and Jorge Nocedal. On the limited memory bfgs method for large scale optimization. Mathematical programming. 45(1-3):503-5555 -288, 1989.
Zichao Long, Yiping Lu, Xianzhong Ma, and Bin Dong. Pde-net: Learning pdes from data. In International Conference on Machine Learning, pp. 3208-3216, 2018.
Lu Lu, Xuhui Meng, Zhiping Mao, and George Karniadakis. Deepxde: A deep learning library for solving differential equations. arXiv preprint arXiv: 2019.
Siamak Mehrkanoon and Johan AK Suykens. Learning solutions to partial differential equations using Is-svm. Neurocomputing, 159:105-116, 2015.
Xuhui Meng, Zhen Li, Dongkun Zhang, and George Em Karniadakis. Ppinn: Parareal physics- informed neural network for time-dependent pdes. Computer Methods in Applied Mechanics and Engineering, 370: 13250, 2020.
Alejandro Molina, Patrick Schramowski, and Kristian Kersting, Padé activation units: End-to- end learning flexible activation functions in deep networks. In International Conference on Learning Representations, 2019.
Guofei Pang. Lu Lu, and George Em Karniadakis. fpinns: Fractional physics-informed neural networks. SIAM Journal on Scientific Computing, 41(4):A2003-A2222 2019.
Maziar Raissi, Paris Perdikaris, and George E Karniadakis. Physics-infonen neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378:686-707, 2019.
Muhammad Asif Zaboor Raja, Jabran Mehmood, Zulqurnain Sabir, Kazemi Nasab, and Muham- mad Anwaar Manzar. Numerical solution of doubly singular nonlinear systems using neural networks- based integrated intelligent computing. Neural Computing and Applications, 31(3): 793-812, 2019.

---

Under review as a conference paper at ICLR 2021
Mohsen Razzaghi and Yadollah Ordokhani. A rationalized haar functions method for nonlinear marcholm-hammeerse integral equations. International journal of computer mathematics, 79(3): 333-343, 2002.
Jie Shen, Tao Tang, and Li-Lian Wang. Spectral methods: algorithms, analysis and applications, volume 41. Springer Science & Business Media, 2011.
Justin Sirignano and Konstantinos Spiliopoulos. Dgm: A deep learning algorithm for solving partial differential equations. Journal of computational physies, 375:1339-1333 1364,, 2018.
Gordon Smith. Numerical solution of partial differential equations: finite difference methods.
Oxford university press, 1985.
Shanshan Tang, Bo Li, and Haijun Yu. Chebnet: Efficient and stable constructions of deep neural networks with rectified power units using chebyshev approximations. arXiv preprint 1rX15:1911.05467. 2019.
Vladimir Vapnik. The nature of statistical learning theory. Springer science business media, 2013.
S Yousefi and Mohsen Razzaghi. Legendre wavelets method for the nonlinear foldeora-rrehhom integral equations. Mathematics and computers in simulation, 70(1):1-8, 2005.
10


Reviews:
[{"review_id": "hgQCu2YW43T", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposed the Legendre Deep Neural Network (LDNN) to solve Volterra–Fredholm–Hammerstein integral equations. Specifically, the network uses Legendre polynomials as the activation in the first layer and uses Gaussian quadrature to discretize the integral operator as a summation. The numerical examples are performed to verify the performance of LDNN. However, the method is not novel and the numerical examples are too simple.\n\nMajor comments:\n\n- The proposed method is the same as the physics-informed neural network (PINN) for solving integral PDEs proposed in https://arxiv.org/abs/1907.04502, but this is not mentioned in the paper. In fact, the method proposed in https://arxiv.org/abs/1907.04502 is more general than the method in this paper and can solve more types of PDEs.\n- The only difference is that here the first layer of the network uses Legendre polynomials as the activation, but there is not any evidence in the paper that if using Legendre polynomials would make a significant difference.\n- The integral equations solved in this paper are very simple. The equations only have one or two integrals, the problem is one dimension, and the solutions are quite smooth. In https://arxiv.org/abs/1907.04502, an integro-differential equation is solved. The same method has also been used to solve the 1D/2D/3D time-fractional/space-fractional/time-space-fractional PDE in a complicated geometry for both forward and inverse problems (https://doi.org/10.1137/18M1229845), which is much harder than the problems solved in this paper.", "extra_scores": {}, "extra_text": {}}, {"review_id": "fzSCunv1a_A", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "** Summary **\n\nThe authors present a neural network based method to solve a special class of integral equations. Their approach involves training a neural network with Legendre polynomial based activation functions to approximate the solution $y(x)$ for a given $x$. The network is trained in a supervised fashion to minimize a loss function with two term- (1) the $\\ell_2$ error between the true solution and $y(x)$ and (2) the residual of the given integral  equation when analysed at $x$. They show impressive numerical results for several instances of VFH-IEs with very low errors. The primary contributions as claimed by the authors are the use of Legendre polynomial based activation functions and creating a differentiable approximation for the integral equation by using Legendre polynomials and Quadrature methods to analyse the integral. \n\n*** Pros ***\n1. The numerical results show great efficiency and same to perform at par or better compared to other numerical methods reported in literature. \n2. The use of Legendre polynomials as an activation function to approximate the input domain is an interesting method to introduce well understood approximations from the numerical methods community.\n\n*** Cons ***\n1. The paper lacks comparisons and ablation studies to show how their model compares to simple supervised training. For example, a simple baseline comparison would be to train a network with similar number of parameters and standard loss functions in a supervised fashion and without the IE residual. This would allow us to analyse the efficacy of the various components of the proposed architecture better.\n\n2. How does the proposed method improve upon traditional numerical methods? I also would like to know the timing comparisons between traditional methods and the proposed neural network method.\n\n3. For Figures 2 and 3, the error between $y_{true}$ and $y_{pred}$ should be plotted as well.\n\nThe paper in its current form is not addressing how and why neural networks improve performance over the traditional methods and is also missing relevant comparisons and ablation studies. I will be willing to change my score if the authors add the required experimental results.", "extra_scores": {}, "extra_text": {}}, {"review_id": "SvxY61gw6H", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Overview: The paper proposes to solve the Volterra-Fredholm-Hammerstein integral equations using a form of neural networks: Legendre Deep Neural Networks which use Legendre polynomials in combination with a neural network to model the solution. The idea and direction of the paper is interesting, however both the theory and numerical results could be improved and specifically it should be made more clear *what* is the added benefit of using this LDNN model as opposed to any other method for solving the integral equations. I suggest some ideas for improvements below. \n\nIdeas for improvement, comments and questions: \n- In general I like the idea of combining collocation, Legendre polynomials and neural networks. \n- For readability (especially for people less familiar with these methods) it would be good to explain some concepts more clearly (see also next points).\n- Are the last parameters \\zeta_1 and \\zeta_2 fixed or trainable? The way it looks in the equation now is since you refer to it as the ‘second network’ that they are trainable, but in Section 3 you mention they are fixed. What are the trainable parameters of the third layer?\n- How are the roots $X_j$ and consequently $\\omega_j$ computed?\n- The numerical results look good and I see the potential of the method. \n- What is the intuition behind the FC layers after the Legendre layer? What would more hidden layers mean in terms of the approximation? Some numerical results on this would be useful.\n- Related to the above, how does the LDNN method compare to the collocation method and are W^2 similar to the role of the coefficients of the basis functions? \n- I think the case of *why* we would be using the LDNN method instead of other more classical approaches should be made a lot stronger for the paper to be of significant contribution. Right now it is not clear for me how the proposed method is better than classical methods. What is the relation to methods which, as you mention, “an attempt is made to obtain the unknown coefficients of the basis functions so that the solution satisfies the equation in a set of candidate points”. \n- A small comment but the English of the paper should be improved. \n- Maybe introduce Legendre polynomials prior to defining the network? \n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All three reviews for this paper were negative, and the authors did not provide rebuttals or comments on the reviews.  The main drawback of this work identified by the reviewers is that the empirical study is not sufficient (e.g., limited comparisons and ablation studies as well as low-dimensional examples).
Decision: Reject