# Parameter Estimation for Dynamical Systems Using a Deep Neural Network

Authors: Tamirat Temesgen Dufera; Yadeta Chimdessa Seboka; Carlos Fresneda Portillo
Journal: Applied Computational Intelligence and Soft Computing
 (2022)

---

## Abstract

The deep neural network (DNN) was applied for estimating a set of unknown parameters of a dynamical system whose measured data are given for a set of discrete time points. We developed a new vectorized algorithm that takes the number of unknowns (state variables) and number of parameters into consideration. The algorithm, first, trains the network to determine weights and biases. Next, the algorithm solves the systems of algebraic equations to estimate the parameters of the system. If the right hand side function of the system is smooth and the system have equal numbers of unknowns and parameters, the algorithm solves the algebraic equation at the discrete point where absolute error between the neural network solutions and the measured data is minimum. This improves the accuracy and reduces computational time. Several tests were carried out in linear and non-linear dynamical systems. Last, we showed that the DNN approach is more successful in terms of computational time as the number of hidden layers increases.


---

## Body

## 1. Introduction

Our environment is surrounded by phenomena that can often be modeled by dynamical systems. For instance, a researcher in mathematical biology [1], ecology [2, 3], or epidemiology [4] trying to understand and predict the interaction between different species may encounter a multidimensional dynamical systems with several parameters. Another example of applications of multidimensional dynamical systems are chemical reactions [5]. These also depend on certain parameters, such as the reaction rate or the equilibrium constant [6, 7].Further applications can be found in economy. For example, when analyzing the dynamics of a certain economic systems for predicting an unfavorable situation, such as a recession or a depression [8]. Understanding the dynamics of such cases enables planning ahead to reduce the impact of possible negative effects.The study of the dynamics of several continuous variables leads to the analysis of a system of ordinary differential equations [9, 10]. The coefficients of these systems usually depend on some parameters which need to be estimated so that the dynamical system is valid, i.e., it provides a reliable mathematical explanation for a certain phenomenon or predictions are reliable. We refer to the estimation of this set of parameters as calibrating the dynamical system.In most cases, estimating the parameters involved in dynamical systems is, in fact, a challenging optimization problem which needs special consideration since it is an iterative method, and at times, there can be issues with the convergence [11, 12]. Some of the standard approaches: the Gauss–Newton method [13], multiple shooting, recursive estimation [14], collocation methods [15], modified multiple shooting algorithm [16], cross-entropy approach [17], a generalized smoothing approach [18], or principal differential analysis [19]; etc. However, most of these methods suffer from one or more of the following issues: small convergence region, convergence to a local minimum instead of the absolute minimum, high computational cost, convergence highly dependent on the initial guesses, see, for example, [20, 21] and further references therein. Despite the variety of methods, this estimation problem remains still a well-known challenging problem.An alternative approach for calibrating dynamical systems is artificial neural networks (ANN). The beginning of ANN is often attributed to the research article by McCulloch [22]. By then, it was less popular due to the capacity of computational machines. Nevertheless, the fast development of computer science and technology in the last decade, has led to an exponential increase of the capacity of machines, for both: storing and processing data.An ANN is defined as “an information-processing system that has certain performance characteristics in common with biological neural networks” [23, 24]. A network is composed of several layers. The first layer is usually called the input layer, whereas the last one is referred as the output layer. Layers falling in between the input and output layers are called hidden layers. Furthermore, each layer have a set of neurons or units. Deep neural networks (DNN) are ANNs with more than one hidden layer [25, 26]. DNNs are widely applied in artificial intelligence. For instance: in computer vision, image processing, pattern recognition, and cybersecurity [27–29]. The successful performance of DNNs is owed to the fact that deep layers are able to capture more variances [30].ANNs and, in particular, DNNs, could potentially address some of the challenges of the aforementioned standard methods. One of these drawbacks is the need of a large training dataset to obtain a sufficiently accurate estimation of the parameters, which entails a high computational cost [31–33]. The ANN approach has been implemented to minimize this limitation. In particular, Dua [34, 35] proposed ANN methods for parameter estimation in systems of differential equations. However, to the best of the knowledge of the authors, we have not encountered in the literature a vectorized DNN algorithm which approximates the solution of a dynamical system with unknown parameters given a set of values of the solution in a set of time points, and thus, it is the main purpose of the paper. Additionally, we provide the following original results.(1)
We extended the algorithm from [36] for systems of differential equations to systems of differential equations with unknown parameters.(2)
We enhance the efficiency and accuracy of the algorithm in the case when the number of unknowns of the dynamical system coincides with the number of unknown parameters.(3)
We show that the DNN algorithm for this problem becomes faster in terms of computational time as the number of layers increase.For the calculation of gradients of the cost functions, we utilized the auto-differentiation technique supported in the code by the Autograd package [37] and for the optimization of the learning rule, we implemented the Adam method [38], which successfully addresses the local minimum problem present in the standard approaches.The paper is structured as follows: (2) mathematical formulation of the problem; (3) DNN model; (4) vectorized algorithm for parameter estimation; (5) numerical experiments; and (6) conclusions and further work.


## 2. Problem Formulation

Let us consider a dynamical system described byn ordinary differential equations involving m unknown parameters,(1)dutdt=ft,ut,p,t∈t0,tend⊂ℝ,ut0=u0,where ut=u1t,u2t,…,untT is a vector field with n components uit, i=1,…,n; p=p1,p2,…,pm∈ℝm is the vector containing the unknown parameters; u0∈ℝn is the initial condition and(2)ft,u,p:=f1t,u1,u2,…,un,p1,p2,…,pmf2t,u1,u2,…,un,p1,p2,…,pm⋮fnt,u1,u2,…,un,p1,p2,…,pm,is a given vector-valued function, not necessarily linear, with n components fit,u1,u2,…,un,p1,p2,…,pm, i=1,…,n. The measured data for the model equation (1) will be denoted by u∗t.Objective: Given measured data atN time points of (1), i.e., tk,u∗tk with k=1,2,…,N, we aim to develop a vectorized deep neural network algorithm that estimates the unknown parameters p and the solution ut for all t.


## 3. Deep Neural Network Model

We consider a deep neural network with a similar architecture as in [36] for non-parametric systems of ODEs depicted in Figure 1. In this network diagram, ti∈t0,tend with i∈1,2,…N represents N time points; akl is the state of the j-th neuron in layer l with l∈1,2,…,L, where L denotes the total number of layers; bl denotes the bias in layer l; and Wl:=wjkl denotes the weight matrix of layer l. Lastly, u^ji denotes the estimated solution of the unknown function ujti, evaluated at the time point ti.Figure 1
Schematic diagram of DNN.The architecture of the DNN is based on the following:(i)
An input layerl=0 consisting of a single neuron corresponding to the time point ti;(ii)
An output layerl=L with n output functions u^ji;(iii)
L−1 hidden layers with hl neurons in each layer, l=1,…L−1.Let us remark that the number of neurons in each hidden layer might not be the same. As a result, the weight matrixW has dimension hl×hl−1, and thus it might not be a square matrix.Moreover, the number of neurons in each hidden layer could be determined based on the performance of the model [36]. Here, by performance of the model, we mean the DNN architecture which gives the best approximation with a small number of iterations and reduced computational time.


### 3.1. Feed-Forward Propagation

The feed-forward network proceeds as follows. For each data pointti, the learning process starts by assigning a10=ti.Then, to obtainal given al−1, we require two steps. The first step is working out a provisional vector zl by multiplying al−1 by the weight matrix Wl, and then we add the bias vector bl, i.e.,(3)zjl≔∑k=1hl−1wjklakl−1+bjl.Alternatively, one can write equation (3) in matrix form as follows:(4)zl=Wlal−1+bl.The second and last step requires applying an appropriate choice of activation functionσl to zl to obtain al. Thus, the values for the next layer are expressed by the following:(5)al=σlzl=σlWlal−1+bl.In short, for each given time pointti, we will obtain a zl and an output al. To illustrate, the dependence from ti, we will add the superscript (i), i.e., ali=σlzli. Therefore, using vector and matrix notation, we can write T=t1,…,tN, Zl=zl1,…,zlNt and Al=al1,…,alN so that we can summarise (4) and (5) for all ti(6)A0=T,Zl=WlAl−1+bl,Al=σlZl,forl∈1,…,L.We will refer to the values of the output layerAL as the output of the network, i.e., NT,WL,bL:=AL or, in component form Njti,WL,bL:=πjAL:=AjLti,W,b, where πj denotes the j-th projection.


### 3.2. Parameter Estimation

In this section, we describe the DNN algorithm for estimating the parameters in the dynamical system (1).Let us suppose that we have measured data atN points of (1), i.e., tk,u∗tk with k=1,2,…,N. For each ti, i=1,…,N, we compute the network output, denoted by AjLti,W,b. Then, the trial solution satisfying the initial conditions is given by the following:(7)u^jti,W,b=u0j+ti−t0AjLti,W,b,j=1,…,n.The expression (7) can be written in the matrix form as follows:(8)u^T,W,b=u0+T−t01NT,W,b.We train the network to findW and b by minimizing the following cost function,(9)J1T,W,b=12∑i=1mu^ti,W,b−ui2.With the optimal learning parametersW∗ and b∗, the approximate solutions of the dynamical systems were obtained. In this paper, Adaptive Moment method (also called Adam method) was applied to update the learning parameters [39]. The updating rule is(10)Mwk=β1Mwk−1+1−β1∇J1Wk,Mbk=β1Mbk−1+1−β1∇J1bk,Vwk=β2Vwk−1+1−β2∇J1Wk2,Vbk=β2Vbk−1+1−β2∇J1bk2,M^wk=Mwk1−β1k,V^wk=Vwk1−β2k,M^bk=Mbk1−β1k,V^bk=Vbk1−β2k,Wk+1=Wk−ηV^wk+ϵM^wk,bk+1=bk−ηV^bk+ϵM^bk,where β1,β2∈0,1 are decay rates for the moment estimates, η is the learning rate, and Vw,Vb and Mw,Mb are the first and the second moment vectors, respectively, that are initialized to be zero. The values η=0.001 , β1=0.9,β2=0.999 and ϵ=10−8 are usually applied in the literature, see e.g., [39].After obtaining the optimal values of the learning parametersW∗ and b∗, we seek the values of parameter p by solving the following non-linear system of algebraic equations,(11)Fp:=du^dtti,W∗,b∗−fti.u^,p=0.When attempting to solve (11), we will distinguish two possible cases: (1) The number of equations equals to the number of parameters and (2) Otherwise.Case 1.
For the particular case in which the right hand side function of the dynamical system (1) has as many parameters as equations, i.e., n=m, the following proposition addresses the existence of a solution in this case. This proposition directly follows from the well-known inverse function theorem.Proposition 1.
If number of unknowns and number of parameters are equal, i.e.,n=m, Fp∈ℝn is C1 in a neighborhood of some initial parameters p0, and if the Jacobian ∂F/∂pp0 is non-singular, then equation (9) has unique solution in the neighborhood of p0 and at point t∗∈t0,tend.Proof.
Lett∗∈t0,tend, then C∗=du^/dtt∗,W∗,b∗ is constant vector, and(12)Fp=C∗−ft∗,u^,p=0,is a system of non-linear algebraic equations having n equations in n unknowns. Hence, the proposition holds by virtue of the inverse function theorem.Remark 1.
The properties ofF in the proposition is determined by the right hand side function f of the dynamical systems (1). Hence, we require the C1 smoothness of f in order to apply Proposition 1.Proposition 2.
The best estimation for the parameterp is obtained if t∗ is chosen according to Proposition 1 such that the sum of the absolute error between solutions of the neural network and the measured data is minimum. That is, choose t∗ such that(13)∑j=1nu^jt∗,W∗,b∗−ujt∗,is minimum.Proof.
Condition (13) ensures the best fit for the solution trajectories.Case 2.
The number of parameters does not equal to the number of equations of the dynamical system. In this case, we can solve the non-linear system (11) by minimizing the following objective function(14)J2p=12∑i=1mdu^dtti,W∗,b∗−fti,u^,p2.Remark 2.
Let us note, that the approach suggested in Case2 can also be applied to Case 1 but not the other way round.


## 3.1. Feed-Forward Propagation

The feed-forward network proceeds as follows. For each data pointti, the learning process starts by assigning a10=ti.Then, to obtainal given al−1, we require two steps. The first step is working out a provisional vector zl by multiplying al−1 by the weight matrix Wl, and then we add the bias vector bl, i.e.,(3)zjl≔∑k=1hl−1wjklakl−1+bjl.Alternatively, one can write equation (3) in matrix form as follows:(4)zl=Wlal−1+bl.The second and last step requires applying an appropriate choice of activation functionσl to zl to obtain al. Thus, the values for the next layer are expressed by the following:(5)al=σlzl=σlWlal−1+bl.In short, for each given time pointti, we will obtain a zl and an output al. To illustrate, the dependence from ti, we will add the superscript (i), i.e., ali=σlzli. Therefore, using vector and matrix notation, we can write T=t1,…,tN, Zl=zl1,…,zlNt and Al=al1,…,alN so that we can summarise (4) and (5) for all ti(6)A0=T,Zl=WlAl−1+bl,Al=σlZl,forl∈1,…,L.We will refer to the values of the output layerAL as the output of the network, i.e., NT,WL,bL:=AL or, in component form Njti,WL,bL:=πjAL:=AjLti,W,b, where πj denotes the j-th projection.


## 3.2. Parameter Estimation

In this section, we describe the DNN algorithm for estimating the parameters in the dynamical system (1).Let us suppose that we have measured data atN points of (1), i.e., tk,u∗tk with k=1,2,…,N. For each ti, i=1,…,N, we compute the network output, denoted by AjLti,W,b. Then, the trial solution satisfying the initial conditions is given by the following:(7)u^jti,W,b=u0j+ti−t0AjLti,W,b,j=1,…,n.The expression (7) can be written in the matrix form as follows:(8)u^T,W,b=u0+T−t01NT,W,b.We train the network to findW and b by minimizing the following cost function,(9)J1T,W,b=12∑i=1mu^ti,W,b−ui2.With the optimal learning parametersW∗ and b∗, the approximate solutions of the dynamical systems were obtained. In this paper, Adaptive Moment method (also called Adam method) was applied to update the learning parameters [39]. The updating rule is(10)Mwk=β1Mwk−1+1−β1∇J1Wk,Mbk=β1Mbk−1+1−β1∇J1bk,Vwk=β2Vwk−1+1−β2∇J1Wk2,Vbk=β2Vbk−1+1−β2∇J1bk2,M^wk=Mwk1−β1k,V^wk=Vwk1−β2k,M^bk=Mbk1−β1k,V^bk=Vbk1−β2k,Wk+1=Wk−ηV^wk+ϵM^wk,bk+1=bk−ηV^bk+ϵM^bk,where β1,β2∈0,1 are decay rates for the moment estimates, η is the learning rate, and Vw,Vb and Mw,Mb are the first and the second moment vectors, respectively, that are initialized to be zero. The values η=0.001 , β1=0.9,β2=0.999 and ϵ=10−8 are usually applied in the literature, see e.g., [39].After obtaining the optimal values of the learning parametersW∗ and b∗, we seek the values of parameter p by solving the following non-linear system of algebraic equations,(11)Fp:=du^dtti,W∗,b∗−fti.u^,p=0.When attempting to solve (11), we will distinguish two possible cases: (1) The number of equations equals to the number of parameters and (2) Otherwise.Case 1.
For the particular case in which the right hand side function of the dynamical system (1) has as many parameters as equations, i.e., n=m, the following proposition addresses the existence of a solution in this case. This proposition directly follows from the well-known inverse function theorem.Proposition 1.
If number of unknowns and number of parameters are equal, i.e.,n=m, Fp∈ℝn is C1 in a neighborhood of some initial parameters p0, and if the Jacobian ∂F/∂pp0 is non-singular, then equation (9) has unique solution in the neighborhood of p0 and at point t∗∈t0,tend.Proof.
Lett∗∈t0,tend, then C∗=du^/dtt∗,W∗,b∗ is constant vector, and(12)Fp=C∗−ft∗,u^,p=0,is a system of non-linear algebraic equations having n equations in n unknowns. Hence, the proposition holds by virtue of the inverse function theorem.Remark 1.
The properties ofF in the proposition is determined by the right hand side function f of the dynamical systems (1). Hence, we require the C1 smoothness of f in order to apply Proposition 1.Proposition 2.
The best estimation for the parameterp is obtained if t∗ is chosen according to Proposition 1 such that the sum of the absolute error between solutions of the neural network and the measured data is minimum. That is, choose t∗ such that(13)∑j=1nu^jt∗,W∗,b∗−ujt∗,is minimum.Proof.
Condition (13) ensures the best fit for the solution trajectories.Case 2.
The number of parameters does not equal to the number of equations of the dynamical system. In this case, we can solve the non-linear system (11) by minimizing the following objective function(14)J2p=12∑i=1mdu^dtti,W∗,b∗−fti,u^,p2.Remark 2.
Let us note, that the approach suggested in Case2 can also be applied to Case 1 but not the other way round.


## 4. Vectorized Algorithm for Parameter Estimation

In this section, we describe the DNN algorithm to estimate the parameters of the system of differential equations (1). It differs from the algorithm in [36]. The current algorithm follows supervised learning, since we have the target output. Additionally, it includes the case where the number of parameters equals the number of unknowns.(1)
Input data: Define the vectorT=t1,t2,…,tm of size 1×m.(2)
Define the deep neural network structure: Determine the numbers of layersL, input layer (having one unit), L−1 hidden layers (having nl units), and the output layer (having n units).(3)
Initialize the network parameters: Choose the weights to have small random entries, whereas the biases and the moment matrices in the Adam algorithm to have zero entries.(i)
W1 has n1×1 dimension,(ii)
Wl has nl×nl−1 dimension, bl has nl×1 dimension,(iii)
WL has n×nL−1 dimension, bL has n×1 dimension,(iv)
Mw and Vw have the same size as the corresponding W, and(v)
Mb and Vb have the same size as the corresponding b.(4)
Forward propagation:(i)
For the input layer start by assigning,A0=T.(ii)
For the hidden layers,1≤l≤L−1,(15)Zl=WlAl−1+bl,Al=σlZl,whereσl is the activation function corresponding to the lth hidden layer.(iii)
For the output layer,(16)ZL=WLAL−1+bL,AL=σLZL.(iv)
Assign the trial solution using equation (8).(5)
Compute the costJ1T,W,b and its gradient (9): Calculate the partial derivatives of the J1 function, given in (9), gradients with respect to T,W and b. To carry out these computations, we apply automatic differentiation techniques [37, 40].(6)
Update the learning parameters using the Adam method described in (10).(7)
Solve the non-linear system of algebraic equations (11) to obtain the parameter p. Recall that there are two possible scenarios.(i)
If the number of unknowns equals that of parameters i.e.,m=n, find t∗ which minimize the absolute error given by (13) and solve the algebraic equation given by (12).(ii)
Ifm≠n; solve the minimization problem (14).


## 5. Numerical Experiments

In this section, we apply the algorithm proposed in the previous Section4, implemented in Python, to four different benchmark problems taken from the literature. The choice of numbers of neurons, numbers of hidden layers, activation functions, and other network parameters are specified within each example. Three out of four of these problems have an analytical exact solution which is then used for validating the model.Example 1.
In this example, taken from [34, 35] and ([6], problem 1), we consider the following system of differential equations related to the mathematical modeling of a chain of irreversible chemical reactions(17)du1dt=−k1u1,u10=1,du2dt=k1u1−k2u2,u20=0.
This system consists of two unknown functions, two parameters. In this case, given the parametersk1,k2=5,1, the analytical expression of the solution is as follows:(18)u1t=e−5t,u2t=54e−5t−1+e4t.
Let us take discrete points from the analytical solution (18), see Table 1.
To solve this problem, we considered a neural architecture with one hidden layer ofh=40 neurons. Furthermore, sigmoid activation functions were applied to each unit and trained with 90000 epochs.
The results of applying the algorithm are shown in Figure2. On the one hand, Figure 2(a) proves the accuracy of the method as we cannot distinguish between the analytical and the approximated solution. On the other hand, Figure 2(b) shows that the absolute error is bounded by 0.00015.
The parametersk1 and k2 were worked out using both approaches, Case 1 and Case 2, (see Remark 2). For the approach described in Case 1, the corresponding system of algebraic equation (11) was solved at t∗=0.6. Table 2 shows the estimated parameters obtained through the approaches described in Case 1 and Case 2, which can be applied due to Remark 2. In this particular example, the approach from Case 1 proved to be more accurate. Moreover, the computational time of Step 7 in the algorithm was 2.5ms CPU time for the first approach, whereas for the second approach, it was 7.6ms.Table 1
Discrete points of the analytical solution (18).
tu1u20.01.0000000.0000000.10.6065310.3728830.20.3678790.5635640.30.2231300.6471100.40.1353350.6687310.50.0820850.6555570.60.0497870.6237810.70.0301970.5829850.80.0183160.5387670.90.0111090.4943261.00.0067380.451427Figure 2
Results of Example1. (a) Exact solution, ANN, and data. (b) Absolute error.
(a)(b)Table 2
Comparison of the two approaches for Example1.
KTrue valuesApproach 1Error of approach 1Approach 2Error of approach 2k15.05.00500.0054.98900.011k21.01.00060.0060.99740.003Example 2.
Dynamical system models a biomass transfer ([41], p. 531).(19)du1dt=−c1u1+c2u2,du2dt=−c2u2+c3u3,du3dt=−c3u3,with initial condition u0=0,0,1T. For c=1,3,5, the analytical solution is given by the following:(20)u1t=158e−5t−2e−3t+e−t,u2t=52−e−5t+e−3t,u3t=e−5t.Following a similar procedure as for the previous application example, we consider 11 points uniformly distributed in the interval from the analytical solution (20), see, Table 3.Table 3
Data generated for Example2.
tu1u2u30.00.0000000.0000001.0000000.10.0557470.3357190.6065310.20.1668500.4523300.3678790.30.2827670.4585990.2231300.40.3811250.4146470.1353350.50.4544160.3526130.0820850.60.5025020.2887800.0497870.70.5285060.2306480.0301970.80.5366410.1810060.0183160.90.5311270.1402410.0111091.00.5157060.1076230.006738In this example, we have three unknown functions and three system parameters. The neural architecture considered has one hidden layer withh=10 neurons, tanh activation functions for each unit. The network is trained with 71800 epochs and the results are shown in Figure 3. On the one hand, Figure 3 shows the estimated solutions using the DNN approach and the analytical solutions. As we can see, Figure 3(a) proves again the accuracy of the method. On the other hand, Figure 3(b) shows that the absolute error is bounded by 0.0002. The corresponding system of algebraic equation (11) was solved at t∗=0.4. Again, the parameter estimation was conducted using approaches from Case 1 and Case 2. The output is shown in Table 4. Last, the computational time of Step 7 in the algorithm was 2.9ms CPU time for the first approach and 10.5ms for the second approach.Figure 3
Plot for comparing the ANN solution and exact solution. (a) Exact solution, ANN and data. (b) Absolute error.
(a)(b)Table 4
Comparison of the two approaches for Example2.
cTrue valuesApproach 1Error of approach 1Approach 2Error of approach 2c111.00240.00240.98960.0103c233.00260.00262.98110.1890c355.01500.01504.96150.0385We conclude that in this example, the approach described in Case1 was again more successful.Example 3.
The next problem is taken from [20] and consists of a dynamical system with initial value condition containing three parameters p1,p2, and p3, see also, ([6], problem 7). The model occurs in several applications such as chemical kinetics, theoretical biology, ecology, etc.(21)du1dt=p1u1−p2u1u2,u10=1.0,du2dt=p2u1u2−p3u2,u20=0.3.
The data are taken from reference [20], as shown in Table 5.
It is to be noted that the number of unknowns and parameters do not coincide in this example. Hence, unique solvability for the corresponding algebraic system (11) is not guaranteed. For solving this ODE system, we considered a single layer neural network with 60 units in the hidden layers, we have chosen 100000 epochs and the activation functions in the hidden layer were tanh functions. Using the approach described in Case 2, we obtain p=p1,p2,p3=0.854,2.201,2.013. The proposed algorithm gives better fit compared to the results reported in [20]. This estimation of the parameters provides an accurate solution; see Figure 4.Table 5
Measurement taken at discrete points Example3.
tu1u20.01.00.30.51.10.351.01.30.41.51.10.52.00.90.52.50.70.43.00.50.33.50.60.254.00.70.254.50.80.35.01.00.35Figure 4
Neural network solution and the measured data for Example3.Example 4.
Consider a dynamical system with three state variables which models methanol-to-hydrocarbon process ([34], Example 5).(22)du1dt=−2k1−k1u2k2+k5u1+u2+k3+k4u1,du2dt=k1u1k2u1−u2k2+k5u1+u2+k3u1,du3dt=k1u1u2+k5u1k2+k5u1+u2+k4u1,with initial conditions, u0=1,0,0, and t∈0,1. We took the data from the reference ([34], Table 6).Table 6
Comparison of shallow vs. deep layers for Example2.
Layers and neuronsk1k2k3Time (sec)Epochs1, (40)1.00442.99364.945120200002, (40, 40)1.00513.01255.051519151073, (40, 40, 40)1.00442.99364.9451169800Using the second approach, we estimated the parametersk1,k2,k3,k4, and k5 for Example 4. Here, the sigmoid activation function with 90000 epochs and one hidden layer having 40 neurons was used. See Table 7 for comparison. Figure 5 shows how the solution trajectories nicely fitted to the data.Table 7
Comparison of the true and estimated parameters for Example4.
kTrue valuesApproach 2Error of approach 2k15.25.1990.001k21.21.2020.002k30.00.00450.0045k40.0−0.01740.0174k50.00.00560.0056Figure 5
Neural network solution and the measured data for Example4.Table8 shows that the results obtained in this paper are in agreement with those from the existing literature.Table 8
Comparison of of results obtained vs. reported in literature.
ExamplesTrue valuesReported in literatureThis workEpochsCPU time (s)Example1k1=55.083 [34]5.0059000079k2=10.981 [34]1.0006Example4k1=5.25.187 [35]5.1999000079k2=1.21.199 [35]1.202k3=000.00145k4=02.028×10−4−0.017k5=000.0056


### 5.1. Shallow vs. Deep Layers

Here, we looked at the effect of adding more layers in the network. Specifically, we compared the network architecture with one, two, and three hidden layers. We considered Examples1 and 2. In the three possible network structure, we take the same activation function, 1/1+e−2.5x and we fix the error tolerance of the cost function to be 10−6. Although further analysis is required, the experimental results from Tables 9 and 6 show that reasonable accuracy on the parameter estimation could be achieved by increasing the hidden layer with less numbers of epochs and computation time.Table 9
Comparison of shallow vs. deep layers for Example1.
Layers and neuronsk1k2Time (sec)Epochs1, (40)5.04600.997919213782, (40, 40)5.05490.999730243583, (40, 40, 40)4.91031.01292616021


## 5.1. Shallow vs. Deep Layers

Here, we looked at the effect of adding more layers in the network. Specifically, we compared the network architecture with one, two, and three hidden layers. We considered Examples1 and 2. In the three possible network structure, we take the same activation function, 1/1+e−2.5x and we fix the error tolerance of the cost function to be 10−6. Although further analysis is required, the experimental results from Tables 9 and 6 show that reasonable accuracy on the parameter estimation could be achieved by increasing the hidden layer with less numbers of epochs and computation time.Table 9
Comparison of shallow vs. deep layers for Example1.
Layers and neuronsk1k2Time (sec)Epochs1, (40)5.04600.997919213782, (40, 40)5.05490.999730243583, (40, 40, 40)4.91031.01292616021


## 6. Conclusions and Outlook

In this paper, we developed a vectorized deep neural network algorithm for estimating the unknown parameters in a dynamical system based on a system of ODEs as well as the solution functions of the system. This problem arises in different sciences as shown in the Numerical Experiments Section. We highlighted that there are two approaches applicable to problems where the number of parameters coincides with the number of ODEs in the system. In this case, the system of algebraic equation arising from the optimal learning parameters (weights and biases) is uniquely solved at a specific point provided that the right side function satisfies the conditions stated in Proposition1. Choosing t∗ significantly improves the accuracy of the parameter estimation.Last, the experimental result shows that, for deep neural network, reasonable accuracy of parameter estimation could be achieved for less number of epochs and hence less running time.For the future work, it is worth comparing the proposed method with the traditional methods in terms of accuracy, computational complexity, and robustness. Furthermore, it is worth also comparing with other neural network architectures. Moreover, the method is yet to be exploited, for instance, for solving delay differential equations, stochastic dynamical systems, or rather more complex problems such as integral equations.



---

*Source: 2014510-2022-04-27.xml*