# Novel FTLRNN with Gamma Memory for Short-Term and Long-Term Predictions of Chaotic Time Series

Authors: Sanjay L. Badjate; Sanjay V. Dudul
Journal: Applied Computational Intelligence and Soft Computing
 (2009)

---

## Abstract

Multistep ahead prediction of a chaotic time series is a difficult task that has attracted increasing interest in the recent years. The interest in this work is the development of nonlinear neural network models for the purpose of building multistep chaotic time series prediction. In the literature there is a wide range of different approaches but their success depends on the predicting performance of the individual methods. Also the most popular neural models are based on the statistical   and   traditional feed forward neural networks. But it is seen that this kind of neural model may present some disadvantages when long-term prediction is required. In this paper focused time-lagged recurrent neural network (FTLRNN) model with gamma memory is developed for different prediction horizons. It is observed that this predictor performs remarkably well for short-term predictions as well as medium-term predictions. For coupled partial differential equations generated chaotic time series such as Mackey Glass and Duffing, FTLRNN-based predictor performs consistently well for different depths of predictions ranging from short term to long term, with only slight deterioration after k is increased beyond 50. For real-world highly complex and nonstationary time series like Sunspots and Laser, though the proposed predictor does perform reasonably for short term and medium-term predictions, its prediction ability drops for long term ahead prediction. However, still this is the best possible prediction results considering the facts that these are nonstationary time series. As a matter of fact, no other NN configuration can match the performance of FTLRNN model. The authors experimented the performance of this FTLRNN model on predicting the dynamic behavior of typical Chaotic Mackey-Glass time series, Duffing time series, and two real-time chaotic time series such as monthly sunspots and laser. Static multi layer perceptron (MLP) model is also attempted and compared against the proposed model on the performance measures like mean squared error (MSE), Normalized mean squared error (NMSE), and Correlation Coefficient (r). The standard back-propagation algorithm with momentum term has been used for both the models.


---

## Body

## 1. Introduction

Predicting the future which has been the goal of many research activities in the last century is an important problem for human, arising from the fear of unknown phenomenon and calamities all around the infinitely large world with its many variables showing highly nonlinear and chaotic behavior. Chaotic time series have many applications in various fields of science, for example, astrophysics, fluid mechanics, medicine, stock market, weather, and are also useful in engineering such as speech coding [1], radar  modeling of electromagnetic wave propagation and scattering [2]. The chaotic interconnected complex dynamical systems in nature are characterized by high sensitivity to initial conditions which results in long-term unpredictability. The dynamical reconstruction seems to be extremely difficult, even in developing era of super computers, not because of computational complexity, but due to inaccessibility of perfect inputs and state variables. Many different methods have been developed to deal with chaotic time series   prediction. Among them neural networks occupy an important place being adequate model of the nonlinearity and nonstationarity.Inspired from the structure of the human brain and the way it is supposed to operate, neural networks are parallel computational systems capable of solving number of complex problems in such a diverse areas as pattern   recognition, computer vision, robotics, control and medical diagnosis, to name just few [3]. Neural networks are an effective tool to perform any nonlinear input output mappings and prediction problem [4]. Predicting a chaotic time series using a neural network is of particular interest [5]. Not only it is   an efficient method to reconstruct a dynamical system from an observed time series, but it also has many applications in engineering problems like radar noise cancellation [6], radar  [7] demodulation of chaotic secure communication systems [8], and spread spectrum/code division multiple access (CDMA) systems [9, 10]. It is already established that, under appropriate conditions, they are able to uniformly approximate any complex continuous function to any desired degree of accuracy [11]. Later, similar results were published independently in [12]. It is these fundamental results that allow us to employ neural network in time series prediction. Since neural networks’ models do not need any a priori assumption about the underlying statistical distribution of the series to be predicted, they are commonly classified as “data-driven” approach, to contrast them with the “model-driven” statistical methods. Neural networks that are the instruments in broad sense can learn the complex nonlinear mappings from the set of observations [13].  The static MLP network has gained an immense popularity from numerous practical application published over the past decade, there seems to be substantial evidence that multilayer perceptron indeed possesses an impressive ability [14]. There have been some theoretical results that try to explain the reasons for the success in [15, 16]. Most applications are based on feed-forward neural networks, such as the back-propagation (BP) network [17] and Radial basis function (RBF) network [18, 19]. It has also been shown that modeling capacity of feed-forward neural networks can be improved if the iteration of the network is incorporated into the learning process [20].Several methods with different performance measures have been attempted in the literature to predict the chaotic time series. It is has been predicted for Mackey-Glass   chaotic time series for short-term ahead prediction with a percentage error of 20% [21]. A new class of wavelet network was developed with a standard deviation of 0.0029 for short-term ahead prediction of Mackey-Glass chaotic time series and annual sunspots for 1 step ahead prediction [22]. By using recurrent predictor neural network  for monthly sunspots chaotic time series for 6 months ahead prediction with EPA (Prediction accuracy) equals 0.992 and ERMSE ( Root mean squared error) equals 4.419, for 10 months ahead prediction with  EPA of 0.980 and ERMSE of 7.050, for 15 months ahead prediction of EPAequals 0.9222 and ERMSE equals 13.658, and 20 months ahead prediction with EPA of 0.866 and ERMSE of 16.79323 [23]. Also by using radial basis function with orthogonal least square Fuzzy model for monthly sunspots with prediction error +6 to -4 and for Mackey-Glass chaotic time series with ERMSE of 0.0015 [24]. It is also attempted with Hybrid network for Mackey-Glass time series with iterative prediction and Normalized Mean Square Error NMSE of 0.053 [25]. By using Elman neural network for yearly sunspots for 1 year ahead prediction with ERMSE   equals 30.2931 and prediction accuracy of 0.9732 [26].From the scrupulous review of the related research work, it is noticed that no simple model is available for long-term prediction of chaotic time series so far. It is necessary to develop a simple model that is able to perform short-, medium- and long-term predictions of chaotic time series with reasonable accuracy. In view of the remarkable ability of neural network in learning from the instances, it can prove as a potential candidate with a view to design a versatile predictor (forecaster) for the chaotic time series. Hence in this paper a novel focused time-lagged recurrent neural network model with   gamma  memory filter is proposed as an intelligent tool for predicting the two non linear  differential equation Mackey-Glass and Duffing time series and two real-time monthly sunspots and Laser chaotic time series not only for short-term but long-term  prediction also  because they  acquire temporal processing ability through the realization of short-term memory and information about the preceding units, which is important when the long-term prediction is required. The Mackey-Glass  chaotic time series was first proposed as a model for  white blood cell production, the Duffing  chaotic time series  describes a specific nonlinear circuit or the hardening spring effect observed in many mechanical problems, monthly sunspots number is a good measure of solar activity which has a period of 11 years, so-called solar cycle. The solar activity has a measure effect on   earth, climate, space weather, satellites, and space missions, and a highly nonlinear laser time series. These chaotic time series are the good benchmark for the proposed model. The various parameters like number of hidden layers, number of processing elements in the hidden layer, step size, the   different learning rules, the various transfer functions like tanh, sigmoid, linear-tan-h, and linear sigmoid, different error normsL1,L2,L3,L4,L5, and L∞, the different memories TDNN, Laguarre and gamma filter, and  different combination of training and testing samples are exhaustively varied and experimented for obtaining the optimal values of performance measures as mentioned in the flow chart. The obtained results indicate the superior performance of estimated dynamic FTLRNN-based model with gamma memory over the MLPNN in various performance measures such as Mean Square Error (MSE), Normalized Mean Square Error (NMSE), and correlation coefficient  (r) on testing as well as training data set. The proposed network is attempted   for training up to 20 000 numbers of epochs for obtaining the improved values of performance measures. The experimentation process is demonstrated in flow chart of Figure 1. This paper is organized as follows in Section 2 the static MLP model is presented, and the learning procedure is explained. In Section 3 the proposed FTLRNN model is explained.  In Section 4 the performance measures and their importance are discussed. Section 5 explains about the significance of the benchmark chaotic time series. Section 6 explains the experimental procedure and analysis. Section 7 summarizes the evaluation results and analyses for the proposed model. Finally concluding remarks on empirical findings are provided in Section 8.Figure 1
Flow Chart.


## 2. Static NN-Based Model

Static Neural networks typically use Multilayer perceptron MLP as a backbone. They are layered feed-forward networks typically trained with static back propagation. MLP solid-based model has a solid foundation [27, 28]. The main reason for this is its ability to model simple as well as complex functional relationships. This has been proven through a number of practical applications [29]. In [11] it is shown that all continuous functions can be approximated to any desired accuracy, in terms of the uniform norm, with a network of one hidden layer of sigmoid or (hyperbolic tangent) hidden units and a layer of linear or tanh output unit to include in the hidden layer. The paper does not explain how many units to include in the hidden layer. This is discussed in [30], and a significant result is derived approximation capabilities of two layer perception networks when the function to be approximated shows certain smoothness. The biggest advantage of using MLP NN for approximation of mapping from input to the output of the system resides in its simplicity and the fact that it is well suited for online implementation. The objective of training is then to determine a mapping from a set of training data to the set of possible weights so that the network will produce predictions   y(t), which in some sense are close to the true outputs y(t). The prediction error approach is based on the introduction of measure of closeness in terms of   mean square error (MSE) criteria:(1)VN(θ,ZN)=12N∑t=1N[y(t)-y(t∣θ)⋀]T[y(t)-y(t∣θ)⋀]=12N∑t=1Nε2(t,θ).
The weights are then found as(2)θ̂=argminθVN(θ,ZN),
by some kind of iterative minimization scheme:
(3)θ(i+1)=θ(i)+μ(i)f(i),
where θ(i) specifies the current iterate (number “i’’), f(i) is the search direction, and μ(i) is the step size.When NN has been trained, the next step is to evaluate it. This is done by standard method in statistics called independent validation [31]. It is never a good idea to assess the generalization properties of an NN-based on training data alone. This method divides the available data sets into two sets, namely, training data set and testing data set. The training data set are next divided into two partitions: the first partition is used to update the weights in the network and the second partition is used to assess (or cross-validate) the training performance. The testing data set are then used to assess how the network has generalized. The learning and generalization ability of the estimated NN-based model is assessed on the basis of certain performance measures such as MSE, NMSE, and the regression ability of the NN by visual inspection of the correlation coefficient characteristics for different outputs of system under study.


## 3. FTLRNN Model

Time-lagged recurrent networks (TLRNs) are MLPs extended with short-term memory structures. Here, a “static” NN (e.g., MLP) is augmented with dynamic properties [14]. This, in turn, makes the network reactive to the temporal structure of information bearing signals. For an NN to be dynamic, it must be given memory. This memories may be classified into “short-term” and “long-term” memories. Long-term memory is built into an NN through supervised learning, whereby the information content of the training data set is stored (in part or in full) in the synaptic weights of the network [32]. However, if the task at hand has a temporal dimension, some form of “short-term” memory is needed to make the network dynamic. One simple way of building short-term memory into the structure of an NN is through the use of time delays, which can be applied at the input layer of the network (focused). A short-term memory structure transforms a sequence of samples into a point in the reconstruction space [33].  This memory structure is incorporated inside the learning machine. This means that instead of using a window over the input data, processing elements (PEs) created are dedicated to store either the history of the input signal or the PE activations.The input   PEs of an MLP are replaced with a tap delay line, which is followed by the MLPNN. This topology is called the focused time-delay NN (TDNN). The focused topology only includes the memory Kernels connected to the input layer. This way, only the past of the input is remembered. The delay line of the focused TDNN stores the past samples of the input. The combination of the tap delay line and the weights that connect the taps to the PEs of the first hidden layer is simply linear combiners followed by a static nonlinearity. Typically, a gamma short-term memory mechanism is combined with nonlinear PEs in restricted topologies called focused. Basically, the first layer of the focused TDNN is a filtering layer, with as many adaptive filters as PEs in the first hidden layer. The outputs of the linear combiners are passed through a nonlinearity (of the hidden-layer PE) and are then further processed by the subsequent layers of the MLP for system identification, where the goal is to find the weights that produce a network output that best matches the present output of the system by combining the information of the present and a predefined number of past samples (given by the size of the tap delay line) [32]. Size of the memory layer depends on the number of past samples that are needed to describe the input characteristics in time. This number depends on the characteristics of the input and the task. This focused TDNN can still be trained with static back propagation, provided that a desired signal is available at each time step. This is because the tap  delay line at the input layer does not have any free parameters. So the only adaptive parameters are in the static feed-forward path.The memory PE receives in general many inputsxi(n) and produces multiple outputs y=[y0(n),…,yD(n)]T, which are delayed versions of y0(n) the combined input,(4)yk(n)=g(yk-1(n)),y0(n)=∑j=1Pxj(n),
where g(·) is a delay function.These short-term memory structures can be studied by linear adaptive filter theory ifg(·) is a linear operator. It is important to emphasize that the memory PE is a short-term memory mechanism, to make clear the distinction from the network weights, which represent the long-term memory of the network.There are basically two types of memory mechanisms: memory by delay and memory by feedback. We seek to find the most general linear delay operator (special case of the Auto Regressive Moving Average model), where the memory tracesyK(n) would be recursively computed from the previous memory trace yK-1(n). This memory PE is the generalized feed-forward memory PE. It can be shown that the defining relationship for the generalized feed-forward memory PE is mentioned(5)gk(n)=g(n)*gk-1(n),k≥1,
where, * is the convolution operation, g(n) is a causal time function, and K is the tap index. Since this is a recursive equation, g0(n) should be assigned a value independently. This relationship means that the next memory trace is constructed from the previous memory trace by convolution with the same function g(n), the memory Kernel yet unspecified. Different choices of g(n) will provide different choices for the projection space axes. When we apply the input  x(n) to the generalized feed-forward memory PE, the tap signals yK(n) become(6)yK(n)=g(n)*yK-1(n),k≥1,
the convolution of y_{k-1}(n) with the memory Kernel. For k=0, we have(7)y0(n)=g0(n)*x(n),
where g0(n) may be specified separately. The projection x(n) of the input signal is obtained by linearly weighting the tap signals according to(8)x(n)=∑k=0Dwkyk(n).
The most obvious choice for the basis is to use the past samples of the input signal x(n) directly, that is, the Kth tap signal becomes yK(n)=x(n-K). This choice corresponds to(9)g(n)=δ(n-1).
In this case   g0(n) is also a delta function δ(n) (delta function operator used in the tap delay line). The memory depth is strictly controlled by D, that is, the memory traces store the past D samples of the input. The time delay NN uses exactly this choice of basis.The gamma memory PE attenuates the signals at each tap because it is a cascade of leaky integrators with the same time constant gamma model. The gamma memory PE is a special case of the generalized feed-forward memory PE, where(10)g(n)=μ(1-μ)n,n≥1,
and g0(n)=δ(n). The gamma memory is basically a cascade of low-pass filters with the same time constant 1-μ. The over all impulse response of the gamma memory is(11)gp(n)=(n-1p-1)μp(1-μ)n-p,n≥p,
where (:) is a binomial coefficient defined by
(12)(np)=n(n-1)⋯(n-p+1)p!.
For integer values of n and p, the overall impulse response gp(n) for varying p represents a discrete version of the integrand of the gamma function, hence the name of the memory.The gamma memory PE has a multiple pole that can be adaptively moved along the realZ-domain axis, that is, the gamma memory can implement only low-pass (0<μ<1) or high-pass (1<μ<2) transfer functions. The high-pass transfer function creates an extra ability to model fast-moving signals by alternating the signs of the samples in the gamma PE (the impulse response for 1<μ<2 has alternating signs). The depth in samples parameters (D) is used to compute the number of taps (T) contained within the memory structure (s) of the network.


## 4. Performance Measures

Three different types of statistical performance evaluation criteria were employed to evaluate the performance of these models  developed in this paper. These are as follows.MSE: the mean square error is given by:(13)MSE=∑j=0P∑i=0N(dij-yij)2N×P,
where P=number of output PEs, N=number of exemplars in the data set, yij=network output for exemplar i at PEj, and dij=desired output for exemplar i  at PEj.NMSE (normalized mean square error)
The normalized mean square error is defined by the following formula, whereP=Number of output PEs, N=Number of exemplars in data set,
(14)NMSE=P×N×MSE∑J=0P((N∑i=0Ndij2-(∑i=0Ndij)2)/N).MSE=Mean square error, dij=desired output for exemplar i at PEj ( jth element of PEs) Correlation Coefficient (r). The mean square error (MSE) can be used to determine how well the network output fits the desired output but it does not necessarily reflect whether the two sets of data move in the same direction. For instance by simply scaling the network output, we can change the MSE without changing the directionality of the data. The correlation coefficient solves this problem. By definition, the correlation coefficient between a network output x and a desired output d is
(15)r=∑i((xi-x̅)(di-d̅)/N)∑i((di-d̅)/N)×∑i((di-d̅)/N),x̅=1N∑i=1Nxi,d̅=1N.
The correlation coefficient is confined to the range [-1,1].


## 5. Benchmark Chaotic Time Series

In science, chaos is used as a synonym for irregular behavior, whose long-term prediction is essentially unpredictable. Chaotic differential equations exhibit not only irregular behavior but they are also unstable with respect to small perturbations of their initial condition. Consequently it is difficult to forecast the future of time series based on chaotic differential equations; they are   the good benchmark for a neural network design algorithm.


### 5.1. Mackey-Glass Time Series

The Mackey-Glass equation is time delay differential equation, which was first proposed as model of white blood cells production [34]. It is often used in practice as a benchmark set because of its nonlinear chaotic characteristics. Chaotic time series do not converge or diverge in time, and their trajectories are highly sensitive to initial conditions. Data are generated by using fourth order Runge-Kutta method .The equation is given by:(16)dxdt=ax(t-τ)1+x(t-τ)c-bx(t),
where a,b,c are constant coefficients and t is time delay, the coefficient we are selected: a=0.2,b=0.1,c=10,τ=17. The Mackey-Glass time series is shown in Figure 2.Figure 2
Mackey-Glass time series.


### 5.2. Duffing Time Series

Duffing time series describes a specific nonlinear circuit or the hardening spring effect observed in many mechanical problems [35]. The Duffing equation is time delay differential equation which is given as(17)dydt=y,dydt={FxCos(τ)-x3-bxy},dτdt=w,
where driving force F=7.5,Xo=initial position 1.0, damping constant (b=0.05), frequency w=1.0, Delay time=0.001. The chaotic time series is as shown in Figure 3.Figure 3
Duffing time series (first 500 samples).


### 5.3. Sunspot Time Series

A sunspot number is a good measure of solar activity which has a period of 11 years, so-called solar cycle. The solar activity has a measure effect on earth, climate, space weather, satellites, and space missions, thus is an important value to be predicted. But due to intrinsic complexity of time behavior and the lack of a quantitative   theoretical model, the prediction of solar cycle is very difficult. Many prediction techniques   have been examined on the yearly sunspots number time series as an indicator of solar activity. However, in more recent studies the international monthly sunspot time series, which has a better time resolution and accuracy, has been used. In particular, a nonlinear dynamics approach has been developed in [36], and prediction results are compared between several prediction techniques from both statistical and physical classes. There has been a lot of work on controversial issue of nonlinear characteristics of the solar activity [36–39]; several recent analyses have provided evidence for low-dimensional deterministic nonlinear chaotic behavior of the monthly smoothed sunspot time series [36–38] which has intense. The data considered   the monthly variations from January 1749 to December 2006. The total samples are 3096 considered and demonstrated in Figure 4. The series is normalized in the range   of -1 to +1. The monthly smoothed sunspot number time series is downloaded from the SIDC (World Data Center for the Sunspot Index) [40].Figure 4
Monthly Sunspot time series.


### 5.4. Laser Time Series

The laser data were recorded from a Far Infrared-laser in a chaotic state. The measurements were made on an 81.5-micron 14NH3 cw (FIR) laser, pumped optically by the P (16) line of an N2O laser, via the  vibrational   aQ(8, 7) NH3 transition. The basic laser setup can be found in [41]. The intensity data was recorded by an Le Croy   oscilloscope. It was made available worldwide during a time series prediction competition organized by Santa Fe Institute and a highly nonlinear data set, since then, it has been used in benchmark studies. The time series has 1000 samples points which has been rescaled to the range of [-1,1].  The time series is shown in Figure 5.Figure 5
Laser time series.


## 5.1. Mackey-Glass Time Series

The Mackey-Glass equation is time delay differential equation, which was first proposed as model of white blood cells production [34]. It is often used in practice as a benchmark set because of its nonlinear chaotic characteristics. Chaotic time series do not converge or diverge in time, and their trajectories are highly sensitive to initial conditions. Data are generated by using fourth order Runge-Kutta method .The equation is given by:(16)dxdt=ax(t-τ)1+x(t-τ)c-bx(t),
where a,b,c are constant coefficients and t is time delay, the coefficient we are selected: a=0.2,b=0.1,c=10,τ=17. The Mackey-Glass time series is shown in Figure 2.Figure 2
Mackey-Glass time series.


## 5.2. Duffing Time Series

Duffing time series describes a specific nonlinear circuit or the hardening spring effect observed in many mechanical problems [35]. The Duffing equation is time delay differential equation which is given as(17)dydt=y,dydt={FxCos(τ)-x3-bxy},dτdt=w,
where driving force F=7.5,Xo=initial position 1.0, damping constant (b=0.05), frequency w=1.0, Delay time=0.001. The chaotic time series is as shown in Figure 3.Figure 3
Duffing time series (first 500 samples).


## 5.3. Sunspot Time Series

A sunspot number is a good measure of solar activity which has a period of 11 years, so-called solar cycle. The solar activity has a measure effect on earth, climate, space weather, satellites, and space missions, thus is an important value to be predicted. But due to intrinsic complexity of time behavior and the lack of a quantitative   theoretical model, the prediction of solar cycle is very difficult. Many prediction techniques   have been examined on the yearly sunspots number time series as an indicator of solar activity. However, in more recent studies the international monthly sunspot time series, which has a better time resolution and accuracy, has been used. In particular, a nonlinear dynamics approach has been developed in [36], and prediction results are compared between several prediction techniques from both statistical and physical classes. There has been a lot of work on controversial issue of nonlinear characteristics of the solar activity [36–39]; several recent analyses have provided evidence for low-dimensional deterministic nonlinear chaotic behavior of the monthly smoothed sunspot time series [36–38] which has intense. The data considered   the monthly variations from January 1749 to December 2006. The total samples are 3096 considered and demonstrated in Figure 4. The series is normalized in the range   of -1 to +1. The monthly smoothed sunspot number time series is downloaded from the SIDC (World Data Center for the Sunspot Index) [40].Figure 4
Monthly Sunspot time series.


## 5.4. Laser Time Series

The laser data were recorded from a Far Infrared-laser in a chaotic state. The measurements were made on an 81.5-micron 14NH3 cw (FIR) laser, pumped optically by the P (16) line of an N2O laser, via the  vibrational   aQ(8, 7) NH3 transition. The basic laser setup can be found in [41]. The intensity data was recorded by an Le Croy   oscilloscope. It was made available worldwide during a time series prediction competition organized by Santa Fe Institute and a highly nonlinear data set, since then, it has been used in benchmark studies. The time series has 1000 samples points which has been rescaled to the range of [-1,1].  The time series is shown in Figure 5.Figure 5
Laser time series.


## 6. Experimental Results

The choice of the number of hidden layers and the number of hidden units in each hidden layers is critical [42]. It has been established that an MLPNN that has only one hidden   layer, with sufficient number of   neurons, acts as a universal approximators of nonlinear mappings [43]. The tradeoff between accuracy and complexity of the model should be resolved accurately [44]. In practice, it is very difficult to determine a sufficient number of neurons necessary to achieve the desired degree of approximation accuracy. Frequently the number of units in the hidden layer is determined by trial and error. To determine the weight values, one must have a set of examples of how the output should relate to the inputs. The task of determining the weights from these examples is called training or learning and is basically a conventional estimation problem. That is, the weights are estimated from the examples in such away that the network, according to metric, models the true relationship as accurately as possible. Since learning is a stochastic process, the learning curve may be drastically different from run to run. In order to compare the performance of a particular search methodology or the effects of different parameters have on a system, it is needed to obtain the average learning curve over the number of runs so that the randomness can be averaged out. An exhaustive and careful experimentation has been carried to determine the configuration of the static MLP model and the optimal proposed FTLRNN model with gamma   memory for short-term and long-term ahead predictions for the benchmark chaotic time series for 60% training, 15% cross-validation and 25% testing samples for the considered benchmark chaotic time series. It is found that the performance of the selected model is optimal for   38, 21, 15, and 43 neurons in the hidden layer with regard to the MSE, NMSE, and the correlation coefficient r performance for the testing data sets for Mackey-Glass,  Duffing, Monthly Sunspots, and Laser time series, respectively, and the different parameters  like transfer function, Learning rule, step size, and momentum values  are mentioned in Table 1 for Mackey-Glass and  Duffing  chaotic time series and in Table 2 for  monthly sunspots and Laser time series.Table 1
Optimal Parameters of FTLRNN for both time series.
Sr. no.ParametersHidden layer for MGHidden layer for DuffingOutput layer for MGOutput layer for Duffing1Number of PEs3821112Transfer functionTanhTanhTanhLin tanh3Learning ruleMomentumMomentumMomentumMomentum4Step size110.10.15Momentum0.80.80.80.8Table 2
Optimal Parameters of FTLRNN for both real-time series.
Sr. no.ParametersHidden layer for sun spot time seriesHidden layer for laser time seriesOutput layer for sunspot time seriesOutput layer for laser time series1Number of PEs1543112Transfer functionTanhTanhTanhTanh3Learning ruleMomentumMomentumMomentumMomentum4Step size110.10.15Momentum0.80.80.80.8When we attempted to increase the number of hidden layer and the number of processing element in the hidden layer, the performance of the model is not to seen to improve significantly. On the contrary it takes too long time for training because of complexity of the model. As there is single input and single output for the given system, the number of input and output Processing Elements is chosen as one. Now the NN models are trained three times with different weight initialization with 1000 iterations of the static back-propagation algorithm with momentum term for these two models.  All the possible variations for the model such as number of hidden layers, number of processing elements in each hidden layer, different transfer functions like tanh, linear tanh, sigmoid, linear sigmoid in output layer, the different supervised learning rules like momentum, conjugant gradient, and quick propagation are attempted for 10-step ahead prediction for Mackey-Glass, Duffing, and Laser time series, and 6 months ahead prediction for the monthly sunspots time series. The results are placed in Table3 for Mackey-Glass and Duffing time series and in Table 4 for real-time monthly sunspots, and Laser time series for different learning rules on testing data set.Table 3
Learning rules variations for the Mackey-Glass and Duffing time series.
Learning ruleMackey-Glass time seriesDuffing  time series(K=10 step ahead prediction)(K=10 step ahead prediction)MSENMSErMSENMSErConjugate gradient0.005450.093210.949650.123320.803650.77838Momentum0.0044750.079400.95960.003350.021850.98910Quick propagation0.005340.094860.94810.006600.043050.97821Table 4
Learning rules variations for the real sunspots and Laser time series.
Learning ruleReal sunspots  time seriesReal Laser time series(K=6 months  ahead prediction)(K=10 step ahead prediction)MSENMSErMSENMSErConjugate gradient0.006240.11350.94500.14500.5111400.79772Momentum0.005540.10080.955280.0095630.3370.8302Quick Propagation0.008660.157570.922560.017870.629950.64371Also the various error normsL1,L2,L3,L4,L5, and L∞, are varied, and FTLRNN model is trained and tested for the optimum transfer function. The results are obtained and placed in Table 5 for artificial Mackey-Glass and Duffing time series and in Table 6 for real-time monthly sunspots and Laser time series. It is clear from Table 5 that for L2 error norm and tanh transfer function the value of MSE is minimum and correlation coefficient r is maximum for the Mackey-Glass chaotic time series. For Duffing chaotic time series the optimal values of MSE, NMSE, and correlation coefficient r is obtained for linear tanh transfer function and L1 error norm can be seen from Table 5.Table 5
Error norms variations for the Mackey-Glass and Duffing time series for FTLRNN model on testing data set.
Error normsMackey-Glass time seriesDuffing  time series(K=10 step ahead prediction)(K=10 step ahead prediction)Transfer function  tanhTransfer function  linear tanhMSENMSErMSENMSErL10.004550.080770.945760.003350.021850.98910L20.004470.079400.95960.003890.027470.9831L30.004840.085950.93450.04880.027560.9821L40.005770.10240.938230.016240.027560.96108L50.006290.11170.93320.016560.093540.95725L∞0.005440.08870.949900.003410.019010.9810Table 6
Error norms variations for the real Sunspots time series and laser time series for FTLRNN model on testing data set.
Error normsMonthly sunspots  time seriesLaser   time series(K=6 months  ahead prediction)(K=10 step ahead prediction)Transfer function  tanhTransfer function  tanhMSENMSErMSENMSErL1  (tanh)0.020160.366730.859150.0061710.21740.9014L2 (tanh)0.005540.10080.955280.0095630.3370.8302L3 (tanh)0.005710.103870.948820.02790.984170.3038L4 (tanh)0.009560.173880.913750.24301.180400.2430L5 (tanh)0.016180.294390.854620.021680.76410.5541L∞ (tanh)0.006530.118890.924820.015360.53930.7887Similarly, for real-time monthly sunspots time series, the minimum value of MSE and maximum value correlation relation coefficientr are obtained for L2  error norm and tanh transfer function. For Laser time series the minimum value of MSE and maximum value of correlation coefficient r resulted for L1 error norm and  tanh transfer function.Then on these resulted optimal parameters the FTLRNN model is trained and tested for short-term (1, 5, 10) and long-term (20, 50, and 100) step ahead prediction. The FTLRNN structure is the MLP extended with the short-term memory structures. So these optimal parameters obtained for FTLRNN model are used for training and testing the MLPNN. Then on the same optimal parameters the static MLPNN model was attempted, and the performance measures like MSE, NMSE, and correlation coefficientr for the short-term (1, 5, and 10) and long-term (20, 50, and 100) step ahead prediction were obtained as stated in Table 7 for Mackey-Glass chaotic time series, in Table 8 for Duffing chaotic time series, in Table 9 for laser time series, and in Table 10 for monthly sunspots time series. It is obvious  from Tables 7, 8, 9, and 10 that for all the time series considered for short-term and long-term ahead predictions, the performance of this FTLRNN model is optimal on the test dataset for the following number of taps = 6, Tap Delay = 1, Trajectory Length = 50 with regards to the value of correlation coefficient r, MSE, and NMSE.Table 7
Performance of MLPNN and FTLRNN on testing data for Mackey-Glass time series.
MLPNN on testing dataFTLRNN on testing dataK (step)MSENMSErMSENMSEr10.0070.010.99330.0004910.009190.9954250.00280.0510.97450.002270.041450.97911100.00560.10080.94850.0044750.079400.9596200.01280.21590.88600.009440.158610.891822500.04110.5990.64060.0243820.356550.808491000.081800.99880.19710.053450.652670.62265Table 8
Performance of MLPNN and FTLRNN on testing data for Duffing time series.
MLPNN on testing dataFTLRNN on testing dataK (step)MSENMSErMSENMSEr10.087500.56660.660350.001130.009570.9954550.076340.555680.684340.002020.013150.99356100.083310.542750.679910.003350.021850.98910200.085800.629750.634540.006320.041500.97982500.100790.745450.545930.009270.065620.966631000.113970.850110.401910.001170.078700.96000Table 9
Performance of MLPNN and FTLRNN for testing data set for Laser time series.
K (step)MLP neural networkFTLRNNMSENMSErMSENMSEr10.018150.6810.598180.002820.1050.9459250.025220.9180.356410.003720.1350.93962100.019570.6890.55870.009560.3370.9014200.030401.0090.112870.016350.5430.71872500.034991.0390.281460.034661.029710.41330Table 10
Performance of MLPNN and FTLRNN for testing data set for monthly sunspot time series.
K (month)MLP neural networkFTLRNNMSENMSErMSENMSEr10.002470.045330.975690.0022270.041620.9816360.012290.22340.89170.005540.10080.9528120.023870.431320.771840.016930.305990.8550180.039890.71740.57410.022660.40750.7977240.05380.96580.35840.030590.548990.71497Then training and testing samples are varied from 10% to 80% as training with the increments of 10% samples and 75% to 5% as testing on the proposed optimal FTLRNN model, the number of training and testing samples was as testing samples with the decrements of 10% keeping exemplars for Cross-Validation (CV) 15% constant. The performance measures were obtained and compared on testing and training datasets and to gauge the performance and robustness of the FTLRNN. The results are obtained and are placed in Table11 for 1-step ahead predictions, Table 12 for 5-step ahead prediction, Table 13 for 10-step ahead prediction, Table 14 for 20-step ahead prediction, Table 15 for 50-step ahead prediction and Table 16 for 100-step ahead prediction for  Mackey-Glass and Duffing chaotic time series.Table 11
ForK = 1, training and testing samples variation for FTLRNN on testing data set.
Time seriesMackey-Glass time seriesDuffing time seriesTraining exemplarsTesting exemplarsMSENMSErMSENMSEr10%75%0.0003150.006270.997070.002300.039720.9801520%65%0.0002470.0048540.997570.001010.007740.9961830%55%0.0004050.0077320.996280.001040.007070.9967140%45%0.0003710.006910.996630.001030.006550.9969250%35%0.0003710.0072830.99640.000870.005550.9973360%25%0.0004910.009190.995420.001130.009570.9954570%15%0.000780.013530.99330.000910.006090.9971280%05%0.0022170.033120.983370.004840.032710.98827Table 12
ForK = 5, training and testing  samples variation for FTLRNN  on testing data set.
Time seriesMackey-Glass time seriesDuffing time seriesTraining exemplarsTesting exemplarsMSENMSErMSENMSEr10%75%0.0005010.017790.991220.004650.085230.9597520%65%0.0010060.0195080.99020.001620.012360.9938830%55%0.0011960.022560.988750.001500.009440.9955240%45%0.001440.026530.986750.001390.008850.9957850%35%0.001670.03220.98380.001700.011540.9943560%25%0.002270.041450.979110.002020.013150.9935670%15%0.003710.062270.96840.001490.009990.9952480%05%0.010890.15020.92260.004350.029410.98926Table 13
ForK = 10, training and testing samples variation for FTLRNN on testing data set.
Time seriesMackey-Glass time seriesDuffing time seriesTraining exemplarsTesting exemplarsMSENMSErMSENMSEr10%75%0.001910.037330.98120.008040.154320.9240520%65%0.002080.039940.979850.003140.023950.9879630%55%0.022610.042050.978880.002020.013150.9935640%45%0.002660.048150.975810.002600.016530.9918550%35%0.034480.064830.9672190.002880.019360.9901460%25%0.0044750.079400.95960.003350.021850.9891070%15%0.007570.122380.93730.003060.020460.9900180%05%0.022040.277960.85370.006550.044290.97835Table 14
ForK = 20, training and testing samples variation for FTLRNN on testing data set.
Time seriesMackey-Glass time seriesDuffing time seriesTraining exemplarsTesting exemplarsMSENMSErMSENMSEr10%75%0.004070.077990.960280.013420.300480.8637420%65%0.004360.081540.958520.005030.038570.9813230%55%0.004690.084990.957210.007320.046500.9698240%45%0.005780.101490.9488020.004270.027250.9819250%35%0.007070.1273020.934910.005160.035180.9827060%25%0.009440.158610.918220.006320.041500.9798270%15%0.015350.231540.879380.004920.031540.9842080%05%0.043790.476880.740820.006620.044810.97861Table 15
ForK = 50, training and testing samples variation for FTLRNN on testing data set.
Time seriesMackey-Glass time seriesDuffing time seriesTraining exemplarsTesting exemplarsMSENMSErMSENMSEr10%75%0.010820.194560.898230.012900.790330.7102120%65%0.011690.203740.893090.010270.083180.9598430%55%0.013160.221150.884380.006580.042030.9790240%45%0.015290.247090.870320.007630.050340.9747050%35%0.018160.290490.845380.007810.054260.9727160%25%0.0243820.356550.808490.009270.065620.9666370%15%0.038290.488260.735430.007430.050200.9756780%05%0.10560.874280.525290.009530.064510.96811Table 16
ForK = 100, training and testing samples variation for FTLRNN on testing data set.
Time seriesMackey-Glass time seriesDuffing time seriesTraining exemplarsTesting exemplarsMSENMSErMSENMSEr10%75%0.0291610.476230.72940.013590.541490.8009220%65%0.028970.456010.745950.009270.072190.9633930%55%0.031820.481560.729380.001160.078100.96230040%45%0.034390.503980.715240.008750.056630.9719650%35%0.040270.547490.687810.008830.060620.9694760%25%0.053450.652670.622650.001170.078700.9600070%15%0.080170.812430.531080.010330.069350.9657080%05%0.210061.476540.248620.011410.077150.96069In similar way, for real-time monthly sunspots and Laser time series, the number of training and testing samples was varied from 10% to 80% as training with the increments of 10% samples and 75% to 5% as testing samples with the decrements of 10% keeping exemplars for Cross-Validation (CV) 15% constant. The performance measures were obtained and compared on testing and training data sets and to gauge the performance and robustness of the FTLRNN. The obtained results are placed in Tables17, 18, 19, 20, and 21 for monthly sunspots and Laser time series for the value K in K-step ahead prediction.Table 17
ForK = 1, training and testing samples variation for FTLRNN on testing data set.
Time seriesSunspot time seriesLaser time seriesK=1Training exemplarsTesting exemplarsMSENMSErMSENMSEr10%75%0.003200.087050.961210.007770.240760.8747520%65%0.002580.68190.966050.010420.307990.8482130%55%0.001880.046080.977910.010540.323610.8465740%45%0.002180.050920.979280.009910.391190.8105750%35%0.002730.057950.978370.001850.092330.9883660%25%0.002270.041620.981630.002820.105970.9459270%15%0.003360.074160.967990.000800.022930.9887380%05%0.001530.050690.975120.001840.042290.97872Table 18
ForK = 6, training and testing  samples variation for FTLRNN on testing data set for monthly sunspot time series and K = 5  for laser time series.
Time seriesSunspot time seriesK=6 MonthsLaser time seriesK=5Training exemplarsTesting exemplarsMSENMSErMSENMSEr10%75%0.007070.191970.917030.007820.240680.8743920%65%0.004510.118880.941530.008110.239690.8802730%55%0.004230.103080.947120.008080.248140.8750440%45%0.004380.101970.950810.006350.253070.8774850%35%0.006540.137440.942520.002670.128980.9451360%25%0.005540.10080.955280.003720.135580.9396270%15%0.004730.102590.953590.005340.148380.9271780%05%0.004990.157250.921950.015790.342970.83272Table 19
ForK = 12 , training and testing  samples variation for FTLRNN on testing data set for monthly sunspot time series and K = 10  for laser time series.
Time seriesSunspot time seriesK=12Laser time seriesK=10Training exemplarsTesting exemplarsMSENMSErMSENMSEr10%75%0.011870.321450.874830.015060.459400.7621020%65%0.011160.293070.849170.013100.383320.8058330%55%0.009670.234850.882210.001400.435070.7711340%45%0.007490.172990.912360.012210.481190.7587950%35%0.009090.190030.910500.004440.207090.8982660%25%0.016930.305990.85500.009560.3370.901470%15%0.008410.178930.920080.010470.282290.8730180%05%0.008710.261190.868360.026310.591650.71077Table 20
ForK = 18 , training and testing  samples variation for FTLRNN on testing data set for monthly sunspot time series and K = 20  for laser time series.
Time seriesSunspot time seriesK=18Laser time seriesK=20Training exemplarsTesting exemplarsMSENMSErMSENMSEr10%75%0.018040.487800.745350.026760.805290.5878820%65%0.018530.484280.752090.023610.687390.6720830%55%0.012870.311690.829910.025690.793760.5619840%45%0.014030.321870.830520.016590.662630.6548250%35%0.027920.579140.736850.007030.307050.8553960%25%0.022660.040750.79770.016350.5430.7187270%15%0.018850.393720.788040.035730.379920.3799280%05%0.007450.215300.897690.003230.592340.59234Table 21
ForK = 24, training and testing  samples variation for FTLRNN on testing data set for monthly sunspot time series and  for laser time series (K = 50).
Time seriesSunspot time seriesK=24Laser time seriesK=50Training exemplarsTesting ExemplarsMSENMSErMSENMSEr10%75%0.032460.377340.492240.040731.182150.1326420%65%0.036570.582970.436160.043101.23643-0.0098030%55%0.022800.528820.693730.030180.988840.2739340%45%0.018390.836220.776590.044251.87215-0.2227350%35%0.040660.419350.539730.025340.948450.2660160%25%0.030190.548990.714970.034661.029710.4133070%15%0.028340.951500.755280.034470.892150.4433080%05%0.023390.577340.80120.088390.965480.28132Next for the optimum values of performance measures obtained from the combinations data partition as  training and testing samples for the chaotic time series as mentioned in Tables11 to 16 for the Mackey-Glass chaotic time series for all the cases of  multistep ahead prediction and Duffing time series  all the step ahead prediction. Similarly, for the real-time monthly sunspots and Laser time series for all the multistep ahead prediction.Then for the optimal data partition of training and testing samples combination resulted for theK-step ahead prediction for the cases of K-value in K-step ahead prediction for all the considered time series. For those combinations, the number of epochs is varied from 2000 to 20000 in a step of 2000, and again the proposed FTLRNN model is trained to observe the more prominent values of performance measures for all the chaotic time series and for all the steps ahead prediction.


## 7. Discussion

It can be clearly observed   that  dynamic FTLRNN model  with gamma memory clearly outperforms the static MLP not only  for short-term prediction but also for long-term prediction for  testing data as well as  training data set. From the results of Table7, for Mackey-Glass chaotic time series, it is noticed that up to the 20-step ahead prediction the Performance measures values of MLP and dynamic FTLRNN are slightly deviating but for long 50- and 100-step ahead prediction the Performance measures values of MSE, NMSE, and correlation coefficient (r) for FTLRNN model are significantly improved as compared to the static MLP. For the Duffing time series from Table 8 it is observed that for short- and long-step ahead predictions the proposed dynamic FTLRNN with gamma memory filter clearly outperforms well as compared to the static MLP with regards to the performance metrics like MSE, NMSE, and correlation coefficient   r.Also for the real monthly sunspots time series, it is observed from Table9 that for the 1, 6, 12, months ahead predictions the performance metrics values of  MLP and dynamic FTLRNN are slightly deviating but for 18 and 24 months ahead prediction, the performance metrics values for FTLRNN are significantly  improved  as compared to the static MLP. For the Laser  time series from Table 10 it is observed that for short- and long-term ahead (K=1,5,10,20,50) prediction  the proposed dynamic FTLRNN with gamma memory filter clearly outperforms well as compared to the static MLP for the values of MSE, NMSE, and   correlation coefficient  r.Next for the resulted optimal parameters the FTLRNN model is trained for different training and testing samples combinations varying from 10% to 80% as a training sample and 75% to 5% as a testing sample and Keeping cross-validation samples constant equal to 15% for short-term and long-term step ahead predictions for finding robustness of model and obtaining the significant results of performance measures for training and testing samples combinations. Tables11 to 16 show the performance metrics values for one-step ahead to 100-step ahead predictions for different combination of training and testing samples for the equation generated Mackey-Glass and Duffing time series.Similarly for real-time series the training and testing samples are varied in combination as mentioned in Tables17 to 21 for one month ahead to twenty four months ahead predictions for monthly sunspot time series and   for one-step to fifty-step ahead predictions for Laser time series.For which training and testing samples the values of MSE NMSE are minimum, and correlation coefficient is closed to unity for short-term and long-term predictions for the chaotic time series for that training and testing samples the FTLRNN with gamma filter is trained for 2000 to20000 epochs in the steps of 2000 for obtaining more significant values and observing the network performance with regards to the performance measures. The results are plotted  in Figure 6 for 1- and 5-step ahead predictions, Figure 7 for 10- and 20-step ahead predictions and Figure 8 for 50- and 100-step ahead  predictions for  performance on MSE and NMSE due to epochs variation, and Figure 9  for performance on  regression due to epochs variation for 1-, 5-, 10-, 20-, 50- and 100-step ahead predictions  for Mackey-Glass chaotic time series.  It is observed that up to 50 steps ahead prediction the performance metrics are slightly deviating but for long-term 100-step ahead predictions above the values of 12 000 epochs training the performance measure like MSE and NMSE values are decreased, and the correlation coefficient r is substantially increased.Figure 6
Performance of epoch’s variation on Errors for 1 and 5 step for Mackey-Glass time series.Figure 7
Performance of epoch’s variation on Errors for 10 and 20 step for Mackey-Glass time series.Figure 8
Performance of epoch’s variation on Errors for 50 and 100 step for Mackey-Glass time series.Figure 9
Performance of epochs variation on Correlation Coefficientr  for 1, 5, 10, 20, 50, and 100 step for Mackey-Glass time series.Also it can be visually inspected closely from Figures10, 11, 12, and 13 that the output of the proposed FTLRNN model closely follows the desired output for 1-, 5-, 10-, 20-step ahead predictions.Figure 10
Desired output and network output for 1-step ahead Prediction for  Mackey-Glass time series.Figure 11
Desired output and FTLRNN output for 5-step ahead  prediction for Mackey-Glass time series.Figure 12
Desired output and FTLRNN output for 10-step ahead prediction for  Mackey-Glass time series.Figure 13
Desired output and FTLRNN output for 20-step ahead prediction for Mackey-Glass time series.From Figure14 and Figure 15, it is clear that, the output of the network is slightly deviating from the desired output for long 50-step and 100-step ahead predictions.Figure 14
Desired output and FTLRNN Output for Testing data  set 50-step ahead prediction for  Mackey-Glass time series.Figure 15
Desired output and FTLRNN Output for testing data 100-step ahead prediction for Mackey-Glass time series.For the Duffing time series the results are shown in Figure16 for 1- and 5-step ahead predictions, Figure 17 for 10- and 20-step ahead predictions and Figure 18 for 50- and 100-step ahead predictions for performance on MSE and NMSE due to epochs variation, and Figure 19 for Correlation coefficient for 1-, 5-, 10-, 20-, 50- and 100-step ahead predictions. It is observed that for short-term prediction and up to 50-step ahead predictions the results are not deviating much. But for 100-step ahead prediction, the performance measures values are improved well.Figure 16
Epochs variation performance on errors forK = 1 and K = 5 for Duffing time series.Figure 17
Epochs variation performance on errors forK = 10 and K = 20 for Duffing time series.Figure 18
Epochs variation performance on errors forK = 50 and K = 100 for Duffing time series.Figure 19
Performance of epochs variation on Correlation Coefficientr  for 1, 5, 10, 20, 50, and 100 step for Duffing time series.From the close inspection of Figure20 and Figure 21 for 1- and 5-step ahead predictions the output of the FTLRNN closely follows the desired output. From Figure 22 and Figure 23 for 10- and 20-step ahead predictions, up to the first 20 samples the FTLRNN output is slightly deviating and after that the actual output closely follows the desired output. Similarly from the Figure 24 and Figure 25 the FTLRNN output follows the desired output for long-term 50- and 100-step ahead predictions.Figure 20
Desired and FTLRNN Outputs for Testing data for 1-step ahead prediction for Duffing time series.Figure 21
Desired and FTLRNN Outputs for Testing data for 5-step ahead prediction for Duffing time series.Figure 22
Desired and FTLRNN Outputs for testing data for 10-step ahead prediction for Duffing time series.Figure 23
Desired and FTLRNN Outputs for testing data for 20-step ahead prediction  for Duffing time series.Figure 24
Desired and FTLRNN Outputs for Testing data for 50-step ahead prediction for Duffing time series.Figure 25
Desired and FTLRNN Outputs for Testing data for 100-step ahead prediction  for Duffing time series.Similarly for real monthly sunspots time series the results are plotted in Figure26 and Figure 27 for performance on MSE and NMSE, respectively, due to epochs variations for 1 and 6 months ahead prediction, Figure 28 and Figure 29 for due to epochs variations for 12, 18, and 24 months ahead prediction and Figure 30 for the correlation coefficient (r) for all the month ahead prediction. For short-term 1 and 6 months ahead prediction the performance values are   slightly   improved. Also for 12 and 18 months ahead prediction the results are slightly deviating but for 24 months ahead prediction for 10 000 epochs training the values of MSE , NMSE, and  Correlation  Coefficient  r  are  significantly improved that is, MSE of 0.00824 and NMSE of 0.23236,  respectively, and the value of  Correlation Coefficient r is  0.88460 where as for the 1000 epochs training the value of MSE and NMSE is 0.0339, 0.57734, and r is 0.8012.Figure 26
Epochs variation performance on MSE for 1 month and 6 months ahead prediction for sunspot time series.Figure 27
Epochs variation performance on NMSE for 1 month and 6 months ahead prediction for sunspot time series.Figure 28
Epochs variation performance on MSE for 12, 18, and 24 months  ahead prediction for sunspot time series.Figure 29
Epochs variation performance on NMSE for 12, 18, and 24 months ahead prediction for sunspot time series.Figure 30
Epochs variation performance on Correlation Coefficientr for 1, 6, 12, 18, and 24 months ahead prediction  for sunspot  time series.Also from the close inspection of  Figure31 , Figure 32 and Figure 33  for  short-term 1, 6, and 12 months ahead prediction the FTLRNN output  closely follows the desired output and from Figure 34 and Figure 35  for long-term 18 and 24 months  ahead prediction the FTLRNN output is slightly deviating from the desired output.Figure 31
Plot between Desired versus Actual FTLRNN outputs for 1 month ahead prediction for FTLRNN modelFigure 32
Plot between Desired versus FTLRNN Outputs for 6-month ahead prediction for FTLRNN Model.Figure 33
Plot between Desired versus Actual FTLRNN Outputs for 12 month ahead prediction for FTLRNN  Model.Figure 34
Plot between Desired versus Actual FTLRNN Outputs for 18 month ahead prediction for FTLRNN Model.Figure 35
Plot between Desired versus Actual FTLRNN Output for 24 months ahead prediction for FTLRNN Model.For Laser time series the results are plotted in Figure36 and Figure 37 for 1-, 5-, and 10-step ahead prediction and Figure 38 and Figure 39 for 20- and 50-step ahead prediction for performance on MSE and NMSE due to epochs variations, and Figure 40 for the performance on Correlation coefficient r due to epochs variation for all the step ahead predictions. It is observed that for short-step ahead prediction the performance metrics are slightly deviating but for 50-step ahead prediction, the error values MSE and NMSE and correlation coefficient r are improved substantially. For 1000 epochs training the values of MSE and NMSE are 0.03447 and 0.89215, and value of r is 0.44330 and when the number of epochs for training is increased to 20 000 the values of MSE and NMSE are 0.154 and 0.412, and the value of r is significantly improved to 0.789.Figure 36
Epochs variation performance on MSE for 1, 5, 10, step ahead prediction for laser time series.Figure 37
Epochs variation performance on NMSE for 1, 5, 10, step ahead prediction for laser time series.Figure 38
Epochs variation performance on  MSE for 20- and 50-step ahead  prediction for laser time series.Figure 39
Epochs  variation performance on NMSE for 20- and 50-Step  ahead  prediction for laser time series.Figure 40
Epochs variation performance on  Correlation Coefficientr  for 1-, 5-, 10-, 20-, and 50-Step  ahead predictions  for laser time series.Also from the close inspection Figures41, 42 and 43 for short-term 1-, 5-, and 10-step ahead prediction the FTLRNN output closely follows the desired output and from Figure 44 and Figure 45 for long-term 20- and 50-step ahead predictions, it is clear that the FTLRNN output is slightly deviating from the desired output.Figure 41
Plot between Desired versus Actual FTLRNN outputs for 1-step ahead predictions for FTLRNN Model.Figure 42
Plot between Desired versus Actual FTLRNN Outputs for  5-step  ahead prediction  for FTLRNN Model.Figure 43
Plot between Desired versus FTLRNN Outputs for 10-step ahead prediction for FTLRNN Model.Figure 44
Plot between Desired versus FTLRNN Output for 20-step   ahead prediction for FTLRNN Model.Figure 45
Plot between Desired versus Actual FTLRNN Output for 50-step ahead prediction for FTLRNN Model.


## 8. Conclusions

It is seen that focused time-lagged recurrent network with gamma memory is able to predict the differential equation generated Mackey-Glass and Duffing time series and real-world monthly sunspots and laser time series quite elegantly in comparison with the Multilayer perceptron (MLP). Static NN configuration such as MLPNN-based model fails to cope up with the underlying nonlinear dynamics of the all the time series for short-term and long-term ahead predictions.  It is observed that MSE NMSE of the proposed focused time-lagged recurrent neural network (FTLRNN) dynamic model for testing data set as well as for training data set are significant better than those of static MLP NN. In addition, it is also observed that the correlation coefficient of this model for testing and training exemplars is much higher than that of MLPNN for the short-term and long-term ahead predictions for the chaotic time series considered. The FTLRNN is trained for different combination of training samples and tested on testing data sets to find the robustness and sustainability of FTLRNN model for all the step ahead predictions for the differential   equation generated and real-time chaotic time series. Also for the proposed FTLRNN model, the output closely follows the desired output and learned the true trajectory for  short-term prediction. For long-term prediction the output is slightly deviating for the considered benchmark chaotic time series as discussed .It is inferred from the experiments that the FTLRNN model with gamma memory has learned the dynamics of   chaotic   time series quite well as compared to multilayer perceptron network for testing data set (data set not used for training) with reasonable accuracy. On the contrary, it is observed that static MLP   performs   poorly   bad, because on the one hand it yields much higher MSE and NMSE on testing data sets and on the other hand the correlation coefficient for testing data set is far less than unity.  Hence the focused time-lagged recurrent neural network with gamma memory filter has outperformed the static MLP-based neural network   better for short-term   and for long-term ahead   predictions. Then the number of epochs is varied from 2000 to 20 000 in the steps of 2000, and the network is trained   for finding out the performance of the model for epoch’s variation.



---

*Source: 364532-2009-07-16.xml*