An improved time series prediction by applying the layer-by-layer learning method to FIR neural networks

-The FIR neural network model was recently proposed for time series prediction and gave good results. However, the learning algorithm used for the FIR network is a kind of gradient descent method and hence inherits all the well-known problems of the method. Recently a new learning algorithm called the optimization layer by layer was proposed for the regular multilayer perceptron network, and showed a great improvement in the learning time as well as the performance of the network. In this paper we develop a new learning algorithm for the FIR neural network model by applying the idea of the optimization layer by layer to the model. The results of the experiment, using two popular time series prediction problems, show that the new algorithm is far better in learning time and more accurate in prediction performance than the original learning algorithm. © 1997 Elsevier Science Ltd. All rights reserved. KeywordsmFIR neural network, Time series prediction, Optimization layer-by-layer, Gradient descent learning algorithm, Nonlinear autoregression, Lorenz equation, Mackey-Glass equation. 1. I N T R O D U C T I O N Neural networks have been applied to many useful and practical problems and have demonstrated good results. Especially in some of the time series prediction problems they have recently shown better performance than statistical methods. Statistical methods and neural networks are two major approaches to time series prediction problems. According to the papers (Chakraborty, Mehrotra, Mohan, & Ranka, 1992; Tang, de Almeida, & Fishwick, 1991) which have reported some comparative analysis of these two approaches, a statistical method is model driven and has many parameters to be chosen before a real problem can be solved. On the other hand, a neural network is data driven and the model selection is easy, although it is not yet possible to systematically describe the complexity and nonlinearity of a given network. Further, it has been said that a neural network approach generally performs better for long-term prediction than a statistical approach. Acknowledgements: This study was supported in part by Korea Telecom under contract 95-55, by a grant 92-21-00-05 from Korea Science Foundation and by SERI under contract 94-32. Address correspondence to Sung-Yang Bang: Department of Computer Science and Engineering, Pohang University of Science and Technology, San 31 Hyoja-Dong, Nam-Gu Pohang, South Korea 790-784. Tel: +82-562-279-2917: Fax: +82-562-279-2299: e-mail: sybang @ vision.postech.ac.kr The goal of time series prediction can be stated succinctly as follows: given a sequence up to time N x(1), x(2), -.-, x(N), find the continuation x(N + 1), x(N + 2), -... The time series may arise from the sampling of a continuous time system and be either stochastic or deterministic in origin. The standard prediction method involves finding an underlying model which gives rise to the observed sequence. One of the most popular methods studied and used in the past is autoregression (AR) which fits the data: T x(t)---E a ( n ) x ( t n ) + e ( t ) = Y c ( t ) + e ( t ) . (1) n = l This AR model represents the prediction ~(t) as a weighted sum of the past time values, More modem technologies employ nonlinear prediction schemes. Some use the same principle as the autoregression but give the estimate k(t) as the output of a neural network N('): x(t) = ~c(t) + e(t) = N[x(t 1), ." , x(t T)] + e(t). (2) This nonlinear autoregression is motivated by Takens' theorem (Takens, 1981). The theorem implies that for a wide class of deterministic systems there exists a diffeomorphism (i.e. one-to-one differential mapping) between a given time series [x(t 1), ..., x(t T)] and the state of the underlying dynamic system. This implies that it is