# A New Hybrid Methodology for Nonlinear Time  Series Forecasting

Authors: Mehdi Khashei; Mehdi Bijari
Journal: Modelling and Simulation in Engineering
 (2011)

---

## Abstract

Artificial neural networks (ANNs) are flexible computing frameworks and universal approximators that can be applied to a wide range of forecasting problems with a high degree of accuracy. However, using ANNs to model linear problems have yielded mixed results, and hence; it is not wise to apply them blindly to any type of data. This is the reason that hybrid methodologies combining linear models such as ARIMA and nonlinear models such as ANNs have been proposed in the literature of time series forecasting. Despite of all advantages of the traditional methodologies for combining ARIMA and ANNs, they have some assumptions that will degenerate their performance if the opposite situation occurs. In this paper, a new methodology is proposed in order to combine the ANNs with ARIMA in order to overcome the limitations of traditional hybrid methodologies and yield more general and more accurate hybrid models. Empirical results with Canadian Lynx data set indicate that the proposed methodology can be a more effective way in order to combine linear and nonlinear models together than traditional hybrid methodologies. Therefore, it can be applied as an appropriate alternative methodology for hybridization in time series forecasting field, especially when higher forecasting accuracy is needed.


---

## Body

## 1. Introduction

Both artificial neural networks (ANNs) and autoregressive integrated moving average (ARIMA) models have achieved successes in their own linear or nonlinear domains. However, none of them is a universal model that is suitable for all circumstances. The approximation of ARIMA models to complex nonlinear problems as well as ANNs to model linear problems may be totally inappropriate and, also, in problems that consist of both linear and nonlinear correlation structures. Theoretical as well as empirical evidences in the literature suggest that, by using dissimilar models or models that disagree with each other strongly, the hybrid model will have lower generalization variance or error. In combined models, the aim is to reduce the risk of using an inappropriate model by combining several models to reduce the risk of failure and obtain results that are more accurate [1]. Typically, this is done because the underlying process cannot easily be determined. The motivation for using hybrid models comes from the assumption that either one cannot identify the true data-generating process or that a single model may not be totally sufficient to identify all the characteristics of the time series [2].Much effort has been devoted to develop and improve the hybrid time series models, since the early work of Reid [3] and Bates and Granger [4]. In pioneering work on combined forecasts, Bates and Granger showed that a linear combination of forecasts would give a smaller error variance than any of the individual methods. Since then, the studies on this topic have expanded dramatically. Combining linear and nonlinear models are one of the most popular and widely used hybrid models, which have been proposed and applied in order to overcome the limitations of each component and improve forecasting accuracy. Chen and Wang [5] constructed a combination model incorporating seasonal autoregressive integrated moving average (SARIMA) model and support vector machines (SVMs) for seasonal time series forecasting. Khashei et al. [6] presented a hybrid autoregressive integrated moving average and feedforward neural network to time series forecasting in incomplete data situations, using the fuzzy logic.Pai and Lin [7] proposed a hybrid method to exploit the unique strength of ARIMA models and support vector machines (SVMs) for stock prices forecasting. Yu et al. [8] proposed a novel nonlinear ensemble forecasting model integrating generalized linear autoregression (GLAR) with neural networks in order to obtain accurate prediction in foreign exchange market. Zhou and Hu [9] proposed a hybrid modeling and forecasting approach based on grey and Box-Jenkins autoregressive moving average models. Valenzuela et al. [10] presented a novel hybridization of intelligent techniques and ARIMA models for time series prediction. Aburto and Weber [11] presented a hybrid intelligent system combining autoregressive integrated moving average (ARIMA) models and neural networks for demand forecasting. Aladag et al. [12] proposed a new hybrid approach combining Elman's recurrent neural networks (ERNN) and ARIMA models for time series forecasting.In all of these methods, the Zhang’s hybrid methodology [13] is applied in order to combine the linear and nonlinear models together. This methodology includes three steps as follows. It must be first noted that, in the Zhang’s hybrid methodology, a time series is considered to be composed of a linear autocorrelation structure and a nonlinear component as follows:(1)yt=Lt+Nt,
where yt denotes original time series, Lt denotes the linear component, and Nt denotes the nonlinear component.(i) Modeling the Linear Component
In the first step, linear component is estimated by ARIMA model and residuals, which will contain only the nonlinear relationship, obtained from the ARIMA model as follows:(2)et=yt-L̂t
where L̂t is the forecasting value for time t of the time series yt estimated by ARIMA model. Zhang [13] claims that any ARIMA model can be selected for the data, as this does not affect the final forecast accuracy [12].(ii) Modeling the Nonlinear Component
In the second step of the Zhang’s hybrid methodology, a multilayer perceptron is used to model the ARIMA residuals. Zhang [13] claims that, by modeling ARIMA residuals by multilayer perceptron, nonlinear relationships can be discovered. With P input nodes, the ANN model for the residuals will be
(3)et=f(et-1,…,et-P)+εt=w0+∑j=1Qwjg(w0j+∑i=1Pwi,jet-i)+εt,
where f is a nonlinear function determined by the multilayer perceptron and εt is the random error. Zhang [13] claims that if the model f is not an appropriate one, the error term is not necessarily random. Therefore, the correct model identification is critical.(iii)  Combining the Linear and Nonlinear Components
In the third step, the linear and nonlinear forecasting values obtained from (2) and (3) denoted as L̂tandN̂t respectively, are combined together as follows:
(4)ẑt=L̂t+N̂t.In the Zhang’s hybrid methodology are jointly used the linear ARIMA and the nonlinear multilayer perceptrons models in order to capture different forms of relationship in the time series data. The motivation of the Zhang’s hybrid methodology comes from the following perspectives. First, it is often difficult in practice to determine whether a time series under study is generated from a linear or nonlinear underlying process; thus, the problem of model selection can be eased by combining linear ARIMA and nonlinear ANN models. Second, real-world time series are rarely pure linear or nonlinear and often contain both linear and nonlinear patterns, where neither ARIMA nor ANN models alone can be adequate for modeling in such cases; hence, the problem of modeling the combined linear and nonlinear autocorrelation structures in time series can be solved by combining linear ARIMA and nonlinear ANN models. Third, it is almost universally agreed in the forecasting literature that no single model is the best in every situation, due to the fact that a real-world problem is often complex in nature and any single model may not be able to capture different patterns equally well. Therefore, the chance in order to capture different patterns in the data can be increased by combining different models.Although the Zhang’s hybrid methodology has been shown to be successful for single models in several studies, it has some assumptions [14] that will degenerate its performance if the opposite situation occurs; therefore, it may be inadequate in some specific situations. In this methodology, it is assumed that (1) the existing linear and nonlinear patterns in a time series can be separately modeled, (2) the residuals from the linear model contain only the nonlinear relationship, and (3) the relationship between the linear and nonlinear components is additive. However, these assumptions may underestimate the relationship between the components and degrade performance if the opposite situation occurs [15], for example, if we cannot separately model the linear and nonlinear patterns in a time series, the residuals of the linear component do not comprise valid nonlinear patterns, or there is not any additive association between the linear and nonlinear elements and the relationship is different (e.g., multiplicative).In this paper, a new methodology is proposed in order to combine the linear and nonlinear models for better modeling both linear and nonlinear components, simultaneously. This methodology has no aforementioned assumptions of the Zhang’s hybrid methodology. The proposed methodology consists of two stages. In the first stage, a linear model such as autoregressive integrated moving average (ARIMA), generalized linear autoregression (GLAR), and so forth is used in order to identify and magnify the existing linear component in data. In the second stage, a nonlinear model such as multilayer perceptron (MLP), support vector machine (SVM), Elman's recurrent neural network (ERNN), and so forth. is used in order to model the preprocessed data. The rest of the paper is organized as follows. In the next section, the formulation of the proposed hybrid methodology is introduced. In Section3, in order to show the appropriateness and effectiveness of the proposed methodology, it is applied to the Zhang’s model and Aladag’s model, respectively, presented in [12, 13] and its performance is evaluated in comparison with the Zhang’s hybrid methodology. Conclusions will be the final section of the paper.


## 2. The Proposed Hybrid Methodology

In our proposed methodology, a time series is considered as function of a linear and a nonlinear component. Thus,(5)yt=f(Lt,Nt),
where Lt denotes the linear component and Nt denotes the nonlinear component. In the first stage, a linear model such as autoregressive integrated moving average (ARIMA), generalized linear autoregression (GLAR), and so forth is used in order to model the linear component. The residuals from the first stage will contain the nonlinear relationship, in which linear model is not able to model it, and maybe linear relationship [14]. Thus,(6)Lt=L̂t+et,
where L̂t is the forecasting value for time t estimated by the linear model and etis the residual at time t from the linear model. The forecasted values and residuals of linear modeling are the results of first stage that are used in next stage. In addition, the linear patterns are magnified by linear model in order to be applied in second stage.In second stage, a nonlinear model such as multilayer perceptron (MLP), support vector machine (SVM), Elman's recurrent neural network (ERNN), and so forth is used in order to model the nonlinear and probable linear relationships existing in residuals of linear modeling and original data. Thus,(7)Nt1=f1(et-1,…,et-n),Nt2=f2(zt-1,…zt-m),Nt=f(Nt1,Nt2),
where f1,f2, andfare the nonlinear functions determined by the nonlinear model. Nandm are integers and often referred to as orders of the model. Thus, the combined forecast will be as follows:(8)yt=f(Nt1,L̂t,Nt2)=f(et-1,…,et-n1,L̂t,zt-1,…,zt-m1),
where f are the nonlinear functions determined by the nonlinear model. n1≤n andm1≤m are integers that are determined in design process of the nonlinear model. It must be noted that anyone of the aforementioned variables ei(i=t-1,…,t-n), L̂t, and zj(j=t-1,…,t-m) or set of them {ei(i=t-1,…,t-n)} or {zi(i=t-1,…,t-m)} may be deleted in design process of the nonlinear model. This may be related to the underlying data-generating process and the existing linear and nonlinear structures in data. For example, if data only consist of pure linear structure, then {ei(i=t-1,…,t-n)} variables will be probably deleted against other of those variables. In contrast, if data only consist of pure nonlinear structures, then L̂t variable will be probably deleted against other of those variables.


## 3. Application of the Hybrid Methodology

In this section, the proposed hybrid methodology is applied to the Zhang’s model and Aladag’s model for Canadian lynx data forecasting. This data consists of the set of annual numbers of lynx trappings in the Mackenzie River District of North-West Canada for the period from 1821 to 1934. The Canada lynx data, which is plotted in Figure1, was also examined by Kajitani et al. [16], beyond the other various studies in the time series literature with a focus on the nonlinear modeling [17]. Following other studies, the logarithms (to the base 10) of the data is used in the analysis. The proposed hybrid methodology is first applied to the Zhang’s model as follows. It must be noted that all calculations are performed on a personal computer that has a “Pentium(R) 4 CPU 3.2 GHz, RAM 1.0 GB” processor and Windows XP.Figure 1
Canadian lynx data series (1821–1934).Stage I.
Using the Eviews package software, the established model is a autoregressive model of order twelve,AR(12), which has also been used by many researchers [4].Stage II.
Using pruning algorithms [18] in MATLAB7 package software, the best-fitted network which is selected is composed of eight inputs, three hidden and one output neurons (in abbreviated form, N(8-3-1)). The structure of the best-fitted network is shown in Figure 2. The performance measures of the proposed methodology for the Zhang’s process modeling for Canadian lynx data are given in Table 1. The estimated values of the proposed methodology for the Zhang’s process modeling for Canadian lynx data set are plotted in Figure 3. In addition, the estimated values of ARIMA, ANN, and the proposed methodology for the Zhang’s process modeling for the last 14 observations (test data) are, respectively, plotted in Figures 4, 5, and 6. In addition, a feedforward neural network, which is composed of seven inputs, five hidden and one output neurons (N(7-5-1)), has been designed to Canadian lynx data set forecast, as also employed by Zhang [13]. Similar to the above, the proposed methodology is applied to the Aladag’s process modeling for Canadian lynx data forecasting. The performance measures of this case are also given in Table 1.Table 1
Comparison of the performance of the proposed methodology.
ModelMSEAutoregressive integrated moving average (ARIMA)0.020Feedforward neural network (FNN)0.020Zhang’s hybrid model0.017Proposed methodology (Zhang’s process)0.009Aladag’s hybrid model0.009Proposed methodology (Aladag’s process)0.006Figure 2
Structure of the best-fitted network (lynx data),N(8-3-1).Figure 3
Results obtained from the proposed model for Canadian lynx data set.Figure 4
ARIMA model prediction of lynx data (test sample).Figure 5
ANN model prediction of lynx data (test sample).Figure 6
Proposed model prediction of lynx data (test sample).Numerical results from Table1 show that both Zhang’s hybrid model and Aladag’s hybrid model by using the Zhang’s methodology outperform the autoregressive integrated moving average (ARIMA) and feedforward neural network (FNN) models for Canadian lynx data. Moreover, Aladag et al. [12] can make a more accurate hybrid model, by replacing the feedforward neural network (FNN) model by Elman's recurrent neural network (ERNN) for Canadian lynx data. However, according to the obtained results, it can be seen that using the proposed hybrid methodology instead of using the Zhang’s hybrid methodology for combination improves the accuracy in both Zhang’s model and Aladag’s model. Our proposed methodology indicates 47.06% and 33.34% improvement over the Zhang’s hybrid methodology in Zhang’s hybrid model and Aladag’s hybrid model in MSE, respectively. In additional, the computation times of the constructed models based on the proposed methodology are as fast as models that are constructed based on the Zhang’s methodology. These evidences clearly show that the performance of the proposed methodology is better than Zhang’s methodology, for hybridization of linear and nonlinear models together for forecasting.


## 4. Conclusions

The Zhang hybrid methodology that decomposes a time series into its linear and nonlinear form is one of the most popular hybrid methodologies, which have been shown to be successful for single models. This methodology are jointly uses the linear and nonlinear models in order to capture different forms of relationship in the time series data. However, some researchers believe that assumptions that are considered in constructing process of this hybrid methodology can degenerate its performance if the opposite situation occurs. In addition, it cannot be generally guaranteed that the performance of the designed models based on this methodology will not be worse than their nonlinear component. These assumptions are as follows.(1)
This methodology assumes that the linear and nonlinear patterns of a time series can be separately modeled by different models.(2)
This methodology assumes that the relationship between the linear and nonlinear components is additive.(3)
This methodology assumes that the residuals from the linear model will contain only the nonlinear relationship.In this paper, we propose a new methodology in order to combine the linear and nonlinear models that has no above-mentioned assumptions of traditional hybrid linear and nonlinear models in order to yield the more general and the more accurate forecasting model. Empirical results with Canadian Lynx data set indicate that our proposed methodology can improve the performance of the designed hybrid models by the Zhang’s methodology. These results confirm this hypothesis that the aforementioned assumptions considered in constructing process of the traditional hybrid linear and nonlinear methodologies will degenerate their performance if the opposite situation occurs. In addition, in contrast to the traditional hybrid linear and nonlinear methodologies, we can generally guarantee that the performance of the designed models by proposed methodology will not be worse than either of the components used in isolation, so that it can be applied as an appropriate methodology for combination linear and nonlinear models for time series forecasting.



---

*Source: 379121-2011-08-02.xml*