Using Synthetic Data to Improve the Long-range Forecasting of Time Series Data

1 INTRODUCTION . Short-range forecasting of time series data has been able to provide some useful information , but its scope of application is limited ( Chatfield , 2000 ; Granger & Newbold , 2014 ) . In most applications , long-range forecasting of time series data is preferred as it allows more time for early intervention and planning opportunities ( Alvarez et al. , 2010 ; Azad et al. , 2014 ) . For example , long-range forecasting of patient ’ s vital signs effectively gives clinicians more time to take action and may reduce the occurrence of potential adverse events ( Luo , 2015 ; Choi & et al , 2016 ) . However , a critical issue affecting the long-range forecasting is that the predictive performance ( e.g. , N -step ahead ) becomes worse as N grows . One practical approach to address this problem is to use synthetic data to shorten the prediction horizon N . For example , in iterative forecasting ( Marcellino et al. , 2006 ; Hamzaçebi et al. , 2009 ) , the previous predictions are used together with the original data as new inputs to evaluate the next prediction . However , the synthetic data ( i.e. , previous predictions ) generated in this recursive and supervised manner is more susceptible to error propagation ( Sorjamaa & et al , 2007 ; Weigend & Gershenfeld , 2018 ) . Overall , the quality of synthetic data is the key to improving long-range forecasting . In recent years , the success of Generative Adversarial Network ( GAN ) ( Goodfellow & et al , 2014 ) in replicating real-world content has inspired numerous GAN based architectures ( Frid-Adar et al. , 2018a ; b ; Shin et al. , 2018 ; Bowles et al. , 2018 ) , in order to generate synthetic data for different purposes ( e.g. , improve classification accuracy ) . However , the utilization of synthetic data to improve long-range forecasting remains unexplored . In this paper , we contribute to the area of long-range forecasting as follows . 1 . We augment the existing conditional Wasserstein GAN with Gradient Penalty with a mean squared error term . This new architecture , called cWGAN-GEP , aims to generate accurate synthetic data which preserves the temporal dynamics between the conditioning input and generated data . 2 . We develop a long-range forecasting method called Generative Forecasting ( GenF ) which consists of three key components : ( i ) a cWGAN-GEP based generator , to generate synthetic data for next few time steps . ( ii ) a predictor which makes long-range predictions based on generated and observed data . ( iii ) an information theoretic clustering ( ITC ) algorithm to better train the cWGAN-GEP based generator and the predictor . 3 . We conduct experiments on three public time series datasets and our results demonstrate that GenF significantly outperforms a diverse range of state-of-the-art benchmarks and classical approaches . In most cases , we find improvements of at least 10 % over the benchmark methods . To the best of our knowledge , our work is the first to propose using synthetic data generated by a GAN based architecture to improve long-range forecasting and the proposed cWGAN-GEP is not explored in any of the previous work . 2 BACKGROUND ON TIME SERIES FORECASTING . In time series forecasting , we use an observation window of historical data to predict the future value ( see Fig . 1 ) . Let M denote the observation window length and N denote the prediction horizon . For those methods that contain generative models , let L denote the synthetic data window length . Let Xi ∈ RK be the observation at the ith time step , where K is the number of features . In Fig . 1 , the prediction horizon is N , indicating that we plan to make predictions N time steps ahead , i.e. , at time step t+N ( e.g. , the circled entry in Fig . 1 ) . Next , we discuss related work in Section 2.1 and shortlist several benchmark methods for comparison in Section 2.2 . 2.1 RELATED WORK . Early methods which use neural networks to perform long-range forecasting include ( Nguyen & Chan , 2004 ; Goswami & Srividya , 1996 ; Al-Saba & El-Amin , 1999 ) . For example , Nguyen & Chan ( 2004 ) propose a group of neural networks to make predictions at different time steps . Several recent works ( Yu et al. , 2017 ; Lai et al. , 2018 ; Qiu et al. , 2020 ; Bouktif et al. , 2018 ; Barsoum et al. , 2018 ) attempt to improve the long-range forecasting by proposing new architectures . For example , Yu et al . ( 2017 ) propose a Long Short-Term Memory ( LSTM ) ( Hochreiter & Schmidhuber , 1997 ) based Tensor-Train Recurrent Neural Network as a module for sequence-to-sequence ( Sutskever et al. , 2014 ) framework , called TLSTM . Moreover , Lai et al . ( 2018 ) propose a Long- and Short-term Time-series network ( LSTNet ) which combines LSTM with autoregressive models to minimize a customized loss function inspired by support vector regression ( SVR ) ( Cortes & Vapnik , 1995 ) . Another recent work ( Cheng et al. , 2020 ) proposes Multi-Level Construal Neural Network ( MLCNN ) which attempts to improve the predictive performance by fusing forecasting information of different future time . One more interesting work to mention is ( Ahmed et al. , 2007 ) , which combines Gaussian processes ( Rasmussen , 2003 ) and neural networks to improve the forecasting performance . Overall , all these methods can be classified into two main classes : direct forecasting and iterative forecasting ( Hamzaçebi et al. , 2009 ; Bontempi et al. , 2012 ) . In direct forecasting , a group of models are trained to directly make predictions at different values of N . The advantage is that all models are independent of each other and hence , models do not interfere with each other . However , its performance tends to becomes worse with N due to the lack of long-range dependencies . In iterative forecasting , the model is trained to make predictions for the next time step ( t + 1 ) only . Then , the same model will be used over and over again and previous predictions are used together with the past observations to make predictions for the next time step . This process is recursively repeated N times to make predictions N time steps ahead . The previous predictions can be considered as synthetic data ( with synthetic data window length L = N − 1 ) to shorten the effective prediction horizon . However , the synthetic data generated in this supervised and recursive manner is more susceptible to error propagation , meaning that a small error in the current prediction becomes larger error in subsequent predictions ( Taieb et al. , 2012 ; Sorjamaa & et al , 2007 ) . 2.2 BENCHMARK METHODS . The taxonomy in Section 2.1 suggests that most existing methods can be classified as either direct or iterative forecasting . This motivates us to use both direct and iterative forecasting as benchmark methods . As mentioned in Section 2.1 , some classical models such as LSTM , Autoregressive Integrated Moving Average ( ARIMA ) ( Box & Pierce , 1970 ) , SVR , and Gaussian Process Regression ( GPR ) ( Rasmussen , 2003 ) are used to form the core of many state-of-the-art methods . Hence , we use these four classical models to implement both direct and iterative forecasting . In addition , three strong baselines described in Section 2.1 , i.e. , TLSTM , LSTNet , and MLCNN , are also selected for comparison as they are reported to provide outstanding long-range forecasting performance in the literature ( Yu et al. , 2017 ; Lai et al. , 2018 ; Cheng et al. , 2020 ) . Moreover , the source code released for these works is well organized to ensure correct implementation . 3 GENERATIVE FORECASTING . In this section , we introduce the idea of GenF ( see Fig . 2 ) and describe its three components : cWGAN-GEP based synthetic data generator , LSTM based predictor and the ITC algorithm . 3.1 IDEA OF GENERATIVE FORECASTING . To improve long-range forecasting , we develop an approach called Generative Forecasting ( GenF ) which consists of a cWGAN-GEP based generator , to generate synthetic data for next few time steps , and a predictor , to make long-range predictions based on observed and generated data . Assuming we use past 3 observations { Xt , Xt−1 , Xt−2 } and make predictions at t+ 10 , we can express GenF as Generate synthetic data X ′t+1 = g ( Xt , Xt−1 , Xt−2 , θg ) , ( 1 ) X ′t+2 = g ( X ′ t+1 , Xt , Xt−1 , θg ) , ( 2 ) X ′t+3 = g ( X ′ t+2 , X ′ t+1 , Xt , θg ) , ( 3 ) Make a long-range prediction : X̄t+10 = f10 ( X ′t+3 , X ′t+2 , X ′t+1 , θ10 ) , ( 4 ) where g denotes the cWGAN-GEP based generator with parameter θg and f10 is the predictor with parameter θ10 . Let X ′t+1 , X ′t+2 and X ′t+3 denote the generated synthetic data for next three time steps and X̄10 denote the prediction at t+ 10 . Note that the value of the synthetic data window length L depends on the quality of the synthetic data and ( 1 ) - ( 3 ) is an example of L = 3 . It can be seen that the generator g generates synthetic data via the strategy of iterative forecasting ( see ( 1 ) - ( 3 ) ) and the predictor makes a prediction via the strategy of direct forecasting ( see ( 4 ) ) . We note that GenF does not belong to either direct or iterative forecasting . The key difference is that GenF leverages synthetic data to shorten the effective prediction horizon while direct forecasting does not leverage any synthetic data . Moreover , unlike iterative forecasting where the synthetic data window length is L = N − 1 , the synthetic data window length L of GenF is much smaller than the prediction horizon N ( L = 3 and N = 10 in the example of ( 1 ) - ( 4 ) ) , so that it can prevent the propagation of error at an early stage . In such a manner , we argue that GenF attempts to balance direct forecasting and iterative forecasting . 3.2 GENERATIVE FORECASTING : THE CWGAN-GEP BASED SYNTHETIC DATA GENERATOR . GAN ( Goodfellow & et al , 2014 ) and its related works have demonstrated promising results in many generative tasks ( Yi et al. , 2017 ; Xu et al. , 2017 ; Haidar & Rezagholizadeh , 2019 ; Zhang et al. , 2017 ; Dong et al. , 2018 ) . The first GAN applied to time series data is C-RNN-GAN ( Mogren , 2016 ) which takes random noise as input and uses LSTM as both the generator and discriminator . One follow-up work , RCGAN ( Esteban et al. , 2017 ) , makes use of the conditioning information to generate synthetic medical data . Along the way , many works have explored generating synthetic data in a diverse range of domains ( e.g. , sensor data ( Alzantot et al. , 2017 ) , biosignal ( Haradal et al. , 2018 ) ) . We note that all these synthetic data are generated for some purpose ( Dai & Le , 2015 ; Larsen et al. , 2015 ; Dumoulin et al. , 2016 ; Srivastava et al. , 2015 ; Yoon et al. , 2019 ) . For example , ( Frid-Adar et al. , 2018a ) makes use of synthetic data augmentation to improve the classification results , and ( Bowles et al. , 2018 ) mixes the synthetic data with training data to enhance the model ’ s predictive ability . However , to the best of our knowledge , none of them have considered using the synthetic data generated by the GAN based architecture , to improve long-range forecasting . We fill in this gap and propose a conditional Wasserstein GAN with Gradient and Error Penalty ( Arjovsky et al. , 2017 ; Mirza & Osindero , 2014 ; Gulrajani & et al , 2017 ) to generator synthetic data which can be used to improve long-range forecasting . The structure of cWGAN-GEP is shown in Fig . 2 ( right ) . As can be seen , the training data G is used as a condition to direct the generation of synthetic data ( Mirza & Osindero , 2014 ) . Then , the synthetic data ( X ′t+1 ) will be concatenated with the training data G at the latest time step , to be evaluated by the discriminator . The network configurations of cWGAN-GEP are given in Table 1 ( left ) . The loss function is summarized as Loss = E X̃∼Pg [ D ( X̃ ) ] − E X∼Pr [ D ( X ) ] + λE X̂∼P X̂ [ ( ||∇X̂D ( X̂ ) ||2 − 1 ) 2 ] + β ( Xt+1 −X ′t+1 ) 2 . ( 5 ) We note that the first two terms in the R.H.S of ( 5 ) correspond to Wasserstein distance which improves learning performance over the Jensen-Shannon divergence used in the original GAN ( Arjovsky et al. , 2017 ; Gulrajani & et al , 2017 ) . The third term is the gradient penalty , corresponding to the 1-Lipschitz constraint . Unique to our work , we introduce the last term which attempts to minimize the mean squared error between the synthetic data and its true value . In such a manner , we argue that the generator can generate accurate synthetic data while preserving the temporal dynamics between conditioning input and generated data by minimizing the difference between the generated data distribution Pg and real data distribution Pr .

Summary: This paper presents a framework for long-range forecasting of time series data using synthetic data. The whole pipeline contains three main components: cWGAN-GEP to predict/generate the following data based on observed ones; an LSTM predictor to forecast results based on real/generated data; and ITC to cluster data points by mutual information which forms training data for the first two components according to clustering results. The proposed framework is evaluated on three public datasets and results seem promising.