A Study of Aggregation of Long Time-series Input for LSTM Neural Networks

1 INTRODUCTION . Predicting the future has always been a task humans found hard to accomplish . In the last decade , many systems have surrounded themselves with multi-variate tracing , recording many variables throughout time . Natural systems such as weather and rivers , financial systems such as the stock market and Bitcoin and global epidemics , all are recorded . This rise of big data allowed machine learning algorithms to predict the future with great accuracy . The mentioned systems can be described using a single variable over time , namely univariate time series , or several variables altogether as a multivariate time series . In both cases , one variable is marked as the important one whom we ’ d like to predict . However , typically , machine learning models are not capable of handling long time series . As the input intake to the model ( more samples ) grows wider , the model is required to adjust more internal weights to learn the dependencies in the data . So while additional data appears to be always an advantage , it affects learning speed but also learning accuracy . To overcome the trade-off between the will to accommodate as much data as possible and the engineering limitation of the learning process it is common to aggregate the data at the model input . Namely , a long data sequence is used , but the input to the model is kept small by partitioning the data to equal size intervals and using the average of the samples in each interval ( Li et al. , 2017 ; Yu et al. , 2017 ; Petersen et al. , 2019 ; Zheng et al. , 2020 ; Farhi et al. , 2021 ) . This enables to preserve the model size modest while still looking far to the past . We observe here that this standard way of aggregation can not be optimal since obviously the data closer to the present ( “ newer ” ) is more important than older data , but it is still aggregated similarly . Thus , we suggest an aggregation method that increases the aggregated interval length exponentially as we look farther back in time . Additionally , we question the use of averaging as the best aggregation function . In this paper , we study our Exponential partitioning and show using seven publicly available datasets that indeed it performs much better than the uniform partitioning that is commonly practiced – an improvement of 6.4-27 % in the RMSE of the prediction . Interestingly , the best results were usually achieved for slow Exponential grows where the exponent base is in the range [ 1.05 , 1.1 ] . We also study other aggregation methods for the intervals ( sample partitions ) and found a more complex picture . Median was the best aggregation function for most of the datasets , however average was still the best for others , and in one case Maximum was the best aggregation function . We also tested a combination of two functions but could only achieve minor improvements . The source-code for our Exponential partitioning is available at http : //www.github.com/TBD . 2 MOTIVATION . To motivate our approach , consider the simplest case of estimating the future value of a number series using a linear estimator , which is based only on two values of the series . To model this , suppose the series is generated by some function and we can sample the function in the interval [ xs , xe ] and want to predict the value at xp > xe . Similarly to the practice in long time series entry to LSTM , instead of selecting the series ( function ) value at two points in the interval , we can calculate the average of sub-intervals . Equation 1 and Equation 2 show the calculated two points , as a function of k , a value within the interval ( not necessarily the middle of it ) . Equation 3 is the linear estimator and Equation 4 calculates the difference between the actual value and the predicted value using a linear estimator . Figure 1 show the calculated differences for the Sine ( sin ( 15 ·X ) ) and Log ( log2X ) functions with the linear interpolation that were done with the interval [ 1 , 7 ] where k varies in the interval ( 1 , 7 ) and the predicted value is at 8 . The graphs clearly show that the optimal partition is not in the middle of the interval and selecting a partition close to the end of the interval is optimal . Similar results were obtained with a second order Lagrange polynomial interpolation . x1 ( k ) = k + xs 2 x2 ( k ) = xe + k 2 ( 1 ) y1 ( k ) = ∫ k xs f ( x ) dx k − xs y2 ( k ) = ∫ xe k f ( x ) dx xe − k ( 2 ) P ( x , k ) = x · y2 ( k ) − y1 ( k ) x2 ( k ) − x1 ( k ) ( 3 ) Diff ( k ) = F ( xp ) − P ( xp , k ) ( 4 ) 3 RELATED WORK . Different architectures implemented an ensemble model that tries to focus both on short term data and long term , for better accuracy . Krstanovic & Paulheim ( 2018 ) constructed an ensemble of LSTM models , each sub-model selected a different window size and other various parameters . Zhao et al . ( 2020 ) also created a cluster of LSTMs that are constructed to separately forecast with different time lag . NNCT weight integration strategy is introduced to determine the weight coefficients of the ensemble model . A more general method was proposed by Choi & Lee ( 2018 ) where an ensemble of LSTM models with varying sequence length ( accordingly to different datasets ) , this allows the model to be capable of handling the dynamic different real-world time series . These methods , although trying to combine between different intervals of time , indirectly try to solve the problem , and do not create a single model that predicts accurately . Changes to the sliding window algorithm were suggested by Laguna et al . ( 2011 ) who proposed a dynamic sliding window , where the size varies and changes according to the last selected window , and after is processed by Dynamic Bayesian Network . It must be noted that this type of pre-processing does not fit LSTM models since the input data shape is not static . Other sliding window methods , mostly for optimization purposes were suggested by Traub et al . ( 2019 ) ; Tangwongsan et al . ( 2017 ) . Similar mechanisms from the field of image vision have been imported into time-series analysis as an attempt to handle temporal data with many features ( such as images ) ( Donahue et al. , 2015 ) . A single model , composed of CNN and LSTM , is used to handle a time-series input , where each instance in time contains many features . In this type of model , time series data is pre-processed by CNN to extract a fixed meaningful vector of the input . We note that unlike this type of model which deals with short window size and many features , this paper deals with long window size . 4 METHODS . In this section we describe how the input is processed for the LSTM model . Data is supplied as a list of time records , where for each time record multiple variables may be available . The data is fed in multiple cycles , where in each cycle only data from a specific time window is used . 4.1 LSTM INPUT . The LSTM architecture accepts its data as a series of timestamps , each can be associated with many features . The following process is employed to allow a time series to be given as input to LSTM . For a single input , we select a window in time and consider the data in this window to predict value of our predicted variable in a constant distance in the future , also called Prediction Horizon ( PH ) . For example , an 8-hour window can be used to predict value that is a 4 hours ahead of the end of that window ( Figure 2 ) . We also note that the figure show a usage of one feature only , when many features could be used in parallel . A relevant window size may contain hundreds of samples , and is too large for an LSTM . To lower the input size , many papers suggest aggregating the windows into larger time interval , each is represented by the average value of all samples in the interval . For example , if the window size is 3 hours and the interval size is 1 hour , a single input to the LSTM is 3 vectors each is an average of 1 hour . The next input to the LSTM is generated by sliding the window by a certain time period , termed a step ( Figure 2 ) . The prediction result can be either a regression , namely an attempt to predict the value , or it can be a classification problem such as predicting that a value will rise above a certain threshold . 4.2 WINDOW PARTITIONING . We suggest to model aggregation as a partition of an interval ( time window ) , as defined in Equation 5 , where x0 and xn are the boundaries of the window , xn−x0 is the window size , and [ xi , xi+1 ] is a sub-interval of the partition . x0 < x1 < x2 < ... < xn ( 5 ) Since each subinterval in time holds a countable number of samples , we will also use the term bin for a subinterval , and use the term bin size for the number of samples in the sub-interval . We will refer to the partition by the sequence of bin sizes ; for example , if the window holds 12 samples , { 3,4,5 } is a partition into three subintervals and the number of samples in the three subintervals is 3 , 4 , and 5 . The common practice ( Farhi et al. , 2021 ; Li et al. , 2017 ; Yu et al. , 2017 ; Petersen et al. , 2019 ; Zheng et al. , 2020 ) is to use a uniform partitioning , namely xi+1 − xi = xj+1 − xj , ∀i , j . We suggest here to use a non-uniform partitioning . 4.3 EXPONENTIAL PARTITIONING . As was explained in 2 , we observe that the data with higher importance is the closer to the present . Thus , we suggest the Exponential partitioning where the bin size grows as we use data deeper in the past . Formally Equation 6 defines the bin sizes of an exponential partitioning . This is done until the penultimate bin , where the last bin holds the reminder of the window . Figure 3 , shows an example of partitioning process , where the window size is 6 records and a list of bins is { 1 , 2 , 3 } . Table 1 shows some of the partitioning we used in the paper and their exponential parameters when applicable . Partition number 0 is the common uniform partition ( uniform baseline ) and partition number 1 is the case where the last n samples are taken as singletons and there is no aggregation at all ( simple baseline ) – both are used as baselines on our experiments . Partition number 2 takes the n − 1 last samples as singletons and aggregates the rest in one bin . This partition is important to understand whether the rest of the history is meaningful where compared to no aggregation ( partition number 1 ) We show in Section 5.2 the impact of aggregating the window size ’ s reminder is significant . Partition number 3 is a linear partitioning ( bin sizes grow linearly ) , and it is closely resemble an Exponential partitioning with b = 1 and ε = 0.45 xi+1 − xi = bb ( 1 + ε ) ic ( 6 )

Summary: The paper studies different windowing schemes and pooling schemes for feeding data into an LSTM encoder for time-series forecasting. The windowing schemes considered are different uniform divisions of the history and exponential window sizes where data further back in the past is aggregated over larger windows. The latter enables the network to get fine grained information from more recent data while also attending to larger history length. The paper also compares different aggregation or pooling functions like max, min, mean and median. Overall there seems to be some value in exponential windowing and median aggregation scheme based on the experiments on 7 datasets, where the task is to predict the values of only one univariate time-series.