A Study of Aggregation of Long Time-series Input for LSTM Neural Networks

1 INTRODUCTION . Predicting the future has always been a task humans found hard to accomplish . In the last decade , many systems have surrounded themselves with multi-variate tracing , recording many variables throughout time . Natural systems such as weather and rivers , financial systems such as the stock market and Bitcoin and global epidemics , all are recorded . This rise of big data allowed machine learning algorithms to predict the future with great accuracy . The mentioned systems can be described using a single variable over time , namely univariate time series , or several variables altogether as a multivariate time series . In both cases , one variable is marked as the important one whom we ’ d like to predict . However , typically , machine learning models are not capable of handling long time series . As the input intake to the model ( more samples ) grows wider , the model is required to adjust more internal weights to learn the dependencies in the data . So while additional data appears to be always an advantage , it affects learning speed but also learning accuracy . To overcome the trade-off between the will to accommodate as much data as possible and the engineering limitation of the learning process it is common to aggregate the data at the model input . Namely , a long data sequence is used , but the input to the model is kept small by partitioning the data to equal size intervals and using the average of the samples in each interval ( Li et al. , 2017 ; Yu et al. , 2017 ; Petersen et al. , 2019 ; Zheng et al. , 2020 ; Farhi et al. , 2021 ) . This enables to preserve the model size modest while still looking far to the past . We observe here that this standard way of aggregation can not be optimal since obviously the data closer to the present ( “ newer ” ) is more important than older data , but it is still aggregated similarly . Thus , we suggest an aggregation method that increases the aggregated interval length exponentially as we look farther back in time . Additionally , we question the use of averaging as the best aggregation function . In this paper , we study our Exponential partitioning and show using seven publicly available datasets that indeed it performs much better than the uniform partitioning that is commonly practiced – an improvement of 6.4-27 % in the RMSE of the prediction . Interestingly , the best results were usually achieved for slow Exponential grows where the exponent base is in the range [ 1.05 , 1.1 ] . We also study other aggregation methods for the intervals ( sample partitions ) and found a more complex picture . Median was the best aggregation function for most of the datasets , however average was still the best for others , and in one case Maximum was the best aggregation function . We also tested a combination of two functions but could only achieve minor improvements . The source-code for our Exponential partitioning is available at http : //www.github.com/TBD . 2 MOTIVATION . To motivate our approach , consider the simplest case of estimating the future value of a number series using a linear estimator , which is based only on two values of the series . To model this , suppose the series is generated by some function and we can sample the function in the interval [ xs , xe ] and want to predict the value at xp > xe . Similarly to the practice in long time series entry to LSTM , instead of selecting the series ( function ) value at two points in the interval , we can calculate the average of sub-intervals . Equation 1 and Equation 2 show the calculated two points , as a function of k , a value within the interval ( not necessarily the middle of it ) . Equation 3 is the linear estimator and Equation 4 calculates the difference between the actual value and the predicted value using a linear estimator . Figure 1 show the calculated differences for the Sine ( sin ( 15 ·X ) ) and Log ( log2X ) functions with the linear interpolation that were done with the interval [ 1 , 7 ] where k varies in the interval ( 1 , 7 ) and the predicted value is at 8 . The graphs clearly show that the optimal partition is not in the middle of the interval and selecting a partition close to the end of the interval is optimal . Similar results were obtained with a second order Lagrange polynomial interpolation . x1 ( k ) = k + xs 2 x2 ( k ) = xe + k 2 ( 1 ) y1 ( k ) = ∫ k xs f ( x ) dx k − xs y2 ( k ) = ∫ xe k f ( x ) dx xe − k ( 2 ) P ( x , k ) = x · y2 ( k ) − y1 ( k ) x2 ( k ) − x1 ( k ) ( 3 ) Diff ( k ) = F ( xp ) − P ( xp , k ) ( 4 ) 3 RELATED WORK . Different architectures implemented an ensemble model that tries to focus both on short term data and long term , for better accuracy . Krstanovic & Paulheim ( 2018 ) constructed an ensemble of LSTM models , each sub-model selected a different window size and other various parameters . Zhao et al . ( 2020 ) also created a cluster of LSTMs that are constructed to separately forecast with different time lag . NNCT weight integration strategy is introduced to determine the weight coefficients of the ensemble model . A more general method was proposed by Choi & Lee ( 2018 ) where an ensemble of LSTM models with varying sequence length ( accordingly to different datasets ) , this allows the model to be capable of handling the dynamic different real-world time series . These methods , although trying to combine between different intervals of time , indirectly try to solve the problem , and do not create a single model that predicts accurately . Changes to the sliding window algorithm were suggested by Laguna et al . ( 2011 ) who proposed a dynamic sliding window , where the size varies and changes according to the last selected window , and after is processed by Dynamic Bayesian Network . It must be noted that this type of pre-processing does not fit LSTM models since the input data shape is not static . Other sliding window methods , mostly for optimization purposes were suggested by Traub et al . ( 2019 ) ; Tangwongsan et al . ( 2017 ) . Similar mechanisms from the field of image vision have been imported into time-series analysis as an attempt to handle temporal data with many features ( such as images ) ( Donahue et al. , 2015 ) . A single model , composed of CNN and LSTM , is used to handle a time-series input , where each instance in time contains many features . In this type of model , time series data is pre-processed by CNN to extract a fixed meaningful vector of the input . We note that unlike this type of model which deals with short window size and many features , this paper deals with long window size . 4 METHODS . In this section we describe how the input is processed for the LSTM model . Data is supplied as a list of time records , where for each time record multiple variables may be available . The data is fed in multiple cycles , where in each cycle only data from a specific time window is used . 4.1 LSTM INPUT . The LSTM architecture accepts its data as a series of timestamps , each can be associated with many features . The following process is employed to allow a time series to be given as input to LSTM . For a single input , we select a window in time and consider the data in this window to predict value of our predicted variable in a constant distance in the future , also called Prediction Horizon ( PH ) . For example , an 8-hour window can be used to predict value that is a 4 hours ahead of the end of that window ( Figure 2 ) . We also note that the figure show a usage of one feature only , when many features could be used in parallel . A relevant window size may contain hundreds of samples , and is too large for an LSTM . To lower the input size , many papers suggest aggregating the windows into larger time interval , each is represented by the average value of all samples in the interval . For example , if the window size is 3 hours and the interval size is 1 hour , a single input to the LSTM is 3 vectors each is an average of 1 hour . The next input to the LSTM is generated by sliding the window by a certain time period , termed a step ( Figure 2 ) . The prediction result can be either a regression , namely an attempt to predict the value , or it can be a classification problem such as predicting that a value will rise above a certain threshold . 4.2 WINDOW PARTITIONING . We suggest to model aggregation as a partition of an interval ( time window ) , as defined in Equation 5 , where x0 and xn are the boundaries of the window , xn−x0 is the window size , and [ xi , xi+1 ] is a sub-interval of the partition . x0 < x1 < x2 < ... < xn ( 5 ) Since each subinterval in time holds a countable number of samples , we will also use the term bin for a subinterval , and use the term bin size for the number of samples in the sub-interval . We will refer to the partition by the sequence of bin sizes ; for example , if the window holds 12 samples , { 3,4,5 } is a partition into three subintervals and the number of samples in the three subintervals is 3 , 4 , and 5 . The common practice ( Farhi et al. , 2021 ; Li et al. , 2017 ; Yu et al. , 2017 ; Petersen et al. , 2019 ; Zheng et al. , 2020 ) is to use a uniform partitioning , namely xi+1 − xi = xj+1 − xj , ∀i , j . We suggest here to use a non-uniform partitioning . 4.3 EXPONENTIAL PARTITIONING . As was explained in 2 , we observe that the data with higher importance is the closer to the present . Thus , we suggest the Exponential partitioning where the bin size grows as we use data deeper in the past . Formally Equation 6 defines the bin sizes of an exponential partitioning . This is done until the penultimate bin , where the last bin holds the reminder of the window . Figure 3 , shows an example of partitioning process , where the window size is 6 records and a list of bins is { 1 , 2 , 3 } . Table 1 shows some of the partitioning we used in the paper and their exponential parameters when applicable . Partition number 0 is the common uniform partition ( uniform baseline ) and partition number 1 is the case where the last n samples are taken as singletons and there is no aggregation at all ( simple baseline ) – both are used as baselines on our experiments . Partition number 2 takes the n − 1 last samples as singletons and aggregates the rest in one bin . This partition is important to understand whether the rest of the history is meaningful where compared to no aggregation ( partition number 1 ) We show in Section 5.2 the impact of aggregating the window size ’ s reminder is significant . Partition number 3 is a linear partitioning ( bin sizes grow linearly ) , and it is closely resemble an Exponential partitioning with b = 1 and ε = 0.45 xi+1 − xi = bb ( 1 + ε ) ic ( 6 )

Summary: This paper looks at how to aggregate time-series inputs to LSTM models, and recommends to use non-uniform aggregation, with small recent bins and larger older bins. Furthermore, instead of simple averaging of time-samples within the bin, it also found value by using non-linear aggregations like median, max and min.  The observations are mostly experimental using a collection of 7 time-series datasets.