A Study of Aggregation of Long Time-series Input for LSTM Neural Networks
Authors: Nitzan Farhi
Yuval Shavitt
Venue: ICLR 2022 Submitted (2022)
Keywords: LSTM
Data Aggregation
Time Series

Abstract: Time series forecasting is the process of using time series data to create a prediction model. 
Long-short term memory (LSTM) models are the state-of-the-art for time-series forecasting.
However, LSTMs can handle limited length input mostly since when the samples enter the model in sequence, 
the oldest samples need to propagate through the LSTM cells self loop for each new sample and thus their data diminishes in this process.

This limits the length of the history that can be used in the training for each time epoch. The common way of handling this problem is by partitioning time records to uniform intervals, averaging each interval, and feeding the LSTM with rather short sequences, but each represents data from a longer history. 
    
In this paper, we show that this common data aggregation method is far from optimal. We generalize the method of partitioning the data, and suggest an Exponential partitioning. We show that non-uniformly partitioning, and especially Exponential partitioning improves LSTM accuracy, significantly. Using other aggregation functions (such as median or maximum) are shown to further improve the accuracy. Overall, using 7 public datasets we show an improvement in accuracy by 6% to 27%. 

Under review as a conference paper at ICLR 2022
A STUDY OF AGGREGATION OF LONG TIME-SERIES INPUT FOR LSTM NEURAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
Time series forecasting is the process of using time series data to create a predic- tion model. Long-short term memory (LSTM) models are the state-of-thee of-theeart for time-series forecasting. However, LSTMss can handle limited length input mostly since when the samples enter the model in sequence, the oldest samples need propagate through the LSTM cells self loop for each new sample and thus their data diminishes in this process. This limits the length of the history that can be used in the training for each time epoch. The common way of handling this prob- lem is by partitioning time records uniform intervals, averaging each interval, and feeding the LSTM with rather short sequences, but each represents data from longer history.
In this paper, we show that this common data aggregation method is far from optimal. We generalize the method of partitioning the data, and suggest Ex- ponential partitioning. We show that non-uniformly partitioning, and especially Exponential partitioning improves LSTM accuracy, significantly. Using other ag- gregation functions (such as median or maximum) are shown to further improve the accuracy. Overall, using 7 public datasets we show an improvement accu- racy by 6% to 27%.
INTRODUCTION
Predicting the future has always been a task humans found hard to accomplish. In the last decade, many systems have surrounded themselves with multi- variate tracing, recording many variables throughout time. Natural systems such as weather and rivers, financial systems such as the stock market and Bitcoin and global epidemics, all are recorded.
This rise of big data allowed machine learning algorithms to predict the future with great accuracy. The mentioned systems can be described using a single variable over time, namely univariate time series, or several variables altogether as a multivariate time series. In both cases, one variable is marked as the important one whom we'd like predict.
However, typically, machine learning models are not capable of handling long time series. As the input intake the model (more samples) grows wider, the model is required adjust more internal weights to learn the dependencies in the data. So while additional data appears to be always an advantage, it affects learning speed but also learning accuracy.
To overcome the trade-off off between the will to accommodate as much data as possible and the en- gineering limitation of the learning process
common to aggregate the data at the model input. Namely, a long data sequence is used, but the input to the model is kept small by partitioning the data to equal size intervals and using the average of the samples in each interval (Li et al.) 2017 Yu et al.) 2017: Petersen et al. 2019 Zheng et al. 2020 Farhi et al., 2021). This enables to preserve the model size modest while still looking far to the past.
We observe here that this standard way of aggregation cannot be optimal since obviously the data closer to the present ("newer") is more important than older data, but it is still aggregated similarly. Thus, suggest an aggregation method that increases the aggregated interval length exponentially as we look farther back in time. Additionally, we question the use averaging as the best aggrega- tion function.

---

Under review as a conference paper at ICLR 2022
Siee Function in range [17] predicingg
Log Function in range [1-7] prodictngg
05
180
Figure 1: The stimation error for optimal Linear estimators as function of the interval partition point.
In this paper, study our Exponential partitioning and show using publicly available datasets that indeed it performs much better than the uniform partitioning that is commonly practiced an improvement of 6.4-27% in the RMSE of the prediction. Interestingly, the best results were usually achieved for slow Exponential grows where the exponent base is in the range [1.05, 1.1].
We also study other aggregation methods for the intervals (sample partitions) and found a more complex picture. Median was the best aggregation function for most of the datasets, however average was still the best for others, and in one case Maximum was the best aggregation function. We also tested combination of two functions but could only achieve minor improvements. The source-code for our Exponential partitioning available whttt::// www. github com/TBD
2 MOTIVATION
To motivate our approach, consider the simplest case of estimating the future value of number series using a linear estimator, which based only on two values of the series. To model this, suppose the series is generated by some function and we sample the function in the interval [Wa, -e] and want to predict the value at xp Le.-
Similarly to the practice in long time series entry to LSTM, instead of selecting the series (function) value at two points in the interval, we can calculate the average of sub-intervals. Equation | and Equation 2 show the calculated two points, function of k, value within the interval (not necessarily the middle of it).
Equation is the linear estimator and Equation 4 calculates the difference between the actual value and the predicted value using a linear estimator. Figure show the calculated differences for the Sine (sin(15- (15 X)) and Log (log, X) functions with the linear interpolation that were done with the interval [1, 7] where i varies in the interval (1, 7) and the predicted value is at 8. The graphs clearly show that the optimal partition is in the middle of the interval and selecting a partition close
to the end of the interval is optimal. Similar results were obtained with second order Lagrange polynomial interpolation.
f [(x) de U1 (k)
Jr f(x) dr
yz(k)
k
2 I2(k)= = Fg + k
k
x
He
x1(k)
(1)
2
P(x, 1c) = 1/2(k) y1 (k:
(3)
x²((6) - (k)
Diff(k) = F(xx)) - P( k)
(4)
2

---

Under review as a conference paper at ICLR 2022
7
Wenten Sime
Hossee
[1, 2, 3]
AVG
11
15
13
Figure 2: Demonstration of the sliding win- dow method.
Figure 3: An example of non-uniform parti- tioning of time series data.
3 RELATED WORK
Different architectures implemented an ensemble model that tries to focus both on short term data and long term, for better accuracy. Krstanovic & Paulheim (2018) constructed an ensemble of LSTM models, each sub-model selected different window size and other various parameters. Zhao et al. (2020) also created a cluster of LSTMs that are constructed to separately forecast with different time lag. NNCT weight integration strategy is introduced to determine the weight coefficients of the ensemble model. A more general method was proposed by Choi & Lee (2018) where an ensem- ble of LSTT models with varying sequence length (accordingly to different datasets), this allows the model to be capable of handling the dynamic different real-world time series. These methods, although trying combine between different intervals of time, indirectly try to solve the problem, and do not create a single model that predicts accurately.
Changes the sliding window algorithm were suggested by Laguna al. (20TT) who proposed dy- namic sliding window, where the size varies and changes according to the last selected window, and after is processed by Dynamic Bayesian Network. must be noted that this type of pre-processing does not fit LSTT models since the input data shape is not static. Other sliding window methods, mostly for optimization purposes were suggested by TTaub al, (2019): Tangwongsan et al. (2017).
Similar mechanisms from the field of image vision have been imported into time-series analysis as an attempt to handle temporal data with many features (such as images) (Donahue et al. 2015).
A single model, composed of CNN and LSTM, is used to handle a time-series input, where each instance in time contains many features. In this type of model, time series data is pre-pree preepeeeeeeed by CNN to extract fixed meaningful vector of the input. We note that unlike this type of model which deals with short window size and many features, this paper deals with long window size.
4 METHODS
In this section we describe how the input is processed for the LSTM model. Data is supplied as list of time records, where for each time record multiple variables may be available. The data is fed in multiple eycles, where in each cycle only data from specific time window is used.
4.1 LSTM INPUT
The LSTM architecture accepts its data as a series of timestamps, each can be associated with many features. The following process is employed to allow a time series to be given input LSTM. For single input, we select a window in time and consider the data in this window to predict value of our predicted variable in a constant distance in the future, also called Prediction Horizon (PH). For example, an 8-hour window can be used to predict value that is 4 hours ahead of the end of that window (Figure D We also note that the figure show usage of one feature only, when many features could be used in parallel.
relevant window size may contain hundreds of samples, and is too large for an LSTM. To lower the input size, many papers suggest aggregating the windows into larger time interval, each is repre- sented by the average value of all samples in the interval. For example, if the window size is hours and the interval size is 1 hour, single input the LSTM is 3 vectors each is average of 1 hour. The next input to the LSTM is generated by sliding the window by a certain time period, termed a step (Figure 2 The prediction result can be either a regression, namely attempt predict the
3

---

Under review as a conference paper at ICLR 2022
Index
partition
base (b)
0
uniform baseline
{6, 6, 6, 6, 6. 6, 6, 6
1
simple baseline
1,
2
1, 1,
41)
3
1,2,3,4,55
7, 20
4
4.4,4.4.4,4. 20
5
2. 29}
6
8, 16]
7
[...3322..3333333
0.25
8
3,1,2.2,3,3,4.32
0.30
9
3,2,2,,,,,333333
0.10
2
10
I.I.I.I..1222.3333.
0.15
2,
0.20
12
2,
0.23
Table 1: Selected partitions for window of 48 samples, with the parameters for Equation 6
value, it can be classification problem such predicting that value will rise above a certain threshold.
4.2 WINDOW PARTITIONING
We suggest to model aggregation partition of an interval (time window), as defined in Equa- tion 5 where 2g and Fn are the boundaries of the window, In 15-2p is the window size, and [Iy 2++1] sub-interval of the partition.
<
Jn
(5)
Since each subinterval in time holds a countable number of samples, we will also use the term bin for a subinterval, and use the term bin size for the number of samples in the sub-interval. We will refer to the partition by the sequence of bin sizes; for example, if the window holds 12 samples, {3,4,5) is a partition into three subintervals and the number of samples in the three subintervals is 3, 3, and 5.
The common practice (Farhi et al. 202 Li et 2017 et al. 2017 Petersen al. 2019 Zheng
et al. 2020) is to use uniform partitioning, namely Ti+1 24 = x5+1 Ij. Vi, J. We suggest here to use non-uniform partitioning.
4.3 EXPONENTIAL PARTITIONING
As was explained in we observe that the data with higher importance is the closer to the present. Thus, suggest the Exponential partitioning where the bin size grows as use data deeper in the past. Formally Equation 6 defines the bin sizes of an exponential partitioning. This is done until the penultimate bin, where the last bin holds the reminder of the window. Figure 3 shows an example of partitioning process, where the window size is 6 records and a list of bins is 2,
Table shows some of the partitioning we used in the paper and their exponential parameters when applicable. Partition number 0 is the common uniform partition (uniform baseline) and partition number 1 is the case where the last n samples are taken as singletons and there is no aggregation at all (simple baseline) both are used as baselines on our experiments. Partition number 2 takes the n 1 last samples as singletons and aggregates the rest in one bin. This partition is important to understand whether the rest of the history is meaningful where compared to aggregation (par- tition number 1) We show Section 5.2 the impact of aggregating the window size's 's reminder is significant. Partition number 3 is a linear partitioning (bin sizes grow linearly), and it is closely resemble an Exponential partitioning with b = 1 and E = 0.45
[b 3)
T4+1 3.
4

---

Under review as a conference paper at ICLR 2022
4.4 AGGREGATION FUNCTIONS
The most common way of aggregating data in a bin is calculating the average of the samples. This done attempt to reduce the dataset size while preserving the remaining data meaningful. Averaging is not necessarily the best way to do so, other functions such as minimum, maximum and median, may be proven more helpful. Indeed, we show that in many scenarios other aggrega- tion functions represent actual data better than average. Other possible methods are calculating the median, sum or any other OWA operator Yager 1988
Herennder al, 2021..
4.5 DATASETS
We used collection of 7 time-series datasets, where records at constant intervals from one another. These datasets are mostly used for regression problems, as none of them are labeled. The datasets used are as follows:
1.
The bike_rental (Fanaee-T & Gama 2014) dataset contains of hourly data of bikes rentals in Washington, D.C., USA over two years. It contains data regarding weather, wind and the amount of casual and registered bike rentals. We used the total amount of bikes rented. Shape (17379, 15).
2. The electricity (ele) dataset contains 15 minute aggregated power consumption (KW) of 321 clients in Portugal over year. The first client was used for prediction. Shape (26304, 321).
3. The exchange (exc) dataset contains the daily exchange rate between American Dollar and other currencies such as AUD, GBP, CAD, CHF, CNY. JPY, NZD and SGD in the years 1990 2016. Australian Dollar exchange rate was used for prediction. Shape (7588, 8).
4. The solar (sol) dataset contains solar power production records for 10 minute intervals in the year of 2006 from 137 photo-voltaic power station plants in Alabama, USA. The first plant was used for prediction. Shape (51560. 137).
5. The traffic (tra) dataset contains hourly data of road occupancy, recorded by different sen- sors in San Francisco freeways throughout the years 2015-2016. The first freeway was used for prediction. Shape (17544, 862).
6. The turkey-power (tur) dataset contains hourly electricity consumption (MWh) of Turkey in the years 2015 2020. Shape (63305, 16).
7. The venezia water (ven) dataset contains hourly measurements from 1983 to 2015 of the water level in Venice (meters). Shape (289272, 1).
Some datasets were needed to be cleaned by replacing extreme or missing values with a default invalid value (in our case -1, occurred only in locations where -1 is indeed an invalid value). Fur- thermore, one-hot encoding of some features were added in turkey-power ppower dataset. Additionally, all datasets were normalized to the range [0.1] LSTM models suffer from the vanishing gradient problem and normalizing the data helps to avoid this problem. Window size was selected to be 48 time instances and prediction horizon of 12 was chosen.
4.6 PERFORMANCE EVALCATION
We use dicting a feature, prediction horizon ahead. This metric captures how different is the predicted curve from the real measurement. Although exponential Partitioning we could not test this since the datasets we used are not labeled. Furthermore, RMSE itself is not sufficient describe improvement it depends on the scale of the data, we compare the decrease in the RMSE, compared to the bascline, as described in Equation g
squared error (RMSE) as evaluating metric (see Equation 7
pre-
also be used for elassification,
RM S'nseline - R.M S Eresult
n
Improu.%
RMSE
ti - ps)
(7)
RM S Ebasetine x 100
(8)
i=1
5

---

Under review as a conference paper at ICLR 2022
5 RESULTS
5.1 SETUP
For the experiments, used k-fold validation, divided each dataset into parts, where for each iteration, we used one part as the test set and the other four as training. Finally, we averaged the results of all iterations as the final result.
The proposed model was implemented using the Keras (Chollet 2015) library with TensorFlow backend (Abadi al. 2016) using Python. Training the model was done on Intel(R) Xeon(R) Platinum 817IM CPU in the Microsoft Azure cloud, where a single prediction takes about 5 mil- liseconds. The model used is a simple LSTM neural network with 10 neurons in order to facilitate fast experimentation. We also report an experiment with a few more complicated models to show that our results are not limited to simple models.
Average Improvment to Average Baseline
Minimum Improvment Avergae Baseline
eatrs Aree
tartay prove
tr
wesoe adder
Sacond Aveage Thid Arange
Fire-llenge Bdeend Iuenge
Median to Average Baseline
Maximum Improvement to Average Baseline
25
wtol suuad Anouge Tled
Tre treng Saecett
Thes
(c)
(d)
Figure 4: Percentage of improvement compared to average uniform baseline in RMSE.
5.2 EXPONENTIAL PARTITIONING RESULTS.
We compared the results of the partitions in Table with uniform baseline aggregated with average function and simple baseline (since all bins are 1, no aggregation is needed). Results of the com- parisons with the uniform baseline can be seen in Figure 4al For all the datasets the top partitions improve the accuracy of the baseline by 3-12%. The top partition improvement was 4-12%. For the electricity and traffic datasets the top partitions are all above 10% improvement.
When adding a different aggregation function to exponential Partitioning improvement changes dramatically (see Figure 4b 4d all the comparisons in the figures are made with the average aggre- gation function using a uniform partition). Median and maximum appears to perform best, where the median shows an improvement of 6-27%, and maximum 5-18%. Note that many cases, the second and even the third best partition has similar improvement the best partition. We will dis- cuss this point later with additional data, but even at this point we can see that the improvement is not sensitive to some special partition. The minimum aggregation function does not show signif- icant improvement in several datasets and others datasets there are larger gaps between the top partitions performance. Thus, is not attractive candidate for aggregation function.
is interesting to isolate the improvement due to the aggregation function. For this purpose we bring Figure 5 and Table Figure compares FompreeS the top three partitions of the three aggregation functions

---

Under review as a conference paper at ICLR 2022
Maximum Inprovereent Maximum
Minimum Impr ownentt Minimmmm Baseline
10:00
5000
rale
ventong aader
like
(8)
Median Improvement to Median Baseline
25.00
30.00
15:00
00.00
-5.00
tarter
wbol stuae
(c)
Figure 5: Percentage of improvement compared to uniform baseline with the same aggregation function in RMSE. The bars for venezia water for the maximum aggregation function were trimmed to obtain better visibility.
Aggregation Functions Comparison
15
baftl
ber Stowe
smoat Like
liee
ma itmeed
Exsira
Gandin
008 M-AR S.7RAA
Figure 6: Comparison of different aggregation functions.
Figure 7: Comparison of 3 partitions used by different models
the baseline partition with same aggregation function. For the minimum function the improvement is always positive in the range of 7-21% %. For the maximum we see even greater improvement in most cases, but for one dataset (exchange) there is negative improvement. For median the differences are smaller. Figure [ shows the improvement of the top partitions, with maximum and median aggregation functions, compared to the top partition with average aggregation.
Table 2 compares the performance of applying the three aggregation functions with the average on the uniform partition. The picture here mixed: in two datasets the average function perform best, in three datasets median is the best and in two datasets maximum is the best, thus all the improvements are negative, in three datasets average is the second best, and only one case all the other three aggregation functions are performing better than average.
Table Sssows the best result achieved for each dataset. Aggregating with the median function the best in out of 7 of the datasets, while averaging is the best for only 2. As for the best partitions, all but one of them have the first bin sizes equal to or 2. There a clear preference for highly skewed partitioning. Thus, when the best partitioning is exponential, it is with very small 0.15)

---

Under review as a conference paper at ICLR 2022
Function DDaaaset
bike rental
electricity
exchange
solar
traffic
turkey powee
water
MAX
1.30
-6.055
11.82
18.14
-3.21
-10.27
-33.55
MIN
-5.62
-19.33
5,48
-3.68
-10.10
-6.94
-18.70
MED
-0.42
-1.82
19.59
7.37
9.05
3.04
-1.70
Table 2: improvement of different aggregation functions over the average aggregation on uniform partitioning.
Improv. from Uniform baseline(%)
Improv. from Simple baseline
Dataset
Function
Partition
1+c
base (b)
bikearental
AVG
(1, 1. 1, 1, 1, 2, 2, 39)
1.15
7.15
8.40
electricity
AVG
1.
1, 1, 2, 2, 39
1.15
1
12.13
11.53
exchange
MED
4, 4, 4, 4, 4, 4, 20
27.08
27.36
solar
MAX
2,
2,
1,10
2
18.72
42.09
traffic
MED
1.
1,41
12.17
14.99
turkey-power power
MED
I, 1,413
6.43
21.66
venezia water
MED
11. 1, 4, 8, 15, 16)
8.67
33.96
Table 3: Top results generic.
Average Increase in RMSE from best (%)
14
20
2.2, 2.2.2.33.33333
3.3.3.3333
1, 2, 1.3.3344.3
1, 2, 2,2,2
1, 2, 2,222,2.0,22 29
1,1.1,2,4 8, 15,16
4.4.4.444444 4,444
1,1,1, 2.1.2.2,.,3.. 2,3,2,2,,,3
6.6.6
1. 6.6. 7.7.7.
6.6.6.
Figure 8: Percentage of increase in RMSE us-
Figure 9: Performance improvement of the partition (1.11.1111111111 over the partition [1.1I.11.1.1.II.......
ing different partitions (baselines are marked in red)
Showing only the best partition for each dataset is not sufficient when wishes to understand how to select good candidate partitions.
As mentioned before, in many minimal. Thus, we tabulated in Figure for each partition, the average increase of RMSE of all datasets, compared the smallest RMSE, in the specific dataset. Interestingly I.I.I.I...11.4417 appears as one of the smallest RMSE, although it appears only twice in Table 3 The next top performing partitions
the difference between the top three partitions is rather
exponential, (1.1.1.2.488.15. and (1.2.1.1.1.222.333) Then there other skewed partitions, mostly exponential. The (4.4.4.4.444.4420) partition appears once (we see in Table 3). It still places significant weight (20) in the last bin. but of the good partitions. In general, by experimenting with few top partitions, one can easily find one that close to the best.
as skewed as the rest
It is already obvious that the partition [1,1,1,,,1111111111 is performing rather well as can be seen in Table 3 and Figure g This partition differ from the simple baseline in the last bin, where the en- tire remaining history is aggregated. Figure 9 shows the performance improvement of the partition (1.1,1,1.11111..41 over (1,1,1,,,1,1,11II;, Despite the fact that all the remaining 40 historical sam- ples aggregated and added to the single last bin the improvement is significant for the majority of the cases. In only four cases the improvement is negative, three of them are when the minimum aggregation function is used, which shown to be the worst function, by far.
8

---

Under review as a conference paper at ICLR 2022
All results far used window size of 48 samples. As mentioned before, larger window size does not mean higher accuracy when using uniform partitioning. 5. the RMSE for the aggregation baseline partition and an exponential partitioning solardataset. In this dataset, when the window size increases the RMSE of the aggregation baseline increases well, but the other hand, when using exponential partitioning, RMSE continues to decrease as the window size grows. Table 4 shows an example of the top 10 partitions for the experiment for the solar and window size of 96. Interestingly, small in the range [0.05,0.1] performs best, smaller than for window size 48.
function of the window size for
compur narping
base
Partition
(b)
[,,5,, 6, 6, 6,
7,7, 29)
1.04
5
4,4, 6, 6, 8, 9, 30;
1.08
4
6,,5,6,,, 7,77
10, 13)
1.08
5
(4,4,5,5, 6,
9, 9, 26}
1.09
4
4,4, 6,
23)
1.1
4
44,44,,, 6, 33)
1.07
4
6,739
1.05
4
5, 6, 7,7, 6,
8,
10,
1.07
5
143
7.7.7. 8, 8, 8, 8, 11}
1.02
7
5, 5, 6. 7, 8, 9, 11, 12, 14, 15}
1.15
4
Figure 10: predicted RMSE using average aggre- gation function function of the window size Table 4: Top 10 partitions for window of size 96.
for solar.
To show that our results are not limited to LSTM, we trained other models, GRU, stacked LSTM, RNN, LSTM Auto-encoder, GRU CNN, CNN and Bidirectional LSTM, solar, with window size of 96 that is aggregated using 3 different partitions: Simple baseline (i.e., all elements are I), Uniform baseline (i.e., all elements 8) and exponential Partitioning with 0.09 and b 4. Figure 7 shows that all models, except RNN (which is not widely used anymore), achieve lower RMSE using exponential Partitioning, compared to the baselines.
CONCLUSION AND FUTURE WORK
In this paper, we showed that the common way of aggregating long time series to feed LSTM net- works, by partitioning the data to equal size intervals (uniform partition) and representing each interval with the average value of the samples in it, is far from optimal. We suggested non-uniform partitioning and the use of other aggregation functions to obtain significant improvement in predic- tion.
Our finding show that while the best data partition depends on the dataset, there are a few partitions that tend to perform best close best for all datasets. Thus, obtain good prediction needs to experiment only with a few candidate partitions, that are based on exponential partitioning with found that almost every partitioning that use smaller bin
small values. In our experiment sizes for the near history and larger ones for the far history, and maintain a non-decreasing bin size order is performing better than uniform partitioning. The cases where the uniform partitioning was performing relatively well, occurred when the differences where very small. We believe that future work should better understand how select the best exponent for the partition.
REFERENCES
Electricity load diagrams.
URL
exchange
rate
URL
http://g@ttab..cm/laaauuu

---

Under review as a conference paper at ICLR 2022
URL
UBL.http.//pess.oot..c..oo
URL http:///wwwaaaaaaaaa.../oorrrora
URL
Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. Tensorflow: A system for large- scale machine learning. In 12th USENIX Symposium on Operating Systems Design and Imple- mentation (OSDI' 16), pp. 265-283, 2016.
Jae Young Choi and Bumshik Lee. Combining LSTM Network Ensemble via Adaptive Weighting for Improved Time Series Forecasting. Mathematical Problems in Engineering. 2018, 2018. ISSN 15635147. doi: 00.11555/018/2470177
François Chollet. keras. http:://ggtauub.ccm/ffoollllelel
2015.
Jeffrey Donahue, Lisa Anne Hendricks, Sergio Guadarrama, Marcus Rohrbach, Subhashini Venu- gopalan, Kate Saenko, and Trevor Darrell. Long-tern recurrent convolutional networks for visual recognition and description. In Proceedings of the IEEE conference on computer vision and pat- tern recognition, pp. 2625-2634, 2015.
Hadi T and Joao Gama. Event labeling combining ensemble detectors and background knowledge. Progress in Artificial Intelligence,
2014.
Nitzan Farhi, Efrat Kohen, Hadas Mamane, and Yuval Shavitt. Prediction of wastewater treatment quality using Istm neural network. Environmental Technology & Innovation, 2021. doi: https: http:r///10..1111/...220000.....
Sascha Krstanovic and Heiko Paulheim. Stacked Istm snapshot ensembles for time series forecast- ing. In International Conference on Time Series and Forecasting, pp. 87-98. Springer, 2018.
Javier Ortiz Laguna, Angel Gareía Olaya, and Daniel Borrajo. A dynamic sliding window approach for activity recognition. In International conference on user modeling, adaptation, and personal- ization, pp. 219-230. Springer, 2011.
Yaguang Li, Rose Yu, Cyrus Shahabi, and Yan Liu. Diffusson convolutional recurrent neural net- work: Data-driven traffic forecasting, arXiv preprint arXiv:1707.01926, 2017.
Niklas Christoffer Petersen, Filipe Rodrigues, and Francisco Camara Pereira. Multi-output bus travel time prediction with convolutional Istm neural network. Expert Systems with Applications, 120: 426-435, 2019.
Jacinto Rivero-Hermández, Annette Vázquez. Ordered weighted aggregation networks for video face recognition. Pattern Recognition Letters, 146:237-243, 2021.
Lester Guerra Denis, and Heydi Méndez-
Kanat Tangwongsan, Martin Hirzel, and Scott Schneider. Low-latency Aliding-window aggregation in worst-case constant time. DEBS 2017 Proceedings of the I 1th ACM International Conference on Distributed Based Systems, pp. 66-77, 2017. doi: 10.1 0..114473093742230099222
Jonas Traub, Philipp Grulich, Alejandro Rodríguez Cuéllar, Sebastian Breß, Asterios Katsifodimos, Tilmann Rabl, and Volker Markl. Efficient window aggregation with general stream slicing. Advances in Database Technology EDBT, 2019-March:97-108, 2019, ISSN 23672005. doi:
10.54411022/e0b22119.00
Ronald R Yager. On ordered weighted averaging aggregation operators in multicriteria decision- making. IEEE Transactions on systems, Man, and Cybernetics, 18(1):183-199 1988.
Rose Yu, Yaguang Li. Cyrus Shahabi, Ugur Demiryurek, and Yan Liu. Deep learning: A generic approach for extreme condition traffic forecasting. In Proceedings of the 2017 SIAM international Conference Data Mining, pp. 777-785. SIAM, 2017.
10

---

Under review as a conference paper at ICLR 2022
Feixiang Zhao, Guo Qiang Zeng, and Kang Di Lu. EnLSTM-WPEO: Short-term traffic flow predic- tion by ensemble LSTM, NÑCT weight integration, and population extremal optimization. IEEE Transactions Vehicular Technology, 69(1):101-113, 2020.
Haifeng Zheng, Feng Lin, Xinxin Feng, and Youjia Chen. A hybrid deep learning model with attention-based conv-LSTM networks for short-term traffic flow prediction. JEEE Transactions on Intelligent Transportation Systems, 2020.


Reviews:
[{"review_id": "nB2bLVnbp2e", "reviewer": "Reviewer_nfPs", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n\n-  Use seven publicly available datasets to evaluate the proposed partitioning procedure.\n-  Source code is available to reproduce the results.\n\nMajor Weaknesses \n\n- In the abstract, authors phrase, “Long-short term memory (LSTM) models are the state-of-the-art for time-series forecasting”, which is a very strong claim to highlight given the recent success of other machine learning architectures in the time series forecasting space. For example, the winning solution of the M5 forecasting competition was based on the LightGBM model [1], whereas the best performing model of the M4 forecasting competition was a hybrid architecture of Recurrent Neural Networks and Holt-winters model (EN-RNN) [2]. The results of these competitions are often used as a yardstick to measure the quality of newly proposed forecasting methods.  The common attribute of these best performing models is that they are trained across a collection of time series and share the same set of parameters across all-time series (Global Forecasting Models). Therefore, a more appropriate statement would be that the current state-of-the-art in time series forecasting is moving from univariate forecasting models to global forecasting models. Refer [3] for more discussions around univariate models vs global models. I agree that the LSTM architecture can be a strong architecture for time series forecasting (preferably in a GFM setting, read [4]), but referring LSTM as “the state-of-the-art” can be misleading here. \n\n-  Therefore, authors have failed to mention key related work applicable to GFMs, which also raises the question about the training mechanism used to train the proposed models in this study (LSTM and other RNN variants such as GRU). Are they univariately or globally trained ?\n\n- Even though authors use seven benchmark datasets (which is commendable) to evaluate the proposed exponential partitioning procedure, they have not compared against current state-of-the-art techniques in the time series forecasting space, or haven’t used the proposed partitioning technique to train modern forecasting architectures such as DeepAR, NBEATS ES-RNN, Transformer, WaveNet etc. Ideally, authors could have used this partitioning procedure to train these architectures with longer time series sequences and attest their respective performance. This would help authors to make strong claims about the effectiveness of this method with current state of the art. \n\n- Authors should use error measures that are commonly used in the forecasting literature (MASE, sMAPE). RMSE can be highly sensitive to outliers. Please refer to [1], where the limitations and drawbacks of the MAPE error measure has been extensively discussed [5]. Moreover, authors also do not show whether the results are statistically significant. \n\n-  Authors also do not discuss the hyper-parameter optimisation method used to optimise the various hyper-parameters of the models (LSTM, CNN)\n\nMinor comments\n\n- Please proofread the paper again (grammar, language use, etc)\n- Make sure to introduce the acronym first, before being used later in the text (e.g CNN it not defined as Convolutional Neural Networks in the text)\n\n\n[1] Makridakis, S., & Spiliotis, E. (2021). The M5 competition and the future of human expertise in forecasting. Foresight: The International Journal of Applied Forecasting, 60, 33–37.\n\n[2] Smyl, S. (2020). A hybrid method of exponential smoothing and recurrent neural networks for time series forecasting. International Journal of Forecasting, 36, 75-85\n\n[3] T Januschowski, J. Gasthaus, Y. Wang, D. Salinas, V. Flunkert, M. Bohlke-Schneider, and L. Callot. “Criteria for classifyingforecasting methods”. In:Int. J. Forecast.36.1 (2020), pp. 167–177\n\n[4] Hewamalage, H., Bergmeir, C., & Bandara, K. (2021). Recurrent neural networks for time series forecasting: Current status and future directions. International Journal of Forecasting 37, 388–427\n\n[5] Hyndman, R.J. and Koehler, A.B. 2006. Another look at measures of forecast accuracy. International journal of forecasting. (2006)\n\n\n\n", "summary_of_the_paper": "In this study, authors propose an exponential partitioning procedure to minimise the vanishing gradient and exploding gradient problems that can occur when forecasting time series with long time series. Using 7 publicly available time series datasets, authors show that the proposed exponential partitioning procedure is able to outperform the traditional uniform partitioning procedures.", "summary_of_the_review": "Even though the motivation of this work is clear, I have multiple concerns about the scientific rigour of the experimental setup (see the review section). Based on the current status of the manuscript, I am recommending to reject this paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "lXCw5_lNkfU", "reviewer": "Reviewer_tJW4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Pros\n- I think the idea is an interesting one with respect to shorter aggregation intervals, its relatively simple but I've not seen it considered elsewhere.\n- Performance over the proposed baseline is convincing over the datasets considered.\n\n## Cons\nI apologies that my main con is that I do not think this question/methodology is of sufficient interest to warrant acceptance to this conference. The work is okay and, assuming the idea of exponential change of window size away from the prediction time has not been considered elsewhere, should be published somewhere. However, I do not think there is sufficient novelty or new ground being broken in this paper to warrant acceptance. \n- It's bold to still call LSTMs state-of-the-art for time series forecasting, check any meta-review these days and there tend to be many better performers. \n- Tables and graphics are non-consistent and not nice on the eyes. Check some papers that get accepted to ICLR/NeurIPS and emulate the table styles, Figures 8 and 9 are the worst graphic offenders since they have a complete change of color scheme from the previous ones that is not pleasant viewing. ", "summary_of_the_paper": "The authors consider the problem of learning from long time series where the model (e.g. LSTM) cannot accept the full time series directly (due to memory/time constraints) and instead must accept some aggregated short form of the time series. The standard method for this is aggregation over equal-sized bins. In this paper, the authors introduce an exponential partitioning, whereby the time series is split into bins but the bin sizes increase as we move further from the current time. The rationale is that the time segments closer to the current time contain more value for future predictions and thus would benefit from aggregation over shorter time scales. \n\nThe authors test the approach for a few different aggregation functions (mean/median/min/max) and show good improvement of the proposed method compared with the simple baseline. ", "summary_of_the_review": "Okay but its a way from being accepted to ICLR or something else top tier. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "_qKJNKluUUI", "reviewer": "Reviewer_Fmnr", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "__Pros.__\n\n1. The paper studies the benefits of exponential aggregations scheme which are often used in practical time-series forecasting applications. This is because due the the sheer volume of data in industrial time-series datasets, it might make sense to store older data in more coarsely aggregated form versus more recent data. This technique is also used in practical seq-2-seq models to ensure that the encoder can attend to a larger history. Even though it is used in practice, I have not seen a systematic study in the literature. So an empirical study would be valuable if done properly. \n\n2. The paper does a decent job of describing the details of each of the experiments performed, so it is should be fairly reproducible. I do need some more clarifications, that are mentioned later in the review.\n\n\n__Cons.__\n\n1. The paper does not discuss or compare with well known advances in architectures that are specifically designed to attend to longer histroy length. For example causal convolution architectures with exponentially increasing architecture (wavenet architectures) have been used successfully for multi-variate time-series forecasting. Some references are: \n\n--- Anastasia Borovykh, Sander Bohte, and Cornelis W Oosterlee. Conditional time series forecasting with\nconvolutional neural networks.\n\n--- Sen, Rajat, Hsiang-Fu Yu, and Inderjit Dhillon. \"Think globally, act locally: a deep neural network approach to high-dimensional time series forecasting.\" Proceedings of the 33rd International Conference on Neural Information Processing Systems. 2019.\n\nThese architectures can attend to history length exponentially increasing in depth and can have a similar effect as exponential aggregation in LSTM. It would be good to cite and compare with these architectures for the sake of completeness.\n\n2. In the introduction the paper states that : _The mentioned systems can be described using a single variable over time, namely univariate time series, or several variables altogether as a multivariate time series. In both cases, one variable is\nmarked as the important one whom we’d like to predict._ This is not true, as in many retail applications we are interested in forecasting the demand of all the items not just one. Indeed the paper considers multivariate forecasting datasets but only predicts one time-series. I think this limits the impact of the study. It would we great if the paper can use the same well-established multi-variate forecasting benchmarks as used in https://arxiv.org/pdf/2103.07719.pdf that is they should report metrics on multi-variate forecasting tasks.\n\n3.  I do not think Section 2 adds any value to the paper and is not directly related to the problem being studied.\n\n4. The whole point of this paper is to display the benefits of longer history but in all the experiments a maximum history length of 48 is considered which is not long and a LSTM architecture might be able to handle it. I would be interested to know the performance of just feeding each point to the network i.e in Table 1 convention {1, 1, ....., 1} (48 ones). Also, what are the basis of choosing the window schemes in rows 2-6 in Table 1?\n\n5. It would be good to see the performance of the best window and aggregation scheme with a regular LSTM on one the standard multivariate forecasting tasks for instance the ones in https://arxiv.org/pdf/2103.07719.pdf, in order to judge whether just a better aggregation scheme is enough to boost the performance of the LSTM architecture into top-k in terms of SOTA performance.\n\n6. In most experiments the task is to predict one our of many time-series in the datasets. In this case are the values of the other time-series used as covariates? If so are they fed into the input layer of the LSTM directly? I could not find a clarification of this in the paper, it would be good if you can point me to it.\n\n7. There might be some additional references:\n--- https://arxiv.org/pdf/2012.08041.pdf -> learnt non-uniform temporal aggregation scheme for action recognition tasks\n--- http://proceedings.mlr.press/v54/bhowmik17a.html --> frequency domain learning of non-uniformly aggregated data\n", "summary_of_the_paper": "The paper studies different windowing schemes and pooling schemes for feeding data into an LSTM encoder for time-series forecasting. The windowing schemes considered are different uniform divisions of the history and exponential window sizes where data further back in the past is aggregated over larger windows. The latter enables the network to get fine grained information from more recent data while also attending to larger history length. The paper also compares different aggregation or pooling functions like max, min, mean and median. Overall there seems to be some value in exponential windowing and median aggregation scheme based on the experiments on 7 datasets, where the task is to predict the values of only one univariate time-series.", "summary_of_the_review": "Overall there is value in studying the problem. However, as evident from the Cons above the paper falls short of doing a thorough empirical study. There are multiple problems like not studying longer history length problems, not comparing on well-known multi-variate forecasting benchmark tasks, and not comparing with a simple baseline that feeds all the granular data (this can be done for history length 48). ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "FC02EpgWX0n", "reviewer": "Reviewer_2ThG", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main idea in the paper -- of using non-uniform aggregation of inputs to LSTM is reasonable, and I am not surprised that it seems to show promise. I have seen a variety of related published ideas about multi-scale aggregation (authors cite a few), but I can't think of a paper that proposed this specific simple idea. Likewise, the idea of using non-linear aggregation functions over bins is also reasonable. The paper is purely experimental, and does show evidence that some form of non-uniform aggregation would be a good idea to try when dealing with long time-series. While the proposal is simple, it could have practical significance. \n\nUnfortunately, the paper has various issues, so I can't recommend its publication:\n1) There are various issues with the experimental setup:\na) There are 7 datasets and 13 different choices of partitions. Many studies report the best among 13 partitions for each dataset -- so it's not surprising that some partition other than the baseline will win the contest for each specific dataset.  The choice of the specific 13 partitions seems fairly arbitrary.  Some are \"exponential-like\", some are uniform with different bin-widths, some are linear,  many are basically uniform with short memory except for the last large bin.\nb) There is some evidence that front-loaded partitions e.g. (1,1,1,1,..., 41) seem to be helpful for several datasets.  However, I think this is basically an LSTM with a window of size 8, which also has an additional longer-term mean feature. It's not really an 'exponential' partition. \nc) There is no attempt at statistical significance. \nd) For datasets which have many time-series (e.g. ele, exc, sol, e.t.c.) the authors only pick 1 timeseries for analysis, and drop the others.\ne) The paper uses 5-fold cross-validation on time-series examples -- no details are provided -- but interpreting it naively it seems to use past and future data in training, and test data is in the middle. This isn't a great choice for time-series.\n\n2) The example in section 2 is not at all convincing (and unclear) -- two specific functions are selected (log and sin), and the authors argue that picking two points in a non-uniform way works better.  For any partition one can select a function for which that partition is optimal. There's no theory.  Notation is unclear.  E.g. \"k\" is undefined.  I assume it's the split point. \n\n3) There is only a very minimal attempt to generalize / interpret the findings. What happens when the input window is not length 48, and when you're not predicting 12 steps ahead? When is max/min/median helpful and why compared to the average. Why is max helpful but min doesn't seem to be?  If you negate your input time-series -- then original mins become maxes -- wouldn't it now make min be a good choice for aggregation now?  What is the typical limit of how many time-steps an LSTM can handle, what does it depend on?  How can you concisely parameterize the choice of interesting partitions to explore? (E.g. the 13 in the paper).  And how to apriori know which partition may be helpful for what kind of time-series (maybe based on auto-correlation / spectral properties). \n\n4) There is no mention / comparison to the rich transformer literature (or other attention models), including the various models addressing long-memory -- e.g. longformers, reformers, e.t.c.\n\n5) Captions in the figures are hard to understand, what does \"Average improvement to average baseline\" mean?  Experimental setup is also not entirely clear, and requires a few passes to understand what's being reported. \n\n6) Various minor grammar issues -- too many to mention here.", "summary_of_the_paper": "This paper looks at how to aggregate time-series inputs to LSTM models, and recommends to use non-uniform aggregation, with small recent bins and larger older bins. Furthermore, instead of simple averaging of time-samples within the bin, it also found value by using non-linear aggregations like median, max and min.  The observations are mostly experimental using a collection of 7 time-series datasets. ", "summary_of_the_review": "This is an experimental paper that argues for using non-uniform (exponential) and non-linear (max and median vs avg) aggregation of long time-series inputs when used with LSTMs.  The ideas are simple, but potentially could be quite effective for some applications. Unfortunately, experimental results are flawed, and only partially convincing. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper addresses unique windowing schemes for the input of an LSTM model for time-series forecasting, in particular an exponential partitioning, where bin sizes increase as moving further from the current time point. Although the basic idea is interesting and motivating and experimental results are strong; as reviewers pointed out, technical significance and novelty are limited because of lack of theoretical or conceptual justification and motivation the proposed approach.  The authors’ claim is primarily based on experiments results. Other critical issues include the lack of comparison with recent advances in specifically designed to attend to longer history length or the discussion of modern approaches. other issues include presentation (e.g., grammatical errors) and the use of acronyms before introducing them.
Decision: Reject