NOVEL FEATURE REPRESENTATION STRATEGIES FOR TIME SERIES FORECASTING WITH PREDICTED FUTURE COVARIATES
Authors: Jimeng Shi
Rukmangadh Sai Myana
Vitalii Stebliankin
Azam Shirali
Giri Narasimhan
Venue: Submitted to ICLR 2023 (2023)
Keywords: time series forecasting
future covariates
shifting
padding
periodicity
deep learning

Abstract: Accurate time series forecasting is a fundamental challenge in data science. Unlike traditional statistical methods, conventional machine learning models, such as RNNs and CNNs, use historical data consisting of previously measured variables including the forecast variable and all its covariates. However, in many applications, some of the covariates can be predicted with reasonable accuracy for the immediate future. Note that the input may also contain some covariates that cannot be accurately predicted. We consider the problem of predicting water levels at a given location in a river or canal system using historical data and future covariates, some of which (e.g., precipitation, tide) may be predictable. In many applications, for some covariates of interest, it may be possible to use historical data or accurate predictions for the near future. Traditional methods to incorporate future predictable covariates have major limitations. The strategy of simply concatenating the future predicted covariates to the input vector is highly likely to miss the past-future connection. Another strategy that iteratively predicts one step at a time can end up with prediction error accumulation. We propose two novel feature representation strategies to solve those limitations -- shifting and padding, which create a framework for contextually linking the past with the predicted future, while avoiding any accumulation of prediction errors. Extensive experiments on three well-known datasets revealed that our strategies when applied to RNN and CNN backbones, outperform existing methods. Our experiments also suggest a relationship between the amount of shifting and padding and the periodicity of the time series.

Under review as a conference paper at ICLR 2023
NOVEL FEATURE REPRESENTATION STRATEGIES FOR TIME SERIES FORECASTING WITH PREDICTED FU- TURE COVARIATES
Anonymous authors Paper under double-biee blind review
ABSTRACT
Accurate time series forecasting is fundamental challenge in data science. Un- like traditional statistical methods, conventional machine learning models, such as RNNs and CNNs, use historical data consisting of previously measured variables including the forecast variable and all its covariates. However, in many applica- tions, some of the covariates can be predicted with reasonable accuracy for the immediate future. We refer to such covariates as predictable future covariates. Note that the input may also contain some covariates that cannot be accurately predicted. We consider the problem of predicting water levels at a given location a river or canal system using historical data and future covariates, some of which (precipitation, tide) may be predictable. In many applications, for some covariates of interest, may be possible to use historical data or accurate predictions for the near future. Traditional methods to incorporate future predictable covariates have major limitations. The strategy of simply concatenating the future predicted covariates to the input vector is highly likely to miss the past-future connection. Another strategy that iteratively predicts one step at a time can end up with predic- tion error accumulation. We propose two novel feature representation strategies solve those limitations - shifting and padding, which create a framework for con- textually linking the past with the predicted future, while avoiding any accumu- lation of prediction errors. Extensive experiments on three well-known datasets revealed that our strategies when applied to RNN and CNN backbones, outper- form existing methods. Our experiments also suggest relationship between the amount of shifting and padding and the periodicity of the time series.
INTRODUCTION
Conventional time series forecasting is widely used to predict a set of target variables at a future time point based past data collected over predetermined length. Next-step forecasting (Montgomery ct al. 2015 Shi et al. 2022) refers to predicting the target variables at a time point one step into the future where the unit of time is the time granularity of the measurements. Multi-horizon forecasting (Quedvlieg 2021) predicts the target variables multiple steps into the future Capistrán et al.) (2110). Accurate forecasting allows people to do better resource management and optimization decisions for critical processes (Cinar et al. 2017 Salinas et al.) 2020 Rangapuram et al.) 2018). Applications include probabilistic demand forecasting in retail (Böse al.] (017)) dynamic assignments of beds to patients (Zhang & Nawata 2018), monthly inflation forecasting, and much more.
forecasting requires historical data of the target variables from which learn long-term patterns. In addition, it also requires measurements from heterogeneous data sources of useful covariates, often from the recent past. However, in many applications, some of the covariates can also be predicted with reasonable accuracy for the immediate future. We refer to such covariates as future covariates. For example, in some applications, a covariate of interest could be "precipita- tion". for which it is possible to use historical data as well reasonably accurate predictions for the near future, which may be obtained from the weather service. Despite its importance, only limited approaches exist that use future covariates to improve time series predictions. Related methods can be mainly categorized into direct strategy using scquence-to-sequeee models (Mariet & Kuznetsov) 2019) and iterated methods using autoregressive models (Sahoo al. 2020).
Good

---

Under review as a conference paper at ICLR 2023
Traditional methods incorporate future covariates have major limitations. We propose two novel feature representation strategies to solve those limitations shifting and padding, which create a framework for contextually linking the past with the predicted future, while avoiding any accumu- lation of prediction errors. Extensive experiments on three well-known datasets revealed that our strategies when applied to RNN and CNN backbones, outperform existing methods.
Iterative methods. The iterative strategy recursively uses a Next-step model multiple times where the predicted values for the previous time step is used as the input forecast the next time step, as in [allnas et al. (2020). For the prediction at time step 6, the target values 2 at the previous time step, the (predicted) covariates I; for the current time step, and the context vectors h that summarize the representation information of all the past time steps are considered as the input to predict target values 2 at the current time step using RNNs. [Rangapuram et al. (2018) adopted a similar approach by parameterizing a per-time-series linear state space model with recurrent neural networks. Related work with the iterative approach is Li al. (2019) where the basic architecture used was the transformer model with convolutional layers.
Direct methods. The direct method typically uses an encoder model to learn the feature represen- tation of past data, which is saved context vectors in hidden state. A decoder model is utilized to intake future covariates and context vectors from the encoder and to then predict the outputs for multi-horizon forecasting, shown in Figure T The multi-horizon Quantile Recurrent Forecaster by Wen et al. (217)) used an LSTT as the encoder to generate context vectors, which are com- bined with predicted future covariates and fed into multi-layer perceptron (MLP) to predict the future horizon. Some works ((Fan et al. 2019 Du et al. (0200) have applied a temporal attention mechanism between the encoder and the decoder. This architecture is able learn the relevance of different parts of the feature representations from historical data by computing attentional weights. The weighted feature representations are then passed into the decoder make predictions for future time steps. In Fan et al. (2019), bi-directional LSTMs are used as the decoder backbone allowing past and predicted features be considered at every future time step. Temporal Fusion Transformer (Lim et al. (202T)) combined gated residual networks (GRNs) and an attention mechanism (Vaswani ctt al. (2017) as an additional decoder on top of the traditional encoder-decoder model. They used GRNs to filter unnecessary information and the additional decoder with attention mechanism to capture long-term dependencies between the time steps.
Prior Known Covariates
Past Unknown Covariates
Target time series
t
w
t+ k
t + k
past
future
future
feature representation
Encoder
Decoder
Figure Direct method using sequence torpencce-----uunnee models.
The iterative as well as the direct methods aim to incorporate future covariates as inputs but suffer from several shortcomings. The iterative methods accumulate prediction errors because the input to each time step is the output from the previous step, causing the model performance to quickly degrade for longer forecasting horizons. On the other hand, direct methods prone to miss some interactions between data from past and future time points. The encoder processes only past data, while the decoder merely concatenates past data and future covariates, which may miss specific rela- tionships between the past and future time points. In this paper, we aim to resolve the shortcomings of both the approaches with novel architecture.
2

---

Under review as a conference paper at ICLR 2023
Our contributions: In this paper, present two novel strategies to combine historical data and predicted future covariates a meaningful manner using the strategies of shifting and padding. The architecture facilitates forecasting by learning useful feature representations from both the past and the predicted future. These two strategies transform the dataset and construct training pairs (input features, labels) used for learning. Their input features are composed of both past information on all features, predicted future information on some covariates (with appropriate time label) and set of covariates for which no accurately predicted future information is available. The goal is directly predict the target variables for the given horizons. The two strategies are briefly described below.
Shifting: Predicted future covariates are shifted past in time and paired with appropriate past time points to create a modified input vector for the predictions. The shift length is hyperparameter of the strategy.
Padding: Covariates that cannot be accurately predicted are simply copied over from the recent past, and provided as additional input parameters after combining them with future predicted covariates. The length of the padding is hyperparameter of the strategy.
2 NOTATION
2()) be N univariate time series of target variables, where Let N, (2).))=1 (z]', 22,...,
z/" E R denotes the value of the n-th target variable at time t. Similarly, let X = (IT:)) = (xT, x22 ...., xx )== be M time-varying observed covariates measured until time 4, and WP 1: = (wi:))g11 (uf. v/. ..... uf v1)0 be Q time series covariates that can be accurately predicted in the near future. To predict target time series at future multiple horizons from t - 1 to t + k. we define forecasting models that use the past data with fixed length of w. i.e., for time - goal of such a forecasting model is to predict the trajectory Z of the target variables at the next time points using the past w time points of all time series (targets and covariates) and & future time points of future predicted covariates. The model is mathematically expressed as follows:
L. The
2++1:0+ F(Z F(Zi-w+10
TM 1- Wo
Wt- θ)
where F(.) a function, denotes the learnable parameters, w represents the length the history used, and l is the length of forecasting horizon.
3 METHODOLOGY
In this section, we provide details on the learning model mentioned in Eq. above. We present two different strategies, shifting and padding, to achieve this.
3.1 THE Shifting STRATEGY
In the shifting strategy, we focus on exploiting the predicted covariates for the future time period of interest. This is achieved by shifting the predictions of the future covariates back in time by time steps, such that the present covariates are aligned and fused with a predicted covariate s time points into the future to produce distinct feature vectors at each time point (Fig. 2 Thus, the input features are composed of all time series (target and covariates) aligned from time points to with future covariates from time points & + 1 1 to + s. Target variables are predicted for the forecasting horizon from / + to / + k: This explicit use of the the predicted future covariates differentiates this approach from traditional forecasting appraoches and is expected to improve the deep learning models since simultaneously learns from the past and the future. The predicted future covariates shown blue dashed trajectory in Fig. 2 The target values computed at time t +++ and later using our shifting approach are expressed as follows:
N 1+1++
M G(Z - -
YO
11-+14 VO t-wt1+sstts's θ),
(2)
where G(-) is the model function, Z Z represents the N target variables to be predicted at the
future time points t+1 t+t & (region shaded brown in Fig. 2 Z,
1+ represent the target variables
from the past time points (brown trajectories in Fig. x2
t-t+ta and V2 (t-+1: represent the
3

---

Under review as a conference paper at ICLR 2023
covariates from the past time points (green and red trajectories from Fig. 2 and yo -r-w+1 tetts represent the predictable future covariates along with predictions from 8 time points into the future and then shifted back by time steps (green trajectories merged with dashed blue trajectories in Fig.
2). and the set of learnable parameters. The shift amount 8 is a hyperparameter of the model.
k
shift
Future predictable Covariates
time steps
Covariates cannot (X)
be predicted
Target time series
(2)
1
past
future
Input: past & shifted covariates Output: future
Figure 2: Input data transformed by shifting the predicted future covariates. Left: Original trajecto- ries of all variables. Right: All input features with the shifted predicted future covariates included.
3.2 THE Padding STRATEGY
The padding approach attempts to extrapolate and incorporate future covariates that cannot be accu- rately predicted by simply making a copy of the values from the previous s time points. The padded values then combined with the values of future covariates that can be accurately predicted. More formally, the padding method makes copy of X and Z,V t-stt> and makes them the padded values (Fig. 3p. Such manipulations
be viewed creating future pseudo-time T (t + 1, s)
where target variables N 1i++. (brown dashed line in Fig. 3 and covariates X Lit+=t+ (red dashed
line in Fig. (3) repeat the previous pattern from f to t. For predictable future covariates, the padding is achieved with the best predictions instead of the copies from the recent past. Eq. (3)
provides a mathematical description of the padding forecasting model:
2N 2-1144 HH 2 1-w+1:t:: 1-wt1it+:: O),
(3)
where H(-) is model function, Z,N 1+++++ represents the N target variables be predicted the future time points + + l (region shaded brown in Fig. 2,N the concatenation of the time series in the range /
2ttwos 1o5-++1l: lit = N - + - sst
with copy of the time series in the AM (-=+1+++ XM 1-=+1# + XM *++1-s: the set of padded covariates corresponding to covariates that cannot be accurately predicted (pink trajectories
range /. +
(brown trajectories in Fig. 3
in Fig. 3 x vo l:t+ is the set of predicted future covariates padded with the predictions for the time range + 1:tt (green trajectory in Fig. (3). and θ the set of learnable parameters.
Future predictable (Y)
Covariates
Covariates
padding
that cannot (X)
be predicted
padding
Target time
series
(2)
past
future
Input:
past
& future
Output: future
Figure 3: Input data transformed using the padding strategy. Pink and brown dotted trajectories represent the padded information the future covariates that cannot be accurately predicted and the target variables being modeled, while the blue dotted trajectories represent the predicted future covariates. Left: Original trajectories of all variables with the predicted future covariates. Right: All input features with the padded values and predicted future covariates and the output trajectory.

---

Under review as a conference paper at ICLR 2023
3.3 NETWORK ARCHITECTURES
To validate the effectiveness of the shifting and padding strategies for feature representation, we constructed simple RNN and CNN architectures that included input the covariates described above in Sections 3.1 and 3.2 The networks were set to forecast future steps in one shot instead of the traditional sequential prediction avoid accumulation of errors.
3.3.1 RNN MODELS WITH SHIFTING
Shifting is implemented by providing the RNN backbone with hidden states shown in Fig. The standard RNNs was further modified to remove the hidden states "t+t....,,he.. to enable one-shot prediction of the & future time points. The input each hidden state h were variables associated with time & = j (i.e., target variable and covariates zj. Yj), as well as the predicted covariates (shifted by 8), Vi+s shown in Fig. 4 RNN model generates the output for target variables (21+1....,, k+++) ) in a one-shot manner. The hidden states are described as follows:
hj ( 1-1 Tj Mj Uÿ+s);
(4)
where f is an activation function; h and h refer to the current and previous hidden states; 2j, Jj and Vj represent the target time series, future covariates that cannot be accurately predicted, and predictable future covariates from the past w steps; and Vj+s denotes the predicted future covariates from steps into the future.
Feth
h
h
ht-
ht-st1
her1
2
Z
Zt-a+1
y1-u++
yt-z
yt-a+1
y1-1
V H-u+l-3
Vi- Ui-
V/1
yeo1
shift left time steps
traditional seq-to-seq
our method
shifting time steps
Figure 4: RNN models with the shifting strategy. Dashed ovals represents predicted future covariates that have been shifted. Solid ovals are historical data or data for which we do have future predictions available. Colors are same as before.
3.3.2 RNN MODELS WITH PADDING
The RNN backbone for padding similar to that for shifting, but extended with extra states for pseudo-times, shown in Fig. 5 A precise mathematical formulation is given below.
f( f(h.- je +
jE It
w 1,
h2
(5)
where f(-) is the activation function; hy and h refer the current and previous hidden states; 2j, Fy Vý represent the target variables, covariates that cannot be accurately predicted, and covariates that can be accurately predicted, all for the past time points [t-w+ 1, and Eg Ty, Øj represent the padded versions of the same variables for the future time range.
3.3.3 CNN MODELS WITH SHIFTING
Convolutional Neural Networks (CNNs) can summarize and learn from the input data using sliding filters that extract features with convolutional computations. The time series aligned in manner similar to how we handled RNNs with shifting. Each CNN filter is used to extract features in parallel by sliding from the first time point the last time point, as shown the left in Fig. 6 The parallel
5

---

Under review as a conference paper at ICLR 2023
S++h
htpp-1
2
222
h;s
he+1
Zp-pol
ft-
St
0L-a-3
yi-s+1
yr-s+2
Figure 5: RNN models with the padding strategy. Dashed ovals represents padded features. Solid ovals before time f. are historical data. Colors are same as before.
action the filters allows for the simultancous learning of past and (predicted) future information. As with RNN, CNN models will output predictions of the target variables for the desired range of time points (i.e., horizon) in one shot.
Different sliding filters in CNN
Shifted future predictable Covariates
Future predictable Covariates
Covariates that cannot be predicted
Target time series
Z
t w +1 t
Input: original past & shifted covariates
Convolutional layer
Output: future
Figure 6: CNN models with the shifting strategy. Left: Input from original past and shifted future. Right: Actual future target time series. Blue line represents prior known future covariates. Pink, brown lines are past unknown future covariates and target time series, respectively.
3.3.4 CNN MODELS WITH PADDING
The time series aligned similar how it organized for RNNs with padding. The sliding filters are similar to that used in CNNs with shifting, but with t + s time points, as shown in Fig. 7 CNNs with padding require more scanning because of the extra padded pseudo-times ++:1:0 8),
but the actual filters are more compact because fewer time series are processed.
4 EXPERIMENTS
4.1 DATASETS
Three real world datasets were used for time series forecasting tasks in this paper. Beijing PM2.5 and Electricity price datasets are publicly available from UCI and Kaggle repositories, respectively. The third one is the Water stage dataset downloaded from the South Florida Water Management District website. More details about dataset descriptions can be found in Appendix A
Beijing PM2.5 includes hourly observed data from January 1, 2010, to December 31, 2014. We consider PM2.5 the target variable predict, other variables such as dew, temperature,

---

Under review as a conference paper at ICLR 2023
Different sliding filters in CNN
Future predictable (Y)
Covariates
Covariates that cannot (x)
be predicted
Target time series
t
Input: past future
Convolutional layer
Output: future
Figure 7: CNN models with the padding strategy. Left shows inputs from the past with padded series. Filters create convolutional layers, which are then used to output the target variables on the right. Colors used in previous figures.
pressure, wind speed, wind direction, snow, and rain are covariates that can be predicted and can influence PM2.5 values. PM2.5 E j0, μg/ wg/m3 8 in this dataset.
Electricity price It was recorded every hour from January 1, 2015, December 31, 2018. Enerqy_dataset. dww.aatttt.t..... includus energy demand, generation, prices, while weather features CSV contains features such as temperature and humidity. Electricity price is the target variable to predict, while prior known covariates are energy demand, generation, and weather features. Electricity price [$9.33, $1 $116.8 in this datase
Water stage This is hourly dataset from January 1, 2010, to December 31, 2020 and includes information on water levels, the height gate opening, water flow values through the gate, water volumes pumped gates, and rainfall measures. Water stage is the target variable while other variables are covariates. Rainfall, gate position, and pump control are future covariates that can be predicted. Water stage [-1.25, 4.05] feet in this dataset.
4.2 TRAINING AND EVALUATION
For each of datasets, used the first 80% training set train the model and the last 20% as a test set to evaluate the performance. During the training phase, common techniques such as normalization, dropout, regularization tune the models for optimal hyperparameters including the number of layers, the number neurons, learning rate, batch size, regularization factor, and the number of epochs. Mean Absolute Error (MAE) and Root Square Mean Error (RSME) were used to evaluate models. Details on the training and evaluation process
used avoid overfitting, Grid search was used fine-
Appendix B
The four deep learning models (with RNNs or CNNs, and with shifting or padding) were tested with the three datasets. Seq-to-Seq models (Du al. 2020). MQRNN (Rangapuram et al.) 2018). DeepAR (Salinas el al.) 2020), Temperal Fusion Transtormer (TFT) (Lim et al. 2021) were com- pared to the four methods mentioned proposed in this manuscript. We used horizon values of k = 6, 12, 24, and 48 hours to forecast with input windows of size w 72 hours and predictable future covariates from time steps ahead.
4.3 RESULTS
The results summarized in Tables 1 and The lowest emror in each column are in bold font. The results show that the shifting and padding methods with CNNs outperform all the other
Although this has no bearing on the analysis conclusions, the currency for the price column in this dataset unavialable. The S sign is used proxy for whatever currency was intended.

---

Under review as a conference paper at ICLR 2023
Table 1: MAE & RMSE for the Beijing PM2.5 dataset. (PM2.5 values E [0, (0.m7m/g/mm)
k
hrs
hrs
k:
hrs
Methods
MAE
RMSE
MAE
RMSE
MAE
RMSE
Seq-to-Seq
25.57
41.66
30.64
49.01
35.61
54.98
MQRNN
39.95
55.66
46.56
60.22
36.63
52.02
Baselines
DeepAR
37.53
57.24
42.04
65.16
49.02
72.88
TFT
33.15
55.65
33.84
53.59
37.69
59.17
Shifting (RNN)
24.80
40.62
30.47
47.74
36.81
55.52
Padding (RNN)
24.73
40.00
29.26
46.89
36.67
55.03
Our methods
Shifting (CNN)
23.79
38.81
28.49
45.58
33.51
51.61
Padding (CNN)
24.29
39.26
29.30
45.87
34.11
51.14
Table 2: MAE & RMSE for Electricity price dataset. (Price [89.33, $116.8)
k 6 hrs
k 12 hrs
h: 24 hrs
Methods
MAE
RMSE
MAE
RMSE
MAE
RMSE
Seq-to-Seq
2.769
3.761
3.350
4.596
3.736
5.169
MQRNN
3.750
4.840
7.019
7.905
8.33
9.609
Baselines
DeepAR
5.444
7.201
6,483
8.823
7.242
9.993
TFT
5.953
8.849
6.618
8.779
7.449
9.986
Shifting (RNN)
2.938
4.042
3.500
4.764
3.721
4.876
Padding (RNN)
2.437
3.454
3.053
4.167
3.541
4.704
Our methods
Shifting (CNN)
2.681
3.577
3.232
4.349
3.667
4.969
Padding (CNN)
2.690
3.710
3.226
4.359
3.583
4.745
Table 3: MAE & RMSE for the Water Stage dataset. (Water Levels E [-1.25, 4.05) ft)
12 hrs
24 hrs
48 hrs
Methods
MAE
RMSE
MAE
RMSE
MAE
RMSE
Seq-to-Sequence
0.123
0.162
0.134
0.174
0.136
0.176
MQRNN
0.123
0.149
0.123
0.225
0.126
0.176
Baselines
DeepAR
0.133
0.194
0.146
0.231
0.127
0.195
TFT
0.057
0.083
0.089
0.139
0.077
0.109
Shifting (RNN)
0.119
0.167
0.132
0.171
0.158
0.207
Padding (RNN)
0.113
0.154
0.133
0.181
0.134
0.177
Our methods
Shifting (CNN)
0.074
0.097
0.080
0.105
0.087
0.115
Padding (CNN)
0.053
0.076
0.068
0.092
0.080
0.105
methods compared in this paper for two of the datasets. The methods with RNNs are sensitive to the forecasting length, k. They outperform the "Baseline" methods for short-term predictions (k = 6, 12 hrs), but often fail to do so for long-term forecasting (k = 24, 48 hrs). Our experiments suggest that the future covariates play significant role in time series predictions. is also surprising that the simple models (RNNs and CNNs) outperform the sophisticated and complicated deep neural networks (labeled "Baseline" in the Tables).
We also experimented with the hyperparameter s, which refers to extent of shifting or padding. As expected, the performance is sensitive to the choice of The best performance appears to be achieved when s = k. In many applications, we may able to reliably predict future covariates for window much larger than the k used here. However, when or deteriorates as is increased. When > w, the performance appears to deteriorate rapidly, Our experiments also allowed us consider impacted by the periodicity of the datasets.
either the performance is flat
8

---

Under review as a conference paper at ICLR 2023
MAE for predicting hours ahead (k=6) 6)
RMSE for predicting 6 hours ahead (k=6) 6)
0.1
0.12
MAE (ft)
0.08
E 0.1
0.11
0.09
0.06
d 0.07
0.08
0.04
0.06
0
12
24
36
48
60
72 78
6
12
24
36
48
60
72 78
MAE for predicting 12 hours ahead (k=12)
RMSE for predicting 12 hours ahead (k=12)
0.12
0.16
MAE (ft)
0.1
RMSE (ft)
0.14
0.12
0.08
0.1
0.06
0.08
0.04
0.06
0
12
24
36
48
60
72
84
12
24
36
48
60
72
84
MAE for predicting 18 hours ahead (k=18)
RMSE for predicting 18 hours ahead (k=18)
0.13
0.17
MAE (ft)
0.11
RMSE (ft
0.15
0.13
0.09
0.11
0.07
0.09
0.05
0.07
12
24
36
48
60
12
MAE for predicting 24 hours ahead (k=24)
RMSE for predicting 24 hours ahead (k=24)
0.14
0.18
MAE (ft)
0.12
RMSE @
0.16
0.12
0.08
0.1
0.06
0.08
0
12
24
36
48
60
72
96
12
24
36
48
60
72
96
Shifting length
Shifting length
Figure 8: MAE & RMSE for different forecasting lengths (k) and shift lengths (s). Left red point of each subplot represents the errors when s k while right red point denotes the errors when s w.
The graphs in Fig. shows the MAE and RMSE values as a function of shift length, 8, for different values of k; making sure to consider values of s that range from smaller than & (prediction horizon) all the way to larger than (input window size).
DISCUSSION AND CONCLUSIONS
Simple modifications to the basic RNN and CNN architectures have allowed us to build deep learn- ing models that outperform more sophisticated models. Our experiments suggest that the utilization of future covariates (whether predictable or not) can enhance performance considerably. Further- more, our experiments have helped us to delineate the relationship between the shifting and padding lengths and the model performance. We have observed that s A results in the best performance since future covariates in exact same forecasting horizon included. If 8 < k, we only get to utilize some of the predicted covariates from the future for the prediction horizon of l time steps resulting in considerably lower performances. If s > w, then we end up dropping some of the future covariates in order align with the input window, which again results in considerably lower performances.
We also observed that there is considerable periodicity and seasonality in the datasets we used in our experiments. However, in the range k <= <= w. the variations in performance was too small to be significant. While there were some local minima in the performance when was multiple of the period p. the improvements were not significant.
In conclusion, the shifting- and calding-based CNNs proposed in this paper outperformed all the baseline deep learning methods considered for our experiments. The corresponding methods with RNNs outperformed the baseline methods for relatively short-term prediction (k- 6. 12 hrs), but was not as accurate for longer term forecasting (k 24, 48 hrs). The critical feature of our methods appears to be the use of future covariates, either by shifting padding. Shifting-based CNNs per- formed best when the shift length is between the forecast horizon and the length input window (k S I w), suggesting sweet spot for how much of the future is needed for good performance. Periodicity in the datasets appears to have small, but insignificant influence on the performance of the shidi-based methods.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Joos-Hendrik Böse, Valentin Flunkert, Jan Gasthaus, Tim Januschowski, Dustin Lange, David Sali- nas, Sebastian Schelter, Matthias Seeger, and Yuyang Wang. Probabilistic demand forecasting scale. Proceedings of the VLDB Endowment, 10(12):1694-1705, 2017.
Carlos Capistrán, Christian Constandse, and Manuel Ramos-Francia
inflation fore- casts using disaggregated data. Economic Modelling, 676)) 2010.
Yagmur Gizem Cinar, Hamid Mirisace, Parantapa Goswami, Eric Gaussier, Ali Ait-Bachir, and Vadim Strijov. Position- based content attention for time series forecasting with sequence-to- sequence RNNs. In International conference neural information processing, pp. 533-544. Springer, 2017.
Shengdong Du, Tianrui Li, Yan Yang, and Shi-Jinn Horn. Multivariate time series forecasting via attention-based mncodrrddecode framework. Neurocomputing, 388:269-27) 279, 2020.
Chenyou Fan, Yuze Zhang, Yi Pan, Xiaoyue Li, Chi Zhang, Rong Yuan, Di Wu, Wensheng Wang, Jian Pei, and Heng Huang. Multi-horizon time series forecasting with temporal attention learning. In Proceedings of the 25th ACM SIGKDD International conference on knowledge discovery & data mining, pp. 2527-2535, 2019.
Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. Enhancing the locality and breaking the memory bottleneck of transformer time series forecasting. Advances in neural information processing xystems, 32, 2019.
Bryan Lim, Sercan Ô Arık, Nicolas Loeff, and Tomas Pfister. Temporal fusion transformers for interpretable multi-horizon time series forecasting. International Journal of Forecasting, 37(4): 1748-1764, 2021.
Zelda Mariet and Vitaly Kuznetsov. Foundations of sequence-to-sequeee modeling for time series. In The 22nd international conference artificial intelligence and statistics, pp. 408-417. PMLR, 2019.
Douglas C Montgomery, Cheryl Jennings, and Murat Kulahci. Introduction time series analysis and forecasting. John Wiley & Sons, 2015.
Rogier Quaedvlieg. Multi-horizon forecast comparison. Journal of Business & Economic Statistics, 39(1):40-555 2021.
Syama Sundar Rangapuram, Matthias W Seeger, Jan Gasthaus, Lorenzo Stella, Yuyang Wang, and Tim Januschowski. Deep state space models for time series forecasting. Advances in neural information processing systems, 31, 2018.
Debashis Sahoo, Naveksha Sood, Usha Rani, George Abraham, Varun Dutt, and AD Dileep. Com- parative analysis of multi-step time-series forecasting for network load dataset. In 2020 I Ith In- ternational Conference on Computing, Communication and Networking Technologies (ICCCTT), pp. 1-7. IEEE, 2020.
David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski. DecpAR: Probabilistic forecasting with autoregressive recurrent networks. International Journal of Forecasting, 36(3): 1181-1191, 2020.
Jimeng Shi, Mahek Jain, and Giri Narasimhan. Time series forecasting (TSF) using various deep learning models. arXiv preprint arXiv:2204.111 11115. 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Ruofeng Wen, Kari Torkkola, Balakrishnan Narayanaswamy, and Dhruv Madeka. A multi-horizon quantile recurrent forecaster. arXiv preprint arXiv:1711.11055 2017.
J Zhang and K Nawata. Multi-step prediction for influenza outbreak by an adjusted long short-term memory. Epidemiology & Infection, 146(7):80--81) 2018.
10

---

Under review as a conference paper at ICLR 2023
A APPENDIX: DATASET
A.1 BEIJING PM2.5 DATASET
This dataset is Beijing Air Quality data from the public UCI Website bettps://archine ics uci. edu datasetss/Beee 1ing+ jingttt/ 5+Data It includes hourly observed data from January I, 2010, December 31, 2014. The data set has 43,824 rows and 13 columns. The first column is simply an index and was ignored for the analysis. The four columns labeled as year, month, day, and hour, were combined into a single feature called "year-month-day
We con- sider PM2.5 as the target variable to predict, other variables such as dew, temperature, pressure, wind speed. wind direction, snow, rain, etc. are the prior known covariates that can influence PM2.5 values. PM2.5 [0, 671] pg/ m in this dataset.
Table 4: Description of Beijing PM2.5 Dataset
VARIABLE
DESCRIPTION
No
row number
year
year of data in this row
month
month of data in this row
day
day of data in this row
hour
hour of data in this row
pm2.5
PM2.5 concentration (ug/m²)
DEWP
Dew Point (á., f) Temperature (ā,
TEMP
PRES
Pressure (hPa)
cbwd
Combined wind direction
Iws
Cumulated wind speed (m/s)
Is
Cumulated hours of snow
Ir
Cumulated hours of rain
A.2 ENERGY (ELECTRICITY) PRICE DATASET
This dataset contains 4 years of electrical consumption, generation, pricing, and weather data for Spain. It is publicly available in the Kaggle website: hittps://wwt kaggle. com/ http://aacchhortteeeeeeeee
It has two hourly datasets from January 1, 2015, to December 31, 2018. Energy dataset.csv includes the information of energy demand, generation, prices, and weather features.csv gives the weather features temperature, humidity, etc. Electricity price is the target variable to predict, while prior known covariates are energy demand, generation, and weather features. Electricity price E [9.33, 116.8) in this dataset.

---

Under review as a conference paper at ICLR 2023
Table 5: Description of Energy (electricity) price dataset
VARIABLE
DESCRIPTION
generation biomass
biomass generation M W coal/lignite generation in MW coal gas generation in MW gas generation in coal generation in MW oil generation in AWW shale oil generation in MW peat generation in MW geothermal generation in MW hydrol generation in MW
generation fossil brown coal/lignite generation fossil coal-derived gas generation fossil gas generation fossil hard coal generation fossil oil generation fossil oil shale generation fossil peat generation geothermal generation hydro pumped storage aggregated generation hydro pumped storage consumption generation hydro run-of-river and poundage generation hydro water reservoir generation marine generation nuclear generation other
hydro2 generation in M W hydro3 generation in MW hydro4 generation in MW sea generation in MW
nuclear generation in MW
other generation MW
generation other renewable
other renewable generation in MW solar generation in M W
generation solar
generation waste generation wind offshore generation wind onshore forecast wind onshore day ahead forecast solar day ahead forecast wind onshore day ahead total load forecast
waste generation in MW
wind offshore generation in MW wind onshore generation in MW forecasted onshore wind generation forecasted solar generation forecasted offshore wind generation forecasted electrical demand actual electrical demand forecasted price EU R/MWh price in EU R/ R/MWh4 h4
total load actual
price day ahead
price actual
12

---

Under review as a conference paper at ICLR 2023
Table 6: Description of weather feature dataset
PART
DESCRIPTION
dt iso
datetime index localized to CET
city _ame
name of city
temp
temperature in K
temp_min min
minimum in K maximum in K pressure hPa humidity in % wind speed m/s wind direction rain in last hour in mm rain last 3 hours in mmm snow last hours mom cloud cover in % Code used to describe weather Short description of current weather Long description of current weather Weather icon code for website
temp_max
pressure
humidity
wind speed
wind deg
rain_ 1h
rain_3h
snow_3h
clouds_all
weather-id
weather main
weather description
weather_icon
A.3 WATER STAGE PREDICTION
Water stage prediction: we downloaded dataset in the real world including the information of water stage, the height of gate opening, the amount of water flowing through the gate, the amount of pumped and rainfall from January 1, 2010, to December 31, 2020. Water stage is the target variable while other variables are covariates. Rainfall information, gate position, and pump control are prior known covariates. Water stage [-1.25, 4.05) feet in this dataset.
Table 7: Description of water stage dataset
PART
DESCRIPTION
WS SI
Water Stage at Station in ft
WS S4
Water Stage at Station 4 in ft
FLOW S25A
The amount of water flowing Station 25A in m
GATE $25A
The height of gate opening at Station 25A
HWS_S25A
Head water stage at Station 25A in ft
TWS_S25A
Tail water stage Station 25A in
FLOW S25B
The amount of water flowing Station 25B in m²/
GATE $25B
The height of gatel opening Station 25B m
GATE_S25B2
The height of gate2 opening Station 25B m
JWSS$25B
Head water stage at Station 25B in ft
TWS S25B
Tail water stage Station 25B in ft
PUMP_S2 S22BB
The amount of pumped water at Station 25B in m
FLOW _S2
The amount of water flowing Station 26 m
GATE_S26.1
The height of gatel opening Station 26
GATE_S26.2
The height of gate2 opening at Station 26 m
S2SB
Head water stage Station 26 in ft
TWS S25B
Tail water stage at Station 26 in ft
PUMP -26
The amount of pumped water at Station 26 in m
MEAN RAIN
Mean value of rainfall of radar rainfall in inch
13

---

Under review as a conference paper at ICLR 2023
B APPENDIX: TRAINING AND EVALUATION
For each of datasets, we used the first 80% as training set to train the model and the last 20% as test evaluate the performance. During the training phase, common skills such normalization, dropout, regularization were used to avoid overfitting. Mean Absolute Error (MAE) and Root Square Mean Error (RSME) are the measurement evaluate models.
B.1 TRAINING DETAILS
Table 8: Training settings for Beijing PM2.5 dataset using RNN and CNN models
Lr
Decay rate
Batch size
Epoch
L1
L2
Early Stopping
Shifting (RNN)
le-4
0.99
512
8000
le-5
le-4
patience=1000
Padding (RNN)
le-4
512
1500
Shifting (CNN)
le-4
0.90
512
8000
le-4
1e-3
patience=500
Padding (CNN)
le-4
512
1000
Table 9: Training settings for Energy (electricity) price dataset using RNN and CNN models
Lr
Decay rate
Batch size
Epoch
L11
1.2
Early Stopping
Shifting (RNN)
le-4
0.99
512
8000
le-5
1e-4
patience=1000 1000
Padding (RNN)
le-4
0.90
512
8000
le-3
1e-2
patience=1000 1000
Shifting (CNN)
le-5
0.90
512
8000
le-4
le-3
patience=500
Padding (CNN)
le-5
0.90
512
8000
le-4
1e-3
patience= 1000
Table 10: Training settings for Water Stage dataset using RNN and CNN models
Decay rate
Batch size
Epoch
L2
Early Stopping
Shifting (RNN)
le-4
512
3000
Padding (RNN)
le-4
512
2000
Shifting (CNN)
lc-4
512
2000
patience=500 patience= 1000
Padding (CNN)
le-4
512
3000
B.2
EVALUATION DETAILS
After models have been trained with the training data (first 80% of the entire dataset), we test the trained models with the (last 20% of the entire dataset). Mean Absolute Error (MAE) and Root Square Mean Error (RSME) are the measurement to evaluate models.
ΣN
15 Vil
MAE
==
(6)
N
MN (1: y:)²
RMSE
N
where is the number of samples in the test set, ÿi is the predicted value of model, y: is the actual value in the test
14

---

Under review as a conference paper at ICLR 2023
C APPENDIX: SHIFTING STRATEGY
For the shifting strategy shown Figure to predict the target variables in future time steps, we shifted the future covariates in the same horizon the past by exact & steps to incorporate enough feature representations of future covariates. That because the future covariates from time t + t k have much more influence on those target variables in the same horizon that we want to predict. However, in the real world, we can get the future covariates with the longer time range (2 k). Therefore, we also tried different shifting lengths using CNNs to explore the relationship between the shifting length, the past length, and forecasting length, & and possible periodicity of dataset, p. Here we used water stage dataset since water stage is influenced by tide whose periodicity roughly 12 hours. The length of past data is still 3 days (i.e., 72 hours). The corresponding MAES and RMSEs are listed Tables and 12 and visualized in Figure 12
C.1 <
If k, it indicated the shorter time range of future covariates shifted the past, which cause covariates from & + s to f + in future k time steps are missed. However, we know those future covariates from time + + have much influence on the target variables in the same horizon to predict. This missing covariates will result in the higher MAEs and RMSEs shown in the left part of each subplot where (Figure 12
w
Future predictable Covariates
Covariates that cannot be predicted
Target time series
shift left
time steps
(s < k)
w
t + &
past
future
w
k
8
Shifted future predictable Covariates
Future predictable Covariates Covariates that cannot be predicted
Target time series
k
Input: past & shifted covariates
Output: future
Figure 9: Input data transformed by shifting connection k) for future covariates. Left: Orig- inal heterogeneous inputs and output. Right: Shifted heterogeneous inputs and output. Dot line represents prior known future covariates.
15

---

Under review as a conference paper at ICLR 2023
C.2 <=
If w < 8 < w + k, it means a few of future covariate from time + 1 to + w are also missed because they are shifted too many steps the past such that exceed the window vision of past data. They would be totally out of the vision window of the past data if 8 = w+ This missing covariates will result in the higher MAEs and RMSEs shown in the right part each subplot where w + & (Figure
w < <=
w
Future predictable Covariates
Covariates that cannot be predicted
Target time series
t
t+ 1 w + k
past
future
w
Out of window vision
shift left time steps
<= <=<=
t
Input: past & shifted covariates
Output: future
Figure 10: Input data transformed by shifting connection (w < <= + k) for future covariates.
Left: Original heterogeneous inputs and output. Right: Shifted heterogeneous inputs and output. Dot line represents prior known future covariates.
16

---

Under review as a conference paper at ICLR 2023
C.3 <<=s<=E
If the shifting length is between the forecasting length and the length of past data (k <- 8 < w), MAE and RMSE lower and fluctuating time steps are incorporated in the window of the past data. We guess the fluctuation is caused by periodicity the dataset since there are local minimum errors the shifting length is approximately equal to the periodicity plus the forecasting length.
small range since all covariates from future
w
Future predictable Covariates
Covariates that cannot predicted
Target time series
shift left time steps
(k <= <= w
past
future
Shifted future predictable Covariates
Future predictable Covariates Covariates that cannot be predicted
Target time series
w
+
Input: past & shifted covariates
Output: future
Figure 11: Input data transformed by shifting connection (k: <= 8 <= w) for future covariates.
Left: Original heterogeneous inputs and output. Right: Shifted heterogeneous inputs and output. Dot line represents prior known future covariates.
17

---

Under review as a conference paper at ICLR 2023
C.4 VISUALIZATION OF MAE & RMSE WITH DIFFERENT SHIFTING LENGTHS
With the Water Stage dataset, we tried different shifting lengths from 1 to w + k for certain forecasting length.
MAE for 6-hrs-ahead prediction
RMSE for 6-hrs-ahead prediction
0.09
MAE (ft)
RMSE (R)
0.1
0.07
0.09
0.08
0.05
0.06
12
36
48
60
36
48
60
72
78
Shifting length
Shifting length
12-hrs-ahead prediction
RMSE
prediction
0.12
0.16
RMSE (f))
MAE
0.08
0.1
VVV
0.04
0.06
12
60
72
12
24
60
72
Shifting length
Shifting length
MAE 18-hrs-ahead prediction
RMSE for 18-hrs-ahead prediction
0.13
0.17
0.15
RMSE (R)
0.13
MAE
0.09
0.13
Wrm
0.05
0.07
Shifting length
Shifting length
MAC for 24-hrs-ahead prediction
RMSE for 24-hrs-ahead prediction
0.14
0.18
MAE (ft)
0.17
RASE (Rt)
016
0.1
0.1
MMM
0.06
Shifting length
Shifting length
Figure 12: MAE RMSE for different forecasting lengths (k) with different shifting lengths (s).
18

---

Under review as a conference paper at ICLR 2023
Table 11: MAE & RMSE for water stage dataset with different shifting lengths.
k=6 hrs
12 hrs
k= 18 hrs
k=24 hrs
Shifting length
MAE
RMSE
MAE
RMSE
MAE
RMSE
MAE
RMSE
0.0841
0.1150
0.1099
0.1478
0.1226
0.1646
0.1360
0. 0.1799
8
2
0.0794
0.1088
0.1000
0.1358
0.1195
0.1608
0.1348
0.1808
8 3
0.0683
0.0952
0.0999
0.1339
0.1066
0.1458
0.1304
0.1770
8 4
0,0594
0.0834
0.0857
0.1177
0.1046
0.1444
0.1228
0.1656
$ 5
0.0612
0.0837
0.0775
0.1084
0.0978
0.1344
0.1125
0.1527
8 6 8 7
0.0572
0.0777
0.0766
0.1068
0.0922
0.1271
0.1129
0.1536
0.0553
0.0751
0.0763
0.1030
0.0919
0.1268
0.1073
0.1465
8 8
0.0531
0.0736
0.0687
0.0936
0.0901
0.1240
0.1030
0.1423
8 = 9
0.0520
0.0714
0.0633
0.0876
0.0844
0.1171
0.1026
0.1397
10
0.0563
0.0755
0.0613
0.0840
0.0786
0.1090
0.0943
0.1304
8 = 1T
0.0555
0.0771
0.0597
0.0820
0.0798
0.1095
0.0931
0. 0.14
12
0.0569
0.0775
0.0581
0.0791
0.0700
0.1001
0.0914
0.
8 = 13
0.0537
0.0729
0.0597
0.0784
0.0728
0.1007
0.0836
0.1174
14
0.0520
0.0771
0.0573
0.0785
0.0608
0.0871
0.0892
0.1227
8 15
0.0534
0.0737
0.0536
0.0738
0.0692
0.0939
0.0809
0.1133
8 16
0.0563
0.0762
0.0574
0.0779
0.0608
0.0853
0.0792
0.10
8 = 17
0.0539
0.0737
0.0588
0.0790
0.0605
0.0848
0.0718
0.1003
$ 18
0.0563
0.0779
0.0532
0.0737
0.0566
0.0778
0.0693
0.0975
8 19
0.0595
0.0812
0.0573
0.0784
0.0638
0.0871
0.0690
0.0948
8 20
0.0599
0.0808
0.0608
0.0822
0.0633
0.0869
0.0693
0.0965
8 21
0.0526
0.0727
0.0565
0.0775
0.0626
0.0859
0.0662
0.0912
8 22
0.0535
0.0753
0.0521
0.0719
0.0613
0.0833
0.0654
0.0896
8 23
0.0516
0.0720
0.0576
0.0785
0.0621
0.0844
0.0655
0.0901
$ 24
0.0527
0.0724
0.0585
0.0807
0.0589
0.0812
0.0617
0.0841
25
0.0518
0.0725
0.0590
0.0803
0.0605
0.0834
0.0638
0.0881
8 26
0.0569
0.0784
0.0595
0.0813
0.0643
0.0877
0.0628
0.0875
8 27
0.0532
0.0732
0.0582
0.0804
0.0591
0.0823
0.0625
0.0871
$ = 28
0,0510
0.0699
0.0530
0.0725
0.0610
0.0846
0.0698
0.0946
$ 29
0.0530
0.0735
0.0559
0.0770
0.0623
0.0855
0.0668
0.0921
$ 30
0,0565
0.0776
0.0573
0.0781
0.0598
0,0830
0.0634
0,0880
$
0.0535
0.0737
0.0570
0.0778
0.0609
0.0840
0.0656
0.0901
8 32
0.0548
0.0753
0.0573
0.0785
0.0563
0.0769
0.0618
0.0866
8 33
0.0586
0.0779
0.0570
0.0781
0.0607
0.0839
0.0636
0.0875
8 34
0.0556
0.0763
0.0571
0.0776
0.0617
0.0847
0.0670
0.0906
$ 35
0.0557
0.0765
0.0598
0.0808
0.0620
0.0844
0.0604
0.0832
36
0.0560
0.0758
0.0561
0.0765
0.0668
0.0902
0.0638
0.0856
S 37
0.0549
0.0758
0.0584
0.0802
0.0600
0.0831
0.0649
0.0914
38
0.0545
0.0746
0.0643
0.0891
0.0614
0.0838
0.0626
0.0871
8 39
0.0533
0.0736
0.0537
0.0740
0.0599
0.0819
0.0646
0.0889
8 40
0.0523
0.0717
0.0607
0.0830
0.0592
0.0815
0.0675
0.0916
8 41
0.0555
0.0750
0.0518
0.0716
0.0635
0.0860
0.0611
0.0840
$ 42
0.0567
0.0765
0.0575
0.0777
0.0667
0.0908
0.0614
0.0837
$ = 43
0.0543
0.0747
0.0579
0.0788
0.0587
0.0812
0.0624
0.0860
8 =
0.0534
0.0727
0.0559
0.0769
0.0559
0.0780
0.0625
0.0863
8 45
0.0576
0.0785
0.0596
0.0816
0.0660
0.0900
0.0627
0.0870
8 46
0.0523
0.0727
0.0596
0.0817
0.0607
0.0833
0.0658
0.0894
8 47
0.0593
0.0804
0.0536
0.0742
0.0598
0.0820
0.0694
0.0943
8 48
0.0583
0.0788
0.0524
0.0723
0.0628
0.0861
0.0593
0.0816
49
0.0548
0.0754
0.0570
0.0777
0.0636
0.0880
0.0664
0.0901
3 50
0.0594
0.0806
0.0553
0.0751
0.0591
0.0811
0.0609
0.0845
19

---

Under review as a conference paper at ICLR 2023
Table 12: MAE & RMSE for water stage dataset with different shifting lengths.
k=6 hrs
k=12 hrs
k=18 hrs
k=24 hrs
Shifting length
MAE
RMSE
MAE
RMSE
MAE
RMSE
MAE
RMSE
51
0.0589
0.0797
0.0573
0.0781
0.0570
0.0793
0.0669
0.0922
8 52
0.0531
0.0727
0.0539
0.0736
0.0619
0.0838
0.0640
0.0863
8 53
0.0587
0.0800
0.0571
0.0775
0.0575
0.0795
0.0654
0.0893
5 = 54
0,0569
0.0771
0.0551
0.0748
0.0627
0.0849
0.0654
0,0880
$ 55
0.0584
0.0790
0.0585
0.0799
0.0645
0.0890
0.0632
0.0869
8 56
0.0568
0.0781
0.0589
0.0792
0.0668
0.0902
0.0642
0.0879
8 57
0.0553
0.0750
0.0622
0.0844
0.0644
0.0881
0.0719
0.0957
8 58
0.0529
0.0723
0.0609
0.0822
0.0653
0.0887
0.0673
0.0904
8 = 59
0.0596
0.0805
0.0580
0.0791
0.0620
0.0853
0.0663
0.0913
60
0.0556
0.0756
0.0596
0.0817
0.0606
0,0836
0.0703
0.0955
8 61
0.0536
0.0733
0.0546
0.0756
0.0596
0.0819
0.0675
0.0924
62
0.0604
0.0819
0.0578
0.0796
0.0641
0.0870
0.0632
0.0865
8 63
0.0571
0.0778
0.0562
0.0761
0.0639
0.0878
0.0673
0.0907
$ 64
0.0584
0.0784
0.0651
0.0874
0.0628
0.0857
0.0713
0.0989
8 65
0,0616
0.0826
0.0583
0.0795
0.0658
0.0894
0.0768
0.1025
8 69
0.0578
0.0770
0.0595
0.0803
0.0613
0.0833
0.0659
0.0901
8 67
0.0596
0.0803
0.0587
0.0794
0.0731
0.0974
0.0670
0.0920
$ 68
0.0607
0.0828
0.0592
0.0788
0.0661
0.0892
0.0768
0.1066
8 69
0.0565
0.0766
0.0624
0.0837
0.0642
0.0875
0.0667
0.0916
8 70
0.0578
0.0787
0.0623
0.0841
0.0654
0.0899
0.0711
0.0952
8 71
0.0553
0.0754
0.0649
0.0874
0.0726
0.0987
0.0682
0.0935
8 72
0.0579
0.0779
0.0615
0.0839
0.0666
0.0910
0.0714
0.0974
8 73
0.0607
0.0817
0.0606
0.0828
0.0626
0.0848
0.0690
0.0927
8 74
0.0591
0.0809
0.0664
0.0896
0.0706
0.0971
0.0700
0.0969
75
0.0607
0.0821
0.0610
0.0832
0.0644
0.0884
0.0727
0.0993
8 = 76
0,0653
0.0887
0.0641
0.0873
0.0663
0.0884
0.0791
O. 1060
8 77
0.0636
0.0885
0.0637
0.0853
0.0682
0.0923
0.0684
0.0931
S = 78
0.0662
0.0895
0.0663
0.0895
0.0808
0.1103
0.0675
0.0913
$ 79
0.0665
0.0913
0.0686
0.0923
0.0679
0.0912
0.0739
0.0981
8 SO
0,0679
0.0926
0.0678
0.0922
0.0693
0.0935
0.0750
0.1023
$
0.0690
0.0946
0.0699
0.0948
0.0704
0.0953
0.0774
0.1042
$ 82
0.0717
0.0977
0.0735
0.0997
0.0725
0.0980
0.0807
0.1092
8 83
0.0755
0.1024
0.0730
0.0990
0.0802
0.1060
8 84
0.0763
0.1028
0.0760
0.1030
0.0748
0.1011
$ = 85
0.0767
0.1040
0.0740
0.1000
0.0780
0.1055
86
0.0826
0.1111
0.0800
0.1080
0.0760
0.1024
S = 87
0.0821
0.1117
0.0810
0.1100
0.0768
0.1050
8 88
0.0831
0.1126
0.0840
0.1110
0.0850
0.1137
8 89
0.0830
0.1120
0.0825
0.1117
8 90
0.0870
0.1170
0.0836
0.1118
8 91
0.0935
0.1242
0.0834
0.1114
$ 92
0.0896
0.1194
0.0868
0.1155
$ = 93
0.0932
0.1233
0.0868
0.1156
8 94
0.0925
0.1229
0.0941
0.1247
8 95
0.0923
0.1231
8 96
0.0970
0.1289
97
0.1021
0.1354
8 98
0,0982
0.1302
99
0.0976
0.
$ 100
0.1004
0. 131338
20


Reviews:
[{"review_id": "gTTtTczsAD", "reviewer": "Reviewer_nHrP", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes two approaches to taking advantage of future values in a time series for prediction (these future values ares ones that can reliably be predicted from data up until the current time), shifting and padding. The paper shows how to use these two approaches with RNNs and CNNs and demonstrates their effectiveness on real data.", "summary_of_the_review": "Overall, I think that this paper is not ready for publication. I encourage the authors to better justify their proposed methods from a theoretical standpoint, and to make their experimental results much more rigorous (very clearly stating how hyperparameter tuning is done so that there is no doubt that the experiments are conducted correctly, adding error bars, adding more datasets).", "strength_and_weaknesses": "Strengths:\n1. The paper is well-explained.\n2. The proposed methods *shifting* and *padding* are easy to understand in terms of how they are computed. The diagrams in particular are very helpful.\n\nWeaknesses:\n1. The two strategies proposed are very straightforward and do not constitute major technical advances.\n2. While I understood how *shifting* and *padding* mechanically work (i.e., I understand how they are computed), I think that the authors should provide more theoretical justification for when and why we expect them to work. For example, in the case of *shifting*, it would be great if more theoretical intuition could be provided for why precisely predicted future values are helpful. Specifically, it seems like if the claim is that all the data up until the current time $t$ can very reliably predict the next $k$ future values for a few specific covariates, then doesn't it mean that the data up until the current time $t$ already contain the information that is in these particular $k$ predicted values, so actually predicting them and feeding the predicted values explicitly as inputs would somehow seem, from a theoretical standpoint, unnecessary? As a concrete example, if the next $k$ predicted values are just represented as a continuation of a line fitted to the most recent $k$ values up to time $t$, then it is unclear what the advantage is of explicitly computing the predicted $k$ values if you already have fitted a line to the most recent $k$ values. Using standard stochastic process terminology, basically I think this could be phrased in terms of the filtration up until time $t$ and that for a few covariates, the future values of these covariates are predictable processes (conditioned on the filtration up until time $t$). Using this sort of standard formalism, providing a clear argument for why shifting definitely helps would be helpful rather than only giving a seemingly heuristic justification. Separately, I think *padding* makes a much larger assumption and one can easily cook up examples where it will fail catastrophically...\n3. For the training, there isn't much of any detail provided regarding how hyperparameters are tuned, which leaves me to find the results obtained to be suspicious. For example, there is no mention of a validation set (that is separate from the held out test set) that is used to help tune hyperparameters. Were hyperparameters then just tuned where one directly gets to see the test set performance (this would constitute extremely poor experimental design)? Assuming that this is not the case, then there should be a lot more details provided for how hyperparameters were selected without ever looking at the test set.\n4. There are no error bars reported for Tables 1 & 2. I'd suggest adding error bars to give readers a sense of the variability of the models evaluated (for instance, you could re-run the experiment using 10 different random seeds and instead report mean/std dev of each metric across the 10 different random seeds). Without these error bars, it's hard to know just how much better one method is doing vs another. Is it just due to the random seed? If we re-run the models, is it the case that the ones currently bolded are always consistently better than the others?\n5. Given how simple the proposed strategies are, I would prefer seeing a much more extensive evaluation (with many more standard publicly available datasets) so that we can better understand when they work and when they don't (i.e., when they don't provide an advantage over baseline(s)). Especially given the comments I mentioned in weakness point #2, I would be concerned that the datasets currently in the paper are too cherry-picked.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "dSLrpvpoU1e", "reviewer": "Reviewer_ZRck", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Two novel feature forecasting strategies – shifting and padding, are proposed to avoid any accumulation of prediction errors while contextually linking the past with the predicted future. These strategies allow to better deal with predictable future covariates. For example, the problem of predicting water levels at a given location in a river or canal system using historical data and future covariates, some of which (precipitation, tide) may be predictable. Experiments performed on three datasets show the proposed strategies outperform existing methods and suggest a relationship between the amount of shifting and padding and the periodicity of the time series.\n", "summary_of_the_review": "While the authors propose two new forecasting strategies with good forecast accuracy compared to baselines, the methodological contributions are limited. Furthermore, the proposed methods do not have a (strong) mathematical foundation. The experiments are limited to three (small) datasets. On the positive side, the paper is well-written and well-presented.\n", "strength_and_weaknesses": "Strengths\n\t- The paper is well-written and clear.\n\t- New strategies for multi-horizon forecasting are needed to avoid the accumulation of prediction errors over the forecast horizon.\n\t- Experimental results show better forecast accuracy compared to baselines.\n\nWeaknesses\n\t- The proposed forecasting strategies are essentially heuristics, without strong mathematical foundations.\n\t- Given the wide variety of existing forecasting strategies, the contribution to the state-of-the-art is rather limited. Also, the methodological contributions of the paper are limited.\n\t- The experiments are only based on three real-world datasets. There are many other forecasting benchmarks (M5, etc) that are more relevant to the forecasting community.\n\n\nComments\n- I suggest citing the paper \"Forecasting: theory and practice\" (https://arxiv.org/pdf/2012.03854.pdf) which discusses forecasting strategies.\n- When reading the results in Tables 1-3, I am not convinced that your method \"outperforms\" other methods, especially seq_to_seq. The differences are rather small.  Tests of significance are needed here.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "EdWNwdp7cUP", "reviewer": "Reviewer_S4dZ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposed two strategies to use future covariates by 1) simply shifting back the covariates with prediction by length $s$ (which is a hyperparameter) and take the shifted covariates as additional input to the model; or 2) for the target and unpredicted covariates, copying the last $s$ observations to pad them into the end of original time series and then take them as input to the model. The experiments on three datasets showed better forecasting accuracy compared to 4 common deep learning baselines.", "summary_of_the_review": "I have concerns about the motivation and the empirical results. Please refer to more detail in the weakness section.", "strength_and_weaknesses": "On the strength, the work studies a less explored area of how to best use the future covariates in a forecasting model, which I think is an important question.\n\nOn the weakness, I have two major concerns. The motivation of the proposed strategies is not well justified and I see some potential problems. The shifting may mess up the interaction between the covariates and target, as the time indexes are mismatched after the shifting. The padding may break the smoothness of the time series. Also, if it is a mere concatenation of past (the encoder output) and future covariates, the decoder could learn the interaction between them. For the covariates that can not be predicted well (or not observed), and if there is relevant information in them, a more principled way for me will be a multivariate method.\n\nThe other concern is lacking detail in the baselines and experiment set up. What is the baseline strategy? Some results for concatenating the covariates with the output of the encoder? Some of the baselines can also use covariates, such as DeepAR. In the experiments, did the authors try to use the covariates, at least the ones that can be predicted accurately, as input for DeepAR? Have the baselines been trained properly? Also, how did the authors generate future covariates? Do you take them from the data or do you actually predict the future covariates? In the end, these deep learning models have quite some variance in the training and some replicates are needed to have robust results. In the end, how to show the effect of the strategy and separating the influence of the model choices? \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject