## 4 Experiments
### Performance Along Different Forecasting Horizons  
How does the forecasting error change when increasing the number of steps to forecast? Are there forecasting methods which are more robust than others to this process? In this subsection we give answers to these questions by exploring to what extent the forecasting errors vary along different horizons. Namely, we will make forecasts at 3, 6, 12, 24, and 48 steps forward. We omit in this case the naive method which for how its nature works is used only for a 24 step forecasting scenario. Clearly, when trying to predict long term dependencies by moving from a few hours up to two days (48 steps) in the future, we expect to make larger and larger errors. The plot in Figure 14 explains this concept pretty well. It has been generated while performing multi-step forecasting using ARIMA with walk-forward validation on a single cell time series. Each boxplot contains the distribution of the mean absolute error for the corresponding step of forecasting from one to 24 steps. The same plot for the case of MLP neural network is shown in Figure 15. Although we do not use the mean absolute error as a performance measure in our study, it come in handy in this case to illustrate the propagation of the forecasting error. In Figure 16 we show some first graphical results of this process regarding the three performance measures for ARIMA method. In the same way, Figure 17 shows the results for CNN-LSTM hybrid model. Figure 14: The distribution of absolute error for ARIMA in multi-step forecasting scenario. As it is natural to see, the absolute error increases with the number of steps forward to forecast. Figure 15: The distribution of absolute error for MLP neural network in multi-step forecasting scenario. The absolute error increases naturally with the number of steps forward to forecast. Finally, in order to summarize the results for all the forecasting methods, we build three more tables as follows. In these tables we show the mean for each performance measure computed across all the cells of the grid. In general, for all the forecasting methods, there is a steady increase in the corresponding forecasting measure of performance when going from a few steps forecast to more steps in the future. Judging from the tables, there are two top-performing methods: the CNN-LSTM hybrid model and ARIMA method. They both show to perform well for MAPE and RMSE given in Tables 2 and 3 respectively. However, as Table 4 shows, the CNN-LSTM model outperforms ARIMA for MAXAE and does a better job in keeping this type of error low in the long term. \begin{table}
\begin{tabular}{l c c c c c} \hline \hline \multirow{2}{*}{**Methods**} & \multicolumn{5}{c}{**RMSE**} \\ \cline{2-6}  & **3 Steps** & **6 Steps** & **12 Steps** & **24 Steps** & **48 Steps** \\ \hline
**LSTM**. & 4.78 & 5.95 & 19.6 & 32.34 & 121.95 \\ \hline
**ETS** & 17.38 & 22.29 & 25.5 & 28.79 & 27.29 \\ \hline
**AR** & 4.29 & 8.26 & 16.5 & 19.72 & 21.34 \\ \hline
**CNN** & 6.15 & 8.37 & 10.22 & 11.99 & 14.23 \\ \hline
**MLP** & 6.35 & 7.68 & 9.48 & 11.63 & 14.06 \\ \hline
**Prophet** & 3.17 & 6.87 & 7.69 & 8.74 & 11.8 \\ \hline
**ARIMA** & 5.65 & 6.26 & 6.66 & 7.44 & 9.21 \\ \hline
**CNN-LSTM** & 6.45 & 6.41 & 5.93 & 6.35 & 7.8 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: The table shows how varies the mean of MAPE error across the cells for each method when transitioning from three steps forecasting up to 48 steps. Both statistical and deep learning methods show good results for 3–6 steps forecast. The deep learning methods such as CNN-LSTM show to be robust by keeping the MAPE low and almost constant in the long term.